Google Inc.

Patent Trials and Appeals Board

Nov 4, 2020

2019002695 (P.T.A.B. Nov. 4, 2020)

UNITED STATES PATENT AND TRADEMARK OFFICE
UNITED STATES DEPARTMENT OF COMMERCE
United States Patent and Trademark Office
Address: COMMISSIONER FOR PATENTS
P.O. Box 1450
Alexandria, Virginia 22313-1450
www.uspto.gov
APPLICATION NO. FILING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO.
15/145,362 05/03/2016 Johnny Chung Lee 1500-G15004-US 6555
125070 7590 11/04/2020
Google LLC
c/o Davidson Sheehan LLP
6836 Austin Center Blvd.
Suite 320
Austin, TX 78731
EXAMINER
SHENG, XIN
ART UNIT PAPER NUMBER
2611
NOTIFICATION DATE DELIVERY MODE
11/04/2020 ELECTRONIC
Please find below and/or attached an Office communication concerning this application or proceeding.
The time period for reply, if any, is set in the attached communication.
Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the
following e-mail address(es):
Google@ds-patent.com
docketing@ds-patent.com
PTOL-90A (Rev. 04/07)
UNITED STATES PATENT AND TRADEMARK OFFICE

BEFORE THE PATENT TRIAL AND APPEAL BOARD

Ex parte JOHNNY CHUNG LEE and STEVEN B. GOLDBERG

Appeal 2019-002695
Application 15/145,362
Technology Center 2600

Before JOSEPH L. DIXON, DAVID M. KOHUT, and
JON M. JURGOVAN, Administrative Patent Judges.

KOHUT, Administrative Patent Judge.

DECISION ON APPEAL
Pursuant to 35 U.S.C. § 134(a), Appellant1 appeals from the
Examiner’s decision to reject claims 1–26.2 We have jurisdiction under
35 U.S.C. § 6(b).
We AFFIRM.

1 We use the word “Appellant” to refer to “applicant” as defined in
37 C.F.R. § 1.42. Appellant identifies the real party in interest as Google
LLC. Appeal Br. 1.
2 Throughout this Decision we refer to the Final Rejection mailed April 19,
2018 (“Final Act.”), the Appeal Brief filed October 17, 2018 (“Appeal Br.”),
the Examiner’s Answer mailed December 27, 2018 (“Ans.”), and the Reply
Brief filed February 8, 2019 (“Reply Br.”).
Appeal 2019-002695
Application 15/145,362

2
INVENTION
The present invention relates to a method and apparatus for “pass-
through display of captured imagery” and “generation of AR [(augmented
reality)] overlays to be displayed in conjunction with the captured imagery.”
Spec. ¶¶ 26, 39; Title (capitalization altered). Claim 1 is representative of
the invention and is reproduced below.
1. A method comprising:
sequentially outputting from a first imaging sensor each
pixel row of a first set of pixel rows of a first image captured by
the first imaging sensor;
buffering a subset of the first set of pixel rows in a buffer,
the subset including a first pixel row of the first set of pixel
rows;
modifying the subset of pixel rows in the buffer based on
augmented reality overlay information associated with a
position of the first pixel row to generate a pixel row
representative of the first pixel row of the first image; and
displaying, at a display device, a pixel row representative
of the first pixel row of the first image prior to a second pixel
row of the first image.
Appeal Br. 13–17 (Claims App.).
REFERENCES
The prior art relied upon by the Examiner is:
Appeal 2019-002695
Application 15/145,362

3
Name Reference Date
De Groot US 2013/0314546 A1 Nov. 28, 2013
Lee US 2014/0240469 A1 Aug. 28, 2014
Margolis et al. US 2014/0375679 A1 Dec. 25, 2014
Williams et al. US 2015/0029218 A1 Jan. 29, 2015

REJECTIONS
Claims 1, 6–9, 11, and 16–19 are rejected under 35 U.S.C. § 103 as
being unpatentable over De Groot in view of Margolis. Final Act. 2–8.
Claims 2, 3, 6, 12, 13, 16, 21, 23,3 24, and 25 are rejected under
35 U.S.C. § 103 as being unpatentable over De Groot, Margolis, and Lee.
Final Act. 8–15.
Claims 4, 5, 10, 14, 15, 20, 22, and 264 are rejected under
35 U.S.C. § 103 as being unpatentable over De Groot, Margolis, Lee, and
Williams. Final Act. 15–20.

OPINION
§ 103 Rejections of Claims 1–8, 10–18, and 20–26
Appellant argues the combination of De Groot and Margolis does not
teach:

3 Although claim 22 is listed in the Examiner’s rejection statement (see Final
Act. 8), claim 23 is actually rejected under De Groot, Margolis, and Lee (see
Final Act. 12–13 reproducing the language of claim 23), while claim 22 is
rejected under De Groot, Margolis, Lee, and Williams (see Final Act. 15,
19). We make this change here and find this to be harmless error.
4 Although claim 24 is listed in the Examiner’s rejection statement (see Final
Act. 15), claim 26 is actually rejected under De Groot, Margolis, Lee, and
Williams (see Final Act. 20 reproducing the language of claim 26), while
claim 24 is rejected under De Groot, Margolis, and Lee (see Final Act. 8,
14). We make this change here and find this to be harmless error.
Appeal 2019-002695
Application 15/145,362

4
“sequentially outputting from . . . [an] imaging sensor each
pixel row of a first set of pixel rows of a first image captured
by the . . . imaging sensor,” “buffering a subset of the first set
of pixel rows in a buffer,” “modifying the subset of pixel rows
in the buffer based on augmented reality overlay information,”
and “displaying . . . a pixel row representative of the first pixel
row of the first image prior to a second pixel row of the first
image,”
as recited in independent claim 1, and similar limitations recited in
independent claims 11 and 23. Appeal Br. 4–5, 7, 9–10; Reply Br. 2–7.
With respect to De Groot, Appellant argues De Groot does not teach or
suggest a “‘prior to’ and ‘before’ temporal relationship” between “reading
out of an ‘image sensor’ and actual display of pixels,” as recited in the
independent claims. Reply Br. 3, 6. More particularly, Appellant argues De
Groot does not teach “the language of claims 1, 11, and 23 directed to
displaying one or more rows of pixels before outputting subsequent rows
from an imaging sensor” and “displaying the modified pixels prior to
capturing further pixels by the image sensor.” Reply Br. 2, 7. Appellant
further argues De Groot’s Figure 5 does not teach the claimed “prior to” and
“before” temporal relationships because
the time period SS1 corresponding to reading of a first image [in
De Groot’s Figure 5] is continuous and reading out of a sensor
continues without regard for displaying happening at D1. . . .
First, as seen on the left side of FIG. 5, in order to reach the
language of claims, 1, 11, and 23 with respect to “prior to” and
“before” reading another pixel row, this timing diagram of
DeGroot would need to not allow readout S1 to proceed until D1
was complete. Such is not illustrated in FIG. 5 and reading out
continues without regard for displaying as the readout lines are
continuously advancing to the right and downward. To reach the
language of claims 1, 11, and 23, the readout and display would
need to proceed stepwise at S, P, and D through the entire image
frame understood by “1” as the image is sequentially read out of
Appeal 2019-002695
Application 15/145,362

5
the sensor. Instead of stepwise, DeGroot describes and illustrates
in FIG. 5 a continuous process for S, P and D. Second, as seen
on the bottom right side of FIG. 5, D1 continues past readout S2.
That is, the right side of FIG. 5 illustrates that readout S2 is
operating without regard to display of rows of pixels of D1. Thus,
not only is display Dx for any row operating independently of its
corresponding readout Sx, the second readout S2 did not wait for
the first display D1 to complete.

Reply Br. 6–7.
With respect to Margolis, Appellant argues Margolis stores entire
images in a camera buffer, describing “a mechanism for an entire image
instead of a row of pixels,” as required by claim 1. Appeal Br. 5–6.
Appellant asserts paragraphs 74 and 75 of Margolis do not describe “the
‘row operations’ of claim 1 with respect to directly passing rows from
sensor to display” including “buffering a subset of the first set of pixel rows
in a buffer,” “modifying the subset of pixel rows in the buffer based on
augmented reality overlay information,” and “displaying . . . a pixel row
representative of the first pixel row of the . . . image prior to a second pixel
row.” Id. at 7.
We do not agree with Appellant’s arguments. Instead, we agree with
the Examiner’s findings. Final Act. 2–3, 5–6, 12–13; Ans. 4–12, 14–24.
Particularly, we agree with the Examiner that De Groot teaches the
“sequentially outputting” and “displaying” limitations of claim 1, and the
similar limitations recited in claims 11 and 23. Ans. 4–6, 11–12, 14–16, 19–
21 (citing De Groot ¶¶ 10, 13, 44, 62–63, Abstract); Final Act. 2, 5. In
particular, De Groot teaches claim 1’s sequentially outputting pixel rows
from an imaging sensor, and claim 11’s imaging sensor having an output to
sequentially output pixel rows of a captured image. See De Groot ¶¶ 13
Appeal 2019-002695
Application 15/145,362

6
(describing “a manner of reading out of the radiation sensitive array in
which the rows of sensor pixels are read out sequentially, i.e., one row after
each other, or a subset of rows after each other” with a “rolling shutter
circuit [that] sequentially provides subsets of pixels to the signal processor”
such that “the rows of sensor pixels are sequentially read out to provide a
sequence of subsets of pixels, with a subset of pixels being processed after
receipt by the signal processor”), 62–63. De Groot’s Fig. 5 (illustrating the
operation of the imaging device described in paragraph 13) shows sequential
output of pixel rows Rn . . . R0 from an imaging sensor: “the readout circuit
140 read[s] out the row Rn (a top row of the radiation sensitive array) at the
beginning of the time period S1 and the row R0 (a bottom row) at its end.
Consequently, during the reading S1, all rows Rn to R0 are read out
sequentially.” See De Groot ¶ 48, Fig. 5.
De Groot also teaches claim 1’s sequential display of pixel rows, by
which a pixel row (representative of a first pixel row of the image) is
displayed prior to a second pixel row being displayed. Ans. 5–6; see De
Groot ¶¶ 13 (“the signal processor, upon receiving a subset of pixels,
processes the subset of pixels, and then provides a result of the processing,
i.e., a processed subset of pixels, to the display. Each processed subset of
pixels is then displayed by the display” such that “the rows of sensor pixels
are sequentially read out to provide a sequence of subsets of pixels, with a
subset of pixels being processed after receipt by the signal processor and
being displayed after receipt by the display”), 48, Fig. 5. De Groot’s Fig. 5
illustrates sequential display as recited in claim 1, by which a first pixel row
(top row Rn of the radiation sensitive array) is displayed prior to a second
pixel row (e.g., row Rn-1 and the following rows, all the way to bottom row
Appeal 2019-002695
Application 15/145,362

7
R0) being displayed. See De Groot ¶¶ 48, 54 (“D1 indicates a time period for
displaying the processed image 162. FIG. 5 shows the display 180
displaying the row Rn at the beginning of the time period D1 and the row R0
at its end.”), Fig. 5. De Groot also teaches claim 23’s display device that
displays imagery including the first pixel row (e.g., top row Rn in De Groot’s
Fig. 5) prior to a second pixel row (e.g., row Rn-1 all the way to bottom row
R0 in Fig. 5) captured by the imaging sensor. See De Groot Fig. 5; Ans. 15–
16. Finally, De Groot teaches claim 11’s display controller that begins
sequential display of pixel rows of the captured image, including the first
pixel row (e.g., top row Rn which is displayed at a time corresponding to an
intersection, in Fig. 5, of display trace D1 with the horizontal trace
representing row Rn) before a last pixel row (bottom row R0, corresponding
to a bottom horizontal trace in Fig. 5) of the captured image is output by the
imaging sensor (bottom row R0 is output by the imaging sensor at a time
corresponding to the end of reading trace S1 intersecting a horizontal trace
representing row R0). See De Groot ¶¶ 48, 52, 54, Fig. 5; Ans. 14–16, 19–
21; Final Act. 5.
We further note that Appellant’s arguments regarding De Groot’s
deficiencies are not supported by corresponding language in claims 1, 11,
and 23. For example, claims 1 and 23 do not recite or require “displaying
one or more rows of pixels before outputting subsequent rows from an
imaging sensor,” and claims 1, 11, and 23 do not recite or require
“displaying the modified pixels prior to capturing further pixels by the
image sensor” as Appellant argues (see Reply Br. 2, 7 (emphasis added)).
Claims 1, 11, and 23 also do not recite or require (i) “not allow[ing] readout
[] to proceed until [display] was complete,” (ii) “readout and display . . . to
Appeal 2019-002695
Application 15/145,362

8
proceed stepwise . . . through the entire image frame,” or (iii) a readout of a
second image to wait for display of the first image to complete, as Appellant
argues (see Reply Br. 6–7).
We are also unpersuaded by Appellant’s arguments regarding
Margolis, and agree with the Examiner that Margolis teaches the claimed
“buffering” and “modifying” limitations of claim 1. Ans. 6–8 (citing
Margolis ¶¶ 54, 57, 59, 61, 65–66, 71, 76–78, 80–81, Abstract); Final Act. 3
(citing Margolis ¶¶ 74–75). In particular, Margolis buffers a subset of a first
set of pixel rows of a camera-captured image, in a buffer—the display out
interface 328, which is a buffer for providing images from forward-facing
cameras to Margolis’ processing unit. See Margolis ¶ 54; Ans. 7 (citing
Margolis ¶¶ 54, 57); Final Act. 3. Appellant’s arguments that Margolis is
deficient because it stores entire images in the buffer and employs “a
mechanism for an entire image instead of a row of pixels” (see Appeal Br.
5–6), are not commensurate with the scope of claim 1. Claim 1 does not
require the buffer to hold less than the entire first image.
As the Examiner further finds, Margolis also teaches the claimed
“modifying” of claim 1. Ans. 6–8; Final Act. 3. In particular, Margolis
modifies a subset of pixel rows (e.g., pixel rows captured by Margolis’
forward-facing cameras 113) in the buffer based on augmented reality (AR)
overlay information (e.g., AR overlay information for generating an
augmented reality image in Margolis, such as an AR image of a frame
surrounding a food box in Margolis’ Figure 6B) associated with a position of
the first pixel row (AR overlay information in Margolis is associated with a
position of the first pixel row because the AR overlay information produces
a portion of the AR image at a real-world location identified by the first
Appeal 2019-002695
Application 15/145,362

9
pixel row captured by Margolis’ forward-facing camera) to generate a pixel
row representative (e.g., row 1 illustrated in Margolis’ Figure 9A) of the first
pixel row of the first image. See Margolis ¶¶ 41, 52, 54, 57, 76–78, Figs.
9A–9D; Ans. 6–8. Margolis generates the augmented reality overlay row-
by-row to produce a modified (AR) pixel row every row period. See
Margolis ¶ 71, Figs. 9A–9D; Ans. 7–8. Thus, contrary to Appellant’s
argument regarding Margolis’ paragraph 75 (see Appeal Br. 7), paragraph
75—referencing Figs. 9A and 9C—teaches the claimed “modifying.”
We are also not persuaded by Appellant’s arguments regarding
Margolis failing to teach claim 1’s “outputting . . . each pixel row” and
“displaying . . . a pixel row representative of the first pixel row of the . . .
image prior to a second pixel row.” Appeal Br. 6–7. Appellant’s arguments
do not address the Examiner’s findings as to the combined teachings of
Margolis and De Groot. The Examiner relies on De Groot for “outputting”
pixel rows and “displaying” pixel rows in a sequential manner, and relies on
Margolis for “buffering” pixel rows and “modifying” pixel rows based on
AR overlay information to generate pixel row representative(s) of a captured
image’s pixel row(s). Ans. 6, 8; Final Act. 2–3. Appellant’s arguments do
not persuade us of reversible error in the Examiner’s findings as to the
combined teachings of De Groot and Margolis, in which the Examiner adds
Margolis’ AR image display with row-dependent duty cycle adjustments to
reduce motion blur, to De Groot’s low latency row-by-row pixel display, “so
as to reduce motion blur when displaying AR images in real time.” Ans. 8–
9.
Accordingly, Appellant’s arguments have not persuaded us of error in
the Examiner’s rejections of independent claims 1, 11, and 23, and we
Appeal 2019-002695
Application 15/145,362

10
sustain the Examiner’s obviousness rejections of claims 1, 11, and 23. We
also sustain the Examiner’s obviousness rejections of dependent claims 2–8,
10, 12–18, 20–22, and 24–26 not separately argued. Appeal Br. 7, 10–12.

§ 103 Rejection of Claims 9 and 19
Claim 9 depends from claim 1 (via claim 8), and further recites “the
first image [from the first imaging sensor] is displayed in a first region of the
display” and “the second image [from a second imaging sensor] is displayed
in a second region of the display concurrent with the display of the first
image.” Appeal Br. 14 (Claims App’x). Appellant contends De Groot does
not teach, but rather teaches away from, displaying a second image in a
second region of the display concurrent with the display of the first image in
a first region of the display, as claimed. Reply Br. 8–10; Appeal Br. 8–9.
More particularly, Appellant argues De Groot’s acquiring two images (of “a
left-hand view and . . . a right-hand view, portions that have a same vertical
position in either image”) and combining the images “does not equate to
displaying two images as in claim 9. . . . this portion [of paragraph 57] of
DeGroot suggests a same position, not displaying a second image ‘in a
second region of the display’” as claimed. Appeal Br. 8.
We do not agree. Rather, we agree with the Examiner’s finding that
[a]lthough the left-hand view and right-hand view [of De Groot]
have the same vertical position, the horizontal position are
different for left-hand view image and right-hand view image.
There is no further description regarding limitation of “first
region” and “second region” in Claim 9 & 19. Therefore, with
the broadest reasonable interpretation, the different horizontal
positions of the left-hand and right-hand view images are
considered in a first region and a second region respectively [as
claimed].
Appeal 2019-002695
Application 15/145,362

11

Ans. 13 (citing De Groot ¶ 57, Fig. 6 (showing images 142 and 242 captured
by imaging sensors 110, 210)); see also Final Act. 5 (citing De Groot ¶¶ 57–
59). That is, under its broadest reasonable interpretation, claim 9 does not
recite or require the “first region of the display” and the “second region of
the display” to be disjoint, non-superimposing regions. As such, De Groot’s
left-hand view and right-hand view images teach and suggest the claimed
first and second images displayed in two (first and second) regions of the
display, since left-hand and right-hand views are typically not identical,
thereby occupying two display regions. Ans. 13. Additionally, De Groot’s
paragraph 62 (cited by the Examiner against claim 1, see Ans. 6, 11)
discloses “image 242 . . . inserted as a so-termed Picture-in-Picture (PiP)
into the image 142,” which also teaches a second image displayed in a
second display region concurrent with a first image displayed in a first
display region as required by claim 9. See De Groot ¶ 62.
Accordingly, Appellant’s arguments have not persuaded us of error in
the Examiner’s rejection of claim 9, and claim 19 reciting similar limitations
and argued for the same reasons. Appeal Br. 8–9. Therefore, we sustain the
Examiner’s rejection of claims 9 and 19.

CONCLUSION
The Examiner’s decision rejecting claims 1–26 under 35 U.S.C. § 103
is affirmed.
DECISION SUMMARY
In summary:
Appeal 2019-002695
Application 15/145,362

12
Claims
Rejected
35 U.S.C. § Reference(s)/Basis Affirmed Reversed
1, 6–9, 11,
16–19
103 De Groot,
Margolis
1, 6–9, 11,
16–19

2, 3, 6, 12,
13, 16, 21,
23, 24, 25
103 De Groot,
Margolis, Lee
2, 3, 6, 12,
13, 16, 21,
23, 24, 25

4, 5, 10,
14, 15, 20,
22, 26
103 De Groot,
Margolis, Lee,
Williams
4, 5, 10,
14, 15, 20,
22, 26

Overall
Outcome
1–26

TIME PERIOD FOR RESPONSE
No time period for taking any subsequent action in connection with
this appeal may be extended under 37 C.F.R. § 1.136(a). See 37 C.F.R.
§ 1.136(a)(1)(iv).
AFFIRMED