Apple Inc.

Patent Trials and Appeals Board

Jul 21, 2021

2020003108 (P.T.A.B. Jul. 21, 2021)

UNITED STATES PATENT AND TRADEMARK OFFICE
UNITED STATES DEPARTMENT OF COMMERCE
United States Patent and Trademark Office
Address: COMMISSIONER FOR PATENTS
P.O. Box 1450
Alexandria, Virginia 22313-1450
www.uspto.gov
APPLICATION NO. FILING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO.
15/495,095 04/24/2017 Chris Y. Chung 082438.032601 5728
139020 7590 07/21/2021
BakerHostetler / Apple Inc.
Washington Square, Suite 1100,
1050 Connecticut Ave, NW
Washington, DC 20036-5304
EXAMINER
NGUYEN, PHUNG HOANG JOSEPH
ART UNIT PAPER NUMBER
2651
NOTIFICATION DATE DELIVERY MODE
07/21/2021 ELECTRONIC
Please find below and/or attached an Office communication concerning this application or proceeding.
The time period for reply, if any, is set in the attached communication.
Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the
following e-mail address(es):
eofficemonitor@bakerlaw.com
PTOL-90A (Rev. 04/07)
UNITED STATES PATENT AND TRADEMARK OFFICE
____________

BEFORE THE PATENT TRIAL AND APPEAL BOARD
____________

Ex parte CHRIS Y. CHUNG, DAZHONG ZHANG,
HSI-JUNG WU, and XIAOSONG ZHOU
____________

Appeal 2020-003108
Application 15/495,095
Technology Center 2600
_______________

Before MICHAEL J. STRAUSS, JAMES B. ARPIN, and
HUNG H. BUI, Administrative Patent Judges.

Opinion for the Board filed by Administrative Patent Judge STRAUSS.

Opinion Dissenting filed by Administrative Patent Judge BUI.

DECISION ON APPEAL
Appellant11 seeks our review under 35 U.S.C. § 134(a) from the
Examiner’s Final rejection of claims 51–56 and 58–69, all of the pending
claims. Supp. Appeal Br. 13–15 (Claims App.). We have jurisdiction under
35 U.S.C. § 6(b).
We REVERSE.2

1 “Appellant” refers to “applicant(s)” as defined in 37 C.F.R. § 1.42.
Appellant identifies Apple Inc., as the real party in interest. Supp. Appeal
Br. 2.
2 We refer to the Appellant’s Supplemental Appeal Brief filed January 9,
2020 (“Supp. Appeal Br.”); the Reply Brief filed March 18, 2020 (“Reply
Br.”); the Examiner’s Answer mailed February 21, 2020 (“Ans.”); the Final
Appeal 2020-003108
Application 15/495,095

2
STATEMENT OF THE CASE
Appellant’s claimed invention relates to “conferencing applications
involving exchange and annotation of video and, specifically, to techniques
for mitigating aberrant rendering artifacts induced by communication delay
among conference participants [in a video exchange system, shown in
Figure 1].” Spec. ¶ 1.
Appellant’s Figure 1 is reproduced below:

Figure 1 shows a video exchange system for exchanging image
annotations between a first terminal 110 and a second terminal 120
(devices), via a network 130, within a streaming session 140.

As shown above in Figure 1, first terminal 110 may send coded video
data (screen content 142) to second terminal 120 where it is decoded and

Office Action mailed July 11, 2019 (“Final Act.”); and the Specification
filed April 24, 2017 (“Spec.”).
Appeal 2020-003108
Application 15/495,095

3
displayed. A viewer at second terminal 120 enters an annotation upon a
frame of the displayed video, which is then coded and coded annotation 144
is transmitted back to first terminal 110 within streaming session 140.
According to Appellant, the annotated frame at second terminal 120 is coded
by predictive coding technique using a reference frame selected from a
decoded video stream (screen content 142) received from first terminal 110
(in a first direction), and is then transmitted back to first terminal 110 (in a
second, opposite direction). Spec. ¶ 37. For example, the Specification
discloses:
[T]he reference frame would be a frame initially used by the
mirroring terminal to code video data sent from the mirroring
terminal to the far end terminal; when used for purposes of
annotation, the mirroring terminal may re-use the reference
frame as a prediction reference for coding of the annotated frame
and transmission back to the mirroring terminal.
Id. (emphasis added).
Claim 51, reproduced below with disputed limitations emphasized, is
representative:
51. A method of annotating data between terminals, comprising:
decoding and displaying coded video received from a
distant terminal in a first direction of communication, the
coding and decoding of the coded video is relative to reference
frames stored by the terminals for decoding in the first
direction of communication,
responsive to an annotation entered by a user, coding
according to predictive coding techniques a displayed frame
being annotated, the coding using a prediction reference from
at least one of the reference frames stored by the terminals, and
transmitting the coded frame and data representing the
annotation to the distant terminal in a second direction of
communication.
Supp. Appeal Br. 18 (Claims App.).
Appeal 2020-003108
Application 15/495,095

4
REJECTIONS AND REFERENCES
(1) The Examiner rejects claims 51, 52, 58, 59, 64, and 65 under
35 U.S.C. § 103 as obvious over the combined teachings of Ahmad (US
2016/0180743 A1; published June 23, 2016), Zhou et al. (US 2013/0223525
A1; published Aug. 29, 2013; “Zhou”), and Junuzovic et al. (US
2014/0063174 A1; published Mar. 6, 2014; “Junuzovic”). Final Act. 2–7.
(2) The Examiner rejects claims 53–56, 60–64, and 66–69 under
35 U.S.C. § 103 as obvious over the combined teachings of Ahmad, Zhou,
Junuzovic, and Ohmori et al. (US 6,339,431 B1; issued Jan. 15, 2002). Final
Act. 7–8.
ANALYSIS
In support of the obviousness rejection of independent claims 51, 58,
and 64, the Examiner finds Ahmad teaches most limitations of Appellant’s
claimed “method of annotating between terminals,” as shown in Figure 1,
including: (1) “decoding and displaying coded video received from a distant
terminal,” (2) “responsive to an annotation entered by a user, coding . . . a
displayed frame being annotated,” and (3) “transmitting the coded frame and
data representing the annotation to the distant terminal.” Final Act. 2–3
(citing Ahmad ¶¶ 14, 19, 59–61, Fig. 2).
Appeal 2020-003108
Application 15/495,095

5
Ahmad’s Figure 1 is reproduced below:

Ahmad’s Figure 1 shows video interaction between local site 110 (1st
terminal) and remote site 130 (2nd terminal), via a network 120, where video
is captured by local doctor 140 at local site 110 and delivered to remote
doctor 150 at remote site 130 (shown in Figures 2–3) to make annotations
directly on live video streams for transmission back to local doctor 140 at
local site 110.
The Examiner also finds Ahmad teaches (1) two-way audio/video
communication including annotation between remote doctor 150 at remote
site 130 and local doctor 140 at local site 110, i.e., remote doctor 150 at
remote site 130 may also “see annotations made by another person (such as
local doctor 140 [at local site 110])” (Ans. 10 (citing Ahmad ¶¶ 53, 57; see
id. ¶ 17) and (2) “[a]udio and video transmitted between local site 110 and
remote site 130 may use any known encoding and compression techniques
(e.g., NTSC, PAL, H.264, SIP, H.323, G.711, G.726, AAC, and triple
streaming encoder).” Final Act. 3 (citing Ahmad ¶ 20).
Appeal 2020-003108
Application 15/495,095

6
The Examiner finds Ahmad does not teach, and, therefore, relies on
Zhou to teach or suggest the use of “predictive coding techniques” to code
live video streams using reference frames stored at terminals, i.e., “the
coding using a prediction reference from at least one of the reference frames
stored by the terminals,” as recited in Appellant’s claims 51, 58, and 64.
Final Act. 4 (citing Zhou ¶ 17; see id. at Figs. 1 and 5). Based on these
references’ teachings, the Examiner finds it would have been obvious to
incorporate the “predictive coding technique” of Zhou into Ahmad in order
to “improve reference frames for video being coded” and “enhance the
communication, better quality for encoder’s prediction.” Id.
To the extent necessary, the Examiner also finds Junuzovic
corroborates Ahmad’s teachings of two-way annotations between local
doctor 140 at local site 110 and remote doctor 150 at remote site 130 in
either direction, as shown in Ahmad’s Figures 2–3. Final Act. 6–7 (citing
Junuzovic ¶¶ 4–6, 103).
Appellant does not dispute the Examiner’s rationale for combining
Ahmad, Zhou, and Junuzovic. Instead, Appellant contends the applied
references does not teach or suggest the disputed limitations of claim 51, 58,
and 64, including:
1. “the coding and decoding of the coded video is relative to
reference frames stored by the terminals for decoding in the first
direction of communication,”

2. “the coding using a prediction reference from at least one of the
reference frames stored by the terminals,” and

3. “transmitting the coded frame and data representing the
annotation to the distant terminal in a second direction of
communication.”
Appeal 2020-003108
Application 15/495,095

7
Appeal Br. 5–8.
In particular, Appellant contends Ahmad does not teach or suggest
that:
“reference frames stored by the terminals for a first direction of
communication” are used to code “according to predictive
coding techniques a displayed frame being annotated,” which is
then “transmit[ed] in a second direction of communication ... to
the distant terminal.”
Id. According to Appellant,
Ahmad certainly has no hint or suggestion that reference frames
from one direction of communication (e.g., an in-bound
direction) would be applied to the encoding of an image in a
second direction of communication (e.g., an out-bound
direction).
Id. at 6.
Appellant also contends Zhou, as a secondary reference, only teaches
or suggests, “a [video] coding system that operates according to a single
direction of transmission,” that is, “coded video proceeds to a distant
terminal only in one direction—out through the channel 250,” as shown in
Figure 2, reproduced below:

Relying on Zhou’s Figure 2, Appellant contends,
Appeal 2020-003108
Application 15/495,095

8
Zhou has no concept of predictively coding an image of a
“displayed frame,” whether annotated or not, using a reference
frame “stored by the terminals for the first direction of
communication,” then transmitting the coded displayed frame
“in a second direction of communication.”
Id. at 6–7.
According to Appellant, “[n]o cited reference teaches or suggests the
concept of coding an image based on reference frames previously used to
code and decode the video from which the image was extracted.” Id. at 9.
Appellant’s contentions are persuasive. As depicted in the figures
below, Zhou discloses encoding video at terminal 110, transmitting the
encoded video transmitted in a first direction to terminal 120, and decoding
the received video for display by terminal 120. Providing for transmission
of an annotated frame in a reverse (i.e., second) direction according to
Ahmad, video coding would be performed by video coder 220 of
Appeal 2020-003108
Application 15/495,095

9
terminal 110, the coded video transmitted in a second direction, back to
terminal 110 for decoding by video decoder 320 of terminal 110. Zhou
discloses the reference pictures are generated by coding engine 220. Zhou,
Abstr., ¶ 21. Although a receiving terminal’s decoder 320 may store
reference frames recovered from decoded video frames “for use in decoding
subsequently-received coded video data” (Zhou ¶ 26), this is not the same as
using the stored reference frames to code annotated video. In particular, the
Examiner fails to provide sufficient evidence or reasoning to support a
finding that receiving terminal 120 uses a reference frame stored on both the
originating terminal 110 and receiving terminal 120 for decoding video and
to subsequently encode an annotated video frame for transmission back (i.e.,
in a second direction) to terminal 110.
Although Zhou teaches or suggests the reuse of reference frames,
Zhou’s reuse is described in connection with encoding separately and apart
from subsequent use by the receiving terminal in decoding. Zhou ¶ 26. That
is, Zhou’s coding engine 220 may reuse the same reference frames that it
generated in a prior coding process in subsequent coding operations. Id.
¶ 21. Likewise, Zhou’s decoding engine 320 may reuse reference frames
recovered in a prior decoding operation to decode subsequently-received
coded video data. Id. ¶ 26. Zhou does not, however, teach a crossover use
of reference frames whereby a reference frame used during decoding is then
used to encode video, much less to encode annotated frames, for subsequent
transmission.
Although not relied upon by the Examiner, Zhou also teaches creating
libraries of prediction patches built from stored reference frames that may be
used to improve coding efficiency. Id. ¶¶ 42–43. According to Zhou, the
Appeal 2020-003108
Application 15/495,095

10
reference patches may be generated from a variety of sources including
“derived from image content stored locally in the terminal that performs the
encoding.” Id. ¶¶ 43–44. However, Zhou does not describe using prediction
patches from a coded video as a reference frame to encode an annotated
frame. Thus, we agree with Appellant that the applied references do not
teach or suggest all of the disputed limitations of claim 51.
Because we agree with at least one of the contentions advanced by
Appellant, we do not reach the merits of Appellant’s other arguments.
Accordingly, we do not sustain the obviousness rejection of
independent claim 51, or the rejection of independent claims 58 and 64,
which include language corresponding to the contested limitations of
claim 51. Nor do we sustain the rejection of dependent claims 52–56, 59–
63, and 65–69, which stand with their respective base claims.
CONCLUSION
On this record, the Examiner errs in rejecting (1) claims 51–52, 58–
59, and 64–65 as obvious over the combined teachings of Ahmad, Zhou, and
Junuzovic; and (2) claims 53–56, 60–64, and 66–69 as obvious over the
combined teachings of Ahmad, Zhou, Junuzovic, and Ohmori.

Appeal 2020-003108
Application 15/495,095

11
DECISION SUMMARY
In Summary:
Claims
Rejected
35
U.S.C. §
Reference(s)/Basis Affirmed Reversed
51, 52, 58,
59, 64, 65
103 Ahmad, Zhou,
Junuzovic
51, 52, 58,
59, 64, 65
53–56, 60–
64, 66–69
103 Ahmad, Zhou,
Junuzovic, Ohmori
53–56, 60–
64, 66–69
Overall
Outcome
51–56, 58–
69

REVERSED

Appeal 2020-003108
Application 15/495,095

12
BUI, Administrative Patent Judge, dissenting.

I respectfully disagree with the majority’s reversal of (1) claims 51–
52, 58–59, and 64–65 as obvious over the combined teachings of Ahmad,
Zhou, and Junuzovic; and (2) claims 53–56, 60–64, and 66–69 as obvious
over the combined teachings of Ahmad, Zhou, Junuzovic, and Ohmori under
35 U.S.C. § 103.
In my view, the combination of Ahmad, Zhou and Junuzovic teaches
or suggests all of the limitations of claims 51, 58, and 64. For example,
Ahmad teaches two-way annotations between local doctor 140 at local site
110 and remote doctor 150 at remote site 130 in either direction, as shown in
Ahmad’s Figures 2–3. Final Act. (citing Ahmad ¶¶ 14, 19, 59–61, Fig. 2);
Ans. 10 (citing Ahmad ¶¶ 53, 57; see also Ahmad ¶ 17). Ahmad also
teaches or suggests that any known encoding and compression techniques
could be used to transmit coded video including annotations between local
site 110 and remote site 130. Final Act. 3 (citing Ahmad ¶ 20).
As a secondary reference, Zhou teaches predictive coding techniques,
as known encoding and compression techniques, for use in a video
coder/decoder (known as a codec), as shown in Figures 2–3, installed in
each terminal to code video using reference frames stored in each terminal,
shown in Figure 5. Zhou ¶ 17. According to the Examiner, Zhou’s
predictive coding technique using reference frames stored in each terminal
could be incorporated into each of Ahmad’s terminals (for example, local
site 110 and remote site 130 of Ahmad’s video annotation system) to yield
predictable results, i.e., capture video at each terminal, whether from a video
source or another terminal, and code the same video, whether annotated or
Appeal 2020-003108
Application 15/495,095

13
not, using reference frames for transmission back to the other terminal.
Ans. 12 (citing Zhou ¶ 15, Figs. 1–2).
In the Reply Brief, Appellant contends Zhou’s “functional blocks
illustrated in FIGS. 2 and 3 support video coding and decoding in one
direction only.” Reply Br. 3 (citing Zhou ¶ 28). To support that argument,
Appellant provides a sketch of Zhou’s Figures 2–3 connecting
transmitter 230 of video coder 220, as shown in Zhou’s Figure 2, to receiver
310 of video decoder 320, as shown in Zhou’s Figure 3, as reproduced
below with additional markings for illustration purposes:

Based on Appellant’s interpretation of Zhou’s Figures 2–3, Appellant
contends Zhou “merely show that independent encoders may be used to code
video with independent content in separate directions” and “does not show
Appeal 2020-003108
Application 15/495,095

14
that reference frames, which are ‘stored by the terminals for decoding in the
first direction of communication,’ also are used for ‘coding . . . a displayed
frame being annotated’ and ‘transmitting the coded frame . . . in a second
direction of communication.’” Reply Br. 5.
I do not agree. Zhou’s video coder 220 and decoder 320, as shown in
Figures 2–3, are not connected in the manner suggested by Appellant.
Instead, both Zhou’s video coder 220 and decoder 320, as shown in
Figures 2–3, are part of an integrated codec 740, as shown in Figure 7 and
the demonstrative Figure prepared by the majority, installed at each terminal
to capture video, whether that video comes from a video source or another
terminal, and then code the same video, whether the video is annotated or
not, using reference frames for transmission back to the other terminal.
Final Act. 4 (citing Zhou ¶ 17); Ans. 12 (citing Zhou ¶ 15).
For example, Zhou’s Figure 7 shows a video coding system including
1st terminal 110 and 2nd terminal 120, as reproduced below with additional
markings for illustration.
Appeal 2020-003108
Application 15/495,095

15

Figure 5 shows a communication protocol between 1st terminal 110 and 2nd
terminal 120, where (1) video coder 220, shown in Zhou’s Figure 2, installed
at 1st terminal 110 may capture source video at step 530, and code that video
using reference frames at step 534 for transmission to the 2nd terminal 120 at
step 540 (in a first direction), and (2) video decoder 320, shown in Zhou’s
Figure 3, installed at the 2nd terminal 120 may receive coded video at step
555, and decode that video using reference frames and reference patches at
step 565 for a visual display. Zhou ¶¶ 35–39. According to Zhou, these
reference frames are designated by video coder/decoder 220, 320 and stored
Appeal 2020-003108
Application 15/495,095

16
in reference frame cache 224, 324 at each terminal 110, 120 for use in
coding later-received video or decoding subsequently-received video. Zhou
¶¶ 17, 21, 25–39. In a simple embodiment of Zhou, these reference frames
may be copied as “prediction patches” (Zhou ¶ 42) and stored in patch
caches 226, 325 (Zhou ¶ 43), and such “prediction patches” “may be
exchanged between terminals” via “signaling” (Zhou ¶ 35), as shown, for
example, in step 515 of Zhou’s Figure 5, as well as “can be updated and
synced between encoder and decoder [installed at terminals 110, 120] by
signaling” (Zhou ¶ 39). For example,
[d]uring active coding, the second terminal 120 may
receive the coded video (box 555) and store any [prediction]
patches contained in the coded video to the patch cache (box
560). The terminal 120 also may decode video using reference
frames and prediction patches as indicated in the coded video
data (box 656) and may display the recovered video (box 570).
Zhou ¶ 38.
Thus, when Zhou’s codec 740, as shown in Figure 5, is incorporated
at each of Ahmad’s terminals (for example, local site 110 and remote
site 130 of Ahmad’s video annotation system) in the manner proposed by the
Examiner, a person skilled in the art would understand that (1) coded video
would be received at remote site 130 from local site 110 (in a first direction)
and (2) in response to an annotation entered by a user at remote site 130,
annotated video would be coded according to Zhou’s predictive coding
techniques, using reference frames included in the coded video received at
remote site 130 from local site 110 (in a first direction) and stored at remote
site 130 and then transmitted back to local site 110 (in a second direction) in
the manner recited in Appellant’s claims 51, 58, and 64.
Appeal 2020-003108
Application 15/495,095

17
As the majority notes, Appellant does not challenge the Examiner’s
rationale for combining Ahmad, Zhou, and Junuzovic.
For these reasons, I would affirm the Examiner’s rejections of (1)
claims 51–52, 58–59, 64, and 65 under 35 U.S.C. § 103 as obvious over the
combined teachings of Ahmad, Zhou, and Junuzovic; and (2) claims 53–56,
60–64, 66 and 69 under 35 U.S.C. § 103 as obvious over the combined
tachings of Ahmad, Zhou, Junuzovic, and Ohmori.