Apple Inc.Download PDFPatent Trials and Appeals BoardJul 21, 20212020003108 (P.T.A.B. Jul. 21, 2021) Copy Citation UNITED STATES PATENT AND TRADEMARK OFFICE UNITED STATES DEPARTMENT OF COMMERCE United States Patent and Trademark Office Address: COMMISSIONER FOR PATENTS P.O. Box 1450 Alexandria, Virginia 22313-1450 www.uspto.gov APPLICATION NO. FILING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO. 15/495,095 04/24/2017 Chris Y. Chung 082438.032601 5728 139020 7590 07/21/2021 BakerHostetler / Apple Inc. Washington Square, Suite 1100, 1050 Connecticut Ave, NW Washington, DC 20036-5304 EXAMINER NGUYEN, PHUNG HOANG JOSEPH ART UNIT PAPER NUMBER 2651 NOTIFICATION DATE DELIVERY MODE 07/21/2021 ELECTRONIC Please find below and/or attached an Office communication concerning this application or proceeding. The time period for reply, if any, is set in the attached communication. Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the following e-mail address(es): eofficemonitor@bakerlaw.com PTOL-90A (Rev. 04/07) UNITED STATES PATENT AND TRADEMARK OFFICE ____________ BEFORE THE PATENT TRIAL AND APPEAL BOARD ____________ Ex parte CHRIS Y. CHUNG, DAZHONG ZHANG, HSI-JUNG WU, and XIAOSONG ZHOU ____________ Appeal 2020-003108 Application 15/495,095 Technology Center 2600 _______________ Before MICHAEL J. STRAUSS, JAMES B. ARPIN, and HUNG H. BUI, Administrative Patent Judges. Opinion for the Board filed by Administrative Patent Judge STRAUSS. Opinion Dissenting filed by Administrative Patent Judge BUI. DECISION ON APPEAL Appellant11 seeks our review under 35 U.S.C. § 134(a) from the Examiner’s Final rejection of claims 51–56 and 58–69, all of the pending claims. Supp. Appeal Br. 13–15 (Claims App.). We have jurisdiction under 35 U.S.C. § 6(b). We REVERSE.2 1 “Appellant” refers to “applicant(s)” as defined in 37 C.F.R. § 1.42. Appellant identifies Apple Inc., as the real party in interest. Supp. Appeal Br. 2. 2 We refer to the Appellant’s Supplemental Appeal Brief filed January 9, 2020 (“Supp. Appeal Br.”); the Reply Brief filed March 18, 2020 (“Reply Br.”); the Examiner’s Answer mailed February 21, 2020 (“Ans.”); the Final Appeal 2020-003108 Application 15/495,095 2 STATEMENT OF THE CASE Appellant’s claimed invention relates to “conferencing applications involving exchange and annotation of video and, specifically, to techniques for mitigating aberrant rendering artifacts induced by communication delay among conference participants [in a video exchange system, shown in Figure 1].” Spec. ¶ 1. Appellant’s Figure 1 is reproduced below: Figure 1 shows a video exchange system for exchanging image annotations between a first terminal 110 and a second terminal 120 (devices), via a network 130, within a streaming session 140. As shown above in Figure 1, first terminal 110 may send coded video data (screen content 142) to second terminal 120 where it is decoded and Office Action mailed July 11, 2019 (“Final Act.”); and the Specification filed April 24, 2017 (“Spec.”). Appeal 2020-003108 Application 15/495,095 3 displayed. A viewer at second terminal 120 enters an annotation upon a frame of the displayed video, which is then coded and coded annotation 144 is transmitted back to first terminal 110 within streaming session 140. According to Appellant, the annotated frame at second terminal 120 is coded by predictive coding technique using a reference frame selected from a decoded video stream (screen content 142) received from first terminal 110 (in a first direction), and is then transmitted back to first terminal 110 (in a second, opposite direction). Spec. ¶ 37. For example, the Specification discloses: [T]he reference frame would be a frame initially used by the mirroring terminal to code video data sent from the mirroring terminal to the far end terminal; when used for purposes of annotation, the mirroring terminal may re-use the reference frame as a prediction reference for coding of the annotated frame and transmission back to the mirroring terminal. Id. (emphasis added). Claim 51, reproduced below with disputed limitations emphasized, is representative: 51. A method of annotating data between terminals, comprising: decoding and displaying coded video received from a distant terminal in a first direction of communication, the coding and decoding of the coded video is relative to reference frames stored by the terminals for decoding in the first direction of communication, responsive to an annotation entered by a user, coding according to predictive coding techniques a displayed frame being annotated, the coding using a prediction reference from at least one of the reference frames stored by the terminals, and transmitting the coded frame and data representing the annotation to the distant terminal in a second direction of communication. Supp. Appeal Br. 18 (Claims App.). Appeal 2020-003108 Application 15/495,095 4 REJECTIONS AND REFERENCES (1) The Examiner rejects claims 51, 52, 58, 59, 64, and 65 under 35 U.S.C. § 103 as obvious over the combined teachings of Ahmad (US 2016/0180743 A1; published June 23, 2016), Zhou et al. (US 2013/0223525 A1; published Aug. 29, 2013; “Zhou”), and Junuzovic et al. (US 2014/0063174 A1; published Mar. 6, 2014; “Junuzovic”). Final Act. 2–7. (2) The Examiner rejects claims 53–56, 60–64, and 66–69 under 35 U.S.C. § 103 as obvious over the combined teachings of Ahmad, Zhou, Junuzovic, and Ohmori et al. (US 6,339,431 B1; issued Jan. 15, 2002). Final Act. 7–8. ANALYSIS In support of the obviousness rejection of independent claims 51, 58, and 64, the Examiner finds Ahmad teaches most limitations of Appellant’s claimed “method of annotating between terminals,” as shown in Figure 1, including: (1) “decoding and displaying coded video received from a distant terminal,” (2) “responsive to an annotation entered by a user, coding . . . a displayed frame being annotated,” and (3) “transmitting the coded frame and data representing the annotation to the distant terminal.” Final Act. 2–3 (citing Ahmad ¶¶ 14, 19, 59–61, Fig. 2). Appeal 2020-003108 Application 15/495,095 5 Ahmad’s Figure 1 is reproduced below: Ahmad’s Figure 1 shows video interaction between local site 110 (1st terminal) and remote site 130 (2nd terminal), via a network 120, where video is captured by local doctor 140 at local site 110 and delivered to remote doctor 150 at remote site 130 (shown in Figures 2–3) to make annotations directly on live video streams for transmission back to local doctor 140 at local site 110. The Examiner also finds Ahmad teaches (1) two-way audio/video communication including annotation between remote doctor 150 at remote site 130 and local doctor 140 at local site 110, i.e., remote doctor 150 at remote site 130 may also “see annotations made by another person (such as local doctor 140 [at local site 110])” (Ans. 10 (citing Ahmad ¶¶ 53, 57; see id. ¶ 17) and (2) “[a]udio and video transmitted between local site 110 and remote site 130 may use any known encoding and compression techniques (e.g., NTSC, PAL, H.264, SIP, H.323, G.711, G.726, AAC, and triple streaming encoder).” Final Act. 3 (citing Ahmad ¶ 20). Appeal 2020-003108 Application 15/495,095 6 The Examiner finds Ahmad does not teach, and, therefore, relies on Zhou to teach or suggest the use of “predictive coding techniques” to code live video streams using reference frames stored at terminals, i.e., “the coding using a prediction reference from at least one of the reference frames stored by the terminals,” as recited in Appellant’s claims 51, 58, and 64. Final Act. 4 (citing Zhou ¶ 17; see id. at Figs. 1 and 5). Based on these references’ teachings, the Examiner finds it would have been obvious to incorporate the “predictive coding technique” of Zhou into Ahmad in order to “improve reference frames for video being coded” and “enhance the communication, better quality for encoder’s prediction.” Id. To the extent necessary, the Examiner also finds Junuzovic corroborates Ahmad’s teachings of two-way annotations between local doctor 140 at local site 110 and remote doctor 150 at remote site 130 in either direction, as shown in Ahmad’s Figures 2–3. Final Act. 6–7 (citing Junuzovic ¶¶ 4–6, 103). Appellant does not dispute the Examiner’s rationale for combining Ahmad, Zhou, and Junuzovic. Instead, Appellant contends the applied references does not teach or suggest the disputed limitations of claim 51, 58, and 64, including: 1. “the coding and decoding of the coded video is relative to reference frames stored by the terminals for decoding in the first direction of communication,” 2. “the coding using a prediction reference from at least one of the reference frames stored by the terminals,” and 3. “transmitting the coded frame and data representing the annotation to the distant terminal in a second direction of communication.” Appeal 2020-003108 Application 15/495,095 7 Appeal Br. 5–8. In particular, Appellant contends Ahmad does not teach or suggest that: “reference frames stored by the terminals for a first direction of communication” are used to code “according to predictive coding techniques a displayed frame being annotated,” which is then “transmit[ed] in a second direction of communication ... to the distant terminal.” Id. According to Appellant, Ahmad certainly has no hint or suggestion that reference frames from one direction of communication (e.g., an in-bound direction) would be applied to the encoding of an image in a second direction of communication (e.g., an out-bound direction). Id. at 6. Appellant also contends Zhou, as a secondary reference, only teaches or suggests, “a [video] coding system that operates according to a single direction of transmission,” that is, “coded video proceeds to a distant terminal only in one direction—out through the channel 250,” as shown in Figure 2, reproduced below: Relying on Zhou’s Figure 2, Appellant contends, Appeal 2020-003108 Application 15/495,095 8 Zhou has no concept of predictively coding an image of a “displayed frame,” whether annotated or not, using a reference frame “stored by the terminals for the first direction of communication,” then transmitting the coded displayed frame “in a second direction of communication.” Id. at 6–7. According to Appellant, “[n]o cited reference teaches or suggests the concept of coding an image based on reference frames previously used to code and decode the video from which the image was extracted.” Id. at 9. Appellant’s contentions are persuasive. As depicted in the figures below, Zhou discloses encoding video at terminal 110, transmitting the encoded video transmitted in a first direction to terminal 120, and decoding the received video for display by terminal 120. Providing for transmission of an annotated frame in a reverse (i.e., second) direction according to Ahmad, video coding would be performed by video coder 220 of Appeal 2020-003108 Application 15/495,095 9 terminal 110, the coded video transmitted in a second direction, back to terminal 110 for decoding by video decoder 320 of terminal 110. Zhou discloses the reference pictures are generated by coding engine 220. Zhou, Abstr., ¶ 21. Although a receiving terminal’s decoder 320 may store reference frames recovered from decoded video frames “for use in decoding subsequently-received coded video data” (Zhou ¶ 26), this is not the same as using the stored reference frames to code annotated video. In particular, the Examiner fails to provide sufficient evidence or reasoning to support a finding that receiving terminal 120 uses a reference frame stored on both the originating terminal 110 and receiving terminal 120 for decoding video and to subsequently encode an annotated video frame for transmission back (i.e., in a second direction) to terminal 110. Although Zhou teaches or suggests the reuse of reference frames, Zhou’s reuse is described in connection with encoding separately and apart from subsequent use by the receiving terminal in decoding. Zhou ¶ 26. That is, Zhou’s coding engine 220 may reuse the same reference frames that it generated in a prior coding process in subsequent coding operations. Id. ¶ 21. Likewise, Zhou’s decoding engine 320 may reuse reference frames recovered in a prior decoding operation to decode subsequently-received coded video data. Id. ¶ 26. Zhou does not, however, teach a crossover use of reference frames whereby a reference frame used during decoding is then used to encode video, much less to encode annotated frames, for subsequent transmission. Although not relied upon by the Examiner, Zhou also teaches creating libraries of prediction patches built from stored reference frames that may be used to improve coding efficiency. Id. ¶¶ 42–43. According to Zhou, the Appeal 2020-003108 Application 15/495,095 10 reference patches may be generated from a variety of sources including “derived from image content stored locally in the terminal that performs the encoding.” Id. ¶¶ 43–44. However, Zhou does not describe using prediction patches from a coded video as a reference frame to encode an annotated frame. Thus, we agree with Appellant that the applied references do not teach or suggest all of the disputed limitations of claim 51. Because we agree with at least one of the contentions advanced by Appellant, we do not reach the merits of Appellant’s other arguments. Accordingly, we do not sustain the obviousness rejection of independent claim 51, or the rejection of independent claims 58 and 64, which include language corresponding to the contested limitations of claim 51. Nor do we sustain the rejection of dependent claims 52–56, 59– 63, and 65–69, which stand with their respective base claims. CONCLUSION On this record, the Examiner errs in rejecting (1) claims 51–52, 58– 59, and 64–65 as obvious over the combined teachings of Ahmad, Zhou, and Junuzovic; and (2) claims 53–56, 60–64, and 66–69 as obvious over the combined teachings of Ahmad, Zhou, Junuzovic, and Ohmori. Appeal 2020-003108 Application 15/495,095 11 DECISION SUMMARY In Summary: Claims Rejected 35 U.S.C. § Reference(s)/Basis Affirmed Reversed 51, 52, 58, 59, 64, 65 103 Ahmad, Zhou, Junuzovic 51, 52, 58, 59, 64, 65 53–56, 60– 64, 66–69 103 Ahmad, Zhou, Junuzovic, Ohmori 53–56, 60– 64, 66–69 Overall Outcome 51–56, 58– 69 REVERSED Appeal 2020-003108 Application 15/495,095 12 BUI, Administrative Patent Judge, dissenting. I respectfully disagree with the majority’s reversal of (1) claims 51– 52, 58–59, and 64–65 as obvious over the combined teachings of Ahmad, Zhou, and Junuzovic; and (2) claims 53–56, 60–64, and 66–69 as obvious over the combined teachings of Ahmad, Zhou, Junuzovic, and Ohmori under 35 U.S.C. § 103. In my view, the combination of Ahmad, Zhou and Junuzovic teaches or suggests all of the limitations of claims 51, 58, and 64. For example, Ahmad teaches two-way annotations between local doctor 140 at local site 110 and remote doctor 150 at remote site 130 in either direction, as shown in Ahmad’s Figures 2–3. Final Act. (citing Ahmad ¶¶ 14, 19, 59–61, Fig. 2); Ans. 10 (citing Ahmad ¶¶ 53, 57; see also Ahmad ¶ 17). Ahmad also teaches or suggests that any known encoding and compression techniques could be used to transmit coded video including annotations between local site 110 and remote site 130. Final Act. 3 (citing Ahmad ¶ 20). As a secondary reference, Zhou teaches predictive coding techniques, as known encoding and compression techniques, for use in a video coder/decoder (known as a codec), as shown in Figures 2–3, installed in each terminal to code video using reference frames stored in each terminal, shown in Figure 5. Zhou ¶ 17. According to the Examiner, Zhou’s predictive coding technique using reference frames stored in each terminal could be incorporated into each of Ahmad’s terminals (for example, local site 110 and remote site 130 of Ahmad’s video annotation system) to yield predictable results, i.e., capture video at each terminal, whether from a video source or another terminal, and code the same video, whether annotated or Appeal 2020-003108 Application 15/495,095 13 not, using reference frames for transmission back to the other terminal. Ans. 12 (citing Zhou ¶ 15, Figs. 1–2). In the Reply Brief, Appellant contends Zhou’s “functional blocks illustrated in FIGS. 2 and 3 support video coding and decoding in one direction only.” Reply Br. 3 (citing Zhou ¶ 28). To support that argument, Appellant provides a sketch of Zhou’s Figures 2–3 connecting transmitter 230 of video coder 220, as shown in Zhou’s Figure 2, to receiver 310 of video decoder 320, as shown in Zhou’s Figure 3, as reproduced below with additional markings for illustration purposes: Based on Appellant’s interpretation of Zhou’s Figures 2–3, Appellant contends Zhou “merely show that independent encoders may be used to code video with independent content in separate directions” and “does not show Appeal 2020-003108 Application 15/495,095 14 that reference frames, which are ‘stored by the terminals for decoding in the first direction of communication,’ also are used for ‘coding . . . a displayed frame being annotated’ and ‘transmitting the coded frame . . . in a second direction of communication.’” Reply Br. 5. I do not agree. Zhou’s video coder 220 and decoder 320, as shown in Figures 2–3, are not connected in the manner suggested by Appellant. Instead, both Zhou’s video coder 220 and decoder 320, as shown in Figures 2–3, are part of an integrated codec 740, as shown in Figure 7 and the demonstrative Figure prepared by the majority, installed at each terminal to capture video, whether that video comes from a video source or another terminal, and then code the same video, whether the video is annotated or not, using reference frames for transmission back to the other terminal. Final Act. 4 (citing Zhou ¶ 17); Ans. 12 (citing Zhou ¶ 15). For example, Zhou’s Figure 7 shows a video coding system including 1st terminal 110 and 2nd terminal 120, as reproduced below with additional markings for illustration. Appeal 2020-003108 Application 15/495,095 15 Figure 5 shows a communication protocol between 1st terminal 110 and 2nd terminal 120, where (1) video coder 220, shown in Zhou’s Figure 2, installed at 1st terminal 110 may capture source video at step 530, and code that video using reference frames at step 534 for transmission to the 2nd terminal 120 at step 540 (in a first direction), and (2) video decoder 320, shown in Zhou’s Figure 3, installed at the 2nd terminal 120 may receive coded video at step 555, and decode that video using reference frames and reference patches at step 565 for a visual display. Zhou ¶¶ 35–39. According to Zhou, these reference frames are designated by video coder/decoder 220, 320 and stored Appeal 2020-003108 Application 15/495,095 16 in reference frame cache 224, 324 at each terminal 110, 120 for use in coding later-received video or decoding subsequently-received video. Zhou ¶¶ 17, 21, 25–39. In a simple embodiment of Zhou, these reference frames may be copied as “prediction patches” (Zhou ¶ 42) and stored in patch caches 226, 325 (Zhou ¶ 43), and such “prediction patches” “may be exchanged between terminals” via “signaling” (Zhou ¶ 35), as shown, for example, in step 515 of Zhou’s Figure 5, as well as “can be updated and synced between encoder and decoder [installed at terminals 110, 120] by signaling” (Zhou ¶ 39). For example, [d]uring active coding, the second terminal 120 may receive the coded video (box 555) and store any [prediction] patches contained in the coded video to the patch cache (box 560). The terminal 120 also may decode video using reference frames and prediction patches as indicated in the coded video data (box 656) and may display the recovered video (box 570). Zhou ¶ 38. Thus, when Zhou’s codec 740, as shown in Figure 5, is incorporated at each of Ahmad’s terminals (for example, local site 110 and remote site 130 of Ahmad’s video annotation system) in the manner proposed by the Examiner, a person skilled in the art would understand that (1) coded video would be received at remote site 130 from local site 110 (in a first direction) and (2) in response to an annotation entered by a user at remote site 130, annotated video would be coded according to Zhou’s predictive coding techniques, using reference frames included in the coded video received at remote site 130 from local site 110 (in a first direction) and stored at remote site 130 and then transmitted back to local site 110 (in a second direction) in the manner recited in Appellant’s claims 51, 58, and 64. Appeal 2020-003108 Application 15/495,095 17 As the majority notes, Appellant does not challenge the Examiner’s rationale for combining Ahmad, Zhou, and Junuzovic. For these reasons, I would affirm the Examiner’s rejections of (1) claims 51–52, 58–59, 64, and 65 under 35 U.S.C. § 103 as obvious over the combined teachings of Ahmad, Zhou, and Junuzovic; and (2) claims 53–56, 60–64, 66 and 69 under 35 U.S.C. § 103 as obvious over the combined tachings of Ahmad, Zhou, Junuzovic, and Ohmori. Copy with citationCopy as parenthetical citation