Ex Parte Nishihara et alDownload PDFPatent Trial and Appeal BoardDec 30, 201612337225 (P.T.A.B. Dec. 30, 2016) Copy Citation United States Patent and Trademark Office UNITED STATES DEPARTMENT OF COMMERCE United States Patent and Trademark Office Address: COMMISSIONER FOR PATENTS P.O.Box 1450 Alexandria, Virginia 22313-1450 www.uspto.gov APPLICATION NO. FILING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO. 12/337,225 12/17/2008 H. Keith Nishihara NG(MS)018018 US PRI 3295 26294 7590 01/04/2017 TAROLLI, SUNDHEIM, COVELL & TUMMINO L.L.P. 1300 EAST NINTH STREET, SUITE 1700 CLEVELAND, OH 44114 EXAMINER BASOM, BLAINE T ART UNIT PAPER NUMBER 2173 NOTIFICATION DATE DELIVERY MODE 01/04/2017 ELECTRONIC Please find below and/or attached an Office communication concerning this application or proceeding. The time period for reply, if any, is set in the attached communication. Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the following e-mail address(es): rkline @ tarolli. com docketing@tarolli.com PTOL-90A (Rev. 04/07) UNITED STATES PATENT AND TRADEMARK OFFICE BEFORE THE PATENT TRIAL AND APPEAL BOARD Ex parte H. KEITH NISHIHARA, SHI-PING HSU, ADRIAN KAEHLER, and LARS JANGAARD Appeal 2014-000743 Application 12/337,225 Technology Center 2100 Before JASON V. MORGAN, DANIEL J. GALLIGAN, CHRISTA P. ZADO, Administrative Patent Judges. ZADO, Administrative Patent Judge. DECISION ON APPEAL Appellants1 file this appeal under 35 U.S.C. § 134(a) from the Examiner’s final rejection of claims 1, 2, 4, 10-13, 19, 41—43, 45^47, and 50-52. We have jurisdiction under 35 U.S.C. § 6(b). We affirm. 1 Appellants identify Northrup Grumman Corp. as the real party in interest. App. Br. 3. Appeal 2014-000743 Application 12/337,225 STATEMENT OF THE CASE2 The Specification and claims are generally directed to a method of providing inputs (i.e., commands) to devices in response to determining that a particular hand gesture has been performed. Spec. 14. The method includes illuminating hand gestures that are performed in the foreground of a background surface, and providing two cameras for generating silhouette images of the hand as the gestures are performed from at least two different perspectives. Id. The silhouette images are used to determine three- dimensional features of the hand to determine what gestures the hand performed. Id. The determined gestures are looked up in a library to determine an associated device input. Id. Claim 1 is exemplary: 1. A method of providing device inputs, the method comprising: illuminating hand gestures performed via a bare hand of a user in a three-dimensional foreground of a background surface with at least one infrared (IR) light source; generating a first plurality of silhouette images associated with the bare hand based on an IR light contrast between the bare hand and the background surface; generating a second plurality of silhouette images associated with the bare hand based on an IR light contrast between the bare hand and the background surface; 2 Rather than repeat the arguments here, we refer to the Appeal Brief and Reply Brief for the positions of Appellants and the Final Office Action and Answer for the positions of the Examiner. Only those arguments actually made by Appellants have been considered in this decision. Arguments that Appellants did not make in the Briefs have not been considered and are deemed to be waived. See 37 C.F.R. § 41.37(c)(l)(iv) (2012). 2 Appeal 2014-000743 Application 12/337,225 determining a plurality of three-dimensional features of the bare hand relative to the background surface based on a parallax separation of the bare hand in the first plurality of silhouette images relative to the second plurality of silhouette images; determining a provided input gesture based on the plurality of three-dimensional features of the bare hand, the provided input gesture comprising the bare hand being positioned as a flat, open hand with fingers and a thumb position in one of together and spread, and the bare hand is at least one of moving in six-degrees of freedom and moving one of upwards and downwards; comparing the provided input gesture with a plurality of predefined gesture inputs in a gesture library; and providing at least one device input corresponding to interaction with displayed visual content based on the provided input gesture corresponding to one of the plurality of predefined gesture inputs, the at least one device input responsive to the provided input gesture comprising at least one of moving a selected portion of the displayed visual content in the corresponding six-degrees of freedom and changing a point-of-view of the displayed visual content one of upwards and downwards, respectively, in at least one of the corresponding six-degrees of freedom. Claims 41—43, 45^47, and 50-52 are similar to claim 1 but recite different combinations of hand gestures and device inputs. PRIOR ART RELIED UPON IN THE REJECTIONS U.S. Patent 5,528,263 U.S. Patent 5,805,167 U.S. Patent 6,208,329 B1 U.S. Patent 6,385,331 B1 U.S. 2002/0186251 A1 U.S. 2006/0026536 A1 U.S. 2007/0252898 A1 U.S. 2008/0005703 A1 U.S. Patent 7,348,963 B2 U.S. Patent 7,598,942 B2 U.S. 2009/0315740 A1 U.S. 2010/0027843 A1 Platzker van Cruyningen Ballare Harakawa Himmel Hotelling Delean Radivoievic Bell Underkoffler Hildreth Wilson June 18, 1996 Sept. 8, 1998 Mar. 27, 2001 May 7, 2002 Dec. 12, 2002 Feb. 2, 2006 Nov. 1,2007 Jan. 3, 2008 Mar. 25, 2008 Oct. 6, 2009 Dec. 24, 2009 Feb. 4, 2010 3 Appeal 2014-000743 Application 12/337,225 U.S. Patent 7,956,847 B2 Christie June 7,2011 Thomas Baudel et al., “Charad: Remote Control of Objects Using Free-Hand Gestures,” Communications of the ACM — Special Issue on Computer Augmented Environments: Back to the Real World, vol. 36, pp. 28—35 (July 1993) (hereinafter, “Baudel”). Koichi Ishibuchi et al., “Real Time Hand Gesture Recognition Using 3D Prediction Model,” Proceedings of the International Conference on Systems, Man and Cybernetics, vol. 17, pp. 324—328 (Oct. 17—20, 1993) (hereinafter, “Ishibuchi”). Lawrence D. Cutler et al., “Two-Handed Direct Manipulation on the Responsive Workbench,” 13D ’97 Proceedings of the 1997 Symposium on Interactive 3D Graphics ACM, pp. 107—114 (1997) (hereinafter, “Cutler”). Jakub Segen et al., “Gesture VR: Vision-Based 3D Hand Interface for Spatial Interaction,” Multimedia ’98 Proceedings of the Sixth ACM International Conference on Multimedia, pp. 455 464 (1998) (hereinafter, “Segen”). Cristina Manresa et al., “Hand Tracking and Gesture Recognition for Human-Computer Interaction,” Electronic Letters on Computer Vision and Image Analysis, Computer Vision Center, pp. 96—104 (2005) (hereinafter, “Manresa”). THE REJECTIONS Claim 1 stands rejected under 35 U.S.C. § 103(a) as being unpatentable over Ishibuchi, Harakawa, Wilson, Segen, and Manresa. Ans. 5—10. Claim 2 stands rejected under 35 U.S.C. § 103(a) as being unpatentable over Ishibuchi, Harakawa, Wilson, Segen, Manresa, and Platzker. Ans. 11—12. 4 Appeal 2014-000743 Application 12/337,225 Claim 4 stands rejected under 35 U.S.C. § 103(a) as being unpatentable over Ishibuchi, Harakawa, Wilson, Segen, Manresa, Platzker, andBallare. Ans. 12. Claims 10-13 stand rejected under 35 U.S.C. § 103(a) as being unpatentable over Ishibuchi, Harakawa, Wilson, Segen, Manresa, and Bell. Ans. 13-14. Claim 11 stands rejected under 35 U.S.C. § 103(a) as being unpatentable over Ishibuchi, Harakawa, Wilson, Segen, Manresa, and Cutler. Ans. 15—16. Claim 41 stands rejected under 35 U.S.C. § 103(a) as being unpatentable over Ishibuchi, Harakawa, Wilson, Hotelling, and Radivojevic. Ans. 16—22. Claim 42 stands rejected under 35 U.S.C. § 103(a) as being unpatentable over Ishibuchi, Harakawa, Wilson, and Baudel. Ans. 22—26. Claim 43 stands rejected under 35 U.S.C. § 103(a) as being unpatentable over Ishibuchi, Harakawa, Wilson, Baudel, and Delean. Ans. 27—28. Claim 45 stands rejected under 35 U.S.C. § 103(a) as being unpatentable over Ishibuchi, Harakawa, Wilson, Radivojevic, and van Cruyningen. Ans. 28—33. Claim 46 stands rejected under 35 U.S.C. § 103(a) as being unpatentable over Ishibuchi, Harakawa, Wilson, and Underkoffler. Ans. 33— 38. Claim 47 stands rejected under 35 U.S.C. § 103(a) as being unpatentable over Ishibuchi, Harakawa, Wilson, Underkoffler, and Christie. Ans. 38—39. 5 Appeal 2014-000743 Application 12/337,225 Claim 50 stands rejected under 35 U.S.C. § 103(a) as being unpatentable over Ishibuchi, Harakawa, Wilson, Segen, and Radivojevic. Ans. 39-44. Claim 51 stands rejected under 35 U.S.C. § 103(a) as being unpatentable over Ishibuchi, Harakawa, and Wilson. Ans. 44-49. Claim 52 stands rejected under 35 U.S.C. § 103(a) as being unpatentable over Ishibuchi, Harakawa, Wilson, Hildreth, and Himmel. Ans. 49-55. ANALYSIS As we discussed above, the claims on appeal relate to methods for providing an input (e.g., command) to a device in response to detecting a particular hand gesture. With respect to the claims on appeal, the Examiner relies on the combination of Ishibuchi, Harakawa, and Wilson for claim recitations relating to illuminating hand gestures, generating silhouette images, using the silhouette images to determine three-dimensional features of the hand gestures, determining input gestures associated with the hand gestures, and comparing the input gestures with predefined gestures in a library. Final Act. 2—7. The Examiner also finds that it would have been apparent to a skilled artisan, in view of Ishibuchi, Harakawa, and Wilson, that “various arbitrary hand gestures [e.g., input gestures] can be predefined to correspond to various arbitrary application [e.g., device] inputs.” Final Act. 7, 12—13, 23—24, 29-30, 35, 41, 46, 51; see also Ans. 69 (citing KSR Inti Co. v. Teleflex Inc., 550 U.S. 389, 401 (2007)). In view of the Examiner’s findings and conclusions, which we adopt herein, the Examiner has shown that claims 1, 2, 4, 10—13, 19, 41—43, 45^47, and 50-51 would 6 Appeal 2014-000743 Application 12/337,225 have been obvious to a skilled artisan, and Appellants have not persuaded us the Examiner erred in rejecting these claims. The Examiner also identifies additional prior art references that describe specific hand gestures and specific device inputs in support of the obviousness rejections. Appellants argue that these additional references fail to disclose particular hand gestures in combination with particular device inputs as claimed. App. Br. 26—59. Such arguments are unpersuasive because, as we discussed above, the Examiner has shown that a skilled artisan would have understood, in view of Ishibuchi, Harakawa, and Wilson, that various arbitrary input gestures could be predefined to correspond to various arbitrary device inputs, and the results would have been predictable. See, e.g., Ans. 69 (citing KSR Int’l Co., 550 U.S. at 401). We, therefore, sustain the Examiner’s rejections of claims 1, 2, 4, 10- 13, 19, 41—43, 45^17, and 50-52. Also, we were not persuaded by Appellant’s arguments that the additional references relied on by the Examiner failed to teach or suggest specific input gestures and input devices as claimed, and we highlight our findings below. 1. Claims 1, 2, 4, 10—13, and 19 With respect to claim 1, the Examiner correctly finds that because the claim recites “at least one of moving in six-degrees of freedom and moving one of upwards and downwards,” (emphasis added) it is satisfied by a teaching or suggestion of a gesture involving moving a bare hand one of upwards and downwards. Ans. 55. With respect to this latter claimed gesture, Appellants argue that Segen’s disclosure of pointing a finger up or down does not satisfy the claim because moving a finger up or down is not 7 Appeal 2014-000743 Application 12/337,225 the same as moving a hand up or down. App. Br. 28. The Examiner finds that under the broadest interpretation of the claim language, “[pjointing a finger up or down can be considered an input gesture comprising the bare hand ‘moving one of upwards and downwards.’” Ans. 55—56. We agree with the Examiner. Neither the claim language nor the Specification limit moving a bare hand in a manner that excludes moving a portion of the hand, such as a finger. We, therefore, sustain the Examiner’s rejection of claim 1, and claims 2, 4, 10-13, and 19 depending therefrom. 2. Claim 41 Appellants argue that although Hotelling and Radivojevic disclose “motion of both the index finger and the thumb to implement zoom commands,” neither reference teaches or suggests performing the claimed hand gesture without contacting the background surface. App. Br. 30. In particular, Appellants argue that Hotelling is directed to gestures for a touch screen device, and discloses that the zoom command is implemented while the thumb and index finger are in contact with the background surface. Id. at 31. We are not persuaded by this argument because the Examiner relies on Radivojevic, not Hotelling, for the feature of not contacting the background surface. Final Act. 19; Ans. 57. We, therefore, sustain the Examiner’s rejection of claim 41. 3. Claim 42 Appellants argue that Baudel’s Start/Stop Auto Play feature fails to teach or suggest the feature of “providing one of a page-up input and a page- down input associated with a document,” as recited in claim 42, because “starting and stopping the auto-play of a presentation is not the same thing as 8 Appeal 2014-000743 Application 12/337,225 performing a page-up and page-down device input.” App. Br. 33. In support of its argument, Appellants identity disclosure in Baudel of a separate hand gesture for performing Next Page and Previous Page device inputs, implying that Next Page and Previous page are page-up and page- down features, whereas Start/Stop Auto Play is not. Id. We are not persuaded by this argument. Appellants do not explain sufficiently why, applying a broadest reasonable interpretation, the next slide of a presentation is not the next page of a document (e.g., a “page-down”). Appellants also argue that claim 42 requires that the hand gesture be performed prior to each “page-up” device input, whereas Baudel’s Auto Play feature automatically advances through a plurality of slides in response to the hand gesture. App. Br. 33. Appellants’ argument is not commensurate with the scope of claim 42. Claim 42 recites providing “at least one device input responsive to the provided input gesture comprising providing one of a page-up input and a page-down input.” In view of the recitation “at least one,” initiating one command responsive to the hand gesture that results in advancing one slide is sufficient to satisfy this claim limitation. Ans. 58. Claim 42 does not preclude additional commands to advance to additional slides in response to the hand gesture. Ans. 58. Appellants also argue that Baudel advances to the next slide only after a predetermined time has lapsed, and therefore the command to advance to the next slide is not “responsive” to the hand gesture. App. Br. 34—35. This argument is not commensurate with the scope of claim 42, which does not restrict the amount of time in which the input to advance to the next page must be provided. Ans. 59. We, therefore, sustain the Examiner’s rejection of claim 42. 9 Appeal 2014-000743 Application 12/337,225 4. Claim 43 Claim 43 depends from claim 42, and further recites a second hand gesture comprising performing the first hand gesture (of claim 42) while concurrently moving the hand forward. Claim 42 also further recites that the “at least one device input” comprises providing a plurality of page-down inputs in response to the second hand gesture. Appellants argue that, given that Baudel’s Start/Stop Auto Play feature is directed to automatically advancing multiple slides, it would not have been obvious to introduce a second hand gesture in Baudel to advance multiple slides. App. Br. 37. We are not persuaded by Appellants’ argument. The Examiner finds that Baudel’s Auto Play feature provides a plurality of commands to advance to the next slide (e.g., “a plurality of page-down inputs”) in response to a hand gesture. Final Act. 25. The hand gesture does not include concurrently moving the hand forward, but the Examiner relies on Delean for the feature of concurrently moving a hand forward. Id. The Examiner also finds that Baudel, in other embodiments, discloses that hand gestures can include motion forward. Id. The Examiner finds that it would have been obvious “to tweak particular gestures so as to incorporate elements from other gestures that are either taught by the prior art or generally known; the resulting gesture would be predictable.” Ans. 60 (citing KSR Int’l Co., 550 U.S. at 401). Appellants have not persuaded us that the Examiner erred in finding that modifying the hand gesture for initiating Auto Play in Baudel to include forward motion would have yielded an unpredictable result or been non-obvious. KSR Int’l Co., 550 U.S. at 401. We, therefore, sustain the Examiner’s rejection of claim 43. 10 Appeal 2014-000743 Application 12/337,225 5. Claim 45 Claim 45 involves a hand gesture comprising alternating from a flat open hand to a closed fist, then back to an open hand. Claim 45 recites providing at least one device input responsive to the claimed gesture comprising “toggling between a list of menu items” in an application shown on a display. Appellants argue that the Examiner erred in finding that Radivojevic’s disclosure of alternating from a flat open hand to a closed fist to toggle between a list of menu items renders obvious claim 45. App. Br. 40. According to Appellants, alternating the hand back to a flat open hand is “recited in claim 45 as being an emphatic gesture step, one that requires effort by a user to maintain, and does not correspond to a ‘neutral pose.’” Id. We are not persuaded by Appellants’ arguments. Figure 6 of the Specification, gesture 252, depicts a hand gesture comprising a “first gesture,” e.g., a flat open hand followed by “second gesture,” e.g., a closed fist, followed by the “first gesture.” The Specification discloses that a user can toggle between menu items each time a user alternates from the first gesture to the second gesture. Spec. 1 53. In light of the Specification, therefore, the gesture of alternating from a flat open hand to a closed fist results in toggling a menu item, and returning the hand to an open hand enables the user to perform the step again of alternating from a flat open hand to a closed fist to toggle to another menu item. Similarly, the Examiner finds that a user of Radivojevic’s system would return the hand to a flat open hand to perform the gesture again of alternating from a flat open hand to a closed fist to continue to toggle between menu items. Ans. 63. We, therefore, sustain the Examiner’s rejection of claim 45. 11 Appeal 2014-000743 Application 12/337,225 6. Claims 46 and 47 Claim 46 depends from claim 47. At issue with respect to claims 46 and 47 is whether panning an image, as taught by Underkoffler, satisfies the claim limitation of scrolling at least one page of a document corresponding to selected visual data as recited in claim 46. Appellants argue that the terms “pan” and “scroll” have different definitions. App. Br. 45. We are not persuaded by Appellants’ arguments. In particular, the Examiner finds that Underkoffler describes a method to navigate a large panoramic display image. Ans. 65. We do not find error in the Examiner’s finding that the image in Underkoffler is “at least one page of a document” as claimed. Id. at 66. The Examiner further finds, and we agree, that Underkoffler teaches that particular hand gestures result in panning the image, and that in the context of Underkoffler, this involves scrolling the image. Id. at 65—66. Appellants’ reliance on on-line definitions taken from a website, without providing evidence that the definitions were applicable in the time frame of the invention date (App. Br. 45), does not address sufficiently the teachings of Underkoffler, and in particular, whether the resulting navigation of a large image is considered “scrolling” as that term is used in the claims and Specification. With respect to claim 47, Appellants further assert that the combination of Underkoffler and Christie fails to teach or suggest the claimed limitations. Id. at 47-48. Appellants, however, do not explain sufficiently why the asserted combination fails to teach or suggest what is claimed, but rather recite the claim language without providing substantive argument. Id. We, therefore, sustain the Examiner’s rejections of claims 46 and 47. 12 Appeal 2014-000743 Application 12/337,225 7. Claim 50 Claim 50 involves performing a particular hand gesture resulting in a device input comprising returning to an immediately preceding web-site in an Internet browser. The Examiner relies on Segen, directed to a controlling movement in a video game, for the claimed hand gesture, and Radivojevic for the feature of returning to an immediately preceding website in response to a hand gesture. Ans. 68—70. The Examiner finds that it would have been obvious to a skilled artisan that various arbitrary hand gestures could be predefined to correspond to various arbitrary device inputs. Id. Appellants argue that the Examiner employed impermissible hindsight in formulating the rejection, and that the Examiner’s determination is merely conclusory. App. Br. 49-50. We are not persuaded by Appellants’ argument. Merely enumerating different combinations of hand gestures and corresponding inputs that are not explicitly mapped in the prior art does not persuade us the subject matter would have been non-obvious in view of the prior art’s disclosure that it was well known to correlate hand gestures with inputs. Appellants have not persuaded us that the Examiner erred in finding that implementing the hand gesture taught in Segen with the device input taught in Radivojevic amounts to a combination of known elements that does nothing more than yield predictable results, and therefore would have been obvious. Ans. 69 (citing KSR Int’l Co., 550 U.S. at 401). We, therefore, sustain the Examiner’s rejection of claim 50. 8. Claim 51 Claim 51 recites, as part of a first gesture, a bare hand rapidly moving in a direction perpendicular to an extended finger, then the bare hand rapidly moving back to its initial position. With respect to this recitation, the 13 Appeal 2014-000743 Application 12/337,225 Examiner relies on Harakawa, which the Examiner finds teaches a hand gesture involving a bare hand rapidly moving in a direction perpendicular to an extended finger. Final Act. 46-47. The Examiner finds that Harakawa also teaches the gesture will naturally involve the user’s bare hand rapidly moving back to its initial position. Id. (citing Harakawa 33:58—61 (“The natural movement of a persons [sic] hand after performing the forward click motion or the backward click motion, is to try to return to the position (neutral position) prior to the click motion.”)). Appellants argue, however, that even if a user returns the bare hand to the neutral position, Harakawa teaches that this motion is not part of the hand gesture that is determined in the claimed method because Harakawa teaches that this motion is ignored in order to avoid mistaking a return to neutral position for a backward or forward click motion. App. Br. 54 (citing Harakawa 33:58—34:12). We are not persuaded by Appellants’ argument. In support of the rejection, the Examiner relies on Harakawa’s disclosure that one method of ignoring the return to neutral position involves storing a distance k corresponding to neutral position and stopping the detection of a click motion until the value of k reaches the value corresponding to neutral position (e.g., it is determined the hand has returned to the neutral position). Final Act. 57 (citing Harakawa 33:58—34:12). Accordingly, this supports the Examiner’s finding that Harakawa teaches that a determination is made of an “input gesture” that includes determining that a hand has been returned to its neutral position. Ans. 70—71. We, therefore, sustain the Examiner’s rejection of claim 51. 14 Appeal 2014-000743 Application 12/337,225 9. Claim 52 Claim 52 involves a first sub-gesture comprising an extended index finger pointing to a portion of displayed visual content and a device input comprising scrolling visual data associated with the portion of displayed visual content in response to the first sub-gesture while a second sub-gesture is maintained. The Examiner relies on Hildreth for the claimed first and second sub-gestures, finding that Hildreth teaches providing a mouse-down event in response to the claimed sub-gestures. Final Act. 52—53. The Examiner acknowledges that Hildreth does not expressly disclose scrolling visual data in response to a mouse-down event, but finds that it was well known to those of ordinary skill in the art that maintaining a mouse-down event, as described in Hildreth, while positioning a cursor over up/down buttons on a scroll bar would cause the visual data associated with the scroll bar to scroll. Id. at 52. In support of the rejection, the Examiner relies on Himmel, which teaches maintaining a mouse-down event to scroll data using a scroll bar. Id. (citing Himmel 120). Accordingly, the Examiner concludes it would have been obvious that performing the mouse-down event in Hildreth in response to the claimed hand gestures would result in scrolling visual data as claimed. Id. at 51—53. Appellants argue that Himmel fails to disclose “anything regarding an input gesture.” App. Br. 57. This argument is unpersuasive because the Examiner does not rely on Himmel for teaching an input gesture. As discussed above, the Examiner relies on Hildreth for teaching the claimed input gestures and for teaching a mouse-down event. The Examiner relies on Himmel to support the finding that a mouse-down event, as described in 15 Appeal 2014-000743 Application 12/337,225 Hildreth, would result in scrolling visual data if the mouse-down event were to occur over up/down buttons on a scroll bar. We, therefore, sustain the Examiner’s rejection of claim 52. DECISION The Examiner’s decision rejecting claims 1, 2, 4, 10—13, 19, 41—43, 45—47, and 50-52 is affirmed. No time period for taking any subsequent action in connection with this appeal may be extended under 37 C.F.R. § 1.136(a)(l)(iv). AFFIRMED 16 Copy with citationCopy as parenthetical citation