Ex Parte Mathan et alDownload PDFPatent Trial and Appeal BoardFeb 25, 201311336152 (P.T.A.B. Feb. 25, 2013) Copy Citation UNITED STATES PATENT AND TRADEMARK OFFICE ____________ BEFORE THE PATENT TRIAL AND APPEAL BOARD ____________ Ex parte SANTOSH MATHAN, MICHAEL C. DORNEICH, PATRICIA M. VERVERS, and STEPHEN D. WHITLOW ____________ Appeal 2011-009567 Application 11/336,152 Technology Center 3700 ____________ Before DONALD E. ADAMS, STEPHEN WALSH, and ULRIKE W. JENKS, Administrative Patent Judges. JENKS, Administrative Patent Judge DECISION ON APPEAL This is an appeal under 35 U.S.C. § 134 involving claims to a rapid serial visual presentation method and system that utilizes user sensitive pacing to classify presented images. The Patent Examiner has rejected the claims for anticipation and obviousness. We have jurisdiction under 35 U.S.C. § 6(b). We affirm. Appeal 2011-009567 Application 11/336,152 2 STATEMENT OF THE CASE The Specification is directed to: [A] method and system for user sensitive pacing in an image triage that is based on rapid serial visual presentation. The physical or cognitive state of a user is monitored during the image triage, and images are shown to the user at a predetermined image presentation rate. The image presentation rate is adjusted during the image triage in response to the physical or cognitive state of the user, so that the rate is adapted to the ability of a user to process images effectively. (Spec. ¶ 0012.) Claims 1-4, 6, 8-13, 26, and 28 are on appeal, and can be found in the Claims Appendix of the Appeal Brief (App. Br. 18-21). Claims 1 and 10 are independent claims. Claim 1 is representative of the claims on appeal, and reads as follows: 1. A method for user sensitive pacing in an image triage, comprising: using one or more sensors to monitor a physical state and a cognitive state of a user; detecting evoked response potential (ERPs) in the user using a plurality of different ERP detection techniques and an integrated real- time ERP feature detection system; using a feature fusion system to classify the ERPs based on a fusion of features derived from each of the different ERP detection techniques; showing images to the user on a display screen and at an image presentation rate; using an analyst sensitive pacing system to adjust the image presentation rate of the images shown on the display screen in real- time, while showing the images to the user, in response to at least one Appeal 2011-009567 Application 11/336,152 3 of the monitored physical state or monitored cognitive state of the user; using an analyst sensitive prioritization system to selectively re- sequence one or more of the images for reexamination by the user in response to at least the monitored physical state; and using the analyst sensitive prioritization system to prioritize the images shown to the user based on the classified ERPs and the monitored physical state and monitored cognitive state of the user. The Examiner has rejected the claims as follows: I. claims 1, 6, 8-13, 26, and 28 under 35 U.S.C. § 102(b) as being anticipated by Parra; 1 and II. claims 1-4, 6, 8-13, 26, and 28 under 35 U.S.C. § 103(a) as unpatentable over Parra in view of Sajda. 2 As Appellants do not argue the claims separately, we focus our analysis on claim 1, and claims 2-4, 6, 8-13, 26, and 28 stand or fall with that claim. 37 C.F.R. § 41.37 (c)(1)(iv). 1 Lucas C. Parra et al., Response Error Correction-A Demonstration of Improved Human-Machine Performance Using Real-Time EEG Monitoring, 11 IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING 173 (2003). 2 Paul Sajda et al., High-throughput Image Search via Single-trial Event Detection in a Rapid Serial Visual Presentation Task, FIRST INTERNATIONAL IEEE EMBS CONFERENCE ON NEURAL ENGINEERING 7 (2003). Appeal 2011-009567 Application 11/336,152 4 I. The Issue The Examiner takes the position that Parra disclosed the invention as claimed. The Examiner asserts that: Parra suggests the use of his system in rapid visual serial presentations, RSVP as indicated by Sajda, for error corrections and the disclosed system uses classified ERPs i.e. the negative fronto-central deflection in EEG signal is classified . . . [and] adjusting HCI or human-computer-interface i.e. prioritizing the information/images presented to the user based on the findings of the study. (Ans. 5.) Appellants assert that Parra is “wholly devoid of any teaching or suggestion of prioritizing the images shown to the user based on classified ERPs, let alone based on classified ERPs and the monitored physical state and monitored cognitive state of the user.” (Reply Br. 5.) The issue is: Does Parra disclose the limitation to “selectively re- sequence one or more images for reexamination” as set out in the claim? Findings of Fact The following findings of fact (FF) are supported by preponderance of the evidence of record. 1. Parra disclosed that: The goal of an adaptive interface is to estimate variables correlated with human performance and adapt the HCI [human- computer interfaces] accordingly (e.g., adjust speed of display, Appeal 2011-009567 Application 11/336,152 5 provide appropriate cues, automatically correct errors. etc.) Several behavioral and physiological measures, such as reaction time, eye motion, and pupil dilation, have been proposed as variables having utility for adapting an HCI [1], [2]. More recently, research in neuroimaging has identified electroencephalography (EEG) signals that are correlated with attention [3], memory encoding [4], motor imagery [5], perceived error and/or conflict [6], perception/recognition [7] and which, therefore, might be useful for such adaptation. (Parra 173; Ans. 5.) 2. Parra discloses “a brain-computer interface (BCI) capable of monitoring a subject's cognitive state associated with specific observable events.” (Parra 173.) 3. Parra‟s subjects performed the following task: [A] high-throughput, alternative forced choice visual discrimination task. In this task, a subject discriminates between two visual stimuli by pressing one of two buttons. When subjects attempt to minimize their response time, they often commit errors that are perceived shortly after the button- push response. Interestingly, such perceived errors are accompanied by a negative fronto-central deflection in the EEG signal. This signal is known as the error related negativity (ERN). (Parra 173; Ans. 4-5.) 4. Parra disclosed that “[t]he goal of measuring the ERN is to monitor a subject's task specific error rate and adapt an HCI to maximize overall performance.” (Parra 176, see conclusion.) Appeal 2011-009567 Application 11/336,152 6 5. Parra disclosed that “[e]xisting methods typically use electrooculogram (EOG) electrodes as a reference. Unfortunately, in addition to eye motion, EOG signals also contain frontal cortical activity which should not be subtracted.” (Parra 173; Ans. 4.) 6. Parra disclosed removing eye blink noise from the EEG data collected. Conventional algorithms detect eye blinks to simply discard the corresponding segment of data. This is not feasible in practice given the frequent eye motion in most real-world situations. A better approach is to subtract the artifacts using linear regression algorithms. . . . We construct a better reference signal using a linear combination of all electrodes, thereby increasing the power of the eye blink activity in the reference. (Parra 173; see also Fig. 1; Ans. 6.) 7. Parra disclosed that “[o]ur method partitions the 64 recorded channels into two sets: 1) the EOG and frontal electrodes containing strong eye blink signals and 2) the remaining parietal, temporal, and occipital electrodes with weaker eye blink signal contributions.” (Parra 174.) 8. Parra disclosed denoising EEG signals of the ERN. “One challenge in detecting the cognitive state of a subject via single trial EEG is the inherently low signal-to-noise ratio (SNR). . . . Here, we propose to estimate noise statistics by modeling such signal properties with a hierarchical probability model.” (Parra 174; see also Fig. 2.) Appeal 2011-009567 Application 11/336,152 7 9. Parra disclosed “[e]xamples of original and HMT [Hidden-Markov- Trees analysis] denoised EEG signals after eye blink removal.” (Parra 175; see also Fig. 3.) 10. Parra‟s disclosed processing sequence is eye blink removal, followed by HMT denoising, or linear classification. (Parra 175-176.) Principle of Law In order for a prior art reference to serve as an anticipatory reference, it must disclose every limitation of the claimed invention, either explicitly or inherently. See In re Schreiber, 128 F.3d 1473, 1477 (Fed. Cir. 1997). To anticipate, every element and limitation of the claimed invention must be found in a single prior art reference, arranged as in the claim. Karsten Mfg. Corp. v. Cleveland Golf Co., 242 F.3d 1376, 1383 (Fed. Cir. 2001). Analysis We agree with the Appellants‟ position that the Examiner has not established that Parra disclosed the limitation to “selectively re-sequence one or more of the images for reexamination by the user in response to at least the monitored physical state,” as required by claim 1. “[A]bsence from the reference of any claimed element negates anticipation.” Kloster Speedsteel AB v. Crucible, Inc., 793 F.2d 1565, 1571 (Fed. Cir. 1986). The preponderance of evidence on this record fails to support Examiner's finding that Parra teaches Appellants' claimed invention. The Appeal 2011-009567 Application 11/336,152 8 rejection of claims 1, 6, 8-13, 26, and 28 under 35 U.S.C. § 102(b) as being anticipated by Parra is reversed. II. The Issue The Examiner takes the position Parra disclosed all the claimed elements but for adjusting the speed for the display in real time. However, the Examiner finds that “[i]t would have been obvious to one of ordinary skill at the time of the invention to use the results of the physical/cognitive state assessment for adjusting speed of display as a way of adapting the human-computer interface with human performance.” (Ans. 8.) Appellants assert that Parra “fails to disclose using the analyst sensitive prioritization system to prioritize the images shown to the user based on the classified ERPs and the monitored physical state and monitored cognitive state of the user” and Sajda does not cure the deficiencies of Parra (App. Br. 12). Appellants assert that “[d]escribing the removal of eye blinks from EEG signals so that ERN signals are not masked is not even remotely similar to detecting ERPs in a user using a plurality of different ERP detection techniques, let alone doing so with an integrated real-time ERP feature detection system.” (Reply Br. 2-3.) The issue is: Has the Examiner established by a preponderance of the evidence that the combination of Parra in view of Sajda renders obvious the method of claim 1? Appeal 2011-009567 Application 11/336,152 9 Additional Findings of Fact 11. The Specification provides that “[a]n ERP [evoked response potential] is a brief change in the brain's electrical potential in response to critical events in the environment.” (Spec. ¶ 0002.) 12. The Specification provides that the fusion detection approach relies on extracting the most informative features for each ERP detected: As shown in Figure 1, these include linear projections 110 and nonlinear matched filter projections 112 of raw data, and wavelet-based time frequency distributions 114 of power in EEG signals. This initial pool of features 120 can be diversified to capture a broad range of critical features present in the EEG signals. These features are then evaluated for their optimally discriminatory value using MI [mutual information] based feature ranking algorithms. Feature extraction using MI techniques can be carried out in conjunction with the system calibration process. Once the optimal feature subset is identified, real-time ERP classification will be extremely efficient. (Spec. ¶ 0025.) 13. Sajda disclosed that “a linear discriminator can be used to detect signatures of visual recognition events, and that such signatures can be successfully used to reprioritize a sequence of images to increase search efficiency.” (Sajda 7.) 14. Sajda disclosed that “EEG was recorded in an electrostatically shielded room using an 87-channel cap containing 79 Ag/AgCl scalp e1ectrodes (Electro-Cap International, Eaton, Ohio). Electrooculogram was Appeal 2011-009567 Application 11/336,152 10 recorded above and below the left eye and at the outer canthi of both eyes.” (Sajda 7.) 15. Sajda disclosed experimental trials in which the subjects were presented images at varying frame rates. Frame rate increased in each trial block; 5 images per sec (200 msec/image), 10 images per sec (100 msec/image), and 20 images/sec (50 msec/image). Subjects were instructed to depress the left button of a generic 3-button mouse with their right index finger while the fixation cross was present, and release the button as soon as they perceived a target. (Sajda 7; Ans. 8.) 16. Sajda disclosed that “as early as between 100-200 msec after image presentation, we see a signature of visual target detection even though an overt response (button release) does not occur until much later (approx 600 msec in subject 1 and 400 msec in subject 2).” (Sajda 8.) 17. Sajda disclosed: We demonstrate[d] that the EEG signatures detected via the linear discriminator can be used to reprioritize the image sequence, placing detected targets in the front of the image stack. . . . . . . [B]oth a visual and motor component provide robust signatures for detection of targets and reprioritization of the image sequence. (Sajda 8.) Appeal 2011-009567 Application 11/336,152 11 Principle of Law “The combination of familiar elements according to known methods is likely to be obvious when it does no more than yield predictable results.” KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398, 416 (2007). If a person of ordinary skill can implement a predictable variation, § 103 likely bars its patentability. For the same reason, if a technique has been used to improve one device, and a person of ordinary skill in the art would recognize that it would improve similar devices in the same way, using the technique is obvious unless its actual application is beyond his or her skill. Id. at 417. It is proper to “take account of the inferences and creative steps that a person of ordinary skill in the art would employ.” Id. at 418. See also id. at 421 (“A person of ordinary skill is also a person of ordinary creativity, not an automaton.”). Analysis Claim 1 recites the limitation that “the ERPs [are] based on a fusion of features derived from each of the different ERP detection techniques.” In addition, claim 1 recites the limitation of a “feature fusion system.” “[D]uring examination proceedings, claims are given their broadest reasonable interpretation consistent with the specification.” In re Hyatt, 211 F.3d 1367, 1372 (Fed. Cir. 2000). The Specification provides that an evoked response potential (ERP) “is a brief change in the brain‟s electrical potential in response to critical Appeal 2011-009567 Application 11/336,152 12 events in the environment.” (FF 11) We interpret this to mean that any detectable change in electrical potential in response to either a positive or negative event in the environment is encompassed by this term. A “„feature fusion system‟” is not an art recognized term (Ans. 10) and the Specification does not specifically define this term. Appellants, however, direct us to Figures 1 and 4, as well as to pages 6, 7, and 15 of the Specification (App. Br. 6) for providing guidance to the meaning of this term. Figure 1 of the Specification provides that “[f]eature fusion: pick optimally discriminative features identified by MI technique during calibration.” (Spec. Figure 1) The mutual information (MI) is: an objective measure of the dependency or nonlinear correlation between two or more random quantities. This suggests that the larger the MI between a set of EEG-based features and the class labels (e.g., background EEG vs. ERP), the better the expected classification accuracy. Hence, the design of a nonlinear projection that maximizes the mutual information between the EEG projection and class labels can be used to create a filter that optimally separates ERP from background EEG activity. (Spec. 5.) Based on the disclosure provided in the Specification, we interpret a “feature fusion system,” or a classification using a “fusion of features” to be a process were the raw EEG signal is gathered and analyzed for the presence of an electrical signal, specifically, an electrical potential that is measured in response to an event in the environment, and where the system separates this signal from the background EEG activity. Here, the stronger the electrical signal the more confident the label that can be applied Appeal 2011-009567 Application 11/336,152 13 to the signal. Thus, the “feature fusion system” takes the collected EEG data and applies different techniques to clean up the data signal and chooses the technique that provides the strongest electrical signal, after separating background noise, as a filter. We agree with the Examiner‟s position set out in the Answer (Ans. 6- 12). We agree with the Examiner that Parra disclosed using one or more sensors to monitor the physical and cognitive state of the subject (FFs 2, 5, 7, 8). Parra disclosed that perceived errors in a visual task by the subjects are associated with EEG deflections (FF 3). These EEG deflections (changes) meet the ERP limitation as defined by the Specification (FF 11). Parra disclosed detecting these EEG deflections (changes), ERP‟s, from the EEG signal using a variety of techniques (FFs 5-10; Ans. 10). Specifically, Parra first removes eye blink signal, and then cleans up the remaining signal using Hidden-Markov-Trees analysis, as well as applying linear classification to the data (FFs 9, 10). In addition, this data analysis is run in real time (Parra 173). While Sajda disclosed that visual detection can be seen as an EEG signal before an overt response is registered by the subject (FF 16). Sajda disclosed using this EEG signal information to reprioritize a sequence of images (FF 13). Sajda disclosed that “the detected EEG signature resulted in a more efficient prioritization of the imagery.” (Sajda 8.) Finally, Para disclosed that the goal of the human-computer-interface is to present the images in real time and to use the EEG information in order to Appeal 2011-009567 Application 11/336,152 14 adjust the speed of the display and automatically correct errors in a visual screening task (FFs 1, 4). We are not persuaded by Appellants assertion that Parra “fails to disclose (or even remotely suggest) any type of feature fusion system.” (Reply Br. 4.) As discussed above, we interpret a “feature fusion system” to be a process step were the raw EEG signal is gathered and analyzed for the presence of an electrical signal, specifically, an electrical potential that is measured in response to an event in the environment. As discussed above, an ERP is any electrical potential that can be measured in response to an environmental stimulus. Parra disclosed measuring an electrical potential associated with a forced choice visual discrimination task (FF 3). When trying to minimize response time to a particular visual task the subjects tended to make errors, and this error can be seen as a deflection (change) in the EEG signal (FF 3). Parra classifies this signal as an error related negativity signal (FF 3). Thus, Parra gathers raw EEG data, and removes the eye blink signal (FFs 9-10) and then subjects this data to additional statically classification in order to detect the signal associated with the error response (FFs 9, 10). Parra disclosed applying this process in real time (Parra 173). Additionally, Parra disclosed that by “constructing a classifier that estimates the class probability, we minimize the number of errors by assigning new trials to the class with the highest probability under the classifier.” (Parra 176.) Parra also disclosed that “[t]he parameters of the linear classifier and the detection threshold must be derived from initial training sequence and Appeal 2011-009567 Application 11/336,152 15 kept constant during operation.” Thus, Parra provides using a combination of analytical methods on the same EEG data in order to detect and classify the signal associated with an environmental event, and provides that the filter thresholds for classification must be set after trial runs, meeting the limitation of a “feature fusion system” as disclosed above. We are not persuaded by Appellants assertion that “the removal of eye blinks from EEG signals so that ERN signals are not masked is not even remotely similar to detecting ERPs in a user using a plurality of different ERP detection techniques.” (Reply Br. 3.) Both Parra and Sajda use EOG as well as EEG to capture electrical potentials during their visual screening experiments (FFs 5, 6, 7, 14). The Specification defines an ERP to be a change in the brains electrical potential in response to an environmental event (FF 11). The EEG signal is evaluated for the mutual information (MI) features and uses those algorithms that provide the optimally discriminatory values (FF 12), between signal and background (Spec. 5). The MI with the strongest signal is used to create a filter to detect ERP from background activity (Spec. 5). Parra applies this same analysis to collected EEG data from the experimental visual task (FF 3). Parra begins by subtracting the background eye-blinks and then cleans up the remaining signal using Hidden-Markov-Trees analysis or by running linear classification on the EEG data (FFs 5, 6, 8, 9, 10). Parra goes on to indicate that the detected ERN can be used to correct errors by classifying the signal and “assigning Appeal 2011-009567 Application 11/336,152 16 new trials to the class with the highest probability under the classifier.” (Parra 176.) We are also not persuaded by Appellants assertion that Parra fails to “disclose using the analyst sensitive prioritization.” The test for obviousness is what the combined teachings of the references as a whole would have suggested to those of ordinary skill in the art. In re Keller, 642 F.2d 413, 425 (CCPA 1981). Parra disclosed that “[t]he goal of an adaptive interface is to estimate variables correlated with human performance and adapt the HCI [human-computer interfaces] accordingly (e.g., adjust speed of display, provide appropriate cues, automatically correct errors. etc.)” (FF 1.) Parra provides that “[w]e are currently exploring other applications of the proposed real-time monitoring. For instance, we argue that one may be able to increase the speed of visual search for image analysts by detecting the activity associated with fast visual recognition.” (Parra 173.) We agree with the Examiner‟s position that Parra teaches that “[u]ser performance is improved by altering the user interface with actions such as adjusting display speed.” (Ans. 11.) The Examiner concludes that the combination of references teaches: Image Reprioritization wherein Sadja [sic] et al. teaches that analyzing evoked response potentials can be used for reorganizing images. Accordingly the combination of ERP classification techniques in Parra to improve error detection in rapid serial visual presentation interfaces as disclosed in Sadja [sic] et al. would include improving other features of the system reliant on ERP detection such as the reprioritization of images. Appeal 2011-009567 Application 11/336,152 17 (Ans. 11-12.) We conclude that the preponderance of the evidence of record supports the Examiner‟s conclusion that the combination of Parra in view of Sajda renders obvious the user sensitive pacing in an image triage of claim 1. We thus affirm the rejection of claim 1 under 35 U.S.C. § 103(a) as being obvious, we also affirm the rejection of claims 2-4, 6, 8-13, 26, and 28. 37 C.F.R. § 41.37(c)(iv). SUMMARY We reverse the rejection of claims 1, 6, 8-13, 26 and 28 under 35 U.S.C. § 102(b) as anticipated by Parra. We affirm the rejections of claims 1-4, 6, 8-13, 26, and 28 under 35 U.S.C. § 103(a) over Parra in view of Sajda. TIME PERIOD FOR RESPONSE No time period for taking any subsequent action in connection with this appeal may be extended under 37 C.F.R. § 1.136(a). AFFIRMED cdc Copy with citationCopy as parenthetical citation