2021005059 (P.T.A.B. Dec. 7, 2021)

International Business Machines Corporation

Patent Trials and Appeals BoardDec 7, 2021

2021005059 (P.T.A.B. Dec. 7, 2021)

UNITED STATES PATENT AND TRADEMARK OFFICE UNITED STATES DEPARTMENT OF COMMERCE United States Patent and Trademark Office Address: COMMISSIONER FOR PATENTS P.O. Box 1450 Alexandria, Virginia 22313-1450 www.uspto.gov APPLICATION NO. FILING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO. 15/258,298 09/07/2016 Paul E. Brennan AUS920160341US1 6234 65362 7590 12/07/2021 TERRILE, CANNATTI & CHAMBERS, LLP IBM Austin P.O. BOX 203518 AUSTIN, TX 78720 EXAMINER BALDWIN, RANDALL KERN ART UNIT PAPER NUMBER 2125 NOTIFICATION DATE DELIVERY MODE 12/07/2021 ELECTRONIC Please find below and/or attached an Office communication concerning this application or proceeding. The time period for reply, if any, is set in the attached communication. Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the following e-mail address(es): USPTO@dockettrak.com tmunoz@tcciplaw.com PTOL-90A (Rev. 04/07) UNITED STATES PATENT AND TRADEMARK OFFICE BEFORE THE PATENT TRIAL AND APPEAL BOARD Ex parte PAUL E. BRENNAN, SCOTT R. CARRIER, and MICHAEL L. STICKLER Appeal 2021-005059 Application 15/258,298 Technology Center 2100 Before JUSTIN BUSCH, BETH Z. SHAW, and JOYCE CRAIG, Administrative Patent Judges. SHAW, Administrative Patent Judge. DECISION ON APPEAL STATEMENT OF THE CASE Pursuant to 35 U.S.C. § 134(a), Appellant1 appeals from the Examiner’s decision to reject claims 1–4, 6–12, 14–18, and 20–23. We have jurisdiction under 35 U.S.C. § 6(b). We AFFIRM. 1 We use the word Appellant to refer to “applicant” as defined in 37 C.F.R. § 1.42(a) (2012). Appellant identifies the real party in interest as International Business Machines Corporation. Appeal Br. 1. Appeal 2021-005059 Application 15/258,298 2 CLAIMED SUBJECT MATTER The claims are directed to verifying ground truth data. Claim 1, reproduced below, is illustrative of the claimed subject matter: 1. A method of verifying ground truth data, the method comprising: receiving, by an information handling system, comprising a processor and a memory, ground truth data comprising a human-curated training set; performing, by the information handling system, annotation operations on the training set using an annotator to generate a machine-annotated training set; assigning, by the information handling system, low entropy relationship mentions from the machine-annotated training set to one or more clusters according to a feature vector similarity measure; analyzing, by the information handling system, the one or more clusters to prioritize clusters based on verification scores computed for each cluster by computing a verification score for each cluster as an Inter Annotator Agreement (IAA) score measuring how consistent annotations of a human subject matter expert (SME) are with annotations from a group of human SMEs for each cluster; and displaying, by the information handling system, machine- annotated training set examples associated with a prioritized cluster as prioritized review candidates to solicit verification or correction feedback from the human SME for inclusion in an accepted training set. Appeal 2021-005059 Application 15/258,298 3 REFERENCES The prior art relied upon by the Examiner is: Name Reference Date Chua US 2013/0024457 A1 Jan. 24, 2013 Buryak US 8,543,576 B1 Sept. 24, 2013 Byron US 2016/0180242 A1 June 23, 2016 Alexandre Davis et al., Named Entity Disambiguation in Streaming Data, Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics 815824 (2012) (“Davis”). REJECTIONS Claims 1–4, 6–12, 14–18, and 20–23 are rejected under 35 U.S.C. § 103 as being unpatentable over Byron, Davis, Buryak, and Chua. OPINION On this record, we see no error in the Examiner’s reliance on the cited references for collectively teaching or suggesting the elements recited in the claims. We adopt as our own the findings set forth by the Examiner in the Final Rejection and Answer. See Final Act. 2–41; Ans. 5–22. First, we find unavailing Appellant’s argument that Davis or Buryak fail to teach or suggest “assigning, by the information handling system, low entropy relationship mentions from the machine-annotated training set to one or more clusters according to a feature vector similarity measure,” as recited in claim 1. Appeal Br. 4–7. As the Examiner finds, and we agree, Davis teaches this disputed limitation. For example, Davis illustrates methods that generate clusters of co-referent entities using known relationships between entities, where information taken from the header of the messages leads to establishing relationships between users and building a co-reference graph. Ans. 13 (citing Davis, 816, 817, and 820). Davis Appeal 2021-005059 Application 15/258,298 4 describes that clusters of co-referent names are then built based on such similarity measures. Id. Davis teaches assigning low entropy relationship mentions to one or more partitions or clusters because its method “minimizes the average entropy of these two partitions.” Id. We also agree with the Examiner’s alternative finding that Buryak also teaches this disputed limitation, as described in the Answer at pages 15– 16. See Buryak, 1:26–30; 3:10–14; 4:7–13; 7:60–64 (describing methods that calculate a similarity score for each cluster, and clusters associated with a label that indicates the topic or issue associated with the majority of user documents included in the cluster). In the Reply Brief, Appellant argues for the first time that the Examiner’s claim interpretation is incorrect. Reply Br. 2–3. These arguments are entitled to no consideration because the Examiner’s claim interpretation was included in the Final Office Action (see Final Act. 2–5) and Appellant’s arguments regarding the claim interpretation were not presented in the opening brief. See 37 C.F.R. § 41.37(c)(1)(iv) (2013); Optivus Tech., Inc. v. Ion Beam Applications S.A., 469 F.3d 978, 989 (Fed. Cir. 2006); accord Ex parte Borden, 93 USPQ2d 1473, 1473-74 (BPAI 2010) (“informative”) (absent a showing of good cause, the Board is not required to address an argument newly presented in the reply brief that could have been presented in the principal brief on appeal); In re Google Tech. Holdings LLC, 980 F.3d 858, 862 (Fed. Cir. 2020) (failure to raise lexicography arguments compels a finding of forfeiture); Ans. 11 (“the above-noted broadest reasonable interpretation is not disputed by Appellant”). We find unavailing Appellant’s arguments (see Appeal Br. 8; Reply Br. 4) against Chua’s teachings for the reasons explained by the Examiner. Appeal 2021-005059 Application 15/258,298 5 In particular, we agree with the Examiner’s finding that Chua does not “teach away” from the claimed solution by merely describing different quality levels of inter annotator agreement on question and answer pairs. See Ans. 17–18. Although Chua notes, in part of paragraph 27, the existence of low inter annotator agreement for question and answer pairs in this particular embodiment, we fail to see—nor has Appellant shown—that Chua criticizes, discredits, or otherwise discourages using inter annotator agreement scores, to teach away from such scores in general. See Norgren Inc. v. Int’l Trade Comm’n, 699 F.3d 1317, 1326 (Fed. Cir. 2012); see also In re Kahn, 441 F.3d 977, 990 (Fed. Cir. 2006). To the extent Appellant contends otherwise, there is no persuasive evidence on this record to substantiate such a contention. Finally, we are not persuaded by Appellant’s general arguments against the combination of references. See Appeal Br. 8–10; Reply Br. 4. The Examiner proposes using prior art elements predictably according to their established functions—an obvious improvement. See KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398, 415–16 (2007); Ans. 20–22. Appellant has not demonstrated that the proposed combination would have been “uniquely challenging or difficult for one of ordinary skill in the art,” and we determine the proposed modification would have been well within the purview of the ordinarily skilled artisan for the reasons stated by the Examiner. Leapfrog Enters., Inc. v. Fisher-Price, Inc., 485 F.3d 1157, 1162 (Fed. Cir. 2007) (citing KSR, 550 U.S. at 418). Appeal 2021-005059 Application 15/258,298 6 Therefore, we are not persuaded that the Examiner erred in rejecting independent claim 1, and we sustain the rejection of claim 1.2 We also sustain the obviousness rejections of claims 2–4, 6–12, 14–18, and 20–23, which are not argued separately with particularity. CONCLUSION We affirm the rejections. DECISION SUMMARY In summary: Claims Rejected 35 U.S.C. § Reference(s)/Basis Affirmed Reversed 1–4, 6– 12, 14– 18, 20–23 103 Byron, Davis, Buryak, Chua 1–4, 6–12, 14–18, 20–23 TIME PERIOD FOR RESPONSE No time period for taking any subsequent action in connection with this appeal may be extended under 37 C.F.R. § 1.136(a). See 37 C.F.R. § 1.136(a)(1)(iv) (2013). AFFIRMED 2 In the event of further prosecution of this application, we leave it to the Examiner to consider a rejection under 35 C.F.R. § 101. Although the Board is authorized to reject claims under 37 C.F.R. § 41.50(b), no inference should be drawn when the Board elects not to do so. See Manual of Patent Examining Procedure (MPEP) §1213.02.