General Electric Company

Patent Trials and Appeals Board

Jan 25, 2021

2020005097 (P.T.A.B. Jan. 25, 2021)

UNITED STATES PATENT AND TRADEMARK OFFICE
UNITED STATES DEPARTMENT OF COMMERCE
United States Patent and Trademark Office
Address: COMMISSIONER FOR PATENTS
P.O. Box 1450
Alexandria, Virginia 22313-1450
www.uspto.gov
APPLICATION NO. FILING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO.
15/397,469 01/03/2017 Lalit Keshav Mestha WABP11531USORG1
552-541US
4124
91959 7590 01/25/2021
The Small Patent Law Group, LLC
225 S. Meramec, Suite 725
St. Louis, MO 63105
EXAMINER
PATTON, SPENCER D
ART UNIT PAPER NUMBER
3664
NOTIFICATION DATE DELIVERY MODE
01/25/2021 ELECTRONIC
Please find below and/or attached an Office communication concerning this application or proceeding.
The time period for reply, if any, is set in the attached communication.
Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the
following e-mail address(es):
Docket@splglaw.com
TransportationPatentCorrespondence@ge.com
john.kramer1@ge.com
PTOL-90A (Rev. 04/07)
UNITED STATES PATENT AND TRADEMARK OFFICE
____________

BEFORE THE PATENT TRIAL AND APPEAL BOARD
____________

Ex parte LALIT KESHAV MESTHA,
JAMES BROOKS, and PETER TU
____________

Appeal 2020-005097
Application 15/397,469
Technology Center 3600
____________

Before ANTON W. FETTING, NINA L. MEDLOCK, and
MATTHEW S. MEYERS, Administrative Patent Judges.

MEDLOCK, Administrative Patent Judge.

DECISION ON APPEAL

STATEMENT OF THE CASE
Appellant1 appeals under 35 U.S.C. § 134(a) from the Examiner’s
final rejection of claims 1, 3, 6–12, and 14–25. We have jurisdiction under
35 U.S.C. § 6(b).

1 We use the term “Appellant” to refer to “applicant” as defined in 37 C.F.R.
§ 1.42. Our decision references Appellant’s Appeal Brief (“Appeal Br.,”
filed March 23, 2020) and Reply Brief (“Reply Br.,” filed June 29, 2020),
and the Examiner’s Answer (“Ans.,” mailed April 28, 2020) and Final
Office Action (“Final Act.,” mailed November 8, 2019). Appellant
identifies GE Global Sourcing LLC as the real party in interest (Appeal
Br. 4).
Appeal 2020-005097
Application 15/397,469

2
We REVERSE.
CLAIMED INVENTION
The claimed invention “relates to monitoring a vehicle operator, and
to a control system for responding to monitored parameters” (Spec. 1).
Claims 1, 11, and 17 are the independent claims on appeal. Claim 1,
reproduced below with bracketed notations added, is illustrative of the
claimed subject matter:
1. A control system comprising:
[(a)] one or more processors configured to receive a first
image from a first imaging device and determine physical,
physiological or psychological changes of an operator of a
vehicle without requiring intervention of the operator based on
the first image received from the first imaging device; and
[(b)] the one or more processors also configured to alert
the operator responsive to the physical, physiological or
psychological changes of the operator of the vehicle indicating
that the operator is inattentive to control of the vehicle;
[(c)] wherein the one or more processors are configured
to receive a second image from one or more of a thermal imaging
device, a midwave infrared (MWIR) camera, or a long wave
infrared (LWIR) camera, and determine thermal changes related
to the operator based on the second image;
[(d)] wherein the one or more processors are configured
to determine the physical, physiological or psychological
changes of the operator using a new joint image that is generated
by warping the second image onto the first image using a
homography; and
[(e)] wherein one or more pixels of the new joint image
of the operator has red (R), green (G), blue (B), and thermal (T)
values.
Appeal 2020-005097
Application 15/397,469

3
REJECTIONS
Claims 1, 3, 6–12, 17, and 21 are rejected under 35 U.S.C. § 103 as
unpatentable over Fung et al. (US 2012/0212353 A1, published Aug. 23,
2012) (“Fung”), Prokoski (US 2010/0191124 A1, published July 29, 2010),
and Thiemo Alldieck et al., Context-Aware Fusion of RGB and Thermal
Imagery for Traffic Monitoring, 16 SENSORS 1–24 (2016) (“Alldieck”)
Claims 14–16, 18–20, 23, and 24 are rejected under 35 U.S.C. § 103
as unpatentable over Fung, Prokoski, Alldieck, and Breed
(WO 2015/175435 A1, published Nov. 19, 2015).
Claim 22 is rejected under 35 U.S.C. § 103 as unpatentable over Fung,
Prokoski, Alldieck, and Boverie et al. (EP 1,788,536 A1, published May 23,
2007) (“Boverie”).
Claim 25 is rejected under 35 U.S.C. § 103 as unpatentable over Fung,
Prokoski, Alldieck, and Kim et al. (US 2016/0272217 A1, published
Sept. 22, 2016) (“Kim”).
ANALYSIS
Independent Claim 1 and Dependent Claims 3, 6–10, and 21
First, by way of background, Fung is directed to a system for
assessing the behavior of a vehicle driver, and adjusting the operation of one
or more vehicle systems to accommodate, for example, a lapse in the
driver’s attention and/or alertness (Fung ¶ 132). Fung discloses that in one
embodiment, the system receives information, e.g., a sequence of images,
from an optical sensing device, e.g., a video camera mounted in the
dashboard of the motor vehicle, and analyzes the driver’s eye movement to
determine if the driver is in a normal or drowsy state (id. ¶ 161). In other
embodiments, a thermal sensing device is used to sense eye movement and
Appeal 2020-005097
Application 15/397,469

4
distinguish between various eyelid positions, e.g., opened and closed
positions, based on variations in the detected temperature of the eyes (id.
¶ 163). “In still other cases, both optical and thermal information could be
received from a combination of optical and thermal devices” (id. ¶ 165).
In rejecting claim 1 under 35 U.S.C. § 103, the Examiner cited Fung
as disclosing limitations (a), (b), and (c), as recited in claim 1. But, the
Examiner acknowledged that Fung does not specifically teach
“determin[ing] thermal changes related to the operator based on the second
image,” and also fails to teach “wherein the one or more processors are
configured to determine the physical, physiological or psychological
changes of the operator using a new joint image that is generated by warping
the second image onto the first image using a homography” and “wherein
one or more pixels of the new joint image of the operator has red (R), green
(G), blue (B), and thermal (T) values,” i.e., limitations (d) and (e), as recited
in claim 1.
The Examiner cited Prokoski as teaching the use of sensor fusion with
pixel intersections as landmarks to map color and temperature changes to
standardized images to track physiological profiles of an imaged subject
(Final Act. 4 (citing Prokoski ¶¶ 32, 81, 106, 468, 567)). And, citing
Alldieck, the Examiner found that “pixel-level fusion of RGB images with
thermal images is the most common method of combining such images and
is commonly implemented by computing a homography between the two
imaging modalities” (id. at 5 (citing Alldieck 3)). The Examiner then
concluded that it would have been obvious at the time of Appellant’s
invention, in view of Prokoski and Alldieck, to modify Fung to arrive at the
claimed invention, as recited in claim 1, because “Fung teaches receiving
Appeal 2020-005097
Application 15/397,469

5
both optical and thermal information from a combination of optical and
thermal devices to determine a drowsiness of a driver” (id. at 4–5).
Further explaining the bases for the obviousness rejection (and
responding specifically to Appellant’s argument that the combination of
Fung, Prokoski, and Alldieck fails to teach or suggest using a homography
to warp a second image onto a first image to generate a new joint image with
one or more pixels of the new joint image of the operator having red (R),
green (G), blue (B), and thermal (T) values), the Examiner, in the Answer,
directs our attention to paragraphs 105 and 106 of Prokoski as teaching
“combining visual and infrared data from imaging sensors into a
standardized composite array in which each pixel has (R, G, B, X, Y, Z, T)
values” using “techniques well known to the art of image registration”
(Ans. 6). We, however, agree with Appellant that rather than disclosing that
a second image is warped onto a first image to generate a new joint image,
the cited paragraphs of Prokoski describe forming a three-dimensional image
by layering inputs from different sensors (Reply Br. 2–3).
Describing its image capture and 3D model construction, Prokoski
discloses that imagery is collected from three sensor types, i.e., thermal,
range, and visual (Prokoski ¶ 101), and notes, “[i]n general, there is not
correspondence between the instantaneous fields of view represented by
pixels in different sensor images” (id.). Prokoski describes that extrinsic
calibration procedures are performed to create correspondence between
elements of the three image arrays (id. ¶ 104), and result in the creation of
layers of a standardized composite array, one layer per sensor (id. ¶ 105). In
other words, as Appellant observes, “a single combined layer that includes
Appeal 2020-005097
Application 15/397,469

6
all of the points matched up with one another and warped together is not
provided” (Reply Br. 3).
In view of the foregoing, we do not sustain the Examiner’s rejection
of claim 1 under 35 U.S.C. § 103. For the same reasons, we also do not
sustain the Examiner’s rejection of dependent claims 3, 6–10, and 21. Cf. In
re Fritch, 972 F.2d 1260, 1266 (Fed. Cir. 1992) (“dependent claims are
nonobvious if the independent claims from which they depend are
nonobvious”).
Independent Claims 11 and 17 and Dependent Claim 12
Independent claims 11 and 17 include language substantially similar
to the language of claim 1, and stand rejected based on the same rationale
applied with respect to claim 1 (Final Act. 8–12). Therefore, we do not
sustain the rejection of independent claims 11 and 17 for the same reasons
set forth above with respect to claim 1. For the same reasons, we also do not
sustain the rejection of claim 12, which depends from claim 11.
Dependent Claims 14–16, 18–20, and 22–25
The rejections of dependent claims 14–16, 18–20, and 22–25 do not
cure the deficiencies in the rejection of independent claims 1, 11, and 17
from which they depend. Therefore, we do not sustain the rejections of
these dependent claims for the same reasons set forth above with respect to
the independent claims.
Appeal 2020-005097
Application 15/397,469

7
CONCLUSION
In summary:
Claims
Rejected
35
U.S.C. §
Reference(s)/Basis Affirmed Reversed
1, 3, 6–12,
17, 21
103 Fung, Prokoski,
Alldieck
1, 3, 6–12,
17, 21
14–16, 18–
20, 23, 24
103 Fung, Prokoski,
Alldieck, Breed
14–16, 18–
20, 23, 24
22 103 Fung, Prokoski,
Alldieck, Boverie
22
25 103 Fung, Prokoski,
Alldieck, Kim
25
Overall
Outcome
1, 3, 6–12,
14–25
REVERSED