X Development LLC

Patent Trials and Appeals Board

Mar 12, 2020

14829716 - (D) (P.T.A.B. Mar. 12, 2020)

UNITED STATES PATENT AND TRADEMARK OFFICE
UNITED STATES DEPARTMENT OF COMMERCE
United States Patent and Trademark Office
Address: COMMISSIONER FOR PATENTS
P.O. Box 1450
Alexandria, Virginia 22313-1450
www.uspto.gov
APPLICATION NO. FILING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO.
14/829,716 08/19/2015 James Joseph Kuffner 14-954-US 3811
138779 7590 03/12/2020
McDonnell Boehnen Hulbert & Berghoff LLP/
X Development LLC
300 South Wacker Drive
32nd Floor
Chicago, IL 60606
EXAMINER
HE, WEIMING
ART UNIT PAPER NUMBER
2612
MAIL DATE DELIVERY MODE
03/12/2020 PAPER
Please find below and/or attached an Office communication concerning this application or proceeding.
The time period for reply, if any, is set in the attached communication.
PTOL-90A (Rev. 04/07)
UNITED STATES PATENT AND TRADEMARK OFFICE
____________

BEFORE THE PATENT TRIAL AND APPEAL BOARD
____________

Ex parte JAMES JOSEPH KUFFNER
____________

Appeal 2019-005983
Application 14/829,716
Technology Center 2600
____________

Before ERIC B. CHEN, JEREMY J. CURCURI, and JAMES B. ARPIN,
Administrative Patent Judges.

ARPIN, Administrative Patent Judge.

DECISION ON APPEAL
Appellant1 appeals under 35 U.S.C. § 134(a), the Examiner’s final
rejections of claims 1, 6, 8–12, 16, 17, 19, and 20, all of the pending claims.
Final Act. 2.2 Claims 2–5, 7, 13–15, 18, and, 21–28 are canceled. Id. We

1 “Appellant” here refers to “applicant” as defined in 37 C.F.R. § 1.42.
Appellant identifies the real party-in-interest as X Development LLC. Supp.
Appeal Br. 1.
2 In this Decision, we refer to Appellant’s Response After Final Action
(“Resp.,” filed June 26, 2018), Appeal Brief (“Appeal Br.,” filed October 16,
2018), Reply Brief (“Reply Br.,” filed March 12, 2019), and Supplemental
Appeal Brief (“Supp. Appeal Br.,” filed August 20, 2019); the Final Office
Action (“Final Act.,” mailed May 4, 2018), the Advisory Action (“Adv.
Act,” mailed July 3, 2018), and the Examiner’s Answer (“Ans.,” mailed
January 23, 2019); and the originally-filed Specification (“Spec.,” filed
August 19, 2015). Rather than repeat the Examiner’s findings and
Appeal 2019-005983
Application 14/829,716

2

have jurisdiction under 35 U.S.C. § 6(b).
We reverse.
STATEMENT OF THE CASE
Appellant’s claimed invention relates to methods and computer-
readable media “involving augmented reality to display virtual
representations of robotic device actions or intents for performing a task.”
Spec. ¶ 4.
As noted above, claims 1, 6, 8–12, 16, 17, 19, and 20 stand rejected.
Claims 1 and 19 are independent. Appeal Br. 12 (claim 1), 14–15 (claim 19)
(Claims App.). Claims 6, 8–12, 16, and 17 depend directly from claim 1;
and claim 20 depends directly from claim 19. Id. at 12–16.
Claim 1 recites “[a] computer-implemented method,” and claim 19
recites “[a] non-transitory computer readable medium having stored thereon
instructions that, upon execution by a computing device, cause the
computing device to perform functions,” substantially as recited in claim 1.
Id. at 12, 14–15. The Examiner relies on the same references and
substantially similar arguments in rejecting claims 1 and 19 (Final Act. 3–6,
10); and Appellant does not contest the rejection of claims 6, 8–12, 16, 17,
and 20 separately from claims 1 and 19 (Appeal Br. 9). Therefore, we focus
our analysis on independent claim 1 and the disputed and overlapping
limitations recited in independent claim 19.

determinations and Appellant’s contentions in their entirety, we refer to
these documents.
Appeal 2019-005983
Application 14/829,716

3

Claim 1, reproduced below with disputed limitations emphasized, is
representative.
1. A computer-implemented method comprising:
receiving information that indicates an action or an intent
of a robotic device to perform a task, wherein the action or the
intent includes one or more of a planned trajectory of the robotic
device to perform at least a portion of the task and an object to
be handled by the robotic device to perform at least a portion of
the task;
receiving a camera feed from a field of view of the robotic
device;
displaying by a computing device on an augmented reality
interface, a virtual representation of the action or the intent,
wherein the virtual representation includes as annotations on the
augmented reality interface at least a portion of the planned
trajectory of the robotic device or highlighting the object to be
handled by the robotic device overlaid onto the camera feed from
the field of view of the robotic device;
providing, by a device remote from the robotic device, a
projection of light along a pathway to modify the action or the
intent of the robotic device; and
in response to providing the projection of light, receiving
from the robotic device one or more of an updated planned
trajectory of the robotic device and an updated object to be
handled by the robotic device.
Id. at 12 (emphasis added).
REFERENCES AND REJECTION
The Examiner relies upon the following references in rejecting the
claims:
Appeal 2019-005983
Application 14/829,716

4

Name3 Number Published Filed
Kobayashi Viewing and
Reviewing How
Humanoids Sensed,
Planned and Behaved
with Mixed Reality
Technology, 7th IEEE-
RAS International
Conference on
Humanoid Robots,
Pittsburgh, PA, USA
Nov. 29–Dec. 1,
2007
NA
Chestnutt Interactive Control of
Humanoid Navigation,
The 2009 IEEE/RSJ
International
Conference on
Intelligent Robots and
Systems, St. Louis,
MO, USA
Oct. 11–15,
2009
NA
Buehler US 2013/0343640 A1 Dec. 26, 2013 Sept. 17, 2012
Specifically, claims 1, 6, 8–12, 16, 17, 19, and 20 stand rejected as
unpatentable under 35 U.S.C. § 103 over the combined teachings of
Kobayashi, Buehler, and Chestnutt. Final Act. 3–11.
Appellant contests the obviousness rejection of independent claims 1
and 19 (Appeal Br. 3–9) and relies on the alleged deficiencies in that
rejection to overcome the rejection of the dependent claims (id. at 9).
Because we determine that reversal of the rejection of independent claim 1 is
dispositive; except for our ultimate decision, we do not discuss the merits of
the rejections of claims 6, 8–12, 16, 17, 19, and 20 further herein. We
review the appealed rejection of independent claim 1 for error based upon

3 All reference citations are to the first named author or inventor only.
Appeal 2019-005983
Application 14/829,716

5

the issues identified by Appellant, and in light of the arguments and
evidence produced thereon. Ex parte Frye, 94 USPQ2d 1072, 1075 (BPAI
2010) (precedential). We address the rejection of claim 1 below.
ANALYSIS
1. Non-Obviousness of Claim 1 Over Kobayashi, Buehler, and
Chestnutt
As noted above, the Examiner rejects independent claim 1 as obvious
over the combined teachings of Kobayashi, Buehler, and Chestnutt. Final
Act. 3–6. In particular, claim 1 recites,
providing, by a device remote from the robotic device, a
projection of light along a pathway to modify the action or the
intent of the robotic device; and
in response to providing the projection of light, receiving
from the robotic device one or more of an updated planned
trajectory of the robotic device and an updated object to be
handled by the robotic device.
Appeal Br. 12 (Claims App.). The Examiner finds that Kobayashi discloses
a footstep planner that computes adaptive footsteps dynamically in order to
avoid the obstacles, such that the planned data can be projected on the
related physical position of a robot, and provides, by a laser projector
mounted on the ceiling, trajectories of the robot on an actual workspace. See
Final Act. 5–6 (citing Kobayashi, pg. 130, col. 1; pg. 131, col. 2.). Further,
the Examiner explains that Kobayashi discloses, “[i]n the footstep planner
example, we can see the interactive results by a change of the goal position.”
Id. at 6 (quoting Kobayashi, pg. 134, col. 2); see Kobayashi, Fig. 8. The
Examiner finds that the disputed limitations merely describe “any device
remote from the robotic deice, can project an updated path on the
environment,” and that “Kobayashi clearly discloses that it is well-known to
Appeal 2019-005983
Application 14/829,716

6

th[os]e skill[ed] in the art to use a laser projector placed on the ceiling
(remote from the robotic device) to project information on the environment
(Zach’s [related] work).” Ans. 4–5 (citing Kobayashi, pg. 131, col. 1).
The Examiner also finds that Chestnutt discloses, “[t]he user simply
draws the overall route that the robot should take using a pointing device.
The robot then plans a footstep sequence along the indicated path and
automatically generates motion which roughly follows the user’s guide.”
Final Act. 6 (quoting Chestnutt, pg. 3519, col. 2). In addition, Chestnutt
discloses, “[w]hile the robot is walking along a previously specified guide
path, the user sketches out a new guide path to follow, ending at the top of
the stairs. Once the new guide path is drawn, the robot generates a footstep
path to walk along it and climb the stairs.” Id. (quoting Chestnutt, pg. 3522,
col. 2.); see Chestnutt, pg. 3521, Fig. 3; pg. 3522, Fig. 4.
The Examiner concludes:
It would have been obvious to one of ordinary skill in the art at
the time of the invention to combine the invention of Kobayashi
and Buehler with the teaching of Chestnutt so that it is easy for
the user to intuitively specify a desired path directly onto the
environment.
Final Act. 6 (citing Chestnutt, pg. 3519, col. 2).
Appellant challenges the proposed combination of the teachings of
Kobayashi, Buehler, and Chestnutt. Appeal Br. 3–9; Reply Br. 1–4. We are
persuaded that Appellant’s challenges identify dispositive Examiner error.
First, Appellant contends that Kobayashi’s footstep planner
only illustrates that footsteps can be overlaid on the actual
environment to show virtual obstacles and planned footprints
avoiding the obstacles. In contrast, the claim language recites
“providing, by a device remote from the robotic device, a
projection of light along a pathway to modify the action or the
Appeal 2019-005983
Application 14/829,716

7

intent of the robotic device,” as in claims 1 and 19.
Reply Br. 1. The Examiner finds that:
Kobayashi further discloses “the interactive results by a change
of the goal position” in [Kobayashi’s] col 2 of p. 134. Here, a
user interaction from a projector on the ceiling can be used to
modify the planned trajectory. It is well-known that a projector
can provide a projection of light along a pathway on a projection
plane (floor in [Kobayashi’s] Fig 8).
Ans. 4. Nevertheless, we are persuaded the Examiner misinterprets
Kobayashi as applied to the disputed limitations of claim 1.
Kobayashi explains:
Recently, an approach that uses a laser projector to indicate
industrial robot trajectories has been proposed. In this approach,
the trajectories of the end of the robot hand are displayed onto
an actual workpiece by the laser projector placed on the ceiling.
This approach allows us to understand and confirm the
trajectories easier than conventional ways,
However, there is a big limitation that this system can
render graphics only onto physical surfaces. In any case, we
note that an industrial robot is significantly simple in comparison
with a humanoid, which should move around autonomously.
Kobayashi, pg. 131, col. 1 (footnotes omitted; emphases added). Thus,
Kobayashi teaches that use of a laser projector was known, but, because of
significant limitations imposed by its use, Kobayashi does not use a laser
projector in its described MR technology. See Appeal Br. 4; Reply Br. 2.
Instead, Kobayashi teaches using MR technology, in which a camera
mounted in a head-mounted display on the humanoid robot, to image the
robot’s environment (see Kobayashi, pg. 131, col. 2) and to allow an overlay
of “what the humanoid robot perceives, plans and controls to the real
environment around it and on its body” (id. at pg. 130, col. 1).
Appeal 2019-005983
Application 14/829,716

8

Kobayashi’s Figure 8 is reproduced below with our annotations.

Thus, Kobayashi’s “footstep planner” overlays footsteps and obstacles onto
an image of the robot’s “actual” environment. See id. at pg. 134, col. 1–2.
The Examiner finds:
As to the first feature “providing, by a device remote from
the robotic device, a projection of light along a pathway to
modify the action or the intent of the robotic device”, examiner
notices that Kobayashi discloses “the footstep planner computes
adaptive footsteps dynamically in order to avoid the obstacles...
In principle, any type of sensed and planned data can be projected
on the related physical position.” in col [2] of p. 130; providing,
by a laser projector on the ceiling, a trajectories of the robot onto
an actual workpiece in col 1 of p.131. Here, Kobayashi discloses
the planned data or modified data can be projected on the floor
in Fig 8 (two different color for planned and modified trajectory”
as reproduced below. Kobayashi further discloses “the
interactive results by a change of the goal position" in col 2 of p.
134. Here, a user interaction from a projector on the ceiling can
be used to modify the planned trajectory. It is well-known that a
Appeal 2019-005983
Application 14/829,716

9

projector can provide a projection of light along a pathway on a
projection plane (floor in Fig 8).
Ans. 3–4 (emphasis added). We disagree.
Referring to Figure 10, the Specification describes
a wand-like device 1014 may be used that includes an [inertial
measurement unit (IMU)] to track gestures/positions of where a
user is pointing or where the device is pointing, and because the
computing device provides an illustration of future movement of
the robotic device, a user may be able to take action to modify
the movement before it has begun.
Spec., ¶ 117. Consequently, we are persuaded instead that Kobayashi
teaches projecting virtual footsteps and obstacles on an image of the robot’s
actual environment, and does not teach “providing, by a device remote from
the robotic device, a projection of light along a pathway [in the robot’s
physical environment] to modify the action or the intent of the robotic
device,” as recited in claim 1.
Second, the Examiner finds that Chestnutt’s “pointing device” teaches
or suggests “a device remote from the robotic device” providing “a
projection of light.” Ans. 7. Chestnutt does not describe the pointing device
with specificity. However, Chestnutt discloses that the pointing device acts
“as a kind of free-floating mouse.” Chestnutt, pg. 3522, col. 2; see Resp. 9;
Appeal Br. 9. According to Chestnutt, “[t]he augmented reality system
operates by localizing cameras within the experimental environment, and
overlaying sensor data or planning information onto the camera image to
aid in debugging or visualizing the planning process.” Chestnutt, pg. 3522,
col. 2 (citing Kobayashi4; footnotes omitted; emphasis added). Referring to

4 The Examiner incorrectly asserts that “Kobayashi cites Chestnutt as prior
art.” Ans. 7. Although Kobayashi cites a 2006 article by a different group
Appeal 2019-005983
Application 14/829,716

10

Chestnutt’s Figures 3 and 4, guide paths, drawn by the operator using the
pointing device, and the resulting robot footsteps are depicted. Id. at
pg. 3521, col. 2 (Fig. 3); pg. 3522, col. 1 (Fig. 4). Thus, like Kobayashi, we
are persuaded that Chestnut teaches generating a path in the image of the
robot’s environment. See (Appeal Br. 9 (“At best the cited ‘guide paths’ are
drawn in response to a motion capture associated with the ‘pointing
device.’”); see also Resp. 9 (“The claimed features provide a ‘projection of
light’, which is in contrast to a virtual pathway or virtual markings as shown
in the primary reference.”). Consequently, we are not persuaded that
Chestnutt teaches or suggests the remote device providing a projection of
light, as recited in claim 1.
Third, as noted above, the Examiner concludes a person of ordinary
skill would have had reason to combine the teachings of Kobayashi,
Buehler, and Chestnutt, “so that it is easy for the user to intuitively specify a
desired path directly onto the environment.” Final Act. 6. For the reasons
given above, neither Kobayashi nor Chestnutt teaches or suggests the remote
device providing a projection of light, as recited in claim 1. Further, the
Examiner’s reason for combining the teachings of the applied references
fails to show why one of ordinary skill in the art would have found their
combined teachings to teach or suggest the disputed limitations. In
particular, although Kobayashi teaches that the use of a laser projector

of authors including “J. Chestnutt” (Kobayashi, pg. 135, col. 2 (reference
[10])), this is not the “Chestnutt” reference upon which the Examiner relies
in this rejection. However, Chestnutt cites Kobayashi regarding debugging
and visualizing the planning process achieved by Kobayashi’s MR
technology. Chestnutt, pg. 3523, col. 2 (reference [24]); see, e.g.,
Kobayashi, pg. 131, col. 1–2.
Appeal 2019-005983
Application 14/829,716

11

involves “a big limitation” in the area of humanoid robot control, causing
Kobayashi to pursue an alternative solution; the Examiner asserts that a
person of ordinary skill in the art would have had reason to combine
Chestnutt’s alleged teaching of a remote light projection device – also in the
area of humanoid robot control – with Kobayashi to achieve the disputed
claim limitations. These references, however, appear to teach away from
their combination to achieve the proposed result. Ormco Corp. v. Align
Tech., Inc., 463 F.3d 1299, 1308 (Fed. Cir. 2006) (“[A] reference that
‘teaches away’ from a given combination may negate a motivation to modify
the prior art to meet the claimed invention.”); see In re Gurley, 27 F.3d 551
Fed. Cir. 1994) (“A reference may be said to teach away when a person of
ordinary skill, upon reading the reference, would be discouraged from
following the path set out in the reference, or would be led in a direction
divergent from the path that was taken by the applicant.”). Thus, we are not
persuaded that the Examiner has shown that Kobayashi, Buehler, and
Chestnutt together teach or suggest the disputed limitations. See Appeal
Br. 9; Reply Br. 4.
Consequently, we are persuaded that the Examiner erred in rejecting
claim 1, and we do not sustain the obviousness rejection of claim 1.
2. The Remaining Claims
As noted above, Appellant challenges the rejection of independent
claim 19 for the same reasons as claim 1. See Final Act. 10; Appeal Br. 3.
Each of claims 6, 8–12, 16, 17, and 20 depends directly from independent
claim 1 or 19. Id. at 12–16 (Claims App.). Appellant does not challenge the
rejection of the independent claim 19 or of the dependent claims separately
from its challenge to the rejection of independent claim 1. Id. at 9; see
Appeal 2019-005983
Application 14/829,716

12

Reply Br. 4. Because we are persuaded the Examiner erred with respect to
the obviousness rejection of claim 1, we also are persuaded the Examiner
erred with respect to the obviousness rejection of claims 6, 8–12, 16, 17, 19,
and 20. For this reason, we do not sustain the rejection of those claims.
DECISIONS
1. The Examiner erred in rejecting claims 1, 6, 8–12, 16, 17, 19, and 20
under 35 U.S.C. § 103 as rendered obvious over the combined
teachings of Kobayashi, Buehler, and Chestnutt.
2. Thus, on this record, claims 1, 6, 8–12, 16, 17, 19, and 20 are not
unpatentable.
CONCLUSION
For the above reasons, we reverse the Examiner’s decision rejecting
claims 1, 6, 8–12, 16, 17, 19, and 20.
In summary:
Claims
Rejected
35 U.S.C. § References Affirmed Reversed
1, 6, 8–12,
16, 17, 19,
20
103 Kobayashi,
Buehler,
Chestnutt

1, 6, 8–12,
16, 17, 19,
20
Overall
Outcome
1, 6, 8–12,
16, 17, 19,
20

REVERSED