VMware, Inc.

Patent Trials and Appeals Board

Oct 26, 2021

2020005576 (P.T.A.B. Oct. 26, 2021)

UNITED STATES PATENT AND TRADEMARK OFFICE
UNITED STATES DEPARTMENT OF COMMERCE
United States Patent and Trademark Office
Address: COMMISSIONER FOR PATENTS
P.O. Box 1450
Alexandria, Virginia 22313-1450
www.uspto.gov
APPLICATION NO. FILING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO.
15/191,415 06/23/2016 Seongbeom KIM C879 5800
152569 7590 10/26/2021
Patterson + Sheridan, LLP - VMware
24 Greenway Plaza
Suite 1600
Houston, TX 77046
EXAMINER
LOONAN, ERIC T
ART UNIT PAPER NUMBER
2137
NOTIFICATION DATE DELIVERY MODE
10/26/2021 ELECTRONIC
Please find below and/or attached an Office communication concerning this application or proceeding.
The time period for reply, if any, is set in the attached communication.
Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the
following e-mail address(es):
ipadmin@vmware.com
psdocketing@pattersonsheridan.com
vmware_admin@pattersonsheridan.com
PTOL-90A (Rev. 04/07)
UNITED STATES PATENT AND TRADEMARK OFFICE
____________

BEFORE THE PATENT TRIAL AND APPEAL BOARD
____________

Ex parte SEONGBEOM KIM,
JAGADISH KOTRA, and FEI GUO
____________

Appeal 2020-005576
Application 15/191,4151
Technology Center 2100
_____________

Before HUNG H. BUI, JEREMY CURCURI, and
PHILLIP A. BENNETT, Administrative Patent Judges.

BUI, Administrative Patent Judge.

DECISION ON APPEAL
Appellant seeks our review under 35 U.S.C. § 134(a) from the
Examiner’s final rejection of claims 1, 3–11, and 13–22, all of the pending
claims. Appeal Br. 13–18 (Claims App.). Claims 2 and 12 have been
cancelled. We have jurisdiction under 35 U.S.C. § 6(b).
We reverse.2

1 “Appellant” herein refers to “applicant” as defined in 37 C.F.R. § 1.42.
According to Appellant, VMware Inc. is identified as the real party in
interest. Appeal Br. 3.
2 We refer to the Appellant’s Appeal Brief filed February 7, 2020 (“Appeal
Br.”); Reply Brief filed July 24, 2020 (“Reply Br.”); Examiner’s Answer
mailed May 26, 2020 (“Ans.”); Final Office Action mailed June 6, 2019
(“Final Act.”); and Specification filed June 23, 2016 (“Spec.”).
Appeal 2020-005576
Application 15/191,415

2
STATEMENT OF THE CASE
Prior art techniques for evaluating resource contention in a non-
uniform memory access (NUMA) computer system having “multiple
memory proximity domains, referred to as ‘NUMA nodes,’” utilize
performance counters to calculate instructions per cycle (IPC) or last level
cache (LLC) hit/miss rates. However, using such counters would be
inconvenient “for the purpose of NUMA management.” Spec. ¶¶ 1, 3.
Appellant’s claimed invention seeks to provide “a more practical
technique to evaluate contention at NUMA nodes for process and memory
placements, so that better process and memory scheduling decisions can be
made in NUMA system.” Spec. ¶ 4. Figure 1, depicting computer system
100 having multiple NUMA nodes 121–122, is reproduced below:

Appeal 2020-005576
Application 15/191,415

3
Figure 1 shows computer system 100 having 1st NUMA node 121 and 2nd
NUMA node 122, each of which comprises a multi-core processor (CPU
cores), memory and memory controller (MC). Spec. ¶ 14. In case of remote
memory access, CPU cores of 1st NUMA node 121 access memory of 2nd
NUMA node 122 through memory controllers of both NUMA nodes 121,
122, via interconnect 123. System software 110 is provided with latency
monitor 111, memory allocator 112, memory rebalancer 113 and rebalancer
114. Spec. ¶ 15. Latency monitor 111 is periodically executed to perform
the probing of latencies of local and remote memory accesses at each
NUMA node 121 or 122. Spec. ¶ 15.
According to Appellant,
“latencies of local and remote memory access are probed at each
NUMA node . . . If the contention at the local NUMA node
increases, the local memory access latency may become large
enough so that it becomes beneficial to allocate/rebalance
memory to a remote NUMA node. The benefits of using the
latency probing technique are its simplicity and accuracy. It does
not consume [prior art] performance counters and reflects inter-
node distance inherently. It is also capable of capturing
contentions on the NUMA interconnect, which cannot be done
by using [prior art] performance counter metrics like IPC or LLC
hits/misses.”

Spec. ¶ 4. For example, “if there is an imbalance in contention of resources
between two NUMA nodes, the NUMA [] [allocator 112] executes either [1]
memory rebalancer 113 to migrate allocated memory pages between two
NUMA nodes or [2] computer rebalancer 114 to migrate processes and their
memory pages between the two NUMA nodes.” Spec. ¶ 15. In particular,
latency monitor 111 (1) reads memory pages from the local NUMA node
121 (step 206, Figure 2), (2) reads memory pages from each of the remote
Appeal 2020-005576
Application 15/191,415

4
NUMA nodes 122 (step 208, Figure 2), and (3) measures the latency of each
memory access for storage in either the local NUMA node 121 or the remote
NUMA node (step 210, Figure 2). Spec. ¶ 18, Figure 2.
Representative Claim
Claims 1, 11, and 19 are independent claims. Representative claim 1
is reproduced below with disputed limitations emphasized for clarity:
1. In a computer system having multiple memory
proximity domains including a first memory proximity domain
with a first processor, a first memory controller, and a first
memory and a second memory proximity domain with a second
processor, a second memory controller, and a second memory, a
method of managing placement of memory pages associated
with a process in one of the memory proximity domains, said
method comprising:
reading memory pages by the first processor from the first
memory using the first memory controller and not the second
memory controller;
reading memory pages by the first processor from the
second memory using the first memory controller and the second
memory controller;
measuring, by a latency monitor, (1) a first one or more
latencies of memory reads during the reading of memory pages
by the first processor from the first memory using the first
memory controller and not the second memory controller, and
(2) a second one or more latencies of memory reads during the
reading of memory pages by the first processor from the second
memory using the first memory controller and the second
memory controller;
comparing the measured first one or more latencies to the
measured second one or more latencies; and
based on the comparing, placing memory pages into the
second memory proximity domain, wherein the placed memory
pages are associated with a first process running on the first
Appeal 2020-005576
Application 15/191,415

5
processor, so that after said placing, the first process is accessing
the memory pages from the second memory during execution.
Appeal Br. 13 (Claims App.).

REJECTIONS AND REFERENCES
(1) Claims 1, 3–5, 8–11, 13–15, and 18–20 stand rejected under
35 U.S.C. § 103 as obvious over the combined teachings of Michalak (US
2014/0136773 A1; issued May 15, 2014) and Schoinas et al. (US 6,347,362
B1; published Feb. 12, 2002; “Schoinas”). Final Act. 2–8.
(2) Claims 6 and 16 stand rejected under 35 U.S.C. § 103 as
obvious over the combined teachings of Michalak, Schoinas, and Brooker et
al. (US 8,918,392 B1; issued Dec. 23, 2014; “Brooker”). Final Act. 8–9.
(3) Claims 7 and 17 stand rejected under 35 U.S.C. § 103 as
obvious over the combined teachings of Michalak, Schoinas, and Vaid et al.
(US 2010/0121865 A1; published May 13, 2010). Final Act. 9–10.
(4) Claims 21 and 22 stand rejected under 35 U.S.C. § 103 as
obvious over the combined teachings of Michalak, Schoinas, and Arimilli et
al. (US 2003/0009640 A1; published Jan. 9, 2003; “Arimilli”). Final Act.
10–11.

ANALYSIS
In support of the obviousness rejection, the Examiner finds the
combination of Michalak and Schoinas teaches all limitations of Appellant’s
claims 1, 11, and 19. Final Act. 2–4. In particular, the Examiner finds
Michalak teaches all steps recited in Appellant’s claimed “computer system”
Appeal 2020-005576
Application 15/191,415

6
including: (1) “reading memory pages . . . from the first memory using the
first memory controller and not the second memory controller” (Final Act. 2
(citing Michalak ¶ 41, Figure 4)); (2) “reading memory pages . . . from the
second memory using the first memory controller and the second memory
controller;” (id. 3 (citing Michalak ¶ 41, Figure 4)); (3) “measuring, by a
latency monitor, (1) a first one or more latencies of memory read [] from the
first memory[] and (2) a second one or more latencies of memory read []
from the second memory[]” (id. 3 (citing Michalak ¶ 46, Figure 4)); (4)
“comparing the measured . . . latencies;” and (5) “based on the comparing,
placing memory pages into the second memory proximity domain.” (id. at 3
(citing Michalak ¶ 10)).
The Examiner acknowledges Michalak’s computer system is not a
NUMA system, but relies on Schoinas for teaching a NUMA system having
multiple memory proximity domains, referred to as “NUMA nodes” each of
which comprises a multi-core processor (CPUs), memory and memory
controller (MC) in order to support the conclusion of obviousness, i.e., “it
would have been obvious . . . to modify the computer system of Michalak to
include a plurality of nodes as taught by Schoinas [] to permit sharing of
main memory among a plurality of nodes.” Final Act. 3–4 (citing Schoinas
1:14–27, 2:56–59, 5:6–7, Figure 1).
Because Michalak’s computer system is not a NUMA system having
multiple NUMA nodes with specific controllers, Appellant argues Michalak
cannot and does not teach the disputed limitations of claims 1, 11, and 19,
which require interactions of specific controllers of a NUMA system,
including, for example:
Appeal 2020-005576
Application 15/191,415

7
reading memory pages by the first processor from the second
memory using the first memory controller and the second
memory controller; [and]

measuring, by a latency monitor, (1) a first one or more latencies
of memory reads during the reading of memory pages by the first
processor from the first memory using the first memory
controller and not the second memory controller, and (2) a
second one or more latencies of memory reads during the reading
of memory pages by the first processor from the second memory
using the first memory controller and the second memory
controller.

Appeal Br. 8–9; Reply Br. 2–3. Likewise, Appellant argues “Schoinas does
not teach or suggest measuring latency and does not cure the deficiencies of
Michalak.” Appeal Br. 9. (Capital emphasis omitted.)
The Examiner does not sufficiently respond to Appellant’s arguments.
Instead, the Examiner takes the position that while Michalak does not teach
“measuring latencies of memory reads to a second memory using both
controllers as claimed,” (Ans. 6) Schoinas teaches access paths of (1)
“reading memory pages by the first processor from the first memory using
the first memory controller and not the second memory controller” (Path 1
of Schoinas) and (2) “reading memory pages by the first processor from the
second memory using the first memory controller and the second memory
controller” (Path 2 of Schoinas) (Ans. 7) as proposed in Schoinas’ marked-
up Figure 1 below.
Schoinas’ Figure 1, depicting the Examiner’s proposed access to local
memory and remote memory in NUMA system 10, is reproduced below
with additional markings for illustration.
Appeal 2020-005576
Application 15/191,415

8

Schoinas’ Figure 1, as reproduced above, depicts NUMA system 10
including a plurality of NUMA nodes each of which includes CPUs 12,
memory 14, and node/memory controller 14. Schoinas 1:6–11; 2:56–62.
According to the Examiner, Schoinas’ CPU 12, shown in Figure 1,
can access paths of (1) “reading memory pages [] from the first memory [16]
using the first memory controller [14]” (Path 1) and (2) “reading memory
pages [] from the second memory [16] using the first memory controller [14]
and the second memory controller [14]” (Path 2). Ans. 7.
We agree with the Examiner’s position regarding Schoinas only in
part. Obviousness is a question of law based on underlying factual findings,
Appeal 2020-005576
Application 15/191,415

9
In re Baxter, 678 F.3d 1357, 1361 (Fed. Cir. 2012), including what a
reference teaches, In re Beattie, 974 F.2d 1309, 1311 (Fed. Cir. 1992), and
the existence of a reason to combine references, In re Hyon, 679 F.3d 1363,
1365–66 (Fed. Cir. 2012). At the outset, we note both Michalak and
Schoinas describe prior art techniques, as acknowledged by Appellant’s own
admitted prior art (AAPA), that utilize performance counters to calculate
instructions per cycle (IPC) or last level cache (LLC) hit/miss rates to
evaluate resource contention in a computer system. Spec. ¶¶ 1, 3.
For example, Michalak teaches “hardware-based page access counters
[] used to record memory utilization [in a mobile processor].” Michalak
¶¶ 1, 7. According to Michalak,
based on the utilizations measured with the page access counters
480, the kernel or operating system routine may change virtual
address to physical address mappings used in the MMU 450 to
dynamically assign (or reassign) pages to the most efficient
memory available in the main memory associated with the
processor 440. In one embodiment, relative efficiencies
associated with different memory types that may be available in
the main memory may be defined based on capacities,
throughputs, latencies, and/or power efficiencies associated
therewith. In particular, with respect to a given memory,
capacity may generally refer to total storage available therein
(typically expressed in GB), throughput may refer to the rate to
transfer data over the memory interface associated therewith
(typically expressed in GB/s or gigabytes per second), latency
may refer to a portion of the cache miss penalty that represents a
time for the MMU 450 to access the memory during a load
instruction from the processor 440, and power efficiency may
refer to the power that the memory consumes during idle, active,
or other I/O states. Accordingly, in one embodiment, the kernel
or operating system routine may be configured with predefined
rules to specify relative efficiencies associated with certain
existing memory types and therefore define the "more efficient"
Appeal 2020-005576
Application 15/191,415

10
and "less efficient" memory types based thereon . . . Alternatively
(or additionally), the kernel or operating system routine may run
one or more appropriate benchmarking processes to measure the
relative capacities, throughputs, latencies, and power efficiencies
associated with the different memory types available in main
memory and define the memory types having greater or lesser
efficiency based thereon.

Michalak ¶ 46 (emphasis added).
As such, all the cited paragraphs 10, 41, and 46 of Michalak pertain to
(1) monitoring page access counters, and (2) dynamically assigning the
pages in a first memory and the pages in a second memory based on the
monitored page access counters. Michalak ¶¶ 10, 41, and 46. As correctly
recognized by Appellant, these cited paragraphs of Michalak do not teach
the steps recited in Appellant’s claimed “computer system,” especially “the
specific measuring of latency [via latency monitor 111 of system software
110, shown in Appellant’s Figure 1] during the specific reads with the
specific controllers as claimed.” Appeal Br. 8.
Like Michalak, Schoinas also teaches monitoring counters in the
context of a NUMA system having multiple NUMA nodes to track certain
types of event. Schoinas 2:13–21, 3:1–5, Figure 2. However, as recognized
by Appellant, “Schoinas does not teach or suggest measuring latency and
does not cure the deficiencies of Michalak” to arrive at Appellant’s claimed
invention. Appeal Br. 9.
As a NUMA system provided with multiple NUMA nodes, we agree
with the Examiner that Schoinas’ CPU 12, shown in Figure 1, is capable of
accessing paths of (1) “reading memory pages [] from the first memory [16]
using the first memory controller [14]” (Path 1) and (2) “reading memory
pages [] from the second memory [16] using the first memory controller [14]
Appeal 2020-005576
Application 15/191,415

11
and the second memory controller [14]” (Path 2). Ans. 7. However, as
recognized by Appellant, “nothing in Fig. 1 of Schoinas or its brief
description in col. 2, lines 55-67, that teaches or suggests managing
placement of memory pages in the NUMA system.” Reply Br. 2.
In fact, neither Michalak nor Schoinas, alone or in combination,
teaches or suggests the use of Appellant’s claimed “latency monitor” as a
software solution for probing and measuring latencies during the specific
reads with the specific controllers, i.e.,
(1) a first one or more latencies of memory reads during the
reading of memory pages by the first processor from the first
memory using the first memory controller and not the second
memory controller, and (2) a second one or more latencies of
memory reads during the reading of memory pages by the first
processor from the second memory using the first memory
controller and the second memory controller
as recited in claims 1, 11, and 19. This way, the use of prior art “counters”
such as disclosed in Michalak and Schoinas could be avoided as
acknowledged by AAPA. Spec. ¶ 3.
Lastly, we also note the Examiner’s rationale to incorporate the
teachings of Schoinas into Michalak is insufficient to support the
combination as required by KSR Int’l Co. v. Teleflex, Inc., 550 U.S. 398, 418
(2007); and In re Kahn, 441 F.3d 977, 988 (Fed Cir. 2006) (“[R]ejections on
obviousness grounds cannot be sustained by mere conclusory statements;
instead, there must be some articulated reasoning with some rational
underpinning to support the legal conclusion of obviousness.”). For
example, the Examiner reasons “it would have been obvious . . . to modify
the computer system of Michalak to include a plurality of nodes as taught by
Schoinas [] to permit sharing of main memory among a plurality of nodes.”
Appeal 2020-005576
Application 15/191,415

12
Final Act. 3–4. However, incorporating Schoinas’ NUMA system into a
mobile computer system of Michalak alone is insufficient to arrive at
Appellant’s claimed “latency monitor” to measure the latency during the
specific reads with the specific controllers in the manner recited in claims 1,
11, and 19.
For these reasons, we do not sustain the Examiner’s obviousness
rejection of independent claims 1, 11, and 19, and the rejections of their
respective dependent claims 2, 8–12, 14, 15, and 17–19, which are not
argued separately.
CONCLUSION
On the record before us, we conclude Appellant has demonstrated the
Examiner erred in rejecting (1) claims 1, 3–5, 8–11, 13–15, and 18–20 as
obvious over the combined teachings of Michalak and Schoinas; (2) claims 6
and 16 as obvious over the combined teachings of Michalak, Schoinas, and
Brooker; (3) claims 7 and 17 as obvious over the combined teachings of
Michalak, Schoinas, and Vaid; and (4) claims 21 and 22 as obvious over the
combined teachings of Michalak, Schoinas, and Arimilli.

DECISION SUMMARY
In summary:
Claims
Rejected
35 U.S.C. § References/Basis Affirmed Reversed
1, 3–5, 8–
11, 13–15,
18–20
103 Michalak, Schoinas 1, 3–5, 8–11,
13–15, 18–
20
6, 16 103 Michalak, Schoinas,
Brooker
6, 16
Appeal 2020-005576
Application 15/191,415

13
Claims
Rejected
35 U.S.C. § References/Basis Affirmed Reversed
7, 17 103 Michalak, Schoinas,
Vaid
7, 17
21, 22 103 Michalak, Schoinas,
Arimilli
21, 22
Overall
Outcome
1, 3–11, 13–
22

REVERSED