Opinion
MDL No. 19-md-2879
2022-05-03
IN RE: MARRIOTT INTERNATIONAL, INC., CUSTOMER DATA SECURITY BREACH LITIGATION Consumer Actions
MEMORANDUM OPINION
Paul W. Grimm, United States District Judge
The Consumer Plaintiffs ("Plaintiffs") of this MDL have filed a motion to certify a number of classes. Among the various classes that Plaintiffs have asked the Court to certify are thirteen classes or subclasses for classwide damages, to be tried along with the injunctive or declaratory relief class and various liability issues for which they also seek certification. Mot. for Class Cert., ECF Nos. 858 (redacted), 859 (sealed), 863 (sealed), 865 (sealed). Certification of the damages classes is sought pursuant to Federal Rule of Civil Procedure 23(b)(3), so it is necessary for Plaintiffs to show that common issues of law and or fact predominate over individual issues, and that class certification would be superior to any other means of determining damages, in the event that liability is found.
All references in this opinion are to these documents. As previously ordered, the parties will be submitting documents to the Court with updated redactions pursuant to the Special Master's resolution of the parties’ confidentiality designation disputes. See ECF No. 1009. Those updates, however, do not affect this opinion.
Plaintiffs rest their hope of certifying classwide damages classes on one of their designated expert witnesses, Jeffrey T. Prince, Ph.D., a highly credentialed economist and Professor at the Kelley School of Business at Indiana University. Defendants Marriott and Accenture ("Defendants") seek to undermine Dr. Prince's classwide damages models with their own highly credentialed economist, Catherine Tucker, Ph.D., a Professor of Management Science and Marketing at Massachusetts Institute of Technology.
Plaintiffs named as defendants Marriott International, Inc., Starwood Hotels and Resorts Worldwide, LLC, and Accenture LLP. Second Amended Consolidated Complaint ("Compl."), ECF Nos. 413 (sealed), 537 (redacted) at ¶¶ 12–14. Marriott International acquired Starwood Hotels and Resorts in September 2016, and I will refer to Marriott International, Inc., and Starwood Hotels and Resorts Worldwide, LLC, using one name—either "Marriott" or "Starwood"—in this opinion.
As might be expected, when the Court and the parties conferred to set a schedule for discovery in advance of class certification motions, deadlines were set for Dr. Prince to file his Federal Rule of Civil Procedure 26(a)(2) expert report, Dr. Tucker to file hers, Dr. Prince to file a rebuttal, and for them both to be deposed. Deadlines also were set for filing class certification motions, and for motions to challenge the admissibility of expert testimony, colloquially referred to as " Daubert " motions. In a perfect world, the deadlines for all these events would line up in harmonious sequence, and, at first, they did. But given the complexity of this case, and issues related to the completion of discovery, the sequencing became less than ideal.
Daubert v. Merrell Dow Pharm. Inc. , 509 U.S. 579, 113 S.Ct. 2786, 125 L.Ed.2d 469 (1993).
Dr. Prince's Rule 26(a)(2) initial report was filed on July 12, 2021. Expert Class Cert. Rep. of Jeffrey T. Prince, Ph.D. ("Prince Initial Rep."), ECF No. 859-4 (sealed). He was deposed on August 13, 2021. Deposition of Jeffrey T. Prince ("Prince Dep. 1"), ECF No. 891-2 (sealed). Dr. Tucker's Rule 26(a)(2) report was filed on September 7, 2021, Expert Rep. of Catherine Tucker, Ph.D. ("Tucker Rep."), ECF No. 894-5 (sealed), and Defendants filed their Daubert motion challenging Dr. Prince's methodology on September 20, 2021. Defs.’ Mot. to Exclude (" Daubert Mot."), ECF Nos. 894 (sealed), 895 (redacted). Dr. Tucker was deposed on September 29, 2021, Deposition of Catherine Tucker ("Tucker Dep."), ECF No. 916-4 (sealed), and Dr. Prince filed his Rule 26(a)(2) rebuttal report on October 12, 2021. Rebuttal Rep. of Jeffrey T. Prince, Ph.D. ("Prince Rebuttal Rep."), ECF No. 916-1 (sealed). Plaintiffs filed their opposition to Defendants’ Daubert motion on October 25, 2021. Pls.’ Opp'n, ECF Nos. 916 (sealed), 922 (redacted). Defendants did not file a reply to Plaintiffs’ opposition to their Daubert motion, but an animated dispute arose regarding Defendants’ contention that Dr. Prince had materially changed his methodology with respect to his two classwide damages models between the filing of his original report, and his rebuttal report. Defs.’ Letter Corresp., ECF No. 930. Plaintiffs were equally animated in denying that Dr. Prince had changed his methodology, and once more the Special Master, Judge (Ret.) John Facciola, was called upon to intervene, following which I ordered that Dr. Prince was to be re-deposed for limited purposes, and that Judge Facciola would preside. Letter Order of November 30, 2021, ECF No. 940. Following Dr. Prince's second deposition, the parties filed letter motions: Defendants seeking to strike Dr. Prince's "new" opinions and methodology, Defs.’ Letter Mot., ECF Nos. 961 (sealed), 963 (redacted); and Plaintiffs opposing this motion, and requesting, in turn, that I "strike" Dr. Prince's second deposition. Pls.’ Letter Mot., ECF No. 964. I did neither, but instead ordered that Dr. Tucker be afforded an opportunity to file a rebuttal report addressing Dr. Prince's rebuttal report and testimony in his second deposition. Letter Order of January 18, 2022, ECF No. 967. Dr. Tucker's second report was filed on February 18, 2022. Suppl. Expert Rep. of Catherine Tucker, Ph.D. ("Tucker Suppl. Rep."), ECF Nos. 984 (sealed), 985 (redacted).
What the above chronology reveals is that the record upon which the parties based their Daubert filings (the last of which was made on October 25, 2021) changed materially after the briefing was concluded: Dr. Prince filed a substantial rebuttal report, he was deposed, and Dr. Tucker filed a substantial report addressing Dr. Prince's rebuttal report and deposition. In addition, I have had the benefit of an expert "tutorial" with both Dr. Prince and Dr. Tucker, during which I was able to question them directly and hear their views with regard to the damages issues. Having carefully reviewed all the experts’ reports, their depositions, Defendants’ motion, and Plaintiffs’ opposition, as well as the academic literature that figured prominently in the development of Dr. Prince's methodology, I am DENYING, for the purposes of ruling on the pending motions for class certification, Defendants’ motion to exclude Dr. Prince's classwide model for determining "overpayment damages" (the difference between what the class plaintiffs actually paid for a Starwood hotel room, and the price that Starwood would have been able to charge in a hypothetical "but-for" world in which consumer willingness to pay for a Starwood room had shifted in response to consumer knowledge of Starwood's inadequate data security), and GRANTING Defendants’ motion to exclude Dr. Prince's model for calculating the "market value" of Plaintiffs’ personally identifiable information ("PII") that was compromised due to the data breach.
During a status conference, held on November 30, 2021, I raised with counsel my interest in appointing John de Figueiredo, Ph.D., Russell M. Robinson II Professor of Law, Strategy, and Economics at Duke University School of Law, as a court technical advisor, to assist me "in understanding the economic theories relied upon by the parties’ experts with respect to damages models." Letter Order of November 30, 2021, ECF No. 940 at 2. As noted in that Letter Order, courts have inherent authority to appoint such a technical advisor to assist the Court in understanding scientific or technical information. See, e.g. , TechSearch, L.L.C. v. Intel Corp. , 286 F.3d 1360, 1377 (Fed. Cir. 2002) ; Ass'n of Mexican-Am. Educators v. California , 231 F.3d 572, 590 (9th Cir. 2000) ; Reilly v. United States , 863 F.2d 149, 156 (1st Cir. 1988) ; Intellectual Ventures I LLC v. Capital One Fin. Corp. , 280 F. Supp. 3d 691, 695 n.1 (D. Md. 2017), aff'd , 937 F.3d 1359 (Fed. Cir. 2019). The parties agreed, and Dr. de Figueiredo provided invaluable assistance in aiding the Court's ability to understand the many complex economic concepts underlying Plaintiffs’ overpayment and market value damages models, and in understanding the many economic publications cited by Dr. Prince and Dr. Tucker in their reports. Dr. de Figueiredo's exceptional assistance made it possible for the Court to have a meaningful dialogue with Dr. Prince and Dr. Tucker during the expert tutorial that took place on March 21, 2022.
At this time, I would like to thank my chambers’ two spring semester judicial interns, Alan Harrison and Michelle Lim, for their able assistance in preparing this opinion.
It is important to understand the limited scope of this ruling. For reasons that I will explain, I conclude that the methodology described by Dr. Prince (which he tested using the limited amount of "New" Data Storage ("NDS") database information then available to him) is sufficient for me to rule on whether Dr. Prince's "overpayment damages" model meets admissibility standards under Federal Rule of Evidence 702 for purposes of ruling on class certification. This is because he has not yet had the opportunity to fully test this methodology on non-bellwether class member data contained in the Starwood NDS database, on which the model depends in order to actually calculate the amount of overpayment damages. Defendants objected to producing NDS database information relating to any but the bellwether class members until after a ruling on class certification, and Dr. Prince was unable to test his overpayment damages model in the context of the members of a specific certified class, which I conclude is necessary in order to evaluate Dr. Tucker's criticisms of the reliability of Dr. Prince's methodology in general, and as to whether the model produces reliable results when applied to the particular facts of a specific certified class. But, I have concluded that Dr. Prince has identified a reliable methodology for determining overpayment damages in general, and tested it on the bellwether plaintiffs’ NDS database information. This is sufficient for purposes of considering his methodology in ruling on Plaintiffs’ motion to certify classwide damages classes. Once Dr. Prince has had access to the NDS database with respect to the non-bellwether class members in the specific classes for which I have granted Plaintiffs’ motion for class certification, and he has calculated his classwide overpayment damages, I will order that Dr. Prince provide a supplemental report setting forth these calculations and allow Defendants to depose him on his final calculations. I will then allow Dr. Tucker to provide a supplemental report and give Plaintiffs an opportunity to depose her. At that time, I will allow limited additional briefing on the admissibility of Dr. Prince's overpayment damages and make a final ruling. If I exclude Dr. Prince's damages model and calculations, I will de-certify the classwide damages classes.
As for Dr. Prince's model for calculating classwide loss of market value of class member compromised PII, however, I agree with Defendants that Dr. Prince's methodology does not meet the requirements of Rule 702, principally because he failed to articulate (and test, using NDS data for the bellwether plaintiffs) a particular method for determining the number of times a particular class member's PII was "sold" in his hypothetical market for PII, without which he cannot determine classwide damages. I explain below the basis for my rulings.
STANDARD OF REVIEW
Defendants challenge the admissibility of Dr. Prince's formulaic models for calculating classwide damages for Plaintiffs. As already noted, they style their motion as a Daubert challenge, based on the now-famous case of Daubert v. Merrell Dow Pharm. Inc. , 509 U.S. 579, 113 S.Ct. 2786, 125 L.Ed.2d 469 (1993) and its progeny, Kumho Tire Co. v. Carmichael , 526 U.S. 137, 119 S.Ct. 1167, 143 L.Ed.2d 238 (1999) and Gen. Elec. Co. v. Joiner , 522 U.S. 136, 118 S.Ct. 512, 139 L.Ed.2d 508 (1997) (collectively, " Daubert "). As a preliminary matter, it is not yet a settled matter that a full-blown Daubert challenge should be entertained by the court when deciding a class certification motion. 3 William B. Rubenstein, et al., Newberg on Class Actions § 7:24 (5th ed. 2021). ("Issues arising out of the use of expert witnesses at the class certification stage have beguiled the federal courts.... The issue is beguiling for the simple reason that certification is generally not the time to decide the merits of the case, yet expert witness testimony relevant to the merits often is proffered as also relevant to a prong of certification inquiry. The problem, if problem it be, is one of a mismatch between evidence and timing, with ultimate merits evidence being weighed before the conclusion of the discovery phase of the lawsuit."). Newberg states that the circuit courts have split in their approach to the applicability of Daubert challenges at class certification stage, with some circuits opting for a "limited" Daubert analysis, while others have required a "full" Daubert examination. Id.
Newberg sums up the score as follows: "[R]eported decisions suggest that courts in the Second, Fourth, and District of Columbia Circuits follow a serious [(i.e., "full")] Daubert approach; courts in the First, Fifth, Sixth, and Tenth Circuits tend to follow a limited Daubert approach." Id. (footnotes omitted). At least three district courts within the Fourth Circuit have considered whether to use a "full Daubert " analysis, ultimately adopting the "necessary to decide class certification" test discussed by Newberg. See, e.g. , Childress v. JP Morgan Chase & Co. , No. 16-CV-298, 2019 WL 2865848, at * 2 (E.D.N.C., July 2, 2019) ("At the outset, the Court notes that there is no controlling precedent which dictates whether to conduct a Daubert analysis at the class certification stage or how focused or full that analysis should be.... The Court is persuaded by authorities which have concluded that where a movant has proffered expert testimony in support of its motion for class certification, and such testimony is critical to the issue of class certification, a full Daubert inquiry is appropriate."); Robinson v. Nationstar Mortg. LLC , No. 14-3667, 2019 WL 4261696, at *13 (D. Md. September 9, 2019) (holding that when an expert's opinions are "critical" to class certification, the court must make a "conclusive ruling" on the evidentiary challenge before it can rule on class certification); Baxley v. Jividen , 504 F. Supp. 3d 539, 542–43 (S.D.W. Va. 2020) (noting that when "courts have required a full or thorough Daubert analysis, they have largely done so when ‘expert testimony is in fact critical to class certification,’ " but declining to do such a full analysis because the expert's analysis was not critical to ruling on the class certification issue).
I find that Dr. Prince's formulaic models for determining overpayment and inherent value damages are essential to Plaintiffs’ efforts to certify classwide damages classes in addition to liability-only classes. Accordingly, I will undertake a "full" Daubert analysis in this case. The starting place for this analysis is Federal Rule of Evidence 702, as amended in 2000 to incorporate the teaching of Daubert and its progeny.
I. Rule 702
Fed. R. Evid. 702 states:
A witness who is qualified as an expert by knowledge, skill, experience, training, or education may testify in the form of an opinion or otherwise if:
(a) the expert's scientific, technical, or other specialized knowledge will help the trier of fact to understand the evidence or to determine a fact in issue;
(b) the testimony is based on sufficient facts or data;
(c) the testimony is the product of reliable principle and methods; and
(d) the expert has reliably applied the principles and methods to the facts of the case.
Fed. R. Evid. 702. The advisory note to the 2000 amendments to Rule 702 ("2000 Advisory Note") is essential reading for judges and lawyers who undertake a Daubert analysis. There are several key takeaways that should be kept in mind. First, Rule 702 was amended in 2000 for the express purpose of incorporating the teachings of Daubert and its progeny, to fulfill their requirement that trial judges act "as gatekeepers to exclude unreliable expert testimony" with regard to all expert testimony, not just that which is science-based. 2000 Advisory Note (case citations omitted).
Second, the 2000 amendment to Rule 702 "provides some general standards that the trial court must use to assess the reliability and helpfulness of proffered expert testimony." Id. To do so, the trial court must apply Federal Rule of Evidence 104(a), which requires that the "proponent [of the expert testimony] has the burden of establishing that the pertinent admissibility requirements are met by a preponderance of the evidence." Id.
Third, the 2000 Advisory Note observes that:
Daubert set forth a non-exclusive checklist for trial courts to use in assessing the reliability of scientific [or any other] expert testimony. The specific factors explicated by the Daubert Court are (1) whether the expert's technique or theory can or has been tested—that is, whether the expert's theory can be challenged in some objective sense, or whether it is instead simply a subjective, conclusory approach that cannot reasonably be assessed for reliability; (2) whether the technique or theory has been subject to peer review and publication; (3) the known or potential rate of error of the technique or theory when applied; (4) the existence and maintenance of standards and controls; and (5) whether the technique or theory has
been generally accepted in the scientific community.
Id.
Fourth, the so-called " Daubert factors" "were neither exclusive nor dispositive. Other cases have recognized that not all of the specific Daubert factors can apply to every type of expert testimony." Id. And, additional factors may be relevant to the inquiry, such as: whether the expert will be testifying about matters that grow "naturally and directly out of research they have conducted independent of litigation, or whether they have developed their opinions expressly for purposes of testifying"; whether "the expert has unjustifiably extrapolated from an accepted premise to an unfounded conclusion"; whether "the expert has adequately accounted for obvious alternative explanations"; whether "the expert ‘is being as careful as he would be in his regular professional work outside his paid litigation consulting"; and whether "the field of expertise claimed by the expert is known to reach reliable results for the type of opinion the expert would give." Id.
Fifth, "[a] review of the case law after Daubert shows that the rejection of expert testimony is the exception rather than the rule. Daubert did not work a ‘seachange over federal evidence law,’ and ‘the trial court's role of gatekeeper is not intended to serve as a replacement for the adversary system’ "—where "[v]igorous cross-examination, presentation of contrary evidence, and careful instruction on the burden of proof are the traditional and appropriate means of attacking shaky but admissible evidence." Id. (internal citations omitted). "Likewise, this amendment is not intended to provide an excuse for an automatic challenge to the testimony of every expert." Id.
If the foregoing Daubert factors provide "up-close" guidance about how a challenge to expert testimony should be addressed by the trial judge, Daubert itself provides a helpful "birds-eye" view, to make sure the court does not overlook the evidentiary forest for the many scientific and technical "trees." As Justice Blackmun helpfully observed in Daubert , there are four related and sometimes overlapping concepts that help guide a trial judge in deciding a Daubert challenge. The expert evidence must be relevant (tending to prove or disprove facts that are consequential to the determination of the case), reliable (sufficiently accurate to be counted on, elsewise it is of no relevance), helpful to the factfinder (otherwise it is entirely unnecessary, as experts are only allowed to offer opinion testimony when the factfinders lack the knowledge and expertise to evaluate the scientific or technical evidence on their own), and must "fit " the facts and issues of the specific case (otherwise it is irrelevant, and unhelpful). Daubert , 509 U.S. at 591–93, 113 S.Ct. 2786 (emphasis added).
Since Rule 702 and Daubert contemplate that the trial judge must be the "gatekeeper" determining whether expert evidence is admissible or not, trial judges would welcome the most focused and helpful analysis of the admissibility factors from the experts retained by the parties and the briefing by counsel on both sides of the admissibility issue. In a very real sense, when an expert witness files his or her Rule 26(a)(2) disclosure, it would be wise for it to be written with the recognition that the trial judge is, in essence, an audience of one who must understand the opinions the expert expresses, the factual support for them, the methodology applied to reach the opinions that will be testified to (and, vitally, why they are reliable), and how the methodology has been applied to the facts of the particular case. Since the Daubert factors include consideration of whether the methodology selected has been subject to peer review or similar scrutiny by other experts in a particular field, it would be of enormous help for the expert and the parties to provide the judge with copies of the most important peer expert articles, and especially so if the key passages have been highlighted to facilitate review by the judge.
Experts retained by the parties to offer opinion testimony at trial are required to make comprehensive pretrial disclosures of the opinions they will express at trial, as well as the facts or data they considered in reaching them. Fed. R. Civ. P. 26(a)(2)(B). They must timely supplement them if they change in any material respect. Fed. R. Civ. P. 26(e)(1)(A). Experts are required to provide the information in accordance with these rules, or to supplement their opinions within the time allowed by the court's scheduling order that imposes deadlines for the completion of discovery. Experts are prohibited from using undisclosed information at a subsequent trial or hearing unless the failure to do so is excused by the court for substantial justification or because it is harmless. Fed. R. Civ. P. 37(c)(1). Thus, when, as here, the court is required to rule on a Daubert motion, the challenged expert's reports filed pursuant to Rule 26(a)(2)(B), as well as the deposition testimony of the expert, become the primary sources of the information that will inform the court's ruling.
Just as importantly, counsel for the proponent and opponent of the expert evidence at issue would demonstrate the most effective advocacy if they organized their memoranda to address the key Daubert factors and the evidence supporting their position with respect to each. Unfortunately, all too often, expert reports are drafted as if their intended audience was another expert in the same field. The reports are filled with undefined technical jargon and calculations likely to be indecipherable to a generalist judge; key peer reviewed literature is neither clearly identified, nor attached as an exhibit (with underlining to facilitate review); and counsel fail to organize their memoranda in a manner that allows the judge to undertake the required analysis.
II. Applicable Case Law
The post- Daubert case law largely tracks the requirements of Rule 702, as amended in 2000. For example, in Hickerson v. Yamaha Motor Corp. , 882 F.3d 476 (4th Cir. 2018), the Fourth Circuit summarized the standards governing a Daubert challenge. The court noted that in fulfilling their gatekeeping duty, trial judges " ‘have considerable leeway’ in excluding evidence," and are required to ensure that "[e]xpert testimony must be ‘based on sufficient facts or data,’ and the expert must arrive at his opinions by properly applying ‘reliable principles and methods’ to the facts." Id. at 480 (internal citations omitted). As for determining the reliability of expert evidence, the court referenced the well-known Daubert factors, namely, whether the methodology: has been tested; has been subjected to peer review; when employed, produces an ascertainable potential error rate (that is not excessively high); is governed by standards controlling its operation; and enjoys general acceptance within the relevant scientific or technical community. Id. at 480–81 (citing Cooper v. Smith Nephew, Inc. , 259 F.3d 194, 199 (4th Cir. 2001) and Daubert , 509 U.S. at 592–94, 113 S.Ct. 2786 ). Further, the Rule 702 inquiry is intended to be a "flexible" one, which means that the Daubert factors are "helpful, not definitive." Hickerson , 882 F.3d at 481 (citing Kumho Tire , 526 U.S. at 150–51, 119 S.Ct. 1167 and Daubert , 509 U.S. at 593, 113 S.Ct. 2786 ). Finally, when applying these standards "courts ‘should be conscious of two guiding, and sometimes competing principles[:] Rule 702 was intended to liberalize the introduction of relevant expert evidence [and] expert witnesses have the potential to be both powerful and quite misleading.’ " Hickerson , 882 F. 3d at 481 (citing Westberry v. Gislaved Gummi AB , 178 F.3d 257, 261 (4th Cir. 1999) ).
In Cooper , the court stated that "a trial judge, faced with a proffer of expert scientific testimony, must conduct ‘a preliminary assessment of whether the reasoning or methodology underlying the testimony is scientifically [or technically ] valid and of whether that reasoning or methodology properly can be applied to the facts in issue.’ " Cooper , 259 F.3d at 199 (citing Daubert , 509 U.S. at 592–93, n.10, 113 S.Ct. 2786 ). Additionally, the proponent of expert evidence must make these required showings by "a preponderance of proof." Id. Because the Rule 702/ Daubert analysis is intended to be flexible, "particular [ Daubert ] factors may or may not be pertinent in assessing reliability, depending on the nature of the issue, the expert's particular expertise, and the subject of his testimony." Cooper , 259 F.3d at 200 (citing Kumho Tire , 526 U.S. at 150, 119 S.Ct. 1167 ). And, importantly, the "trial judge must have considerable leeway in deciding in a particular case how to go about determining whether particular expert testimony is reliable." Cooper , 259 F.3d at 200 (citing Kumho Tire , 526 U.S. at 152, 119 S.Ct. 1167 ).
While the focus of Daubert was scientific evidence, Kumho Tire made it clear that Rule 702 and the Daubert standards applied to all expert evidence within the scope of Rule 702, whether scientific, technical, or specialized. Kumho Tire , 526 U.S. at 149–50, 119 S.Ct. 1167.
Finally, and particularly relevant in this case, when the filing of a class certification motion occurs prior to full discovery having been afforded to the party seeking to certify one or more classes (and the moving party relies on the opinions of an expert to do so), the inability of the expert to express opinions that are fully supported by all the facts does not provide a basis for striking the expert's opinions. As Judge Chuang of this court aptly has noted "where the [expert's] broad methodology is sound, the lack of considerations of unproduced data cannot provide a basis to strike the expert's testimony. Rather than striking the testimony, the Court may need to consider permitting supplemental discovery to correct for the lack of relevant data not previously made available" to the expert. Robinson , 2019 WL 4261696, at *14. Here, Defendants objected to producing NDS database information for any plaintiffs other than the bellwether plaintiffs prior to determination of class certification. And that limited production was provided to Dr. Prince a mere two days before his initial report was due. In his rebuttal report and his deposition testimony, Dr. Prince explained why his opinions regarding the overpayment damages model were cabined by the limited class discovery produced by Defendants. Therefore, because, for the reasons explained below, I find that Dr. Prince's "broad methodology" for determining overpayment damages is "sound," and that he still requires full class member access to the NDS database to fully calculate overpayment damages, his opinions regarding this model are appropriate to consider in ruling on the class certification motion.
Having discussed the standards that govern the admissibility of expert testimony in general and at class certification, I now turn to each of the damages models that Dr. Prince developed for measuring class wide damages.
DISCUSSION
I. Overpayment Damages
In his initial report, Dr. Prince "considered whether it is possible to formulaically estimate the amount of any change in market price for the value of the rooms that Plaintiffs booked in a ‘but for’ world" in which consumers were aware that Marriott was not protecting consumer PII appropriately. It is an empirical question whether consumer demand for Marriott hotel rooms would have been reduced in the but-for world in which it had been disclosed to potential customers that Marriott's data security practices were inadequate. Prince Initial Rep. at ¶ 7. Relying on the results of a conjoint consumer survey prepared by another of Plaintiffs’ experts, Sarah Butler, Dr. Prince stated:
Dr. Prince repeatedly refers to "Marriott" in his written reports and deposition testimony when describing his overpayment damages model, but that was as a convenience, since Marriott is one of the defendants. It is clear that all of the overpayment damages were incurred from stays at Starwood properties between 2014–2018. Because Dr. Prince referred to "Marriott" throughout, I will also use Marriott and Starwood interchangeably for the purposes of this ruling.
Defendants designated their own expert, Olivier Toubia, Ph.D., who criticized Ms. Butler's survey methodology and results, Expert Rep. of Olivier Toubia, Ph.D. ("Toubia Rep."), ECF No. 885-4 (sealed), but they have not sought to exclude Ms. Butler's methodology or the results of her consumer survey. For her part, Ms. Butler vigorously defended her methodology and results in the face of Dr. Toubia's criticisms. See Expert Class Cert. Rep. of Sarah Butler ("Butler Initial Rep."), ECF No. 859-3 (sealed); Rebuttal Rep. of Sarah Butler ("Butler Rebuttal Rep."), ECF No. 905-5 (sealed). Defendants have not sought to exclude Ms. Butler's methods and survey results, and my own review of the reports and depositions of Ms. Butler and Dr. Toubia leads me to conclude, for purposes of this motion, that Dr. Toubia's criticisms go to the weight of Ms. Butler's findings, not their admissibility. See, e.g., Johannessohn v. Polaris Indus., Inc. 450 F. Supp. 3d 931, 972 (D. Minn. 2020) (rejecting challenges to a conjoint survey conducted by Ms. Butler that were similar to those raised by Dr. Toubia, on the basis that they did not preclude admissibility, but were better reserved for cross examination.); In re Whirlpool Corp. Front-Loading Washer Prods. Liab. Litig. , 45 F. Supp. 3d 724, 753 (N.D. Ohio 2014) (rejecting challenges to the admissibility of another conjoint survey conducted by Ms. Butler that were similar to those raised here by Dr. Toubia, concluding " ‘[a] survey need not be perfectly conducted for testimony concerning its results to be admissible. So long as the expert's testimony and the underlying survey have probative value after all the survey's deficiencies are taken into account, testimony concerning the results for the survey that meets the basic requirements of usefulness and reliability is admissible into evidence, and the trier of fact may accord it the weight it deems proper.’ " (citing 4 Jack B. Weinstein and Margaret A. Berger, Weinstein's Federal Evidence § 702.06 (2d ed. 1997)). Therefore, I will accept that Dr. Prince appropriately relied on Ms. Butler's findings in developing his overpayment model.
I can use accepted scientific methods to estimate prices that Plaintiffs would have paid in a but-for world where Marriott had disclosed its inadequate data security practices by drawing on the empirically estimated reduction in consumer demand from Ms. Butler's results. The estimates of these but-for prices can be obtained through econometric modeling of the pricing decisions that Marriott likely would have made upon observation of lower consumer demand and the responses of Marriott's rivals to any change in Marriott's prices. That is, I can use econometric modelling techniques employed in economic literature to estimate equilibrium pricing decisions that account for both demand and supply responses to reduced consumer demand.
Id. at ¶ 8.
Dr. Prince initially tested his model using current (as of 2021) hotel pricing information (as opposed to "historical" hotel pricing information relating to the relevant time period that the data breach took place, between 2014–2018) for Marriott hotels and their competitors in four cities. Prince Dep. 1 at 68:1–17; 79:16–80:5. He did this not for purposes of calculating actual overpayment damages, but rather just to demonstrate that his model could be used for the purpose of calculating overpayment damages on a formulaic basis. Id. at 90:1–3; 164:17–165:1. But Dr. Prince was confident that, when called upon to calculate class member damages relating to hotel stays on a particular date during the data breach period and a specific Marriott "focal hotel," he would be able to locate the historical hotel pricing and competitor information needed to do so. Id. at 205:1–21; 210:2–211:10. He subsequently did so in his rebuttal report, when he tested his overpayment model for six Marriott focal hotels using historical pricing information and NDS database information for bellwether plaintiffs. Prince Rebuttal Rep. at ¶¶ 44–51.
By "formulaic," Dr. Prince means that his overpayment model may be used by each member of a certified class to calculate their individual overpayment damages. See Prince Dep. 1 at 94:9–11. Since the exact same model may be used without modification by each class member (albeit, by inputting the data regarding each of their particular stays at a Marriott hotel during the data breach period), Plaintiffs assert that overpayment damages may be calculated classwide. This concept is discussed in more detail in the Memorandum and Opinion ruling on class certification. See Section III.
It should be acknowledged that Dr. Prince's first report is not an easy read for lay persons lacking a solid (if not advanced) knowledge of economics. And he did not advance his cause by his unnecessarily guarded, equivocal, and at times evasive testimony during his first deposition. But what can be pieced together from his first report, first deposition, rebuttal report, and second deposition, is that he selected his economic model for measuring overpayment damages from a series of well-regarded economic publications (some of which were studies of the hotel industry, and employed methodology identical or similar to that used by Prince in his overpayment model) including: Barry, Levinsohn, and Pakes (1995) ("BLP") (discussed at length in Prince Initial Rep. at ¶¶ 95–120); Nevo (2000), Mazzeo (2002), Cho (2018), Petrin (2002), and Wilson (2011). See, e.g., Prince Rebuttal Rep. at ¶ 15 (citing Mazzeo and BLP), ¶¶ 19–21 (citing Nevo), ¶ 43 (citing Wilson); Prince Initial Rep. at ¶ 64 (citing Cho), ¶ 117, n.184 (citing Petrin). In addition, his model was designed to replicate Marriott's own internal program for pricing hotel rooms against the competition, known as "One Yield," Prince Rebuttal Rep. at ¶¶ 14, 18, 36, 40, as well as the testimony of a Marriott 30(b)(6) witness familiar with Marriott's pricing of hotel rooms to account for competitors’ pricing, Id. at ¶¶ 14, 34 (citing Marriott Rule 30(b)(6) Deposition of Russell Vereb) (May 26, 2021)).
Steven Berry, James Levinsohn, and Ariel Pakes, Automobile Prices in Market Equilibrium , 63 Econometrica, 841 (1995). This article introduced a new method to estimate demand and supply structurally in oligopolistic differentiated product markets. Id. It relied on aggregate level consumer and supplier data and undertook to demonstrate that market demand and supply curves can be determined on the basis of product-level and aggregate consumer-level data. Id. Although the market studied was the automobile industry, the underlying principles have widely been followed for other oligopolistic differentiated product industries, such as the hotel industry. Id.
Aviv Nevo, A Practitioner's Guide to the Estimation of Random-Coefficients Logit Models of Demand , 9 J. of Econ. & Mgmt. Strategy 513 (2000). This is a very sophisticated and highly cited article aimed at advanced degree economists that extensively discusses how to implement the BLP model and similar models. Id. It concludes that the researcher does not need individualized information or data, but only market level price data, quantity data, and product characteristics data. Id. It analyzes competition based on product characteristics. Id. Moreover, it demonstrates that only a limited number of product characteristics, determined by the researcher, is needed to accurately estimate the model. Id. And, it provides a detailed discussion about how to determine the total market size, which is necessary to identify an "outside option." Id. Although this study focused on the cereal market, it has extensively been used for many other types of markets. Id.
Michael J. Mazzeo, Competitive Outcomes in Product-Differentiating Oligopoly , 84 Rev. of Econ. and Stat. 716 (2002) (describing a structural model methodology of supply and demand in hotel markets and providing guidance as to how to measure prices and competition in the hotel industry, measures that Dr. Prince discussed and adopted as part of his overpayment methodology).
Sungjin Cho et al., Optimal Dynamic Hotel Pricing, Working Paper (2018). Although not a peer reviewed article, it is a working paper co-authored by economics professors at Seoul National University and Georgetown University. It is a sophisticated study of the hotel industry showing that hotel prices reflect co-movement, which was relied on by Dr. Prince in developing his overpayment model. Id.
Amil Petrin, Quantifying the Benefits of New Products: The Case for the Minivan , The J. of Pol. Economy (2002) (discussing, extending, and enhancing the BLP model).
Nathan E. Wilson, Branding, Cannibalization, and Spatial Preemption: An Application to the Hotel Industry , FTC Bureau of Economics Working Paper No. 309 (using the BLP demand estimation analysis when studying the hotel industry).
With respect to the methodology most heavily relied on by Dr. Prince, the BLP model, the articles cited by Dr. Prince and description given by him in his written reports and deposition testimony, it appears that his overpayment model would operate formulaically in three analytical steps, as follows.
See Prince Initial Rep. at ¶¶ 95–119.
Step One: Gather data relating to each Marriott focal hotel where a class member stayed during the data breach period, for an appropriate time period (discussed below), which includes: identifying a Marriott focal hotel (where a class member stayed); identifying a set of competitors for that focal hotel (up to a maximum of nine, based in part on Marriott's One Yield program); identifying the core set of hotel features for the focal hotel and its nine competitors (collectively, the "choice set") (selected based on the results of the Butler conjoint consumer survey, academic literature, and historical information regarding the hotel tier, location, amenities, brand, and prices of each of the hotels in the choice set); identifying the minimum price charged by the ten hotels in the choice set for the appropriate time selected to run the model (taken from the Mazzeo paper and data collected from historical sources); identifying the appropriate number of hotel rooms for the particular market (measured by total rooms or number of rooms sold); and, finally, calculating the "outside option" (meaning the statistical probability that a consumer would choose a hotel room other than from the ten choice set hotels in that particular market, a step apparently informed by the BLP, Nevo, and Jones and Chen studies, which does not prescribe any single immutable method of selecting an "outside option," provided the manner chosen generates a positive statistical probability that a consumer would choose lodging other than from the choice set for that market).
In determining how to calculate the outside option in this case, Dr. Prince was guided by a paper that analyzed the hotel industry in Las Vegas, Nevada, Peter Jones & Meng-Mei Chen, Factors Determining Hotel Selection: Online Behaviour by Leisure Travelers, Tourism and Hospitality Rsch. (2011), which he cited in his reports. Prince Initial Rep. at Appx. 3, n.209, Prince Rebuttal Rep. at ¶ 21, n.18. Dr. Tucker also referenced it. Tucker Rep. at ¶ 88, n.179–81. This paper studied consumer practices in selecting a hotel to stay in during leisure travel and concluded that the process is done in two stages. The first narrows all possible hotels to a "consideration set." At the second step, these hotels were filtered to obtain a "choice set," from which the one hotel was selected for the stay. Dr. Prince was influenced by this paper in choosing the consideration set and choice set in his BLP model, which are necessary to establish the outside option.
Step Two: Factor in the results of the Butler conjoint study as they relate to a consumer's willingness to pay for a hotel room in the hypothetical but-for world (that is, consumer willingness to pay for a hotel room when aware that Marriott may not have adequate data security protections, which could lead to a loss of consumer PII—a price that is lower than the actual price paid by the class member in the "real world," when he or she purchased a hotel room from Marriott at a particular focal hotel, at a particular time). This step is intended to simulate market dynamics (i.e., supply and demand dynamics) that generate the "overcharge" caused by the Marriott data breach. In other words, the percentage difference in price for the hotel room actually paid by the class member (the "real world price") and the reduced price that a consumer would be willing to pay in the "but-for" world, as determined by the Butler conjoint survey.
Step Three: Determine the actual overcharge damages for a particular class member and a particular stay during the data breach period. Only the following data would be required for each class member, all of which Prince stated could be derived from the Starwood NDS database: the class member for each stay, the focal Marriott hotel of the stay, the price that class member paid for the Marriott focal hotel room, and the dates of the stay.
Defendants, of course, challenge this methodology on a variety of bases. The clearest explanation of these objections is found in Dr. Tucker's supplemental report filed on February 18, 2022. Tucker Suppl. Rep. Dr. Tucker is particularly critical of Dr. Prince's overpayment model because, she contends, he only proposes to analyze the market for each focal Marriott hotel at a single point in time, despite the fact that class members may have stayed there at multiple times during the data breach period. This, Dr. Tucker contends, is economically unsound because the competitive market for the same focal hotel can change in material ways over a four-and-a-half year period. Id. at ¶¶ 3, 6, 11, 16, 18, 20–22.
It is true that Dr. Prince generated considerable confusion about the frequency with which he would test the competitive market for each Marriott focal hotel. In his first deposition, he testified that, once he had access to the full Starwood NDS database, he planned to run his overpayment model for each class member stay at each focal hotel during the entire data breach period, meaning that he would have to run his model numerous times for each individual market. Prince Dep. 1 at 93:15–94:11; 97:21–98:16; 102:2–6. And, with as many as 1200–1400 focal hotel locations that might need to be examined, that could amount to an enormous number of calculations. In his rebuttal report, Dr. Prince clarified that only one run of the model is required for each focal hotel's competitive market. Prince Rebuttal Rep. at ¶ 43. But it mischaracterizes Dr. Prince's position to insist, as Defendants do, that his belief that he only needed to test each focal hotel's competitive market a single time was immutable. To the contrary, he clearly stated in his first deposition (and again in his second deposition) that it might be necessary to run the model more than once for each market, but he would need the full NDS database in order to determine this. See, e.g., Prince Dep. 1 at 102:2–6 ("Well, I haven't gone to parsing it by time period, but I would say there may be a need to do analysis at different points in time. I just, sitting here now, don't want to go so far as to say what the segments in time would be"); Prince Dep. 2 at 359:1–14 ("Well, I think for each—with each market, what I would do is look to see if there's a change of competitors. If there was, I can assess multiple runs of the model for changes in the competitors. And then, I also can do sensitivity checks to runs at multiple points in time, even if there was the same number of competitors.... So, what I'm saying is that one is required, but that doesn't preclude me from doing checks to see if multiple runs providing any information about the usefulness of running multiple times.").
Prince Dep. 1 at 103:5–9. Of course, because I have only been asked to certify liability and damages classes for class members from California, Maryland, and New York under the overpayment theory, the number of Marriott focal hotel stays that must be examined once Dr. Prince has access to the NDS database will be substantially reduced. In other words, he will NOT be called upon to examine the competitive markets for each of the 1200–1400 Marriott focal hotels for each class member for all 50 states for the full 4.5-year data breach period. Rather, he would need to examine only the markets for the hotels where the relevant class members stayed for only the relevant time periods.
Moreover, Dr. Prince justified his impression that a single run of his overpayment model for each focal hotel could be sufficient by his explanation that there was support in the economic literature that hotel prices within the same competitive market reflect "co-movement" in prices over time. Prince Dep. 2 at 360:9–14 ("[Co-movement is] the simple idea of prices amongst, in this case, competing hotels, moving in a correlated way. So, one hotel's price goes up; other hotel's prices tend to also go up. One hotel's price goes down the other hotels’ prices also tend to go down"); 361:9–24. Based on this characteristic of the hotel industry, Dr. Prince concluded that "predictable patterns of pricing" could be derived allowing him to assume that within a particular Marriott focal hotel market, prices would co-move at the same rate throughout the data breach period, allowing him to apply the same percentage of overcharge for each stay of a class member at that focal hotel for the entirety of the data breach period. He based this assumption on a working paper published by four economists (at least one of whom, John Rust, is highly respected amongst economists).
Prince Dep. 2 at 361:5–17.
Sungjin Cho, Gong Lee, John Rust, and Mengkai Yu, Optimal Dynamic Hotel Pricing (Soc'y for Econ. Dynamics Meeting Papers 179, 2018), https://economics.sas.upenn.edu/index.php/system/files/2018-04/hp_final_update.pdf (cited by Dr. Prince at Prince Rebuttal Rep. at ¶ 43, n.46).
As noted, Dr. Prince primarily relies on the empirical fact of "co-movement" of hotel prices to support his conclusion that he may need as few as one run (but, if more than one run, only a few) of the model for each Marriott focal hotel's competitors, rather than for each stay of each class member, as Marriott contends is necessary. See, e.g., Prince Rebuttal Rep. at ¶¶ 40, 43, n.47; Prince Dep. 2 at 412:8–413:1; 421:1–25; 437:1–25. In one instance, he also contends that co-movement of hotel prices would allow him to "impute" the prices charged by Marriott focal hotel competitors during the more than four-year duration of the data breach, if he were unable to find the prices those competitor hotels actually charged. See Prince Rebuttal Rep. at ¶ 43, n.49 ("To the extent that price data are not available for competitors in an area for a comparable set of [Marriott] booking and stay dates, even these situations can be addressed through the calculation of an imputed price.... According to the results of empirical studies such as Cho et al. (2018) [the study that found strong evidence of co-movement of prices within the hotel industry], it would be reasonable to estimate an imputed price for [a competitor hotel based on the prices of other competitor hotels whose actual prices were ascertainable]....").
Dr. Tucker strongly disagreed, of course, and criticized Dr. Prince for assuming co-movement of hotel prices without actually testing for it with respect to the Marriott focal hotels. Tucker Suppl. Rep. at ¶¶ 13–20. If Dr. Prince had reached final conclusions regarding the overpayment damages for all of the class members in this case, that criticism might doom his calculations. But, for purposes of class certification, that was NOT what he was asked to do. He was asked to develop a model for formulaically measuring overpayment damages that could be used to calculate these damages for each member of a specific certified class. He developed his model, initially tested it using current information, and later tested it again using historical data for bellwether plaintiffs in six cities. He supported his assumption that hotel prices reflect co-movement (thereby enabling him to assume that one run of his model for each focal hotel competitive market was sufficient) by reference to supporting economic literature. Dr. Prince candidly acknowledged that he might need to adjust the application of his model once he received the additional information from the NDS database, and he was confident that his model would allow him to do so. As he made clear:
I note that there are many practical alternatives, none of which require individualized analysis at the class member level.... But it is not possible to know which calibration will be most accurate without reviewing information that Marriott has not yet had to produce: i.e., information on Marriott's non-class representatives customers.... The markets upon which the model is applied ultimately depends upon the information that Marriott will provide for all class members at the damages phase. I do not have access to the complete list of stays by proposed class members, and it is my understanding that Marriott has not been required to turn over this information before plaintiffs are certified as a class.
Prince Rebuttal Rep. at ¶¶ 21, 25.
And the same applies to Dr. Tucker's many other criticisms of Dr. Prince's methodology, including his method of calculating the outside option, changes in the competitive market for each focal hotel market, changes in hotel characteristics (amenities, size, tier, and brand), as well as the errors that Dr. Tucker found in some of Dr. Prince's data used to test his methodology. Tucker Suppl. Rep., Section III.
I am persuaded, therefore, that Dr. Prince has met the requirements of Rule 702 for purposes of identifying a model for formulaically calculating overpayment damages for the discrete liability and damages cases for which Plaintiffs seek certification. First, there is no question that Dr. Prince is highly qualified. Second, he supported his model with sufficient factual data for its limited intended purpose (developing a model for future use to calculate damages formulaically once additional discovery was received from Marriott). Third, he developed his methodology by reference to reliable economic literature. And fourth, he tested his methodology with both current and historical information relating to the bellwether plaintiffs, to whose NDS database information he was given access, thereby applying his model, to the extent then possible, to the facts of this case.
Moreover, once he receives the additional discovery he needs to run his model for the class members that have been ascertained following classwide notice, and he has access to the NDS database for those class members, he will be able to run his model for those specific class members for their stays at Marriott focal hotels during the data breach period. This undoubtedly will entail far fewer calculations than those Dr. Prince was challenged with by Defendants during his depositions. It also will enable him to test whether he only needs one run of his model for each focal hotel during the entire data breach period, or whether more are needed (for example, because prices for that particular market did not co-move as anticipated throughout the data breach period, or the competitor set for that focal hotel changed in some significant way). And he will be able to calculate the outside option in a manner that has been accepted as reliable by the economic literature. This testing will allow error rates to be assessed by both Dr. Prince and Dr. Tucker, and depending on their magnitude, the results of Dr. Prince's calculations may be evaluated for admissibility at trial. But that is a ruling for another day.
II. Market/Inherent Value Damages
Dr. Prince's second category of damages "considered whether damages stemming from the inherent value of [Plaintiff's] ... data can be estimated formulaically for all Plaintiffs." Prince Initial Rep. at ¶ 10. Dr. Prince concluded that consumers "attribute value to preserving the privacy of their personal information. This value stems from consumers’ incentives to protect their personal information" to avoid such undesirable consequences as identity theft, receipt of unwanted marketing communications (such as spam or telemarketing calls), or public disclosure of sensitive personal information. Id. Dr. Prince also opined that there was market demand for consumer personal information, and that "these economic incentives to protect personal information, combined with demand for consumer information, produce market value." Id. From these assumptions, Dr. Prince derived "a method for estimating the revenue one could reasonably expect to collect for various combinations of personal data that were plausibly stolen from the class members." Id. at ¶ 11. He then used "public information about market prices for personal data," and estimated the value that could be obtained from the market for selling Plaintiffs’ data" that was compromised by the Marriott data breach. Id. And, because "individual records can be sold multiple times ... one can then multiply these prices by an estimated quantity of sales to obtain a formulaic calculation of the damages for each Plaintiff." Id. Dr. Prince concluded by stating:
I can apply the above damages methodology formulaically to all members of the proposed classes to whom a jury finds that Marriott is liable.... I understand that the Court has ruled that Marriott does not need to produce data for all proposed class members at this time, thus below I demonstrate the way in which my methodology could be applied to all proposed class members if Marriott does produce such data in the future.
Dr. Prince also noted that Marriott had "produced a significant amount of data about the bellwether plaintiffs two days ago." Prince Initial Rep. at ¶ 12. Inasmuch as his initial report was dated July 12, 2021, and his rebuttal report was dated October 12, 2021, Dr. Prince had over 90 days in which to test his methodology for determining market/inherent value on the NDS database information for each of the bellwether plaintiffs. He did not, and the significance of this failure to do so is discussed below.
Unfortunately, due to some inartful descriptions by counsel for Plaintiffs of this second category of damages as "diminishment of" the value of Plaintiffs’ PII, a good deal of time, energy, and resources of the parties was devoted to resolving a dispute about whether Dr. Prince had changed his methodology between his first and second reports. Defs.’ Letter Mot., ECF Nos. 961 (sealed), 963 (redacted); Pls.’ Letter Mot., ECF No. 964; Letter Order of January 18, 2022, ECF No. 967. I ultimately ordered that Dr. Prince be deposed a second time for the limited purpose of determining whether he had changed his methodology between his first and second reports, and the Special Master, Judge (Ret.) Facciola, capably presided over this deposition. Letter Order of November 30, 2021, ECF No. 940. Thereafter, counsel exchanged letter motions addressing whether Dr. Prince's explanation of his market/inherent value damages model (in his rebuttal report and second deposition) should be stricken, as well as Plaintiffs’ motion that portions of Dr. Prince's second deposition be stricken due to questioning by counsel for Defendants that Plaintiffs’ counsel argued exceeded the scope of the order allowing his second deposition. Defs.’ Letter Mot.; Pls.’ Letter Mot. I denied both motions but ordered that Dr. Tucker could file a supplemental report addressing Dr. Prince's second report and second deposition, ECF No. 967, which she did, ECF No. 984, filed February 18, 2022.
I am satisfied that the Consolidated Amended Complaint does sufficiently plead that Plaintiffs suffered the loss of the economic value of their PII as a result of the Marriott data breach, such that what Dr. Prince undertook to quantify was indeed a category of damages that Plaintiffs had pleaded. See, e.g. , Compl. at ¶¶ 5 (alleging loss of the "inherent value" of PII), 270 (alleging loss of the inherent value of PII), 304 (same), 1349 (same).
Dr. Prince's second deposition provides the clearest and most concise explanation of his methodology for formulaically determining Plaintiffs’ market/inherent value damages. For ease of reference, I will refer to this for the remainder of this opinion as Plaintiffs’ "market value" damages methodology.
In his second deposition, Dr. Prince unequivocally stated that his methodology for determining market value damages did NOT measure the change in price (in other words, diminution in value of PII caused by the data breach) of Plaintiffs’ PII that was compromised in the Marriott data breach. Prince Dep. 2 at 450:22; 451:8; 492:18–22 ("Q. Your methodology doesn't calculate whether the market price of data decreased as a result of ... the market price of data decreased as a result of the data breach, correct? A. Yes, that's right"). Rather, he explained that he undertook to calculate the revenue that could be estimated from the market value of Plaintiffs’ hacked PII. Id. at 451:13–452:1 ("I'm looking in terms of revenues, so price times quantity. So, I'm not measuring whether or how the price has changed. But I am thinking in terms of what it means in terms of a quantity change. Q. When you say ‘quantity change,’ can you clarify what you mean by that? A. So, what I mean is the number of sales one could reasonably expect those data could attain. Q. Right. And so, when you say you're talking about revenue, you're talking about the number of sales times—multiplied by an assigned market value; is that fair? A. By an assigned market price, yes").
Thus, it is essential to the operation of Dr. Prince's market value model that he determine the revenue ("R") that would be generated by the total number of sales (i.e., quantity, or "Q") of an individual plaintiff's PII at a particular market price ("P"). Expressed formulaically, his model can be described as follows: R (revenue) = P (market price) × Q (total number of sales of each plaintiff's particular "bundle" of hacked PII (for example, name, address, email address, credit card number, credit card expiration date, credit card security code, passport number, etc.)). Therefore, Dr. Prince would need to determine the specific "bundle" of PII for each class member that was compromised by the data breach, determine the market value for that particular bundle, and, finally, determine the number of times that a plaintiff's bundle was "sold" in order to reach the total market value loss for that plaintiff. In response to skillful questioning by counsel for Accenture, Dr. Prince testified that there were four steps required under his methodology.
Step One: Dr. Prince agreed that the first step of his market value model required him to identify the data (PII) of each class member that was compromised by the breach. Id. at 456:2–12 ("Q. Okay. So it seems to me that the first step in your methodology is that you need to identify the data that are at issue for a given class member; is that fair? A. So, I would identify which types of data, for example, you know, home address, phone number, credit card number, that were compromised for a given class member, yes."). When asked how he would do this, Dr. Prince testified that he would get that information from the NDS database. Id. at 456:10–12 ("Q. How do you propose doing that? A. I believe that information, we would be able to get that from the NDS data."). Importantly, when Dr. Prince was asked, "Have you done that analysis for any potential class member to date?", he answered, "I don't believe I have damages calculation for a specific class member for the inherent value thus far." Id. at 456:13–17.
Recall that Dr. Prince stated in his first report, which was dated July 12, 2021, that he had recently received the NDS data for the bellwether plaintiffs. Prince Initial Rep. at ¶ 12. His rebuttal report was dated 90 days after his initial report (on October 12, 2021), and his second deposition occurred on December 23, 2021. Thus, Dr. Prince had more than five months in which to test his market value model on the NDS data for the bellwether plaintiffs in order to determine whether it could accurately calculate market value damages. But he did not.
When the obvious follow-up question was posed to Dr. Prince—"[w]hy didn't you attempt to perform that at least for one or more of the bellwether plaintiffs?"—he answered:
I'm trying to remember. I know we had some issues with the data the first time around. So, in the opening report, the data were basically unworkable for us until two days before the report was due. So that wasn't really a possibility. And then for the rebuttal, I think the method—the method, I think, is very clear. It's, I guess, it would be easy to do for any individual. But I think the method I thought was sufficiently clear.
Id. at 456:21–457:5.
On further questioning, he amplified his answer, saying "[b]ut, you know, I guess in this case it's—I feel like it's eminently clear. If you give me an individual and tell me they lost their address and phone number and credit card information, applying the method produces the number with very minimal steps." Id. at 457:15–20. Thus, Dr. Prince had the data he needed to test his market value methodology with respect to the NDS data for the bellwether plaintiffs (which he did with regard to his overpayment model). He had ample time before he filed his rebuttal report to do so. It would have been easy to do so. And he could have done so "with minimal steps." Id. But because he felt his method was "sufficiently clear," he never bothered to apply it to the facts of this case. As will be emphasized below, this is one reason (but not the only reason), why Dr. Prince's untested market value model is inadmissible.
As already noted, Dr. Prince did test his assumptions using bellwether plaintiff NDS data when he formed his opinions regarding the overpayment damages, so his failure to do the same for the market value model is contrary to his own approach. Inasmuch as Rule 702 and the Daubert factors place particular emphasis on the importance of testing expert assumptions, as opposed to conclusory assumptions, so that they will produce accurate results, the failure to test his assumptions regarding the market value model is especially concerning.
Neither Defendants nor the Court is obligated to accept the untested ipse dixit pronouncement of an expert that he is satisfied that his method is reliable and will produce accurate results, without the inconvenience of having to test it to show that it is. This is precisely the type of opinion that Rule 702 (b)–(d) guards against, by requiring as a condition precedent to admissibility that the expert's conclusions be supported by "sufficient facts or data" and be "the product of reliable principles and methods," and that the expert "has reliably applied the principles and methods to the facts of the case. " Fed. R. Evid. 702 (b)–(d) (emphasis added). And, while at the class certification stage, it is sufficient for Dr. Prince to show that he has a model that may reliably be applied to whatever class members are ultimately certified, this does not relieve him of the requirement to show that it can in fact produce reliable results by actually testing it. This is particularly so when, as here, Dr. Prince had access to the NDS data of the bellwether plaintiffs, ample time to test his model to demonstrate its reliability and the accuracy of its results, and it would have been "easy" for him to do so.
Step Two: Dr. Prince was asked, "So, is it fair to say that the next step here in your analysis would be to determine once you have an individual with a set of data, whether each component of that data set has value?" Prince Dep. 2 at 460:7–10. He answered, "Well, what I would say is I'd look for whether each component ... I would look for in the market is there a comparable data for each type for which there is a market price. And for each one that has a market price, that would be included in the set that I would search over." Id. at 460:11–16. He amplified, explaining that he and his team first looked at "different entities that sell data, and what kind of data is it, what kind of prices do they charge." Id. at 461:14–16. Then, they "looked for subsets of those data that, at least, would be considered reasonably comparable to the type of data that, [in] my understanding, was breached from Marriott." Id. at 461:16–19.
As with his first step, Dr. Prince touted the ease with which he would be able to determine the market value of the bundle of PII of any particular class member, as documented in the NDS database. He was asked, "Have you performed that analysis for data pertaining to any of the potential class members in this case?" Id. at 463:13–15. He responded:
So, looking at, for a given class member, what data was stolen from them, and then what's the value of those data? [Defendants’ counsel said "Yes," and Dr. Prince continued] As I said ... that wasn't even feasible with the opening report. And then the method, I believe it's so straightforward that, you know, literally you hand me any person, you tell me what the data fields are, name, address, phone number, I apply the method to that and the number pops out very easily."
Id. at 463:16–464:1. When Defendants’ counsel followed up by asking, "And so, the answer is, no, you have not done it, at least to this point, as to any individual potential class member?", Dr. Prince replied, "I haven't." Id. at 464:2–5. Thus, as with the first step of his model, Dr. Prince had the bellwether plaintiff NDS data and the time to test the second step of his model, but he elected not to do so.
Step Three: Defendants’ counsel then asked, "And then it seems to me that the third step in your methodology, Dr. Prince, is that you then determine the value of that bundle; is that fair?" Id. at 464:9–12. He answered, "I provide ways for measuring the value of that bundle." Id. at 464:13–14. He was asked how he would do this and responded that he described two approaches in his report. One was to set a "lower-bound" estimate, the other was a "less conservative" higher-bound value. Id. at 464:22–465:6. Defense counsel then asked whether Dr. Prince had "determined whether it's feasible to perform that analysis, the lower-bound and upper-bound analysis for all of the NDS data fields?" Id. at 465:6–9. He responded:
I guess I'm—I'm hard pressed to see what would preclude me from. being able to do that for all the NDS data fields. So, either if the data field doesn't have an associated price, then it wouldn't be included. If it has an associated price, then its included, and then I just apply the method. And literally I click "Go," and it would produce all the measures.
Id. at 465:10–17.
When he was asked by Defense counsel what he meant by "click ‘Go,’ " Dr. Prince explained that he and his team already did market analysis to determine the relevant data prices and had put together a table that summarized the prices so the remaining step would be to "execute the algorithms that I've laid out, and it will produce the damages figures." Id. at 466:15–467:1. But when Defense counsel asked him whether the data source tables included in Dr. Prince's report "reflect all available fields of data that exist in the NDS?" Dr. Prince responded, "I can't say that they necessarily cover all the fields in [the] NDS. So, you know, that—as I said, in the opening report, we didn't have the data in time to do that kind of analysis. For the rebuttal, I felt like the algorithm was clear and there was nothing further to elaborate on that." Id. at 467:7–15.
Algorithms are not omniscient, omnipotent, or infallible. They are nothing more than a systematic method of performing some particular process from a beginning to an end. Paul W. Grimm et al., Artificial Intelligence as Evidence , 19 NW. J. TECH. & INTELL. PROP. 9, 11 (2021) (defining algorithm). If improperly programmed, if the analytical steps incorporated within them are erroneous or incomplete, or if they are not tested to confirm their output is the product of a system or process capable of producing accurate results (a condition precedent to their admissibility), then the results they generate cannot be shown to be relevant, reliable, helpful to the fact finder, or to fit the circumstances of the particular case in which they are used. As with the failure to test the first two steps of his market value model against the NDS data for the bellwether plaintiffs, as he so easily could have done, Dr. Prince's willingness to rely on his own untested conclusion that his model could reliably be applied to the facts of this case is insufficient to meet the requirements of Rule 702.
See, e.g., Fed. R. Evid. 901(b)(9).
Step Four: Dr. Prince was questioned by Defense counsel regarding the final step of his market value model: "It seems to me that the final step of your methodology here is what you lay out at paragraph 138 of your original report. And that's to multiply the bundle price by what you call the quantity of sales, I believe: An estimated quantity of sales; is that right? A. Yes." Prince Dep. 2 at 471:4–10. When asked to explain how he was "going to make the determination of what that quantity of sales would be," Dr. Prince responded, "So I lay out the possibilities here. But the key thing that I emphasize is that none of these requires individual inquiry. It would be using—so, one of the papers I cite is the [OECD] paper in footnote 208 [Prince Initial Rep. at ¶ 138], which talks about exactly this type of issue." Prince Dep. 2 at 471:14–19. Dr. Prince stated that for "data-heavy" companies whose "value is largely determined by the data that they hold" he could:
Citing Org. for Econ. Coop. & Dev. ("OECD"), Exploring the Economics of Personal Data: A Survey of Methodologies for Measuring Monetary Value , at 25, OECD Dig. Econ. Papers No. 220 (Apr. 2, 2013), http://dx.doi.org/10.1787/5k486qtxldmq-en. For ease of reference, I will also refer to this publication as the "OECD paper."
[S]imply do the ratio between the value of the company and the amount of data that they have as an approximation for, you know, how much—what kind of turnover they get out of the data that they have, which would then feed into the overall value of the company. So that's one way of going about it. That paper [OECD] also talks about looking at revenues, for example. But the bottom line is, there are multiple ways to think about that. And they're all very general, not requiring individual inquiry.
Prince Dep. 2 at 471:23–472:9. But when asked, "[S]itting here today, it sounds like you don't know precisely which direction you're going to take on this quantity of sales, right?", Dr. Prince replied, "I would say I would probably start with exactly what I just described." Id. at 472:10–14. When asked whether his model assumed that the same quantity of sales applied for "each and every class member," Dr. Prince replied:
It needn't necessarily be. I mean, one could potentially allow for a different estimate for Q [the number of sales of data for each class member] for different combinations of data. But one could very plausibly just estimate kind of a baseline Q that would be applied to everyone as well. That would be another reasonable approach.
Id. at 472:17–22.
This exchange reveals a number of serious (and unanswered) concerns about the reliability of the methodology Dr. Prince would use to arrive at the number of times the data for each class member was sold, or "Q", without which he simply cannot calculate classwide market value damages. First, the OECD study, on which Dr. Prince seems entirely to rely in determining the quantity of sales of class member PII, clearly explains the drawbacks of each of the methods Dr. Prince appears to accept (at face value) as "plausible" market valuation methods to calculate the total number of data sales—summarized conveniently in a chart at p. 20 of the OECD paper. The chart makes clear that the methods described by Dr. Prince may be used as measures of value of personal data ("indicators based on market valuation"), not methods of determining the total number of sales of personal data of any individual consumer. And Dr. Prince fails entirely to explain how methodology designed to be used for one purpose (valuation of personal data) reliably can be used for a completely different purpose (estimating the total quantity of sales of one person's personal data). This fails to comply with the requirement of Rule 702(d) that the proponent of expert opinion testimony has "reliably applied the principles and methods to the facts of the case. " Fed. R. Evid. 702(d) (emphasis added).
The earlier exchange demonstrates the off-the-cuff nature of Dr. Prince's approach to his market value damages methodology. He makes it quite clear in his deposition testimony and reports that he relied heavily on the OECD paper in developing his methodology. Dr. Prince claims that the multiple possible models for calculating the value of individual items of PII identified in the OECD paper (each with its own drawbacks and potential for erroneous conclusions) would yield a different quantity of PII sales for any or every class member and could account for different values for each class member depending on the combination of comprised data for each class member. Prince Dep. 2 at 472:17–22. However, all financial valuation methods proposed in the OECD paper (market capitalization, revenue, profit) create equal value for each data record, no matter what the content, and all methods infer a common "Q" for each record. Dr. Prince has failed to identify how he would be able to formulaically calculate a separate "Q" for each individual record using the methods proposed. But even if Dr. Prince could justify this economic legerdemain, he failed to test it using the available NDS database for any bellwether plaintiff, which itself falls short of complying with Rule 702.
Second, and importantly, the OECD paper prefaces its discussion of various means of estimating the value of personal data based on market valuation with a clear caveat:
There are two important points that should be kept in mind in this context. Firstly, it must be highlighted that there is no single, perfect measure of value of personal data. Each of the presented measures might suffer from certain methodological biases and measurement errors. It should also be reemphasized that it is crucial not to isolate personal data from the underlying context of the business model in question.
OECD Paper at 19. Turning to the first method discussed in the OECD paper—"financial results per data record" (the one that Dr. Prince said he "probably would start with")—it is described as "[a]ggregated market cap (revenues, or net income) of a company divided by the total number of personal data records used by this company." OECD Paper at 20. Its benefits are described as "relatively easy to identify, reflects actual economic value added generated through personal data." Id. But its potential drawbacks are described as follows: "Likely to be inaccurate, as numerous other components impact market cap/revenues/income of a company; [p]ossible synergy effects could lead to overestimates for firms with larger datasets; [a]ppropriateness of this approach depends on what portion of turnover is directly tied to personal data." Id. But nowhere in either his initial report (where he mentioned the OECD paper in a footnote on the next to last page of his report), his rebuttal report, or either of his depositions, does he explain how he determined that this methodology's drawbacks did not outweigh its benefits, or that it would be appropriate for use in estimating "Q"—total sales of class member personal data for purposes of calculating market value damages. And just as concerning, he did not attempt to test this methodology using the NDS data of any bellwether plaintiff, despite the fact that he had this data and months in which to do so.
The OECD paper provided more detail about the use of the "financial results per data record" method of calculating the value of personal data. It stated, relevantly: "The first method for estimating the value of personal data looks at the financial results of a company such as market capitalization, revenues, and net income on a per-user or per-record basis as a way to capture the market values of the data in each record. This type of analysis only works for firms that either derive most, if not all, of their revenues from personal data or firms that separate out revenues for personal data activities. Many companies that work with personal data are not publicly traded or do not report information in sufficient detail to perform these analyses." OECD Paper at 20. Dr. Prince does not explain how he determined that this model—which he stated was the one he "probably" would start with, was appropriate for estimating the quantity of class member data sales, given its limitations as noted in the paper. Thus, he fails to meet the requirements of Rule 702(c) (expert testimony is the product of reliable principles and methods).
Similarly, Dr. Prince's testimony that the OECD paper also allows him to determine the market value of PII by "looking at revenues" fails to account for the serious shortfalls of this methodology identified in the OECD paper itself. It states:
Prince Dep. 2 at 471:20–472:6.
There are several limitations related to the use of financial results (e.g., market cap, revenues or net income) divided by the number of users or records as a measure for value of personal data. There are numerous components other than personal data that impact financial results of a company, such as its human and physical capital stock, volume of other intangible assets and expertise. Consequently, this measure could significantly overestimate the value of personal data, particularly in industries where personal data are not the main input factor of production.... The measure of revenues per personal datum/user does have limitations because revenues only contribute to economic growth insofar as they generate added value (or surplus).... In addition, the relationship between the volume of personal data and financial outcomes may not be linear, even for companies that rely heavily on personal data. For example, it is likely that the use of personal data in an economic activity could include synergistic effects, i.e., when for a given company the economic value of a single isolated data record is smaller than the economic value of the same data record in a large, consistent dataset. Such synergies could in turn lead to significant estimation biases.
OECD Paper at 24–25 (emphasis added). Dr. Prince does not explain how he accounted for these shortfalls in his model, and, of course, he failed to test it with the NDS database information for any of the bellwether plaintiffs. This fails to comply with the requirements of Rule 702.
Another of the methods for estimating value of personal data based on market valuation discussed in the OECD paper (not specifically referenced by Dr. Prince, but presumably included in his reference to the OECD article as presenting "multiple ways" to determine quantity of class member data sales) is described as "market prices for data." Id. at 20. It is described as "price per personal data entry offered on the market by data brokers." Id. Its benefits are referenced as "relatively easy to identify, reflects market value of a given specific data entry," and its potential drawbacks as "[a]part from the data value, it includes the cost of data search and processing. It also neglects the context in which the data is sold, which has a large influence on the demand (and price) for data." Id. As with the "financial results per data record" method, Dr. Prince failed to explain how this method of estimating the value of personal data based on market data could be used to estimate the total number of sales of each class member's NDS data, a completely different purpose. And, as already noted, Dr. Prince did not feel the need to test this method using bellwether plaintiff NDS data and actual market prices for this data. Accordingly, he likewise fails to meet the requirements of Rule 702 with respect to its use.
The same may be said of the remaining valuation methods listed in the OECD paper, which Dr. Prince did not discuss in any detail. And whether considered individually or collectively, Dr. Prince has failed to identify and test with available bellwether NDS data any method of determining "Q"—the number of sales of class member personal data, which is an essential component of his model to formulaically determine classwide damages for loss of market value of class member PII. His failure to test his assumptions with the NDS data that he had for the bellwether plaintiffs (even though he repeatedly testified that it would be easy for him to do so) is in sharp contrast to the methodology he applied to his overpayment model—where he first tested his model with historical data (because he had not received NDS data) and later re-tested using NDS data for the bellwether plaintiffs, once it was made available. Rather, for his market value damages, Dr. Prince was content to just trust his assumptions that his model would work, and that his algorithm was equal to the task of performing the calculations that were needed. And his vagueness about how he would determine Q, the quantity of sales for each class member's PII—an essential component to his ability to calculate the revenue those sales would have produced (which is what the damages would be)—leaves half of his formula both unexplained as well as untested. Moreover, his failure to address the warnings in the OECD article about the shortcomings of each of the potential methods he said he could possibly use to calculate market revenue damages fails to demonstrate that any of these methods would produce reliable results. Thus, at each of the four steps of his market value damages model, Dr. Prince fails to meet the requirements of Rule 702. Failure at any of the four steps would be sufficient to disqualify the market value model. Therefore, I am GRANTING Defendants’ motion to exclude this category of damages.
Neither did he explain how to overcome the substantial shortcomings of each of the valuation methods identified in the OECD article itself, and, of course, he failed to test his assumptions using bellwether plaintiff NDS data. Taken individually or collectively, these failures further demonstrate that Dr. Prince's market value model fails to meet the requirements of Rule 702.
Dr. Tucker raises many other valid deficiencies in Dr. Prince's market value model in her rebuttal report, but I need not discuss them because the above discussion amply demonstrates why Dr. Prince's model does not pass muster under Rule 702. See Tucker Suppl. Rep.
CONCLUSION
In conclusion, Defendants’ motion to exclude Dr. Prince's opinions regarding the overpayment damages model is DENIED for purposes of ruling on the class certification motion. Further discovery will proceed, under the capable supervision of the Special Master, with the production of relevant NDS database information regarding the class members for which the court has granted class certification. A supplemental report will be prepared by Dr. Prince once he has had the opportunity to test his overpayment model with the forthcoming discovery. Dr. Prince may be deposed again regarding his supplemental report. Dr. Tucker may submit a supplemental rebuttal report to address the supplemental report by Dr. Prince, and Dr. Tucker may be deposed. Thereafter, if Defendants seek to renew their challenge to Dr. Prince's overpayment methodology, they may submit a three-page letter setting forth the anticipated grounds for doing so, and a scheduling conference will be held with respect to briefing the motion.
Defendants’ motion to exclude Dr. Prince's opinions regarding the market value damages model is GRANTED. A separate order memorializing this opinion will follow.