Opinion
CIVIL ACTION H-14-1189.
05-04-2017
HOUSTON FEDERATION OF TEACHERS, LOCAL 2415, et al., Plaintiffs, v. HOUSTON INDEPENDENT SCHOOL DISTRICT, Defendant.
Manuel Alfonso Quinto–Pozos, Martha Powell Owen, Deats Durst & Owen PLLC, Austin, TX, for Plaintiffs. Chelsea Gabrielle Glover, James C. Ho, Kyle D. Hawkins, Gibson Dunn and Crutcher, Dallas, TX, Russell H. Falconer, Lisa R. McBride, Thompson & Horton LLP, Houston, TX, for Defendants.
Manuel Alfonso Quinto–Pozos, Martha Powell Owen, Deats Durst & Owen PLLC, Austin, TX, for Plaintiffs.
Chelsea Gabrielle Glover, James C. Ho, Kyle D. Hawkins, Gibson Dunn and Crutcher, Dallas, TX, Russell H. Falconer, Lisa R. McBride, Thompson & Horton LLP, Houston, TX, for Defendants.
AMENDED SUMMARY JUDGMENT OPINION
Stephen Wm. Smith, United States Magistrate Judge
This case presents a matter of first impression in this circuit—the use of privately developed algorithms to terminate public school teachers for ineffective performance. Of course, an employer's impulse to quantify employee performance is neither new nor inherently objectionable. The difficulty, as this case illustrates, is the tension between the understandable secrecy surrounding proprietary algorithms developed by private commercial enterprises, on the one hand, and the Fourteenth Amendment due process protections against substantively unfair or mistaken deprivations of life, liberty, or property, on the other.
See Frederick Winslow Taylor, The Principles of Scientific Management (1911).
Courts are beginning to confront similar due process issues about government use of proprietary algorithms in other contexts. See State v. Loomis, 371 Wis.2d 235, 881 N.W.2d 749 (2016), petition for cert. filed No. 16–6387 (U.S. Sup. Ct. Oct. 12, 2016) (use of computerized risk assessment tool as non-determinative factor in sentencing did not violate due process).
At issue here is the constitutionality of the "value-added" teacher appraisal system used by the Houston Independent School District during the 2011–15 school years. Plaintiffs include the Houston Federation of Teachers, a labor union with over 6,100 members that represents teachers and other HISD employees. Nine individual HISD teachers have also joined the suit. Before the court is HISD's motion for summary judgment (Dkt. 65). Having considered the parties' submissions and argument of counsel at a hearing on December 5, 2016, HISD's motion is denied in part and granted in part.
The parties have consented to magistrate judge jurisdiction for all purposes, including final judgment (Dkt. 13).
Background
In 2010, HISD began its transition to a "data driven" teacher appraisal system, with the goal of "having an effective teacher in every HISD classroom." The new system was implemented in 2011–12, evaluating teachers based on three components: (1) instructional practice; (2) professional expectations; and (3) student performance. The weight given each component has varied over the years. The focus of this litigation is on the third criterion, student performance, particularly HISD's new method of rating teacher effectiveness based on proprietary algorithms belonging to a private company.
P.Ex. 25.
From 2012 through 2014, HISD counted student performance as the most significant component, comprising 50% of the teacher's overall appraisal. For the 2014–15 school year, student performance counted 30%, instructional performance 50%, and professional expectations 20%. P.Ex. 27. There is a fact dispute whether the student performance and instructional performance criteria actually overlap, given HISD's "alignment" policy, discussed infra.
The basic idea behind the new appraisal system is that a teacher's impact on student performance, for better or worse, can appropriately be measured by student growth on standardized tests. This is generally referred to as the "value-added model" (VAM) for evaluating teacher effectiveness. Under HISD's new policy, student growth will whenever possible be calculated by a value-added statistical model called the Educational Value–Added Assessment System (EVAAS), developed by private software company SAS and licensed for use by HISD. The EVAAS system measures teacher effectiveness by attempting to track the teacher's impact on student test scores over time. The details are more complicated, but in general a teacher's EVAAS score is based on comparing the average test score growth of students taught by the teacher compared to the statewide average for students in that grade or course. The raw EVAAS score is generated by SAS's proprietary software and is then converted to a test statistic referred to as the "Teacher Gain Index" (TGI), based on the ratio of the EVAAS score to its standard error. The TGI is sorted into one of five "value-added" effectiveness ratings, described in the following table:
P.Ex. 25.
Id. Not all subjects and grade levels have sufficient standardized test data to enable value-added analysis, so not all teachers can be given an EVAAS score. See P.Ex. 62.
HISD represents that it discontinued its contract with SAS in 2016, did not calculate EVAAS scores for the 2015–16 school year, and currently does not use a value added model in the evaluation process. Dkt. 66 at 10 n.1. The voluntary cessation of allegedly illegal conduct does not render a case moot, however. United States v. W. T. Grant Co., 345 U.S. 629, 632, 73 S.Ct. 894, 97 L.Ed. 1303 (1953). HISD concedes that it is investigating all of its options for value-added modeling going forward, including SAS's EVAAS product. Dkt. 86 at 43.
P.Ex. 62; P.Ex. 64 (Rothstein Report).
P.Ex. 64 at 13–16. Standard error is a measure of statistical significance at various confidence levels.
Dkt. 71 at ¶ 28.
VALUE-ADDED EVAAS ®TGI RELATIONSHIP TO RATING EXPECTED AVERAGE GROWTH Well above Equal to or greater than 2 Students on average substantially exceeded expected average growth Above Equal to or greater than 1 but Students on average less than 2 exceeded average growth No detectable Equal to or greater than -1 but Students on average met difference less than 1 expected growth Below Equal to or greater than -2 but Students on average fell less than -1 short of average growth Well below Less than -2 Students on average fell substantially short of expected average growth
SAS's source codes and other information underlying the EVAAS statistical methodology are proprietary trade secrets unavailable to plaintiffs or HISD.
P.Ex. 5 (Defendants' responses to requests for production); Dkt. 73 at 10.
Plaintiffs challenge the use of EVAAS under various aspects of the Fourteenth Amendment, including:
1. procedural due process, due to lack of sufficient information to meaningfully challenge terminations based on low EVAAS scores;
2. substantive due process, because there is no rational relationship between EVAAS scores and HISD's goal of employing effective teachers;
3. substantive due process, because the EVAAS system is too vague to provide notice to teachers of how to achieve higher ratings and avoid adverse employment consequences; and
4. equal protection, because HISD has a policy of aligning teachers' instructional performance ratings with EVAAS scores.
HISD has moved for summary judgment on all counts. Additional facts will be discussed as relevant to the analysis below.
Summary Judgment Standards
Summary judgment is appropriate if no genuine issues of material fact exist, and the moving party is entitled to judgment as a matter of law. FED. R. CIV. P. 56(c). The party moving for summary judgment has the initial burden to prove there are no genuine issues of material fact for trial. Provident Life & Accident Ins. Co. v. Goel, 274 F.3d 984, 991 (5th Cir. 2001). Dispute about a material fact is "genuine" if the evidence could lead a reasonable jury to find for the nonmoving party. In re Segerstrom, 247 F.3d 218, 223 (5th Cir. 2001). "An issue is material if its resolution could affect the outcome of the action." Terrebonne Parish Sch. Bd. v. Columbia Gulf Transmission Co., 290 F.3d 303, 310 (5th Cir. 2002).
If the evidence presented to rebut the summary judgment is not significantly probative, summary judgment should be granted. Anderson v. Liberty Lobby, Inc., 477 U.S. 242, 249–50, 106 S.Ct. 2505, 91 L.Ed.2d 202 (1986). In determining whether a genuine issue of material fact exists, the court views the evidence and draws inferences in the light most favorable to the nonmoving party. Id. at 255, 106 S.Ct. 2505.
Analysis
1. Plaintiffs' protected property interests
The Fourteenth Amendment prohibits a state from depriving any person of life, liberty, or property without due process of law. When these constitutionally protected interests are implicated, the right to some kind of prior hearing is paramount. Board of Regents v. Roth, 408 U.S. 564, 569–70, 92 S.Ct. 2701, 33 L.Ed.2d 548 (1972). To evaluate such a claim, a court must first consider whether there is sufficient evidence implicating a protected property right in plaintiffs' employment. Cleveland Brd. of Educ. v. Loudermill, 470 U.S. 532, 538, 105 S.Ct. 1487, 84 L.Ed.2d 494 (1985).
HISD employs teachers under probationary, term, and continuing contracts. A continuing contract has no definite term and can be terminated only for good cause. TEX. EDUC. CODE § 21.154. Teachers employed under continuing contracts have a protected property interest in continued employment. Frazier v. Garrison I . S . D . , 980 F.2d 1514, 1529 (5th Cir. 1993). Probationary and term contracts create a protected property interest in employment during the term of the contract. Id. at 1531. The record is clear that most teachers represented by HFT have a constitutionally protected property interest in continued employment during their current term, and, for those with continuing contracts, beyond the current term.
HISD no longer enters continuing contracts with teachers, but it is undisputed that hundreds of teachers remain employed under continuing contracts. Dkt. 66 at 8; Dkt. 71–1 at ¶ 3.
Defendants do not contest HFT's associational standing. Dkt. 86 at 29.
HISD argues that a due process plaintiff must show actual deprivation of a constitutional right, as opposed to a mere conspiracy to deprive, citing Villanueva v. McInnis, 723 F.2d 414, 418–19 (5th Cir. 1984). However, Villanueva arose in a very different context—a § 1983 suit for damages against a former district attorney (McInnis) who conspired to murder the plaintiff (Villanueva). The Fifth Circuit reversed a jury award of money damages to Villanueva, because the defendant's co-conspirator (Rodriguez) tipped off the FBI so soon that "there was no actual threat that the 'conspiracy' would be carried out." Id. at 418. Key to the court's holding was the nature of the relief sought:
That the agreement between McInnis and Rodriguez was illegal or even 'unconstitutional' in an abstract sense such as might be posed by a quo warranto inquiry does not answer the question raised by this private suit for money damages. We are unable to identify in this inchoate 'agreement' an actual deprivation of any constitutional right of Villanueva.... While Villanueva's liberty or life interests may for a brief period of time have been sufficiently threatened to warrant injunctive relief the distance to a deprivation of liberty or life was here too great to lend definition to the constitutional right allegedly lost.
Id. at 418–19 (emphasis added to last sentence). In other words, while an actual deprivation might be necessary to support a damages award, a threatened deprivation is sufficient to support injunctive relief.
Plaintiffs here do not seek money damages, but rather a declaratory judgment and permanent injunction against the use of EVAAS scores in termination or nonrenewal of teacher contracts. It is well settled that "[o]ne does not have to await the consummation of threatened injury to obtain preventive relief." Pennsylvania v. West Virginia, 262 U.S. 553, 593, 43 S.Ct. 658, 67 L.Ed. 1117 (1923). To pursue a claim for preventive relief such as an injunction, plaintiffs need only show "a realistic danger of sustaining a direct injury as a result of the [policy's] operation or enforcement." Pennell v. City of San Jose, 485 U.S. 1, 8, 108 S.Ct. 849, 99 L.Ed.2d 1 (1988). As the following summary judgment evidence convincingly demonstrates, HISD's value-added appraisal system poses a realistic threat to deprive plaintiffs of constitutionally protected property interests in employment.
In 2012, shortly after adopting the EVAAS model for rating teacher effectiveness, HISD altered its teacher nonrenewal policy to add a new reason for nonrenewal—"insufficient student academic growth as reflected by value-added data." That same year, HISD declared a district-wide goal of ensuring that "no more than 15% of teachers with ratings of ineffective are retained district-wide." HISD explicitly defined teacher "effectiveness" by reference to their value-added scores. The school board monitored whether the District met its goal of "exiting" a sufficient percentage of "its ineffective teachers, defined by the Board Monitoring System as teachers with EVAAS ≤ -2.00." Principals and other administrators were likewise evaluated on whether the District met its goal of exiting ineffective teachers. HISD reportedly progressed toward those goals for the three school years 2011–14, in which 20.3%, 24.4%, and 25.0% of ineffective teachers were "exited," respectively.
P.Ex. 14 (HISD Policy DFBB, Reason No. 35, issued 10/15/12). Although strictly speaking non-renewal only affects teachers with term contracts, the record shows that Reason 35 has been cited as "good cause" for terminating teachers with continuing contracts as well. D.Ex. HH (Dkt. 73–20, under seal); Dkt. 71–1 (Capo Aff.) at ¶¶ 4–5; Dkt. 76–2 (Capo Supp. Aff.); P.Ex. 36; P.Ex. 40.
P.Ex. 29 (Human Resources Update, Nov. 7, 2012, Bates no. 10800).
P.Ex. 30 (2014 District Teaching Effectiveness).
P.Ex. 31 at 4 (Board Monitoring System: Teachers).
P.Exs. 31, 32.
P.Ex. 31, at 2.
While HISD maintains that teachers were not terminated solely on the basis of low value-added scores, the record indicates otherwise. HFT president Zeph Capo, based on his review of HISD documents (including one labeled "Status of Low Three-year EVAAS Teachers"), identified 12 HFT members whose continuing contracts were terminated for low value-added scores between 2012–14. All twelve teachers resigned, retired, or were reassigned without going through termination proceedings. HISD confirms that among these twelve are four teachers (two HFT members) given letters stating that the HISD Board had accepted the Superintendent's recommendation of their termination "based on significant lack of student progress and/or insufficient academic growth as reflected by value-added scores." HISD has also approved severance packages for dozens of teachers in 2012 and 2013 who were proposed for termination or nonrenewal due to "performance and a significant lack of student progress attributable to the educator."
P.Ex. 1 (Capo Aff.) ¶¶ 4–5; Capo. Supp. Aff. (Dkt. 76–2).
Affidavit of Tonnis Hilliard and Ex. 1–14 (Dkt. 81–1–14). See Holden v. Knight, 155 Fed.Appx. 735, 739–740 (5th Cir. 2005) (employee facing "the Scylla of voluntary resignation and the Charybdis of forced termination" has a cognizable due process claim).
D.Ex. HH (Dkt. 73–20) (under seal).
P.Exs. 39, 40.
From this evidence the court concludes that HISD's value-added appraisal system for teachers poses a realistic threat to protected property interests sufficient to withstand summary judgment on their claim for injunctive relief under the Fourteenth Amendment.
2. Procedural due process
Once it is determined that the Due Process Clause applies, the question remains what type of process is due. Cleveland Bd. of Educ. v. Loudermill, 470 U.S. 532, 541, 105 S.Ct. 1487, 84 L.Ed.2d 494 (1985). The core requirement of procedural due process is the opportunity to be heard at a meaningful time and in a meaningful manner. Mathews v. Eldridge, 424 U.S. 319, 333, 96 S.Ct. 893, 47 L.Ed.2d 18 (1976). The Supreme Court has emphasized that procedural due process has two related but distinct goals:
The purpose of this requirement is not only to ensure abstract fair play to the individual. Its purpose, more particularly, is to protect his use and possession of property from arbitrary encroachment—to minimize substantively unfair or mistaken deprivations of property.... For when a person has an opportunity to speak up in his own defense, and when the State must listen to what he has to say, substantively unfair and simply mistaken deprivations of property interests can be prevented.
Fuentes v. Shevin, 407 U.S. 67, 80–81, 92 S.Ct. 1983, 32 L.Ed.2d 556 (1972) (emphasis added). See also Carey v. Piphus, 435 U.S. 247, 262, 98 S.Ct. 1042, 55 L.Ed.2d 252 (1978) ("[The] purpose of procedural due process is to convey to the individual a feeling that the government has dealt with him fairly, as well as to minimize the risk of mistaken deprivations of protected interests .") (emphasis added)). In short, due process is designed to foster government decision-making that is both fair and accurate.
The standards of due process are not wooden absolutes, and must be judged according to the demands of the particular situation. Morrissey v. Brewer, 408 U.S. 471, 481, 92 S.Ct. 2593, 33 L.Ed.2d 484 (1972). In deciding what process is constitutionally due in various contexts, the Supreme Court has stressed that "procedural due process rules are shaped by the risk of error inherent in the truth-finding process...." Carey, 453 U.S. at 259, 101 S.Ct. 2748 (quoting Mathews v. Eldridge, 424 U.S. 319, 344, 96 S.Ct. 893, 47 L.Ed.2d 18 (1976) ). Among the factors considered is "the risk of an erroneous deprivation of such interest through the procedures used, and the probable value, if any, of additional or substitute procedural safeguards." Mathews, 424 U.S. at 335, 96 S.Ct. 893.
In the context of public school teacher terminations, the Fifth Circuit has long required "timely notice and an opportunity to answer charges so as to minimize the likelihood of an erroneous discharge." Findeisen v. North East Independent School Dist., 749 F.2d 234, 239 (5th Cir. 1984). The minimum standards of procedural due process in such cases were specified in Ferguson v. Thomas, 430 F.2d 852 (5th Cir. 1970). These include the right to:
(1) be advised of the cause for his termination in sufficient detail so as to enable him to show any error that may exist;
(2) be advised of the names and testimony of the witnesses against him;
(3) a meaningful opportunity to be heard in his own defense within a reasonable time;
(4) a hearing before a tribunal that possesses some academic expertise and an apparent impartiality toward the charges.
Id. at 856 (as restated in Levitt v. University of Texas at El Paso, 759 F.2d 1224, 1228 (5th Cir. 1985) ). Within these boundaries the State has discretion to adopt the procedures it finds most appropriate. Levitt, 759 F.2d at 1228.
Under Texas law, a teacher proposed for termination has the right to (i) be represented by a representative of her choice; (ii) hear evidence on which the proposal to terminate her contract is based; (iii) cross-examine each adverse witness, (iv) present evidence on her own behalf, as part of a hearing in front of a hearing examiner, and (v) make oral argument to the Board of Trustees before any final ruling on her employment status. TEX. EDUC. CODE §§ 21.255 – 21.256.
Plaintiffs argue that these procedures are constitutionally inadequate for teachers threatened with termination based on low value-added scores, because they are denied access to the computer algorithms and data necessary to verify the accuracy of their scores. As a consequence, they say, HISD flunks the minimum procedural due process standard of providing the reason for termination "in sufficient detail to enable [the teacher] to show any error that may exist." Levitt , 759 F.2d at 1228 ; Ferguson, 430 F.2d at 856. The summary judgment record amply supports their claim.
For present purposes, "accuracy" simply means that the EVAAS score is correctly calculated according to the vendor's own algorithms, using the right data (e.g., correct test scores for the teacher's own students as well as all other students with whom they are compared) and executed by properly performing software that has been suitably tested and maintained according to appropriate quality control measures. Whether the EVAAS score reliably measures what it purports to measure, i.e. teacher effectiveness, is a separate question, dealt with later in this opinion.
HISD does not itself calculate the EVAAS score for any of its teachers. Instead, that task is delegated to its third party vendor, SAS. The scores are generated by complex algorithms, employing "sophisticated software and many layers of calculations." SAS treats these algorithms and software as trade secrets, refusing to divulge them to either HISD or the teachers themselves. HISD has admitted that it does not itself verify or audit the EVAAS scores received from SAS, nor does it engage any contractor to do so. HISD further concedes that any effort by teachers to replicate their own scores, with the limited information available to them, will necessarily fail. This has been confirmed by plaintiffs' expert, who was unable to replicate the scores despite being given far greater access to the underlying computer codes than is available to an individual teacher.
P.Ex. 28 at 6.
P.Ex. 62 at 3 ("EVAAS/Value–Added Frequently Asked Questions").
HISD has consistently denied discovery requests for this information on the grounds that "it requires or seeks the production of proprietary, trade secret information not in the custody, control, or possession of the District." P.Ex. 5. HISD has also denied repeated Public Information Act requests for this information on the same grounds. P.Ex. 1 ¶ 7(Capo Affidavit).
P.Ex. 6 (Defendant's Amended Objections and Answers to Plaintiffs' First Set of Interrogatories No. 22).
Dkt. 86 at 10; P.Ex. 21 (Stevens Depo. 195–96, discussing P.Ex. 61)
P.Ex. 64 (Rothstein Report 58–60).
The EVAAS score might be erroneously calculated for any number of reasons, ranging from data-entry mistakes to glitches in the computer code itself. Algorithms are human creations, and subject to error like any other human endeavor. HISD has acknowledged that mistakes can occur in calculating a teacher's EVAAS score; moreover, even when a mistake is found in a particular teacher's score, it will not be promptly corrected. As HISD candidly explained in response to a frequently asked question, "Why can't my value-added analysis be recalculated?" :
Id.; P.Ex. 62 at 3. See generally Steven M. Bellovin, Matt Blaze, Sandy Clark, & Susan Landau, Lawful Hacking: Using Existing Vulnerabilities for Wiretapping on the Internet, 12 Northwestern Journal of Technology and Intellectual Property 1, at 27–30 (2014) ("The ability to produce error-free code is the Holy Grail of systems development: heavily desired but unattainable.").
P. Ex. 62, at 3.
Once completed, any re-analysis can only occur at the system level. What this means is that if we change information for one teacher, we would have to re-run the analysis for the entire district, which has two effects: one, this would be very costly for the district, as the analysis itself would have to be paid for again; and two, this re-analysis has the potential to change all other teachers' reports.
Id. (emphasis in original).
The remarkable thing about this passage is not simply that cost considerations trump accuracy in teacher evaluations, troubling as that might be. Of greater concern is the house-of-cards fragility of the EVAAS system, where the wrong score of a single teacher could alter the scores of every other teacher in the district. This interconnectivity means that the accuracy of one score hinges upon the accuracy of all. Thus, without access to data supporting all teacher scores, any teacher facing discharge for a low value-added score will necessarily be unable to verify that her own score is error-free.
Value-added teacher evaluation systems such as EVAAS are a relatively recent development, and no Fifth Circuit case has addressed a procedural due process challenge to such a system. Plaintiffs rely most heavily upon Banks v. Federal Aviation Admin., 687 F.2d 92 (5th Cir. 1982), in which two air traffic controllers challenged their dismissal for drug usage after their urine samples were destroyed and unavailable for independent testing on their behalf. The lab tests showed traces of cocaine, the only evidence of drug use in the record. The Fifth Circuit overturned their discharges, agreeing that the controllers were denied due process because their inability to evaluate the critical lab samples rendered the administrative hearings fundamentally unfair:
The laboratory tests here were the only meaningful evidence resulting in the discharges. The accuracy of those tests, including the possibility that the samples were mixed-up, damaged, or even inaccurately tested, was the likely determinant of the entire case. Indeed, challenging the laboratory reports was probably the only way the controllers could succeed in their appeal.
Id. at 94. Mere description of the lab's general testing methods and results was not good enough, the court declared. "We hold that due process required an opportunity by the controllers to test on their own behalf to evaluate the accuracy of the government-sponsored tests." Id. at 96. Plaintiffs assert that Banks is controlling here, and that due process similarly requires an opportunity by teachers to test on their own behalf the accuracy of their HISD–sponsored value-added scores. The court agrees.
HISD's efforts to distinguish Banks fall wide of the mark. It is true that HISD provides some information about EVAAS to teachers—such as an overview of value-added growth as a measure of student learning, a general description of the EVAAS test methods and how they are applied in HISD, and how to read the EVAAS teacher value added report. A teacher is also provided a list of the students linked to that teacher and the percentage of their instruction for which the teacher was responsible. Contrary to HISD's claim, however, this information does not "surpass" the Banks standard. The controllers in Banks were similarly advised about the lab's general testing methods, and even had the opportunity to cross-examine the laboratory director. Yet, the Fifth Circuit held these measures fell short of due process, because without the test samples it was not possible to verify or replicate the controllers' particular test results. Banks , 687 F.2d at 94. The same is true here. HISD's own discovery responses and witnesses concede that an HISD teacher is unable to verify or replicate his EVAAS score based on the limited information provided by HISD.
D.Ex. P.
D.Exs. Q & S.
D.Ex. R.
P.Ex. 6 (Defendant's Answer to Plaintiffs's First Set of Interrogatories No. 9). Defendant's brief contends that teachers also have access to student test scores, but the record citations do not support this assertion. See Dkt. 81 at 3, citing D.Exs. P & R. At most, a teacher "may only view data for students who are currently at their own campus." D.Ex. V at 18; D.Ex. R (Dkt. 73–4 at 5) (teachers can access past testing history of students taught in the grade and subject in the most current testing year). But such limited data would be of little use without full access to all test scores used in the EVAAS calculation, including those of all the teacher's students, past and present, as well as the other students providing the standard of comparison statewide.
P.Ex. 64 (Rothstein Report at 58–60); P.Ex. 6 (Defendant's Answer to Plaintiffs's First Set of Interrogatories No. 9) ("Teachers cannot calculate their own growth measure, or NCE (Normal Curve Equivalent) or their own standard error."); P.Ex. 21 (Stevens Depo. at 195–96) ("[T]hey're not going to come up with the value-added score that would be run by using the much more statistically robust methods that SAS EVAAS uses. So in other words, don't try to calculate your own data. You can try, but you're not going to come up with the same answers.").
HISD argues that Banks did not require access to proprietary information of the independent testing laboratory used by the FAA to perform the analysis. As defendant's brief correctly observes, "the Due Process Clause does not empower Plaintiffs to put SAS out of business" by requiring disclosure of its trade secrets. By the same token, SAS's trade secrets do not empower, much less compel, HISD to violate the constitutional rights of its employees. When a public agency adopts a policy of making high stakes employment decisions based on secret algorithms incompatible with minimum due process, the proper remedy is to overturn the policy, while leaving the trade secrets intact.
Dkt. 73 at 10.
Moreover, in at least one respect the teachers' due process argument is stronger than the controllers in Banks. A drug test is a widely accepted, routine procedure to detect the presence of a physical substance in the body. By contrast, the EVAAS score purports to measure an intangible, job-related trait ("effectiveness") using a recently invented method that by HISD's own admission is the subject of vigorous academic debate. No similar controversy attends the detection of illicit drugs based on urine samples. Given the same urine sample, independent verification of a positive drug test is possible. But independent verification of a negative EVAAS score is impossible. According to the unrebutted testimony of plaintiffs' expert, without access to SAS's proprietary information—the value-added equations, computer source codes, decision rules, and assumptions—EVAAS scores will remain a mysterious "black box," impervious to challenge.
Of course, drug labs need to be monitored and tested to assure quality control, and most states have standards and procedures in place to assure that high stakes testing is done accurately and correctly. See, e.g., California v. Trombetta, 467 U.S. 479, 498–90, 104 S.Ct. 2528, 81 L.Ed.2d 413 (1984) (describing state testing and certification procedures to insure accurate and reliable breathalyzer results). None of this is (yet) true for value-added ratings for teachers.
P.Ex. 64 (Rothstein Report 58–60).
While conceding that a teacher's EVAAS score cannot be independently verified, HISD argues that the Constitution does not require the ability to replicate EVAAS scores "down to the last decimal point." But EVAAS scores are calculated to the second decimal place, so an error as small as one hundredth of a point could spell the difference between a positive or negative EVAAS effectiveness rating, with serious consequences for the affected teacher.
Dkt. 81 at 1.
See, e.g., P.Ex. 37.
P.Ex. 64 at 15 ("There is no meaningful difference between a teacher with a TGI of –2.01 and one with a –1.99, but they receive different categorical ratings.").
Finally, HISD contends that, unlike in Banks where the drug tests "controlled" the outcome of the hearings, the EVAAS score is merely "one factor the hearing officer might or might not consider." This argument casually elides the compelling summary judgment evidence recounted above, such as HISD's aggressive goal of "exiting" 85% of teachers with "ineffective" EVAAS ratings, as well as its amended policy adding low value-added scores as grounds for nonrenewal. Given these official policies endorsed by the HISD Board, it beggars belief that any HISD hearing officer would (or could) freely disregard the very score used by HISD to identify "ineffective" teachers.
Dkt.81 at 2 n.1.
P.Ex. 29
P.Ex. 14.
On this summary judgment record, HISD teachers have no meaningful way to ensure correct calculation of their EVAAS scores, and as a result are unfairly subject to mistaken deprivation of constitutionally protected property interests in their jobs. HISD is not entitled to summary judgment on this procedural due process claim.
3. Substantive due process: rational basis
A successful substantive due process claim requires evidence that the challenged law or practice is not "a rational means of advancing a legitimate governmental purpose." Finch v. Fort Bend Independent School Dist., 333 F.3d 555, 563 (5th Cir. 2003). Plaintiffs contend that EVAAS violates their rights to substantive due process because there is no rational relationship between EVAAS scores and HISD's stated "goal of having an effective teacher in every HISD classroom so that every HISD student is set up for success."
P.Ex. 25.
Rational basis scrutiny presents a very demanding standard for plaintiffs, and a very forgiving standard for policymakers. See Reid v. Rolling Fork Pub. Util. Dist., 854 F.2d 751, 753 (5th Cir. 1988) ("equal protection violation does not arise if there is any basis for a classification or official action that bears a debatably rational relationship to a conceivably legitimate government end" (emphasis added)). Ultimately, whether the necessary rational relationship exists is a question of law. Hidden Oaks Ltd. v. City of Austin, 138 F.3d 1036, 1044 (5th Cir. 1998).
Plaintiffs argue that EVAAS is not a rational evaluation tool "because it is sytematically biased against large categories of teachers on the basis of the type and size of classrooms they teach, is highly volatile, is highly variable on the basis of which models or tests are used, and is highly divergent from other measures of teacher effectiveness." As discussed above, it is also highly secretive and impossible to replicate.
Dkt. 71 at 47.
HISD counters that 42 states and the District of Columbia use some measure of student performance in teacher evaluations, and value-added models have been throughly vetted and endorsed by much of the academic community. HISD also cites a trio of recent cases outside the Fifth Circuit rejecting similar substantive due process challenges to value-added teacher rating schemes.
Dkt. 66 at 12 (citing D.Ex. F at 3).
The Eleventh Circuit considered a value added model based on student scores on the Florida Comprehensive Assessment Test (FCAT VAM) in Cook v. Bennett, 792 F.3d 1294 (11th Cir. 2015). The plaintiffs were teachers who either did not teach tested subjects, or taught students who did not take the FCAT. The Cook plaintiffs conceded that the "FCAT VAM is—or at least a rational policymaker could believe it is—capable of measuring some marginal impact that teachers can have on their own students or on the overall school environment." Id. at 1302. The Eleventh Circuit affirmed the dismissal of plaintiffs' substantive due process and equal protection claims because "[w]hile the FCAT VAM may not be the best method—or may even be a poor one—for achieving [the government's goal of improving instruction], it is still rational to think that the challenged evaluation procedures would advance the government's stated purpose." Id. at 1301.
While plaintiffs vehemently deny that EVAAS passes rational review, at least one of their experts has made a concession similar to that in Cook :
Q. There is pretty good evidence that a teacher's EVAAS score is correlated with that teacher's effectiveness, if we define teacher effectiveness as causal impact on student learning growth, as measured by performance on standardized tests?
A. Yes.
D.Ex.D (Rothstein Dep. at 111–12). Similarly, Dr. Amrein–Beardsley testified that the question of whether school districts should use VAMS in general is "debatable." D.Ex. E (Amrein–Beardsley Dep. at 62–64). The parties disagree about the meaning of "debatable" in the context Amrein–Beardsley was using the term, and the question was not focused on EVAAS specifically.
In Wagner v. Haslam, 112 F.Supp.3d 673 (M.D. Tenn. 2015), the court addressed Tennessee's state-mandated statistical method for evaluating teacher effectiveness, TVAAS. The plaintiffs criticized TVAAS, and value-added models in general (called VAAs in Wagner ), as severely flawed due to failure to control for outside impacts on performance and due to high margins of error leading to low statistical confidence. As in Cook, the Wagner plaintiffs challenged only the constitutionality of using of school-wide TVAAS scores to evaluate the performance of teachers of subjects not tested on state-wide standardized tests, a claim not at issue here. However, the Wagner court explained at length the extremely limited role a court has in applying the rational basis standard:
Like EVAAS, TVAAS is a product of SAS.
[O]ne can conceive of performance metrics that would be truly irrational, such as basing a Tennessee teacher's evaluation on the test scores of students in Arizona, whether the Nashville Sounds baseball team had a winning season that school year, or the State of Tennessee's economy on evaluation day. It is inconceivable that a Tennessee teacher's 'value added' to a student's performance would bear any relationship to those metrics.
Most recently, in Trout v. Knox Cty. Brd. of Educ., 163 F.Supp.3d 492 (E.D. Tenn. 2016), teachers who did not receive annual performance bonuses asserted that use of TVAAS violated their constitutional rights to due process and equal protection. Although not the primary basis for its ruling, the court discussed plaintiffs' claims based on the statistical shortcomings of TVAAS. The court ruled as a matter of law that it is not irrational for the state to rely on a 68% confidence level in TVAAS results, stating "were this case to go to trial, its merits would be subject to a preponderance of the evidence standard. That is, the Court could make its decisions based on a 51% confidence level. Surely then, a 68% confidence level must pass rational basis review." Id. at 503. The court further pointed out that it was undisputed that "TVAAS algorithms have been validated for their accuracy in measuring a teacher's effect on student growth." Id. at 504. The court concluded that while TVAAS may be a "blunt tool," that is all the constitution requires. Id. at 505.
The Trout court held that plaintiffs did not have a protected property interest in bonuses, and that it was not irrational to use system-wide test data to evaluate teachers who do not teach subjects covered by standardized tests, or to use data from only a small number of a teacher's students. 163 F.Supp.3d at 501–505. Plaintiffs argue the Trout court's rational basis review is merely dicta because the court dismissed the substantive due process claims due to the lack of a protected property interest. The court looks to Trout only for its persuasive, not precedential, value.
It is certainly disputed here whether EVAAS algorithms have been validated, and plaintiffs offer up numerous other ways in which EVAAS falls short. Even accepting plaintiffs' criticisms at face value, the loose constitutional standard of rationality allows governments to use blunt tools which may produce only marginal results. HISD's motion for summary judgment on this substantive due process claim is granted.
A concise summary of the "severe flaws" plaintiffs allege in EVAAS can be found in plaintiffs' supplemental response. Dkt. 76 at 9–10.
4. Substantive due process: vagueness
Plaintiffs' claim that EVAAS is unconstitutionally vague also arises from the Fourteenth Amendment's guarantee of substantive due process. The applicable test for unconstitutional vagueness requires plaintiffs to show that EVAAS "fail[s] to provide the kind of notice that will enable ordinary people to understand what conduct it prohibits" or "authorize[s] and even encourage[s] arbitrary and discriminatory enforcement." City of Chicago v.Morales, 527 U.S. 41, 56, 119 S.Ct. 1849, 144 L.Ed.2d 67 (1999). While the doctrine of vagueness has been extended to civil cases, and specifically to the discharge of public employees, it is recognized that "lesser degrees of specificity are required to overcome a vagueness challenge in the civil context than in the criminal context... because the consequences in the criminal context are more severe." San Filippo v. Bongiovanni, 961 F.2d 1125, 1135 (3d Cir. 1992).
In San Filippo, a tenured professor sued after being dismissed by Rutgers University for failure to maintain "standards of sound scholarship and competent teaching." The Third Circuit rejected San Filippo's argument that these regulations were unconstitutionally vague because they do not specify exactly what conduct is prohibited, holding that broad and general regulations are not necessarily vague. Id. at 1137. A vague standard is one that does not specify any standard at all, not one that merely proscribes a wide range of not-specifically-enumerated behaviors. Id. at 1137–38 ; see Ford Motor Co. v. Texas Dep't of Trans., 264 F.3d 493, 507 (5th Cir. 2001) ("An economic regulation is invalidated only if it commands compliance in terms so vague and indefinite as really to be no rule or standard at all").
HISD teachers, like Rutgers professors, can "evaluate their behavior's conformity to the dismissal standard" provided by regulations implementing EVAAS. Id. Teachers are advised that a low EVAAS score can lead to termination, and given general information about the EVAAS system and how it measures teacher effectiveness. While teachers may not be able to verify the accuracy of their EVAAS scores, a suitably definite rule or regulation is not rendered unconstitutionally vague simply because it may be unfair or prone to error. HISD's motion for summary judgment on plaintiffs' vagueness claim is granted.
5. Equal protection
To state a claim under the Equal Protection Clause, a § 1983 plaintiff must allege either that (a) a state actor intentionally discriminated against [him] because of membership in a protected class, or (b) he has been intentionally treated differently from others similarly situated and that there is no rational basis for the difference in treatment. Gibson v. Texas Dep't of Ins., 700 F.3d 227, 238 (5th Cir. 2012). Plaintiffs' claim is of the latter type. They allege that HISD wrongly classifies teachers according to their EVAAS scores, and then subjects them to different treatment based on the classification. Plaintiffs further contend that HISD has a policy of requiring supervisors to bring instructional performance ratings into alignment with EVAAS scores. This practice of alignment allegedly subverts the independence of the instructional practice score. This appears to be a novel claim, and the court has found no authority addressing an equal protection claim in an analogous context.
See Dkt. 71 at 28–30; P.Exs. 49, 50. HISD denies it has such a policy. Dkt. 73 at 9 n.3. The record establishes a genuine issue of material fact on this point.
See D.Ex. I (Amrein–Beardsley rebuttal report at 9–14).
Equal protection claims were raised in Cook, 792 F.3d at 1301, and Wagner, 112 F.Supp.3d at 690, but the theories advanced were different.
As with substantive due process, an equal protection claim is subject to rational basis review. This requires that the classification system used to justify subjecting some teachers to different treatment than others must be rationally related to a legitimate governmental objective. City of Cleburne v. Cleburne Living Center, 473 U.S. 432, 439, 105 S.Ct. 3249, 87 L.Ed.2d 313 (1985) ; Mahone v. Addicks Util. Dist. of Harris Cty, 836 F.2d 921, 933 (5th Cir. 1988) (an equal protection challenge focuses on three elements: (i) the classification, (ii) the purpose the classification is designed to serve, and (iii) the fit between the classification and the purpose.). As plaintiffs point out, the court must review "the classification system itself."
Dkt. 71 at 67.
--------
Plaintiffs' factual allegations simply do not fit the mold of an equal protection claim. The court fails to see how the practice of aligning instructional practice ratings with EVAAS scores is a classification system. Even if it were, the court has already determined that the EVAAS system satisfies rational basis review. Thus, this claim necessarily fails along with the substantive due process claim discussed above. HISD's motion for summary judgment on plaintiffs' equal protection claim is granted.
Conclusion
For these reasons, HISD's motion for summary judgment (Dkt. 65) is denied with respect to the procedural due process claim, but granted on all other claims.