Summary
weighing expert use of differing techniques with varying degrees of standardized error in retracing a bullet's trajectory
Summary of this case from Krause v. Cnty. of MohaveOpinion
CASE NO. 3:17-CR-00226-JO
06-20-2018
UNITED STATES OF AMERICA, Plaintiff, v. W. JOSEPH ASTARITA, Defendant.
AMENDED OPINION AND ORDER :
This matter is before the court on defendant's motion in limine to exclude from trial the reports, testimony and exhibits of government experts Frank Piazza, Victoria Dickerson, Michael Haag, Kevin Turpen, and Toby Terpstra [# 65] Defendant challenges the reliability of this evidence and contends it should be excluded under Fed. R. Evid. 702 and case law following Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579 (1993), which as of June 1, 2018, has been cited 2149 times by the Supreme Court and in published opinions of the federal Courts of Appeals. In addition, defendant asserts that presenting this evidence to a jury would pose a danger of unfair prejudice that substantially outweighs its probative value so that it should be excluded under Fed. R. Evid. 403. For the following reasons, the motion is granted in part and denied in part
BACKGROUND
On the afternoon of January 26, 2016, members of the FBI Hostage Rescue Team ("HRT") and the Oregon State Police Special Weapons and Tactics Team ("OSP") stopped two vehicles carrying the leaders of the armed takeover of Malheur National Wildlife Refuge. One of the vehicles, a white Dodge pickup driven by Robert Lavoy Finicum sped away from the scene of the stop and headed north on Highway 395. OSP and HRT set up a roadblock on Highway 395 to stop Finicum's truck and apprehend its occupants, including Finicum, Ryan Bundy and Shawna Cox.
As Finicum approached the roadblock at speeds up to 70 mph, an OSP officer ("OSP Operator 1") fired three rounds, hitting Finicum's truck in the hood, grill, and driver's side mirror. Finicum drove off the road to the left of the roadblock and plowed into a deep snow bank, narrowly missing Special Agent JN. Finicum opened the driver's side door and emerged from the truck with his hands spread. At that approximate time, two additional shots were fired ("shots 4 and 5"), one of which perforated the roof of Finicum's truck and shattered the left rear window. The other did not hit anything of significance for the present purposes.
Finicum confronted two OSP operators, disregarding repeated commands to stop and get down on the ground. Two OSP officers shot Finicum a total of three times as he appeared to reach for a weapon in a pocket of his coat, where officers subsequently found a loaded handgun. Finicum died at the scene. The shooting occurred at approximately 4:30 p.m.
These events were recorded by aerial video taken from two FBI fixed-wing aircraft circling approximately two miles from the scene ("FBI video"). Contemporaneously, Shawna Cox filmed the events from the back seat of Finicum's truck with a SLR camera that recorded both audio and video ("Cox video").
Approximately nine hours later, Deschutes County Deputy Sheriff Kevin Turpen, Oregon State Police Forensic Scientist Victoria Dickers on and others arrived to measure and reconstruct the shooting scene. They used a total station device to measure the location of Finicum's truck and the other evidence at the scene. Turpen used these measurements to create a computer assisted diagram of the scene. Examination of Finicum's truck revealed the bullet hole from shot 4 or 5 in the roof ("Impact W"), in addition to the three impacts from shots fired by OSP Operator 1 as the truck approached the roadblock. Dickerson estimated the trajectory of each round, including the round that caused Impact W. Turpen added Dickerson's trajectory estimates to his diagram to indicate the likely location from which each shot was fired. He did not make his own trajectory estimates.
In the course of the investigation, all the shots fired by OSP operators were accounted for; the three that hit the front of Finicum's truck as it approached the roadblock and the three that hit Finicum after he emerged from his track. No one admitted firing the shot that hit the roof of Finicum's truck or the one that missed. The allegations against defendant in this case are that he fired shots 4 and 5 and then falsely denied doing so, not that he shot Finicum.
The government hired Frank Piazza as an audio/video expert to examine the FBI videos and the Cox video, synchronize the recordings and create a side-by-side playback. The government retained Michael Haag to examine Finicum's track and determine whether a reliable trajectory estimate could be made for the bullet that caused Impact W.
The government retained Toby Terpstra to take the data provided by Piazza, Haag, and others to reconstruct the scene into a 3-D animation model. Terpstra took 3-D measurements of Finicum's truck and the shooting scene. He used twelve "camera matches" from photographs or still frames from videos to position the vehicles in his model and three camera matches to position the individuals. He then added Haag's trajectory data for the shot that caused Impact W. Terpstra's 3-D animation model places defendant within the area shown as the likely origin of the shot that caused Impact W.
During a four-and-a-half day Daubert hearing, the court heard testimony from Piazza, Dickerson, Turpen, Haag, Terpstra and Professor Jeffrey Smith. The defense put on experts Bruce Koenig, Andrew Bray, Matthew Noedel, Eugene Liscio and Clifford Mugnier, to challenge the work of the plaintiff's experts, but not to create reconstructions of their own.
LEGAL STANDARDS
A person qualified as an expert by knowledge, skill, experience, training or education may provide expert testimony if his or her specialized knowledge will help the trier of fact to understand the evidence or to determine an issue of fact. Fed. R. Evid. 702. The expert testimony must be both relevant and reliable. Daubert, 509 U.S. at 589. The proponent of the expert testimony bears the burden of establishing its admissibility by a preponderance of the evidence. Daubert, 509 U.S. at 592 n.10.
The Ninth Circuit Court of Appeals recently summed up Daubert and its progeny in Murray v. Southern Route Maritime SA, 870 F.3d 915, 922-23 (9th Cir. 2017):
The Court in Daubert . . . constructed a flexible test examining the "reliability" and "fit" of the offered expert testimony. See id. at 589-92, 113 S.Ct. at 2786.
The question of reliability probes "whether the reasoning or methodology underlying the testimony is scientifically valid." Id. at 592-93, 113 S.Ct. 2786. To give shape to the inquiry, the court identified four factors that may bear on the analysis: (1) whether the theory can be and has been tested, (2) whether the theory has been peer reviewed and published, (3) what the theory's known or potential error rate is, and (4) whether the theory enjoys general acceptance in the applicable scientific community. See id. at 593-94, 113 S.Ct. 2786. But the Court was quick to emphasize that the factors are not "a definitive checklist or test" and that the reliability analysis remains a malleable one tied to the facts of each case. Id. at 591, 593, 113 S.Ct. 2786. Later cases have reiterated that the Daubert factors are exemplary, not constraining. Kumho Tire Co. v. Carmichael, 526 U.S. 137, 150, 119 S.Ct. 1167, 143 L.Ed.2d 238 (1999); id. at 159, 119 S.Ct 1167 (Scalia, J., concurring) ("[T]he Daubert factors are not holy writ . . .").870 F.3d at 922-23.
It is important to remember that the factors are not "equally applicable (or applicable at all) in every case." Daubert v. Merrell Dow Pharm., Inc., 43 F.3d 1311, 13176 (9th Cir. 1995). Applicability "depend[s] on the nature of the issue, the expert's particular expertise, and the subject of his testimony," Kumho Tire Co., 526 U.S. at 150, 119 S.Ct. 1167 (citation omitted). A district court may permissibly choose not to examine factors that are not "reasonable measures of reliability in a particular case." Id. at 153, 119 S.Ct. 1167.
Because of the fluid and contextual nature of the inquiry, district courts are vested with "broad latitude" to "decid[e] how to test an expert's reliability" and "whether or not [an] expert's relevant testimony is reliable." Id. at 152-53, 119 S.Ct.1167. District judges play an active and important role as gatekeepers examining the full picture of the experts' methodology and preventing shoddy expert testimony and junk science from reaching the jury. See Daubert, 509 U.S. at 595-97, 113 S.Ct. 2786.
This gatekeeping function applies to any expert testimony, not just scientific opinions. Primiano v. Cook, 598 F.3d 558, 564-65 (9th Cir. 2010). The district court's role is limited to ensuring the soundness of the expert's methods and does not include finding facts about the correctness of the expert's conclusions or the credibility of the witness. Primiano, 598 F.3d at 564-65; Daubert, 43 F.3d at 1318. Shaky but admissible evidence is to be attacked at trial by cross examination, impeachment, opposing expert testimony, contrary evidence, and proper application of the burden of proof, not by pretrial exclusion. Primiano, 598 F.3d at 564.
ANALYSIS
I. Frank Piazza
The government seeks to elicit the expert opinion of Frank Piazza, an audio engineer who owns and operates a forensic audio-video company. The government summarized his work as follows.
Mr. Piazza identified where in the Shawna Cox video the fourth and fifth shots rang out. He relied on more than just visual cues in making that determination. He listened critically to the audio track on the Cox video using professional grade headphones in a controlled environment. He listened at different playback speeds, and coupled what he heard with spectrographic analysis and visual cues from the video - shrapnel or debris coming through the headliner of Finicum's pickup truck, and the left rear window shattering - to locate the gunshots,
Mr. Piazza synchronized the Cox video and the FBI video. He converted both to a format suitable for editing, set both to run at 30 frames per second, and used professional editing software to narrow the time frame in both videos to focus on pertinent events. He looked for and matched fixed points visible in both videos, including Finicum's truck passing a post on the left side of the road, and Finicum opening the driver's door and emerging from the truck while raising his hands, He produced a single video that plays the synchronized videos together, with the Cox video playing in the lower left comer of the FBI surveillance video.
He ... noted the elapsed time counter for the matching frames in each video - to 1/100th of a second. Professor Smith reviewed Mr. Piazza's work and confirmed that his synchronization was well within an acceptable range of error. He also confirmed that Mr. Piazza's methodology in locating shots four and five on the Cox video, and in synchronizing the Cox video with the FBI video, were scientifically sound. Mr. Piazza initially estimated an error rate for the synchronized video of ± 3 frames. In preparing for the hearing, however, he discovered a glitch in the Cox video that occurred approximately 20 seconds after the last rounds were fired. At that point, the two videos were
off by approximately 10 frames (1/3 of one second). When shots four and five were fired, the two videos were properly synched.
Mr. Piazza enhanced the FBI surveillance video to aid in tracking the movements of the operators at the shooting scene. He adjusted the lighting, contrast, and color. He manually stabilized the video, sharpened the video (being mindful of the adverse effect caused by over-sharpening), slowed its playback, and placed colored circles around the various operators. The enhanced videos are reliable, relevant, and helpful to the jury, because they help track the movements of various operators at the scene, Notably, neither the synchronized video nor the video Toby Terpstra used to create his camera-matched 3-D scene re-creation were enhanced.
The defense contends Piazza erroneously synchronized the FBI and Cox videos and altered the edges of objects in the video he sharpened. The defense summarized their objections as follows.
Deputy Turpen's diagrams and Mr. Terpstra's model purport to position people and other objects at the precise moment at which the shot that struck the Finicum truck was fired. Both Turpen and Terpstra admitted that their analyses depend on accurately selecting the precise frame from the overhead FBI video that captures that moment. The government hired Frank Piazza to identify that frame by analyzing the "Cox video" and synchronizing that video with the FBI video . . . He never submitted a meaningful report documenting his methods and failed to maintain notes of his work or screen captures documenting his settings on any of the (at least three) software programs he used.
In its brief, the government promised that Mr. Piazza would testify that his sync "may be off by a frame at most, and likely less" (emphasis in original). But Mr. Piazza admitted on direct examination that his sync might actually be off by as many as ten frames in either direction. On cross-examination, that error rate expanded to eleven frames in either direction (a 23-frame range). And another government expert, Professor Jeff Smith, was only willing to label as "reliable" a synchronization range of 34 frames. This is critical because the positions of all of the individuals in those frames—not just Special Agent Astarita—are essential to understanding who may have been in a position to shoot. And during the 34-frame loop that the government played during the hearing, several people were seen materially changing position, underscoring the need to identify the precise frame depicting those
individuals' positions when the key shot was fired. But neither Mr. Piazza, Dr. Smith, nor any other expert was able to do so.
RULING
Piazza is qualified by training and experience to identify sounds on audio recordings, synchronize video recordings depicting the same content from different vantage points, and to apply filters and other software tools to enhance video images. He is qualified to provide expert testimony with respect to those matters.
With respect to the synchronization error of one-third of a second, I suspect asking whether a person can move an appreciable distance in that brief instant is akin to an intellectual speculation from the old adage "how many angels can dance on the head of a needle." On the other hand I take judicial notice under Fed. R. Evid. 201, that the world record for the 100 yard dash is only 9.58 seconds, demonstrating a person can travel quite a bit in the briefest of times, at least when unencumbered by police tactical gear. Because of a potential error range, the prosecution must show that the operators depicted in the synchronized video did not alter their positions materially during the pertinent time.
The corrected Piazza synchronized video demonstrates at 34:46 the first three shots hit Finicum's truck, at 34:54 shots 4 and 5 can be heard, and at 35:06 the three shots that killed Finicum occurred. The locations of individuals at the time of shots 4 and 5 is to be determined by the jury.
Piazza's use of a narrow-band spectrograph does not warrant exclusion. Piazza found the gunshots readily identifiable on the spectrogram by high sound pressure that appeared "like a brick wall" coupled with the simultaneous sound of gunshots and visual cues. There were no N waves, echo, or reverberation on the Cox audio track. In sum the defendant's objections to Piazza's report and testimony go to the weight of the evidence. The jury will make the ultimate determination after hearing all the admissible evidence in the case.
II. Victoria Dickerson
The government seeks to elicit the expert opinion of Victoria Dickerson, a senior forensic scientist employed by the Oregon State Police. The government summarized her work as follows,
[W]hen Ms. Dickerson placed a ballistic rod in Impact W, she found that it lacked stability due to the size of the hole. Accordingly, she used a centering cone to stabilize the rod. The rod itself passed through the hole and followed the bullet's path as it traveled through the headliner material and into the cab of Finicum's truck. Importantly, the rod lined up perfectly with the lead-in mark and pinch point of Impact W, resulting in an accurate horizontal azimuth angle that was within three degrees of the angle measured by Michael Haag, Ms. Dickerson's vertical angle measurement differed from Mr. Haag's, but that was because she measured the angle of the path the bullet took as it passed through the headliner, rather than the angle at which it was traveling when it first made contact with the truck's roof. She acknowledged the bullet likely deflected once it struck the roof. ...She found no reason to deviate from the industry-standard margin of error of ± 5° when measuring Impact W." ECF [102]
The horizontal azimuth angle of Impact W is the critical measurement in this case, since all of the operators were standing on the ground when shots 4 and 5 were taken.
The defense contends Dickerson's trajectory measurement is unreliable because she used a centering cone to assist in the placement of the trajectory rod and that she measured the deflected path of the bullet through the roof of Finicum's truck and not the path the bullet took from the muzzle of the gun to the truck. The defense summarized their objections as follows
"[T]he evidence established that Ms. Dickerson's trajectory measurement (which was incorporated into Deputy Turpen's diagrams) is unreliable for two main reasons. First, as Ms.
Dickerson and Mr. Noedel explained, by placing a "centering cone" in Defect W, Ms. Dickerson necessarily assumed that the bullet passed through the center of the defect, even though Defect W was nearly twice the width of the .223 caliber round that the government alleges created that defect. The government did not present evidence that the method satisfies any of the Daubert factors: there is no evidence that the method has been tested or subjected to peer review, that it has a known error rate, that there are published standards governing its application, or that it is generally accepted in the field. Indeed, the Oregon State Police do not have a standard operating procedure for the use of centering cone trajectory analysis.
Second, as both Ms. Dickerson and government expert Michael Haag acknowledged, Ms. Dickerson's method—passing a trajectory rod through both Defect W and one of three holes under the roof's insulation and liner—measured (albeit imprecisely) the bullet's trajectory after it struck the roof and deflected, rather than before it struck the roof. The importance of that distinction is apparent in Ms. Dickerson's estimation of the vertical angle of the shot (depicted in yellow, below), which passes far over the heads of all of the potential shooters:
Image materials not available for display.
Ms. Dickerson brushed aside the significance of that deflection error by baldly asserting—with no forensic or other supporting evidence whatsoever—that although the round clearly deflected vertically, it did not deflect horizontally at all. But government expert Michael Haag's book makes clear that there is no basis for such an assumption:
Image materials not available for display.
In short, as Matt Noedel explained and Mr. Haag acknowledged, Ms. Dickerson's estimate of post-impact trajectory (which is itself flawed because of the "centering" process) is not a reliable proxy for the bullet's pre-impact path, which is the only path that matters." [ECF #103]
RULING
Ms. Dickerson is a senior forensic scientist who has been employed by the Oregon State Police (OSP) for over thirteen years. She is the technical leader for the entire forensic division. During her time with the OSP, she has conducted over 100 crime scene investigations, received ongoing training, written and published numerous instructions and articles on a variety of crime scene investigation topics, held memberships and leadership positions in professional forensic organizations and appeared as an expert witness in nineteen cases. She is a well-qualified expert in the field of bullet trajectory and bullet path reconstruction.
In her report, Dickerson stated that the truck was embedded in the snow with the passenger side angled downward toward the highway at an approximate 14 to 15 degree angle. She observed that three of the four impacts were made by bullets traveling from the front of the vehicle to the rear. The fourth, Impact W, was made by a bullet traveling from the rear of the vehicle to the front. During the Daubert hearing, Dickerson testified that the size of Impact W measured approximately 10 mm. in width and 20 mm. in length and described Impact W as having a longer pinch point than sometimes can be seen but not out of the ordinary. She noted that when she inserted the trajectory rod through Impact W, the rod lined up with both the lead-in mark and the pinch point. Using the two-point method, Dickerson estimated the trajectory of the bullet that entered Finicum's vehicle at Impact W as traveling at a downward angle of 20° and from left to right at an angle of 35° left of midline. She said her estimates had a certainty of ±5°. At the Daubert hearing, Dickerson acknowledged that her measurements reflected the trajectory of the deflected bullet after it perforated the roof of the vehicle.
Presumably, all two-hole trajectory measurements record a bullet's deflected path if the first impact is through a substance solid enough to affect the path of a bullet.
Defense expert Noedel asserts that Dickerson's use of the centering cone made her measurements unreliable because a centering cone assumes the bullet passes through the center of the opening created by the bullet and the trajectory rod aligns through a single position making the trajectory rod swivel and rotate because the ends of the rod are not anchored to anything. However, he stated that the use of centering cones is not as much of a method as an assist to a multi-point assessment. Indeed, when Dickerson used a centering cone to stabilize her trajectory rod, the rod passed through two distinct points W and W2. Dickerson testified she used the two-point method.
The parties cite numerous articles and books analyzing bullet trajectories. The references report actual test data using various methods to measure bullet trajectory, including the two-point method used by Dickerson. The bullet trajectory literature reveals that the two-point method can be tested, has been tested, and has been subjected to peer review and publication.
E.g., Dean H. Garrison, Jr. "Practical Shooting Scene Investigation" Universal Publishers/Upublish.com 2003; Erwin J.A.T. Mattijssen, & Wim Kerkhoff, Bullet trajectory reconstruction - Methods, accuracy and precision, 262 Forensic Science Int'l 204 (2016); Michael G.Haag & Lucien C. Haag, Shooting Incident Reconstruction (Academic Press 2008) (2006).
Noedel asserts that the two defects used by Dickerson are too close together providing an unreliable angle estimate. Neither party provided a measurement of the distance between Impact W in the roof of Finicum's truck and the hole in the cloth headliner where the bullet exited the roof of the truck and entered the cab. But, Dickerson testified that in order to measure the vertical angle using an inclinometer, she needed to attach an extension to the trajectory rod. This indicates that the two points were not exceedingly close.
Dickerson applied an error rate of ±5° and saw no reason to apply a different error rate because she used the centering cone. According to government expert Michael Haag, the ±5° error rate has served as an "industry standard" for many years based on empirical testing and collective experience. Even defense expert Noedel noted that the error rate of ±5° was probably appropriate for the two-point method, although he would apply a larger range of error given the use of the centering cone.
The two-point method received broad acceptance among bullet trajectory experts. As Noedel stated in his Declaration, "The most common and widely accepted method for bullet path reconstruction is ... the two-point trajectory rod method." Declaration of Matthew Noedel p. 2 During the hearing he testified that the two-point method is clearly accepted and used by everybody who is practicing bullet path analysis.
The parties agree that Dickerson's method would measure the bullet's deflected path from the time the bullet struck Finicum's truck until it exited through the headliner. The parties do not agree as to the direction the bullet deflected or if it deflected at all after striking Finicum's truck. That issue is a matter for the jury. It goes to the weight of the evidence and not the reliability of the method Dickerson used. III. Michael Haag
As noted, the prosecution seeks to elicit the expert opinion of Michael Haag, a forensic scientist specializing in the reconstruction of shooting scenes. The government summarized his work as follows.
Michael Haag is a published author and recognized authority in the field of shooting incident reconstruction. He is extremely well qualified in the fields of ballistics and trajectory analysis, and used reliable and relevant techniques to conduct a valid trajectory analysis in this case. He has trained ballistics and trajectory analysts nationally and internationally for many years.
Mr. Haag noted that each shooting scene is unique; no single measurement technique works in all situations. An experienced ballistics and trajectory expert will select an appropriate method of measurement based on the nature of the impact, and the examiner's training and experience. Every trajectory measurement method - even the "two-point" method described by Matthew Noedel - involves some degree of subjectivity.
Mr. Haag found nothing particularly unusual or atypical about Impact W. Many of its characteristics are commonly seen in low-angle impacts. In fact, he was able to reproduce many of Impact W's characteristics in testing he did in connection with this case (Ex. DH 29-32). A German police ballistics laboratory was able to do so as well (Ex. DH 34-46).
Mr. Haag used the rocker point method to measure Impact W, as it was the best and most reliable measurement method given the circumstances. He first learned of that technique in 2004, and has been training others in its use since 2006. He listed over 20 law enforcement agencies from Los Angeles to Boston that currently use the rocker point method to measure low-angle impacts like Impact W, and find it reliable. He demonstrated the technique in court, using low-angle bullet holes in sheet metal.
Mr. Haag conducted empirical studies using low-angle impacts in sheet metal and the same caliber ammunition involved in this case. He measured the trajectories of each of those impacts using the rocker point method, and documented his measurements with a 3-D laser scanner. Each measurement was within five degrees of the actual angle of origin (Ex. DH 21-26, 33). In addition, he pointed to peer-reviewed literature and ballistics textbooks that describe the rocker point method using different names (such as the lead-in method, or placing the ballistic rod against the "shoulder" at the leading edge of the perforation). While Mr. Haag coined the term "rocker point method," the technique itself is recognized and used by other experts in the field.
Mr. Haag described the differences between how he measured Impact W, and how Ms. Dickerson measured it. She measured the angle of the bullet's path after it penetrated the truck's roof; he measured its angle when it first contacted the roof. Thus, their vertical angle measurements differed. However, for the horizontal azimuth angle, which is the critical measurement for Impact W, they both measured along the lead-in mark and the pinch point, and those measurements were within three degrees of each other. Mr. Haag testified to his measurements to a reasonable degree of ballistics certainty, and found no reason to depart from the industry-standard error range of ± 5°.
The defendant contends that Haag's rocker point method is an unreliable methodology for estimating trajectory and that Haag's claimed range of error has no basis. The defense summarized their objections as follows.
Mr. Haag measured the bullet's pre-deflection trajectory by using his "rocker point" method. But Mr. Haag admitted—and Mr. Noedel confirmed—that unlike the more reliable "two-point" method, his rocker point method relies heavily on the "subjective feel" of the examiner. Ms. Dickerson testified that the Oregon State Police do not have a standard operating procedure describing the method, nor was she even aware of it until Mr. Haag trained her in October 2016. Mr. Noedel confirmed that the method is not described in any detail in the literature or generally accepted in the community, and Mr. Haag admitted that it is not even described in his own book or training materials. According to Mr. Haag, the method—which in this case involved the use of duct tape and a clamp—is a "minor technique" amounting to only a "small part" of the classes he teaches.
Mr, Noedel explained that the method's reliability is further undermined by its extreme sensitivity to small errors. Because (as Mr. Haag confirmed) only approximately 1 centimeter of the trajectory rod's tip is used to take the measurement, even miniscule mistakes can result in dramatic errors: Mr. Noedel explained that a one-millimeter error (roughly the diameter of the point of a pencil) can throw off the measured trajectory angle by 14 degrees.
* * *
Ms. Dickerson assumed that a ± 5° margin of error applied to her centering cone measurements. But the record is utterly devoid of any support for that assumption, and Ms. Dickerson herself did not cite any foundational basis for relying on this error rate. None of the government's experts offered any studies, tests,
or other evidence of the error rate that should properly be applied to centering cone measurements.
Similarly, there are no reliable, statistically valid data supporting the application of a ± 5° margin of error to Mr. Haag's rocker point method. In its brief, the government relied primarily on a 2008 paper authored by Mr. Haag ("2008 Study") to support that error rate. But as defense expert Dr. Andrew Bray testified, the 2008 Study was plagued by fatal structural and statistical errors: (1) lack of "blinding" (i.e., students knew the answers before taking their measurements); (2) conditions were not held constant (i.e., no way of knowing which or how many of the ~ 450 shots were measured using the rocker point method versus some other method, were fired into sheet metal versus some other medium, were fired at shallow versus steep angles, etc.); (3) contrary to NIST recommendations, "outlier" values were simply discarded from the dataset; (4) operations were mis-ordered; (5) the study failed to take a weighted average of results across all of the various scenarios; and (6) Mr. Haag improperly assumed that his results followed a bell-shaped, or "Gaussian," distribution.
Although Mr. Haag testified that he agreed with many of Dr. Bray's criticisms of his study and the way in which he handled his data, Mr, Haag stubbornly stood by his ± 5° margin of error assumption for two reasons, neither of which was supported by any reliable, statistically valid data. First, Mr. Haag pointed out that he and others used that standard even before he published his 2008 paper. But as Mr. Noedel explained, the evolution of that "standard"—derived from little more than folklore derived from people connecting two points with string and pencils in walls—only highlights how necessary the 2008 Study was in introducing statistical rigor to a field that did not have it before. Mr. Haag wrote his paper for a reason, and without it, the empirical support for the standard collapses. Second, Mr. Haag contended that his ± 5° margin of error is supported by the 19 shots he fired into sheet metal in preparation for the hearing, each of which he apparently measured within 5° of the true value. But the Court should be suspicious of the results of a test performed by the leading proponent of the method, specifically for purposes of the litigation at hand, without any blinding. Moreover, despite the cautionary note in Mr. Haag's 2008 Study that "[g]ood scientifically defensible methods often require an in-depth statistical analysis," Mr, Haag failed to offer any statistical analysis of those 19 shots. In order to calculate a reliable margin of error, Mr. Haag would have had to consider the distribution of the data points, calculate an appropriate standard deviation, and apply other applicable statistical criteria. Mr. Haag did none of this. Mr. Haag's 2008 Study also notes that "a larger cone [than ± 5°] may be necessary
for shallow angle shots" like those at issue here. But there are no reliable studies establishing what the size of that cone should be. (Footnotes omitted)
RULING
Haag is a highly qualified ballistics and trajectory analyst who has trained law enforcement personnel on the subject for many years and has published papers based on studies of the subject. He is well qualified to opine on the procedures and techniques used for making trajectory estimates. Notably, he has no qualification in statistical analysis to assess the degree of statistical certainty or probability associated with measurement procedures.
Haag was a pioneer in what he calls the "rocker point" method that he used to estimate the trajectory of the bullet that caused Impact W. He opined that the trajectory of that shot had a downward angle of nine degrees relative to the roof of Finicum's truck. He said the shot had a horizontal azimuth angle of 122° relative to a straight line running lengthwise down the center of Finicum's track. The general direction of flight of the shot that caused Impact W was from the right rear toward the left front. Haag opined that his trajectory measurements were reliable to ±5°.
Dickerson estimated the horizontal azimuth of 35° from the midline of the vehicle. Haag's estimate, if done from the midline would be 32°.
Haag demonstrated his rocker point technique in court with Government's Exhibit 37 using shallow angle bullet holes in sheet metal on a box and inserting a rod to demonstrate trajectory. He has instructed numerous law enforcement agencies in using the technique for shallow angle bullet impacts since approximately 2006. Haag identified standards controlling how the rocker point method is to be applied. There must be a low angle impact with a lead in mark of sufficient length that a centimeter or more of the trajectory rod settles into and stabilizes in the primary bullet defect when the evaluator applies pressure. Although these standards may appear ill-defined to a lay person, I am satisfied that an experienced ballistics and trajectory specialist with proper training would be able to apply the procedure and that Haag faithfully did so in making his trajectory estimate for the shot that caused Impact W.
The underlying methodology can be repeated and tested for accuracy and precision. Haag performed empirical testing of the method by firing shallow angle shots 14 times into sheet metal and 5 times into a vehicle. He estimated the trajectory of these shots using the rocker point method and his results were accurate within ±5°. Haag's empirical testing method was not ideal, however, because of the small sample size and because he is the proponent of the method and failed to blind himself from the true trajectory when testing its accuracy.
Defense experts Andrew Bray and Matthew Noedel described how they would design studies to measure the accuracy of the rocker point method. They admit such testing is feasible, but point out that the record does not show that it has been accomplished. They claim Haag's 2008 article titled "Accuracy and Precision of Trajectory Measurements" failed to properly assess the accuracy of the rocker point method. I agree with Bray and Noedel that the study does not provide a valid statistical assessment of the accuracy of the rocker point method. Nevertheless, Bray's impressive statistical evaluation of Haag's study does not overcome the collective knowledge derived from use of the technique in the field by law enforcement personnel for at least a decade. Of note, the many law enforcement groups Haag has trained who report regularly using the rocker point method for determining trajectory for low angle shots include the Toronto Police Department, Washington State Patrol, San Diego Police Department, Sacramento County Crime Lab, Salt Lake City Police Department, Texas Rangers, Mesa Police Department, Johnson County Kansas, Boston Police Department, Contra Costa County, New Mexico State Crime Scene Team, Philadelphia Police Department, Montana State Crime Lab, West Valley Utah Police Department, Phoenix Police Department, Minnesota Bureau of Criminal Apprehension, Richland County South Carolina Sheriff's Office, and others.
The record also demonstrates further degree of acceptance of the rocker point method in the ballistics and trajectory measurement field. In a 2016 article titled "Bullet trajectory reconstruction - Methods, accuracy and precision," Mattijssejn and Kerkoff attempted to measure the relative accuracy and precision of three methods for estimating bullet trajectory. They found the "lead in" method, which, like Haag's rocker point method, involved estimating trajectory by aligning a probe with the lead in portion of the bullet defect, was more accurate than other methods tested for low angle shots.
Haag also asserts that his trajectory measurements are within ±5° based on a longstanding industry standard for all trajectory measures regardless of method. This basis is somewhat suspect because the rocker point method is a relatively new technique. Haag began teaching the rocker point method in approximately 2006 and Dickerson, a professional forensic scientist, had not heard of it until 2016. Nevertheless Haag has taught this method to hundreds of law enforcement trainees and presumably it has been used extensively in the field.
On the record before the court, it does not appear that the rocker point method has been subjected to significant publication, but no single factor is essential under Daubert. Defendant objects to the subjectivity of the operator's "feel" involved in the rocker point method. Here, the "feel" is palpable when placing the ballistic rod so that it settles into the lead in part of a low angle bullet impact, such as those provided in Government's Exhibit 37. As noted, not every or any of the Daubert factors are essential if I am satisfied that the methodology passes muster, though imperfect. Haag may testify at trial and give his opinion on trajectory, subject to rigorous cross-examination by the defense and contrary expert testimony.
IV. Kevin Turpen
The government seeks to elicit the testimony of Kevin Turpen as both a fact witness and an expert witness. He measured the scene using a total station measuring device. With the assistance of diagramming software, he used that data and the measurements provided by Dickerson and others, to create a scaled diagram of the shooting scene (Ex. DH 14). Although he lacks an advanced degree and has not published any scientific articles, he has taken courses in crash investigation, crash reconstruction, and been trained on the use of diagramming software for crash and crime scene documentation. He has participated in over 75 criminal crash scene investigations.
RULING
Turpen is qualified by specialized training and experience to offer lay and expert opinions on his crime scene reconstruction. He produced diagrams using the methodology commonly used by crash scene investigators and reconstruction experts. With respect to shot trajectory, his opinion and diagrams are based on Dickerson's trajectory estimate. If her trajectory estimate is off, his diagrams will be off as well. Any weaknesses in his testimony or diagrams can be addressed through cross-examination and other expert testimony.
V. Toby Terpstra
The government seeks to elicit the expert opinion of Toby Terpstra, a 3-D computer animation and reconstruction expert. The government summarized his work as follows.
Toby Terpstra and his company, Kineticorp LLC, produced a 3-D recreation of the shooting scene immediately before shots four and five were fired. He analyzed surveillance video from two FBI planes, a synchronization of the Shawna Cox
and FBI videos, video footage taken near the scene by a private citizen, and still images taken near the time of the shooting. In addition, he documented Finicum's truck and the shooting scene with 3-D laser scanners and a total station measuring device. He and two other Kineticorp employees used computer software to process over 192,000,000 3-D data points to build a virtual model of the scene. They then used camera matching photogrammetry to place the vehicles and personnel into the virtual model, and positioned them as they were just before shots four and five were fired.
Mr. Terpstra used 12 different camera matches to place the vehicles and three camera matches to place the operators into the 3-D model. He also placed Mr. Haag's trajectory cone into the model. Mr. Terpstra demonstrated the precision and accuracy of his camera match by showing how even minor changes to the orientation and placement of the vehicles or operators resulted in misalignment.
Defendant criticized Mr. Terpstra for placing the virtual camera at the wrong height and using the wrong focal length lens. However, Mr. Terpstra demonstrated that so long as the virtual camera is on the same line of sight - and it was - the height of the camera and focal length of the lens could be adjusted without distorting or changing the view of the scene. Defendant also criticized Mr. Terpstra for failing to correct for lens distortion. However, Mr. Terpstra did correct for lens distortion where he had information about the lens that was used. He could not correct for lens distortion for the FBI video because he had no access to information on the lens used to create the video. In any event, he saw no signs of lens distortion in the FBI video.
Finally, defendant criticized Mr. Terpstra for using the wrong frame to create his 3-D model. The confusion arose because the video synchronization Mr. Terpstra attributed to Frank Piazza was actually done by someone else, and it differed from Mr. Piazza's synchronization by nine frames - a total of 0.3 seconds. Thus, Mr. Terpstra's model accurately depicts the vehicles and operators 0.3 seconds before shot five. Although defendant claims that some of the operators were moving at the time of the shots, they could not have moved much in 0.3 seconds, particularly when weighed down by tactical gear and winter clothing. Furthermore, the video shows that defendant stood still, in a shooting stance,
with his rifle trained on Finicum's truck, while the other two operators moved away from the trajectory cone.
Professors Jeff Smith and Catalin Grigoras of the Univeristy of Colorado Denver's National Center for Media Forensics reviewed Mr. Terpstra's report and methodologies. They found that Mr. Terpstra's report was "thorough" and "well documented," that he properly applied principles of 3-D scene reconstruction using scan data, computer modeling, and image analysis, and that his range of certainty was "scientifically sound." Mr. Terpstra used accepted, published, and reliable methods in his work, and created a 3-D model that is reliable, relevant, and helpful to the jury.
Mr. Terpstra has used the analytical camera matching software favored by Mr. Liscio, but found that it reported "false" camera matches that were visually wrong, requiring manual correction.
The defense contends Terpstra is not an expert in forensic science and that, in addition to using the wrong frame from the FBI video as the basis of his analysis, the methodology he used to make his model is unreliable. The defense summarized their objections as follows.
Although Mr. Terpstra claims to have used "photogrammetry," he is admittedly not an expert in that field; he is an animator with no bachelor's degree in any discipline, much less in any field of forensic science. He also conceded that, while he may have used millions of datapoints to create an accurate depiction of the road and other landmarks (the chessboard, essentially) and to construct models of the trucks (some of the chess pieces), the critical task of arranging the pieces (the people and trucks) on that board was a manual, subjective process.
Terpstra also conceded that he used the wrong frame from the FBI video as the foundation of his analysis, a problem that, by his own admission, renders every significant diagram in his report unreliable. He did not, as he claimed in his report, rely on Mr. Piazza's synchronization to choose that frame. Instead, he relied on a synchronization whose methods and error rates are not described anywhere in the record. And he admitted that, even if he had used Mr. Piazza's sync, his model failed to account (and should have accounted) for where each individual was located over the entire error range identified by Mr. Piazza, not merely the one (wrong) frame Mr. Terpstra analyzed.
Furthermore, in creating his model, Terpstra admittedly did not apply his "camera matching" method correctly:
• He admitted that: (1) he relied on lane markers for his matching without realizing that those markers had changed since the time of the shooting; (2) this affected the accuracy of his model; but (3) he could not quantify this error.
• Studies cited by Terpstra in his own report emphasize that "the accuracy [of camera matching] depends on the ability of the user to accurately place the [virtual] camera in the correct position with the correct focal length" (emphasis added). Nevertheless, Terpstra admitted that with respect to the FBI Video—the one picture that really matters—he placed his "virtual camera" over 3,500 feet away from where it should have been and assumed a focal length that was completely wrong. He conceded that there is absolutely nothing in the literature to suggest that erroneous camera positioning and focal length do not matter so long as the virtual camera is placed along the same line of sight as the actual camera. To the contrary, Terpstra demonstrated that changing a camera's distance and focal length can distort the resulting image even if the camera remains on the same line of sight with the correct camera positioning. But his model in no way accounts for this error.
• He agreed that his analysis assumes that the truck remained in precisely the same position during the entire 20-minute period during which the 12 photos used for his camera matching were taken after the shots were fired. He had no idea that the truck had settled in the snow after Round 5 was fired and did not take this fact into account at all.
• Although Terpstra has written at least three papers cautioning about errors that can result in camera matching when lens distortion goes uncorrected, he admitted that he: (1) did not correct for lens distortion in nine of the twelve images used in his analysis; (2) omitted this fact from his report; and (3) does not know anything about the FBI cameras that might support his assumption that it was safe to completely ignore lens distortion instead of correcting for it.
Finally, even if Terpstra had faithfully applied accepted camera matching techniques, he admittedly failed to account for significant errors that are inherent in those techniques. He admitted that, although his report does not assume any accuracy errors at all, camera matching itself does have an error rate—some distance between where camera matching says an object is versus where it actually is. He further conceded that the applicable error rate changes from scenario to scenario, photograph to photograph, and object to object, For example, the "Coleman Study" cited in
Terpstra's report—which relied on much clearer pictures than those at issue here, taken at much closer ranges—still recorded errors of up to 19.6 cm (nearly 8 inches). Mr. Terpstra's own study, which he discussed when he was recalled to the stand, relied on measurements taken by multiple professional photogrammetrists employed by Mr. Terpstra's company, each of whom believed they had used camera matching to "align" the photographs correctly, exactly as Terpstra claims to have done here. Again, the images in that study were of much higher quality, and taken at much closer ranges, than the images at issue in this case. Despite all of that, the average error rate among those professional photogrammetrists was 13 inches (33cm). But Mr. Terpstra did not build in an error rate of 8 or 13 inches into his analysis. He admittedly did not build in any error rate at all, instead inexplicably assuming that the error rate was zero.
Terpstra admitted that his "range of certainty" is not an error rate. Rather, it is a rough measure of precision (how much he could "wiggle" the model with it still subjectively looking right to him), rather than a measure of accuracy (how far away Terpstra is from the right answer).
RULING
Mugnier, a giant in the field of analytical photogrammetry, described Terpstra's method as "graphic art" rather than photogrammetry. But this criticism misses the mark. Terpstra does not claim to be an expert in analytical photogrammetry. He did not have the opportunity to take multiple clear photographs of the shooting scene from different angles. Accordingly, analytical photogrammetry could not possibly be used in this situation. If it could, perhaps Mugnier's criticism would be apt.
In creating his 3-D animation model, Terpstra took the imperfect images that were available and applied a camera matching technique with some similarity to photogrammetry to create a 3-D animation model that fit the images as closely as he could. In doing so he exercised his expertise as a forensic 3-D animator. Notably Mugnier acknowledged being pretty much a layman regarding camera matching techniques. What he condemned Terpstra's work as, "graphic art," is what it is - 3-D animation is graphic art. That does not mean it is inaccurate.
At this juncture, I am persuaded that the government failed to meet its burden of showing Terpstra's animation model should be admitted in full at trial. Terpstra placed vehicles in his model with 12 cross referenced camera matches of relatively clear images. This method can be tested and repeated as shown by Liscio's critique. That part of Terptra's model can be presented at trial and challenged by cross examination.
Terpstra relied on only three camera matches to place the individuals in his model. The still frames he used did not provide clear images of the individuals, only fuzzy smudges. The images were of such poor quality I am unconvinced that this methodology could accurately place the location of the individuals and the positions in which they are posed in the model. The clear image of the model depicting defendant with his rifle shouldered and trained on Finicum's track was not the product of a reliable methodology and involved excessive subjectivity. To present it to a jury would suggest a degree of certainty that cannot be justified and would be unduly prejudicial under Rule 403.
Surprisingly there are a scant number of cases dealing with 3-D animation or computer simulation. Most are unpublished trial court opinions or not relevant. The two decades old Ninth Circuit case Byrd v. Guess, 137 F.3d 1126, 1134 (9th Cir. 1998) (superseded on other grounds) upheld limited use of computer animation under Rule 403. --------
Accordingly, the government will not be permitted to demonstrate Terpstra's model to the jury unless it proves useful to illustrate the testimony of eye witnesses who may be called to identify the positions of people or vehicles at the time of the shooting. However, without such identification by witnesses, Terpstra's animation model may not be presented as an accurate representation of what is depicted in the FBI still frames that look like fuzzy smudges.
COMMENT
All counsel have performed diligently demonstrating the highest level of competence and professionalism in preparing and presenting the complex issues to the court, I look forward to our final pretrial conference to commence July 16 and trial July 24.
IT IS SO ORDERED.
DATED this 20th day of June, 2018.
/s/_________
Robert E. Jones, Senior Judge
United States District Court