- Population mean
- Sample mean
- Population standard deviation
- Sample standard deviation
- Standard error
- Administrative data
- Survey data
- Point estimate
- Confidence interval
- Sampling bias
- Reporting bias
Now, let’s begin. Here is a graph of estimates of cumulative fatalities in Puerto Rico over time.
Figure 1: Estimates from Santos-Lozada and Jeffrey Howard (Nov. 2017) for September and October (calculated as difference of midpoint estimates), and Nashant Kishore et al. (May 2018) for December 2017 (blue triangles), and Roberto Rivera and Wolfgang Rolke (Feb. 2018) (red square), and calculated excess deaths using average deaths for 2015 and 2016 compared to 2017 and 2018 using administrative data released 6/1 (black square), and Santos-Lozada estimate based on administrative data released 6/1 (large dark blue triangle), end-of-month figures, all on log scale. + indicate upper and lower bounds for 95% confidence intervals. Orange + denotes Steven Kopits 5/31 estimate for range of excess deaths through September 2018. Cumulative figure for Santos-Lozada and Howard October figure author’s calculations based on reported monthly figures. [Figure revised 6/3]
Recall, the official count of Hurricane Maria related deaths was as of yesterday 64 (and perhaps still is).
The estimates are derived in different ways. How does one understand the strengths and weaknesses of each?
First, a definitional issue. The 64 count is I believe an estimate of direct deaths due to the Hurricane (e.g., flying debris, drowning, etc.). It does not include for instance deaths due to electricity supply interruption (which could be categorized as “indirect”, see discussion here).
With that prefatory remark made, what is true is that most researchers attempted to identify excess deaths associated with Hurricane Maria, both direct and indirect. In order to do that, one needs to determine what is the normal rate of deaths, and measure what is the currently observed.
With administrative data, the latter is in some sense easy, assuming away biases in reporting and lags that might occur because of bureaucratic and/or communication constraints. The former requires a model — say averaging over the previous six years — as in Santos-Lozada and Howard. Even then, recalling the PR population is declining over this time, means that this averaging procedure is not necessarily uncontroversial. The sample mean (arithmetic average) of the monthly data over the years (say October) is an estimate of the true population mean (I talking frequentist here, not Bayesian), which is unknown. Santos-Lozada and Howard used administrative data, and in fact used the upper 95% confidence interval (based on the calculated standard error, which in turn is calcluated using the sample standard deviation, which is a function of the population standard deviation) for their estimate of a given month’s mean deaths, in order to be conservative (I used actual 2017 reported minus estimated mean in my calculations in the above graph). Had I reported their more conservative estimate, then the cumulative thru-October number would be below Santos-Lozada’s current estimate for cumulative excess deaths through December.
The Rivera and Rolke study uses administrative data, but assumes a different distribution (i.e., does not assume a Normal distribution), and has to estimate part of the October numbers, so differs, but not essentially from the Santos-Lozada and Howard estimate.
Both of these studies based on administrative data have confidence intervals, but it’s important to understand that the confidence intervals have to do with the error in sampling assuming the data are reported correctly (no measurement error).
The big outlier is the Harvard School of Public Health study published in the EJMR, in terms of point estimates. The point estimate of 4565 is surely headline grabbing. However, the authors explicitly noted the 95% confidence interval, from 793 to 8498 (and in reputable publications like the NYT, reported. Here, the number of respondents is small relative to the population, so the sampling error is large, and the resultant estimate of excess deaths corresondingly imprecise.
So is the Harvard School of Public Health “garbage”? I would point out that in their survey design (the team didn’t just go out to Puerto Rico and willy-nilly interview people), they could control for things like remoteness.
In the administrative data, one in principle could do that, but to date, all I know is that the deaths are recorded as the information comes in. We (or at least I) do not know if there is a bias in reportage — are deaths incompletely reported because telephones are not working, and very few people can travel over broken bridges and swept away roads? I think this is potentially very important, given the distribution of these effects is not random. (The lags in reportage is probably not random either; and even in the US in non-crisis situations, the QCEW data lags CES estimates by about six months!)
We found a strong positive association between remoteness and the length of time without electricity, water, or cellular telephone coverage (Fig. 3A). On average, households went 84 days without electricity, 68 days without water, and 41 days without cellular telephone coverage after the hurricane and until December 31, 2017. In the most remote category, 83% of households were without electricity for this entire time period (Table S2 in the Supplementary Appendix).
The distributions around these estimates were often bimodal (Fig. S5 in the Supplementary Appendix), particularly in remote regions, which suggests that households either recovered services relatively quickly or not for several months. Many survey respondents were still without water and electricity at the time of sampling, a finding consistent with other reports.
This suggests to me we may see continued rise in fatality counts attributed to months past, perhaps surging when full power and road traversability is restored — something that might be delayed considerably should another hurricane or tropical storm hit Puerto Rico.
Finally, for those who still have a visceral distate for survey data — if you have been citing the labor force participation rates and employment numbers at the state level, it would behoove you to know you are relying upon survey data, and not particularly large samples, at the state level.
Update, 6/3 6:40PM: NotTrampis refers me to Andrew Gelman’s excellent analysis of the Harvard SPH study, and I am glad to share it with you.