An interesting new working paper, “Quarterly GDP Revisions in G-20 Countries,” by Manik Shrestha and Marco Marini, documents the fact that at the end of 2008, statistics understated the true extent of U.S. GDP decline.
The Surprise in 2008Q3-Q4 and Assessing the ARRA’s Impact
Source: Shrestha and Marina, “Quarterly GDP Revisions in G-20 Countries,” IMF Working Paper No. 13/60. The observations pertain to the first release (1st), the release three months out (3m), one year (1y), and two years (2y).
Notice how the the 2008Q3 estimate moved from positive growth to negative. As the authors observe:
We can appreciate how earlier estimates of quarter 2008:Q4, epicenter of the crisis, have been revised downward in both the U.S.A. and the U.K. … this pattern has continued for all the quarters in 2009 in the U.S.A. …
Why is this of interest? Recall the oft-repeated It’s important to understand that the Bernstein-Romer estimate of the impact of the ARRA was conditioned upon the understanding of the extent of the downturn as of January 9, 2009, at which time only the 2008Q3 estimates were available (the 2008Q4 advance release would be published at end of January). The surge in unemployment (c.f. Okun’s law) as output dived was taken as failure of the ARRA, rather than the outcome of conditioning on an overly optimistic assessment of the state of the economy. This is true despite the fact that Bernstein and Romer write explicitly “Table 1 shows that we expect the proposed recovery plan to have significant effects on the aggregate number of jobs created, relative to the no-stimulus baseline.” (p. 3).
So one cannot directly assess the impact of a policy without considering on what assumptions the baseline forecast was conditioned on, as discussed here. This point also applies to other projections. 
Implications for Assessing Forecasts
From the conclusion to the paper:
The second finding is that overestimations in the first release of quarterly GDP have been more frequent for the 2008–2009 quarters compared with the quarters of the previous years. Negative [mean revision] values are found for eight countries in the post-2008 period, compared with only four countries in the pre-2008 period. More negative values of the MR statistics after 2008 are noted for countries with a significant fall in the GDP during the years 2008–2009 (in particular U.S.A., and Japan), while for the U.K. the mean revision to the first estimate of quarterly GDP growth after two years is confirmed to be around zero despite the strong economic downturn …
The result that pre-2008 revisions are approximately mean zero for the U.S. is consistent with Faust, Rogers and Wright (JMCB, 2005) (ungated wp version).
This finding of negative mean revisions post-2008 (although based on a relatively few observations) suggests that one needs to be very careful about making strong conclusions regarding the state of the economy on the basis of advance GDP data, e.g., in or out of recession, at least for now. As economic conditions stabilize, there is no guarantee that the tendency toward negative mean revisions will persist.
Moreover, it would be wrong to conclude there is no information in U.S. advance GDP data, as the growth rate from the advance release correlates with the growth rate from the estimate 2 years out 88% of the time, even in the post 2008 sample (and 94% of the time in the pre-2008 sample). This finding is illustrated in Figure II.16b:
Source: Shrestha and Marina, “Quarterly GDP Revisions in G-20 Countries,” IMF Working Paper No. 13/60.