There has been some discussion recently about discrepancies between different government estimates of the state of the labor market. Although a legitimate issue has been raised, there has also been a bit of misunderstanding.
The Bureau of Labor Statistics provided a great boon to business cycle researchers when it began publishing the Business Employment Dynamics data. The BED data divides establishments into two categories: (a) firms that are either new establishments or are hiring more workers compared to the previous quarter, and (b) firms that either go out of business or are hiring fewer workers compared to the previous quarter. The number of net job additions from firms in the first class is referred to as “gross job gains,” whereas the number of net jobs lost from firms in the second class is referred to as “gross job losses.” The BED numbers for gross job gains and gross job losses come from the Quarterly Census of Employment and Wages.
These BED data are collected separately from (and reported with a considerably longer delay than) the Current Employment Statistics data. The CES data are instead simple counts of the total number of people working at surveyed establishments. The CES numbers are collected and reported monthly and provide the basis for the “nonfarm payroll employment” numbers that get most of the coverage in the press.
You’d think in principle that if you took the difference between the gross job gain and loss numbers from the BED, you’d get the same number as the change in the number of people working according to CES. Historically you typically did get roughly comparable numbers, but the estimates differ somewhat for the most recently available quarter, 2006:Q3. Barron’s Alan Abelson (via Barry Ritholtz) reports the following claims attributed to Philippa Dunne and Doug Henwood of the Liscio Report:
compared with a gain for the [2006:Q3] quarter of 442,000 jobs reported in the so-called establishment survey, the Business Employment Dynamics, or BED, reckoning was a scant 19,000 additions. In manufacturing, the 9,000 jobs lost according to the payroll figures balloon into a loss of 95,000 jobs in the BED data; the improbable 20,000 additions in construction (think: housing) turns into a loss of 77,000 by BED’s measure; the 507,000 gain in private services shrinks to 108,000. And so it goes. Or, more accurately, so goes the job mirage.
One likely culprit, Philippa and Doug suggest, is that curious concoction known as the “birth/death” model used by the Bureau of Labor Statistics to estimate the gains/losses in jobs from the launching and demise of businesses. Thanks to this voodoo calculation, 156,000 were added in last year’s third quarter and a hefty 388,000 in the opening four months of this year.
Bloomberg reported some slightly different numbers attributed to Ray Stone of Stone & McCarthy Research Associates:
The new [BED] data revealed a seasonally adjusted third-quarter private payroll gain of only 19,000, in sharp contrast to the BLS’ published monthly payroll [CES] increase of 498,000 for the quarter.
Calculated Risk has also mentioned the discrepancy between BED and CES, though he did not present the particular numerical calculations repeated above.
The 442,000 jobs growth number for 2006:Q3 was apparently arrived at by Dunne and Henwood by taking the average of the seasonally adjusted CES-reported levels of employment for July, August and September and subtracting the average seasonally adjusted values for April, May and June. However, temporally averaging CES data is not conceptually the correct way to get a number comparable to the BED value, since the latter is intended to reflect the situation on the 12th day of the third month of the quarter. The 498,000 figure calculated by Stone is better, being based on the difference between the September and the June CES values.
But the glaring error by either analyst was in trying to compare the 442,000 or 498,000 CES figure with a number of 19,000 from the BED. The 19,000 figure was arrived at by first taking the BED raw count of gross job gains and seasonally adjusting it, and then taking the BED raw count of gross job losses, and seasonally adjusting it separately. The seasonal patterns of these two series are quite different, and when you seasonally adjust them separately, and then take the difference between those seasonally adjusted sums, you get an artifact that should in no way be construed as the seasonally adjusted net employment gains.
The correct procedure, if you want to know whether the two surveys have come up with the same count, is to use the seasonally unadjusted values for each.
The change in the actual, seasonally unadjusted CES count of the number of people working at private establishments between June and September was a loss of 298,000 jobs. The difference between the seasonally unadjusted BED job gains and losses in 2006:Q3 was a net loss of 453,000 jobs. The discrepancy between the two is therefore 453 – 298 = 155 thousand jobs, not the 479 thousand job discrepancy claimed by Stone nor the 423 thousand job discrepancy claimed by Dunne and Henwood. In other words, about 2/3 of the discrepancy claimed by these analysts resulted from a misuse of the data by the analysts rather than a problem with the data that BLS reported.
For perspective, the graph below plots the cumulative seasonally unadjusted net employment change over 4 quarters as calculated by BED and over 12 months as calculated by CES. The two series track each other pretty well.
 |
Even so, a discrepancy of 155,000 workers within the single quarter 2006:Q3 is more than it should be, and suggests something is clearly wrong with one of the measures.
BLS has reported an analysis of some of the prior discrepancies between the BED and CES figures. The report investigated a number of possibilities. Some firms answer one survey but not the other, some firms give different answers to the same question on different surveys, and there are possible errors that can arise from either means of data collection. The analysis did not find evidence that such factors could account for big differences between the two measures. Instead, the most likely factor identified in the BLS report is indeed the birth/death model fingered by Abelson above.
I do not know what Abelson’s definition of “voodoo” might be, but I doubt that many statisticians would want to describe the BLS birth/death model with such a term. Details of how it works can be found here. Perhaps “voodoo” to Abelson means “something with math in it.” The basic idea behind and motivation for the birth-death model is quite simple. The CES number of people working is based on counts received from firms who filed a report. Unfortunately, not all firms file reports on time. One case in which firms often do not file a timely report is if they have gone out of business. If you didn’t receive data from a firm this month, that could mean that the firm has gone out of business, or it could mean that they just didn’t get their report filed this month.
If a firm filed a report last month but did not file this month, is your estimate of the number of people working there now equal to zero? Or is your estimate the number the firm reported for the most recent available month? If you gave either answer, please go back and retake Stat 101. The best guess of the number of people working would come from forming some estimate of how many of the nonresponders represent business “deaths” and how many represent data errors. The way you would form such an estimate would be to look at historical data for what are the odds that a missing observation represents a business death rather than just a late report.
The other kind of firm that you’re going to miss with the CES survey is one whose existence you did not know about at the time you set up the survey design. Again, is your statistical estimate of the number of people working at new firms this month zero? Mine would be based on looking at what the past numbers for jobs coming from new firms have been and how they correlate with things I currently know.
And that is what the BLS birth/death model is all about. It is not voodoo and it is not mysterious. It is just an effort by the BLS to use the data it has to estimate the data it does not have.
Now, that is not to say that the statistical basis for the birth/death adjustments could not be improved, nor is it to claim that even the very best conceivable model could always tell you accurately the values for numbers you did not observe. As Calculated Risk has emphasized, at the moment, with a weak economic environment in general and a very troubled housing sector in particular, it seems very likely that a higher fraction than usual of the nonreporting establishments have in fact gone out of business, and that there are fewer new businesses starting than would be typical.
 |
For this reason, it seems very likely to me that recent CES data have been overstating the extent of employment in residential construction. The next question is, what should we do about it?
The birth-death issue strikes me not as a problem with the CES data construction, but instead is a fundamental limitation of any data gathered directly from firms. This in my view is another good reason to be using the BLS household data, for which a surveyor goes to a particular residential address to collect data on how many people there are working, as a supplement to the CES establishment data. Yes, I know, the household count has problems of its own, and it would be an even bigger error to rely on it alone. But the good news is that the household survey gives us an estimate that at least is not contaminated by the birth/death issue. And the household data would lead you to conclude that employment growth in the first half of this year has been weaker than the CES estimates suggest.
Given these concerns, beginning next month I will be increasing the weight I place on the BLS household survey from 10% to 20% and decreasing the weight on the CES establishment data from 80% to 70%.
I conclude that the problems with the CES data are more significant than I had been estimating, though substantially less severe than some analysts have suggested. I continue to believe that the best way to deal with such problems is not to throw out data, but instead to widen the set of variables that are regarded as informative. That approach supports the inference that U.S. employment growth in the first half of this year was likely less robust than is currently reported, particularly in the construction sector.
Technorati Tags: BLS,
employment,
birth death model,
CES employment,
BED employment
Its not the deaths, but the births that are so problematic with BLS.
Prior to 2001, the B/D model created relatively few new jobs as part of the overal MEASURE of job creation. Since then, B/D hypotheses are an increasingly large proportion of the total BLS NFP job numbers. Last year,it was 42% of total non-farm payrolls.
Note that the establishment survey is an actual measure (which then gets extrapolated) versus BD, which is rather hypothetical as opposed to measured.
Indeed, because of this, the model all but guarantees the Birth portion of the B/D enhancement will be wrong at turning points.
I agree, Barry. Turning points are definitely the weakness of any birth/death modeling effort.
Unfortunately, the turning points are the most critical points (for policy decisions, esp. those of a monetary nature).
Sigh.
And since turning points cannot be reliably identified as they occur, we cannot know when the birth/death plug is most liable to mislead until after the fact. sigh
Birth/death assumptions becoming dated seems to be an unavoidable problem.
Why not revise the benchmarks every 6 months instead of once per year? They would have to appropriate more money to the BLS, hire more statisticians, etc. But there is nothing magical about the March QCEW data that is used to benchmark. If they had the resources, I don’t see why they couldn’t benchmark the second half of the year to the preceding September QCEW data.
It wouldn’t do away with the problem, but it would alleviate it.
Great post. What seems amazing from the BD model is that new companies(i guess there maybe 2% more new companies every year and mostly are pretty small) are generating the largest part of employment within a very weak economic enviroment.
Very nice synthesis, Professor. Thank you.
Given that we have a dual-mandate Fed and that businesses make financing and expansion plans based on their read of the general economy, we clearly need better employment information at these inflection points.
I’d rather have a model that indicates economic performance second derivative sign changes, instead of a model that attempts to have an accurate total count during all the steady-as-she-goes economic times….
These measures are equivalent to blocking the car windshield with a blinking sign that says “you have not hit a wall yet”….
Kevin Depew at Minyanville said that the BLS refuses to allow academic or commercial economists access to the models used for birth/death additions (but I don’t know if this is true). Also, consider what economist Paul Kasriel says:
“I am a big believer in the notion that the private sector can do most things more efficiently than can the government sector. This notion extends to measuring economic activity. Perhaps the ADP estimate of payrolls, which is based on actual company payroll data processed by ADP and which is not adjusted for the birth and death of small businesses, presents a more accurate picture of employment growth in America than the BLS Establishment and Household Surveys does.”
An excellent post, I agree. How is that model coming along RP? I have faith that someday the model that encompasses enough variables to truly track the economy will be developed due to the efforts of fine economists such as JDH. When this happens, a chimpanzee can set monetary policy by simply taking the second derivative. Chimps should be able to do simple calculus by that time.
I’m not positive Kasriel is right on that point. ADP is not predicting payrolls. They’re predicting what the BLS will say payrolls are. So if they’re looking at how their monthly data compare to historical BLS payroll numbers, then ADP is taking into account the B/D plug.
> How is that model coming along RP?
I suspect a model can be built either from noting an increasing divergence of the various steady state models, or perhaps whenever revisions start to be on the same order of magnitude as the earlier numbers.
But I have a paying job, so any such model would have to be a hobby.
First, thanks for that enlightening post. Extremely helpful. Having crunched my own numbers and driven the assessments with YoY differences I still didn’t see what you bring out. That said despite the remaining data gathering problems the various series seem to me to behave similarly (hats off to the BLS for a difficult task that gets nothing but critques). In other words we can see the directional and turning point characteristics and watch the trends. Particularly looking at YoY changes it’s very apparant graphically.
BtW the ADP numbers track well with the Private SA numbers of the CES.
Good reconcilation–for sure the CES payroll estimates appear to be overstating growth. The question is largely one of degree. I suspect that the problem is more than the Births/deaths model. I think that the BLS has some inherent problems in the “probability-based” sample as it relates to very small firms, including those in construction. The BLS sample for construction has an average firm size of 51 employees, compared to the UI data, which has an average firm size of about 9 workers. I have sliced and diced these data in a number of fashions. Using the raw QCEW data it looks to me as if the change in payroll growth per the CES estimate for the period Mar06 to Sep06 is 200K to 300K. But because of differences in seasonality I also looked at the 12 month change (Sep06/sep05), here the discrepancy appears to be about 350K. Using the seasonally adjusted BED data I come up with a difference of about 325K.
Ah, but aren’t the inputs to the ADP model all from payroll checks? ADP adjustments may aim at picking up seasonal factors as reflected in BLS data, but does ADP really bother looking at the B/D plug? Assuming I’m right that all the ADP inputs are just paychecks counted, that would be one great value of ADP. Whenever the relationship falls apart, we have reason to think that BLS is getting fewer responses or imputing hiring or something. ADP never has trouble getting firms to report jobs.
My understanding is that the ADP series is based not exactly on the number of paychecks, but rather on the number of payees. Back in 1991 there was a very large downward benchmark revision to payrolls, and this actually stemmed from ADP reporting paychecks rather than payees. In many cases overtime pay or bonus pay was associated with a separate paycheck, which resulted in a double counting of workers. What the guys at Macroeconomic Advisors do in estimating the ADP series is simply regress the ADP payroll counts against the BLS postbenchmark series. After establishing the historical relationship, they simply use the current ADP payroll count and employing the historical relationship, calculate a estimate of the BLS change. They contend (correctly so) that their series is a better gauge of the 2nd, 3rd, and post-benchmark estimate of payrolls than teh 1st closing. The ADP payroll growth is actually materially faster than the BLS series, so that the trajectory of ADP growth is scaled back to more closely ressemble the BLS series
JDH said: “…Even so, a discrepancy of 155,000 workers within the single quarter 2006:Q3 is more than it should be, and suggests something is clearly wrong with one of the measures…”
A discrepancy of 155,000 workers may suggest something wrong with one of the measures, but with a workforce in the neighborhood of 140 million *is it meaningful enough to make/change policy, an economic forecast, or an investment strategy*? Especially when the BED and CES numbers (in the first chart) track so closely?
“Massaging” data to make it a better “fit” for the *perceived* conditions, instead of simply accepting the data and questioning whether I’m perceiving the conditions properly has almost invariably ruined my forecasts. “Who are you going to believe, me or your lying eyes?”:)
Sebastian
THis is a very interesting discussion. You might want to look at our article forthcoming in the Journal of Business and Economic Statistics
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=608221
which tries to get at this “true birth” and “true death” issue using UI wage records. Here’s the abstract
This paper uses a novel approach to measure firm entry and exit, mergers and acquisition. It uses information about the flows of clusters of workers across business units to identify longitudinal linkage relationships in longitudinal business data. These longitudinal relationships may be the result of either administrative or economic changes and we explore both types of newly identified longitudinal relationships. In particular, we develop a set of criteria based on worker flows to identify changes in firm relationships – such as mergers and acquisitions, administrative identifier changes and outsourcing. We demonstrate how this new data infrastructure and this cluster flow methodology can be used to better differentiate true firm entry/exit and simple changes in administrative identifiers. We explore the role of outsourcing in a variety of ways but in particular the outsourcing of workers to the temporary help industry.
Julia
In comparing the BED and CES I would actually think it’s better to use the seasonally adjusted data, since they cover somewhat different seasons. The BED is measuring net changes in employment from July 1 to September 30th, while the CES would be giving us survey data based on the pay periods that included June 10th and September 10th. The seasonal changes in employment over this period are very large, so comparing unadjusted data that has roughly 40 non-overlapping days could give a very distorted picture.
Dean, according to BLS, “Both the CES survey and the QCEW program use the same pay period including the 12th of the month reference period for employment.”
JDH, For reasons that I don’t fully understand the seasonality of the QCEW counts and the CES estimates are closely aligned, but not identical. The QCEW data for the first 2 months of each quarter are believed to be less than accurate. The thinking is that when firms fill out the UI forms following each quarter, they make rough estimates for the 1st 2 months of the quarter, but have an accurate reading for the final month. But, one would think that the seasonal behavior over the full 3 month period, say June to September, would be nearly identical for both the CES and QCEW series. The only justification for slippage might be those workers not covered by UI, but included in the CES estimate. About a year ago there was a lot of talk about merging the QCEW program and the CES program at the BLS. The thinking was that the CES data might be effectively benchmarked once a quarter (with a lag), instead of once a year. This was before the big March 2006 benchmark, and what now appears to be a large benchmark revision in the opposite direction. The problem in doing quarterly benchmarks is that offseting discrepancies from quarter to quarter become apparent instead of being smoothed away.
Thanks JDH, I had noticed the how seasonal adjutsments of the data did not match up, as Ray indicates and had assumed that this could be explained by the difference in periods. Oh well.
The correlation between quarterly changes in the BED and CES data (taking three-month averages of the monthly data, which is close enough for these purposes – this isn’t physics) was 0.88 going into the most recent quarter. So normally, the BED tracks the benchmarked CES data very closely. It didn’t in the most recent quarter, which is news, no?
Surely it’s worth noting what the BLS has to say about the BED vs the CES. They point out that they are for DIFFERENT uses. The BED tracked enterprises are longitudinally linked so employment changes based on start-ups, closings, expansions and contractions will “show the dynamic labor market changes that underlie the net enployment change statistic.” They go on to say explicitly that “As such, data users interested particularly in the net employment change and not the gross job flows underlying this change SHOULD REFER TO THE CES DATA….” (my caps). Why isn’t the the BLS view of these data series ever mentioned??
.
The BED data are directly derived from the QCEW or what used to be called es202 data. It is these data that the BLS uses to benchmark the payroll estimate each year. The BED data is simply a sorting of these data by expanding and contracting firms, along with new firms and firms that have gone out of business. The nice thing about the BED data is that the BLS publishes a seasonally adjusted verison, whereas such isn’t available in the raw QCEW data. There are a couple difinitional differences. the QCEW and BED data are slightly more inclusive than the CES data. For example, there are some agricultural, logging type, and household type workers that are included in the QCEW and BED data but are excluded from the CES. Effectively, anyone who is eligible for UI is included in the QCEW. The CES includes some workers not cover by UI including certain railroad workers, clergy etc. I think that you might find still a better fit between the BED and CES simply by taking the 3-mo change in the CES series for the quarter end months, and comparing these with the net BED changes (as oppose to the Q3 average vs teh Q2 average).
Thanks, JDH, for all this follow up on B/D model and other BLS data. I remain convinced that perhaps some things are so complex (almost chaotic) that they can’t really be measured like this type of activity. Another example is the fact that no one can ever really predict a recession yet we know they happen – but not until long after the fact. Geez, Nostradamus could do that! I enjoy the academic repartee though.