Today Econbrowser is pleased to host this guest contribution from Professor Angus Deaton of Princeton University describing some of his experience with political attacks on academic research.
On weights and coding errors: odd coincidence or dress rehearsal?
by Angus Deaton
The recent criticism of Carmen Reinhart and Ken Rogoff’s work on debt and growth has resembled nothing so much as a public pillorying. According to their critics, a team at the University of Massachusetts, their results are vitiated by a coding error and by their choice of weights. According to Reinhart and Rogoff, the critics’ results are identical to their own preferred results, but which were ignored by their critics. Yet none of this has deflected a firestorm of public criticism and the public dismissal of Reinhart and Rogoff’s work.
I do not comment on the substance of this debate here, leaving it in Reinhart and Rogoff’s capable hands. Instead, I note an earlier and less-known incident in which my own (joint) work came under criticism by economists at UMass– one of whom, Michael Ash was involved in both cases– and where there are a number of almost uncanny similarities to the more important current debate.
The substantive question
In 2003, Darren Lubotsky and I published a paper in the journal Social Science and Medicine. Our topic was whether income inequality is a health risk to people who live in particularly unequal cities or states. The idea is that income inequality is like a toxic pollution, harming everyone who lives with it. My own view, then and now, is that there is no such effect, though I also believe that the extreme income inequality that we see in the US today is a threat to public health, but through quite different (essentially political) mechanisms.
Our analysis showed that the correlation between higher mortality and income inequality arises from a failure to control for the racial composition of the population in each city or state. In cities (states) with a larger fraction of blacks, the difference between the average incomes of blacks and whites is larger, perhaps because employers do not regard blacks and whites as fully substitutable in production. This drives a link from fraction black to income inequality. But blacks also have worse health than whites– in part because of an apartheid healthcare system that treats blacks less well than whites– so that the fraction black is also linked to overall mortality. Those two links induce a strong positive correlation between income inequality and mortality. That this correlation is spurious is documented by its vanishing when we control for fraction black, by the fact that mortality rates of blacks and whites separately are uncorrelated with income inequality, and by the fact that income inequality and mortality are uncorrelated across space in other settings where race is not a salient factor.
The topic has acquired some political baggage. The inequality as pollution story is often favored on the left, particularly in Britain, and those who argue against it have sometimes been accused of doing so on political grounds. Of course, this is but a pale shadow of the political importance of the debt questions in Reinhart and Rogoff’s work.
A challenge
In December 2005, I received a letter from Michael Ash and Dean Robinson at UMass asking questions about the data in our paper and saying that they could not replicate our results. All of the data that we had used were (and are) publicly available on government websites, but they require processing and organizing to be useful. Such replication queries from other scholars are routine, and while we believed that our procedures were clearly enough specified in the paper to permit replication, we were happy to help guide others who are less familiar with this kind of calculation. Darren Lubotsky, who had done the original data assembly, corresponded with Professor Ash over a period of time, and provided explanations, code, and data to allow him to replicate what we had done. Correspondence then stopped, and we assumed that the matter had been resolved.
We were not so fortunate. Almost three years later, Ash wrote to us to say that Social Science and Medicine was going to publish their paper criticizing our work, as was later confirmed by the editor. Different journals may have different policies, but most journals seek the opinion of the authors being criticized prior to making the publication decision. Of course, authors cannot be given a veto, but they will often be able to persuade an editor that the comment is worthless– as was the case here– and thus prevent unnecessary public controversy.
Ash and Robinson’s note claims that Lubotsky and I had made a coding error in specifying the weights in our regressions, and that without the error, inequality retains its significant positive effect on mortality. There was no coding error, but we had chosen weights that downplayed the larger cities and states compared with the weights that Ash and Robinson had chosen. And it turns out that except for one case (in one specification in one data period and with only one of their alternative weighting schemes) all of our results are unaffected by the change in weights. In spite of this, Ash and Robinson make the extraordinarily misleading statement in the abstract that “correcting the error changes the basic results of the paper with respect to inequality and mortality in a relevant and substantive way,” Ash and Robinson’s preferred specification has exactly the same result as our paper, that income inequality is not a health hazard.
Our response, which was published together with Ash and Robinson’s note, makes all of this clear. To our knowledge, there has been no subsequent controversy.
General implications
It is hard to imagine any applied paper that would be immune to this sort of attack. Weighting can always be argued over: in standard regression analysis, it is not supposed to matter, and when it does, it is usually unclear what is being estimated. So if you want to debunk a paper, working through it equation by equation, trying out a range of weights, you will eventually find something that changes. You can then cry “coding error” and hope that the rhetoric shifts the burden of proof back to the original authors.
In our case, as in Reinhart and Rogoff, neither the coding error (in our case there was none) nor the choice of weights has any effect on the main results. In our case, Ash and Robinson simply ignored the results that did not support their charges, and claimed that their results were different from ours in a “relevant and substantive way.” With Reinhart and Rogoff, they referred only to an early paper, ignoring updated results. But the effect is the same, to magnify a tiny or non-existent problem and claim that it threatens the whole enterprise whereas, in fact, nothing of the sort is true.
There is also the question of publication. In our case, I believe that Social Science and Medicine should have shown us the paper prior to the publication decision, but they did allow us to publish a response alongside the critique. In the Reinhart and Rogoff case, Ash et al did not submit their paper to a journal where there might have been a chance of an appropriate professional response, but sent it directly to the world press, copying it to Reinhart and Rogoff on the same day.
Such smear methods appear to work, and provide a handy template for others on how to disguise political attacks as legitimate scientific commentary. While it is naive to think that science can ever be insulated from politics, if these methods of attack are widely replicated, and if journals and newspapers are prepared to abet them, it will make it much more difficult for serious policy-relevant researchers to do their job. Scholars will also be much less willing to share data than is currently the case; doing so allows anyone who is unscrupulous enough to turn your cooperation against you.
This post was written by Angus Deaton
Wow. Thanks for publicizing this story. It’s a shame that UMass Econ feels itself exempt from common courtesy. If they try this trick more often, they’re likely to find that turnabout is fair play.
There are three ideologies. You are a classical liberal. Ash et al are egalitarians. The notion of credibility and reasoned discussion are themselves liberal concepts. Thus, if you are a liberal, you will tend to believe that your actions make rational sense and that your arguments should be judged on their own merits. There is a certain naiveté and optimism in all this. You have the confidence that either i) your reasoning is sound or ii) that you can personally survive if your reasoning is wrong. That’s a pretty high level of confidence.
Egalitarians by contrast operate under the assumption of weakness. Thus, they are always under threat and over-powered by the intelligent and well-educated, from their point of view. Therefore, there is an attempt to avoid the engaging in good faith dialogue–because it involves granting of property rights. I recall Nixon saying something to the effect that, “You can’t trust those North Vietnamese. They lie and never keep their word.” (Alas, I am old enough to remember this.) I always found this curious. Why wouldn’t the North Vietnamese keep their word?
Because granting one’s word is equivalent to signing a contract: it acknowledges property rights. If you agree to something, you have granted the counterparty property rights. If property is theft, then credibility is also theft.
So the presumed means of argumentation is not symmetrical. You are assuming that the argument has its own logic, to be argued in good faith on its own terms. Egalitarians assume that they can’t win in open combat, that they are the underdogs to your overlord. Therefore, they have to play the cards they have, which includes misinformation and propaganda. Really, what is egalitarianism? It is taking away things that other people own through political force. What did they try to do to R&R? What did they try to do to you?
But don’t take it personally. It’s the way the system operates. It derives from a political ideology. From your perspective, it’s a moral failing. From their perspective, it’s an essential tool. Deal with it on its own terms.
But keep in mind, they don’t think like you do.
Except, Kopits, exactly the same kind of issues pop up in the global warming arena with the political sides reversed.
Professor Deaton To quote the old Wendy’s commercial, “Where’s the beef?” By your own account it seems that academia eventually came down on your side. At least that’s how I interpret your statement: “Our response, which was published together with Ash and Robinson’s note, makes all of this clear. To our knowledge, there has been no subsequent controversy.” Sounds like victory was yours and Prof. Ash was left with egg on his face. No?
In all fairness you should note that Prof. Ash didn’t actually do the research work or write the R-R critique in any meaningful way. The credit (or blame…take you pick) belongs to the grad student Thomas Herndon…and I guess we should include his girlfriend. The accounts I’ve read suggest that Prof. Ash’s involvement in the paper amounted to putting his name alongside his graduate student’s, which is pretty much standard operating procedure at a lot of universities.
Also, notice that R-R did not handle the situation in the way that you did. The original R-R paper said one thing and got a lot of attention. A later (error corrected) paper said something a little different. I think most people would expect some kind of explanation and not just a revised paper with different numbers as though the first paper never happened. It sounds like you at least acknowledged a small error in your original paper and successfully explained why it did not upset your original results. About a year ago I presented a draft paper to a symposium. Further research caused me to revise (in some cases significantly) some of the conclusions as I refined my methodology and yes, caught a few dumbass errors. But I also felt it was important to point out and explain the differences between the draft version and the final published version. I don’t think we got that from R-R.
I would also disagree with your claim that the revised R-R numbers are close to the HAP numbers. This is comparing apples and oranges. The R-R “preferred result” uses the median. The HAP paper did not replicate the R-R median but the R-R mean, which is the figure that got all of the attention. I just don’t think you can say that results are similar just because the mean and the median happen to be close. And this is especially true when the whole argument was over a comparison of the means.
I do agree that one can have legitimate disagreements over weighting techniques used in regressions. It sounds like the academic community eventually agreed with your weighting scheme rather than Prof. Ash’s. But that does not mean the R-R weighting technique should be off limits to critics. It was afterall a rather insane weighting technique. Giving a fixed effect based on 19 observations the same weight as a fixed effect based on one observation is just silly. And even in their revised work the weighting scheme is crazy. Apparently the US of 1800 is more like the US of today than is any of today’s OECD countries. That’s nuts.
Steven Kopits,
I’m a rational egalitarian who is now going to obliterate your “argument” by pointing out that many Nobel Prizing-Winning economists are (were) also rational egalitarians, including Gunnar Myrdal, Amartya Sen, James Heckman, Joseph E. Stiglitz, Paul Krugman, and Peter A. Diamond. One could arguably add K. Arrow, J. Tinbergen, P. Samuelson, R. Solow, T. Koopmans, and several others.
I doubt that any of these economists “assume that they can’t win in open combat, that they are the underdogs to your overlord.” Your own violation of the principles of rational argument is to generalize from one ill-mannered article to all “egalitarians.”
One last comment. Sometimes ugly fights have redeeming but unintended consequences. And the R-R fight is one of those cases. The basic R-R claim that debt hurts growth was largely unchallenged for 3 years. Then along came the HAP paper and within a month the scales fell from the eyes of many economists and we were subsequently treated to a plethora of excellent papers going far beyond the HAP critique. So even if you thought the HAP paper was an unfair hack job (I don’t), then you still have to admit that it spurred economists to really put the R-R paper under the microscope. And I think it was these further critiques that really buried the R-R thesis and not the HAP paper. So maybe academic advances resemble sausage making…not always pretty.
Professor Deaton,
Thanks very much for sharing your own story. Not surprising I suppose that the same rascals were involved in a similar drive by.
You raise some important points about the effects of these politicized attacks. It’s necessary for economists to share data and methods if the field is to progress. Yet, as you point out, if your own data might be used against you to unfairly damage your reputation, you’ll think twice about sharing it. But it’s even more important that economists contribute to the policy debate. Economics already has a bad reputation in the public mind as a field in which two economists can get the Nobel Prize for making opposite statements. That’s a misunderstanding of course but it doesn’t help when economists unfairly attack one another in a public forum. It’s not just the reputation of the economist being attacked that gets sullied but really the whole field. People are less inclined to take anyone seriously.
But I think there are some other important lessons to take away from the Reinhart-Rogoff episode. What you and R&R have in common is that you were both attacked because you had politically important results that were backed up by real evidence. When that’s the case, you have to expect people to attack, often unfairly.
I don’t think it’s realistic to expect newspapers or political partisans to refrain from this behavior. If you are saying something important and have evidence to back it up, you have to expect to be assaulted. And when that happens, you have to defend yourself aggressively.
Unfortunately, I think R&R failed to defend themselves effectively. They responded in a very measured way that was respectful of the UMASS economists. They pointed out the academic reasons why the researchers were wrong. It’s a tribute to their character and objectivity as researchers that they responded this way. It pains me to say this, since I have only the highest respect for R&R. But this only helped to twist the knife.
When 3 hitmen from Massachusetts mug you on the pages of a international newspaper and on a comedy tv show, you are no longer in a polite academic dispute. No one who is watching the spectacle understands the nuances of estimators or macroeconomics. You have to respond in kind. The message people need to hear in no uncertain terms is that the muggers are WRONG, WRONG, WRONG as well as being foolish and silly. Point out that they are not experts in the field and have therefore made basic mistakes. Point out that they have missed perfectly obvious results they should have seen if they had read all the literature. Point out that their weighting argument reveals an undergraduate understanding of statistics. Make sure people understand that they have no credibility. If you don’t do that, if you treat them like equals, people will conclude that there might be something to it and will see you as being on the defensive, especially when there’s a full-blown witch hunt on.
People who engage in smear tactics need to pay the price in terms of their own damaged reputations. So do newspapers who publish the smear tactics. And so do faculty members at your own university who aided and abetted the smear.
If they do pay the price, these smear tactics won’t work. Thus, I’d ask you to qualify your assertion that these smear campaigns apparently work. Yes, they do–if you don’t respond to them properly. But it’s not inevitable that they work.
When one shows the results are driven by one data point, one indeed has shown the entire enterprise to be rotten to the core.
I am not sure what you are complaining about…
It seems to me that you are complaining about the scientific method.
Another scientist wants to verify your conclusions: he tries to replicate the analysis and highlights the assumptions (because that’s what they are) he is unhappy about. He makes different assumptions and he gets a different result. What’s wrong with that? Other people will then be able to judge which assumptions are more reasonable much more easily…
Testing other people conclusion for errors is not a disservice in any way. It is a very useful (if somewhat less creative) effort. All theories are only true until falsified. Sometimes the error might not be material… Not testing is not an option…
“While it is naive to think that science can ever be insulated from politics,..” says Deaton.
Deaton is wrong, that can be done easily. Well, Krugman can’t tell you the “how to”, but Bill McBride could tell you.
Question remains if economics should be treated as a science profession, I question that.
Rich Stryker:
Yet, as you point out, if your own data might be used against you to unfairly damage your reputation, you’ll think twice about sharing it.
Take a look around to see how global warming dissidents reacted to this statement made by Phil Jones of UEA climate science:
“Why should I make the data available to you, when your aim is to try and find something wrong with it?”
Slightly off topic, but it wouldn’t be a complete surprise if Angus got a call from the Nobel committee sometime soon.
Steven Kopits,
Thank you for setting the term “liberal” in its proper context when it comes to academia. Liberals do in fact seek wisdom and knowledge rather than power. They assume that wisdom and knowledge will lead to being given the power they deserve.
As you point out, using the term “egalitarian,” there are those who seek equality rather than wisdom and knowledge, and they intuitively understand that this means that they must use power to overwhelm wisdom and knowledge.
I tend to use the term Progressive because of Woodrow Wilson’s appeal to the idea, but Wilson’s concept of “Progressive” fits perfectly with your “egalitarianism” which is also illustrated by the contrast of the American Revolution (based on liberty and freedom) with the French Revolution (based on equality).
Your post is a very thoughtful addition to the post of Professor Deaton.
It might be important to point out that MIT does have a history of using criticism of the hard work and research of others to generate “academic” papers pretending to be rigorous (Paul Krugman).
Greg –
Egalitarianism is based on the declining marginal utility of wealth and income, right?
I don’t believe I’ve ever seen it argued that way. And there’s a reason for that. But, hey, I’d be pleased to do so.
By the way, what’s a rational egalitarian?
I think there’s another interpretation of my comment. I am reminding Prof. Deaton that his work is also ideological, but in a different way.
Unspoken but ever-present is the primacy of the value of scarce resources and a certain indifference about who owns these resources. Thus, efficiency is more important than equity, or put another way, there is the implicit belief that efficiency more or less is best for everyone. And, further, the professor believes that the act of arguing itself is neutral. The argument can be divorced from its advocate and the case argued on its own merits. He, as an individual, does not perceive himself to be at risk as a consequence of debating a point. Nor does he perceive that any rational outcome of a debate is likely to be personally threatening. The conclusions are dissociated from the participants.
But viewed from lower down the deck, such an argument may be viewed as having greater stakes. Liberals were historically criticized for their “devil take the hindmost” philosophy–and if you’re the hindmost, you may have a different view of the merits of the argument. Cheating, stealing and lying don’t look so bad if that’s what it takes to keep you from the clutches of the devil.
So, the point I was making was that liberals operate under a specific ideological regime buttressed by certain assumptions. Egalitarians have their own regime and assumptions. These are not the same.
And, of course, I have not said anything about social conservatives yet.
Regarding sharing of data. Professor Deaton writes: “Scholars will also be much less willing to share data than is currently the case; doing so allows anyone who is unscrupulous enough to turn your cooperation against you.”
Live by the sword, die by sword. If you want open, rational argument, I think data sharing will be essential. But you need to realize that others (egalitarians, in this case) will have access to types of argumentation that you do not. But so be it. Warfare is not always symmetrical.
This topic was, in fact, widely discussed at a conference I attended in Sicily in August. Although I was formally on the energy panel, I found the climate panel more interesting, and spent much of the week with McKitrick, McIntyre, Monckton, Lindzen and the elusive Svensmark. (Yes, it was fun.)
One of the recurring themes was access to data used by the AGW crowd. (Another was the threats that leading AGW academics had made to young researchers in the climate discipline. It was quite eye-opening, and well beyond what I would have expected.)
Based on these discussions, I think we’ll see increasing standards of access and documentation related to data and methodology. I personally think it’s unavoidable, and academics should upgrade their expectations and practices accordingly.
Steven Kopits,
You write, “Egalitarianism is based on the declining marginal utility of wealth and income, right? I don’t believe I’ve ever seen it argued that way. And there’s a reason for that. But, hey, I’d be pleased to do so.”
There are several forms of egalitarianism: a utilitarian view that depends on the diminishing marginal utility of income; John Rawls’ theory of “justice as fairness”; the luck egalitarianism of Ronald Dworkin and others; the capabilities egalitarianism of A. K. Sen and Martha Nussbaum; and several other strands. So, your first task is familiarize yourself with some of this literature.
You also write, “So, the point I was making was that liberals operate under a specific ideological regime buttressed by certain assumptions. Egalitarians have their own regime and assumptions. These are not the same . . . By the way, what’s a rational egalitarian?”
This paragraph also suffers from a lack of familiarity with the relevant literature. John Rawls is a liberal and an egalitarian, so is Ronald Dworkin, and both build on some premises shared by non-egalitarian, or less egalitarian, liberals. In your original post, you claimed that, since egalitarians can’t win arguments, “they have to play the cards they have, which includes misinformation and propaganda.” All of the egalitarians I’ve mentioned, economists and otherwise, offer reasons for their views; they are “rational egalitarians.”
Economics is a field where I frequently find that the theory drives the data. Climate science suffers from the same problem. People’s careers and beliefs are tied to a particular theory, and the goal of their research is to justify that theory, not follow the data to a conclusion. Rather than Steven Kopits’s liberal vs egalitarian distinction, I find a tendency to a near religious belief in a particular theory in any field (e.g. social sciences) which is not subject to exact repeatable observation and experiment. In macro economics I’ve found that by adjusting assumptions and presenting a model correctly it’s possible to prove any hypothesis, and I’ve seen examples of economists of all schools interpreting data to fit their preferred theory.
In the case of the R+R paper, I noticed two points after the initial controversy came out. The first was a graph of the raw data, which (if accurate) looks at first glance like a fairly uniform scatter plot of points. It should be obvious from that distribution that growth is not strongly associated with debt and any trend analysis should be taken with a large grain of salt.
The other point confirms my suggestion that theory drives data. As soon as the initial error came out, which said that growth doesn’t decline as strongly with high debt (but still confirmed that growth declines with debt) I started seeing claims that this now proves that the opposite is true — high debt actually encourages growth (with some elaborate analysis which I didn’t have time to look at, but seemed to be cherry picking the data to prove the point).
RB,
I think you are missing the point. This isn’t about politics or whose ox is being gored. I believe that all researchers should turn over their data and methods. Research should be “open source.” Results need to be checked and stress tested. I have no problem with people vigorously challenging a researcher with his own data. That’s how knowledge progresses.
But just as researchers have an obligation to make their data and methodology public, people who receive the data have obligations too. They have an obligation to present their findings to the original researcher and give him a chance to explain or rebut. They should fairly characterize any disagreement. They should not accuse the original researcher of fraud or incompetence without tremendously strong evidence.
I think Professor Deaton’s point is that Ash did not live up to his obligations. Ash and his coauthor did not give Deaton a chance to review and rebut their findings but instead surprised him with an article that was about to come out. They mischaracterized their disagreement as a “coding error,” implying a mistake on Deaton’s part. And they did not fairly acknowledge that their criticisms, even if true, did not matter for the final result.
What Ash and company did to R&R was much more egregious. They surprised R&R with a hit piece in the Financial Times that implied that R&R were dishonest and incompetent. But when you look at the substance of Ash et al’s charges, they were ridiculous.
I think Professor Deaton was just bemoaning the fact that when researchers behave the way Ash and company did, it poisons the atmosphere, causing researchers to be wary about giving out data, which is bad for progress.
All science should be open source. Climate skeptics should ask for data and climate scientists should hand it over. Climate skeptics should vigorously challenge the scientists. I have no problem with that as long as the climate skeptics live up to their ethical obligations too.
Rick,
… it poisons the atmosphere, causing researchers to be wary about giving out data, …
I agree with this.
On the other hand ..as long as the climate skeptics live up to their ethical obligations …, assumption of fraud and incompetence is often the case with the dissidents, not many of whom can truly be called “skeptics.”
Interestingly, McIntyre saw parallels between R&R and his bete noire Michael Mann. More here .
Dean Baker has a post up discussing this guest post.
http://www.cepr.net/index.php/blogs/beat-the-press/the-inconvenienced-economist
The description of the R & R controversy in this narrative is simply untrue. Five minutes of random searching on academic blogs will give you a better view of the controversy than this story does.
Greg,
Please, educate me on how any of your thinkers contradict the notion of making transfers from the rich to the poor based on declining marginal utility of wealth and income. Did any of these thinkers endorse transferring money from the poor to the middle class or rich? Please cite an example and explain.
And, yes, you do argue like an egalitarian:
“All of the egalitarians I’ve mentioned, economists and otherwise, offer reasons for their views; they are ‘rational egalitarians.'”
So if you provide a reason for your thinking, that qualifies as ‘rational’? I don’t have my homework because the dog ate it? That constitutes ‘rational’ in the context of this discussion?
‘Rational’, in this sense, I would think means that transferring funds from the rich to the poor makes social sense because it increases aggregate utility. Now that’s rational. But that’s exactly what I contend: that egalitarianism (indeed, civil society) is built on the declining marginal utility of wealth and income. That’s why such transfers make sense.
And let me tell you what’s wrong with that from an egalitarian’s point of view. Declining marginal utility is a three-edged sword. First, in a static analysis, it’s absolutely true that transferring money from the rich to the poor increases aggregate social utility. It has to (if it’s in cash).
But it’s also true that utility increases with income. So I can trade off growth against redistribution. And I can do it quantitatively.
And finally, declining marginal utility also means that inequality at higher income levels is less important than at lower income levels. And I can quantify that, too.
So if we use declining marginal utility of wealth and income (OK, let’s give a nod to Scott Sumner: income) as our model for equality, then I can offset any incentive scheme for equality against one based on GDP growth and debt sustainability (the FAA).
In other words, I can take two of my three objective functions for government and put them on an apples-to-apples basis. And that’s why egalitarians don’t use the declining marginal utility of income in their argumentation. Because they then have to argue for equality in the open, on equal terms. And they just don’t feel confident they can win that debate.
Rick –
Ash has no responsibility to Deaton. That’s what I am arguing.
You state:
“…people who receive the data have obligations too. They have an obligation to present their findings to the original researcher and give him a chance to explain or rebut. They should fairly characterize any disagreement. They should not accuse the original researcher of fraud or incompetence without tremendously strong evidence.
I think Professor Deaton’s point is that Ash did not live up to his obligations.”
You are making a socially conservative case, (our third objective function, dealing with the allocation of rights and responsibilities within the group).
You are arguing that Deaton and Ash are part of the same group–academics–and they therefore should share common cultural norms. Deaton is upset specifically because he feels these norms have been violated.
I am arguing that they are, in fact, not in the same cultural group. Deaton is a liberal, Ash is an egalitarian. This is not an intra-cultural disagreement, it’s an inter-cultural one.
In an inter-cultural agreement, you may use tools which you would not in an intra-cultural one. For example, you might write a critical post on a widely read economics blog.
Steven Kopits,
You write, “Please, educate me on how any of your thinkers contradict the notion of making transfers from the rich to the poor based on declining marginal utility of wealth and income.”
Sorry, but you’ve got to take responsibility for your own education. I mentioned lots of egalitarian thinkers whose work you could Google, or even look up on Wikipedia. Until you do that, you’ll remain stuck in your echo chamber, arguing against a simple-minded caricature of egalitarianism that you’ve created out of whole cloth.
I’ll probably regret this, but you might take a look at http://works.bepress.com/greg_hill/3/. Good luck.
“In our case, as in Reinhart and Rogoff, neither the coding error (in our case there was none) nor the choice of weights has any effect on the main result”
Actually, it did. Once corrected, the 90% debt-to-gdp ratio cliff became insignificant and causality was shown to go the other way – that poor economies lead to high debt-to-gdp rather than high debt hindering economic growth.
I am baffled that economists are still implying that R&R results validate fiscal consolidation in a time of severe AD deficiency.
as a PhD in molecular biology, I find it literally unbelievable that you economists, when you publish a paper, don’t have an excel file (excel is the most portable, and can easily handle a few million data points) with all of the data, going from whatever the primary was to your endpoint, and when you get a request, you just email or dropbox the excel…to say that the data is on a gov’t website is just BS and is not acceptable.
For instance, sometimes people say that http://www.bls.gov is a reference.
Have you ever been to bls.gov ?
or, just as bad, people will say some page xxx.gov is a reference, and when you go to that page, there are links to 20 diff data sets, and you don’t know,explicitly which dataset has been used (much less, has the data set changed over time …)
what is WRONG with you people !!!
how on earth do you people call yourselves scientists when you don’t deposit the RAW data on acceptance of your MS ????
Steven,
I guess I’m talking about what ought to be obligations rather than what people perceive their obligations to be. Maybe Ash doesn’t think he has those obligations. But I bet if you asked him he’d say he does and would mean it sincerely.
I think there is something to what you are saying in general. But in this particular case, I have a more pedestrian view of what happened. I think HAP, motivated by ideological bias, believed that they found evidence of errors and dishonesty on the part of R&R and rushed to tell the world. But what they thought they found was mostly a reflection of their own ignorance. I think we can see that if we look in detail at the 3 claims they made.
First, they found a spreadsheet error in the data. That really did happen and R&R acknowledged that. These kinds of errors happen to everyone. They are not evidence of chicanery and the error didn’t change the results materially.
Second, they found that some countries had not been included that would have changed the results. That was true but was a mistake on HAP’s part. They just didn’t realize that the data wasn’t available at the time to be included at the R&R paper was written. Had they been more careful and checked with the authors, HAP could have avoided making that mistake.
Third, they claimed that the R&R weighting method was “unconventional” and “non-standard.” Here they reveal their ignorance of econometrics. You’ll have to bear with me as I explain this, as it is technical.
The issue that HAP didn’t realize they were talking about is a problem in panel data econometrics, in which you have both time series and cross sectional data. The HAP way of weighting is ordinary least squares in which country specific differences are assumed away and only differences across all countries matter. R&R assumed what’s known as “fixed effects” which goes to the other extreme, assuming that country specific differences are fixed constants. As JDH has already pointed out, these are the two extremes of the more general random effects model. A random effects estimate will lie somewhere between R&R and HAP.
How do we know where the random effects estimate will end up? Basically, it depends on this quantity:
Z = v(u(i,t))/(v(u(i,t) + Tv(a(i))
where v(u(i,t)) is the variance of a random error that depends on country i and time t, v(a(i)) is the variance of a random variable that depends only country i, and T is the number of time observations.
If Z = 1, then you get the HAP estimator. If Z = 0, you get R&R. In practice, Z will lie somewhere in the middle. Let’s look at the extremes.
If the number of observations T is small, then Z goes to 1 and you get HAP. That makes sense intuitively since if you don’t have enough data in time, you can’t isolate the country specific differences. On the other hand, if T is large, Z goes to 0 and you get R&R. Intuitively, if T is large, the across country differences wash out.
If V(a(i)) is small, then Z goes to 1 and you get HAP. Makes sense because then there is no country specific variation, just what HAP assumes.
If V(u(i,t)) is small, then Z goes to 0 and you get R&R. Makes sense since R&R are assuming that only country specific differences matter.
So, ultimately where you end up depends on how much data you have relative to the size of the variation of country-specific and across-country differences.
Steven, I apologize for writing down this technical detail but I wanted it be clear that this is really a well-understood technical question.
If you look at the HAP analysis on the R&R weighting in their paper, you will find no discussion like this. They make hand waving arguments and pronouncements that R&R is “unconventional.” They also make an irrelevant point about serial correlation. But they never even come close to showing that they understand the real issue.
The reality is that HAP had no clue what the real issue is when they wrote their critique of the R&R weighting. They just a simple estimate that gave them an answer they liked better and then they ran with it.
What’s worse, they assumed that the mistake they made and the ignorance they exhibited were evidence of mistakes and ignorance on R&R’s part. And then they told the world on the pages of the Financial Times and on the Colbert Report. Meanwhile Krugman at the NY times repeated their analysis uncritically, giving HAP a big credibility boost.
I don’t think what HAP did had had much to do with feeling over powered by the well-educated R&R. I think they really believed they had discovered major flaws in R&R and were in the right. HAP’s egalitarianism was undoubtedly a factor in that it provided the ideological motivation for them to jump to the conclusion that R&R were mistaken and dishonest. But in the end it was HAP who were mistaken and dishonest.
Rick Stryker If each of the countries had 19 episodes of debt > 90%, then that would have been one thing. But that’s not what happened. A lot of the countries had only one or two episodes of debt > 90%. Attempting a fixed effect model with only one or two observations is an exercise in self-deception. A mean based on one observation is next to meaningless when you know that other countries with many observations showed a lot of variation around the country mean.
Also note that they subjectively grouped countries into emerging and developed countries. I don’t have a problem with that, but presumably they did so because of different fixed effects. But if they believed in their fixed effect approach, then why did they feel they had to first group countries into two somewhat arbitrary larger categories? And don’t you find it a little strange that the US in 2009 is treated as being more like the US of 1946 than Germany, France or the UK of 2009? I’ll get back to this point a little later.
If you will read HAP’s reply to JDH’s post, you will see that they are not saying their approach is without its problems. In fact, they specifically said that a better estimate would lie somewhere between their approach and R-R’s approach. Go read their reply. In fact, their whole point about serial correlation was that this was just one example of why a researcher might not want to give each country/year observation equal weight. In other words, they were offering up a reason not to weight things they way they did.
There’s also this problem about what R-R did versus what they said they did. Here’s how R-R described their weighting procedure:
The annual observations are grouped into four categories, according to the ratio of debt to GDP
during that particular year as follows: years when
debt to GDP levels were below 30 percent (low
debt); years where debt/GDP was 30 to 60 percent
(medium debt); 60 to 90 percent (high); and
above 90 percent (very high).
Anyone familiar with the English language would interpret R-R as saying they were giving equal weight to each country/year observation. R-R did not say that they were taking an average of each country’s annual observation, they said that they took an average of the annual observations and grouped all of those annual observations into four categories. Even if you don’t think their weighting procedure was unconventional, you would have to agree that their misuse of the English language was most unconventional.
HAP also pointed to another curious omission in the R-R paper. As R-R noted, the decade in which a country’s debt-to-GDP ratio exceeds 90% is “significant.” Their word. Strangely, even though they say the decade is “significant,” apparently it’s not significant enough to bother controlling for it in their analysis. Didn’t this bother you when you read their paper? Afterall, in applied econometrics it’s common to include some time variable. For example, when measuring production or cost efficiencies in stochastic frontier models it is usually assumed that firms become more efficient over time, so those models will usually try and account for that fact. Why didn’t R-R since they found the temporal factor “significant”? Well, perhaps they didn’t include it in their paper because doing so would have severely undermined their results. One of the things that the HAP results highlighted was how irrelevant the debt-to-GDP ratio becomes over time. If you look at the last two decades there is no difference in the growth rates across all four categories.
Just a technical note about random effects models. In general they are preferred over fixed effects models; however, random effects models are not always feasible. The data have to cooperate. The composite error cannot be correlated with the explanatory variable.
Rick Stryker I don’t think the R-R paper fell into the dustbin of economic analysis solely because of the Left’s embrace of the HAP paper. The HAP paper really prompted a lot of economists who had previously accepted R-R’s finds (e.g., Miles Kimball) to take a second look. And it was those follow-on papers that really crippled the R-R thesis. Way back in college I took a course on the Faust tradition in Western literature. One of the recurring Faust themes was that of Mephistopheles as a “pike in a carp pond.” There was an old belief that activity and movement improved the taste of European Carp (considered a delicacy), so it was common to throw in a pike (considered a garbage fish) into the pond to keep the otherwise slothful carp active. I see the HAP paper as a pike that wakened the economics community from intellectual laziness. So I guess in that sense R-R are right to complain about “carping” critics.
Steven Kopits wrote:
First, in a static analysis, it’s absolutely true that transferring money from the rich to the poor increases aggregate social utility.
Steven,
There was a time when I accepted this but not any more. To make this true it requires more than static analysis. It requires a suspension of reality and a fixed point in time. If such transfers are seen over time not only is there a decline in the marginal utility of wealth and income there is a decline in social utility.
Removing resources from the productive in society reduces both finished goods and factors of production. That means that there is a general or aggregate reduction in wealth not just the utility of wealth but an actual reduction in wealth. This means that the poor actually see a social decline because they have less access to goods.
An obvious example of this is the poor of the United States compared to the poor in most of the rest of the world.
If by static you mean frozen in time perhaps in theory it could be correct but such a freeze must be theoretical because transfers imply time. In any flow of time such forced transfers reduce both the marginal utility of wealth and the actual supply of wealth, leading to a reduction in social utility.
Rawls thinking is only meaningful in an alternative universe of his own making. I would refer Greg to Robert Nozick.
Ezra Abrams,
Unfortunately, it’s not as simple as that. R&R did make their data available on their website. However, HAP (Henrndon, Ash, and Pollin) were not able to reproduce the calculations. So they requested that R&R send the spreadsheet that implemented the calculations. That’s what led to the controversy.
Replication isn’t just about access to data. I have my own replication horror story to illustrate. Years ago I came across a multivariate GARCH model on financial data in a paper. I thought the model would be useful for something I was doing and I decided to start by replicating it.
The data were clearly defined and publicly available so I didn’t have that problem. Also the model was clear. So I coded it up in matlab, using matlab’s optimization toolbox.
And I didn’t get the authors’ results.
Even if I started the maximization at the published estimated values, I didn’t get the author’s results using the exact data set. I thought it might be a problem with the optimization toolbox. I looked at that code and tweaked it. Nothing.
At that point, I thought that maybe I need to go to better optimization code. So I re-wrote the estimation in c++ and used the IMSL c-library for optimization. No luck. I then tried the NAG c-library. Still can’t do it.
At this point, I was determined to replicate the GARCH results. Since I didn’t have the source code to NAG or IMSL, I decided to write my own optimizers. I implemented quasi-Newton and trust region algorithms with numerical derivatives, but that didn’t work any better than anything else.
To avoid derivatives, I coded up the Nelder-Mead simplex. Nope–not any better. Maybe I better switch back to a derivative-based method? Since I thought the problem might be rounding error on the derivatives, I implemented a c++ class that did arithmetic with arbitrary precision to get around numerical derivative round off error. Didn’t work.
Then I implemented automatic differentiation in c++ to compute exact derivatives of the likelihood function. That worked no better than anything else.
I gave up finally.
A couple of years after that I ran in to the one of the authors of the paper and told my story. He said the optimization was a beast and they couldn’t do it either. So they implemented some judgmental overides in the optimization code. Unfortunately, that wasn’t described in the paper though.
These sort of problems are endemic and occur in other fields such as biology and medicine. The book Wrong: Why experts* keep failing us–and how to know when not to trust them *Scientists, finance wizards, doctors, relationship gurus, celebrity CEOs, … consultants, health officials and more is an interesting layman’s account.
These sorts of problems occur even in the hardest of sciences. People know the story about how Eddington led an expedition to Africa to photograph stars near the sun during an eclipse in order to confirm the General Theory of Relativity. When Eddington confirmed the theory, Einstein became world famous. What’s not commonly known is simultaneously Eddington also sent an expedition to Brazil to do the same experiment. That experiment confirmed Newton’s theory of gravity. Eddington believed that the General Theory was correct and so suppressed the Brazil results.
Kopits: “Really, what is egalitarianism? It is taking away things that other people own through political force.”
You talking about the Republican hostage taking????
There is such a lot of sophisticated name calling going on here. I am learning great stuff! Thank you all.
If I had gotten a doctorate in economics, would I have learned all these great name calling techniques? Or would I have learned economics? Whatever the hell that is.
Did Ash and his colleagues criticize R&R and Deaton? Apparently, so.
Did R&R update their conclusions? Apparently, so.
Did Deaton? Apparently, not.
Are any of these questions settled? Not as far as I can see.
So, bottom line, Deaton does not like how Ash criticizes him or R&R.
BFD.
Anonymous Rawls thinking is only meaningful in an alternative universe of his own making.
Just to be clear, Rawls never said that justice as fairness required absolute equality. It’s entirely possible that a radical redistribution of income from the wealthy to the poor might make the poor worse off; and it is for that reason that Rawls very early on introduces the “difference principle” in A Theory of Justice. Under the “difference principle” unequal income distribution is allowed only if it makes the worst off better off. So inequality is conditionally allowed. Of course, Rawls doesn’t deal with the problem of what happens if making those at the very bottom better off also makes those that are next to the bottom worse off. Rawls assumes that the “difference principle” operates monotonically, which may not always be the case.
Rick Stryker Interesting story. Off topic, but since you brought up multivariate GARCH models, I’m curious if you’ve ever found any practical real world use for them. Anything beyond the most simple MGARCH model is computationally intractable. And I say this as someone who has dedicated a fair amount of his professional reputation defending the practical value of plain old vanilla GARCH models. A few years ago I had a deskside with the Assistant SECDEF explaining why GARCH models should be incorporated in multi-echelon logistics models. The savings were modest (~$45M/year), but very real. My colleague and I showed that including GARCH terms dampened the dreaded “bullwhip effect” in multi-echelon models, but we were never able to find much value in extending that approach to an MGARCH model.
2slugbaits,
Just where do Ash and Pollin acknowledge in their reply to JDH “that a better estimate would lie somewhere between their approach and R-R’s approach.” They didn’t do that because they didn’t understand the relationship between their proposed estimator and R&R when they wrote their paper and they didn’t understand it when they replied to JDH.
If they had understood this, they would have never proposed to do the equal weighting they did. In panel data, the question is whether to use fixed or random effects. If T is large, it doesn’t make any difference because they are essentially the same. But in the R&R case, when T is not large, it’s a more difficult question. I tend to agree with JDH that random effects would be better–but you really have to try it to see.
HAP called what R&R did an “unsupportable statistical technique” in their FT article. They called it “unconventional” and “non-standard” in their paper. And they called it “unusual” in their reply to JDH. But what HAP did is non-standard. You don’t start with ordinary least squares (OLS) as HAP did in a panel data context. If they had understood this issue better, they might have argued that R&R’s estimate should be closer to OLS by appealing to the sort of arguments I raised above, rather than making their garbled serial correlation point. But even if they did that, they can’t claim that their OLS estimate is really better than R&R’s without having done some detailed empirical work using a random effects estimator. They never did that.
2slugbaits,
I never did find a practical use for multivariate GARCH models either. Univariate GARCH models are very useful though.
I don’t know anything about the application you mention but I’m sure you’re right that MGARCH doesn’t work for what you are doing.
Rick Stryker Here is where they say that:
We do state in our paper that due, for example, to issues of serial correlation, one might not want to give fully 19 times the weight to the UK experience relative to the one New Zealand year. Just to make sure this is clear, here is what we say on pp. 7-8 of our working paper:
RR does not indicate or discuss the decision to weight equally by country rather than by country-year. In fact, possible within-country serially correlated relationships could support an argument that not every additional country-year contributes proportionally additional information.
How else do you interpret the words “…one might not want to give fully 19 times the weight to the UK experience relative to the one New Zealand year”? In other words, they fully recognize that their approach is not without its problems and that some approach mid-way between what they did and what R-R did was something to consider. So I don’t see where they are disagreeing with JDH’s view at all. JDH did not see their example of serial correlation as convincing, but that’s a secondary issue. The main point is that they agreed with his larger point that some in-between approach might be better than either the R-R or HAP approach.
HAP never claimed to be using a panel data approach. They were clearly using something akin to a cross-sectional approach. And in a second analysis they included decade variables to capture the time effects. It turns out that this was important because ALL of the effects of debt on growth were attributable to the decade in which that debt occurred. Perhaps that’s why R-R found the decade “significant” but for some reason decided against including it in their analysis. What was unconventional and downright weird about the R-R panel data was to pretend you could get precise estimators with a fixed effect sample of one.
I don’t know which econometrics textbooks you follow, but back in the day I was always taught that the decision whether to use a fixed effect model or a random effects model is really driven by the following considerations:
(1) If you’re primarily interested in differences in the mean, then you will probably want to use a fixed effect approach. If you are mainly interested figuring out whether or not an observed effect is the result of a random draw from some probability distribution that includes that effect, then use a random effects approach. Obviously, in the R-R case a random effects approach was infeasible because you cannot establish confidence intervals when you have one observation in many of the groupings.
(2) If you are estimating a fixed effect OLS model by adding dummy variables, then you are likely to quickly chew up degrees of freedom. You can get around this problem by subtracting a population mean, but this means you are also subtracting away a lot of information. A random effects model can get around this problem and is one reason why it is oftentimes preferred.
(3) Increasing the size of T does not solve the problem of composite error terms being correlated with the explanatory variables. That’s why every econometric study using random effects will include a Hausman test. You can only use a random effects OLS model if the composite error term is uncorrelated with the explanatory variables.
Going back to Prof. Deaton’s piece about Prof. Ash not adhering to a certain academic code of conduct, I think Prof. Deaton likely has a point in his particular experience. Afterall, that was literally an academic fight. But I don’t think that carries over to the R-R/HAP debate. Remember, the R-R paper was not peer-reviewed. The only reason it even got published in the AER is because of their sterling reputation. Then R-R moved out of the ivy walls and ivory towers and entered the realm of public intellectual. They testified before Congress, knowing full well how their paper was being (ab)used. They wrote op-ed pieces. The went on the talk show circuit. Once they entered the world of public intellectual they forfeited the right to expect the kind of kid glove treatment they enjoyed in academia. When the topic becomes central to a lively public policy debate you are no longer entitled to quiet behind-the-ivy discussions between academics: Abandon all hope ye who enter here.
@JDH Thanks for publishing this.
Anonymous,
You write, “Rawls thinking is only meaningful in an alternative universe of his own making. I would refer Greg to Robert Nozick”
When Rawls says, “inequalities are just insofar as they work to everyone’s advantage,” do you think this isn’t “meaningful,” or do you just disagree with it?.
And, as for Nozick, if you find his Lockean story of how individuals acquire a right to parts of nature they “mix their labor with,” then I’m torn between replying, “really?” or “does it follow that labor deserves the full right to its product?” You might be interested in Nagel and Murphy’s book, “The Myth of Ownership: Taxes and Justice.”
2slugbaits,
No, Ash and Pollin did not acknowledge JDH’s point in their reply. They quoted themselves, repeating their serial correlation argument, and then concluded in their reply that “In short, doing simple country-year weighting strikes us as more reliable in this case than taking country averages.”
As usual, you have not responded to any of my points but rather with tangents and irrelevancies. My points were, to repeat:
1) There is nothing “unusual,” “non-standard,” or “uncoventional” about starting with fixed effects as R&R do. In fact, starting with OLS in panel data as HAP do is what’s unconventional. If you think that using fixed effects in panel data is unusual as HAP apparently do, then make an argument.
2) I wrote down the conditions under which the estimator would converge to HAP’s OLS estimator. If you want to defend’s HAP’s use of OLS, you need to show how their nebulous serial correlation argument results in a combination of small T, small variance of the country-specific effects, and large variance of the across country error, such that OLS would be justified.
3) It’s difficult to see how this justification could be done a priori. You would need to do empirical work with the random effects model, which neither you nor HAP have done.
I think you are ignoring Professor Deaton’s point. He’s saying that if we tolerate smears, how can we expect serious policy-oriented scholars to inform the public debate? I agree with his point.
Greg,
I apologize. I was the Anonymous who responded to your Rawls commnets.
First, thank you for the link to your paper. I found it to be a very good analysis of Rawls’ thinking.
I agree that the utilitarian argument fails on the grounds Rawls writes about. Nozick also is weak when he makes the argument from a Lockean point of view, but Nozick is closer than Rawls.
Rawls makes a static assumption. He assumes that income inequality always means MWO become more wealthy than LWO, but in a growing economy this is not valid. If at time 1 MWO has a TV but LWO does not and then at time 2 MWO has a wide screen TV but LWO now has a smaller TV, though MWO is more well off, LWO is not really also more well off. Where Rawls makes his error is at a similar point to where Keynes makes his error. Monetary income does not equal wealth. If one acquires more goods at a lower cost, monetary income or monetary measurement of value could decline, but obviously more goods means would mean greater wealth. The poorest today live significantly better than the richest of 300 or even 200 years ago.
Related to this, Rawls does not recognize the inter-relationship between the MWO and the LWO. In a free market world the MWO is a greater producer than LWO that is what created the income differential. When resources are taken from MWO and given to LWO, but the production of goods declines the whole society including LWO becomes less well off. This is not a utilitarian analysis of the society as a whole but of each individual having fewer goods.
For Rawls analysis to be valid he would have to prove that LWO is always harmed with increased prosperity. There is actually evidence concerning this. The poorest in the United States compared to the poorest in the rest of the world.
Rawls reasons from faulty assumptions.
I have posted a response to Angus Deaton’s October 9, 2013, Econbrowser post, “On weights and coding errors: odd coincidence or dress rehearsal?” which critiques my 2009 paper with Dean Robinson, at http://people.umass.edu/maash/deaton-2013/response-to-deaton.html and http://people.umass.edu/maash/deaton-2013/response-to-deaton.pdf.
Professor Ash,
I’ve never used stata, but it seems to me you are misinterpreting the stata manual. I believe the manual is saying that if you want to weight a regression by n, that that’s mathematically equivalent to running OLS in which you multiply the data by sqrt(n).
Look at this stata reference for example. The technical note below example 7 shows that a regression by weighted by n is equivalent to multiplying the data by the square root of n. Also, look at example 7. The example shows that if you want to weight the regression by population, you run the command
. regress drate medage i.region [w=pop]
where they note that w is equivalent to aweight in this context. Thus, if you want to weight the regression by the square root of the population, it would seem to me that this reference is saying that you would set aweight = sqrt(pop), just as Professor Deaton did.
Like I said, I’ve never used stata but it looks to me like Professor Deaton did the right thing and there is no coding error. Can you explain further if you think there is?
Professor Ash,
To put the point more formally, a weighted least squares regression, with weights w(i) is the solution to the problem:
minimize with respect to b0, b1, b2, …bk
sum(i = 1 to n) w(i)(Y(i) – b0 – b1*x1(i) – b2*x2(i) – … – bk*xk(i))^2
If we define the matrix W to be a diagonal matrix with weight w(i) on the diagonal, X to the be matrix of independent variables, Y to be the be the matrix of dependent variables, and b the vector of coefficients, we have the normal equations for the weighted regression:
(X’WX)b = X’WY (where X’ is the matrix transpose)
So, the estimate b for the weighted regression is
b = inv(X’WX)X’WY
Now suppose we define Z = W^(1/2) to be the square root of W, where the weights on the diagonal are the square roots of w(i). Obviously, Z*Z = W. Let’s multiply the regression through by Z, which means that we multiply the data in the regression by the square root of the weights. Our estimate of b is
b = inv((ZX)'(ZX))(ZX)'(ZY)
= inv(X’ZZX)(X’ZZY)
= inv(X’WX)(X’WY)
This is the same as the weighted regression.
This shows that a weighted regression is equivalent to doing OLS where you multiply the data by the square root of the weights. That’s what the stata manual meant.
It seems to me that you have misinterpreted the stata manual. If Professor Deaton wanted to do a weighted regression where the weights are the square root of the population, he would be correct to set the weight to the square root of the population, not to the level of the population as you asserted.
Moreover, it would seem that if you have misinterpreted what the stata command is actually doing, you have also reported incorrect results in your replication article.
Slug,
I never thought I would say this but your October 12, 2013 12:10 PM post on Rawls is excellent especially your comment, “Rawls assumes that the “difference principle” operates monotonically, which may not always be the case.”
My comment: Rawls thinking is only meaningful in an alternative universe of his own making. If LWO never benefit from redistribution then his entire theory fails. He must first prove his assumption. He asks a question then answers it without any proof that his answer is valid. And as you note he analysis is incomplete as it relates to the entire population.
The key claim being made by Professor Ash in his response is that when Deaton and Lubotsky (D&L) claimed that they were weighting the regression by the square root of the population, that meant that they were multiplying the data by the square root of the population. As I’ve suggested in my comment above, that view would be a misinterpretation of what weighted least square is. Still, since I don’t know anything about this literature, I thought I would check to see if D&L were somehow implying that they were doing what Ash and Robinson (A&R) think they were doing.
So, I looked at D&L’s (2009) reply to A&R. There is no ambiguity.
In equation 1, D&L assert that their estimator is
b = inv[sum(i=1 to n)w(i)x(i)x'(i)][sum(i = 1 to n)w(i)x(i)y(i)]
where w(i) is a scalar weight for the ith observation, x(i) is a column vector of explanatory variables for the ith observation, and y(i) is the ith observation of the dependent variable. x'(i) is the transpose of a column vector to a row vector.
It is easily verified that this estimator is equivalent to what I wrote in my comment above in which the weights are on the squared innovations, not on the raw data. It is standard weighted least squares. Thus D&L are correct to set the weight to the square root of the population in the stata code. There is no coding error.
There are some interesting similarities and differences in the R&R and D&L cases. What’s similar is that in both cases the critiques were based on a misunderstanding of econometrics. In the R&R case, it was based on HAP not understanding panel data econometrics. And in D&L’s case, it’s apparently based on A&R not understanding how weighted regressions work.
Neither case need have happened. If HAP and A&R had come to R&R and D&L with their criticisms privately and discussed them, I would hope that we would have not ended up with these false claims.
But the difference is instructive too. The R&R discussion was conducted in newspaper editorials and on a comedy show. But in D&L’s case, there was a journal involved that was supposed to be an independent arbiter. In a dispute like this, I would think that A&R should have sent their code and data to D&L for discussion. If A&R couldn’t agree with D&L about whether there really was a coding error, then the editor should have gotten involved along with the referees to make a judgment. Amazingly enough, it doesn’t sound like that happened here.
So, the moral of these stories is:
Quis custodiet ipsos custodes?