I’ve been tapped to teach the second course in the statistics/econometrics sequence at the La Follette School. I need examples of excruciatingly bad econometrics to discuss. Please post your suggestions as comments. In 2016, I assigned students to examine the empirical content of “business conditions” indices, like Stephen Moore’s/ALEC Economic Outlook rankings.
Some dissections here:
The Phoenix Center “estimates” fiscal multipliers.
The Phoenix Center links the regulatory burden and slow growth.
Less energy regulation causes faster growth?
Well, Menzie, knowing me you’d expect my first comment in this thread to be something smart-A$$ and non-constructive, yes?? I’m wondering if Sabia withholding his data set is kind of like trump/Pompeo withholding State Department documents and witnesses?? Feel free to use your most wicked humor when responding.
If worse comes to worse, you can disconnect CoRev’s dog leash and say “raw data!!! fetch!!! raw data!!! fetch!!!!”
This is the closest I can get to actually helping your students (OK, sad, I know). As far as textbooks and student learning aids go, $35 is pretty cheap yes?? (Would have been very cheap in my college days for either a text or a study guide. I’m assuming it’s even worse for textbook costs in the year 2019. And this is their second course, right?? So they should already be better than Uncle Moses in knowing variables and crunching equations. Have them pick up this book as an optional course material on your class syllabus:
https://www.amazon.com/Mostly-Harmless-Econometrics-Empiricists-Companion/dp/0691120358/ref=sr_1_1?crid=3DC09DZYAM7HD&keywords=angrist+mostly+harmless+econometrics&qid=1574312640&sprefix=angrist%2Caps%2C152&sr=8-1
If they are extra good at searching out free stuff online like Uncle Moses is, they might even be able to pick up the ENTIRE text for free. [ Clears throat in suspicious manner, cough!!!! cough!!!! Google search link, cough!!!! cough!!!! ] Of course….. I’m not endorsing such indecent and naughty behavior, just saying that’s possible for lower income students.
Kids, think search links of the .cn variety here, ok?? You can thank Uncle Moses by finishing your 4-year degree and throwing rotten tomatoes and rotted out fruits at nearby frat houses as you pass by. But not necessarily in that order.
“To see how the Phoenix Center came to its conclusion, now let ΔY be employment growth, ΔX be investment growth, and ΔZ be government consumption and investment growth, and run the regression over the 1960-2011 period et voilà! Proof positive that multipliers for government spending are zero while those for investment very large.”
Pretty bad but try this. Regress change in private spending (C + I + net exports) against change in output and change in government purchases. Complete crowding out proved by an identity!
Carmen Reinhart and Ken Rogoff, “Growth in a time of debt,” American Economic Review, 100, May 2010.
Carmen Reinhart and Ken Rogoff, “Growth in a time of debt,” American Economic Review, 100, May 2010.
Hahahaha, I have to confess this made me laugh (because there’s some truth in it). Of course they will say it was a spreadsheet “error”, rather than a bad model design. My problem was that the “error” just so happened to benefit their line of argument. Was that really 50/50 chance the error would happen that way without being caught?? But I know that Prof Hamilton and Professor Chinn overall respect these two folks, and I will let it lay there at that.
Years ago Martin Feldstein published a controversial paper supporting his designs to reform (make that deform) Social Security. It seems his results fell apart when people discovered his research assistant had coded the key data incorrectly.
As far as regular readers of this blog are concerned, I’ve made my feelings on RAs clear enough. Among a very bad group of Reaganites, who in my grumpier moods I would be just as apt to describe as rodents, I consider Martin Feldstein to be among the best in the group (i.e. I have a limited and very begrudging respect for the man). I find it very low behavior to make RAs do the hardest parts of papers/books and then turn around and blame them when the data/numbers are wrong. I’m going to assume you at least partly agree with me, even if you wouldn’t take it to my “extreme” view.
Obviously I do not have the academic credentials to be a Dean of a dept. But if this happened with a professor/researcher in a department I was dean of, “a Rogoff” or “a Feldstein” would be called into the office privately, told to take 100% of the blame for said “error”, give FULL co-author status to the RA being blamed, or hand me their resignation. Those would be the 3 choices on the table, to be decided before they left that office.
Now you can describe that action I would take as dean of a dept any way you like—but I can tell you this much—research errors “committed by RAs” would go down to ZERO overnight.
i am on the same page with you regarding blame on RA’s. if find it morally lacking to have somebody else do the grunt work, publish the work under your own name, and then blame others for when the work you published is shown to be flawed-especially when the others are not coauthors. i was very fortunate to have an extremely well known supervisor, who let students be first author when they did most of the work. he never published under his name only, unless he did ALL of the work. and he never placed any blame for mistakes on the students-as he was the one responsible for supervising the work. this is not always the experience when working with well known researchers, especially those trying to be the superstar. i got lucky. he was a good guy, and probably the best in the world at what he did. he could have gotten away with poor behavior if he wanted to. he chose the higher ground.
that said, moses, your “approach” as a dean is also extremely flawed and unrealistic. for a tenured faculty member, they can simply laugh at all three of your options. do i like the system as is? no. but that is where we are today. and that is what allows some faculty to take the credit/blame game to the unethical extreme. i will say, most faculty are quite ethical and would not get caught in this trap. but it is the dishonest and unethical ones that tend to capture the media attention in these situations. there are arguments both for and against tenure. this would be one against. however, a heavy handed chair or dean is an argument for tenure protections. tenure works best through faculty governance, not administrator governance. but that assumes factually take their responsibility seriously.
@ baffling
Your point is valid as far as my scenario being very unrealistic. I would think if you have a situation where there’s been an error and pivoting blame on people in more vulnerable positions (RAs), maybe possibly the dean could leverage faculty members or leadership to make some moves. But you’re right, my example under the system (tenure, etc) is pretty fantasyland. So I have to concede you got me on that one.
This is the very best breakdown I have seen online of the Rogoff-Reinhart “spreadsheet error”, and frankly other than looking at the paper itself with some statistical software tools, I dare anyone to find a better breakdown:
https://rooseveltinstitute.org/researchers-finally-replicated-reinhart-rogoff-and-there-are-serious-problems/
Sorry, I broke my word, I said I would let the issue go but I thought the link could be beneficial to Menzie’s students.
Fabulous
Econometrics by charlatans
needs a powerful
Expose
A 24/ 7 monitoring site
Would help
A collective effort
by a numerate pack of lamp carriers
If you want to look at causation versus correlation and selection bias, then a great piece is from The Correspondent, “The new dot com bubble is here: it’s called online advertising.” https://thecorrespondent.com/100/the-new-dot-com-bubble-is-here-its-called-online-advertising/13228924500-22d5fd24
They embed bad econometrics, plainly explained, in a context where the both sides of the bargain – customers purchasing clicks, and media selling clicks – want bad econometrics. A subtitle could be “There is no truth in (buying/selling) advertising.”
Any empirical work that does not spend time checking data sources, methods and accuracy. Teach the kids to stroke and love the data.
“Any empirical work that does not spend time checking data sources, methods and accuracy.”
By this reasonable standard, 98% of the work on the valuation of closely held entities would be bad econometrics, while 99.5% of the work on transfer pricing would be bad econometrics. Of course, I may be a bit off with my percentages here so let me recheck my data for accuracy!
Bob Flood: Sage words! Unfortunately, I sound like Grumpy Old Man when tell my students I used to have to manually type in data from hard copies of International Financial Statistics into the computer, in order to run a regression…(I have, however, not recounted stories of having to have tapes mounted on the mainframe…
I remember having to punch cards, send them to the data processing center and pick-up the green and white printouts the next day. And I wish those damn kids would stay off the lawn.
I was an AF supply officer from 1972 to 82. We operated an Univac 1050-II, 32K RAM! We had lots of punch cards……… and teletype input devices for transactions and queries.
At one of the commands where I worked at the Army hired deaf keypunchers because people that weren’t hearing impaired couldn’t tolerate the constant noise of the keypunch machines.
2slug,
Amen, brother. There are a lot of cases like that.
J.
You are just a young whippersnapper, Menzie. I remember not only that, but that the program and data were on a bunch of Hollerith cards that one carried around in boxes, and one had to go from the UW Social Science building (now Sewell) over to the computer science center to its basement, where one submitted the whole deck and then waited for them to give you back a giant paper printout, there being no desktop or laptop computers to type anything into. Yes, those were the days!
It was quite a day when I finally threw out the boxes containing the cards for all the stuff I did for my diss because there were no longer any readers of the cards on campus.
I remember those days. The day I learned I could sit at a TV screen called a desktop computer and type into something that had a memory in it, I cheered PROGRESS. Of course getting International Financial Statistics data from a computer tape was even more progress even if the damn programming was a COBOL file.
What? No slide rule?
When I was in high school I belonged to the slide rule club. We used to walk around with our slide rules attached to our belts, sort of like cowboys with their pistols in holsters. Wow, we were such nerds.
I remember those days too. But at least you had a typewriter. In my early days, we had to key punch cards.
Yes I’m definitely a Grumpy Old Man!
Menzie, IFS is not country-source data – do not use it. EVER. The IMF stat dept takes the country-source data and massages it. Have Dooley tell you the story of his hiring at the IMF. The IMF MD was embarrassed by Volcker’s telling him that IFS had US M1 wrong. At the IMF we use the WEO dagabase, which was consructed to get around IFS.
I think most of us get that by now. There are so many other ways of getting country source data these days that was not readily available when Menzie, Barkley, and yours truly were kids! One of my favorite moments as a grad student was when my mentor make a joke about the IFS’s defining something called the “world money supply”. He generally was not that funny but this one cracked up the whole audience!
Not as bad as when we carried around big boxes of punch cards.
We would number them with a felt tip pin so in case we ever dropped the box
we would not have to start all over again to get them in the correct sequence.
Since socialism and communism seem to be hot topics among some US residents, is there an econometric model comparing theory and actual for socialism and communism. With a brief internet search, I could not find an answer.
Perhaps of interest may be how leaders such as Mao and Stalin affect the implementation of theoretical communist economics by various social control methods. I heard the results of a recent survey saying that a significant number of “young” folks have never heard of Mao or Stalin. A proper understanding of socialism and communism may be one of the most important issues of the day for the young folks among us.
A significant number of “young folk” have never heard any critique of modern, Les Misérables global corporate capitalism, perhaps we should be mandating the reading of Veblen and Polanyi instead of studying the state capitalism of Stalin and Mao…
Let’s Take the Con Out of Econometrics by Ed Leamer:
https://rady.ucsd.edu/faculty/directory/valkanov/pub/classes/mfe/docs/Leamer_1983.pdf
Note that Dr. Leamer’s excellent paper starts with empirical studies involving farming. CoRev should read this but of course he likely has not. He ends with a discussion of some of the con jobs we get from folks like John Lott on the alleged deterrent benefits from capital punishment.
pgl: I too think everybody should read Leamer’s paper. It is why every empirical paper worth its salt has a robustness section or appendix. I think I recommended it (but did not require it) the last time I taught the La Follette econometrics course.
I’m old enough to remember when Leamer was still promoting an early draft of this classic paper. I distinctly recall one rather Republican but Keynesian economist liked the paper but got all agitated that charlatans like Art Laffer could publish their garbage and yet Leamer was asking the rest of us to be more careful. And yes – this was just after the Reagan tax cuts. I guess not every Republican economist was sold by supply-side “magic”.
Anything ever written by John Lott. To say you time, here’s a critique of one Lott study:
https://www.cato.org/blog/fatal-flaw-john-r-lott-jrs-study-illegal-immigrant-crime-arizona
This one is on immigrants and crime, not guns, but you can confidently lack confidence in any of Lott’s gun studies.
ED Leamer’s paper that I linked to was in large part motivated by Lott’s intellectual garbage.
“According to Lott, the data allowed him to identify “whether they [the prisoners] are illegal or legal residents.” This is where Lott made his small error: The dataset does not allow him or anybody else to identify illegal immigrants.[i]”
I wonder how much of Stephen Miller’s “policy” briefs are based on the “empirical work” of Lott!
You know his family must be proud:
https://www.nytimes.com/2019/11/18/us/politics/stephen-miller-white-nationalism.html
But But But!!!! Mary Rosh was his RA!!!
That’s funny. I take a wee bit of objection to Moses going after RAs as professors like Feldstein are ultimately responsible for the data they use in their papers. But for John Lott – we will make an exception!
Now my ribs hurt…
There is the pernicious problem of people making time-series conclusions based on cross-section data. An example I have been deeply involved involves the ongoing controversy over the Easterlin Paradox, that in many nations, while income is correlated with reported happiness, the latter does not rise as national income rises. So we have Betsey Stevenson and Justin Wolfers in their heavily cited 2008 NBER WP 14287, “Economic growth and subjective well-being: Reassessing the Easterlin paradox,” claiming to refute this argument by looking at cross-section data across nations.
Riochard Easterlin himself provides a reply in a paper I published by him, “Paradox Lost?” Review of Behavioral Economics, 2017, 4(4), 311-339.
BTW, the Stevens-Wolfers paper got a lot of publicity in the meda, with the twwo of them sort of being media stars, a supposedly cool econ couple. But I note that the paper has yet to appear in a refereed economics journal, for all its favorable publicity accompanied by journalists sneering at Easterlin.
PGL beat me to it.
Damn
Lies, Damned Lies and Statistics
https://www.york.ac.uk/depts/maths/histstat/lies.htm
‘A few years ago I thought that I had successfully tied down the origin of this quotation. I concluded that it came from Lord Courtney in 1895 as explained below, but it now appears virtually certain that, whoever first thought of it, it was not Lord Courtney. The origin is still uncertain, but if it originated with any one well-known figure, the most likely candidate is Sir Charles Dilke.’
There’s more – enjoy!
From my link to the origin of Lies, Damned Lies, and Statistics:
“Accountant, The in 1886
Whereupon counsel on the other side was heard to explain to his client that there were three sorts of liars, the common or garden liar … the damnable liar who is fortunately rather a rara avis in decent society, and lastly the expert”
This cracked me up!
I thought it was due to Mark Twain, but maybe he just repeated and publicized it.
When Mark Twain used this phrase in 1904 he attributed it to British Prime Minister Benjamin Disraeli.
How about research discarding possibly useful continuous data for a zero/one coding to fit into a canned regression program. This made sense when programming was time intensive. It’s not now. Have the kids write their own programs and study the histograms of residuals.
The application of econometric techniques to data often results in confusing policy recommendations. If the data sets used in two different analysis are the same, but different econometric methods are used to summarize the series, then the results and policy implications will differ.
The question is ‘why do researcher apply different techniques to the same data sets’?
To start, the researcher must determine if the data set(s) are time series or cross-sectional data. If the data are time series, then the data series must be summarized using time series techniques. If the data are cross-sectional, then the data series may be summarized using regression techniques.
If regression techniques are applied to time series, almost invariably one or more of the underlying assumptions about the error terms are violated. Hence, the estimated equations are incorrect, and the policy implications are incorrect.
Second and closely related to the first, the RHS variables must be theoretical drivers of the LHS variable. Arbitrary selection of the RHS variables results in non-sensical policy implications.
If either the first or second steps are violated, then the policy implications are doomed.
Trump directs $10 billion pentagon cloud contract to microsoft and away from amazon , because trump dislikes bezos. Is this the crony capitalism we despised corev, peakloser and rick stryker? Or business as usual under the trump regime. I notice the crony capitalism does not seem to bother the forever trumpers.
Devin nunes running the republican show was caught trying to dig dirt from………ukraine. How convenient. No conflict of interest there, eh? What a dirt ball nunes is. No wonder he is such an adamant defender of trump. Guilty of same crimes.
Nunes denies this story. Of course Nunes lies often than his blink his eyes.
nunes refuses to directly answer the question.
and now trump is influencing navy decisions with back room deal making to override military law and order. he simply creates chaos wherever he goes. trump is stripping down the military chain of command and promoting the same kind of back room dealings that got him caught in the ukraine bribery scandal to begin with. hey rick stryker, is this the type of behavior you continue to condone from the commander in chief? or as his lawyers refer to him, the king who is beholden to nobody and is immune from the law.
another off topic but informative piece
https://www.cnn.com/2019/11/25/perspectives/gm-electric-cars/index.html
i have argued with steven kopits about the future of electric vehicles. the ceo of gm seems to be in agreement with my arguments. i will argue his timelines are a little slow. he thinks it will be about a decade before ev and gas vehicles have cost parity (assume he means in terms of lifetime costs). my bet is even faster. and it only gets better with time for ev. gas vehicles have really stalled out in their incremental improvements of efficiency. the future of gas powered vehicles will decline faster than most people expect. unfortunately it will bring about a significant drop in the economy of texas, unless we can somehow increase the value and use of natural gas (or increase exports). the rise of renewables will help offset. but the change is going to be rough on the energy corridor. other oil patches without natural gas deposits will suffer greatly.
cnn is on a roll today
https://www.cnn.com/2019/11/25/politics/supreme-court-climate-scientist-michael-mann-national-review/index.html
courts are going to let michael mann sue the national review for defamation, of which it is guilty. hope the fine puts them out of business. the unethical behavior of the climate deniers should be costly, and it would be my pleasure to see them rot in financial purgatory. next mann should focus on losers like corev, who have parroted the claims of the national review.
I would teach them this paper:
https://www.sciencedirect.com/science/article/abs/pii/0304407674900347
Spurious regressions in econometrics
C.W.J.Granger P.Newbold
https://doi.org/10.1016/0304-4076(74)90034-7
First, thank you Menzie and regular commenters for this blog. I have learned much by reading the back and forth between all of you.
As an example of bad econometrics I suggest this paper,
https://www.stumblingontruth.com/~/media/aqr/stumbling-on-truth/files/its-not-the-heat-its-the-tepidity-20150312.pdf
After claiming to be “…pretty experienced at statistics and inference.”, the authors fit a simple linear trend to global temperature data from 1880 to 2014. After analyzing this data they conclude “If things continue along the way they have for the last 135 years, the point at which we reach dangerous temperatures is a very very long time from now.” (pg 2) And “If the next 135 years repeat the experience since 1880, we are in good shape,” (pg 6)
It’s been a long time since I claimed to be a practicing econometrician, but when I see a data set this long my first thought is “I wonder if there has been any change in the process generating the data?” And, when I see a regression line fitted to data, my first thought is, “Are there any patterns in the residuals?”
Because both thoughts gave me reason to question their conclusions, I split the data in two and did a simple Chow test on the two regressions. Sure enough, the coefficients are statistically different in the two regressions. I stopped there because I am not a climate scientist and I proved to myself that the “if” in their conclusion was doing a lot of heavy lifting. There is more to explore in the remainder of their paper.