You should run away from that “policy analyst” as fast as you can. The above is a quote from Steven Kopits, who styles himself a policy analyst. His entire comment is here.
When I teach statistics for policy analysis, many of the examples of what not to do come from Mr. Kopits (a recent compendium). Click here to see the many instances where he (1) declares recession in 2022H1, (2) calculates ratios of chain-weighted quantities, (3) argues that the employment numbers were cooked before the election, (4) argues that government interest payments were 13.6% of GDP (I get 3.7% in 2023), (4) mistook a graph of estimated recession probabilities for underlying VMT data, (5) claims oil caused the 2010-13 euro crisis, (6) asserts macroeconomists ignore demographics, (7) argues that minimum wage increases in NYC would cause an employment disaster, (8) thinks potential GDP should be thought of as a “speed limit”.
(7) argues that minimum wage increases in NYC would cause an employment disaster,
After all – monopsony power cannot exist according to the world’s worst policy analysis. And raising oil prices by collusion is perfectly AOK.
Perhaps, perched as I am in the lower branches of the knowledge tree, I can help clarify this “confidence interval” thingie for my fellow low-branch denizens.
In finance, one occasionally hears the expression “risk-adjusted return”. What, one wonders, does ‘risk adjustment’ mean? Something to do with the odds of being able to anticipate actual returns, based on what we already know. It’s important to know how likely you are to get the returns you expect, because low confidence might change your decision about the investment.
Now, what might that have to do with policy analysis? Well, let’s imagine that policymakers would like to know how likely a policy is to succeed before committing resources to it. See how that is analogous to knowing how likely a particular return on investment is? Of course you do; even those of us perched on the low branches understand that.
Those who flap their way to the higher branches? Some, apparently, are too worn out with the effort to do such thinking.
I think Kopits has proven once again why he sits on the right (as in right vs left) side of the aisle politically. Many members of the right celebrate their own ignorance. Kopits seems to be very proud he doesn’t even have undergrad/101 knowledge of statistics.
There is in fact a stark difference between just being ignorant and celebrating ignorance. Those who have some moderate amount of embarrassment about their own ignorance will be driven to learn more. Those who bathe in ignorance like a happy pig wallowing in mud will tell you all sorts of reasons why they can’t be bothered.
Steven Kopits I don’t understand why you would not want to provide your clients with a confidence interval. I would think most clients would insist on seeing the confidence interval associated your estimate of the mean. Confidence intervals might be symmetric, but your clients’ risk tolerance probably isn’t.
Do you know how to interpret a confidence interval? That’s a serious question. A long time ago you made some comment that suggested a confidence interval meant that some outcomes were more likely or less likely than others depending upon where they fell along the x-axis. That’s a very common misunderstanding. Confidence intervals give you the range that covers the true population mean. The true population mean could be anywhere within that range. The confidence interval approaches a normal distribution regardless of the underlying distribution of each sample. You take lots and lots and lots and lots of samples and you can be 95% confident (or whatever level you choose) that the true population mean will lie within the confidence interval. Professional statisticians will no doubt slap their heads after reading this, but that’s a rough-and-ready explanation of confidence intervals.
All good points and questions but know this. Stevie does not have clients he cares about. All he cares about is getting appearances on Fox and Friends.
I think the confidence interval has been discussed in the past on Econbrowser.
I always need to re-review the concept. I am sure 2slugbaits’ math skills are much more advanced than mine, so the comment below is meant as a collegial addition to the discussion.
In statistics, a 95% confidence level means that if we were to take multiple samples and compute a confidence interval for each sample, then 95% of those confidence intervals would contain the true population mean, as 2slugbaits mentions.
However, it’s important to note that this doesn’t mean that there’s a 95% probability that any one specific interval contains the population mean. Once the interval is calculated, the population mean is either in that interval or it isn’t. The 95% confidence level refers to the long-term behavior of the method for calculating the interval.
It depends, Slugs. Let’s take the specific example of the EIA’s oil price forecast which Menzie posted on May 7.
https://econbrowser.com/archives/2024/05/may-2024-short-term-energy-outlook-wti-gasoline-prices
Here, the 95% CI is at around $40 WTI on the low end at $130 on the high end for year end 2026. For most investors, those bands are effectively useless. If you’re a shale operator, should you hedge your production or not? A band that wide does not help you at all.
If you were pricing, say, out-of-the-money options and believed the future would be like the past, then confidence intervals could have a place. But note, the EIA bands appear to be derived from options prices, not vice versa. Thus, the CIs appear to be forward-looking based on current market prices of oil futures options, not historical data. Thus, the probabilities (CI’s) are imputed from options prices, not the probabilities computed from historical oil markets data. These are not statistical intervals based on historical data, but rather the statistical interpretation of current market sentiment, which is something different. But perhaps I am reading it wrong.
In the case of Hurricane Maria, the Harvard Study suggested some 4500 people had died in the hurricane, while the PR mortality data, as of year end, suggested 200-400. Given that the hurricane occurred on October, if I recall correctly, this suggested a massive number of dead people had not ended up in the morgue. If that interpretation were true, you’d better get the cadaver dogs and rescuers to go out and try to find the bodies, which may be trapped in homes, under landslides or washed out to sea. So that’s an example of statistics driving action. As I recall, the lower CI limit for the Study was around 700-800, suggesting that perhaps 400 bodies were missing, as of the preliminary PR Dec. mortality report.
As it turned out, the excess mortality was on the order of 1200-1400, this the result of delayed reporting from PR authorities wrt to deaths. (Why does it take weeks to months to determine mortality in PR? I would think they should have networked computers that can track all this in real time, but apparently not.)
If you combine the data delays from PR with the CI from Study, you could have an overlap. That is, the CI encompasses the actual mortality ultimately reported by the government there. So what then is the CI worth? Does it inform action on the part of PR decision makers? Or it is a nice artefact without practical meaning? It seems the latter.
I would also add that the 4500 imputed deaths were literally 3x the actual. If you’ve blown the mean by that much, how good are your intervals? You can calculate the CI using a stats package, sure. But if your underlying data is garbage — and it was — then you have a CI no better than the flawed data on which it was premised.
So, in short, if you’re using a 95% CI, then you are including, in effect, all but ‘black swan’ events. That’s not particularly helpful for most day-to-day decision-making.
Steven Kopits Here, the 95% CI is at around $40 WTI on the low end at $130 on the high end for year end 2026. For most investors, those bands are effectively useless. If you’re a shale operator, should you hedge your production or not? A band that wide does not help you at all.
But in your example the CI does help your client make a decision. It might be a tough decision made even tougher by having more information, but that’s life. If you only reported the sample mean, then you’d be misleading your clients regarding the price risk.
Anyone can analyze policy. I do it quite a bit, often with relevant insight. Professionally, though, I would probably consider myself a strategic management consultant. My goal is to help people solve problems, not comment on policy per se, although I think writing about policy is one important pillar in the process of encouraging change.
“Anyone can analyze policy. I do it quite a bit, often with relevant insight.”
You analyze policy like a retarded monkey and a typewriter writes Shakespeare. Total gibberish is not insight. And you are the world’s worst consultant. Oh wait – you will likely come back with “do you know who I am”? I do – a moron and a liar.
‘Anyone can analyze policy.’
Anyone? Someone (like you) who is incompetent at basic statistics?
Someone who knows nothing about basic economics? Oh wait – that’s Stevie!
Someone who does not international law or even anti-trust law? That would be you again.
Any village idiot can analyze policy I guess. Exhibit A is Princeton Stupid Steve!
Restaurant scene in LA
https://www.msn.com/en-us/money/markets/the-shocking-state-of-the-restaurant-industry-we-can-t-afford-to-be-open-we-can-t-afford-to-be-closed/ar-BB1mqdgW
Here’s the situation in New York:
https://www.nytimes.com/2023/10/20/opinion/restaurant-industry-shutdown-inflation.html
“Standard restaurant logic dictates that your dining room be as big as possible, but we cut ours in half (to just 35 seats) ” If price increases, quantity demanded decreases.
“Our restaurant is walk-in only, so we don’t pay an online reservation platform or lose money on no-shows. Our tiny menu is efficient and minimizes waste.”
Decrease service levels to account for increased costs.
“It’s becoming common for us to do more than four turns on a table at night. The faster we turn tables, the stronger our sales are. ” The consumer is getting less, in this case, less time at the table.
“You might have noticed more top-rated restaurants have summoned the nerve to charge you reservation fees or for entire meals up front and more prix fixe dinners instead of the be-everything-to-everyone menus we’ve gotten so used to. You’ll probably see some offbeat service models.”
Surcharges, again, demand destruction. Smaller selection of food.
So, how is the restaurant scene doing in New York? Well, if you have enough money for reservation fees, then all is good. If you’re lower down the ladder, then you can have in-and-out pasta at the place noted above. The consumer is paying more for less. That’s the lesson.
And congestion pricing isn’t even factored in yet.
Oh brother – Stevie’s LA story has all sorts of canards. Gee prices have risen by 20% or so in the last 3 plus years. But wait nominal GDP has risen by 28% over the same period. Oh – the movie strike hurt these food establishments. Well duh – Stevie claims this is why the food workers should work for slave labor.
Hey Stevie – this is an economist blog. Your right wing trash belongs somewhere else.
You are quoting pasta chef Andrew Strong on a labor economics issue? Are you effing kidding me? BTW – Manhattan has always had small eating establishments because the landlords charge sky high rents not because workers are getting a decent wage.
And you claim you are a policy analyst? No you are a very dumb right wing MAGA moron. And that’s your best attribute.
Steven, If there is a large confidence interval, then anybody arguing their prediction is accurate is simply using the “do you know who i am approach” to defend their prediction. Because the data indicates you cannot know accurately the precise value of interest. It is not a scientific argument. It is a salesman approach to pressure a client into believing the analyst knows more than he actually does. the salesman wants you to believe he has a crystal ball. He does not. The salesman is simply gambling with other people’s money.