Ever since I read the hysterically incorrect interpretation of a confidence interval from a person who purports to be a policy analyst, I’ve been looking for a succint explanation from a statistician, as a handy reference. Here it is (h/t David Giles via Mark Thoma):
The specific 95 % confidence interval presented by a study has a 95 % chance of containing the true effect size. No! A reported confidence interval is a range between two numbers. The frequency with which an observed interval (e.g., 0.72–2.88) contains the true effect is either 100 % if the true effect is within the interval or 0 % if not; the 95 % refers only to how often 95 % confidence intervals computed from very many studies would contain the true size if all the assumptions used to compute the intervals were correct. It is possible to compute an interval that can be interpreted as having 95 % probability of containing the true value; nonetheless, such computations require not only the assumptions used to compute the confidence interval, but also further
Source: Greenland et al. (2016).
assumptions about the size of effects in the model. These further assumptions are summarized in what is called a prior distribution, and the resulting intervals are usually called Bayesian posterior (or credible) intervals to distinguish them from confidence intervals.
Here is another simplified version: https://www.statista.com/statistics-glossary/definition/328/confidence_level/
Bruce Hall I didn’t find the statista.com definition particularly helpful. Confidence intervals are all about the percent coverage of many separate individual interval or range estimates. The problem with the statista.com definition is that it makes it sound like 95% of the point estimates of the mean are within a certain range. It would be closer to the mark to say that 95% of many, many sample intervals around the sample means also cover the true population mean.
Statista was founded in 2007 by a couple of Germans to sell data. Anyone who thinks it is the guru source for actually understanding statistical analysis has to be kidding himself. Alas this is par for the course for Bruce Hall. I’d go with Dave Giles any day!
pgl I’m not disputing that the explanation is Statista is not totally precise. In fact, the disclaimer on the bottom of the page says:
Please note that the definitions in our statistics encyclopedia are simplified explanations of terms. Our goal is to make the definitions accessible for a broad audience; thus it is possible that some definitions do not adhere entirely to scientific standards.
Perhaps this https://www.khanacademy.org/math/ap-statistics/estimating-confidence-ap/introduction-confidence-intervals/v/confidence-intervals-and-margin-of-error would make you feel better. And there are related videos as well.
There is a problem here: in the drive to be absolutely technically correct, there is often the result that nothing gets communicated well and the absolute technically correct explanation simply gets ignored by the listener/reader. Try using the Greenland et al. in a presentation to a politician or a CEO and the best you can communicate is the old standard “Trust me, I’m from the economics department”. You’re simply going to get massive eye rolls. The same might be said of a scientist trying to explain the intricacies of CRISPR to an accountant.
So, if someone says to that politician that they have 95% (or high degree of) confidence that the response or average value of a sample is within the “likely expected range” of responses if everyone or everything were included, that politician will understand that:
• the sample is directional, not perfect
• the sample’s likely result still has room for error
• if we ask people questions to get our likely outcome, we might be asking the wrong people and the wrong questions, but the sample is directional.
It is kinda a mouthful isn’t it?? Maybe people should stay away from verbal explanations of it unless they are teaching a class or tutoring, and have the textbook definition handy, because unless you do the statistics relatively often, it’s probably relatively hard to spit it out correctly or thoroughly.
Correct me if I’m wrong, but a standard deviation is a relatively similar concept. If you told the average person who had never taken college level math what a standard deviation was in only verbal form, first could you get it right, and 2nd, if you got it right would they understand it after you said it?? Without a whiteboard handy etc, it’s best to stay away from these things in a TV interview or in casual conversation unless you think the audience you’re speaking to is well-educated.
Walk up to random people on the street (not on a college campus, maybe at an upscale shopping mall, upscale would be more educated than Wal-Mart right??) and ask them to tell you what two standard deviations from the mean is. Without repositioning the words you just told them in that question/sentence, 9 out of 10 are going to have a hell of a time doing it.
Bill Gross is an interesting cat. I strongly suspect he has always been a different kind of cat, and no, I don’t mean that in a flattering way. But, who knows???—maybe at one time he was a good guy and the accumulation of mass amounts of money changed him over the years. I assume his marriage was bad long before the filing of divorce, and some documents seem to point in that being a correct assumption. Be that is it may, this seemed to escape the news cycle this week (or at least escaped my radar), so I thought there are few who can say it as well as Tracy Alloway can, so I encourage you to fit this into your weekend reading or quiet time:
http://www.tracy-alloway.com/?p=865
Bill Gross, in better days, slightly simpler times, with a mustache:
https://youtu.be/i9GsiIQ6jUc?t=611
Context please? “Ever since I read the hysterically incorrect interpretation of a confidence interval from a person who purports to be a policy analyst”.
Could you link to this “intrepretation” so we see what prompted this?
pgl: I have to hunt it down again, but you have the usual suspects…
People who want to talk about probability of parameter being in a certain interval should talk in terms of credible intervals.
However, then you have to tell them about priors. And imagining how people can misuse priors is a truly scary nightmare – they don’t really change their opinions in light of the incoming data even without using priors to justify that