Data Compendium, Updated

Another update of The Data Will Set You Free , so you won’t go around quoting Zerohedge.

In an era of easily accessible databases, am constantly amazed that people write stuff that is easily falsifiable.  Or ask me for the “raw” data when it’s freely available via a aggregating database I’ve provided the hyperlink (most galling is when they then accuse me of misquoting data sources).

Just to remind the frequent commenters on this blog, freely available and documented data available here:

Historical

Surveys, Forecasts

Uncertainty Indices

High frequency and nowcasting data

Usually I cite FRED or BEA and/or BLS via FRED, or from the above data sources. In certain cases, I have written papers using specialized data sources. Below are links to those data sources.

 

55 thoughts on “Data Compendium, Updated”

  1. pgl

    Wow – CoRev has been barking like a mad dog since April 2019 or before? Your co-host got the 5-star award for best comment:

    JBH
    April 15, 2019 at 8:30 am
    Menzie: If a researcher could gather the data, probably a well-nigh prohibitively costly endeavor, might they find that the amount of fluoride in the water correlates to lack of careful reading? (Part tongue in cheek, part not.)

    1. AndrewG

      The headline would read: “Econometrics textbook author mocks buffoon”

      I think 2019 was around when the psychic declassification began. Hillary deleted all the evidence, unfortunately.

  2. AndrewG

    Now this is a *great* list of data sources!

    Some I would add for history, mostly US:

    ICPSR
    https://www.icpsr.umich.edu//web/pages/

    IPUMS (focus is on individual-level data; not data all can be accessed by the public)
    https://www.ipums.org/

    Data.gov (basically a dumping ground for US gov’t data, including EPA; not great in all honesty, but can help get that bit of data you can’t find elsewhere)
    https://data.gov/

    US Census (already mentioned; just want to highlight that they have historical stuff in quick/useful formats that the other sources above might not)
    https://www.census.gov/

    Historical data (and related research) outside the US/UK is growing very fast too. Not my wheelhouse but if you’re curious, know this stuff exists!

    .

    US geographic/climate/agricultural data:

    NHGIS (IPUMS) (mostly historical boundaries, commonly used; elsewhere there are publicly available crosswalks for harmonizing the county borders for data analysis)
    https://www.nhgis.org/

    USGS National Map (includes the Geographic Names Information System)
    https://apps.nationalmap.gov/datasets/

    NOAA National Centers for Environmental Information
    https://www.ncei.noaa.gov/

    USDA (though I think a lot of this you can find at the Census and elsewhere)
    https://www.usda.gov/content/usda-open-data-catalog

    Bureau of Land Management (mostly maps; some very specialized stuff, but it is for public consumption; includes old Land Office surveys and patents!)
    https://www.blm.gov/about/data

    .

    Miscellaneous:

    NBER Economic Indicators and Releases (really just a handy compilation of links)
    https://back.nber.org/releases/

    Gapminder (I’m not sure how active they have been recently since founder Hans Rosling passed away)
    https://www.gapminder.org/

    Wikidata (kind of hot right now; not in any way a handy compilation, but still cool)
    https://www.wikidata.org/wiki/Wikidata:Main_Page

    .

    I am a preternaturally lazy commenter, but maybe listing all these sources will get me to use them here more. And maybe you too!

  3. joseph

    What a tedious and exhausting list of sources. Why would anyone waste their time plowing through all that for the “facts” when they can have it all distilled to a couple of succinct paragraphs in Zerohedge? Alternative facts rule.

    1. AndrewG

      Entire paragraphs? What are you, special? All I need is the chyron on Fox News. OMG Hunter Biden’s laptop AGAIN!

      1. AndrewG

        Not even. Bare minimum for a MAGA dopamine hit is the disgusted expression on Tucker Carlson’s face + the word “woke” on the screen. The chyron is just gravy.

  4. Macroduck

    Off topic, More European energy stuff –

    Putting aside the various agenda-driven points of view in comments from the prior post, inquiring minds may still want to think about the economic and financial implications of Europe’s energy situation.

    Y’all realize, I’m sure, that commodity markets are highly financialized – derivatives based on commodity prices amount to many times the value of the underlying commodities. Kinda like the financialization of housing through mortgage-backed securities and derivatives. See where this is going?

    So there is a related, additional risk to Europe’s (and other) economies from energy markets beyond the budgetary, inflationary and production effects. That risk is financial, and is transmitted through energy derivatives and contracts. Price swings are one element of the risk. Delivery is a second element. Payment (cue spooky music) is another. Payment risk is a nightmare. If I don’t get paid, I can’t pay you. If you don’t get paid… And so on. That was the critical transmission mechanism for the Lehman shock.

    This risk is getting attention from European governments. Maybe they can help:

    https://news.yahoo.com/energy-trade-risks-collapsing-over-092509271.html

    Now, let’s see if grown-ups can have a conversation without some political toddler interrupting.

    1. AndrewG

      All very good points, but there’s another nuanced interpretation of the Lehman shock which Ben Bernanke doesn’t want to talk about: It was Ben Bernanke’s fault. (You may be familiar with Bernanke’s lame excuses, as documented by Larry Ball.) If they had bailed out Lehman like they did Bear, we may not have had a panic. Panics are about confidence that “deposits” are going to be safe. The solution (as I put it in my quizzes on LOLR) is “more money,” i.e., lender of last resort facilities. (I will spare you all the corny Floyd Mayweather jokes I make in that quiz that apparently today’s 20-year-olds don’t get.)

      The other component is, for “depositors”, surprise + lack of clarity about actual exposure, then using some broad heuristic to bail one’s self out of the market that is so broad that if everyone does it, the market collapses (and of course is self-fulfilling, because if everyone else does it …). This is like 2008 “depositors” jumping from funds with any exposure to real estate at all. Again, another way to look at a panic. This is Gorton/Holmstrom stuff, except leaning on the side of “more information good”.

      So as I see it the government(s) have a couple options to stop any panic: 1) be a LOLR, and 2) do some sort of auditing/stress testing like you would for banks. Governments are actually good at this – so long as they commit publicly and follow through.

      There’s a new paper in the works by Jaremski about FDR’s bank holiday – most relevant to #2 – and it’s a good one.

      Amazing how the ghosts of 2008 come back to haunt us! But the risk of panics is always with us. It’s in the nature of finance (as we’ll never be in a 100% reserve world).

      1. baffling

        “If they had bailed out Lehman like they did Bear, we may not have had a panic. ”
        I guess the question is, where does the domino stop? say they had bailed out Lehman like Bear, what happens next? I guess it is important to remember, at that point all options are suboptimal. somebody is always going to feel cheated that somebody else was saved. personally, I think if they had bailed out Lehman, then somebody else would have fallen. like Goldman. probably a more influential firm, with even more dramatic impacts. on the other hand, I think LOLR works most effectively once people truly understand what you are trying to save. if the government had stepped in as LOLR prior to Lehman, how do you think it plays out? people understood the reasons after Lehman. not sure how they would have responded if Lehman had not failed. the leverage in the system was still there either way.

        “So as I see it the government(s) have a couple options to stop any panic: 1) be a LOLR, and 2) do some sort of auditing/stress testing like you would for banks. Governments are actually good at this – so long as they commit publicly and follow through.”
        mostly agree. especially on item 2. once a firm or industry reaches a size that could become systemic to the economy, it should be treated differently.

        1. AndrewG

          “I think if they had bailed out Lehman, then somebody else would have fallen. like Goldman. probably a more influential firm, with even more dramatic impacts.”

          That’s hard to believe since Goldman didn’t fall even after Lehman was left holding the bag and we had a panic. Whatever institutions would have fallen in the counterfactual (bail out Lehman) case would have been even more likely to fall in the actual (don’t bail out Lehman) case. Even healthy institutions are threatened during a panic.

          “the leverage in the system was still there either way.”

          Yes, but the leverage isn’t what causes a panic; it’s “depositors” not knowing their true exposure conditional on there being a panic. That’s why LOLR is important. It’s as much psychological as anything. The way finance econ people talk about it (as I’m sure you know) is illiquid vs. insolvent. Panic about the insolvent can drive runs on the merely illiquid; even rumors about the illiquid can drive runs on healthy firms. That’s one reason why panics are so devastating – ex ante healthy firms fail.

          On 1) (LOLR) it could be a special discount window or other special facility to allow distressed borrowers to show up, perhaps anonymously to avoid setting off runs. Doesn’t have to be wholesale monetary policy pushed to zero or something. Just guessing here, but a special facility may allow the central bank to continue its tight policy for the rest of the economy (as might be needed in Europe today – huge hike today!).

          Good point about SIFI/firm size, but I’m not sure what to think about that stuff myself. Sink or swim is potentially playing with fire (as Lehman may teach us). But there is a (conservative) argument that treating SIFIs partially (starting in the 70’s, the story goes) is what gets us in the mess of perpetual SIFIs/TBTFs in the first place. If a firm knows it’s backed up by the government, their standards are likely to fall. But then again, I guess increased scrutiny would factor in here, as you mention. Part of the Dodd-Frank story.

          1. baffling

            first off, not arguing with you. I appreciate the discussion. this was an important episode, and we still do not have all the answers. hope we continue to learn. at least I am interested in doing so.

            “That’s hard to believe since Goldman didn’t fall even after Lehman was left holding the bag and we had a panic.”
            here is the thing. right after Lehman went bankrupt, the rest of the big banks were potentially in a boat load of trouble. Merrill Lynch was going under. AIG was the next big domino, even though it is not a bank. they were bailed out, and that undoubtedly saved Goldman and others. there was a panic, but at that time we had LOLR in action. as you noted, that helps to control the depths of the panic. personally, if the government does not step in when they did, most of the big financial institutions would have been in very serious trouble. including Goldman. but I am not picking on Goldman here, just using them as an example of a big, supposedly strong firm.

            I simply find it hard to believe we could have gotten through this episode with no big bankruptcy, even with LOLR.

            “Even healthy institutions are threatened during a panic.”
            I don’t think Goldman was healthy. Goldman was lucky. Maybe with good friends in high places? they have made me money over the years, so I do not hold a grudge against them with these comments.

            “On 1) (LOLR) it could be a special discount window or other special facility to allow distressed borrowers to show up, perhaps anonymously to avoid setting off runs”
            agreed. and we effectively established such a system eventually. this is a good lesson to apply going forward.

          2. Rick Stryker

            Baffled,

            Goldman was never in any real danger. Remember that Warren Buffet made a sizable investment in Goldman that he would have lost if Goldman failed. Buffet apparently thought Goldman was a good risk.

            One bank that was in trouble for a while in 2008 was Morgan Stanley. I think they came close but pulled it out with some adroit liquidity risk management.

          3. baffling

            rick, buffet invested after Goldman recharacterized as a bank, and aig was bailed out. why do you think Goldman took that action if they were in great shape?

            but the government appears to disagree
            https://www.reuters.com/article/goldman-aig/goldman-was-exposed-to-aig-losses-govt-report-idUSN189618120091118

            Goldman was quite exposed to aig. at the time, the controversy was why let Lehman fail (a competitor of Goldman) but rescue AIG? the conspiracy was to bail out Goldman. now I am not really buying the conspiracy theory here, I don’t that is what drove the action to bail out aig. it was bailed out because it was tbtf, and the government decided to backstop. but if you think that Goldman was not in any real danger, given their exposure to AIG, then rick you don’t understand the financial market the way you claim to on this blog.

            now Goldman has gone on the record saying they would not have been exposed if aig had failed, because they were hedged against it. in theory that may be true. but in reality, if aig had failed and defaulted on its obligations, it probably would have been the end of all the parties involved in the Goldman hedges. so Goldman would not have actually collected. of course, that did not happen because the government bailed out aig. and for good reason. it seems to me, the only really large financial institution to have avoided danger was jp morgan. and unfortunately I did not own them at the time.

          4. Rick Stryker

            Baffled,

            It’s important to be skeptical of reporting by the financial media, especially regarding the 2008 financial crisis. Read the report Reuters refers to yourself and you’ll see that Reuters is exaggerating what the report actually said. The crucial point that Reuters left out that the report acknowledged was that the hedges were also collateralized. And it was reported a few years later in the media that the counterparties that sold the hedges were quite diversified on top of that.

          5. baffling

            rick, i am not trying to bash goldman. but if you think goldman was in good health and would have been ok had the government let aig fail, like it did lehman, i got a bridge to sell you. goldman received nearly $13 BILLION of the aig bailout in counterparts payouts. how much would they have collected if the government had not bailed out aig? your guess is as good as mine. but i think it is fair to say that an aig bankruptcy would not have been healthy for goldman at all.

            one lesson we learned during that financial crisis is that there was a stunning lack of diversity in hedges. financial institutions were simply hedging with other financial institutions. everybody was simply playing with different neighbors, but all from the same neighborhood. maybe that is simply the way it must be in finance. but it means diversity is not all its cracked up to be.

          6. Rick Stryker

            Baffled,

            You do not properly appreciate the importance of having the hedges collateralized. That reduces the risk very substantially and diversification of hedge providers decreases the risk even more. If you want me to explain more, I’m happy to. But I’m not interested in having a pointless debate.

          7. baffling

            rick, i understand that goldman received $13 BILLION in counterparty payouts, as part of the AIG bailout. that was the largest payout for any company. i also understand that diversifying your hedges amongst a bunch of other financial firms that are subject to the same extreme financial event is not really diversification.

          8. AndrewG

            @baffling No, no argument here! I very much appreciate your perspective.

            In my mind, the key is that panics fundamentally change the game. LOLR doesn’t make things better at the depth of the panic: it’s designed to prevent one in the first place. (If you’re at the depth of the panic, you’ve missed the whole game.) *Not* bailing out Lehman was a sign that the government was *not* going to be a lender of last resort (which isn’t just about the Fed pumping “liquidity” into the economy). So, as the literature goes, you have to talk about ex ante and ex post solvency. Goldman may well have been ex ante solvent, but ex post insolvent *due to the panic*.

            There’s another, another, another perspective, which is that bailing out Bear actually made the moral hazard problem *worse*, driving up risky practices. The argument here is that the government *shouldn’t* have bailed out Bear! There is a story of a leading financial economist at Jackson Hole a couple years after the crisis making exactly this argument – and getting audible groans from the audience, which included Ben Bernanke himself!

            Personally I don’t know where I stand on all of this SIFI/TBTF/LOLR policy stuff. Not my wheelhouse anyway. But I am sure that panics are not truly functions of true insolvency – the true health of a financial firm. They are functions of sentiment and game theory. Even rumors can set off a panic. An example of this? The euro crisis. Both countries with crappy fiscal policy (Greece) and great fiscal policy (Spain) got hammered, at least until Draghi’s famous utterance. De Grauwe is a great read on the euro panic.

          9. baffling

            “There’s another, another, another perspective, which is that bailing out Bear actually made the moral hazard problem *worse*, driving up risky practices. The argument here is that the government *shouldn’t* have bailed out Bear! ”

            Andrew, my thinking is that there needs to be a failure and there needs to be a bailout, in that order. if bailouts occur without at least one failure, the moral hazard issue is true. but a failure will require a bailout, when we have something as significant as the financial crisis. too many financial firms were actually quite unhealthy, and something was going to happen. so the reality is, the government is going to have to do some picking of winners and losers. that is the tricky part. perhaps letting bear fail would have been better, followed by a bailout of Lehman (or whoever fell next). but bear was smaller, and perhaps they naively thought it was a small price to pay to preserve the economy. the problem was simply bigger than they realized. aig was the big problem at the time. we knew it was too big to let fail. Lehman seemed to check all the right boxes. I have often wondered what would have happened had bear failed first, or we bailed out bear and Lehman, and then what? hypotheticals, of course, and not sure if it change the overall outcome. but interesting to think about.

            and I do agree with much of what you say about panics. so they question becomes, how do you scare people just enough to change their ways, without inducing a panic? glad I am not in charge of that decision making.

  5. Macroduck

    You’ll notice that European high-yield interest rate spreads are wider than U.S. spreads for the first time since 2013:

    https://fred.stlouisfed.org/graph/?g=TuHP

    The ECB is about half-way through a review of bank’s preparedness for interest rate and credit spread shocks:

    https://www.bankingsupervision.europa.eu/press/publications/newsletter/2022/html/ssm.nl220817_1.en.html

    Yeah, credit spreads and energy-sector margin calls aren’t the only source of financial stress. The ECB and other central banks are causing ripples, too. Makes it more expensive to finance derivative positions. Good thing governments know not to wait for the results of this review to take action.

  6. Macroduck

    Off topic, Financing Denialism –

    Several member of our local troll choir are climate-change deniers. They aren’t necessarily overt deniers – that’s not how “fake science” tactics work. They just throw dust in the air whenever the issue of climate change comes up and hope that doubt and confusion will do the rest. Calling into question other commenters’ or even our host’s understanding of issues, when the deniers have themselves demostrated no real grasp of the science, insisting on “weather, not climate” in the oddest of situations – you’ve seen it.

    Makes ya wonder who pays for this dreck. Well, have you ever heard of the Hearland Institute?:

    https://www.propublica.org/article/barre-seid-heartland-institute-hillsdale-college-gmu

    1. pgl

      Seid has funded climate denialism as well as a national network of state-level think tanks that promote business deregulation and fight Medicaid expansion. He’s also supported efforts to remake the higher education system in a conservative mold, including to turn one of the nation’s most politically influential law schools into a training ground for future generations of right-wing judges and justices.

      It figures that the same person who has funded CoRev’s annoying barking also wants women to be nothing more than baby machines.

  7. Anonymous

    data is record of past.

    the future is mostly independent of the past.

    as yogi berra the great yankee once said

    ‘predictions are hard, especially about th future’

    that said fred has something for everyone!

    1. Macroduck

      “the future is mostly independent of the past.”

      Howdy Do! That’s the most profound thing I have ever read.

      So the borders of the Great State of Pennsylvania today are mostly independent of what they were yesterday? Spending today is mostly independent of the bisget adopted yesterday? Birth is mostly independent of pregnancy? Russia invaded Ukraine early this year, but that really has little to do with conditions in Ukraine today?

      It’s trippy when the “just make stuff up” crowd decides to go Big Picture.

      2. AndrewG

        As far as I’m concerned, all the births I’ve been maliciously associated with are completely independent of whatever pregnancy-related action occurred before. Correlation does not equal causation. You see, Yogi Berra once said …

    2. AndrewG

      “the future is mostly independent of the past.”

      Have you heard of out-of-sample predictions? They’re a thing. One of the defining features of our current world. You’re not saying anything wise, just confused and pointlessly fatalistic.

      1. Noneconomist

        You have to like “data is record of past” followed by “the future is mostly independent of the past.”
        That means we have no data on Aaron Judge’s performance in games over the coming weekend! Who knew?
        Even worse, although he has 55 home runs, we have no idea if he’s even capable of hitting another.
        Thanks, anonymous. You’ve altered my world view considerably.

        1. AndrewG

          Baseball, borders, and births. All proof that the past contains little or no actionable intelligence on any paternity lawsuits I may or may not be subject to at present.

  8. Ivan

    With Russian troops having been moved south to defend positions in the Kherson area, Ukraine just took back a 400 square kilometer chunk of land in the north in Karkiv oblast.

    https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-september-7

    The critical Russian distribution node of Izium is now in real danger. I doubt Putins deadline for occupation of all of Donetsk in a week will be fulfilled. Will Russian generals start flying out of windows, or will Putin yet again extend the deadline ?

          2. Moses Herzog

            I always thought of Econned as more of a Bobby Riggs type. Or possibly Russell Westbrook??
            https://theathletic.com/3324047/2022/05/24/russell-westbrook-lakers-offseason/

            https://finance.yahoo.com/news/russell-westbrook-longtime-agent-thad-162742096.html

            https://finance.yahoo.com/news/appetite-league-wide-russell-westbrook-221130992.html

            Or possibly……. Trae Young “the leader in assists and points”. People, guess what you have to do to be the leader in assists and points?? Think hard now:
            https://twitter.com/TheTraeYoung/status/1540807198099619841

            Trae Young will NEVER win an NBA title, unless he is a 41 year old coming in off the bench~~and frankly I doubt the latter would happen either. You can’t win a tittle when you are a two guard who bullies his coaches/team into letting him play point guard. Two guard and point guard are different roles on a team. And if the two guard forces/bullies his way into being point guard, and that point you’ve lost both ball rotation and two of your five players become useless Which is why NO TEAM wants age 33 Westbrook’s contract, and in not many years, the same will happen to Trae Young. He’s useless as a teammate.

      1. baffling

        yes econned, ie j zanfino, is basically a troll and a jerk. all wrapped up in one. highly accomplished. why he is so jealous of prof. chinn is a mystery. perhaps he is a former student who do not do well in the class? or a failed academic? not sure. but a sad story nonetheless. apparently he has an issue with the posting of economic data sources on an economics blog.

        1. SportsEconnedDatabase’d

          That’s weird. Where does Menzie state this is a posting of “economic data sources”? Also, I honestly thought the comment section of this blog was primarily used for off-topic ramblings. But, of course, I’m not off topic. Seems it is you who has an issue with sports databases.

          2. DavidEconnedChappelle

            Menzie Chinn,
            Sentient humans above the age of 5 years old who have experienced even the slightest bit of social interactions often use what is called “humor” (like making comments that are so obviously jokes that even a book smart academic, above the age of 5, should have the ability to grasp).

          3. Menzie Chinn Post author

            DaveEconnedChappelle aka Econned: Please excuse me. Since you seemed like the most humorless person I’ve ever encountered, I was surprised by your attempt at humor. Apologies.

          4. baffling

            “I honestly thought”
            j zanfino, as soon as you start thinking, that is when you should stop. it only makes you look like a fool.

          5. AndrewG

            It was kind of hard to interpret your humor here.

            Now I get it and I chuckle – but not before. I have learned from the past!

          7. baffling

            zanfino, your lack of friends may be related to your lack of quality humor. you cannot seem to pull it off, without sounding like a prude. maybe you should work on that aspect of life improvement. just a helpful hint from somebody looking to help the less fortunate.

          8. Econned

            baffling,
            It’s you who is getting bent out of shape over the comments of a random entity on the internet. Self-reflection would do you well.

Comments are closed.