With commodity prices exhibiting wide fluctations over the past few years, it’s no wonder that many are interested in determining what procedure best forecasts. A recent New York Fed blog post by Jan Groen and Paolo Pesenti tackles this issue. In a horse race between various economic, time series, and futures-based approaches…
there is no obvious winner. Information from large panels of global economic variables can help, but their forecasting properties are by no means overwhelming. It all depends on the choice of the specific index and the forecasting horizon. …
…For example, for one specific commodity price index, PLS regressions provide significantly better predictions than both autoregressive and random walk benchmarks when used to forecast one-month and one-quarter-ahead commodity prices. But when the forecasting horizon is six months or longer, the forecast performance of PLS regressions is no better than the statistical benchmarks. PLS does perform relatively better with aggregate commodity price indexes than with commodity subindexes such as metals or energy.
If we focus on specific subsets of explanatory variables—as emphasized by the “true believers”—we do find some, but not overwhelming, evidence for the notion that commodity currencies are useful predictors. We find even less empirical support for the notion that commodity futures have strong predictive power.
Ultimately, the basic message is one of inconclusiveness. No easy generalization or pattern emerges, and the results look almost random. In fact, we are unable to generate forecasts that are, on average, more accurate and robust than those based on autoregressive or random walk specifications. If a policy lesson can be drawn from our results, it is that one should be very cautious when interpreting the forecast of a forthcoming commodity price surge as an early signal of recrudescence in global headline inflation. As forecasts of commodity prices provide only highly noisy hints about their actual future trajectories and persistence, excessive confidence in such forecasts may bias policymakers’ views and beliefs about future inflation risks in the direction of a premature—and unwarranted—tightening of the global policy mix.
…
The paper upon which the study is based is here.
This study provides an interesting complement to a 2010 study Oli Coibion and I wrote (post here). The Groen-Pesenti study pertained to commodity price indices, while our study pertained to spot prices of individual commodities. We concluded:
Commodity prices have long played an important role in accounting for economic fluctuations.
Forecasting changes in commodity prices is therefore an important component for forwardlooking
policy-makers. The growing use of futures markets has raised the question of how much
information these prices incorporate about future movements in spot prices. We show that while
energy futures can adequately be characterized as unbiased predictors of future spot prices, there
is much stronger evidence against the null of unbiasedness for other commodities, especially
once one explicitly takes into account time-varying heterogeneity in shocks variances. In part,
this failure of futures markets for many commodities to satisfy the unbiasedness hypothesis
likely reflects the fact that these markets suffered from only light trading volumes. In recent years, as the depth of these markets has increased, we find much weaker evidence against the
null of unbiasedness. In addition, futures prices have frequently outperformed random walk
predictions since 2003, despite substantial and protracted price changes, and vastly outperform
reduced form statistical models of price changes. This result leads us to be cautiously optimistic
about the broader use of futures prices as predictors of subsequent spot price movements,
particularly for those markets which continue to be actively used by a wide range of financial
and real market participants.
This suggests to me one reason why futures might have differing predictive power relative to other approaches, for certain commodities or commodity classes. But that’s just a conjecture.
As an aside, I find it amusing to repeat some conclusions Yin-Wong Cheung, Antonio Garcia Pascual and I came to in our 2005 JIMF paper, regarding exchange rate models:
We re-assess exchange rate prediction using a wider set of models that have been proposed
in the last decade: interest rate parity, productivity based models, and a composite specification.
The performance of these models is compared against two reference specifications e purchasing
power parity and the sticky-price monetary model. The models are estimated in
first-difference and error correction specifications, and model performance evaluated at forecast
horizons of 1, 4 and 20 quarters, using the mean squared error, direction of change
metrics, and the “consistency” test of Cheung and Chinn [1998. Integration, cointegration,
and the forecast consistency of structural exchange rate models. Journal of International
Money and Finance 17, 813-830]. Overall, model/specification/currency combinations that
work well in one period do not necessarily work well in another period.
“We show that while energy futures can adequately be characterized as unbiased predictors of future spot prices, there is much stronger evidence against the null of unbiasedness for other commodities…”
This raises the obvious question: why is unbiasedness a desirable way of measuring error in commodity markets? Or for that matter why use any symmetric error measure like MSE or MAE when evaluating over the hold-out period? For example, I recently read a paper where the author found that a biased estimate tended to generate greater returns in the stock market. As I recall the author tested some flavor of an EGARCH model (log variance and a threshold parm to correct for skew and GARCH to correct for kurtosis) with the usual GED distribution. The forecast was still slightly biased, but it outperformed other unbiased forecasts over the hold-out sample period. And as it happens, I’m also working on a technical report that found EGARCH models outperform other forecasts of Army demands if you measure errors asymmetrically where the loss function trades-off inventory holding costs against supply performance costs where backorders are estimated as a Lagrangian shadow price. That last qualification is critical. I’ve really become a big fan of asymmetric error measures…as was JDH’s former colleague Clive Granger. I found some quote from Granger on the importance of measuring errors in a way that matches how decision makers actually evaluate errors instead of automatically using textbook symmetric error measures.
“…especially once one explicitly takes into account time-varying heterogeneity in shocks variances.”
So even if you use a symmetric error measure, doesn’t this imply that the metric should be heteroskedasticity corrected (e.g., HMAE) if you’re going to test for significance?
2slugbaits: There are two reasons (at least) one could be interested in the coefficient on the basis. The first is that it is associated with a maintained hypothesis. Under the joint or composite hypothesis of rational expectations and risk neutrality, the coefficient should be unity (after taking into account cost of carry, etc.). The second is that one would want to exploit the existing correlation to make money; in that case, one would not care about the particular coefficient estimate — just that one could exploit it.
In that latter case, I think it would make sense to investigate not only alternative metrics, but whether one could make money after accounting for transactions costs.
There is a large literature regarding statistical metrics versus profit metrics. Lucio Sarno has done some recent work on this — see here — although the literature goes way back to at least Boothe and Glassman in the mid-1980s. Also, Edison, West and Cho, in a mean-variance optimization framework.
Shorter version:
The markets don’t know how to predict the weather.
Given all the recent attention, have economists and analysts made any progress in their ability to predict movements in commodity prices? In this post, we find there is no easy answer. We consider different strategies to forecast near-term commodity price inflation, but find that no particular approach is systematically more accurate and robust.
D- for intellectual honesty.
file this under the “yes and in other news the sun will also rise”
If there were a model that could accurately forecast commodities prices, then there would be a hedge fund attempting to using it (there are many attempting to make money from personal experience!). And if there were hedge funds making money then there would be other hedge funds sprouting up with former traders from the first… and then pretty soon the models would no work.
my experience is that the commodities markets are pretty efficient these days.