Authored by Salil Mehta via Statistical Ideas blog,
There is a 2/3 chance that both competing Federal Reserve 2017 Q1 GDP nowcasts are wrong! That’s an audacious prediction for the storied NY and Atlanta institutions (one of them led by my former big boss Timothy Geithner), and yet there is no way around the current confusion they are in.
This is also critically important as one is showing a robust 3.2% growth reading, while the other is at 0.9% (the 2nd lowest reading in nearly 3-years) and essentially indicates that we are descending towards recession.
[NOTE - NOWCAST just downgraded their forecast dramatically from 3.19% to 2.83%]
Are we descending towards recession? It"s unlikely but zero-growth is certainly in the cards and not reflected by these two nowcasts, and we certainly think there is only a single digit probability of a >3% GDP. How could the NY Fed plausibly give such a madly high estimate (which if true would be the second highest in 2-years)? Yet there you have it, two extreme readings, and a 2.3% (3.2%-0.9%) chasm between them. We show here that the Federal Reserve’s conclusions are somewhat ridiculous, though shouldn’t be since they impact the open market committee monetary decisions that the world looks to. And there are humbling lessons from these nascent Big Data, overfit models.
The chart here shows some basic information regarding the current GDP nowcasts. As we via the two blue bars, we have the Atlanta nowcast on the left (the bar was recently as high as 3.4% earlier this year). And the NY nowcast on the right (the bar was recently as low as 1.5%). That’s right, both nowcasts passed each other, while aggressively moving further in the opposite direction! The large swings in each are also doubtful, given each nowcast’s eventually advertised, margin of error (concordant paradigm in various forecasts by Taleb). For a good chronology of these nowcast reports, refer to MishTalk.
Each nowcast boasts a margin of error of just ~1%, and this clearly poses an issue since the average of these two nowcasts (shown in orange at 2.1%) is clearly outside of both the Atlanta and the NY stated margin of error! As supportive reference, we also show (in green) that the current 2016 Q4 GDP is nearby at 1.9%. Now we should ask some important questions about how we keep getting into more strange nowcasts in the past year that they both have operated. The first thing to appreciate is that the nowcasts are supposed to predict very tight errors that are uncorrelated to the variance in the actual GDP itself. And good nowcasts should have errors independent of one another, except since the NY and Atlanta Fed operate independent of one another there is a good chance that there may be some modeling similarities. We modestly assume this and derive through the variance formula (VarianceAtlanta+VarianceNY+2?Atlanta?NY?Atlanta,NY) that the margin of error of the difference between the models is just less than 1% (silver vertical interval arrows in chart above). This is a highly plausible tight expected variance. Sample size is also trivial here as we don"t have the true expectation to model a limit from. And with this, the probability of seeing an inadvertent 2.3% difference between the two correct Federal Reserve models is <5%. Or that their publicized margin of error is awkwardly too low (to the point we’ll show later that randomly guessing the GDP would be safer).
We also have the probability that one of the nowcasts is correct, which due to symmetry means applying one of the nowcast stated margin of errors to the other nowcast value. Or that the one nowcast is unintentionally correct, which would be like the <5% probability above. So, in total the probability of one of the Federal Reserve nowcasts being correct and the other being wrong is ~10% (not the >½ that many may generously assume as they critique these divergent outputs).
That leaves us with two other possible outcomes still! That is to split up the remaining 85% probability that both models are individually wrong into: (a) the average of the two models is still correct, and (b) even the average of both models is wrong. This leaves us with no choice but to conclude that the probability that there is ~40% probability that the correct 2016 Q1 GDP is nowhere near either nowcasts nor the average of the two, and a >½ probability that the GDP is near the 2.1% average that inappropriately happens to be well outside both two nowcasts’ margin of error. And between those two we can safely claim that there is a 2/3 chance that both models are total wrong (and merely <5% chance they are still both right). There is perhaps a 30% chance one can smartly use both models in a deliberate way, though this is conditional on how they use the information and not at their endorsed specified face value.
Now a more practical assumption of this is that the margins of error should be more than doubled (to 2.6%!), in which case no one would even use such nowcast models. However, the probability breakdown in such a scenario is this:
<30% chance both models -with their current 2.3% chasm- are correct
~60% chance one of the models is wrong and one is correct
~10% chance both models are still wrong individually, though the average in rare cases is correct
In all cases, all three probabilities sum to 100% as they should. And they lend a healthy sense of respect that incessantly observing each of these GDP nowcasts is commonly a waste of time, and that rarely will one gain insight from it other than from ex post luck. It’s the same as the more senior open market committee models vainly attempt to forecast other macro-economic variables. Sometimes simply looking at the most recent quarter’s GDP (in this case ~2%) is as good of a guess as any. So is giving a little more weight to the near-0 probability most have of an outright contraction this quarter. As a business CEO, one would want to be arranged for anything at this point, which a genuine 2.6% margin of error about GDP infers.
