In 1996, Yale economist Robert Shiller looked around, considered the historical record, and concluded that the US stock market was overvalued. In the past, whenever price-earnings ratios were high, future long-run stock returns were low. But now prices on the broad index of the S&P 500 stood at 29 times the average of the past decade's earnings.
On the basis of econometric regression analyses carried out by Shiller and Harvard University's John Campbell, Shiller predicted in 1996 that the S&P 500 would be a bad investment over the next decade. In the decade up to January 2006, he argued, the real value of the S&P 500 would fall. Even including dividends, his estimate of the likely inflation-adjusted returns to investors holding the S&P 500 was zero -- far below the roughly 6 percent annual real return that we have come to think of as typical for the US stock market.
Shiller's arguments were compelling. They persuaded Alan Greenspan to give his famous "irrational exuberance" speech at the American Enterprise Institute in December 1996. They certainly convinced me, too.
But Shiller was wrong. Unless the US stock market collapses before the end of this month, the past decade will have seen it offer returns that are slightly higher than the historical averages -- and much, much greater than zero. Those who invested and reinvested their money in the US stock market over the past decade have nearly doubled it, even after taking account of inflation.
Why was Shiller wrong? We can point to three factors, each of which can take roughly one-third of the credit for annual real returns of 6 percent, rather than zero, over the past decade:
The high-tech revolutions behind the very real "new economy," which have accelerated US companies' productivity growth.
Shifts in the distribution of income away from labor and toward capital, which have boosted corporate profits as a share of production.
Increasing risk tolerance on the part of stock market investors, which appears to have raised long-run price-earnings ratios by around 20 percent.
None of these three factors was obvious in 1996 (although there were signs of the first and inklings of the third for those smart or lucky enough to read them). In 1996, betting on Shiller's regression analyses was a reasonable and perhaps intelligent thing to do. But it was also an overwhelmingly risky thing to do, as anyone who followed the portfolio strategy implicit in Shiller's analysis learned.
This takes nothing away from the importance of the question that Shiller addressed. Just why is it that stock markets around the world are subject to fits of "irrational exuberance" and "excessive pessimism?" Why don't rational and informed investors take more steps to bet heavily on fundamentals and against the enthusiasms of the uninformed crowd?
The past decade offers us two reasons. First -- if we grant that Shiller's regression analyses correctly identified long-run fundamentals a decade ago -- betting on fundamentals for the long term is overwhelmingly risky. Lots of good news can happen over a decade, enough to bankrupt an even slightly leveraged bear when stock prices look high; and lots of bad news can happen, enough to bankrupt an even slightly leveraged bull when stock prices look low.
Thus, even in extreme situations -- like the peak of the dotcom bubble in late 1999 and early 2000 -- it is very difficult even for those who believe that they know what fundamental values are to make large long-run bets on them. It is even more difficult for those who claim to know this and who want to make large long-run contrarian bets to convince others to trust them with their money.