Skip to content
The Best Interest » The Needle in the Haystack

The Needle in the Haystack

One of the first lessons in long-term investing involves the challenge of picking single stocks. We’re told to avoid searching for the needle in the haystack. Instead, just buy the whole haystack!

Because it’s amazing just how rare the needles are.

This study by Arizona State professor Henrik Bessembinder perfectly illustrates the rarity of needles. Below we’ll cover the highlights of the study.

Most Stocks Can’t Beat Treasury Bonds

We recently discussed the “risk-free rate” here on The Best Interest. Short-term U.S. Treasury notes provide that risk-free rate, which is why T-bonds are frequently used as a benchmark against which other investments are measured.

How do individual stocks compare against T-notes?, asked Professor Bessembinder.

Not well.

Since 1926, 4 out of every 7 stocks underperformed T-notes. The majority of stocks lose to the risk-free rate, despite stocks having significantly more risk than bonds. Yuck. And there were 25,300 different stocks included in Professor Bessembinder’s study, stretching from 1926 to 2016.

If you want to outperform the risk-free rate, picking individual stocks is a hard way to do so.

How Many Winners Are Needed to Make Up for the Losers?

We now know 4 out of 7 stocks (or about 57%) have historically underperformed the risk-free rate. If we owned all of those underperformers, our portfolio would look awful!

So let’s start adding in some winners to get back to even. How many winners do we need to balance the losers and achieve the risk-free rate of return?

Answer: we need the next best 39% of stocks. The bottom 57% of the stock market plus the next best 39% of the stock market would have provided the same return as the risk-free rate.

From St Louis Trust Company – Most stocks underperform the market!

That’s 96% of the total market. You could have owned 96% of the stock market (the bottom 57% plus the next 39%) for the past 100 years, and only received the same return as someone holding zero-risk Treasuries.

But Where Does Stock Outperformance Come From?

Despite these sobering stats, we know that stocks have crushed bonds over the long run. But so far in this article, we’re not seeing it!

Turns out, the top 4% of all stocks account for all stock outperformance in the past 100 years. If you didn’t own that 4% sliver, you missed out on the magic of the stock market.

That 4% is the needle. The other 96% is hay (most of which is rancid). The graph below is for illustrative purposes only. In short, it shows:

  • the risk-free rate, in dotted orange
  • the 57% of underperformers in red
  • the 39% of overperformers, in blue, needed to counteract the underperformers
  • and finally, the 4% of “super stocks” from which all net outperformance comes from

Just Focus on the 4%, Then!

If only it was easy to identify the 4% of stocks that are needles, or at least that other ~39% of stocks that are better than risk-free.

But it’s not easy.

Academic studies of mutual fund performance (a.k.a. professional stock pickers) show that only about 1/3 of fund performance show repeatability, or skill. The rest is luck. Most professionals, in other words, don’t discover needles. They stumble upon them. And stumbling upon a 4% needle – a 1-in-25 stock – is improbable in and of itself.

The only way to reliably stuff your portfolio full of valuable needles is to buy the entire haystack. Don’t try to find needles. Don’t hope you stumble upon them. Instead, you need to guarantee you own all the needles by buying a small sliver of every stock out there.

Sure, buying the whole haystack means you’ll be buying that bottom 96% of stocks too, with returns only as good as the risk-free rate. But you’re certain to capture the sought-after 4% of stocks providing all the outperformance of the stock market.

How Hard Could It Be, Though?

Commenting on Bessembinder’s study, Jason Zweig wrote:

Traditional advice on diversification says you should own at least 15 to 30 stocks in order to reduce your risk. But, in a sequel to his new research paper, Prof. Bessembinder found that a portfolio of 25 stocks still has a 64% chance of underperforming the total market. Superstocks are so scarce that you need to hold hundreds, even thousands, of companies to be near-certain of matching the market’s return.

Jason Zweig

Small portfolios miss too many needles. The odds are stacked against you. You need to own hundreds or thousands of stocks to ensure you own the needles.

But by the time you hold hundreds or thousands of stocks, you’re essentially building your own index fund. Why bother going through the time and effort to do so when you can simply buy an index fund with a single click?

This, in short, is the argument behind index investing – especially in more efficient markets, like the U.S. stock market.

The stock market has few needles. Missing out on them negates the purpose of stock investing in the first place. Your best bet – literally – is buying the whole haystack.

Thank you for reading! If you enjoyed this article, join 8000+ subscribers who read my 2-minute weekly email, where I send you links to the smartest financial content I find online every week.


Want to learn more about The Best Interest’s back story? Read here.

Looking for a great personal finance book, podcast, or other recommendation? Check out my favorites.

Was this post worth sharing? Click the buttons below to share!

Leave a Reply