EP 191: Charley Ellis on Why Most Investors Fail (And What to Do Instead)

by Peter Lazaroff | Feb 12, 2025 | Podcast

Watch Now

Listen Now

Charley Ellis has spent decades studying investment markets, advising institutional investors, and writing some of the most influential books on investing. In this episode of The Long Term Investor, we dive deep into his investment philosophy, why active investing has become a loser’s game, and the behavioral pitfalls that cost investors dearly. If you’ve ever wondered why outperforming the market is so difficult—or what to do instead—this conversation is a must-listen.

Most investors believe they can beat the market—but what if the smartest move is to stop trying altogether?

In this episode of The Long Term Investor, I sit down with legendary investor Charley Ellis, author of Winning the Loser’s Game and Rethinking Investing, to discuss why active investing is a losing battle and how individual investors can build wealth more effectively.

Charley has spent decades advising institutional investors and analyzing the financial markets, and his research leads to a simple but powerful conclusion: most investors fail not because they lack intelligence, but because they make costly, preventable mistakes.

Here are notes from my conversation…

Sign up for my newsletter so you can easily reply to my emails with your thoughts or questions for the podcast:

Why Most Investors Lose the Investing Game (02:30)

Investing is often framed as a game of intelligence—where the sharpest minds and savviest strategies lead to the biggest wins. But what if success isn’t about brilliance at all?

According to Charley Ellis, legendary investor and author of Winning the Loser’s Game, today’s market is so competitive that investing has become a “loser’s game”—one where most participants are destined to underperform.

To explain why, Ellis draws a surprising parallel to amateur tennis. In professional tennis, players win points by hitting precision shots their opponents can’t return. But in amateur tennis, most points aren’t won—they’re lost. Players hit balls out of bounds, double fault, or make other unforced errors. The key to winning isn’t playing aggressively—it’s minimizing mistakes.

Ellis argues that investing works the same way. Most investors don’t underperform because they lack intelligence or effort, but because they make preventable mistakes:

Trading too frequently, leading to high fees and taxes.
Reacting emotionally to market swings, buying high and selling low.
Chasing past performance, assuming yesterday’s winners will continue rising.
Paying excessive fees for strategies that rarely outperform.

The best investors don’t try to outmaneuver the competition—they focus on avoiding costly errors and staying invested.

“If you want to win the match, just keep the ball in play and let the other person make mistakes.” — Charley Ellis

For decades, the financial industry has promoted active stock picking and market timing as the path to financial success. But the investing landscape has changed dramatically, making these old strategies far less effective.

The Evolution of Investing: Why Active Management No Longer Works (04:45)

For decades, the financial industry sold the idea that with enough skill, research, and effort, you could consistently beat the market. And for a time, this was true.

In the mid-20th century, investing was a winner’s game—one where skilled professionals had a significant edge over the competition. Back then, 90% of stock trading was done by individual investors, many of whom had limited information and little financial expertise. These amateur investors made frequent mistakes, creating mispricings that Wall Street professionals could easily exploit.

During this era, investment professionals had clear advantages:

Insider Access – They could call corporate executives and gather information before it was widely available.
Exclusive Research – They had access to detailed financial reports that the public rarely saw.
Limited Competition – The majority of market participants were inexperienced, making it easy for professionals to find mispriced stocks and profit.

Back then, beating the market wasn’t just possible—it was relatively easy for those with the right tools and connections.

How the Game Changed: The Rise of Professional Investors

Fast forward to today, and the investing landscape has completely transformed. What was once a winner’s game has become a loser’s game, where even the most skilled professionals struggle to outperform.

Ellis outlines several key shifts that have made active investing far less effective:

1. The Market Is Now Dominated by Professionals

Decades ago, individual investors made up most of the market. Today, over 90% of trading is driven by institutions—hedge funds, mutual funds, and algorithmic traders.
These players aren’t making amateur mistakes. They are highly trained, well-funded, and have access to cutting-edge technology.
Competing against these professionals is like trying to outplay an NBA team in a pickup game—you’re facing elite competition at every turn.

2. Technology Has Erased Information Advantages

In the past, early access to company data gave professionals an edge.
Today, SEC regulations ensure that all material information is released publicly at the same time, eliminating private insights.
AI-driven research, Bloomberg Terminals, and real-time data allow investors worldwide to process new information instantly—so by the time an opportunity looks obvious, the market has already adjusted.

3. High-Speed Trading and Quantitative Models Have Raised the Bar

Decades ago, traders used slide rules to make calculations.
Now, algorithmic hedge funds use machine learning and predictive models to trade in microseconds.
The moment a stock appears undervalued, quant funds have already exploited the inefficiency—leaving little room for human investors to act.

4. Globalization Has Increased Market Efficiency

In the past, most investors focused on domestic stocks, leaving many inefficiencies in international markets.
Today, institutional investors analyze every asset class, in every market, across the globe, ensuring that no mispricing lasts for long.

5. The Cost of Competing Has Skyrocketed

Outperforming the market today requires massive resources—top-tier talent, state-of-the-art technology, and research budgets that cost millions, if not billions, of dollars.
For an individual investor, trying to compete against Wall Street’s best is like racing against Formula 1 cars on a bicycle.

Why Beating the Market Is Nearly Impossible

Ellis acknowledges that while markets aren’t perfectly efficient, any mispricings that do exist are small, fleeting, and nearly impossible to exploit consistently.

Even professional investors, armed with high-frequency trading models, PhDs in finance, and supercomputers, struggle to outperform.

“Markets today are so endowed with information, talent, and competition that prices, while not always perfect, are so close to correct that your chances of beating them over time are minuscule.” — Charley Ellis

A handful of elite investors might still succeed. But for the vast majority—even among hedge funds and mutual funds—trying to beat the market is a losing game.

So, if even Wall Street’s best can’t reliably win at active management, what should individual investors do?

The answer lies in playing a different game altogether—one that doesn’t require predicting markets or outsmarting professionals.

Why Most Prices Are (Almost) Always Right–And Why That Matters (12:30)

One of the biggest myths in investing is that markets are full of hidden opportunities waiting to be exploited. Many investors believe they can spot an undervalued stock before the rest of the market catches on, earning easy profits.

But here’s the reality: if you think you’ve found a bargain, the market has likely already priced it in.

Charley Ellis explains that modern markets are so efficient that by the time an individual investor identifies what seems like an obvious opportunity, professionals have already reacted—adjusting prices almost instantly.

The stock market isn’t a wild guessing game—it’s a battleground where millions of professionals, hedge funds, and AI-powered algorithms are constantly processing new information. The moment fresh data becomes available, these market participants adjust stock prices accordingly, making it incredibly difficult for anyone—especially individual investors—to consistently outguess the market.

The Reality of Market Efficiency

Ellis doesn’t claim that markets are perfectly priced at all times—only that they are efficient enough to make beating them nearly impossible. Here’s why:

1. Markets Process Information Instantly

Every stock price reflects the collective knowledge of millions of investors analyzing earnings reports, economic data, and global events.
Unlike decades ago, when some traders had access to private information, today’s strict SEC regulations require companies to disclose all material data to the public at the same time.
The moment new information emerges, traders react immediately, updating prices in real time.

2. Technology Has Eliminated Most Advantages

In the past, an investor who dug deep into financial reports or made calls to executives might have gained an edge.
Today, sophisticated hedge funds and algorithmic trading firms use AI, machine learning, and predictive analytics to process data and execute trades in milliseconds.
By the time an individual investor spots an opportunity, the market has likely already adjusted, leaving no room for easy gains.

3. If a Stock Looks Cheap, It’s Probably for a Reason

Many investors assume they’ve found a “hidden gem” when they see a stock trading at a discount.
But in an efficient market, low prices usually reflect real risks—such as declining earnings, poor management, or industry challenges.
The market isn’t always right, but it’s right enough that most attempts to outguess it fail.

Why Even the Best Investors Struggle to Outperform

If markets are so efficient, what happens to professional investors who try to beat them? Most fail.

Studies show that over 80-90% of actively managed funds underperform their benchmarks over long periods.
Even the rare funds that outperform in one decade rarely repeat their success in the next.
The competition is too intense, the costs of active management are too high, and the margin for error is too small.

“Markets today are so endowed with information, talent, and competition that prices, while not always perfect, are so close to correct that your chances of beating them over time are minuscule.” — Charley Ellis

Since the vast majority of active investors fail, Ellis argues that the smartest strategy isn’t to try to win—it’s to avoid losing. Instead of chasing outperformance, successful investors focus on what they can control:

Lowering costs – High fees eat away at returns. Choosing low-cost index funds gives investors an immediate advantage.
Avoiding unnecessary trading – Frequent buying and selling leads to poor timing decisions and tax inefficiencies.
Diversifying wisely – Holding a broad mix of assets reduces risk and improves long-term success.
Staying disciplined – The biggest mistake investors make is letting emotions drive their decisions.

Rather than trying to predict what will happen next, the best investors embrace a long-term strategy based on patience, discipline, and avoiding unnecessary risks.

The smartest investors don’t waste time hunting for mispricings—they recognize that the real opportunity isn’t beating the market, but owning it. Instead of chasing illusions of undervalued stocks, they focus on low-cost index investing, diversification, and staying the course.

The Smartest Investors Play a Different Game (17:50)

After decades of research, Charley Ellis has come to a simple but powerful conclusion: trying to beat the market is a losing battle. The financial industry is filled with highly skilled professionals, armed with advanced technology, working around the clock to exploit every possible inefficiency. Even they struggle to outperform consistently—so what chance does the average investor have?

Rather than playing a game they’re unlikely to win, the smartest investors play a different game altogether—one that doesn’t require predicting markets or outsmarting professionals. Instead of trying to make brilliant investment moves, they focus on avoiding costly mistakes and letting the power of compounding do the work.

What Playing a Different Game Really Means

Wall Street thrives on the idea that smart investors must always be doing something—picking stocks, timing trades, chasing the next hot trend. But the real secret to investing success isn’t about action—it’s about discipline.

The best investors understand that success isn’t about outthinking the market; it’s about tilting the odds in your favor by controlling what you can. That means:

Lowering Costs: The One Sure Way to Improve Returns

High fees erode returns. Every dollar paid in fees is a dollar that won’t compound over time.
Research shows that low-cost index funds consistently outperform most actively managed funds simply by keeping more money invested.
As Warren Buffett put it: “Performance comes, performance goes. Fees never falter.”

Avoiding Unnecessary Trading: The Hidden Cost of “Doing Something”

Many investors feel the urge to act when markets move—but excessive trading often leads to bad timing decisions (buying high, selling low).
A Dalbar study found that the average investor underperforms the market by several percentage points per year due to emotional trading mistakes.
Long-term success isn’t about reacting—it’s about staying the course.

Diversification: Your Best Defense Against the Unknown

Instead of betting on a handful of stocks, successful investors spread their risk across asset classes, industries, and geographies.
A well-diversified portfolio helps investors ride out market volatility and capture long-term growth without relying on luck.
Owning the entire market means you don’t have to predict which stocks will win—you own the winners by default.

Staying Disciplined: The Hardest (But Most Important) Habit

The biggest challenge in investing isn’t the market—it’s our own emotions.
Fear and greed push investors to abandon sound strategies at the worst possible times.
The most successful investors aren’t necessarily the smartest—they’re the most patient and disciplined.
As Ellis puts it: “Most investors don’t need to outsmart the market. They need to outsmart themselves.”

“The best investors don’t try to outplay the market—they let the market work for them.” — Charley Ellis

Why Index Investing Is the Best Bet for Long-Term Wealth (22:15)

If beating the market is so difficult, why do so many investors keep trying? Charley Ellis explains that the answer lies in overconfidence, misinformation, and the financial industry’s marketing machine. Many investors believe they can pick the right stocks or identify a fund manager who will outperform, but decades of research show that index investing is the best strategy for long-term success.

The Overwhelming Case for Index Investing

For years, studies have compared actively managed funds to index funds. The results are remarkably consistent: most active managers fail to beat their benchmarks over time. Here’s why:

1. Most Active Managers Underperform Their Benchmarks

Research from SPIVA (S&P Indices Versus Active) shows that over 85% of actively managed funds underperform their benchmarks over a 15-year period.
Even the few that do outperform in one decade rarely repeat their success in the next.
The competition among professionals is so intense that even the best fund managers struggle to maintain an edge.

2. High Costs Erode Returns

The average actively managed fund charges fees of 1% to 2% per year, while many index funds cost 0.05% or less.
A difference of just 1% in fees may seem small, but over a 30-year investing horizon, it can cost an investor hundreds of thousands—or even millions—of dollars in lost returns.
Research consistently shows that keeping costs low is one of the most reliable ways to improve investment returns.

3. Index Funds Remove the Emotional Pitfalls of Investing

Active investors tend to chase past performance, buying high and selling low in an attempt to time the market.
Index investors, by contrast, own the entire market and avoid the behavioral mistakes that lead to underperformance.
By sticking with a diversified index fund, investors avoid costly misjudgments and unnecessary stress.

4. Market Returns Are More Than Enough

The stock market has historically delivered average annual returns of 7-10%.
While some investors believe they need to “beat” the market, most who try end up underperforming instead.
Investing in a total market index fund allows you to capture the full return of the market with minimal effort and cost.

The Power of Simplicity in Investing

The beauty of index investing is its simplicity. Investors don’t need to analyze earnings reports, time the market, or guess which fund manager will outperform next. Instead, they follow a straightforward plan:

Buy low-cost, broad-market index funds.
Stay invested for the long haul.
Let compounding do the heavy lifting.

As Ellis puts it, investing success isn’t about complexity—it’s about consistency.

“The real challenge isn’t figuring out which stocks to buy. It’s sticking with a simple strategy and avoiding unnecessary mistakes.” — Charley Ellis

In the next section, we’ll explore why even the most sophisticated institutional investors struggle to beat the market—and what that means for everyday investors.

Why Yale’s Investing Model Won’t Work For You (21:40)

Many investors look at the Yale Endowment and its extraordinary returns under David Swensen and wonder, “Why shouldn’t I invest like that?” After all, Yale’s portfolio includes private equity, hedge funds, venture capital, and other alternative investments that have outperformed the stock market in the past.

But Charley Ellis argues that individual investors who try to mimic Yale’s investing model are making a critical mistake: they’re playing the wrong game.

Why Yale’s Strategy Worked—for Yale

David Swensen revolutionized institutional investing by shifting Yale’s portfolio away from traditional stocks and bonds and toward alternative assets. His approach made sense for Yale for several reasons:

Access to the Best Funds – Yale had access to exclusive hedge funds, private equity firms, and venture capital funds that are closed to most investors. These elite managers had a track record of delivering above-market returns.
Long Investment Horizon – With a time horizon measured in perpetuity, Yale could afford to be highly illiquid, investing in assets that might not generate returns for decades.
Large Research Teams – Yale had a team of analysts and advisors capable of evaluating the most complex investments. They had the expertise to identify superior funds and negotiate favorable terms.
Scale and Negotiating Power – As a multibillion-dollar endowment, Yale had the ability to secure lower fees, better deal terms, and early access to investments that individuals simply can’t match.

These advantages were critical to Yale’s success. But for individual investors, trying to replicate this approach is a losing battle.

Why the Yale Model Fails for Individual Investors

While Yale’s strategy has been widely admired, the reality is that it simply doesn’t translate to personal investing. Here’s why:

1. Most Investors Lack Access to Elite Funds

The top-performing hedge funds, private equity firms, and venture capital funds are closed to new investors or only accept institutional capital.
Many funds that do accept individual investors charge high fees, have rigid lock-up periods, and underperform public markets after costs.
The average retail investor is left with second-tier options, which don’t provide the same returns Yale achieved.

2. Liquidity Needs Are Different

Yale can lock up its money for decades, but individual investors typically need access to their savings for life events, retirement, or emergencies.
Alternative investments often have multi-year lock-up periods, meaning investors can’t withdraw their money when they need it most.
This lack of liquidity makes alternative investments impractical for most individuals, who require flexibility in their portfolios.

3. High Costs and Complexity Reduce Returns

Hedge funds and private equity charge significant fees—often 2% annually plus 20% of profits. These fees eat into returns, especially when compared to low-cost index funds.
Research shows that after fees, most hedge funds underperform simple index portfolios over time.
Private equity funds have long holding periods, high capital commitments, and uncertain cash flows, making them unsuitable for investors without substantial wealth and patience.

4. Private Investments Don’t Guarantee Outperformance

Many investors assume that private markets offer better returns, but not all private equity or hedge fund investments are winners.
The best-performing funds are oversubscribed and inaccessible, while lower-tier funds often deliver mediocre or even below-market returns.
Yale’s success was due to its ability to pick the top-tier managers—something individual investors simply don’t have the resources to do.

What Individual Investors Should Do Instead

Rather than chasing Yale’s returns with illiquid, high-cost investments, Ellis argues that individual investors should focus on strategies that actually work for them. That means:

Owning the Market – A globally diversified portfolio of low-cost index funds provides broad exposure without the risks of picking individual assets.
Keeping Costs Low – Minimizing fees is one of the few guaranteed ways to improve long-term returns.
Maintaining Liquidity – A portfolio that allows easy access to funds is essential for financial security.
Avoiding Complexity – Yale’s strategy requires deep expertise, exclusive access, and patience that most investors simply don’t have.

The Key Takeaway: Play a Game You Can Win

While Yale’s model worked for an elite institution with unique advantages, it doesn’t work for the average investor. Instead of chasing hedge funds, venture capital, or private equity, individual investors should embrace a simple, transparent, low-cost strategy that maximizes long-term returns without unnecessary risk.

As Ellis puts it:

“The smartest investors don’t try to play Yale’s game. They play a game they can actually win.”

In the next section, we’ll explore why even institutional investors struggle to beat the market—and what that means for everyday investors.

The Biggest Mistakes Investors Make (31:45)

The hardest part of investing isn’t picking the right stocks or finding the perfect strategy—it’s managing your own behavior. Even the most knowledgeable investors fall victim to fear, greed, and overconfidence, making costly mistakes that drag down their returns.

Decades of research confirm what many advisors already know: the average investor earns far less than the market itself.

Morningstar’s Mind the Gap study consistently finds that investors underperform the very funds they invest in—not because the funds are bad, but because people make poor timing decisions.

What causes this gap? Emotions override logic. When markets rise, investors chase performance. When markets fall, they panic and sell. This cycle leads to self-inflicted losses that compound over time.

Ellis identifies five of the most common investing mistakes—and how to avoid them.

1. Chasing Performance

Investors love to buy what’s been doing well recently—whether it’s a hot stock, a booming sector, or a high-flying mutual fund.
But by the time most people jump in, the best gains have already been made, and a downturn is often around the corner.
This pattern repeats: buy high, panic when it drops, sell low.

“If you chase past performance, you’re always reacting to yesterday’s news—while smart investors are positioning for the future.” — Charley Ellis

2. Trying to Time the Market

Investors believe they can predict when to get in and out, avoiding downturns while capturing rallies.
But study after study shows that even professional traders fail to time the market consistently.
Missing just a handful of the market’s best days can devastate long-term returns.
For example: If you missed just the 10 best days in the S&P 500 over the past 20 years, your total return would have been cut by more than half.

3. Overconfidence in Stock Picking

Many investors believe they can outsmart the market—picking individual stocks that will outperform.
But even professional fund managers, with all their research and technology, fail to beat the market most of the time.
Picking stocks isn’t just about finding winners—you also need to know when to sell and avoid holding onto losers too long.

4. Panic Selling During Market Drops

When markets fall, emotions take over. Investors see their portfolio value shrinking and feel the urge to sell to “stop the bleeding.”
But history shows that the worst days in the market are often followed by the best days.
Selling after a crash locks in losses and makes it much harder to recover when the market rebounds.

“The best investors aren’t the ones who always make the right moves. They’re the ones who avoid making the biggest mistakes.” — Charley Ellis

5. Ignoring the Power of Compounding

Many investors focus too much on short-term gains and losses, losing sight of the long-term power of compounding.
The secret to wealth isn’t hitting home runs—it’s letting your money grow steadily over decades.
The earlier you start and the longer you stay invested, the more powerful compounding becomes.

Most investing mistakes come from emotion-driven decisions. The best way to protect yourself? Use a disciplined, rules-based approach.

Follow a long-term investment plan—and don’t abandon it when emotions run high.
Ignore market noise—don’t let headlines push you into knee-jerk reactions.
Stick with low-cost, diversified investments—instead of trying to outguess the market.
Think in decades, not days—because time in the market beats timing the market.

Ellis’s biggest takeaway? Successful investing isn’t about brilliance—it’s about discipline.

“The greatest investors aren’t the smartest ones. They’re the most patient.” — Charley Ellis

The Best Investment Strategy? Let the Market Work for You (38:15)

After decades of studying financial markets, working with institutions, and writing some of the most influential books on investing, Charley Ellis has arrived at a simple but powerful conclusion: the best way to win the investing game is to stop trying to outguess the market.

Most investors waste time searching for the next great stock, attempting to time the market, or picking fund managers who promise to outperform. But Ellis explains that the odds of consistently beating the market are slim—not because investors lack skill, but because the competition is too strong, and the market is too efficient.

Rather than trying to outmaneuver Wall Street professionals, successful investors embrace a long-term, low-cost strategy that allows them to capture the market’s growth without taking unnecessary risks.

Why Index Investing Wins Over the Long Run

If outperforming the market is nearly impossible, the most logical move is to own the market itself.

Most actively managed funds fail to beat their benchmarks over time, largely due to high fees, excessive trading, and fierce competition among professionals. By contrast, index investors benefit from:

Lower costs – Minimizing fees allows more of an investor’s money to compound over time.
A built-in advantage – Since index funds track the market, they eliminate the need to predict which stocks or sectors will outperform.
Fewer behavioral mistakes – Investors who stick with index funds avoid common pitfalls like performance chasing and panic selling.

Ellis explains that index investing works not just because of its cost advantage, but because it removes the temptation to tinker with investments—a major reason why so many investors underperform.

The Power of Staying the Course

One of the most overlooked investment advantages is simply staying invested. Market downturns and financial headlines create the illusion that investors need to react, but history shows that those who remain patient tend to do far better than those who try to time the market.

Ellis points to legendary investors like Warren Buffett, John Bogle, and David Swensen as examples of individuals who built wealth not by outguessing the market, but by maintaining discipline and allowing compounding to work over time.

If you’re investing for decades, short-term fluctuations are irrelevant. What matters is sticking to a strategy that keeps you invested and lets time do the work.

Keeping It Simple: The Key to Long-Term Wealth

Many investors assume that complexity leads to better results, but Ellis argues that the opposite is true. The best strategies are simple, repeatable, and easy to stick with.

That means:

Avoiding unnecessary complexity – Hedge funds, alternative investments, and market timing often add more risk than reward.
Minimizing fees and trading costs – Every dollar spent on expenses is a dollar that won’t compound for the future.
Ignoring market noise – Successful investors don’t react to short-term events. They build a plan, stay disciplined, and let the market do its work.

By keeping things simple and focusing on what matters—low costs, diversification, and a long-term mindset—investors set themselves up for success while avoiding the stress and uncertainty of trying to predict market movements.

Final Takeaway: The Best Investment Strategy Is One You Can Stick With

At the end of the day, successful investing isn’t about beating the market. It’s about finding a strategy that works and sticking with it.

Ellis’s advice is clear:

Choose a low-cost, diversified investment approach.
Commit to it for the long term.
Let time and compounding do the rest.

By following these principles, investors can achieve strong returns without the stress of constant decision-making. The challenge isn’t in knowing what to do—it’s in having the patience and discipline to stay the course.

For those looking for an easy-to-follow investment framework, Ellis’s latest book, Rethinking Investing, distills these key lessons into a concise, actionable guide.

Resources:

The Long Term Investor audio is edited by the team at The Podcast Consultant

Submit Your Question For the Podcast

Do you have a financial or investing question you want answered? Submit your question through the “Ask Me Anything” form at the bottom of my podcast page.

Support the Show

Thank you for being a listener to The Long Term Investor Podcast. If you’d like to help spread the word and help other listeners find the show, please click here to leave a review.

I read every single one and appreciate you taking the time to let me know what you think.

Free Financial Assessment

Do you want to make smart decisions with your money? Discover your biggest opportunities in just a few questions with my Financial Wellness Assessment.