It can be difficult to disentangle the difference between luck and skill by examining performance on its own.
We simulate the returns of investors with different prediction accuracy levels and find that an investor with the skill of a fair coin (i.e. 50%) would likely under-perform a simple buy-and-hold investor, even before costs are considered.
It is not until an investor exhibits accuracy in excess of 60% that a buy-and-hold investor is meaningfully “beaten” over rolling 5-year evaluation periods.
In the short-term, however, a strategy with a known accuracy rate can still masquerade as one far more accurate or far less accurate due to luck.
Further confounding the analysis is the role of skewness of the return distribution. Positively skewed strategies, like trend following, can actually exhibit accuracy rates lower than 50% and still be successful over the long run.
Relying on perceptions of accuracy alone may lead to highly misguided conclusions.
The only thing sure about luck is that it will change. — Bret Harte1
The distinction between luck and skill in investing can be extremely difficult to measure. Seemingly good or bad strategies can be attributable to either luck or skill, and the truth has important implications for the future prospects of the strategy.Source: Grinold and Kahn, Active Portfolio Management. (New York: McGraw-Hill, 1999).
Time is one of the surest ways to weed out lucky strategies, but the amount of time needed to make this decision with a high degree of confidence can be longer than we are willing to wait. Or, sometimes, even longer than the data we have.
For example, in order to be 95% confident that a strategy with a 7% historical return and a volatility of 15% has a true expected return that is greater than a 2% risk-free rate, we would need 27 years of data. While this is possible for equity and bond strategies, we would have a long time to wait in order to be confident in a Bitcoin strategy with these specifications.
Even after passing that test, however, that same strategy could easily return less than the risk-free rate over the next 5 years (the probability is 25%).
Regardless of the skill, would you continue to hold a strategy that underperformed for that long?
In this commentary, we will use a sample U.S. sector strategy that isolates luck and skill to explore the impacts of varying accuracy and how even increased accuracy may only be an idealized goal.
The (In)Accurate Investor
To investigate the historical impact of luck and skill in the arena of U.S. equity investing, we will consider a strategy that invests in the 30 industries from the Kenneth French Data Library.
Each month, the strategy independently evaluates each sector and either holds it or invests the capital at the risk-free rate. The term “evaluates” is used loosely here; the evaluation can be as simple as flipping a (potentially biased) coin.
The allocation allotted to each sector is 1/30th of the portfolio (3.33%). We are purposely not reallocating capital among the sectors chosen so that the sector calls based on the accuracy straightforwardly determine the performance.
To get an idea for the bounds of how well – or poorly – this strategy would have performed over time, we can consider three investors:
The Plain Investor – This investor simply holds all 30 sectors, equally weighted, all the time.
The Perfect Investor – This investor allocates with 100% accuracy. Using a crystal ball to look into the future, if a sector will go up in the subsequent month, this investor will allocate to it. If the sector will go down, this investor will invest the capital in cash.
The Anti-Perfect Investor – This investor not merely imperfect, they are the complete opposite of the Perfect Investor. They make the wrong calls to invest or not without fail. Their accuracy is 0%. They are so reliably bad that if you could short their strategy, you would be the Perfect Investor.
The Perfect and Anti-Perfect investors set the bounds for what performance is possible within this framework, and the Plain Investor denotes the performance of not making any decisions.
The growth of each boundary strategy over the entire time period is a little outrageous.
Annualized Return
Annualized Volatility
Maximum Drawdown
Plain Investor
10.5%
19.3%
83.9%
Perfect Investor
42.6%
11.0%
0.0%
Anti-Perfect Investor
-20.0%
12.1%
100.0%
Source: Kenneth French Data Library. Calculations by Newfound Research. Past performance is not a guarantee of future results. All returns are hypothetical and backtested. Returns are gross of all fees. This does not reflect any investment strategy offered or managed by Newfound Research and was constructed exclusively for the purposes of this commentary. It is not possible to invest in an index.
A more informative illustration is the rolling annualized 5-year return for each strategy.
Source: Kenneth French Data Library. Calculations by Newfound Research. Past performance is not a guarantee of future results. All returns are hypothetical and backtested. Returns are gross of all fees. This does not reflect any investment strategy offered or managed by Newfound Research and was constructed exclusively for the purposes of this commentary. It is not possible to invest in an index.
While the spread between the Perfect and Anti-Perfect investors ebbs and flows, its median value Is 59,000 basis points (“bps”). Between the Perfect and Plain investors, there is still 29,000 bps of annualized outperformance to be had. A natural wish is to make calls that harvest some of this spread.
Accounting for Accuracy
Now we will look at a set of investors who are able to evaluate each sector with some known degree of accuracy.
For each accuracy level between 0% and 100% (i.e. our Anti-Perfect and Perfect investors, respectively), we simulate 1,000 trials and look at how the historical results have played out.
A natural starting point is the investor who merely flips a fair coin for each sector. Their accuracy is 50%.
The chart below shows the rolling 5-year performance range of the simulated trials for the 50% Accurate Investor.
Source: Kenneth French Data Library. Calculations by Newfound Research. Past performance is not a guarantee of future results. All returns are hypothetical and backtested. Returns are gross of all fees. This does not reflect any investment strategy offered or managed by Newfound Research and was constructed exclusively for the purposes of this commentary. It is not possible to invest in an index.
In 59% of the rolling periods, the buy-and-hold Plain Investor beat even the best 50% Accurate Investor. The Plain Investor was only worse than the worst performing coin flip strategy in 6% of rolling periods.
Beating buy-and-hold is hard to do reliably if you rely only on luck.
In this case, having a neutral hit rate with the negative skew of the sector equity returns leads to negative information coefficients. Taking more bets over time and across sectors did not help offset this distributional disadvantage.
So, let’s improve the accuracy slightly to see if the rolling results improve. Even with negative skew (-0.42 median value for the 30 sectors), an improvement in the accuracy to 60% is enough to bring the theoretical information coefficient back into the positive realm.
Source: Kenneth French Data Library. Calculations by Newfound Research. Past performance is not a guarantee of future results. All returns are hypothetical and backtested. Returns are gross of all fees. This does not reflect any investment strategy offered or managed by Newfound Research and was constructed exclusively for the purposes of this commentary. It is not possible to invest in an index.
The worst of these more skilled investors is now beating the Plain Investor in 41% of the rolling periods, and the best is losing to the buy-and-hold investor in 13% of the periods.
Going the other way, to a 40% accurate investor, we find that the best one was beaten by the Plain investor 93% of the time, and the worst one never beats the buy-and-hold investor.
Source: Kenneth French Data Library. Calculations by Newfound Research. Past performance is not a guarantee of future results. All returns are hypothetical and backtested. Returns are gross of all fees. This does not reflect any investment strategy offered or managed by Newfound Research and was constructed exclusively for the purposes of this commentary. It is not possible to invest in an index.
If we only require a modest increase in our accuracy to beat buy-and-hold strategies over shorter time horizons, why isn’t diligently focusing on increasing our accuracy an easy approach to success?
In order to increase our accuracy, we must first find a reliable way to do so: a task easier said than done due to the inherent nature of probability. Something having a 60% probability of being right does not preclude it from being wrong for a long time. The Law of Large Numbers can require larger numbers than our portfolios can stand.
Thus, even if we have found a way that will reliably lead to a 60% accuracy, we may not be able to establish confidence in that accuracy rate. This uncertainty in the accuracy can be unnerving. And it can cut both ways.
A strategy with a hit rate of less than 50% can masquerade as a more accurate strategy simply for lack of sufficient data to sniff out the true probability.
Source: Kenneth French Data Library. Calculations by Newfound Research. Past performance is not a guarantee of future results. All returns are hypothetical and backtested. Returns are gross of all fees. This does not reflect any investment strategy offered or managed by Newfound Research and was constructed exclusively for the purposes of this commentary. It is not possible to invest in an index.
You may think you have an edge when you do not. And if you do not have an edge, repeatedly applying it will lead to worse and worse outcomes.2
Accuracy Schmaccuracy
Our preference is to rely on systematic bets, which generally fall under the umbrella of factor investing. Even slight improvements to the accuracy can lead to better results when applied over a sufficient breadth of investments. Some of these factors also alter the distribution of returns (i.e. the skew) so that accuracy improvements have a larger impact.
Consider two popular measures of trend, used as the signals to determine the allocations in our 30 sector US equity strategy from the previous sections:
12-1 Momentum: We calculate the return over an 11-month period, starting one month ago to account for mean reversionary effects. If this number is positive, we hold the sector; if it is negative, we invest that capital at the risk-free rate.
10-month Simple Moving Average (SMA): We average the prices over the prior 10 months and compare that value to the current price. If the current price is greater than or equal to the average, we hold the sector; if it is less than, we invest that capital at the risk-free rate.
These strategies have volatilities in line with the Perfect and Anti-Perfect Investors and returns similar to the Plain Investor.
Using our measure of accuracy as correctly calling the direction of the sector returns over the subsequent month, it might come as a surprise that the accuracies for the 12-1 Momentum and 10-month SMA signals are only 42% and 41%, respectively.
Even with this low accuracy, the following chart shows that over the entire time period, the returns of these strategies more closely resemble those of the 55% Accurate Investor and have even looked like those of the 70% Accurate Investor over some time periods. What gives?
Source: Kenneth French Data Library. Calculations by Newfound Research. Past performance is not a guarantee of future results. All returns are hypothetical and backtested. Returns are gross of all fees. This does not reflect any investment strategy offered or managed by Newfound Research and was constructed exclusively for the purposes of this commentary. It is not possible to invest in an index.
This is an example of how addressing the negative skew in the underlying asset returns can offset a sacrifice in accuracy. These trend following strategies may have overall accuracy of less than 50%, but they have been historically right when it counts.
Consistently removing large negative returns – at the expense of giving up some large positive returns – is enough to generate a return profile that looks much like a strategy that picks sectors with above average accuracy.
Whether investors can stick with a strategy that exhibits below 50% accuracy, however, is another question entirely.
Conclusion
While most investors expect the proof to be in the eating of the pudding, in this commentary we demonstrate how luck can have a meaningful impact in the determination of whether skill exists. While skill should eventually differentiate itself from luck, the horizon over which it will do so may be far, far longer than most investors suspect.
To explore this idea, we construct portfolios comprised of all thirty industry groups. We then simulate the results of investors with known accuracy rates, comparing their outcomes to 100% Accuracy, 100% Inaccurate, and Buy-and-Hold benchmarks.
Perhaps somewhat counter-intuitively, we find that an investor exhibiting 50% accuracy would have fairly reliably underperformed a Buy-and-Hold Investor. This seems somewhat counter-intuitive until we acknowledge that equity returns have historically exhibit negative skew, with the left tail of their return distribution (“losses”) being longer and fatter than the right (“gains”). Combining a neutral hit rate with negative skew creates negative information coefficients.
To offset this negative skew, we require increased accuracy. Unfortunately, even in the case where an investor exhibits 60% accuracy, there are a significant number of 5-year periods where it might masquerade as a strategy with a much higher or lower hit-rate, inviting false conclusions.
This is all made somewhat more confusing when we consider that a strategy can have an accuracy rate below 50% and still be successful. Trend following strategies are a perfect example of this phenomenon. The positive skew that has been historically exhibited by these strategies means that frequently inaccurate trades of small magnitude are offset by infrequent, by very large accurate trades.
Yet if we measure success by short-term accuracy rates, we will almost certainly dismiss this type of strategy as one with no skill.
When taken together, this evidence suggests that not only might it be difficult for investors to meaningfully determine the difference between skill and luck over seemingly meaningful time horizons (e.g. 5 years), but also that short-term perceptions of accuracy can be woefully misleading for long-term success. Highly accurate strategies can still lead to catastrophe if there is significant negative skew lurking in the shadows (e.g. an ETF like XIV), while inaccurate strategies can be successful with enough positive skew (e.g. trend following).
How Much Accuracy Is Enough?
By Nathan Faber
On March 4, 2019
In Craftsmanship, Portfolio Construction, Trend, Weekly Commentary
Available as a PDF download here.
Summary
The distinction between luck and skill in investing can be extremely difficult to measure. Seemingly good or bad strategies can be attributable to either luck or skill, and the truth has important implications for the future prospects of the strategy.
Source: Grinold and Kahn, Active Portfolio Management. (New York: McGraw-Hill, 1999).
Time is one of the surest ways to weed out lucky strategies, but the amount of time needed to make this decision with a high degree of confidence can be longer than we are willing to wait. Or, sometimes, even longer than the data we have.
For example, in order to be 95% confident that a strategy with a 7% historical return and a volatility of 15% has a true expected return that is greater than a 2% risk-free rate, we would need 27 years of data. While this is possible for equity and bond strategies, we would have a long time to wait in order to be confident in a Bitcoin strategy with these specifications.
Even after passing that test, however, that same strategy could easily return less than the risk-free rate over the next 5 years (the probability is 25%).
Regardless of the skill, would you continue to hold a strategy that underperformed for that long?
In this commentary, we will use a sample U.S. sector strategy that isolates luck and skill to explore the impacts of varying accuracy and how even increased accuracy may only be an idealized goal.
The (In)Accurate Investor
To investigate the historical impact of luck and skill in the arena of U.S. equity investing, we will consider a strategy that invests in the 30 industries from the Kenneth French Data Library.
Each month, the strategy independently evaluates each sector and either holds it or invests the capital at the risk-free rate. The term “evaluates” is used loosely here; the evaluation can be as simple as flipping a (potentially biased) coin.
The allocation allotted to each sector is 1/30th of the portfolio (3.33%). We are purposely not reallocating capital among the sectors chosen so that the sector calls based on the accuracy straightforwardly determine the performance.
To get an idea for the bounds of how well – or poorly – this strategy would have performed over time, we can consider three investors:
The Perfect and Anti-Perfect investors set the bounds for what performance is possible within this framework, and the Plain Investor denotes the performance of not making any decisions.
The growth of each boundary strategy over the entire time period is a little outrageous.
Source: Kenneth French Data Library. Calculations by Newfound Research. Past performance is not a guarantee of future results. All returns are hypothetical and backtested. Returns are gross of all fees. This does not reflect any investment strategy offered or managed by Newfound Research and was constructed exclusively for the purposes of this commentary. It is not possible to invest in an index.
A more informative illustration is the rolling annualized 5-year return for each strategy.
While the spread between the Perfect and Anti-Perfect investors ebbs and flows, its median value Is 59,000 basis points (“bps”). Between the Perfect and Plain investors, there is still 29,000 bps of annualized outperformance to be had. A natural wish is to make calls that harvest some of this spread.
Accounting for Accuracy
Now we will look at a set of investors who are able to evaluate each sector with some known degree of accuracy.
For each accuracy level between 0% and 100% (i.e. our Anti-Perfect and Perfect investors, respectively), we simulate 1,000 trials and look at how the historical results have played out.
A natural starting point is the investor who merely flips a fair coin for each sector. Their accuracy is 50%.
The chart below shows the rolling 5-year performance range of the simulated trials for the 50% Accurate Investor.
Source: Kenneth French Data Library. Calculations by Newfound Research. Past performance is not a guarantee of future results. All returns are hypothetical and backtested. Returns are gross of all fees. This does not reflect any investment strategy offered or managed by Newfound Research and was constructed exclusively for the purposes of this commentary. It is not possible to invest in an index.
In 59% of the rolling periods, the buy-and-hold Plain Investor beat even the best 50% Accurate Investor. The Plain Investor was only worse than the worst performing coin flip strategy in 6% of rolling periods.
Beating buy-and-hold is hard to do reliably if you rely only on luck.
In this case, having a neutral hit rate with the negative skew of the sector equity returns leads to negative information coefficients. Taking more bets over time and across sectors did not help offset this distributional disadvantage.
So, let’s improve the accuracy slightly to see if the rolling results improve. Even with negative skew (-0.42 median value for the 30 sectors), an improvement in the accuracy to 60% is enough to bring the theoretical information coefficient back into the positive realm.
Source: Kenneth French Data Library. Calculations by Newfound Research. Past performance is not a guarantee of future results. All returns are hypothetical and backtested. Returns are gross of all fees. This does not reflect any investment strategy offered or managed by Newfound Research and was constructed exclusively for the purposes of this commentary. It is not possible to invest in an index.
The worst of these more skilled investors is now beating the Plain Investor in 41% of the rolling periods, and the best is losing to the buy-and-hold investor in 13% of the periods.
Going the other way, to a 40% accurate investor, we find that the best one was beaten by the Plain investor 93% of the time, and the worst one never beats the buy-and-hold investor.
Source: Kenneth French Data Library. Calculations by Newfound Research. Past performance is not a guarantee of future results. All returns are hypothetical and backtested. Returns are gross of all fees. This does not reflect any investment strategy offered or managed by Newfound Research and was constructed exclusively for the purposes of this commentary. It is not possible to invest in an index.
If we only require a modest increase in our accuracy to beat buy-and-hold strategies over shorter time horizons, why isn’t diligently focusing on increasing our accuracy an easy approach to success?
In order to increase our accuracy, we must first find a reliable way to do so: a task easier said than done due to the inherent nature of probability. Something having a 60% probability of being right does not preclude it from being wrong for a long time. The Law of Large Numbers can require larger numbers than our portfolios can stand.
Thus, even if we have found a way that will reliably lead to a 60% accuracy, we may not be able to establish confidence in that accuracy rate. This uncertainty in the accuracy can be unnerving. And it can cut both ways.
A strategy with a hit rate of less than 50% can masquerade as a more accurate strategy simply for lack of sufficient data to sniff out the true probability.
Source: Kenneth French Data Library. Calculations by Newfound Research. Past performance is not a guarantee of future results. All returns are hypothetical and backtested. Returns are gross of all fees. This does not reflect any investment strategy offered or managed by Newfound Research and was constructed exclusively for the purposes of this commentary. It is not possible to invest in an index.
You may think you have an edge when you do not. And if you do not have an edge, repeatedly applying it will lead to worse and worse outcomes.2
Accuracy Schmaccuracy
Our preference is to rely on systematic bets, which generally fall under the umbrella of factor investing. Even slight improvements to the accuracy can lead to better results when applied over a sufficient breadth of investments. Some of these factors also alter the distribution of returns (i.e. the skew) so that accuracy improvements have a larger impact.
Consider two popular measures of trend, used as the signals to determine the allocations in our 30 sector US equity strategy from the previous sections:
These strategies have volatilities in line with the Perfect and Anti-Perfect Investors and returns similar to the Plain Investor.
Using our measure of accuracy as correctly calling the direction of the sector returns over the subsequent month, it might come as a surprise that the accuracies for the 12-1 Momentum and 10-month SMA signals are only 42% and 41%, respectively.
Even with this low accuracy, the following chart shows that over the entire time period, the returns of these strategies more closely resemble those of the 55% Accurate Investor and have even looked like those of the 70% Accurate Investor over some time periods. What gives?
This is an example of how addressing the negative skew in the underlying asset returns can offset a sacrifice in accuracy. These trend following strategies may have overall accuracy of less than 50%, but they have been historically right when it counts.
Consistently removing large negative returns – at the expense of giving up some large positive returns – is enough to generate a return profile that looks much like a strategy that picks sectors with above average accuracy.
Whether investors can stick with a strategy that exhibits below 50% accuracy, however, is another question entirely.
Conclusion
While most investors expect the proof to be in the eating of the pudding, in this commentary we demonstrate how luck can have a meaningful impact in the determination of whether skill exists. While skill should eventually differentiate itself from luck, the horizon over which it will do so may be far, far longer than most investors suspect.
To explore this idea, we construct portfolios comprised of all thirty industry groups. We then simulate the results of investors with known accuracy rates, comparing their outcomes to 100% Accuracy, 100% Inaccurate, and Buy-and-Hold benchmarks.
Perhaps somewhat counter-intuitively, we find that an investor exhibiting 50% accuracy would have fairly reliably underperformed a Buy-and-Hold Investor. This seems somewhat counter-intuitive until we acknowledge that equity returns have historically exhibit negative skew, with the left tail of their return distribution (“losses”) being longer and fatter than the right (“gains”). Combining a neutral hit rate with negative skew creates negative information coefficients.
To offset this negative skew, we require increased accuracy. Unfortunately, even in the case where an investor exhibits 60% accuracy, there are a significant number of 5-year periods where it might masquerade as a strategy with a much higher or lower hit-rate, inviting false conclusions.
This is all made somewhat more confusing when we consider that a strategy can have an accuracy rate below 50% and still be successful. Trend following strategies are a perfect example of this phenomenon. The positive skew that has been historically exhibited by these strategies means that frequently inaccurate trades of small magnitude are offset by infrequent, by very large accurate trades.
Yet if we measure success by short-term accuracy rates, we will almost certainly dismiss this type of strategy as one with no skill.
When taken together, this evidence suggests that not only might it be difficult for investors to meaningfully determine the difference between skill and luck over seemingly meaningful time horizons (e.g. 5 years), but also that short-term perceptions of accuracy can be woefully misleading for long-term success. Highly accurate strategies can still lead to catastrophe if there is significant negative skew lurking in the shadows (e.g. an ETF like XIV), while inaccurate strategies can be successful with enough positive skew (e.g. trend following).