Have you ever seen a bad back-test?
Investment professionals have been jokingly asking that question for years, and the answer remains the same: of course not. That is because no one will likely visit your office to discuss a new product designed to be smart beta, strategic beta, or what I’ll call factor-based whose simulated history only offers mediocre performance.
Why? Because few would buy it.
Which brings us to how investment products are (often) made and how you can determine whether they are worth your clients’ money.
Factor-based products are often developed when asset managers examine historical data to try to determine what attributes of securities may have driven outperformance over time. Before a product is launched, a rules-based methodology may be implemented and applied to historical data as if the methodology had begun earlier, hence the term back-tested. What one person might consider research another might call data mining, and there can be a fine line between the two. But however you think about it, there is a difference between finding a random anomaly and identifying a viable rules-based strategy.
As a fun example of this, a few years ago my colleagues Joel Dickson and Chuck Thomas ran a hypothetical simulation that compared the performance of the S&P 500 Index with an equity portfolio that had an equally weighted combination of all stocks with tickers that began with S, M, A, R, or T. As the figure below shows, this simple, rules-based strategy did very well over a long period of time. However, let’s be honest, there is no sound reason to justify why it would be a good idea to pursue this strategy in the future.
Annualized return of S.M.A.R.T. beta strategy from December 31, 1994, to October 31, 2013
Source: Vanguard.
Note: The S.M.A.R.T. beta strategy is hypothetical in nature and does not represent the returns of any Index or Investment vehicle. It is constructed with equal-weighted components of all current securities in the S&P 500 whose tickers begin with the letters S.M.A.R.T. and rebalanced monthly.
While in most cases, it is hard to eliminate the risks of data mining entirely, there are a number of ways to help improve your confidence in the potential of a simple rules-based strategy to produce a return premium for a client in the future.
I invite you to download our one-page checklist for evaluating back-tested strategies, which identifies common biases that can occur when products are created. The questions in the checklist are ones you may want to ask any time you are considering a factor-based product.
Read the full article here by Doug Grim of Vanguard
