How to Use Backtests Effectively

Updated on

Everyone interested in managing their own money ought to keep an old idiom in mind: if it seems too good to be true, it probably is. Unfortunately, greed is a powerful motivator. It’s tempting to see a new model with an incredible backtest and think this could be the answer.

Experienced investors know that there’s often a drastic change between a model’s backtest and its first live run. You can usually find that point by checking for where the 45-degree angle increase in value drops off into sideways movement (and generally underperforms the market).

This week, we’ll take a deeper dive into how you can minimize these problems using professional techniques.

Review

Our last Stock Exchange revisited a common theme: making stock picks according to a set time frame. Our models suggested finance and software stocks in the short-term, and energy for the long-term.

Let’s turn to this week’s ideas.

This Week— *crickets*

We usually arrive to find the gang happily enjoying their weekly poker night. Instead, all we’ve got are Felix and Oscar’s weekly rankings with a “gone fishin’” note on the counter. Strange behavior.

We decided to give Vince Castelli a call to investigate. Vince is our modeling guru, a brilliant scientist who spent the bulk of his career as a civilian employee for the U.S. Navy. During his time there, he’s had hands-on experience with modeling techniques vital to national security – not something you can find in the classroom. He knows these models better than anyone; after all, he designed them.

Jeff: Vince! What is this about giving everyone the week off?

V: I didn’t give them the week off. There were new no new fresh signals.

J: Is there something wrong with the gang? They encompass five different methods. How can there be no fresh ideas?

V: A key feature of all models is recognizing the best times to trade. When volatility increases trades become less predictable.

J: What do you mean? The VIX is lower this week. Volatility is down.

V: That measure is for amateurs. Trading volatility includes both upside and downside. That bogus fear gauge emphasizes only the downside. Predictions are affected by extreme movements in either direction.

J: Since we do not have current picks to feature, maybe we could discuss how you developed these models.  There was an excellent recent article from Ben Carlson about what you cannot learn from a backtest. It reminded me of the TV ads suggesting that anyone can discover a system and trade their way to a fortune.

V:  If only it were that easy.

J:  Ben’s description made me think of some suburbanites wandering into the woods with massive chain saws.  The power of the tools far exceeds their skill in using them.

V: I see it all of the time.

J:  I would like to take up a few of Ben’s points and get your reaction.  How about this:  How many bad backtests came before the good ones?

V:  This is a great question.  Most people do not know the right questions to ask the model developer.  That one is crucial.

J:  How would you answer?

V:  Our method preserves multiple out-of-sample periods.   We develop models on our development data, saving pristine time periods for the test.  We verify that the strength of results continues.  You cannot just look at backtest results; you must know the developer’s method.

J:  Here is another good point — Data availability at the time.   Isn’t it easy to “peek ahead” or to exclude data from failing company?

V:  It certainly is.  You need to have data that includes the failed and merged companies.  The average person at home will not pay up to get this.  It introduces a deceptive, positive bias.

J: Ben also raises an interesting point about friction.  He writes:

It’s almost impossible in a backtest to completely account for costs and frictions such as taxes, commissions, market impact from trading, market liquidity, etc. Sure, you can estimate these frictions, but you never truly understand how these things will affect your bottom line until you actually have to execute buy and sell orders.

V:  This is the first point where I really disagree with him.  Why is it almost impossible?  You should definitely include commissions and a slippage factor.  If your trades are a small percentage of the market volume, the impact from trading is negligible.  Taxes vary by the type of account and the investor.

J:  Interesting point!  “Almost impossible” is strong language.  For someone who knows the ropes, this kind of test might represent real edge.

V:  That is what I do with each of my creations!

J:  Some of Ben’s other points relate to psychological factors.  The trader bailing out of the system in the face of losses.  Or concern about real money.

V:  That is strictly a matter of confidence in the system.  If it has been developed properly, you should not do a lot of fretting.

J: Thanks for joining us, Vince. I’m sure your comments will help readers make more sense of our series.

V: Any time!

Conclusion

One important point was not mentioned in Ben’s article – simplicity.  The temptation for the untrained modeler is to introduce as many variables as possible, hoping to find correlations that others have missed.  What they find is misleading. Computers are powerful enough to discover apparent links between variables when there is actually no relationship. A great model uses as few variables as possible.  The backtest may not seem as good, but the real-time trading will be much better.

Quantitative modeling is an extraordinarily complicated field. In some ways, the way to find success here is similar to finding success in the investment world as a whole. Find the right experts, learn their methods, and try to make sense of the data for yourself. Backtesting can be effective or dangerous – it depends on the skill of the developer.

Background on the Stock Exchange

Each week Felix and Oscar host a poker game for some of their friends. Since they are all traders they love to discuss their best current ideas before the game starts. They like to call this their “Stock Exchange.” (Check it out for more background). Their methods are excellent, as you know if you have been following the series. Since the time frames and risk profiles differ, so do the stock ideas. You get to be a fly on the wall from my report. I am the only human present, and the only one using any fundamental analysis.

The result? Several expert ideas each week from traders, and a brief comment on the fundamentals from the human investor. The models are named to make it easy to remember their trading personalities.

Questions

If you want an opinion about a specific stock or sector, even those we did not mention, just ask! Put questions in the comments. Address them to a specific expert if you wish. Each has a specialty. Who is your favorite? (You can choose me, although my feelings will not be hurt very much if you prefer one of the models).

Getting Updates

We have a new (free) service to subscribers to our Felix/Oscar update list. You can suggest three favorite stocks and sectors. Sign up with email to “etf at newarc dot com”. We keep a running list of all securities our readers recommend. The “favorite fifteen” are top ranking positions according to each respective model. Within that list, green is a “buy,” yellow a “hold,” and red a “sell.”  Suggestions and comments are welcome. Please remember that these are responses to reader requests, not necessarily stocks and sectors that we own. Sign up now to vote your favorite stock or sector onto the list!

Leave a Comment