Separating Skill From Luck In Investment Performance by Covenant Capital Management
“So, how did you guys do last month?”
It seems like the most natural and logical question to ask an investment manager. But is it the right question to ask? I suspect that most managers either cringe or rejoice internally upon hearing this question depending on how they actually performed the previous month. Almost implied in the question is the notion that the manager’s recent performance is some sort of indicator of his ‘skill’ at investing. But was it skill or was it luck (good or bad) that constituted that month’s performance? And how can one possibly separate the two?
A Laboratory for Luck and Skill
Luckily (or should we say skillfully), there is a scientific way to approach the problem. We can consider a game of chance with some known degree of ‘inherent skill’. For instance imagine a trader who has a 50% chance of winning each trade he takes. Suppose the trader makes twice as much on his winning trades than he loses on his losing trades. This is equivalent to being paid 2 to 1 on a fair coin flip. The long run expected outcome of each trade is $0.501. This $0.50 expectancy is the ‘inherent skill’ of the trader. But an outside observer does not have this knowledge. The outside observer only has the trader’s track record and must use it to somehow decipher the trader’s abilities. So how is this best accomplished?
Another way to pose the question above is to say “How should one look at a track record in a way that will minimize the effects of randomness?” In the scenario above, randomness is manifested in whether or not a series of actual trades turns out to be winners or losers. Even though the chance of winning each trade is 50%, this does not mean that in 10 trades exactly 5 will be winners. There is some possibility that the trader will only win 3 of the next 10 trades – it is the exact same possibility that 10 coin flips will yield only 3 heads. If the trader wins 3 (or fewer) trades over the course of 10, his actual performance per trade is negative (-$0.10 per trade for 3 wins) as opposed to his positive $0.50 expectancy. Thus his actual performance over these 10 trades is vastly different from his long term expectancy.
The mathematical tool for determining the likelihood of observing 3 or fewer wins in 10 trades if the underlying chance of winning is 50% is the binomial distribution2. It turns out that the chance of winning 3 or fewer trades in this scenario is 17.2%. Put another way, even though the trader has a skill which implies he should expect to make $0.50 per trade, there is roughly a 1 in 5 chance that over the course of 10 trades he will lose money – simply due to randomness. Similarly, there is some chance that the trader will win 6 or more trades out of 10, in which case his realized performance is well above his $0.50 expectancy. According to the binomial distribution, the chance of the trader winning 6 or more out of 10 trades is 37.7%. That is there is a 37.7% chance that the trader’s performance over 10 trades will be better than his long term expectancy.
Time Will Tell
The key to separating ‘skill’ from ‘luck’ lies in the fact that the odds in the scenarios given above actually change depending on the number of (independent) observations3. For instance while the chance of winning 3 or fewer trades out of 10 is 17.2%, the chance of winning 6 or fewer trades out of 20 is only 5.8%. Figure 1 illustrates how the odds of observing a winning percentage of less than or equal to 30% for a situation with a true winning percentage of 50% diminish as the number of observations increases.
From the exercise above it should be clear that if a trader truly has skill, it will be more obvious over the long run as opposed to the short run. Put another way, the short term good and bad luck offset over the long run and all that remains is the manager’s skill or ‘market edge’.
Focusing on Short-Term Performance can be Dangerous
Placing undue focus on short term performance is a very slippery slope – it can be hazardous to one’s trading health. A seemingly ‘anomalous’ bad (or good) month may cause a manager to try to avoid (or replicate) his actions in that particular month going forward, when in reality, this ‘anomaly’ may have been nothing more than typical short term randomness. Yet by changing his actions the manager may lose some of his inherent ‘market edge’. It is not only dangerous to managers and the psychology they take into trading, but it is also dangerous to investors.
Let’s look at one more example to drive this last point home. Imagine two groups of traders – five in ‘Group A’ and five in ‘Group B’. The five traders in Group A win 50% of their trades, gain $1.20 on winning trades and lose $1.00 on losing trades. The expectancy for each trader in Group A is $0.10. The five traders in Group B also win 50% of their trades, but gain only $0.80 on winning trades and lose $1.00 on losing trades. Thus, the expectancy for each trader in Group B is -$0.10. Clearly over the long run the traders in Group A will earn money while those in Group B will lose money. But let’s look at a simple simulation of a single month in which each trader does five distinct (random) trades.
Figure 2 shows the results of this simulation. The realized averages are in some cases very different from the long term expectation. In fact, two of the traders in Group A actually lose money while four of the traders in Group B actually make money. Now imagine a portfolio manager who keeps traders who made money the previous month and does not use traders who lost money the previous month. After the month shown in Figure 2 the manager would have four traders with negative expectancy included in his portfolio for the next month, and would not include two traders with positive expectancy.
In fact, after each month the portfolio manager would almost always be keeping traders who recently outperformed their long term expectancy and removing traders who recently underperformed their long term expectancy. The point is clear – focusing too heavily on short term performance not only carries little information regarding a trader’s skill, but it can be a very dangerous activity for both traders and managers alike.
About Covenant Capital Management
Covenant Capital Management is a boutique CTA that has been managing client assets for over 15 years. CCM has offices in Nashville and Chicago, and employs a systematic trading methodology across global futures markets. The goal of Covenant Capital Management is to provide clients with the highest risk-adjusted returns available in the market. You can reach CCM at their website www.covenantcap.com or on twitter @covenantcap.
See full PDF below.
