Don’t Get Fooled Again

Updated on

Explore how investors are commonly fooled by current performance metrics. Learn how to correct for these common mistakes and allocate to managers that have the highest probability of repeating good performance and avoid redeeming from high-quality managers that have experienced some bad luck.

Three Forces Contributing to Type I Errors

  • There are a huge amount of Type I errors. Why?
    • Evolutionary propensity
      to tolerate Type I error
    • Randomness
      With enough tests, something will look “significant”
    • Rare effects
      We incorrectly ignore prior beliefs leading to a high error rate


  • We have a very high tolerance for Type I error
  • There is a tradeoff of Type I and Type II errors
  • For example, if we declared all patients pregnant there would be a 0% Type II error, but a very large Type I error

Evolutionary Foundations

  • High Type I error (low Type II error) animals survive
  • This preference is passed on to the next generation
  • This is the case for an evolutionary predisposition for allowing high Type I errors
  • Pigeons put in cage. Food delivered at regular intervals – feeding time has nothing to do with behavior of birds.
  • Results
    • Skinner found that birds associated their behavior with food delivery
    • One bird would turn counter-clockwise
    • Another bird would tilt its head back
    • A good example of overfitting – you think there is pattern but there isn’t
    • Skinner’s paper called:
      ‘Superstition’ in the Pigeon, JEP (1947)
    • But this applies not just to pigeons or gazelles…
  • Coins the term Apophänie. This is where you see a pattern and make an incorrect inference. He associated this with psychosis and schizophrenia.
  • Apophany is a Type I error (i.e., false insight)
  • Epiphany is the opposite (i.e., true insight)
    • Apophany may be interpreted as overfitting

“…nothing is so alien to the human mind as the idea of randomness.” – John Cohen

Performance Metrics

Performance Metrics

Performance Metrics

Article by Campbell Harvey, PhD – Research Affiliates

See the full slides below.

Leave a Comment