Top Performers Are Not The Most Impressive When Extreme Performance Indicates Unreliability

Updated on

Jerker Denrella and Chengwei Liub highlight the relationship between performance and ability is a central concern in the social sciences: Are the most successful much more able than others, and are failures unskilled? Prior research has shown that noise and self-reinforcing dynamics make performance unpredictable and lead to a weak association between ability and performance.

Extreme performance attracts people’s attention. People tend to believe the most successful are the most skillful and that failures lack skill (1, 2). A tendency to imitate the most successful has also been argued to be a basic universal trait that is shaped by evolution and promotes adaptiveness (3, 4). However, is success necessarily an indication of skill and worthy of praise and imitation and failure an indication of lack of skill?

Clearly, observed performance is not always a reliable indicator of skill. Chance events outside the control of individuals often influence performance (5–7). Moreover, such chance events rarely average out over time. Instead, due to “rich-get-richer” dynamics and “Matthew effects” (8), success usually breeds success and failure breeds failure. For example, individuals with early success might be given more resources and instruction, or consumers may favor products with a high market share (9, 10). Prior research has shown how such processes can amplify chance events and produce a weak association between performance and ability (11–13), leading to a distribution of outcomes that is both unpredictable and highly unequal (14). In such settings, extreme success and failure are, at best, only weak signals of skill. The highest performers may be more able than others and the lowest performers less able than others, but one should not expect their skill level to be very far from the mean (15).

These prior contributions show that performance and skill may be weakly associated due to noise and rich-get-richer dynamics, but they do not challenge the idea that higher performers are likely more skilled and worthy of imitation. Even if the highest performers are only marginally more skilled than others, it makes sense to imitate them. In this paper, we show that noise and richget-richer dynamics can have more counterintuitive implications that go beyond the conventional understanding of regression to the mean.

Noise and rich-get-richer dynamics not only introduce unpredictability but also change how much one can learn from extreme performances and whether higher performance indicates higher skill. In particular, we show that when noise and rich-getricher dynamics can strongly influence performance, extreme performances can be relatively uninformative about skill. As a result, higher performance may not indicate higher skill. The highest performers may not be the most skilled and the lowest performers may not be the least skilled. The implication is that one should not imitate the highest performers nor dismiss the worst performers.

More generally, we show that whether higher performance indicates higher skill depends on whether extreme performance could be achieved by skill or requires luck. The intuition behind our results is that an extreme performance may be more informative about the level of noise and the strength of rich-get-richer dynamics than about skill. People often have to infer the degree of skill from performance without knowing the extent to which performance is subject to noise or the extent to which past performance influences future performance. Extreme performance indicates that the level of noise is high and that past performance strongly influences future performance, because extreme performances are more likely then. In settings with high levels of noise and when past performance strongly influences future performance, however, observed performance is a less reliable indicator of skill because chance events and early success strongly influence performance. Because extreme performances are less informative about skill levels than moderate levels of performance, a rational person should regress more to the mean when observing extreme performances, implying that the association between performance and ability can be nonmonotonic.

We develop two models to formalize this intuition. The first model assumes that current performance depends on skill but also on past performance and evaluators are uncertain about how much past performance matters. The second model assumes that performance depends on skill and noise and evaluators are uncertain about the extent to which noise matters. For both models, we show that higher performance does not indicate higher skill if luck is essential to achieving extreme performance. On the other hand, when luck is unlikely to result in extreme performance, we show that extreme performances, high or low, can be especially informative about skill.

Full article via pnas.org

Leave a Comment