We discuss the role of intuition in decision making as well as efforts to work on noise reduction and improve judgement.
Phil Tetlock: Intuition and Noise Reduction
Nate Silver was actually that was an accurate forecast. It was not the right outcome. He didn't call it right there. But you can't say at 70% probability.
Well, Nate Silver wasn't in the, you know, he's a famous guy. He's a celebrity. And he, he was more accurate than almost all of the polls, right. He aggregated his polls. We aggregated his polls before the 2016 election using an algorithm not unlike the one that I described here, extremism, but he used that algorithm, he was coming up with a probability around 90-95% of Hillary winning, sampling at Princeton went public with 90-95% probability. Nate Silver looked at that.
He said, You know, that's too extreme. I'm worried about correlated measurement error in the polls in the swing states of Michigan and Pennsylvania and Wisconsin, if there's correlated measurement error, and those days are pretty close, one of them goes against Hillary, the others are likely to go as well. And she could have had a big Electoral College loss, even though she's ahead in the confidence vote.
He said, I'm not going to turn my 95% into, you know, 90% Trump's gonna win, I'm going to downgrade to 70 or 70% Hillary victory is close to 95%. So he was actually using intuition and judgement to moderate his algorithm. The algorithm would otherwise have directed him to do based on the aggregate of the polls.
Faulty polls and behavioral finance
Are you, it sounds like you're in the client camp on Danny Kahneman and Gary Klein on intuition. You rely on intuition a little bit more than Kahneman would.
When you have two people who are that smart. I think it's a good idea to try this.
So okay, actually, can we talk since week since I brought up Kahneman. Can you talk a little bit about your work with him that you're doing on noise reduction right now? And from what I understand,
That's the work he's doing totally separately from totally separate. I'm aware that he's doing that. Well, we have a project that I might be useful to him. Let's see, I think the interesting thing that you know that I'm excited about the book that is working.
Danny Kahneman working on noise because it strikes me such a valuable complement to is a profoundly influential book on bias such as Thinking Fast and Slow. Bias is about the systematic error, right? You're systematically under confident and systematically overconfident, you can measure it, you can see it. Noise is random. By its nature, it's very, very difficult to observe. But there are ways of observing it by creating things like noise audits, for example.
So if you were to take three of your financial analysts, and show them the same report, and ask them if this report were true, what price should this company be valued at? That's almost a textbook MBA sort of exercise at the end if the three financial analysts each gives you very different information from reading the same report, and Assuming that before it is true, that means you have noise. Got it. And that's an error in your system that you'd like to suppress. And that happens all the time on Wall Street as well.
So some people would argue that the lowest hanging fruit from proven judgement maybe noise reduction, mean bias reduction and from a Kahneman point of view is very difficult for that because of that meal or liar illusion I showed you the bias either very, very sticky, it's much more natural to think what a happy couple they're going to stay together then. [inaudible] type of cognition comes much more naturally to people. You have to pull up the ruler formula liar, right, figure out which lines are longer, but you still see them. You still see them incorrectly even when after the ruler is removed.
noise reduction and the economy
So that makes it that makes that economy fight pessimistic about bias reduction, but he may somewhat more optimistic about noise reduction. And it may be that noise reduction will be the topic does your opponents comes out in 2020 years.
How do I present fundamental information to people without getting them to just migrate to the centre to set the most, you know, the centre of that path?
That's a really good question as hard. We're working on that in a new IARPA tournament. I don't like binary questions I don't like Yes. No questions. I don't like yes, maybe no questions. I prefer questions that do measure probability distributions and more or less continuously. And I think you should put people forecasters on notice that trying to gain this system and assuming that the right centred each distribution around the correct what we think is the correct answer is a good way to lose their you don't want them to take away the implicit message that we created this probability distribution with the middle in mind.
Got it so have randomness to the path that we're demonstrating in all these fundamentals.