A previous healthy memory blog post, “Would You Rather Be Popular or Accurate,” summarized Philip Tetlock’s book, Expert Political Judgment. Tetlock summarized several decades of research on experts’ political predictions. He found that their predictions were virtually indistinguishable from chance, in other words these experts were not experts. However, he was able to classify these experts into two categories, which he labeled hedgehogs and foxes. Hedgehogs were characterized by big ideas. In other words, they were ideologues. However, the judgments of foxes were more nuanced with qualifications and conditions. Even though the judgments of foxes were poor, they were still better than the judgments of hedgehogs. What it is disturbing is that the hedgehogs get more air and print time, so we are wasting our time listening to these experts. Nevertheless, these experts make a good living at being wrong.
Tetlock summarized his new research in Superforcasting: The Art and Science of Prediction by Philip Tetlock and Dan Gardner. This research involved the recruitment of literally thousands of volunteers. These volunteers were given tasks such as predicting if and when, North Korea would conduct a nuclear test, if and when peace would break out in Iraq, if and when Iran would agree to a nuclear ban, etc. The volunteers would research these topics and revise their predictions whenever they thought that new information warranted a revision. The volunteers reported their predictions using subjective ratings. Remember that these were volunteers working without pay. Anyone could volunteer. I believe that token gift certificates were presented.
This research was sponsored by the Intelligence Advanced Research Project Agency (IARPA). I imagine that some readers are asking two questions. One question might be why did IARPA not use expert intelligence experts? The second question might be, why conduct all this research, why not simply ask the experts how they do their analysis?
With respect to the first question I would remind readers of the previous study, where presumable experts were not found to be experts. There also is no means of identifying the experts. Most reports do not include subjective numerical estimates that are amenable to statistics. Nor is their a system that tracks the accuracy of these reports. Moreover, given Tetlock’s previous research where hedgehogs receive the attention and foxes are ignored, it might be that the wrong analysts are being promoted and receiving attention. The foxes might be laboring ignored in obscurity.
With regard to the second question, the answer is that you could not rely on what they tell you. The vast majority of cognitive processing occurs below our level of awareness, and research has shown that at times what people report is why they did something is not consistent with the empirical evidence (see the healthy memory blog post, “Strangers to Ourselves”). To a certain extent it is as useful as asking someone how they ride a bicycle.
It was only a very small percentage of this group who could be classified as “superforecasters.” Moreover, identifying this group presented statistical challenges. The question was whether these high performers more knowledgeable or lucky. After all, lottery winners are lucky individuals who are rewarded for doing something stupid.
What was characteristic of these superforecastors? Well, first of all I believe that all participants could be regarded as having growth mindsets (see the immediately preceding post). The supercasters tended to use relatively precise subjective estimates, which the frequently revised. Moreover, these revisions were done in the spirit of Bayesian analysis (see the healthy memory blog post, “Organizing Information for the Hardest Decisions”), even if they didn’t explicitly use Bayes Theorem. There are many more results and conclusions, but too many to summarize. If interested, I recommend reading the book.
© Douglas Griffith and healthymemory.wordpress.com, 2015. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Douglas Griffith and healthymemory.wordpress.com with appropriate and specific direction to the original content.