Posts Tagged ‘Foxes’

Messing with the Enemy

December 13, 2019

Messing with the Enemy is an excellent book by Clint Watts. He is a Robert A. Fox Fellow in the Foreign Research Institute’s Program on the Middle East as well as a senior fellow at the Center for Cyber and Homeland Security at the George Washington University. He is graduate of the U.S Military Academy and in addition to his work as an Army officer, he also served in the F.B.I. He founded the Combating Terrorism Center at the Military academy.

He used the internet to study, or as he writes, mess with extremists half a world away. He observed their debates, gauged their commitment to terrorist principles, and poked them with queries from a laptop at home. He was also able to pose as a fellow terrorist.

The internet provided assistance to al-Qaeda operatives when Osama bin Laden was forced out of Tora Bora, Afghanistan. Hunted by the entire international community, his aides and deputies were constantly on the run. The internet allowed for communication between and control of these aides and deputies. Throughout the mid-to late nineties, websites and email chains provided a communications leap forward to terrorists (and the rest of the world), but they had a major limitation: they were one-way modes of information sharing. Bin Laden could only broadcast to audiences. They could not easily follow up with those inclined to join the ranks. All that changed with the dawn of the new millennium. With the emergence of vBulletin, commercially available software allowing group discussions and Yahoo groups, audiences now had a direct window to communicate with Islamist webmasters, clerics, and leaders. In 2001, the Global Islamic Media Front started a Yahoo Group and a related website. They required users to acquire a password to access the discussion page. Many others featuring general Islamist discussions with a sprinkling of jihadi messaging popped up and down toward the end of the decade. Watts writes that none endured for long before rumors of intelligence operatives penetrating them squelched their dialogue and counterterrorism arrests of forum administrators led to their closure. Two-way communication between al-Qaeda leaders and hopeful jihadis increased, but more content needed to follow to sustain audience engagement.

al-Qaeda created an official media group, al-Sahab, to fill the void and gain greater control of jihadi discussions. Bin Laden recognized the value of jihadi websites and began sending audio and written statements from top al-Qaeda leaders directly to al-Qaeda in the Arabian Peninsula’s leader, Yusuf al-Uyayri, and his site Al Neda. Websites and forums served as principal communication points for those around the world inspired by the incredible success of the 9/11 attacks and seeking to join bin Laden’s ranks.

Replication of sites and duplication of content became key features of online survival for al-Qaeda supporters. Openly available software and hosting services meant websites and forums could be created by anyone in minutes, and accessed by anyone around the world with an Internet connection. This lowered technical boundary for mainstream internet users meant relatively novice jihadis now had the power to create their own safe havens online.

In his book The Wisdom of Crowds, James Surowiecki describes how the internet provided a vehicle for crowds to make smarter decisions than even the smartest person in the crowd, working alone, could make. On-line Watts made the prediction on January 2, 2011 that Osama bin Laden would be killed that year. He made this prediction to work as a vehicle for crowdsourcing an important question. What would al-Qaeda and the world of terrorism be if bin Laden were no more? He used this New Year’s prediction to provoke the audience to answer this question. Watts was disappointed to find that rather than yielding great wisdom of important insight from experts, the results instead returned a pattern of answers of no consequence, “Nothing will change,” and “It doesn’t matter” became patent answers from the best thinkers in the field, regardless of the question.

So Watts took recourse in research that has been reported in previous healthymemory posts on Philip Tetlock. In his 2005 book, Expert Political Judgment he reported the survey of hundreds of experts in political thought over two decades. He determined that, en masse, experts were no more successful at predicting future events than a simple coin toss. He identified two kinds of forecasters. He borrowed from a Greek saying, “The fox knows many things, but the hedgehog knows but one big thing.” He classified those good predictors as “foxes” and poorer performers as “hedgehogs.” What differentiated the two groups’ success was how they thought, not what they thought. Tetlock’s foxes were self-critical, used no template, and acknowledged their misses. By contrast, hedgehogs sought to reduce every problem to a single theory, were not comfortable with complexity, we’re overconfident in their assessments, and placed their faith in one big idea, pushing aside alternative explanations. He saw a lot of hedgehogs in his online surveys,and occasional foxes to get insights. He developed a techniques to identify, in advance, foxes.

At this point, there will be a break in this narrative to mention that Tetlock has conducted additional research into intelligence analysis using a very large sample of analysts. There he was able to identify analysts who performed better than chance, and these analysts were, of course, foxes. These posts can be found by entering “Tetlock” into the search box at healthymemory.wordpress.com.

Returning to the current post on Alan Watts, he used the research of Daniel Kahneman and Amos Tversky (two authors oft cited in this blog) Their research identified a series of heuristics and noted the circumstances where biases emerged to make incorrect judgments. Long ago they identified the predictive missteps Watts had observed in his polls. Status quo bias, a belief that tomorrow will most likely look like today, ruled the responses. Loss aversion, a tendency to avoid anticipated losses rather than pursue equally likely gains, filled the results of counterterrorism policy questions. Herding, the tendency of large groups of people to behave the same way and pursue groupthink, drove Watt’s social media recruits to the same set of answers.

Watts changed his approach using Tetlock’s insigts and Kahneman and Tversky’s heuristics and biases. Instead of asking simple yes-no questions, he flooded respondents with as many potential outcomes as he could think of, making it challenging for non experts to wade though the responses. He identified novices and less innovative thinkers by playing to the status quo bias. Every question had a “no change” response option, surrounded by responses imitating common thinking stripped from Google searches, newspaper headlines, and cable news pundits. With every question he offered survey takers a comment box or allowed them to craft an “other”response.

The prediction he made was confirmed when on May 2, 2012, U.S. Navy SEALS killed Osama in Laden in Abbottabad, Pakistan. The result was that his Twitter feed of only a couple hundred followers suddenly became more active than usual. For a brief Google search period, news of bin Laden’s death brought a world of visitors to his New Year’s prediction. His small blog suddenly had an audience, and he had a new opportunity for rater perspectives from a larger crowd.

Finally, Hope on the Prediction Front

October 15, 2015

A previous healthy memory blog post, “Would You Rather Be Popular or Accurate,” summarized Philip Tetlock’s book, Expert Political Judgment.  Tetlock summarized several decades of research on experts’ political predictions.  He found that their predictions were virtually indistinguishable from chance, in other words these experts were not experts.  However, he was able to classify these experts into two categories, which he labeled hedgehogs and foxes.  Hedgehogs were characterized by big ideas.  In other words, they were ideologues.  However, the judgments of foxes were more nuanced with qualifications and conditions.  Even though the judgments of foxes were poor, they were still better than the judgments of hedgehogs.  What it is disturbing is that the hedgehogs get more air and print time, so we are wasting our time listening to these experts.  Nevertheless, these experts make a good living at being wrong.

Tetlock summarized his new research in Superforcasting:  The Art and Science of Prediction by Philip Tetlock and Dan Gardner.  This research involved the recruitment of literally thousands of volunteers.  These  volunteers were given tasks such as predicting if and when, North Korea would conduct a nuclear test, if and when peace would break out in Iraq, if and when Iran would agree to a nuclear ban, etc.  The volunteers would research these topics and revise their predictions whenever they thought that new information warranted a revision.  The volunteers reported their predictions using subjective ratings.  Remember that these were volunteers working without pay.  Anyone could volunteer.  I believe that token gift certificates were presented.

This research was sponsored by the Intelligence Advanced Research Project Agency (IARPA).  I imagine that some readers are asking two questions.  One question might be why did IARPA not use expert intelligence experts?  The second question might be, why conduct all this research, why not simply ask the experts how they do their analysis?

With respect to the first question I would remind readers of the previous study, where presumable experts were not found to be experts.  There also  is no means of identifying the experts.  Most reports do not include subjective numerical estimates that are amenable to statistics.  Nor is their a system that tracks the accuracy of these reports.  Moreover, given Tetlock’s previous research where hedgehogs receive the attention and foxes are ignored, it might be that the wrong analysts are being promoted and receiving attention.  The foxes might be laboring ignored in obscurity.

With regard to the second question, the answer is that you could not rely on what they tell you.  The vast majority of cognitive processing occurs below our level of awareness, and research has shown that at times what people report is why they did something is not consistent with the empirical evidence (see the healthy memory blog post, “Strangers to Ourselves”).  To a certain extent it is as useful as asking someone how they ride a bicycle.

It was only a very small percentage of this group who could be classified as “superforecasters.”    Moreover, identifying this group presented statistical challenges.  The question was whether these high performers more knowledgeable or lucky.   After all, lottery winners are lucky individuals who are rewarded for doing something stupid.

What was characteristic of these superforecastors?  Well, first of all I believe that all participants could be regarded as having growth mindsets (see the immediately preceding post).    The supercasters tended to use relatively precise subjective estimates, which the frequently revised.  Moreover, these revisions were done in the spirit of Bayesian analysis (see the healthy memory blog post, “Organizing Information for the Hardest Decisions”), even if they didn’t explicitly use Bayes Theorem.   There are many more results and conclusions, but too many to summarize.  If interested, I recommend reading the book.

© Douglas Griffith and healthymemory.wordpress.com, 2015. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Douglas Griffith and healthymemory.wordpress.com with appropriate and specific direction to the original content.

Would You Rather Be Popular or Accurate?

July 16, 2014

That is, if you were a political pundit, would your rather be popular or accurate? To answer this question we need to review research done by Philip Tetlock, a professor of psychology and political science. In 1987 he started collecting predictions from a broad array of experts in academia and government on a wide variety of topics in domestic politics, economics, and international relations. He asked theses experts to make predictions on a periodic basis about major events. This study spanned more than fifteen years and was published in his 2005 book, Expert Political Judgment. Regardless of their backgrounds, these experts did barely better than random chance, and had done even worse than rudimentary statistical methods at predicting future political events. About 15 percent of the events they predicted to have no chance of occurring, happened, and about 25% of those they said were absolutely sure things failed to occur. At this point you might have decided against a career as a pundit, but remember many pundits manage to make a living, and some pundits make a very good living.

Tetlock was able to classify his pundits into two classes that he called hedgehogs and foxes, The Greek poet Archilochus had written, “The fox knows many little things, but the hedgehog knows one big thing.” Hedgehogs believe in Big Ideas, in governing principles about the world that behave as though they were physical laws and underlie most every interaction in society. Hedgehogs tend to be specialized, stalwart, stubborn, order-seeking, confident, and ideological. These are all traits that make hedgehogs weaker forecasters.

On the other hand, foxes are scrappy creatures who believe in many little ideas and in taking a multitude of approaches towards a problem. Foxes are multidisciplinary, adaptable, self-critical, tolerant of complexity, cautious and empirical. These are all traits that make foxes better forecasters. “Better” is used in a relative context as the overall performance was quite poor.

So, would you rather be a fox or a hedgehog? Hedgehogs tend to be much more popular on TV talk shows as they are strong spoken and sure in their beliefs. They tend not to equivocate, even though the issues are complex and they are quite likely to be wrong. This is likely a contributing factor to the polarization of society. In his 1970 book, Future Shock, Alvin Toffler predicted future technology would lead to the polarization of society. This is one of the mechanisms by which the polarization occurs.

© Douglas Griffith and healthymemory.wordpress.com, 2014. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Douglas Griffith and healthymemory.wordpress.com with appropriate and specific direction to the original content.