A new technique can better extract correct answers from large groups of people. For a given question, people are asked two things: What they think the right answer is, and what they think popular opinion will be. The variation between the two aggregate responses indicates the correct answer. (Credit: Christine Daniloff/MIT)
Is Philadelphia the capital of Pennsylvania? The answer may surprise you. Even more important, just how many people are surprised might help us figure out what the right answer is, even when most people get it wrong.
New work by CIFAR Advisor Sebastian Seung
(Princeton University) and colleagues may help correct a major flaw in the “wisdom of crowds” technique, which tries to figure out correct answers by averaging the answers of many non-experts.
In 1907, Sir Francis Galton published a paper in Nature that used a weight-guessing competition for an ox to demonstrate the principle of group wisdom. Galton found the median of the 800 ballots was also the most accurate guess. The collective wisdom of the group proved to be more accurate than any individual, even an expert. Since then the technique has been successfully used for determining everything from number of beans in a jar to the results of elections.
Why ‘surprisingly popular’ answers should be correct, illustrated by a simple model of the Philadelphia question with ideal respondents.
But one of the problems with the technique is that non-experts can be systematically wrong about the answers to some questions. For instance, although Harrisburg is the capital of Pennsylvania, most people will choose the more historically significant city of Philadelphia. Seung and his colleagues addressed the problem by asking a second question: What answer do you think most people will choose?
The surprisingly popular, or SP, method is unique because it exploits the popular opinion. The people who said that, yes, Philadelphia is the capital, believe that most people will agree with them. The people who know that the correct answer is Harrisburg, on the other hand, also understand that most people don’t know that.
In one study, respondents judged the market price of 20th century artworks. a, Roshan Houshmand, Rhythmic Structure. b, Abraham Dayan, dance in the living room. c, Matthew Bates, Botticelli e Filippino. d, Christopher Wool, Untitled, 1991, enamel on aluminum, 90′′ × 60′′ ©Christopher Wool; courtesy of the artist and Luhring Augustine, New York. e, Anna Jane McIntyre, Conversation With a Spoonbill. f, Tadeusz Machowski, Abstract #66. Reprinted by permission from Macmillan Publishers Ltd: [Nature] copyright (2017).
By comparing how many people predicted that the most common answer would be ‘yes’ to the actual number of people who answered ‘yes, ’ researchers saw that the ‘no’ vote was the “surprisingly popular” answer – that is, ‘no’ was considered correct much more often than people predicted it would be, and was in fact correct. Overall, the algorithm reduced error by 21.3 per cent relative to the majority vote.
“The SP method is elitist in the sense that it tries to identify those who have expert knowledge,” Seung says. “However, it is democratic in the sense that potentially anyone could be identified as an expert. The method does not look at anyone’s resume or academic degrees.”
Researchers tested the SP method through four studies ranging from surveys to medical diagnoses. In the first study, students were asked true or false questions about U.S. state capitals, the probability of their answer being correct and the percentage of people who thought it would be true. The SP method reduced the number of incorrect decisions by 48 per cent relative to the majority vote. The method was less effective for situations where participants had similar levels of expertise. For example, when dermatologists were asked to diagnose skin lesions there was not a large gap among the group.
Seung and his colleagues acknowledge the value and limitations of democratic methods like Galton’s.
“People are not limited to stating their actual beliefs; they can also reason about beliefs that would arise under hypothetical scenarios. Such knowledge can be exploited to recover truth even when traditional voting methods fail,” the authors write.
“If respondents have enough evidence to establish the correct answer, then the surprisingly popular principle will yield that answer; more generally, it will produce the best answer in light of available evidence.”
“A solution to the single-question crowd wisdom problem,” was published in Nature Jan. 26. It was co-authored by Dražen Prelec and John McCoy from the Massachusetts Institute of Technology.