Expert Elicitation & Collective Intelligence

Structuring expertise and aggregating judgement transparently.

Individual judgements spread along a scale, with disagreement kept visible A horizontal scale carries a scatter of markers. Most cluster at one point, the aggregate; a second, looser group sits further along, and one marker stands well apart from both. A dashed span covers the whole range of judgements, and a dashed ring holds the separate group, so the disagreement is shown rather than averaged away.

At a glance

Produces
Structured judgements with their uncertainty attached, and an account of where informed people disagree.
Depends on
A quantity defined precisely enough to be estimated, and contributors whose calibration has been examined.
Fails when
The aggregation rule is chosen to manufacture a consensus the contributors do not hold.

Expert elicitation is the systematic collection and combination of expert judgement under conditions of incomplete evidence. Collective intelligence extends this to the question of how groups form, weight and revise judgements together - and how disagreement is preserved rather than smoothed away.

How we approach it

The value of elicitation comes from structure. Anything else produces an anecdote with a sample size.

  • Elicitation is designed against the specific quantity required, whether that is a probability, a range, a ranking or a qualitative judgement.
  • We work on calibration and on the acknowledgement of uncertainty. Experts who never say "I do not know" are not being helpful.
  • Aggregation is treated as a methodological choice with consequences. We report how different weighting and combination rules would change the result.
  • Dissent is documented rather than resolved. Minority judgements are often the most informative thing an elicitation produces.