Combining Experts’ Judgments: Comparison of Algorithmic Methods using Synthetic Data

Abstract

Expert judgment (or expert elicitation) is a formal process for eliciting judgments from subject-matter experts about the value of a decision-relevant quantity. Judgments in the form of subjective probability distributions are obtained from several experts, raising the question how best to combine information from multiple experts. A number of algorithmic approaches have been proposed, of which the most commonly employed is the equal-weight combination (the average of the experts’ distributions). We evaluate the properties of five combination methods (equal-weight, best-expert, performance, frequentist, and copula) using simulated expert-judgment data for which we know the process generating the experts’ distributions. We examine cases in which two well-calibrated experts are of equal or unequal quality and their judgments are independent, positively or negatively dependent. In this setting, the copula, frequentist, and best-expert approaches perform better and the equal-weight combination method performs worse than the alternative approaches.

Replaced by

James K. Hammitt, and Yifan Zhang, “Combining Experts' Judgments: Comparison of Algorithmic Methods”, Risk Analysis, vol. 33, n. 1, January 2013, pp. 109–120.

Reference

James K. Hammitt, and Yifan Zhang, “Combining Experts’ Judgments: Comparison of Algorithmic Methods using Synthetic Data”, TSE Working Paper, n. 12-293, March 2012.