We address these questions by examining data collected during several geopolitical forecasting tournaments. What makes a good forecaster? Is it someone who is highly knowledgeable regarding the particular subject being forecasted? Someone who excels at probabilistic reasoning and inference? Perhaps a combination of both? Or perhaps neither, and forecasting is a unique skill unto its own. Unlike strictly intellective tasks, driven entirely by domain-specific knowledge, or judgmental tasks that relate to preferences, values and opinions, forecasting features a combination of intellective and judgmental processes ( Reference Fan, Budescu, Mandel and HimmelsteinFan et al., 2019 Reference Stael von HolsteinStael von Holstein, 1970 Reference Wallsten and BudescuWallsten & Budescu, 1983). Human forecasting is a unique psychological process. While data-driven approaches to prediction have grown in popularity, human forecasting remains important, especially in data-sparse settings.

Finally, we demonstrate the predictive validity of these results on out-of-sample data, and their utility in producing performance weights for wisdom-of-crowds aggregations.įrom ancient oracles to modern prediction markets, humanity has long been fascinated by the prospect of predicting future events. When past performance information is limited, psychometric traits are useful predictors of future performance, but, as more information becomes available, past performance becomes the stronger predictor of future accuracy. By parsing the data at different time points during the competitions, we assess the relative importance of each information source over time. Our results suggest these methods perform better at assessing forecasting skill than simpler methods employed by many previous studies. Next, using methods adapted from classical test theory and item response theory, we model latent forecasting skill based on the forecasters’ past accuracy, while accounting for the timing of their forecasts relative to question resolution.

Our first analysis, which compares psychometric traits assessed prior to forecasting, indicates intelligence consistently predicts accuracy. But how does one identify skilled forecasters effectively? We address this question using data collected during two seasons of a longitudinal geopolitical forecasting tournament. A growing body of research indicates that forecasting skill is a unique and stable trait: forecasters with a track record of high accuracy tend to maintain this record.

Beyond Binary by Lee Mandelo