Measuring Probability Numeracy

Probability numeracy is the ability of individuals to think in probabilistic terms and use probabilities effectively in everyday life. Measuring probability numeracy helps to understand how well individuals can plan for uncertainty, such as the likelihood of losing a job over the next year, and to identify groups who make suboptimal life decisions because of their limited understanding of probabilities. In this paper, we explore different batteries of questions to measure levels of numeracy. We do so by examining questions asked in the American Life Panel Financial Crisis Survey. We describe methods to create various versions of the numeracy score based on Item Response Theory. We evaluate the performance of alternative scores in predicting the quality of answers to subjective probability questions in the survey. We identify a four-item battery that captures relevant variation in probability numeracy; it takes about 90 seconds of survey time. We show that adding demographic covariates (e.g. gender, age, education, race) significantly improves properties of the score, but some additions such as education pose challenges for comparisons across cohorts or countries because of the differences in the meaning of these measures.