Scoring multilevel regression and poststratification based population and subpopulation estimates

by Lauren Kennedy, Aki Vehtari, Andrew Gelman

Keywords:

Abstract

Multilevel regression and poststratification (MRP) has been used extensively to adjust convenience and low-response surveys to make population and subpopulation estimates. For this method, model validation is particularly important, but recent work has suggested that simple aggregation of individual prediction errors does not give a good measure of the error of the population estimate. In this manuscript we provide a clear explanation for why this occurs, propose two scoring metrics designed specifically for this problem, and demonstrate their use in three different ways. We demonstrate that these scoring metrics correctly order models when compared to true goodness of estimate, although they do underestimate the magnitude of the score.