From Narratives to Numbers: Valid Inference Using Language Model Predictions from Verbal Autopsy Narratives

In settings where most deaths occur outside the healthcare system, verbalautopsies (VAs) are a common tool to monitor trends in causes of death (COD).VAs are interviews with a surviving caregiver or relative that are used topredict the decedent's COD. Turning VAs into actionable insights forresearchers and policymakers requires two steps (i) predicting likely COD usingthe VA interview and (ii) performing inference with predicted CODs (e.g.modeling the breakdown of causes by demographic factors using a sample ofdeaths). In this paper, we develop a method for valid inference using outcomes(in our case COD) predicted from free-form text using state-of-the-art NLPtechniques. This method, which we call multiPPI++, extends recent work in"prediction-powered inference" to multinomial classification. We leverage asuite of NLP techniques for COD prediction and, through empirical analysis ofVA data, demonstrate the effectiveness of our approach in handlingtransportability issues. multiPPI++ recovers ground truth estimates, regardlessof which NLP model produced predictions and regardless of whether they wereproduced by a more accurate predictor like GPT-4-32k or a less accuratepredictor like KNN. Our findings demonstrate the practical importance ofinference correction for public health decision-making and suggests that ifinference tasks are the end goal, having a small amount of contextuallyrelevant, high quality labeled data is essential regardless of the NLPalgorithm.