The Statistical Methods for Health Equity Series is a monthly online series co-hosted by the Data Science for Health Equity community, the Alan Turing Institute Health Equity Interest Group, and the Department of Statistical Science at University College London.

---

We are excited to announce the next instalment of our webinar series with speaker Amanda Coston (UC Berkeley).

Title: Evaluating Predictive Models Under Selection Bias and Covariate Shift

Abstract:

Understanding how a prediction model will perform in a new environment before deployment is essential to preventing harm when algorithms inform decision-making. Two common sources of model performance degradation are (i) covariate shift, where the target covariate distribution differs from the source, and (ii) selective labels, where the observability of outcomes depends on historical decisions. We study model evaluation under the joint presence of covariate shift and selective labeling. In particular, we present a double machine learning procedure for estimating the target risk of an arbitrary black-box prediction model under a general loss function. We show identification of this estimand under standard assumptions and derive a bias-corrected estimator based on the influence function of the target risk. Finally, we evaluate our proposed estimator in controlled synthetic experiments and semi-synthetic experiments using the eICU database, showing that it tracks the true target risk more accurately than methods developed to address either selective labels or covariate shift alone, as well as baselines that combine standard plug-in approaches.

This is joint work with Annie Ulichney.

Speaker:

Amanda Coston is an assistant professor of statistics at UC Berkeley. Her research addresses real-world data problems that challenge the validity, reliability, and equity of algorithmic decision support systems and data-driven policy-making. Her work draws on techniques from causal inference, machine learning, and nonparametric statistics. 

She earned her PhD in machine learning and public policy at Carnegie Mellon University and subsequently completed a postdoc at Microsoft Research on the Machine Learning and Statistics Team. She also holds a Bachelor of Science in Engineering from Princeton in computer science and a certificate in the Princeton School of Public and International Affairs.

--

Please direct any questions about the webinar series to info@datascienceforhealthequity.com.