Correcting for case-mix shift when developing clinical prediction models.

Background: When developing a clinical prediction model (CPM), a case-mix shift could occur in the development dataset where the distribution of individual predictors changes, potentially affecting model performance. This study exploits the case-mix shift that is already observed in the development dataset to address the case-mix shift between the development and deployment phase of a CPM.

Methods: We propose a Membership-based method to correct for case-mix shift in the development phase of CPMs. This method uses a probabilistic similarity metric to re-weight data samples in the source set (before the case-mix shift) to more closely match the target set (after the case-mix shift), assuming the target set reflects the target population. We apply the proposed method in a real-world dataset of myocardial infarction patients with out-of-hospital cardiac arrest within 90 days as the outcome. We design nine scenarios (including case-mix shift and no case-mix shift with a range of target/source sets sample sizes) to explore the impact on predictive performance of CPM developed with the proposed method in comparison to CPMs developed by either using all data samples but ignore the shift, or only using the most recent data. We report calibration and discrimination on development and 200 bootstrap samples.

Results And Conclusions: The proposed method shows promise in accounting for case-mix shift when developing a CPM, particularly when the target set sample size is insufficient. In a partial case-mix shift scenario with an insufficient target sample size, the Membership-based model achieved an optimism-adjusted calibration slope (c-slope) of 0.98, outperforming other models. Conversely, when the target set sample size is sufficient, the Unweighted model on target data only had an optimism-adjusted c-slope of 0.95, compared to 0.92 for the Membership-based model. In complete case-mix shift cases, the Membership-based and Unweighted on target data only models performed similarly. Both achieved an optimism-adjusted c-slope of 0.77 with insufficient target sample size, and optimism-adjusted c-slope of 0.94 with a sufficient target sample size. Further investigation and testing are needed, as well as accounting for other types of data distribution shift to improve model fit for the latest distribution shift in the development dataset.