Stability of Clinical Prediction Models Developed Using Statistical or Machine Learning Approaches скачать в хорошем качестве

Stability of Clinical Prediction Models Developed Using Statistical or Machine Learning Approaches

Clinical prediction models are a hot topic. They use statistical or machine learning methods to estimate an individual’s risk of a particular health outcome, conditional on their values of multiple predictors (features). In this talk, I present research undertaken (paper forthcoming) with Prof Gary Collins (University of Oxford), to raise the concern that many models are developed using small datasets that lead to instability in the model and its predictions (estimated risks). I define four levels of model stability in estimated risks moving from the overall mean to the individual level. Then, through simulation and case studies of statistical and machine learning approaches, I show instability in a model’s estimated risks is often considerable, and ultimately manifests itself as miscalibration of predictions in new data. I recommend researchers should always examine instability at the model development stage and propose instability plots and measures to do so. This entails redeveloping the prediction model (repeating exactly the same development steps as used originally) in each of multiple (e.g., 1000) bootstrap samples and deriving (i) a prediction instability plot of bootstrap model predictions (y-axis) versus original model predictions (x-axis), (ii) a calibration instability plot showing calibration curves for the original model’s predictions in the bootstrap samples; and (iii) the instability index, which is the mean absolute difference between individuals’ original and bootstrap model predictions. Case studies illustrate how these instability assessments help inform a model’s critical appraisal (risk of bias rating), fairness assessment (e.g. by comparing different ethnic groups) and further validation requirements. Hope you find it useful!