A paper on the dynamic assessment of mortality prediction models using Statistical Process Control (SPC) methods was presented at the 16th Congress of the International Society of Burn Injuries (ISBI) in Edinburgh.
[This work won the best poster prize of the American Burn Association meeting, Palm Springs, April 2013]
Introduction: Many mortality prediction models have been developed for use to predict outcome in thermally injured patients. Of these the Abbreviated Burn Severity Index (ABSI), Belgian Outcome of Burn Injury (BOBI) score  and Baux Score (BS)  have shown promise in validation studies on independent data. However the performance quality of these prediction models has not been assessed in the midst of changing temporal conditions. Statistical Process Control (SPC), which is widely used in industry to control quality in critical processes, has recently been used to monitor outcomes in surgery and intensive care. This technique has not been used to measure the performance quality of prediction models in burns over time.
Methods: A database of 48,410 acute thermally injured patients admitted to UK burn centres between 2003 and 2011 (inclusive) was constructed from the international Burn Injury Database (iBID). These were chronologically arranged by date of admission into 24 sequential groups to establish a time-sequenced dataset. Each of the groups comprised 2000 patients apart from the last group, which included 2410 patients. The prediction performance of ABSI, BOBI and BS was evaluated over time by applying these models to the 24 time periods. The C-index (AUC on ROC analysis) was used to track the quality of the prediction models.
Results: Twenty-four chronological c-indices for the 3 scoring systems (ABSI, BOBI, BS) were derived. The mean c-indices for the scoring systems were BS 0.952 (95%CI 0.918-0.986), ABSI 0.931 (95%CI 0.892-0.971) and BOBI 0.826 (95%CI 0.812-0.843) with BS providing the best measure of outcome. However X-bar charting with 3-sigma upper and lower limits for the c-indexes showed deterioration of BS with transient loss of control over time, which was not seen with ABSI or BOBI.
Conclusions: This study supports the novel use of SPC to detect significant changes in prediction model performance over time. SPC has the potential to detect changes in model performance that could remain unnoticed using current quality control measures.