A paper on the dynamic assessment of mortality prediction models using Statistical Process Control (SPC) methods was presented at the 16th Congress of the International Society of Burn Injuries (ISBI) in Edinburgh.
[This work won the best poster prize of the American Burn Association meeting, Palm Springs, April 2013]
Introduction: Many mortality prediction models have been developed for use to predict outcome in thermally injured patients. Of these the Abbreviated Burn Severity Index (ABSI), Belgian Outcome of Burn Injury (BOBI) score  and Baux Score (BS)  have shown promise in validation studies on independent data. However the performance quality of these prediction models has not been assessed in the midst of changing temporal conditions. Statistical Process Control (SPC), which is widely used in industry to control quality in critical processes, has recently been used to monitor outcomes in surgery and intensive care. This technique has not been used to measure the performance quality of prediction models in burns over time.
Methods: A database of 48,410 acute thermally injured patients admitted to UK burn centres between 2003 and 2011 (inclusive) was constructed from the international Burn Injury Database (iBID). These were chronologically arranged by date of admission into 24 sequential groups to establish a time-sequenced dataset. Each of the groups comprised 2000 patients apart from the last group, which included 2410 patients. The prediction performance of ABSI, BOBI and BS was evaluated over time by applying these models to the 24 time periods. The C-index (AUC on ROC analysis) was used to track the quality of the prediction models.
Results: Twenty-four chronological c-indices for the 3 scoring systems (ABSI, BOBI, BS) were derived. The mean c-indices for the scoring systems were BS 0.952 (95%CI 0.918-0.986), ABSI 0.931 (95%CI 0.892-0.971) and BOBI 0.826 (95%CI 0.812-0.843) with BS providing the best measure of outcome. However X-bar charting with 3-sigma upper and lower limits for the c-indexes showed deterioration of BS with transient loss of control over time, which was not seen with ABSI or BOBI.
Conclusions: This study supports the novel use of SPC to detect significant changes in prediction model performance over time. SPC has the potential to detect changes in model performance that could remain unnoticed using current quality control measures.
The Informatics Group of the NNBC has considered reports concerning various telemedicine projects in burn care from across England and Wales. There are three such projects, two of which are network based and one national.
1) A detailed presentation from the South East network was received concerning the development of a tele-referral system using a secure website on which participating emergency departments would register and be able to send text and images for consideration by the burn care team. This has been piloted in East Grinstead (TRIPS) and it is hoped a roll the project out across the network in a phased fashion through 2012.
2) Discussions are underway in the Northern network concerning the potential using telemedicine in a variety of roles but with no specific projects having been identified. A meeting to consider the options was held in Wakefield in early September and future discussions will be held.
3) As part of the iBID software redevelopment a follow-on project was described at its initiation in 2007 to build into the software the ability to accommodate a tele-referral system. With finalisation the iBID software and distribution in 2012 it is anticipated the tele-referral component would then go into full-time development.
The Informatics Group has undertaken to keep all such projects under review and report their progress to the NNBC.
Further to an invitation from the organising committee of the Australian Royal College of Surgeons, a presentation was given by Ken Dunn to the meeting in Adelaide in which he provided an overview of the discussions underway in the UK about the funding of trauma care, with particular reference to burn services.
Additional discussions were had concerning the nature of the burn injury registries in both the UK and Australasia and the potential for collaborative data sharing and research between the two organisations. It was agreed that such data sharing would be considered in detail once be new version of the iBID software was in use in 2012.
The Informatics Group of the NNBC has considered the first draft of information about burn services that could be made publicly available. This work was initiated at the request of the NNBC and is in line with Department of Health policy and that of Commissioners of Specialised Services. The difficulties of providing information about specialised services that is comprehensive but understandable and meaningful to a non-medical readership poses significant challenges.
It is intended to provide a first draft for NNBC consideration in the autumn, after which a period of consultation with stakeholders will follow. The potential for providing information in a visual manner, in the form of maps was considered, using some of the existing examples:
Further to an invitation from the Central and Eastern Europe Burns Forum, iBID presented information about the ongoing debate in the UK, and elsewhere, concerning the funding of trauma services with particular reference to burn services. The various methods used in various health economies for burn service remuneration and the different costing methodologies in use in Europe were discussed. The difficulties of providing a high cost and low volume service in a financially challenging environment were universally acknowledged.
Some of the profound organisational challenges being experienced by some services were also detailed during what was a most enjoyable and informative meeting. The ability of iBID to provide information in line with its professed aims was applauded by the meeting but the difficulty of collecting standardised information across so many language, cultural and organisational boundaries was appreciated.
Further to advice from the Informatics Group, the NNBC has decided to ask HQIP (www.hqip.org.uk/) to carry out a review of the organisation and function of iBID, with a view to making recommendations concerning its future. This is in order to provide a stable and sustainable registry of burn injury into the future and to meet the needs of specialised commissioning and of the health service in general.
The review is intended to conclude by the summer of 2011 after which consideration will be given to the procurement of a long-term partner to provide the stability of sustainability required. It is accepted that the infrastructure costs of the database will therefore increase. Explicit reassurance has been given that the clinical nature and ownership of the registry will not be lost or weakened.
In January an application was made by the National Network for Burn Care to be the host for a Payment by Results (PbR) Development Site project. The intention the project is to develop an alternative currency for the remuneration of burn care within the NHS.
In March we heard that we have been successful, and that development and expansion of the patient level costing pilot scheme under way in South Manchester is to be supported by the PbR team.
The first meeting was held of the newly created Informatics Group, which is a formal subcommittee of the National Network for Burn Care (NNBC). The terms of reference in this group are to coordinate and advice on all aspects of data collection and analysis pertaining to burn injury and to develop a means of communicating the nature of burn injury to health care professionals and the general public.
Information pertaining to the Informatics Group is available from the downloads section.
A new collaboration has been initiated to learn lessons from the patient level costing pilot being run at the specialised rehabilitation service at Northwick Park Hospital under the leadership of Professor Lynn Turner-Warwick. The intention of the work is to utilise some of the models and methods developed there and translate their usage into the burn care arena. Of specific interest is the rehabilitation dependency scoring system specifically developed to reflect the wide ranging medical and non-medical interventions required for the rehabilitation of patients following complex injury.
The Study Day was held on the 10th of December 2009 at University Hospital of South Manchester and proved to be a useful opportunity to share ideas and aspirations concerning improvements in the current data collection software as well as some of the reports being sent back to burn services. It also proved a useful discussion group for design features going into the next version of the database software. Participants had an opportunity to consider these before the meeting and to discussed them in some detail.
There was broad agreement and consensus that the planned way forward made sense for both the burn services and for the aims and intended uses of the burn database information. Valuable additions and design features were offered and accepted with an indication that areas of further work when necessary, particularly in relation to the recording of surgical activity and aspects of physical therapy outcome measurement.
Other areas covered included the design of the recently reconfigured error and validation report (report E) with an acceptance that the new version of the software should make this report redundant as real-time validation and error checking is to be written into the software. The innovation of updating menu lists automatically from the central server was accepted as a major improvement and a way of keeping the database design contemporary and clinically valid.
A recently produced report was introduced, which is renewed each month and is an indication of the volume of data being transferred by each burn service into the database server. It was emphasised that this is now available to any registered user of the database and gives an indication of the volume of successfully uploaded data. The database currently contains approximately 66,000 records.
Throughout the discussions it was accepted that one of the major drivers for the database design has been the wish to incorporate patient level costing methodology. This will undoubtedly increase the volume of data collection but in doing so make the record a more clinically relevant indication of the care provided.
It was discussed and appreciated that acceptance of the database as a commissioning tool will be a complex matter and based on successful negotiations with a number of groups, including the NHS Payment by Results team and the data team at the NHS Information Centre.
Additionally agreement needs to be reached with Specialised Commissioning Groups (SCG) within the NHS who need to accept the information from iBID as valid commissioning currency. This is expected to be discussed at the next national group meeting in Feb 2010 and subsequently at the SCGs Directors meetings.
Burn service attendees were reminded that all of the reports produced about their services remain available using the service specific username and password previously circulated to clinical leads and available from burn network managers. If any member of staff requiring access to these reports is unable to access them, then contact their network manager for the necessary access rights.