Prospective external validation of the automated PIPRA multivariable prediction model for postoperative delirium on real-world data from a consecutive cohort of non-cardiac surgery inpatients

Source of data and participants

The delirium risk calculated using the PIPRA model was routinely collected, together with the delirium screening results, during a quality improvement project (QIP) in a Swiss, private 335-bed hospital.20 The project included all inpatients aged 60 and above undergoing non-cardiac, non-intracranial surgery with an admission date from 1 May to 30 June 2023. In 2023/2024, the hospital had a mix of 43.0% public and 57.0% private or partially privately insured patients, 79.4% of patients were outpatients and, of the inpatients, 82.4% were elective. The QIP aimed to increase delirium screening, treatment and prevention.

Comparison of validation to original development data

The development data originated from an IPDMA of clinical studies18 19 21 performed in several countries, whereas the validation data for this study consisted of real-world data collected as part of a QIP in a single private hospital in Switzerland. The eligibility criteria in this QIP were generally broader than those in the studies captured by the IPDMA. The delirium outcome was also measured heterogeneously in the IPDMA data, with only one study (contributing less than 10% of patients) that used the Delirium Observation Screening Scale (DOSS) as a diagnosis tool, although in combination with Diagnostic and Statistical Manual of Mental Disorders (DSM-IV) criteria. In the QIP presented here, the DOSS was used as the primary diagnostic tool, in combination with any routine diagnosis performed by the physician according to DSM-V criteria. In the development dataset, the predictors were collected specifically to investigate predictors of POD, while here they were collected during the routine preanaesthesia consultation.

Outcome

The outcome predicted by the model was POD, which was defined as delirium occurring up to 7 days after surgery. In this dataset, a patient was deemed delirious if they had a score of 3 points or more for the DOSS or had the ICD-10 diagnosis code for delirium for their hospital stay. The DOSS is a 13-point checklist for the identification of delirium.22 The data were collected in routine clinical practice, with delirium risk present in the electronic health record of the patient and viewable by the nurses performing the DOSS. The nurses were not informed about the validation project and were instructed to perform the DOSS on all patients as part of the QIP. Compliance with DOSS screening was assessed by comparing the observed number of DOSS screenings to the expected number. We anticipated one DOSS screening per patient per shift, with three shifts per day. Thus, compliance was calculated as a percentage:

Predictors

All PIPRA variables (age, body mass index, American Society of Anesthesiologists Physical Status Classification System (ASA) score, number of prescribed medications, cognitive impairment, history of delirium, surgery risk, laparotomy/thoracotomy, optional preoperative C reactive protein (CRP) value) except CRP were routinely collected. CRP is optional and therefore was only used where available.

Predictors were not collected specifically for the delirium risk. Instead, they were all part of clinical routine and were collected by the anaesthesiologist during the preanaesthesia consultation.

In addition to the main predictors, we recorded patient sex and the Self-Care Index (SPI) for further characterisation. The SPI, assessed daily, measures self-care abilities (eg, hygiene, mobility, elimination), producing a total score from 10 (maximum impairment) to 40 (full self-care).23

Sample size

A sample size was not calculated since it was a QIP. However, with a POD incidence of 10% and a targeted area under the curve (AUC) of 0.75, a 95% CI for the AUC would have a width of 0.12 with the 866 subjects enrolled in the study. We consider this to be precise enough to provide meaningful information about model performance.

Missing data

Missing data were only imputed for PIPRA predictors. Following the use proposed in the development paper, mean/mode imputation was used. When CRP was not available, the PIPRA submodel without CRP was used for prediction rather than imputing CRP, as in the development paper.

Statistical analysis methods

Subject and procedure characteristics were summarised by median (IQR) or frequency (percentage), dependent on data type. Differences in clinical parameters across groups (eg, POD/no POD) were explored using t-tests or χ² tests, accordingly. All analyses were performed using R Statistical Software (V.4.3.2).24

This external validation was performed using data completely independent of the model development. To compute the predicted risk on the new data, the following equation was applied: p=1/(1+exp(−lp)), where p is the predicted risk and lp is the linear combination of the individual predictor variables multiplied by the log odds coefficients (including the intercept).

The participant and procedure characteristics were summarised and compared across the development and validation studies using the same methods as for comparing across POD. They were also visualised using violin plots or bar charts.

Both model discrimination (AUC) and calibration (calibration-in-the-large, calibration slope, calibration plot) were assessed on the validation data.

Risk groups

Patients were stratified into low (PIPRA <10%), intermediate (PIPRA from 10% to 19.9%), high (PIPRA from 20% to 34.9%) and very high (PIPRA >35%) risk groups as per the original publication.18 Patients with high risk or very high risk have more chance of developing delirium than the average elderly population. For this population, it is essential that some preventive perioperative measures are taken. An exploratory analysis was added, in which a stratified table was created based on risk group. The number of delirious patients was identified, and the number of potentially preventable cases was estimated, assuming prevention effectiveness rates reported in the literature.17

Participants

5279 patients were admitted during the timeframe of the validation study. Excluding patients under 60 years of age, outpatients and non-surgical or cardiac surgery patients, 866 patients remained (figure 1). The characteristics of the patients showed worse outcomes for delirium patients, including on average sixfold more time spent in the intensive care unit (ICU), twice the length of stay and threefold more nursing time (table 1). Delirium patients had, on average, a 10-fold higher mortality; however, only seven patients died in this cohort. Patients experiencing POD were generally older, with more cognitive impairment, a greater history of delirium, less able to take care of themselves (lower SPI), with higher number of prescribed medications, higher CRP values and higher ASA status (table 1).

While the participants in the development study and this validation did not display clinically meaningful differences in age, BMI or history of delirium, they did differ in other ways (figure 2, online supplemental table S1). In the validation, fewer patients underwent higher risk or open procedures, and the patients themselves were more frail (higher ASA). All patients in the validation study were on at least one medication.

The mean compliance to delirium screening was 60.6%. 79 patients (9%) received no screening, and these patients were younger, with lower ASA, smaller surgeries and lower length of stay (online supplemental table S2). Anecdotal feedback suggested the night shift had the lowest compliance, though we were not able to retrieve this level of data due to ethical considerations.

Overall, data were mostly missing for CRP (table 1). However, this was anticipated in the model design, and the submodel without CRP was used for these patients. Missingness was also relatively high for cognitive impairment, history of delirium and number of medications (table 1).

Model performance

The validation analysis included 866 eligible subjects observed from May 2023 to the end of the study. Of these, 100 subjects (11.5%, 95% CI 9.6% to 13.8%) were identified as having POD. According to the risk groups created at development, 59.2% of subjects were considered low risk, 21.6% intermediate risk, 10.4% high risk and 8.8% very high risk (figure 3a).

At external validation, the PIPRA model was found to discriminate well between those with and without POD (AUC=0.77 (95% CI 0.72 to 0.82)). There were no major violations to calibration, although there was a tendency to slight overprediction for higher risk patients (figure 3b,c). The diagnostic accuracy was dependent on the chosen threshold and is displayed in detail in table 2. For example, at the threshold of 10% or ‘medium’ risk, the sensitivity was 74% and specificity 64%.

Potential effect

An exploratory analysis was performed to estimate the preventative potential. This analysis suggested that three-quarters of delirium cases could be prevented, under the assumption of a constant prevention effect from the literature, and when focusing exclusively on patients classified as medium risk or higher (online supplemental table S3).

Limitations

The QIP was not a clinical trial and, therefore, there were no sample size calculations, and there were strict limitations on exploring the data in depth due to ethical considerations.

While adherence to delirium screening protocols meets acceptable standards for clinical practice (91% of patients were screened), higher compliance is anticipated in research settings, with only 61% of the total expected screenings performed. The DOSS is simple and does not require direct patient interaction; it has limitations in both sensitivity and specificity.25 To mitigate this issue, we provided nurses with training to accurately identify and diagnose delirium, stressing that the DOSS is a mandatory supplementary tool. Additionally, we emphasised in the training that the tool is prone to missing hypoactive delirium.

The risk prediction model was originally developed on data collected by IPDMA. In the development data, delirium was predominantly assessed by CAM. Only one study used the DOSS; however, it was used in conjunction with DSM-IV criteria. Differences in setting, outcome measurements and patient characteristics between the development and validation data could have caused the model to underperform; however, the model performed well, and this further confirms that the risk prediction model is robust.

Since the validation was performed in normal clinical practice, there was no blinding of the nurses to the prediction. While they were not involved or informed of the validation project, it is still possible that this introduced bias.

Strengths

Despite the limitations discussed above, we see the model performing well on real-world data, showing the model to be robust. This was expected, since the original model was built on a diverse dataset of IPMDA data. To the best of our knowledge, this is the only POD risk prediction model that has been built on IPDMA data and so far the only POD risk prediction model that is CE-marked and approved for clinical use.

A key strength of using real-world data is the broader demographic representation for subpopulations often under-represented in trials. All patients were included in the study, and there was no bias introduced through a consenting process, where often those patients who are most vulnerable are lost (eg., those with low education or mild cognitive impairment—known risk factors for delirium).19 However, more than half the patients were privately insured, which is not representative of most hospitals.

Interpretation

The AUC from this external validation was excellent in comparison to other external validations for POD prediction. In the original study, a pilot external validation on 359 patients showed an AUC of 0.74, which is close to the 0.77 observed here. A project undertaken by Wong et al, using a dataset of 292 patients to test various delirium prediction models head-to-head revealed C-indices ranged from 0.52 to 0.74, where even the highest scoring algorithm was below the AUC of 0.77 observed here.26 Strikingly, in that review, the highest self-reported AUC from the assessed models was 0.94;27 however, this depreciated to 0.61 in the external validation. This illustrates the importance of external validation on new patient cohorts from different settings. It also compares favourably to a risk prediction model using only age, in both discrimination and calibration (online supplemental figure S1).

The use of real-world data has opposing strengths and limitations to the use of data from clinical trials. There was a preliminary external validation in the original study,18 and a clinical trial to validate the algorithm further is underway, which will complement this study well.

Implications

Identification of patients at risk of delirium is essential for effective, targeted prevention strategies and early treatment of delirium. The PIPRA tool was developed as a consensus tool, using data from many authors and a number of large studies through a collaborative IPDMA, and this real-world validation study confirms that we have a robust and reliable POD risk prediction algorithm.