Identifying long-term conditions in New Zealand general practice using structured and unstructured data: a cross-sectional study

Introduction

Multimorbidity (MM), the presence of two or more long-term conditions in an individual, is associated with reduced quality of life,1 increased healthcare needs, hospitalisations and premature mortality.2–4 In 2017, an MM index, M3, was developed and validated for 1-year mortality risk from all adult New Zealand (NZ) residents using 61 long-term condition categories from routine hospital admission data.5

This index estimated that MM affected approximately 8% of the NZ general population, with around 91% of affected individuals aged 65 years or older.2 International evidence indicates a strong link between socioeconomic factors and MM. For instance, systematic reviews indicate that lower education is associated with a roughly 64% increased risk of MM, while individuals in the lowest income bracket are over four times more likely to develop MM compared with those in the highest income bracket.6 Moreover, findings from Scotland, based on primary care records, demonstrate that the onset of MM occurs 10–15 years earlier in the most deprived populations, with deprivation particularly linked to MM involving mental health disorders.7 However, because the M3 index relies on International Classification of Disease 10th edition Australian Modification (ICD-10-AM) diagnostic coding following hospital admissions, it under-represents long-term conditions that are primarily managed in general practice settings such as dietary controlled diabetes mellitus, gout, dementia or angina.8

For this study, ‘general practice data’ refer to the electronic health records (EHRs) maintained by NZ general practices. These records capture comprehensive patient information through both structured coding systems (mainly Read codes9) and unstructured free-text entries. Recent analyses using general practice EHRs have highlighted these limitations. For example, a study of 454 367 general practice patients in NZ10 found that 24% of individuals diagnosed with ischaemic cardiovascular disease by their general practitioners (GPs) had had no corresponding hospital admission. Similarly, data from the QRISK3 GP database in the UK revealed that 23% of angina and 55% of transient ischaemic attack outcomes were solely documented in GP records.11 While extraction of structured fields can be performed accurately and consistently, manual extraction from unstructured free-text is labour-intensive, prone to human error and presents significant challenges, especially in large-scale studies of diverse chronic diseases.12–14

We aimed to develop a robust procedure for extracting both structured and unstructured data from general practice EHRs in NZ. This approach will facilitate research comparing MM representation between secondary and general practice settings, quantifying MM in the community and investigating the association of MM with cardiovascular disease outcomes.

Methods

We investigated the classification of long-term conditions in a large cohort of general practice patients. This involved identifying relevant Read codes (the clinical coding system used in NZ general practice to document patient diagnoses and procedures) for each ICD-10-AM diagnostic code within the 61 long-term condition categories of the M3 index.5 Additionally, we developed a programmable text-based classification procedure to extract conditions associated with non-standard Read codes and free-text documented in general practice EHRs.

Implementing the categorisation strategy

To preprocess data, (a) all text entries were converted to lowercase to enable case-insensitive searches; (b) uninformative entries such as ‘not applicable’ or ‘unknown’ were removed as unnecessary whitespace; (c) special characters were removed, except for question marks, a minus sign (ie, −ve representing ‘negative’), and the pound sign (#, representing a fracture). Numbers within the text were retained.

Excluding the 4500 records that had already undergone keyword detection, a random sample of 1000 preprocessed text records from groups 1, 2 and 3 was taken (figure 1). Code was developed by YCC in R statistical software to perform computer-generated classification of M3 condition categories based on entry terms and negation rules. This programmatic classification procedure will be referred to as the ‘R_M3Text_Classification’ in this and future publications.

Figure 1

Request permissions

Text-data mining and validation processes (*Information extracted from the patient management systems (PMS) underwent simultaneous review and classification by both software and a clinical expert).

The accuracy of the classification of free-text entries into M3 categories was assessed by comparing its output with manual classification performed by a clinical expert (SW). In each iteration, the classification process involved evaluating how well the identified keywords and negation phrases corresponded to the M3 categories, with attention to cases where anatomical context was relevant. For example, entries with ‘aneurysm’ were initially categorised under ‘aortic and other aneurysms’ (eg, dissection of aortic aneurysm), but if they were combined with ‘cerebral’, they were reclassified under ‘cerebrovascular disease’ (eg, cerebral artery aneurysm). This review process was iterated 10 times, with a new random sample of 1000 entries reviewed in each iteration. As new keywords, synonyms, abbreviations, spelling mistakes and negation phrases were identified, they were incorporated into the classification model. Any questions or uncertainties about the meaning of a text entry or corresponding M3 categorisation were resolved by clinical consensus (SW and KP).

To evaluate the performance of the ‘R_M3Text_Classification’ against manual classifications, we used standard metrics after each iteration: sensitivity (true positive rate), specificity (true negative rate), positive predictive value (percentage of positive results that are true positive) and F₁-score. The F₁-score represents the harmonic mean of positive predictive value and sensitivity and serves as an indicator of result quality, with 1 indicating optimal performance and 0 indicating poor performance. We aimed to optimise sensitivity without compromising specificity. All methodologies were performed using R Statistical Software V.4.3.1 (the R codes are available on request).

• Sensitivity: TP/(TP+FN), where TP is true positive, FN is false negative.

• Specificity: TN/(TN+FP), where TN is true negative, FP is false positive.

• Precision (or positive predictive value): TP/(TP+FP).

• F₁-score: (2×precision×sensitivity)/(precision+sensitivity).

Results

A total of 623 475 patients, aged ≥18 years and enrolled in ProCare PHO as of 1 January 2014, were included in the study (median age 45 years (IQI 32–58); 53% female). Following the patient management extraction process, this number was reduced to 374 071 patients (60% of the original cohort; median age 46 years (IQI 33–58); 55% female), due to technical challenges related to the variability of PMS and differing coding systems. Deidentified individual patient data from the disease classification portion of the PMS were investigated, yielding a total of 7 154 762 PMS entries (an average of 19 entries per person). These records accounted for 99.6% of the total records and were characterised by the presence of Read codes, non-standard Read coding and/or free-text notes in the disease classification fields (figure 2).

Of the 7 154 762 entries with information in the disease classification portion of the clinical record, 24.1% (n=1 726 168) were identified as either non-standard or incompletely coded or stored as unstructured free-text notes (groups 2 and 3).

There were 224 888 records that were only unstructured free-text, with each sentence containing a maximum of 25 words and a median of 3 words (IQI 1–4). Cohen’s kappa for the inter-rater reliability of assessing keywords was 0.95 (95% CI 0.92 to 0.97), 0.90 (95% CI 0.87 to 0.93) and 0.99 (95% CI 0.98 to 1.00) for groups 1, 2 and 3, respectively, indicating a high level of agreement among the assessors’ judgements.

Assessment of negation terms revealed five dominant patterns: (1) preceding terms that appear before disease findings, (2) following terms that appear after disease findings, (3) pseudo-negation terms that indicate negation but actually represent double negatives (eg, not ruled out), (4) ambiguous phrasing (eg, possibly, unlikely) and (5) specific terms that only act as negation indicators for particular disease conditions. An example in figure 3 shows sentences that may be used to describe colitis as a bowel disease, including typical negation terms.

The ‘R_M3Text_Classification’ procedure provided the sensitivity, specificity, positive predictive value and F₁-score for each testing iteration (table 1). The sensitivity of identifying patients with one or more long-term conditions was lowest in the first iteration at 72.6% (95% CI 64.6% to 79.7%) and improved with all subsequent iterations (mean 90.2% (95% CI 88.6% to 91.9%)). Specificity ranged from 98.6% (95% CI 97.5% to 99.3%) to 100% (95% CI 99.5% to 100%), consistently surpassing the sensitivity in all iterations. The positive predictive value ranged from 92.3% (95% CI 86.6% to 96.1%) to 100% (95% CI 98.0% to 100%) and F₁-score from 81.5% to 94.7%. The primary reason for incorrect classifications was simple spelling errors in free-text notes.

The prevalence of 61 M3 conditions in the cohort of 374 071 patients was 42.1% based solely on Read coded data. After incorporating the text-based information using the procedure developed in this study, the prevalence of M3 conditions increased to 46.3%. Not considering text-based classifications would misclassify 15 972 patients as not having an M3 long-term condition category and 12 626 patients as having no MM.

Discussion

This study highlights the importance of incorporating unstructured free-text data from general practice EHRs to provide a more accurate representation of the burden of long-term conditions. Our analysis, which represents the first large multipractice NZ study of MM in general practice, revealed a substantial number of multimorbid patients whose conditions were not captured in Read coded data. By augmenting structured data with free-text analysis, our ‘R_M3Text_Classification’ procedure improved the identification of M3 index conditions, achieving very high specificity, sensitivity, positive predictive value and F₁-score.

The sensitivity of the first iteration was 72.6%. This initial value was primarily due to overdetection of conditions that were acute or short term, related to family history, or resulted from routine screening. For example, people with gestational diabetes, a family history of gynaecological cancer, negative HIV or breast cancer screening tests and previous benign breast biopsy results were often misclassified as having a long-term condition. To improve sensitivity while minimising false positives, the programme was refined to accurately identify when keywords indicated conditions were not long term. The highly variable nature of free-text, particularly in a database sourced from multiple clinicians, meant that it was not possible to identify all permutations of phrasing. Despite these challenges, our methodology achieved consistently high specificity—minimising false positives—which is crucial for accurate condition identification in general practice.

Two clinical experts collaboratively developed key terms, resolved ambiguities and refined the classification rules, while our statistical expert translated those decisions into logical programmable steps. This iterative process has demonstrated that even a relatively straightforward text classification approach can yield robust performance without the resource-intensive requirements of advanced natural language processing (NLP) techniques.

Our findings align with similar studies that aimed to harness unstructured free-text data from EHRs to enhance disease classification and management. Recent research15–18 employing advanced NLP and machine learning techniques has successfully analysed unstructured patient medical records, enabling effective identification of patients with specific diseases. However, these methods often necessitate extensive computational resources and large datasets for training, which can limit their applicability in resource-constrained settings. In contrast, our study demonstrated that a more straightforward text classification approach, such as the ‘R_M3Text_Classification’ procedure, can yield high performance metrics without the complexities of NLP techniques, particularly in general practice free-text data. In line with recent work by Hossain et al,19 which highlighted challenges with misclassification due to ambiguous phrasing and negation in free-text entries, our study reinforces the notion that while sophisticated algorithms can enhance accuracy, simpler methods can also be effective when tailored to the specific characteristics of the data. These findings suggest that our approach may serve as a practical alternative for general practice settings, where clinician time and resources are often limited.

While our study yielded promising results, it is important to acknowledge limitations such as dependence on the quality of free-text records and lack of specific keywords for some conditions, particularly when the condition descriptions are broad. For example, it was difficult to identify keywords for venous insufficiency and uncomplicated hypertension that were specific enough to avoid false positives. Another limitation arises from the presence of pseudo-negations, such as double negatives (eg, not ruled out) or ambiguous negations, which can pose challenges for the procedure and potentially lead to misclassification.

While challenges remain, including dependency on the quality of free-text records and the difficulty of capturing all phrasing variations, our study provides a practical alternative for general practice settings. For example, consider a patient whose coded records indicate atrial fibrillation and coronary disease. However, on incorporating free-text entries from the same records, we identified that the patient had also developed heart failure—a condition not captured by structured data alone. This case exemplifies the broader implications of our approach, demonstrating that augmenting structured data with free-text analysis enhances MM detection and provides a more accurate representation of patients’ health status. Studies such as Owen et al20 have shown that the temporal sequence of MM diagnoses can significantly affect life expectancy, underscoring the clinical importance of capturing the full disease burden for optimal decision-making. Integration of our procedure into the PMS could improve medical documentation precision, enhance clinical decision-making and ultimately support better patient outcomes and health policy decisions. External validation across diverse patient populations is needed to further assess the applicability and effectiveness of this approach. This would provide valuable insights into its functionality in different general practice settings.

Conclusions

Unstructured clinical data from NZ general practice records is a valuable addition to the identification of patients with multiple conditions that may not be readily apparent in structured data. Manually extracting information from these records is time-consuming and prone to human error. We have developed a programme classification procedure that efficiently and consistently extracts information from extensive clinical notes for 61 long-term conditions. The approach not only provides a more accurate identification and count of adults with MM but also facilitates future research on MM and outcomes by eliminating the need for manual review of free-text data. Future research can build on this methodology by exploring how the improved identification of MM affects treatment outcomes and long-term health trajectories.