Defining asthma and assessing asthma outcomes using electronic health record data: a systematic scoping review
- Mohammad A. Al Sallakh1,2⇑,
- Eleftheria Vasileiou2,3,
- Sarah E. Rodgers1,4,
- Ronan A. Lyons1,4,
- Aziz Sheikh2,3,4 and
- Gwyneth A. Davies1,2
- 1Swansea University Medical School, Swansea, UK
- 2Asthma UK Centre for Applied Research, UK
- 3Usher Institute of Population Health Sciences and Informatics, The University of Edinburgh, Edinburgh, UK
- 4The Farr Institute of Health Informatics Research, UK
- Mohammad A. Al Sallakh, Data Science Building, Swansea University, Singleton Park, Swansea, SA2 8PP, UK. E-mail: M.A.AlSallakh{at}swansea.ac.uk
Abstract
There is currently no consensus on approaches to defining asthma or assessing asthma outcomes using electronic health record-derived data. We explored these approaches in the recent literature and examined the clarity of reporting.
We systematically searched for asthma-related articles published between January 1, 2014 and December 31, 2015, extracted the algorithms used to identify asthma patients and assess severity, control and exacerbations, and examined how the validity of these outcomes was justified.
From 113 eligible articles, we found significant heterogeneity in the algorithms used to define asthma (n=66 different algorithms), severity (n=18), control (n=9) and exacerbations (n=24). For the majority of algorithms (n=106), validity was not justified. In the remaining cases, approaches ranged from using algorithms validated in the same databases to using nonvalidated algorithms that were based on clinical judgement or clinical guidelines. The implementation of these algorithms was suboptimally described overall.
Although electronic health record-derived data are now widely used to study asthma, the approaches being used are significantly varied and are often underdescribed, rendering it difficult to assess the validity of studies and compare their findings. Given the substantial growth in this body of literature, it is crucial that scientific consensus is reached on the underlying definitions and algorithms.
Abstract
Inconsistent/underreported asthma case definitions are common in research, limiting reproducibility/comparability http://ow.ly/knmc30a0ltp
Introduction
Asthma is in clinical practice a diagnosis based on patient history, examination and objective tests [1]. It is, however, increasingly considered to represent a heterogeneous group of disorders with different phenotypes and endotypes [2]. The clinical definitions of asthma and its key outcomes, including disease severity, control and attacks/exacerbations, have been the subject of vigorous debate [3–8].
Particular challenges arise in the context of epidemiological studies where validated operational definitions are needed [9, 10]. These studies are, increasingly, being undertaken using electronic health record (EHR)-derived data, which adds a further layer of complexity as the use of valid and reliable approaches is essential to ensure the reproducibility of research findings [11].
In order to assess current approaches, we systematically interrogated the recent EHR-based asthma literature. Our specific objectives were to: 1) describe the different methods of defining asthma and assessing disease severity, control and exacerbations in EHR-based studies; 2) investigate whether authors reported on the validity of those methods; and 3) assess their reporting practices.
Methods
We conducted a systematic scoping review based on the Arksey and O'Malley [12] five-stage framework, including identifying the research question, identifying relevant studies, study selection, data charting, and collating, summarising and reporting the results. The research questions were: 1) how were asthma and its key outcomes defined using EHR data in the recent literature, 2) how did authors report on the validity of their EHR-based algorithms and 3) how clearly were the EHR-related methods reported?
Eligibility criteria and search strategy
We searched PubMed using a broad query (supplementary table E1) to retrieve asthma studies that used EHR-derived data and were published between January 1, 2014 and December 31, 2015. The search query was iteratively improved by adding many variations and equivalents of the keywords “EHR” and “routinely collected data” as well as named data sources found in the literature. Only articles written in English were included.
Study selection
We excluded nonrelevant articles by reviewing titles and abstracts, referring to the full text when needed. We included only articles where asthma was a main finding. For the purpose of this scoping review, we limited the concept of EHR-derived data to coded, objective, individual-level data that were generated as a byproduct of routine healthcare.
Data extraction and synthesis
From each of the eligible articles, we extracted and summarised information from the full text and online supplements, including basic bibliography, setting (country) and design; names and types of EHR-derived data sources used; algorithms to identify asthma patients, assess disease severity, assess control and define exacerbation; and how authors reported on algorithm validity. In this context, we referred to “validation” as any attempt to assess the algorithm's concurrent or construct validity. We used the RECORD Statement's 13-item checklist to assess the clarity of reporting of other EHR-related aspects such as clinical code lists used in the algorithms and the implications of using EHR data in asthma research. The RECORD Statement is a recently introduced extension to the STROBE Statement which helps improve the reporting of observational studies conducted using routinely collected data [13]. Supplementary table E2 describes the data extraction and charting tool. Article screening and data extraction were performed independently by two authors (M.A. Al S. and E.V.), with a third author arbitrating (G.A.D.).
Role of the funding sources
The funding sources had no role in study design, in the collection, analysis and interpretation of data, in the writing of the report, or in the decision to submit the paper for publication.
Results
We included 113 articles in the scoping review. Figure 1 shows the study selection process. Most studies were conducted in the USA, Taiwan and Canada (supplementary table E3), and employed longitudinal designs (supplementary table E4). The most commonly used data types were health insurance claims, followed by medical record repositories and dispensing databases (supplementary table E5).
Defining asthma
We identified 66 different algorithms to define asthma under seven diagnostic labels (supplementary table E6).
“Persistent asthma” was defined over 12 and 24 months using the US Healthcare Effectiveness Data and Information Set (HEDIS) criteria [14], which involved assessing for any of the following asthma-related events: 1) emergency department visit, 2) hospitalisation, 3) outpatient visit and two asthma prescriptions or 4) four asthma prescriptions [15–18]; by HEDIS criteria except “four asthma prescriptions” [19]; and by any asthma encounter (hospitalisation or emergency department visit) or using oral corticosteroids (OCSs) for ≥3 days [20].
“Current asthma” was defined by any asthma encounter in the last 3 years [21].
“Current general practitioner-reported and diagnosed asthma” was defined as any asthma encounter in the last 12 months; “current general practitioner-reported, diagnosed and treated asthma” as the same plus any asthma prescription in the same period [22].
Patients with treated asthma were otherwise required to have at least three dispensing events of asthma treatments in three different quarters of the year [23].
“Acute asthma” was defined using any asthma diagnosis codes in emergency department or inpatient data [24].
In the remaining studies, the label “asthma” was defined using various algorithms, some of which were similar to those of the aforementioned more specific labels.
The intervals over which asthma diagnostic/management and prescription codes where queried were specified in 31 and eight studies, respectively. The positions of diagnostic codes in the encounter (i.e. primary or secondary) were specified in 37 studies.
We identified the following approaches in these algorithms: requiring diagnostic/management events, prescription events or both (supplementary table E7). In addition, to exclude likely nonasthma patients, some studies applied additional nonasthma criteria to restrict the study population based on age (supplementary table E8) and/or comorbidities (supplementary table E9).
Assessing asthma severity
18 studies used 20 different algorithms to assess asthma severity (supplementary table E10) as binary (i.e. severe versus nonsevere asthma) [15, 23, 25–38] or ordinal variables (mild, moderate and severe asthma [39]; low-, moderate- and high-risk asthma [40]). The algorithms were based on one or more of the following asthma-related variables: number and/or dosage of prescriptions (i.e. short-acting β2-agonist (SABA), inhaled corticosteroid, OCS and leukotriene receptor antagonist), and number of hospitalisations, emergency department and outpatient visits. Almost all algorithms (n=17) used prescriptions (either alone or with other variables), while one algorithm was based only on hospitalisations and emergency department visits [36]. The intervals over which asthma severity was assessed were 3 [29], 6 [38], 12 [15, 23, 28, 30–32, 34, 36, 37, 39, 40] or 24 months [33, 35], or unclear [26, 27].
Assessing asthma control
Nine studies assessed asthma control using 11 algorithms, in nine of which the interval was 12 months, in one 1–3 months and in the remaining study this was unclear (supplementary table E12). Uncontrolled asthma was defined by a minimum number/dose of SABA prescriptions [30, 31, 39, 41, 42]; any or short-course OCS prescriptions [30, 31, 41–44]; any hospitalisation or emergency department visit with either diagnosis of asthma [27, 30, 31, 41–43, 45] or (in already diagnosed asthma patients) diagnosis of status asthmaticus, pneumonia, dyspnoea or respiratory insufficiency [30]; unscheduled outpatient visits for asthma or lower respiratory tract infections (LRTIs) [31]; and general practitioner consultations for LRTIs requiring antibiotics in asthma patients [31]. Asthma impairment was defined based on the required SABA use, namely an average of more than two salbutamol puffs per day [31]. One study assessed asthma control based on number of OCS and SABA prescriptions per year (without giving any further details about the actual algorithm) [41].
Defining exacerbations
24 studies defined exacerbations using EHR-derived data (supplementary table E11) as a dichotomous variable (absent versus present) [16, 17, 23, 27, 30–32, 35, 37–39, 42–44, 46–54] or stratified into absent, moderate and severe [55]. OCS prescriptions were used as a marker for exacerbations in 17 studies, either alone [23, 30, 31, 35, 39, 42, 47, 48, 53] or with a concurrent asthma encounter (e.g. a general practice, outpatient or emergency department visit, or hospitalisation within 5 or 7 days) [16, 17, 32, 37, 38, 46, 52, 54]. In one study, exacerbations were defined by a minimum of six SABA prescriptions per year [47]. Other definitions included an outpatient code of “asthma exacerbation” [52], asthma hospitalisation [23, 30, 32, 35, 37, 39, 43, 44, 46, 48, 50, 51, 53–55], asthma emergency department visit [16, 30–32, 35, 37, 38, 43, 44, 46, 48, 51–54], or hospitalisation with diagnosis of status asthmaticus or (in already diagnosed asthma patients) diagnosis of pneumonia, dyspnoea or respiratory insufficiency [30].
Clarity of reporting
Overall, the reporting of methodological aspects of using EHR-derived data was suboptimal. The majority of studies presented no information on algorithm validity. Among studies that reported on the validity, we identified 10 practices of reporting or justifying on the validity of algorithms (table 1): 1) performing validation or concordance analysis in the same study against other measures based on different data sources (e.g. medical record review or patient-reported measures); 2) referring to previous validation of similar algorithms in the same or 3) different databases; 4) referring to previous validation of similar algorithms for different diseases in the same or 5) different database; 6) using algorithms “consistent” with previous studies in the same or 7) different databases; 8) using nationally developed algorithms; 9) using algorithms based on clinical guidelines; and 10) relying on previous validation of the database content. Some studies did not provide clear algorithms for asthma severity or control, but only referred to their components [23, 35, 37, 38, 41].
Of the 113 reviewed studies, 40 studies used record linkage, of which 17 mentioned it in the abstract and 28 provided at least some explanation in the full text. The geographical region, time frame of data and types or names of the data sources were mentioned in 83, 91 and 104 abstracts, respectively. 83 studies reported their extent of access to the data sources. The intervals over which the algorithms were applied were often not reported. 111 studies touched on the implications of using EHR data to study asthma. Of these, 64 and 63 studies discussed the risk of misclassification bias and unmeasured confounding, respectively. Six studies acknowledged the possible changes over time in data quality and coding practices, and the entailing changes in case definition eligibility and accuracy. Five studies explained their data cleansing procedures. Finally, no study shared the programming codes of data preparation and analysis.
Discussion
Statement of principal findings
This systematic analysis of the contemporaneous asthma literature found evidence of considerable international activity in using EHR-derived data to study a variety of asthma populations and outcomes. Importantly, we also found wide variations in the approaches used, with limited attention being paid to the validity of the underlying algorithms used and suboptimal reporting of studies. This poses a major challenge to the interpretation and reproducibility of this important, emerging body of research inquiry.
Strengths and limitations
To the best of our knowledge, this is the first systematic exercise to investigate the quality of reporting on EHR-based studies, especially the validity of measures, in the context of asthma. In undertaking this work, we used robust approaches which involved two people independently selecting studies and undertaking data extraction. The findings may also apply to other chronic diseases. This scoping review had no geographic limits, but it was confined to assessing the recent literature. Examining the most recent asthma literature is most likely to provide meaningful insights on current practices. A limitation is that we did not systematically check whether the references provided to support the claimed validity of algorithms in question actually provided sufficient evidence of validity. For example, differences might exist between the algorithms used in a given study and those previously validated.
Interpretation in the light of previous studies
Although EHR-derived data are convenient resources for research, they are originally collected for other purposes, and usually suffer from missing or incorrect data and potential biases [56–58]. In addition, EHR systems usually fail to capture complete and accurate clinical information at the point of care due to design limitations and inefficient use of these systems by clinicians to document clinical data [59, 60].
These issues impose challenges on their use to assess a complex and heterogeneous condition such as asthma. For example, asthma diagnosis codes, which are commonly used solely for patient identification, may be recorded after a trial or wrong diagnosis and do not capture undiagnosed patients [61]. In addition, many EHR-derived databases often lack important variables, such as lung function, indication of dispensed medications, adherence to treatment and lifestyle, which are vital for identifying and assessing asthma patients. These challenges are, however not insurmountable. In this scoping review, we found several techniques intended to improve algorithm accuracy, such as age limitation, comorbidity exclusion and diagnosis position restriction.
Ideally, algorithms should be validated in the databases in which they are used. However, this was often not the case. Instead, using algorithms with only reasonable face validity based on clinical guidelines or clinical judgement is a very common practice in EHR-based studies [62, 63]. These approaches assume that clinical codes in the database accurately represent the patient's actual healthcare events [62].
Underreporting on implementation details and methods' validity compromises transparency and reproducibility, a crucial issue in medical research. It has been previously found that in EHR-based studies, full lists of clinical codes were often not reported [64]. A recent, large-scale reproducibility exercise identified similar challenges due to suboptimal reporting of EHR-based studies, particularly sharing code lists and algorithms [65].
The significant methodological heterogeneity we found in EHR-based asthma assessment algorithms reflects, in addition to the content differences between the databases used, the lack of consensus on the clinical definitions in the first place despite continuous standardisation efforts [5, 6, 66, 67]. The focus of our work was to examine asthma definitions and their validity specifically in the context of EHR, but this highlights the fundamental need to reach consensus on clinical asthma definitions and the appropriate validation of asthma diagnosis. For example, there is still an active debate on whether lung function is essential to establish asthma diagnosis [7, 8]. A recent study also found significant variation in algorithms to assess asthma severity from health insurance data [68]. Unjustified interstudy variation in the operational definitions of the same clinical concepts creates challenges for comparability, meta-analysis and evidence synthesis. These issues have been raised for asthma [69] and other allergic conditions, such as peanut allergy [70, 71] and anaphylaxis [72], where wide variations in findings were potentially attributed to inconsistent case definitions.
Implications for policy, practice and research
This scoping review sheds light on the opportunities offered by the increasingly ubiquitous EHRs, but also highlights considerable heterogeneity and suboptimal reporting of EHR-based asthma assessment algorithms and the implications of these practices on comparability and reproducibility of studies.
Developing reliable algorithms to assess asthma outcomes using EHR data is a nontrivial challenge. In addition, standardising such algorithms across different populations may be impractical since databases differ in content, validity may not hold across different populations and no best practice currently exists [68]. Similar challenges arise when comparing asthma epidemiology between multiple populations [73]. These methodological issues, in addition to suboptimal reporting, should be considered when interpreting and synthesising evidence from geographically dispersed studies.
With the accelerating availability of EHR-derived data and their use to study asthma, we believe that consideration needs to be given to convening an international task force to work on the harmonisation of those algorithms under uniform and consistent clinical labels, while considering the differences between populations and databases. In addition, validation of these algorithms in the respective populations should be given a high priority. Furthermore, to allow more accurate assessment of asthma from EHR data, efforts are needed to improve the capture and coding of asthma-related data at the point of care [74], which requires more efficient EHR systems [59, 60]. In addition, emerging data sources such as patient-generated data and wearables need to be harnessed [75]. Finally, to improve the clarity of reporting on EHR-related methodological aspects, we strongly advocate the adoption of the RECORD Statement as an extension of the STROBE Statement by both authors and journal editors [13]. Optimal reporting should include complete code lists, detailed algorithms and validity assessment. Implications of using EHR-derived data to study a complex condition such as asthma should be clearly communicated to enable judgement of internal and external validity.
In summary, we found that there is considerable international interest in exploiting EHR-derived data to study asthma, but that there are considerable variations in the approaches used. These variations are compounded by suboptimal reporting of methods, which makes it difficult to assess the reproducibility of research. Given the substantial investments taking place in EHRs globally, this body of work is likely to grow significantly in the coming years. It is therefore important that the asthma-interested research community works to place it on a solid footing in order to ensure the quality and reproducibility of this work.
Supplementary material
Supplementary Material
Please note: supplementary material is not edited by the Editorial Office, and is uploaded as it has been supplied by the author.
Supplementary material ERJ-00204-2017_Supplement
Disclosures
Supplementary Material
A. Sheikh ERJ-00204-2017_Sheikh
Acknowledgements
Author contributions: M.A. Al Sallakh, S.E. Rodgers, G.A. Davies, A. Sheikh and R.A. Lyons developed the concept and methods. M.A. Al Sallakh conducted the literature search. M.A. Al Sallakh and E. Vasileiou independently reviewed the studies with G.A. Davies arbitrating. All authors contributed to the development of methods, interpretation of findings and manuscript writing, and critically reviewed and approved the final manuscript.
Footnotes
This article has supplementary material available from erj.ersjournals.com
Support statement: This work was funded by Health and Care Research Wales and Abertawe Bro Morgannwg University Health Board. It was carried out with the support of the Asthma UK Centre for Applied Research (AUK-AC-2012-01). We also acknowledge the support from The Farr Institute of Health Informatics Research. The Farr Institute is supported by a 10-funder consortium: Arthritis Research UK, the British Heart Foundation, Cancer Research UK, the Economic and Social Research Council, the Engineering and Physical Sciences Research Council, the Medical Research Council, the National Institute of Health Research, the National Institute for Social Care and Health Research (Welsh Assembly Government), the Chief Scientist Office (Scottish Government Health Directorates), and the Wellcome Trust (Medical Research Council grants CIPHER MR/K006525/1, Scotland MR/K007017/1). Funding information for this article has been deposited with the Crossref Funder Registry.
Conflict of interest: Disclosures can be found alongside this article at erj.ersjournals.com
- Received January 27, 2017.
- Accepted March 9, 2017.
- Copyright ©ERS 2017