Building Learning Healthcare Systems for Critical Care Medicine

A SCOPUS database search was conducted by medical library staff and one author for articles outlining how to implement a learning healthcare system in critical care. A variety of terms were utilized with learning health to narrow the search specifically to the ICU setting. The specific search terms were as follows:

(TITLE-ABS-KEY (registry OR database OR datamart) W/5 TITLE-ABS-KEY (creat* OR introduc* OR implement*) AND TITLE-ABS-KEY (“critical care” OR icu OR “intensive care unit”)), (TITLE-ABS-KEY (learning AND health) AND TITLE-ABS KEY (registry OR database OR datamart) AND TITLE-ABS KEY (creat* OR introduc* OR implement*) AND TITLE-ABS-KEY (“critical care” OR icu OR “intensive care unit”))

The search was then expanded to all articles with mention of “learning health” and those published in the Learning Health Systems journal. Titles and abstracts were screened for relevance for learning healthcare systems in general, followed by learning healthcare systems in critical care medicine, with a focus on frameworks and implementation strategies. A narrative review of relevant articles was carried out following the planned search strategy. Despite a broad search (1,487 articles identified on “learning health” and 202 in Learning Health Systems), there were no articles that addressed the framework and implementation of a learning healthcare system in the critical care space specifically, but articles were identified outside the field of critical care medicine, and focused on theory or frameworks, specific aspects of data collection and curation, process improvement and implementation strategies, stakeholder engagement, and measures of success.

The learning healthcare system model is driven by the idea that data generated from an electronic health record can be utilized to better inform clinical knowledge and decisions in real or near-real time.  Larger-scale learning healthcare systems tend to utilize the electronic health record as their primary data source. Data elements in an electronic health record are diverse in their types, generated from various medical devices, information technology systems, and manual input by personnel. In their raw form, while voluminous, these data elements can be incomplete and may not be stored in a format that is immediately suitable for analysis (e.g., a Glasgow Coma Scale score entered manually into a flowsheet or as free text in a note). This poses challenges when building a data infrastructure, since desired elements should be easily accessible to end users in order to seamlessly interpret and extract knowledge. An understanding of how each data element in an electronic health record is generated is important for understanding the quality, accessibility, and ability to utilize those data in a learning healthcare system, particularly because these data were originally generated for a clinical or administrative purpose, rather than for the sole purpose of a learning healthcare system. This necessitates a thorough data management strategy mapping the entire life cycle of data—from collection to utilization—to allow the flexibility and speed that are necessary for the learning health cycle. Additionally, mechanisms to maintain security within the data infrastructure are essential to protect protected health information that is extracted from the electronic health record.

The initial step of the learning health cycle is data to knowledge, where internal and external data are systematically gathered and evaluated by stakeholders. For collection of internal data that are created within a health system, appropriate informatics systems must be in place, as discussed in the previous section. Once an appropriate informatics system allows curation of data from an organization’s electronic health record system in a validated fashion, understanding of key data elements from these systems, as well as application of rigorous analytic methods, serve as cornerstones for moving from data to knowledge for a learning healthcare system.

For internal data from within a health system, ascertainment can be from various systems including the organization’s administrative systems resulting in claims data (International Classification of Diseases [ICD] diagnosis, procedure codes, diagnosis-related groups, and health system charges) and electronic health record data. While ICD diagnoses and procedure data are not generated and collected for learning health purposes, they can serve as a relatively easy-to-use source of discrete data, with validated definitions of diagnoses, procedures, and comorbidity summaries such as the Elixhauser, Charlson, or van Walraven scores.  Additional sources of claims data are hospital charges, which reflect billings for the variety of patient care delivered including operating room utilization, inpatient bed utilization, pharmacy charges, and any supplies that are used for patient care during their hospitalization. Both ICD and charges are codified within hospital claims forms, such as the UB04 form which are submitted to payers for reimbursement for hospital services.

In addition to claims data, electronic health record data may allow the ascertainment of the most granular data. Data can include clinical characteristics, vital signs, laboratory data, pharmacy data, imaging, flowsheets, processes of care data, and clinical outcomes. While electronic health record data may add significant granularity to understand a patient’s hospital experience, the volume of data (especially for critically ill patients) can be large, with resultant complexity in data cleaning and computational power needed for analyses. While these data sources serve as a rich repository of patient characteristics, care delivery, and clinical outcomes, all of these sources can be subject to missingness and misclassification and care must be taken to make sure that data are fully validated in order to avoid the analysis of data resulting in biased inferences. Domain expertise is required to understand the workflow that led to data generation, potential sources of noise, and incorrect conclusions.

In addition to collection of internal data, the data to knowledge step also involves collection of data and results of studies external to an organization. External data refers to peer institutional data, as well as the vast biomedical research data created through international research enterprises and published in the peer-reviewed literature. The choice of external data must be rigorous, with close evaluation for bias, causal inference, and feasibility or applicability to the local culture. In our learning healthcare system (Duke Critical Care Medicine Collaborative, Durham, North Carolina), we choose studies that have high practical relevance, are well-conducted, and have the potential for minimal bias (such as large, multicenter randomized-controlled trials in critical care medicine, meta-analyses of randomized control trials with minimal heterogeneity, and observational research employing strong causal inference methods).

After data collection, curation, and analysis, information presentation is the next important step to disseminate newly generated local and external knowledge. Stakeholders in the critical care space vary in their needs: (1) high-level information comparing key metrics across ICUs (hospital-acquired infections, mortality, and so forth) within a health system are generally needed for hospital leadership (i.e., hospital president, chief medical and nursing officers, health system quality improvement leadership); (2) more granular information within an individual ICU (processes of care, patient outcomes, healthcare utilization) is generally appropriate for individual unit leaders (i.e., ICU medical and nursing directors); and (3) additional information on evidence-based practice (i.e., rigorous local and external studies) is needed by front-line critical care providers and staff (i.e., attending physicians, trainees, advanced practice providers, nurses, therapists, dieticians). Information presentation must be individualized for the relevant stakeholder (hospital administration, ICU leadership, individual providers, and allied healthcare personnel), and may include crude data, basic risk adjustment, and high-level causal inference analyses. Importantly, reports must be interpretable, be actionable, and show trends over time.

After the data to knowledge step, the next step in the learning health cycle involves moving to knowledge to practice. This critical step moves beyond rigorous data collection and presentation, and involves concepts from social and organizational psychology, implementation science, and behavioral economics to transition from simply observing the data to engaging with the data and implementing changes in clinical practice. An initial step is setting up the appropriate forums (committees, provider meetings, conferences, and so forth) at multiple levels (health systems leadership, local unit leadership, and frontline providers) for data engagement and discussion. Examples in critical care medicine include detailed evaluation of reports relating to various processes of care across ICUs (i.e., tidal volumes per predicted body weight, plateau pressures, blood product utilization, laboratory or imaging utilization, high-cost pharmacologic therapies, hemodynamic management, glucose control, hospital acquired infections, and so forth). Based on these data streams, priority areas are defined through a multidisciplinary process of relevant stakeholder engagement, regular data review, identification of gaps or improvement opportunities, identification of local resources, and detailed strategic planning.

After data engagement and discussion, established systems can help with implementation of changes in response to the data. Systems such as Six Sigma, Plan-Do-Study-Act, Lean, and Theory of Constraints have all been applied to healthcare settings to provide a framework for the continuous improvement cycle within learning healthcare system.  While each has a different area of focus, such as decreasing unwanted variability in the case of Six Sigma and Plan-Do-Study-Act, reducing waste in the case of Lean, and increasing throughput in the case of Theory of Constraints, they all provide a system of steps to prioritize problems, identify their root causes, create change, and measure the effect. A key feature of learning health, measurement, requires an analytics infrastructure, which is discussed separately. These systems describe iterative steps that can be used to manage change within a single environment.

While applicable to health care, many of these systems were originally designed for manufacturing, and do not intrinsically provide methods to address resistance to change in complex, interprofessional environments. There are many tools from the field of implementation science that can be used by stakeholder groups to investigate these challenges to adoption or dissemination. One such tool is the Theoretical Domains Framework, which was originally developed to investigate barriers to implementing evidence-based practice among healthcare professionals. It includes 14 “domains” related to behavior change such as knowledge, skills, social or professional role and identity, beliefs about capabilities, and beliefs about consequences. While more detailed study methods from qualitative research can be used, this tool can also be used in less formal settings. Specifically, change leaders can select elements from the Theoretical Domains Framework domains to guide interviews, focus groups, or surveys to identify barriers and facilitators to behavior change at the individual and group level. 

An effective learning healthcare system will be attuned to factors that influence motivation. While the Theoretical Domains Framework does include domains that affect motivation, such as goals, social influences, and emotion, the field of behavioral economics offers additional tactics to overcoming resistance to change. Two effective strategies include selecting defaults, in which the default behavior is made to be identical to the desired behavior, and social reference points, in which healthcare providers are given feedback on their performance in the context of their peers. While these apply to the electronic health record, their principles can be used in nondigital applications as well.

After the knowledge to practice step, the final step in the learning health cycle involves moving to practice to data. This important step aims to study implementation of new interventions within a health system and provides an opportunity for rigorous analysis of outcomes, in order to help guide decisions around (1) wider implementation of beneficial interventions, (2) deimplementation of nonbeneficial interventions, and (3) refinement of particular interventions. This evaluation generally involves concepts from population health research methods, including epidemiology, health services research, health economics, and pragmatic clinical trials to generate results that support a causal inference that the applied intervention led to improved clinical outcomes.

Observational methods are utilized to evaluate results and support causal inference, through analyses of real-world data generated from a health system. One method to exploit the heterogeneity of intervention implementation across units in a health system is through the use of natural experiments, which are defined as exogenous events not controlled by researchers.  By using these exposures as “as if” randomizing events, selection bias may be reduced.³⁰  To enhance the evaluation of natural experiments (above and beyond pre–post descriptive evaluations), additional approaches to evaluation can be employed including interrupted time series  segmented regression and differences in differences methods employing unexposed units as “control” groups.  In addition to natural experiments, causal inference using observational data can be enhanced through use of directed acyclic graphs for mapping causal pathways comprehensive risk adjustment, and the use of propensity score methods.  While the previously discussed analytic methods help support a causal inference quantitatively using observational data, it must be appreciated that drawing a causal inference also involves a qualitative exercise of the existing data using frameworks such as the Bradford–Hill criteriawhich examine (among others) strength of association, consistency, specificity, appropriate temporal sequence, dose–response, and biologic plausibility.

Rigorous evaluation of local data additionally allows near real-time analysis of practice patterns and processes of care, which can be used to inform and refine current improvement efforts and provide surveillance for the possible need for new improvement efforts. This approach has been taken to support disease surveillance, decision support, and clinical outcomes research.Additionally, these approaches have led to limiting ventilator-induced lung injury building predictive models for ICU readmission and monitoring ventilator bundle compliance.  Thus, the observational data generated can support multiple methods for surveillance, analysis, and implementation.

In addition to employing rigorous observational methods, a learning healthcare system provides a platform to test hypotheses using novel methods for pragmatic clinical trials. One method that has gained increased interest in the field of critical care medicine in the last few years is the cluster randomized controlled trial design. This type of trial design allows the unit of randomization to be at the unit level (rather than at the individual patient level), thus providing greater efficiency in conducting a pragmatic study across multiple ICUs and negating the need for active individual patient recruitment. Additionally, a well-integrated electronic health record can be leveraged to collect data on the intervention, covariates, and outcomes, without the need for personnel to actively fill case report forms. This strategy has successfully been employed in critical care to study several pragmatic interventions, such as fluid choice and extubation support.  Additionally, this type of design can also facilitate implementation by having all clusters (in this case, ICUs) have the intervention fully implemented at the end of the trial, using concurrent methods such as a stepped-wedged design.

Drawing on available literature and expert opinion based on the experience of the authors, we summarize a framework for a critical care learning healthcare system (fig. 1) and recommendations for building a learning healthcare system for critical care medicine (table 1). These principles involve an integrated and multidisciplinary approach with key considerations for each step of the learning health cycle. To turn data into knowledge, we recommend building an infrastructure capable of aggregating and validating high volumes of data from the desired sources and applying rigorous methods to process and succinctly present that data to stakeholders. To move the knowledge generated into clinical practice, consider the motivations and barriers to change (resistance, education, technology, and so forth) and apply the best-fit framework to guide stakeholders through that transition. Finally, utilize integrated informatics systems as well as high fidelity observational research methods and clinical trials (embedded into daily practice) to track and evaluate new changes. Through each of these steps, all members of the healthcare team should be supported as they become accultured to a learning healthcare system. This will generate new data to inform the decision to refine, discontinue, or expand new processes and continue the cycle of learning health. These principles involve an integrated approach, using the principles outlined throughout this review.

Learning healthcare systems present a new opportunity in the field of critical care medicine for rapid generation and implementation of the best evidence-based findings into daily clinical practice, ultimately aimed at improving patient outcomes. Within this system, it is possible to make use of the massive amounts of data generated in the ICU as well as include patients, providers, researchers, and other stakeholders into the process. There is a significant gap in the literature pertaining to the implementation of such systems in the critical care space, and this represents a significant opportunity for further research in critical care.