What’s Measured Matters: Measuring Performance in Anaesthesia

Br J Anaesth. 2015;114(6):869-871.

Authors: F. Kiernan; D. J. Buggy

Delivering value for money is an increasingly insistent demand in health policy, driven by financial constraint, increased patient expectations, an ageing population, and more expensive technologies. Health performance measurement is central to ensuring that the healthcare we provide is of value to both those who pay for it, and those who use it. It improves health systems, by collecting information on population health, thereby allowing the appropriate allocation of resources based on these levels of health, and provides the stimulus to improve the quality of healthcare provided. Health Performance Measurement is a necessary part of ensuring that the health system is accountable to its citizens.

The concerns voiced over the role of performance measurement in healthcare, rarely relate to the idea of that clinical data should be collected in order to ensure a minimum standard of care. Instead, clinician concerns are focused on the choice of indicators that is used to assess this quality of care, and how the results should be used. Our surgical colleagues have been at the forefront of debates regarding the public reporting of results of surgical performance measurement. Measuring clinician performance is now commonplace in many high-income countries and the public reporting of anaesthesia-related outcome measures is sure to follow.

Part of our hesitation in measuring and comparing the performance of anaesthetists as physicians and healthcare providers, is because of a long standing belief that high quality data cannot be collected in large enough amounts, to allow accurate comparison. The collection of either small amounts of high quality data, or large amounts of poor quality data, is known to lead to inaccuracies and potential gaming. However, our ability to collect accurate and appropriate measures of performance has changed with the advent of Big Data. Big Data allows us to collect data from heterogeneous datasets, integrate this data, and use predictive analytics to determine most efficient and effective means of care. As a result of major advances in digital technology, we can analyse and compare cost and clinical effectiveness data across jurisdictions and over time, perform accurate risk-adjustment, and deliver the results to those who use them. The obstacles to high quality data collection are no longer technological.

Perhaps the two remaining barriers to accurate, relevant data collection are uncertainty over choice of indicators that best reflect patient outcome, and optimum use of that data. When incentives, either financial or reputational, are attached to an indicator, it is perceived as more important than unmeasured aspects of care. The indicator subsequently receives a disproportionate amount of attention from the provider, a phenomenon known as ‘what’s measured is what matters’. Evidence from primary care studies has shown a decrease in quality for patients with asthma and heart disease, whose care was not associated with an incentive. This highlights that delivering appropriate care requires that we measure what matters to patients, rather than merely the data that it is easiest to collect. While data on process indicators in anaesthesia, (e.g. the administration of anti-emetics, application of thermoregulatory devices, the performance of an airway assessment etc.), may be easy to collect, they matter less to patients than the true outcome measures of a normal recovery, no morbidity and absence of vomiting and pain. Indeed, qualitative data on patient reported outcome measures, have demonstrated that postoperative vomiting is ranked as the highest concern for patients undergoing ambulatory surgery, more so even than postoperative pain.  It would seem that emesis might be the ‘Holy Grail’ of outcome measures – it has been validated as a relevant patient outcome, and is easy to collect for a large population of patients. However, to take a single example of how performance measurement data may be abused, reporting vomiting outcomes on a patient group undergoing ambulatory surgery only takes into account this low-risk group, who would be expected to have low mortality, low morbidity, and no hospital stay. In reality, a systematic review of 108 commonly used anaesthesia quality indicators, found that only 40% were truly validated.

If outcome data instead focuses on the high-risk groups, then mortality, morbidity, delayed discharge, and unplanned ICU admission would seem to be suitable alternative measures of performance. Analysis of postoperative outcomes for the elderly suggests that the risk of myocardial infarction, stroke, delirium, and pulmonary complications can be decreased through quality improvement methods. However, a natural hesitation in using these outcome measures, rests on evidence that they may not have a strong relationship with the quality of care provided. Indeed the ‘signal-to-noise’ ratio for mortality has been shown to be too low for it to be used as a marker of quality. Furthermore, as health is multidimensional, and in-hospital care is multidisciplinary, mortality cannot be related to the performance of an individual anaesthetist, or team of anaesthetists. Indeed mortality is often related to factors beyond the control of clinicians, including the level of education of nursing colleagues and the management structure of the hospital. In addition, there is a lack of consensus even among anaesthetists regarding the appropriateness of these measures. While unplanned admissions to intensive care after surgery, have been described by some as being an inappropriate way of examining quality others have suggested that they are an accurate means of measuring performance and improving quality.

The second obstacle concerns how these data are used. Berwick’s framework for quality improvement describes how the public reporting of outcome data leads to improvements in quality from one of two methods – either patients select better providers of care, or the data provides information on areas of underperformance, leading to a stimulus for improvement from the deficient providers. This assumes, however, that the care provided is actually amenable to improvements in quality. The OECD estimates that only half of deaths from ischaemic heart disease can be considered amenable to healthcare, while death from hypertensive diseases can be prevented by improvements in quality within the entire healthcare system. Attaching a high-powered incentive to these outcome measures, could result in perverse outcomes. The BMA demonstrated that attaching incentives to waiting times in Emergency Departments had a negative effect on care in other areas of the hospital, including the cancellation of emergency lists. Misrepresentation of data is a known feature of healthcare systems that rely on the public reporting of data. While Gaming, (defined as the influence of mortality by deliberately altering variables other than clinical quality), has rarely been described in anaesthesia, 41% of the reported reductions in surgical mortality, in the New York Cardiac Surgery Reporting System (CSRS) were because of gaming rather than improvements in care. ‘Cream-skimming’, (i.e. the refusal of treatment to high risk patients, who are more likely to have poor outcomes), is another potential mis-use of healthcare performance measurement, with the CSRS also finding that 62% of cardiac surgeons refused to treat higher-risk patients, because of reputational risks from public reporting of potential adverse outcomes.

Providing information on performance measures, without adequate explanation of the context, is a flawed means of providing accountable healthcare, and indeed is little more than a reward-punishment system for clinicians. Comparisons between the public and private sectors have found that publication of raw healthcare performance measurement indices, has a particularly perverse effect on the public sector, by adversely affecting the professional attitude of providers leading to a destruction of the patient-doctor relationship.

Furthermore, while the provision of data in the form of a league table is a response to the belief that providing patients with more information will help them make better decisions, in reality, information asymmetry in health care, (i.e. the discrepancy in knowledge that exists between patients and clinicians) means that patients are unlikely to be able to distinguish between relevant and irrelevant information. Particular attention should be paid to the needs of those who are least likely to use league tables to make choices – the elderly, recent immigrants and those with lower educational levels. Clinicians should not only be involved in determining the indicator used to assess their performance, but also the manner in which this is communicated to the public and their patients.

Ultimately, high quality data collection, performance measurement, and appropriate reporting of that performance measurement are necessary to ensure continued improvement in accountable, equitable healthcare outcomes. In addition, an eagerness to report these measures publicly may demonstrate that the healthcare system is committed to a culture of accountability and transparency. However, improvements in the healthcare system, both in quality and resource allocation, should be based on clinical assessments, and accurate data collection, rather than political decisions. ‘What’s measured matters’, therefore we need to pay close attention to indicators that matter to most to patients, but which are also preventable. Efficient data collection must accurately reflect performance, and must include measures to decrease perverse outcomes such as up-coding, or recording patients as being higher risk than their co-morbidities suggest, and cream-skimming.

The care provided by anaesthetists is complex, as are our patients. Therefore the analysis of our care should not be considered to be any less complex. While process level data may be easy to collect, it provides little information on the effect of treatment on patients. In the era of ‘big data’, we can move towards the collection of outcome measures using accurate information, enabling us to assess outcomes that are potentially preventable, and that matter to patients.