Although inconsistencies among anesthesiologists have been demonstrated in the American Society of Anesthesiologists (ASA) physical status classification system, one study from the State University of New York (SUNY) Downstate Medical Center, in New York City, has found that agreement between anesthesiologists and nonanesthesiologists is exceedingly weak.
These discrepancies are particularly evident in patients with higher ASA physical status scores, which may open the door to more adverse events.
According to Diana N. Romano, MD, currently a resident at the Icahn School of Medicine at Mount Sinai in New York City, although the reliability and predictability of the ASA physical status classification have occasionally come under fire, it remains a standard communication among anesthesiologists. Nonanesthesiologists and nurses also use the scale in their patient assessments before sedation outside the operating room. Since higher ASA scores tend to come with a greater likelihood of adverse events, some institutions limit non–anesthesiologist-directed sedation to patients with low scores; new computer-assisted sedation devices designed for use by nonanesthesiologists also are limited to patients with an ASA physical status score of I or II.
To help determine the level of agreement between nonanesthesiologists at SUNY Downstate and two anesthesiologists blinded to their counterparts’ ASA status assignments, the researchers had all participants review the records of 768 adult patients undergoing procedural sedation at the institution between 2007 and 2012. Data were tabulated for each case, including ASA physical status, diagnosis, procedure, age, comorbidities and adverse events.
Interrater Reliability Not Good
As Dr. Romano reported at the 2015 annual meeting of the ASA (abstract A2103), nonanesthesiologists tended to score patients much lower than the two anesthesiologists, both of whom were board certified and had at least five years of experience.
The interrater reliability of ASA physical status by nonanesthesiologists compared with each of the two anesthesiologists ( weighted <03BA>) revealed no meaningful evidence of absolute agreement. “The interrater reliability between the first anesthesiologist and the nonanesthesiologists was only 0.02 [95% CI, 0.00-0.04],” Dr. Romano explained. “And between the second anesthesiologist and the nonanesthesiologists, it was 0.04 [95% CI, 0.01-0.06].”
Most telling among the various differences observed were those instances in which the nonanesthesiologists assigned a score of I or II and anesthesiologists assigned a score of III or IV. “ The differences here are pretty large,” Dr. Romano explained. “There were 221 cases where nonanesthesiologists gave a II and an anesthesiologist gave a IV. And there were 21 cases where an anesthesiologist gave a IV and nonanesthesiologists gave a I.
“It’s even more pronounced when we look at ASA IVs alone,” she added. “ Out of the 768 cases, nonanesthesiologists only ascribed a total of 17 ASA IVs. But looking at the same information about the same patients, the first anesthesiologist ascribed a total of 365 IVs, while the second anesthesiologist ascribed 310 ASA IVs.” (Tables 1 and 2).
Table 1. Distribution of ASA Scores for Nonanesthesiologists Versus Anesthesiologist 1.a
Nonanesthesiologists ASA I ASA II ASA III ASA IV Total
ASA I 1 8 25 21 55
ASA II 1 34 239 221 495
ASA III 1 8 79 111 199
ASA IV 0 0 5 12 17
Total 3 50 348 365 766
a Numbers underlined along the diagonal indicate agreement. ASA, American Society of Anesthesiologists.
Table 2. Distribution of ASA Scores for Nonanesthesiologists Versus Anesthesiologist 2.b
Nonanesthesiologists ASA I ASA II ASA III ASA IV Total
ASA I 9 8 24 14 55
ASA II 21 37 251 188 497
ASA III 4 12 84 99 199
ASA IV 0 1 7 9 17
Total 34 58 366 310 768
b Numbers underlined along the diagonal indicate agreement. ASA, American Society of Anesthesiologists.
Of note, the study also revealed low interrater reliability between the two anesthesiologists, a finding consistent with previous studies. Indeed, the weighted <03BA> was only 0.32 (95% CI, 0.27-0.39). “Most of the discrepancies were observed between ASA scores of III and IV, which could probably have been predicted,” Dr. Romano explained. “But when we pooled scores III and IV, the weighted <03BA> rose to 0.5, which is considered moderate agreement. It’s not necessarily something to be proud of, but it’s surely better than 0.02 and 0.04.”
Sedation-related adverse events—unarousable sleep, desaturation, apnea, need for intubation and/or positive pressure ventilation, severe change in heart rate/blood pressure or the use of a reversal agent—were rare, documented in only seven cases. Of these, four were scored lower by nonanesthesiologists than anesthesiologists; two had the same score between all parties; and one was scored lower than the first but the same as the second anesthesiologist.
How To Fix: Education and Examples
“We know the ASA physical status classification system has fairly poor reliability among anesthesiologists,” Dr. Romano noted. “However, when you compare anesthesiologists to nonanesthesiologists, it’s pretty abysmal, at least in this particular study. And we think this suggests a need for focused education for nonanesthesiologists performing moderate sedation, especially with the emergence of these new computerized sedation systems, which require lower ASA physical status for safe use.
“We would really like to do a prospective trial to further characterize the relationship between discordance in ASA physical status and adverse events,” she added.
Erin Hurwitz, MD, assistant professor of anesthesiology and pain management at the University of Texas Southwestern Medical Center in Dallas, did not find the results particularly surprising. “The ASA physical classification system has been shown to have poor interrater reliability between physician anesthesiologists who use it every day,” Dr. Hurwitz said. “So the fact that nonanesthesiologists have poor reliability is not surprising. However, I didn’t anticipate the extent to which the results would be discordant.”
Where the results become concerning, Dr. Hurwitz continued, is in how they may affect patient safety. “It’s very common for nonanesthesiology physicians to use the ASA physical classification system as a way to determine eligibility for moderate sedation,” she explained. “But if these physicians tend to underestimate patient comorbidities, there may be situations where patients are put at undue risk.”
Preventing potential adverse events begins with focused education efforts, particularly with respect to recent changes in the way ASA physical status classifications are described. “I would venture to say that the majority of nonanesthesiology physicians are aware that the ASA has recently approved examples for each physical status classification,” Dr. Hurwitz said. “And I think that educating them that these examples exist will likely improve reliability, because it takes some subjectivity out of the assignment.”