graphic

Imagine that your C-suite senses that complaints from the OR are becoming too frequent and want this addressed. They “ask” you if you would put money at risk by surveying their top-five high-volume surgeons when your group services 50 surgeons and proceduralists. Would you agree? How would you make an argument against it?

During hospital contract renewal, addressing gaps in care and service may come to the forefront. The challenge is that we may not be the sole driver, and we need to demonstrate that to our hospital partners. We might, however, be in the unfortunate position of not being able to say no. Nevertheless, there are avenues to navigate these situations, and we will explore a data-driven approach to evaluating and executing these requests.

Cliches are cliches because they are true more often than not. Such is the case with, “it starts with the data.” There are three issues: what is our plan to capture the necessary change, how much data do we need to reflect it, and how do we deal with outliers that may skew our results?

On the surface, asking “how can we capture change” seems straightforward, yet there are really multiple questions. How confident are you that what you are measuring will reflect performance? Who is documenting: Is it self-reported? Will the hospital use secret shoppers? Are we sampling or measuring every instance? How many charts will be reviewed each month? In what cadence will the data be shared? Outcome measures such as length of stay, infection rates, or satisfaction measures rely on either the hospital or an outside vendor and can be cumbersome from an acquisition and analysis perspective, affecting turnaround time. In general, data is shared either monthly, like on-time starts or turnover time, or quarterly. Once shared, the other party should have a defined limit of time to review and either accept or dispute the data. These all need to be defined contractually.

With Merit-based Incentive Payment System measures, we have the infrastructure to measure a whole population, but what if you must rely on sampling? What size sample do I need to feel comfortable that my measurement is reflecting what is occurring in my population?

Let’s look at what CMS uses for HCAHPS (Hospital Consumer Assessment of Healthcare Providers and Systems). They require a minimum of 300 survey samples over a 12-month period for publicly reported hospital ratings, which works out to 25 responses a month (asamonitor.pub/4eeIbYk). This gives you an idea what the government considers “statistical precision” and what you might consider as a minimum monthly sample size for your institution. Another consideration is to use any number of online calculators, such as Survey Monkey, to determine your sample size for your population estimate for a given standard error of the mean (asamonitor.pub/3yRNyfP). Recall that standard error of the mean (SEM) is how close your sample mean will be to your population mean – the smaller the better (asamonitor.pub/3xddNgj). There are no hard and fast rules on what is an acceptable SEM, as this varies by industry and context, but at least you can have an idea what level of reliability you can expect for your sample size. Keep in mind that we are talking about business metrics, and these guides are nowhere near the scientific rigor required for randomized controlled studies.

So, to our original question about surveying the top five surgeons of a population of 50. You plug in the population and sample size with a 95% confidence limit, and you get an SEM of 43%! That is extremely large for any industry, and the opinions of your top five surgeons aren’t likely to reflect your level of service to all of the staff.

What are outliers, and how do you deal with them? Data entry errors? Data that doesn’t belong in the population? True “black swan” events? (asamonitor.pub/3z1Eg0M). There are four options in dealing with outliers: verify that it wasn’t a data entry error, keep it, remove it, or replace it (asamonitor.pub/3z4whju). It’s important to review any preexisting data, looking for outliers and querying the data acquisition and entry process. You can certainly keep the value, but as there may be money at risk, an extreme data point might shift the favor in one direction. One way you could keep it and lessen an outlier’s impact is to use a geomean instead of an arithmetic mean. A geomean is where you multiply “n” numbers by each other and then take the nth root (asamonitor.pub/3VCu5ZK). This is more commonly used in economics and finance but can be useful for length-of-stay metrics (asamonitor.pub/3VzXXWz). Regarding removing data points, have a consensus on acceptable data points with your hospital partner. An unexpected, prolonged length of stay or a 2.5-hour turnover time due to staff or supply shortages can obscure your improvement efforts. As far as replacing values, that is beyond the scope of this article, and it is suggested you consult a more comprehensive source. These are but a few ways of dealing with outliers for business metrics, so do your research and choose a method both you and your hospital partner are comfortable with.

What about benchmarks? How do we quantify success? In 2012, OR Manager published OR efficiency benchmarks from McKesson’s OR Benchmark Collaborative, which can be used for multiple efficiency measures (OR Manager 2012;28:1-5). CMS publishes Quality Payment Program benchmarks yearly; however, it’s difficult to find anesthesia-related metrics that aren’t topped out and usable (asamonitor.pub/3XfkIQT). So, what are your options for defining success/failure? “Guessing” what an improvement looks like is a consideration, but you might be too high and unachievable or too low and inadvertently guaranteeing success. Keep in mind that as there is a potential exchange of money, metrics that were viewed as either too easy or impossible to achieve may be problematic during review for compliance and could be construed as a kick-back.

Another option is to consider benchmarking against your current performance. There are no fixed standards, but typically one would compare year to year as this will reduce the effect of seasonality. Once you have examined the data and dealt with outliers, one can either use a half or one standard deviation reduction/improvement as a goal, depending on the magnitude of the spread. These are targets based on prior performance and can easily be justified if being scrutinized.

Let’s look at first case on-time starts (FCOTS) as an example. Michigan State University recently did a retrospective review of FCOTS at a level II community teaching hospital and is a great source of data for us to interpret (Spartan Med Res J 2022;7:36719). In their analysis, surgeons are the prime driver of case delays (57%), followed by preop delays at 18% and room delays at 13%, for a total of 88% of delayed cases. Anesthesia delays only accounted for 7%, and patient factors accounted for 5%. One could argue why we would take on the risk of something being driven by factors out of our control. The reality is that you may not have a choice. Why are on-time starts so important to our hospital partners? Often, on-time starts are a key performance metric that’s reported at the board level as an indicator of OR efficiency. Someone’s (the CEO, COO, CNO?) bonus or job may be on the line, so “no” may not be an option.

How would you navigate this? First off, is there really a problem? According to the 2012 OR Benchmark Collaborative, the median FCOTS was 64.3%, and the 90th percentile was 88.3%. Gather your data, determine the extent of the problem, and delineate the root causes. You want to make sure you are taking on a task that you can impact. Pareto charts are useful tools in identifying and demonstrating the magnitude of root causes and can be meaningful to someone who may not be familiar with the intricate workings of an OR.

If in your case late starts are driven by surgeon arrival, with a small contribution from your department, you still might not be off the hook.

One option includes participating in a clinical practice improvement project. Use your Pareto chart to demonstrate that this is a three-department problem – namely, nursing, anesthesia, and surgery. A “three-legged stool” analogy works best here, meaning if the three legs aren’t working together, it’s not going to function as designed. These departments need to work in concert to affect meaningful, lasting change. The other option is to parse out verified anesthesia delays from a year’s worth of data and set either one or a half standard deviation reduction as an improvement goal. These are some strategies and a palatable alternative to “no.”

It doesn’t always have to be a conflict. When meeting with the administration, ask them about their top four or five initiatives and how you can help. There might be opportunities that neither has thought of and that demonstrate your department’s desire to be collaborative. Ensure that you analyze the data and make your case for impacting change while resisting the urge to say “no.”