Guest post from Linda W. Xu, MD
Neurosurgical Resident, Stanford University Hospital and Clinics
Palo Alto, CA
Online ratings fill the Internet, and it is almost impossible to avoid them. So it is no surprise that for patients facing the difficult decision to have surgery, it is natural to gather information about their surgeon. Until recently, most physician reviews relied on patient opinion and individual experiences. Last year two organizations, ProPublica and Consumers’ Checkbook, released surgeon scorecards that claimed to provide objective data on a given surgeon’s quality. Quality patient care must be a priority for all health care providers, including neurosurgeons, however, these groups’ rankings clearly misled the public. To better understand just what consumers got from these efforts, let us:
- Explain the flawed methodologies used, and
- Test the quality of the data that was reported.
The Methodology
Both websites used data released by the Centers for Medicare and Medicaid Services (CMS) on surgeries performed from 2009-2012/2013. Using this data, both websites attempt to provide an assessment of a surgeon’s volume and complication rate.
Consumers’ Checkbook provides:
- 3, 4, or 5-star rating based on “complication rate” for particular procedures compared to other surgeons, with complication defined as inpatient mortality, prolonged risk-adjusted length of stay, 90-day postdischarge mortality and 90-day readmission;
- Surgeons with above average complication rates (potential 1 or 2-star ratings) have no stars listed; and
- A rating of whether surgeons are low, medium, or high volume on particular procedures compared to their peers.
ProPublica provides:
- A numerical value of how many of a particular type of procedure the surgeon has performed to assess volume; and
- Adjusted “complication rate” for each procedure compared to other surgeons, with complication defined as inpatient mortality, 30-day readmission for a principal diagnosis believed related to the surgery by a panel of reviewers, adjusted by patient age, sex, health score, and whether surgeries were multilevel in the case of spine surgery.
Quality of the Data
To test the quality of data provided patients, we searched over 500 surgeons from top institutions specializing in orthopaedics and spine with the study results published in the Journal of Neurosurgery. Critical findings include:
- 72 percent of surgeons were not listed on Consumer’s Checkbook, and 31 percent had no data listed on ProPublica;
- Only 9 percent of surgeons at top hospitals had low complications listed on ProPublica;
- Only 14 percent of surgeons at top hospitals were given a 5-star rating indicating expected better outcomes; and
- Although the assessment of complication is based on the same data, there was no significant correlation between the number of stars on Consumers’ Checkbook and raw complication rates listed on ProPublica.
A valid assessment of a surgeon’s outcomes must fulfill three basic tenets:
- Use of high-quality data;
- Accurate complication and risk assessment; and
- Adequate sample size.
When looking at the methods for both these organizations, our analysis found that they fail to meet any of these criteria to provide meaningful information for our patients (Click here to read the complete study results). From a patient’s perspective, the ideal surgeon is one with a low complication rate and high volume compared to their peers. It’s telling that in this search of more than 500 surgeons, only four surgeons would have met this criteria. Although providing more information and transparency is a laudable goal, we have to be careful about how misleading data can be interpreted and how this can affect both neurosurgeons and our patients.
