Machine Learning Increases Accuracy of ASA Physical Status Scores

NEW ORLEANS—Interindividual variability in assigning ASA physical status scores to patients may one day become a thing of the past, thanks to the development of a new machine learning algorithm (ML-ASA) that determines the score using widely available patient data, according to new research.

This hospital registry cohort study concluded that by using such data, the ML-ASA can predict 30-day mortality with an accuracy that is superior to that of anesthesiologist-led ASA scores, a finding that made the investigators foresee broad-based applicability of the system.

“When a patient is booked for a surgical procedure, it is usually the surgeon who needs to decide on that patient’s ASA score,” said Karuna Wongtangman, MD, the anesthesiology digital health laboratory chief at Montefiore Medical Center, in New York City. “Some surgeons actually make an assessment of the patient’s ASA physical status, while others do not, and simply input a random number.

“Nevertheless, that ASA score subsequently determines the care and lab tests that the patient will get,” she said. “So, if we can have an automated tool determine ASA physical status, it will help streamline the patient assessment process and ensure that everyone is receiving the appropriate preoperative care and testing.”

Big Disagreement in ASA Scores

The investigators built a machine learning algorithm to derive ASA physical scores and then compared its predictive value for 30-day mortality with anesthesiologist-assessed ASA scores. The study included data obtained between 2005 and 2021 from adult surgical patients presenting at both Beth Israel Deaconess Medical Center (BIDMC), in Boston, and Montefiore Medical Center (MMC).

The researchers first used a random forest model to train the machine learning algorithm with 35 preoperatively available candidate predictors in a set of 361,602 BIDMC patients. The model was further validated using an internal BIDMC data set of 90,400 patients, as well as an external data set of 254,412 patients from MMC. Logistic regression was used to compare the ML-ASA scores with anesthesiologist-derived ASA scores for the prediction of 30-day mortality.

In a presentation at the 2022 annual meeting of the American Society of Anesthesiologists (abstract A1087), Wongtangman reported that there was disagreement between the machine learning model and anesthesiologist-derived ASA scores in 42.8% of total cases. In low-acuity patients, anesthesiologists’ rating of the ASA score was lower than the ML-ASA score in each of ASA I (mean difference, 0.43), ASA II (mean difference, 0.15) and ASA III (mean difference, 0.08) patient groups. On the other hand, the mean anesthesiologist-derived score was greater in high-acuity (ASA IV) patients than it was with the ML-ASA score (mean difference, 0.22) (P<0.001).

It was also found that the ML-ASA algorithm performed comparably to anesthesiologists’ assessment of ASA scores on the specific task of predicting 30-day mortality, with area under the curve scores of 0.85 (95% CI, 0.84-0.85) and 0.84 (95% CI, 0.83-0.84), respectively.

“We think this is good,” Wongtangman said. “At least the tool is doing the same job as an anesthesiologist in patients who don’t get to see one before surgery. So, we feel confident that we can use this score to triage patients accordingly.” The performance of the ML-ASA algorithm and anesthesiologists’ ratings are compared in the Table.

As Wongtangman noted, the ML-ASA algorithm also reclassified patients into higher ASA classes relative to anesthesiologists’ assessments of these individuals.

“We found that the model successfully reclassified 35% of patients who experienced 30-day mortality into the high-risk category,” she said in an interview with Anesthesiology News. “It also showed better accuracy and better identification.”

Nevertheless, the researchers don’t see the ML-ASA as a supplement to clinical expertise, but rather as an adjunct to it.

“We want to make sure that patients are assessed early on, before they might see an anesthesiologist, because most surgical patients will not see an anesthesiologist until the day of surgery or the day before,” Wongtangman said. “By that time, there is no opportunity to go back and order more lab tests because that will either cause case cancellations or delay the operating room.”

External Validation Makes Results Powerful

For Jonathan P. Wanderer, MD, MPhil, who led similar research back in 2018 (J Med Syst 2018;42[7]:123), the current research helps demonstrate how machine learning can make anesthesiologists more efficient in the processes that power healthcare.

“The idea that you can look at a patient’s health records to get their ASA score with a pretty high degree of accuracy makes a ton of sense,” said Wanderer, a professor of anesthesiology and biomedical informatics and the associate director of Vanderbilt Anesthesiology and Perioperative Informatics Research at Vanderbilt University, in Nashville, Tenn. “What I like about this project is that there is immediate applicability to the operating room—you can take the algorithm and plug it into a triage process and apply it there.”

Another benefit of the current research, Wanderer said, is that the model was developed at one institution and then tested at another. “So, you have this external validation, which is powerful,” he noted. “It demonstrates that maybe we can take this algorithm from their institution and implement it elsewhere. You might have to tweak it a bit, but I think ultimately it can be very helpful.”

By Michael Vlessides

---

Wanderer and Wongtangman reported no relevant financial disclosures. The abstract was selected as one of the meeting’s featured abstracts.