Unraveling the Truth about AI Confidence Scores in Healthcare

Confidence Scores Explained in the Context of AI

Confidence scores in AI are numbers that indicate an AI tool's certainty about an output, such as a diagnosis or a medical code. These scores typically come from a statistical confidence interval, which calculates the probability of an AI output's accuracy based on its training model. Just like a dating app's match score, they can mislead users into thinking they're reliable. For clinicians using generative AI summaries, a displayed confidence score can lead to unintended errors if they trust the technology over their own judgment.

For example, an off-the-shelf AI tool might give a high confidence score for a diagnosis based on population-level training, but it doesn't account for the specific clinician's population or local health patterns. This leaves clinicians with an incomplete picture and can lead to mistakes.

A Flawed Approach for Grading AI Output

AI confidence scores often appear as percentages, suggesting a certain likelihood of a code or diagnosis being correct. However, for healthcare professionals not trained in data science, these numbers can seem deceptively reliable. There are four significant risks associated with relying on these scores:

1. Misunderstanding of context: AI workflows only contain population-level training and don't account for a provider's specific demographic. This leads to broad assumptions and an incomplete picture for clinicians.

2. Overreliance on displayed scores: A 95% confidence score can make clinicians assume there's no need to investigate further, oversimplifying data complexities and encouraging them to bypass their own critical review.

3. Misrepresentation of accuracy: The intricacies of healthcare don't always match statistical probabilities. A high confidence score might match population-level data, but it can't diagnose a particular patient with certainty, creating a false sense of security.

4. False security generates errors: If clinicians follow an AI recommendation too closely based on high scores, they might miss other potential diagnoses, leading to delayed critical interventions or billing mistakes.

A Better Way of Helping Users Understand the Strength of AI Output

To create trustworthy AI outputs, it's better to use the following methods:

1. Localize and update AI models often: Tailoring AI models to include local data, such as specific health patterns and demographics, makes the output more relevant. For example, there are more patients with Type II Diabetes in Alabama than in Massachusetts, and timely, localized data is crucial. Regular retraining and audit processes ensure the models reflect current standards and discoveries.

2. Thoughtfully display outputs for the end user: Consider how each user interacts with data and design outputs to meet their needs. Instead of a single confidence score, show contextual data such as how often similar predictions have been accurate in specific populations or settings. Comparative displays help users weigh AI recommendations more effectively.

3. Support, but don't replace, clinical judgment: The best AI tools guide users without making decisions for them. Use stacked rankings to present a range of diagnostic possibilities with the strongest matches on top, allowing clinicians to use their professional judgment.

Clinicians need tech tools that support their expertise and discourage blind reliance on confidence scores. By blending AI insights with real-world context, healthcare organizations can provide safer patient care and build smoother workflows.

Brendan Smith-Elion is VP, Product Management at Arcadia. With over 20 years in the healthcare vendor space, his passion is product management. He has experience in business development and BI engineer roles. At Arcadia, he's dedicated to driving transformational outcomes for clients through data-powered, value-focused workflows. He started his career at Agfa, led the cardiology PACS platform, and later worked at Chartwise and athenahealth. His most recent role was at Alphabet/Google, working on a healthcare data platform.

This post appears through the MedCity Influencers program. Anyone can publish their perspective on business and innovation in healthcare on MedCity News through MedCity Influencers. Click here to find out how.