Target validity matters more than headline predictive accuracy.
AI value · Black share identified for extra care
Verified
46.5% if ranked by actual health need
Counterfactual repair in one system; vendor disputed use characterization.
Before
Has a limited number of intensive nursing and coordination slots to allocate. → Use clinical knowledge to identify patients with complex needs.
After
Ranks patients by predicted future cost as a proxy for health need. → Use the score among selection inputs and allocate scarce extra care.
Human boundary
Programs own enrollment; clinicians retain judgment.
Why it matters
A predictable proxy can still allocate scarce care unjustly.
How the work changed
Before
How the work ran before the change.
Step 1 of 2
Care-management program
Has a limited number of intensive nursing and coordination slots to allocate.
ControlThe Science study does not provide a dated pre-algorithm workflow; this is a bounded resource-allocation baseline.
Step 2 of 2
Clinicians and program staff
Use clinical knowledge to identify patients with complex needs.
ControlHuman judgment, constrained by program capacity.
What changed
A predictable proxy can still allocate scarce care unjustly.
Decision rightHuman authority remains at the consequential boundary
After
How the same work runs now.
Step 1 of 2
Commercial risk algorithm
Ranks patients by predicted future cost as a proxy for health need.
ControlTop 97% risk threshold at the studied site; proxy choice embeds unequal access and spending.
Step 2 of 2
Program staff and clinicians
Use the score among selection inputs and allocate scarce extra care.
ControlClinicians can supplement the score, but scaled allocation is score-shaped; health-need counterfactual exposed bias.
Process model built from the published workflow evidence for Large US health system / Optum algorithm. Every step, actor, and control appears in full below.Every step, actor, and control
Exception path
Clinical judgment can add patients, but the queue is score-shaped.
Decision authority
Programs own enrollment; clinicians retain judgment.
Before
#
Actor
Action
Control
01
Care-management program
Has a limited number of intensive nursing and coordination slots to allocate.
The Science study does not provide a dated pre-algorithm workflow; this is a bounded resource-allocation baseline.
02
Clinicians and program staff
Use clinical knowledge to identify patients with complex needs.
Human judgment, constrained by program capacity.
After
#
Actor
Action
Control
01
Commercial risk algorithm
Ranks patients by predicted future cost as a proxy for health need.
Top 97% risk threshold at the studied site; proxy choice embeds unequal access and spending.
02
Program staff and clinicians
Use the score among selection inputs and allocate scarce extra care.
Clinicians can supplement the score, but scaled allocation is score-shaped; health-need counterfactual exposed bias.
Work that left the path
Some manual population prioritization
Human role before
Clinicians and care-program staff used clinical knowledge and limited program capacity to select patients for intensive care management; the study does not provide a dated pre-algorithm queue.
Human role after
Program teams set enrollment thresholds and clinicians may supplement the score.
AI role
Ranks patients by predicted future cost as a proxy for need.
Outcomes
Black share identified for extra care
Verified
17.7% under cost proxy→46.5% if ranked by actual health need
Published 2019 · 43,539 White and 6,079 Black patients
Counterfactual repair in one system; vendor disputed use characterization.
What leaders can reuse
Anti-pattern
Do not call cost prediction inaccurate; the target was wrong.
Questions
01Where is the operating threshold set and who can override it?
02What measured result would trigger rollback or retraining?
03Which residual decisions must remain human-owned?
Portability conditions
Need-based targets
Subgroup audits
Override capacity
Reputation risk
high
Evidence and authority
What the public record supports.
Current · updated
1 independent, 1 peer reviewed; publication outcomes are verified.
Bundle 1.0.0 · reviewed 2026-09-06 · stable ID 81c7b1bc482d6d8f
