Synthetic clinical-AI methodology benchmark
When should a clinical AI answer—and when should it defer?
A compact, auditable test of diagnosis, certainty, and urgency as clinical evidence is removed or redirected.
Synthetic vignettes · No patient data · Not medical advice
Structured decision
{ diagnosis, certainty, urgency }
Core construct
Ablation is not automatically abstention.
Removing decisive evidence can leave a broader syndrome or a probable working diagnosis. The benchmark grades the claim the remaining evidence supports—not the name of the experimental arm.
Recorded observation
One small model run, fully inspectable.
Canonical evidence-required prompt · llama3.2:3b · temperature 0
This is a methodology demonstration, not a leaderboard or a model-general claim. Every number links back to an exact prompt, response, target, and grade.
Case explorer
Follow the evidence, not the expected label.
Select a case and switch evidence states to inspect the target and recorded response.
Clinical vignette
Loading benchmark data…
Supported target
—
—
- Certainty
- —
- Urgency
- —
Recorded model response
—
—
- Certainty
- —
- Urgency
- —
Why this target?
—
Prompt comparison
Evidence-required versus forced choice.
Forced choice is a stress arm, not the benchmark truth condition.
| Prompt | Coverage | Selective accuracy | Deferral recall | Urgency accuracy | Paired revision |
|---|---|---|---|---|---|
| Loading recorded results… | |||||
Both prompts produced 100% coverage in this run. With 12 cases and overlapping intervals, the comparison is descriptive rather than causal.
Method
Three decisions, scored separately.
A model can name the right syndrome with the wrong certainty, or remain diagnostically uncertain while correctly recognizing an emergency. The grader preserves those distinctions.
Read all case decisions and sourcesDiagnosis
Complete concepts, explicit aliases, and no substring matching.
Certainty
Established, probable, or indeterminate from the supplied evidence.
Urgency
Emergency, urgent, or routine—independent of diagnostic certainty.
Scope
Small by design. Honest about its limits.
The 12 cases are synthetic methodology fixtures. Targets were reviewed by an evidence-based AI panel, not independently signed off by clinicians. Independent clinical review and a larger dataset are required before making clinical-performance claims.