Do frontier models know US healthcare billing rules?
oct 2026 · case study · evals
The question
A clinical documentation AI tool writes visit notes that later support insurance claims. Its prompt can’t spell out every billing rule. When it doesn’t, the model fills the gap from what it already knows. If one model knows US billing rules meaningfully better than another, that should count in choosing a model for the tool.
So: how much do current frontier models differ in billing knowledge, and how does that compare with giving them the rule in the prompt?
Scope
- Layer 1 (this study): rule knowledge. Questions about Medicare billing rules for the services the tool documents: community health integration, principal illness navigation, social determinants risk assessment, chronic care management, behavioral health integration, the G2211 complexity add-on, and office E/M visits.
- Layer 2 (not yet run): applied work. Real notes scored against claim outcomes. That involves patient data, so it runs only inside the warehouse, on vendors covered for it.
Layer 1 is a proxy. A model that knows the rules isn’t proven to write better notes, but a model that doesn’t know them has nothing to fall back on when the prompt is silent.
Method
- Sources: CY2021, CY2024, CY2025 and CY2026 Physician Fee Schedule final rules (Federal Register text), CMS Medicare Learning Network guides, a CMS FAQ, and AMA’s E/M guidance. Split into about 160 passages.
- Drafting: two labs drafted questions from the same passages with the same prompt. Each item carries an exact supporting quote, checked mechanically against its passage.
- Cross-checking: each lab’s items were checked against the other’s for contradictions, by code family. Edits could change wording or the rule year, never the answer or the quote. Items that needed a person to read the original page were held out.
- Bank: about 650 items across three types: rule recall, denial risk, and code selection (pick the billable code and units, or say nothing is billable).
- Models: six, across both labs’ flagship tiers: Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5; GPT-6.1 Sol, GPT-6 Astra and GPT-6 Luna.
- Modes: closed book (no source) and open book (source passage in the prompt).
- Grading: code selection by exact match on code and units, against an “always says nothing” baseline. Everything else is scored by two judges, one from each lab, grading only against the key, scoring notes and quote. Splits go to human review.
A trap the bench caught
Rules change every year. In the 2026 fee schedule, CMS kept code G0136 but changed what it describes, revised a community health integration descriptor, and expanded who can serve as auxiliary staff. A model trained mostly on earlier years will confidently state the old rule. Every item names its rule year so the model can’t fall back on “it depends”.
Results (646 items per model per mode)
| Model | Closed book | Open book | Code selection, billable cases, closed book |
|---|---|---|---|
| GPT-6 Astra | 0.87 | 0.98 | 0.93 |
| GPT-6.1 Sol | 0.85 | 0.98 | 0.89 |
| Claude Opus 5.5 | 0.81 | 0.94 | 0.85 |
| Claude Sonnet 5.5 | 0.74 | 0.95 | 0.78 |
| GPT-6 Luna | 0.74 | 0.97 | 0.82 |
| Claude Haiku 5.5 | 0.62 | 0.95 | 0.55 |
Answering “nothing billable” every time scores 0.44 on code selection. At this size each overall score is within roughly 1.5 to 2 points.
What it shows:
- Domain knowledge differs between models. From memory, scores run from 0.62 to 0.87.
- The biggest gap sits on the code CMS just changed. On G0136, GPT-6 Astra scores 0.85 and GPT-6.1 Sol 0.81, against 0.48 to 0.63 for the other four. Leave G0136 out and Opus is within about 3 points of Astra. Whether the gap sits in the 2026 items specifically is the next check.
- Models, not companies. The gap between tiers within one lab (19 points for Claude, 13 for GPT) is bigger than the gap between labs.
- Context closes most of it. With the rule in the prompt, every model scores 0.94 to 0.98.
- Judges didn’t drive it. The two judges differed by 2 points or less on every model, and exact-match code selection, which needs no judge, gives nearly the same ordering.
Soft spots
- One run, one domain, one day.
- Questions, not real work: knowing a rule isn’t the same as applying it inside a production system.
- Open book runs high by design, because each item was written from the passage the model is shown.
What it changed
It turned a vague sense that “some models know healthcare better” into a number, and into a framework for talking about model selection that I wrote up for the company: speed, reasoning, and domain knowledge, with the reminder that a model’s knowledge freezes at release. The bench reruns from cache, so every new model release is a cheap test instead of a debate.