You manage AI now

When everyone can write code

A year ago, building with AI took an engineer. Now a PM wires an LLM into a workflow in an afternoon, an analyst runs one over ten thousand notes from a notebook, and anyone can swap one model for another by changing a single line. Code is cheap. Judgment isn’t.

That changes the senior job. You stop being the person who writes the most code and become the person who manages the AI that writes it. It’s the same skill as managing a team. You don’t do every task. You own the few calls that decide whether the work is any good, and you build cheap ways to check it.

There are four calls I think every orchestrator owns:

1. Which model does the work.
2. How you know the work is right: evals you can rerun, not one-off spot checks.
3. What the AI is allowed to decide and where a person has to step in.
4. What the AI knows about your business: the context you give it, so it isn’t guessing.

Each one used to belong to a specialist. Now it gets made every day by whoever is building, so it has to become a shared skill with a simple framework behind it. This essay takes the first call.

The first call: which model

Most model choices are made by habit: the model the team already uses, the one that just launched, the one someone said was best. That works until it doesn’t, and when it doesn’t, nobody can say why the choice was made.

I want model selection to be something a PM can do as well as an engineer: a simple framework and a cheap way to test it. This is mine.

Three axes

1. Speed. Is someone waiting on the output? If a person sits through the call, latency is friction, and the smaller, faster tiers get weighed first.

2. Reasoning. How much nuance and ambiguity does the task carry? Complex judgment, multi-step logic and storytelling push toward larger models.

3. Domain knowledge. What does the model already know about your domain before the prompt tells it anything?

The first two get public benchmarks and launch-day charts. The third almost never does, because it’s specific to your domain, so nobody measures it. I think it’s the one teams most often get wrong.

A model’s knowledge freezes at release

Domain knowledge comes from how a model was built: what it was pretrained on, how it was post-trained, and the engineering behind both. Then training ends, and the knowledge is fixed. A model learns nothing from being used. What changes over time is which model sits behind the name, when a lab ships a new version, and what we put around it: the prompt, retrieval, tools.

That matters because prompts are never perfect. Every production prompt leaves gaps, and the model fills them with what it knows. In a domain whose rules change every year, like US healthcare billing, a model’s built-in knowledge goes stale on a schedule. A gap filled with last year’s rule produces a confident error that looks right.

Prompts can make up the difference, but only for what they actually carry. Where a prompt is silent, the model’s own knowledge decides, and models differ a lot in what they know.

How to test it (anyone can run this)

1. Write the questions from documents, not from a model. If a model writes the test, the test measures that model. Tie every question to a quoted sentence in a primary source, and check the quote mechanically.
2. Have more than one lab draft, then cross-check. Two vendors draft from the same passages, and each checks the other’s work against the source. Edits can change wording, never an answer or its quote.
3. Test closed book and open book. Closed book: the model answers from what it knows. Open book: the source is in the prompt. The gap tells you how much the model’s own knowledge is doing, and how much good context would replace it.
4. Compare models, not companies. Two tiers from one lab can differ more than flagships from two labs.
5. Grade with judges from more than one lab, and measure them. Check whether any judge favors its own lab’s models.
6. Cache everything and rerun on every release. Then a new model only needs its own answers run, and the test becomes a habit instead of a project.

What I found

I ran this on Medicare billing rules: 646 questions built from CMS rules, and six current models across both major labs’ flagship tiers. From memory, the models scored between 0.62 and 0.87. With the rule in the prompt, every one scored 0.94 or higher. The biggest single gap fell on a code whose description CMS had just changed. Full write-up: the full write-up.

Two lessons carry past billing. First, the gap between tiers within one lab was bigger than the gap between labs, so “we’re a Claude shop” isn’t a model choice. Second, good context closed nearly all of the gap, which is the fourth call on the list above doing its job.

Best practices I’d hand any team

• Name the axis that matters before you pick. Is the constraint speed, reasoning or knowledge? Most bad choices skip this.
• Assume the model doesn’t know your rules unless you’ve tested it. If a rule matters, put it in the prompt.
• Watch the calendar. When your domain’s rules change, the model you picked last year may now be wrong in confident ways.
• Test models, not brands.
• Make the test cheap to rerun. A bench that reruns from cache gets used. A one-off study gets cited once and forgotten.
• Write down why you chose. The next person to swap the model should know what they’re giving up.

It’s a management skill

Picking a model works a lot like hiring. You define the job before you look at candidates. You test for the skill the job actually needs, not the one on the résumé. You check again when the job changes. And you write down why you chose, so the next manager isn’t guessing.

Managing AI well is mostly the same discipline: clear calls, cheap checks, decisions written down. Model selection is the first one. The other three (evals, decision rights, and context) are the next essays.