AI doesn't have bad days

Okay, it has 529s. Anyone who has watched a batch job fill up with “overloaded” errors knows the API has bad days. The model doesn’t.

My position

For high-volume judgment work (reviewing timesheets, investigating complaints, auditing notes), AI is better than people. It’s more consistent, it reads every case in full, and it applies the same standard to the ten-thousandth case as to the first. It doesn’t fade after lunch or go easy on a Friday.

The only serious argument against this is that people handle ambiguity better. I think that argument is really about context. If a human reviewer makes a better call than your model, it’s because the human knows something the model doesn’t: the policy exception, the history with this person, what “normal” looks like for this role. Give the model that context and the gap closes. If it doesn’t close, your pipeline isn’t good enough yet. That’s an engineering gap, not an AI gap.

I’ve spent most of the last two years building the systems that prove this.

Two systems the company runs on

Workforce quality assurance

I built, and still own, the primary quality assurance system for a large provider workforce, from the decision logic to the AI investigators. Data engineering partnered with me to move it into the warehouse. It runs against the entire population, not a sample.

• Every provider’s activity is evaluated against the company’s quality standards on a regular schedule.
• When something goes wrong, like a missed session or negative patient feedback, an AI investigator reads that provider’s full relationship history (messages, calls, notes, care plans, support tickets) and writes up the evidence. A human doing this would skim. The model reads everything, every time.
• Issues become flags. At three flags, the system issues a strike automatically, and a person reviews it before it stands.
• I own the decision logic, the investigator prompts, and the exemptions, and I’ve led that logic through three major versions.

Before this existed, the work took a team of reviewers and still only covered the cases someone had time for. Now every case gets the full read.

Billing integrity

I built the primary billing integrity system: the one that checks every timesheet submission providers make, every weekday.

This is the data science-heavy one:

• Reading the notes. An LLM classifies the free text of every submission: what kind of work it describes and whether the time claimed fits that work. That’s semantic analysis at a scale no review team could match.
• Baselines per person, not just per population. Statistical norms for how long each kind of work usually takes, and for each provider’s own patterns, so the system can tell “unusual for everyone” apart from “unusual for this person.”
• A decision matrix, not a single score. The model’s classification and the statistical signals come together in a rules matrix that decides what gets escalated. Each rule is plain enough for operations to read and challenge.
• Built as a guardrail, not a ranking. It surfaces what’s unusual and sends it to a person. It never scores anyone as a good or bad provider, because the moment an outlier detector becomes a leaderboard, people start optimizing against it.
• Model choice on evidence. I reclassified the full submission history with candidate models and picked the winner through blind adjudication. The faster model also won most of the disputes.
• Validated against experts. Operations reviewers checked its calls, agreed strongly, and endorsed every escalation.

It later moved upstream into real-time checks that catch problems before a timesheet is even submitted. I was the data scientist on it, start to finish. When it moved into production, I wrote the detection plan, thresholds, data contract, and test cases, and oversaw the engineers who built it into the backend.

Why consistent beats careful

AI does make mistakes. The advantage isn’t that it’s always right. It’s that its mistakes are consistent, so you can find a cause and fix it once, for every future case.

When human reviewers disagree, there’s no code to fix. You retrain, re-audit, and hope. When a model is wrong, it’s wrong the same way every time, so the pattern shows up in the data. Three real examples:

• A model that obeyed too well. We had to move to a new model under a hard deprecation deadline, and it disagreed with production more than expected. It looked like hallucination. It wasn’t: the new model was following contradictions in our prompt literally, where the old one had smoothed them over. The fix was the prompt. We treated the gap as a list of specific fixes, not a go/no-go decision, and closed most of it before cutover.
• Overturns that weren’t the model’s fault. When people disputed outcomes and won, I traced why. Most overturns came from gaps in the deterministic logic around the model, not from its judgment. Every one was fixable, and each fix was context the model had been missing.
• Two systems, one event. Two separate AI pipelines could each penalize the same incident. I audited every path across three codebases, reviewed every past collision by hand, and we adopted one rule: one event, one penalty. Guards in both pipelines now catch nearly all historical collisions on replay.

A human review team would have buried all three inside individual judgment calls. The AI system put them in plain view, and every fix made the system permanently better. That’s the engineering gap closing.

The guardrails are the job

The model is the easy part. Trust comes from the system around it:

• Plain rules before the model. Anything that should never depend on judgment (approved leave, platform outages, cases outside the policy window) is handled in deterministic logic before a model sees the case.
• The AI investigates; people confirm. Automated strikes get human review, and there’s an appeal path.
• Evidence you can audit. Every decision keeps a snapshot of exactly what the model saw.
• Prompts are code. Versioned, tested and reviewed like code.
• Migrations measured against production. A new model has to be shown no worse on the cases that matter.
• A review standard for person-affecting logic. Any change that decides something about a person gets an invariant review before it merges.

AI’s own bad days

The 529s are a joke with a serious side. AI systems fail in their own ways, and you design for them:

• Silent stalls. One batch pipeline stopped writing results for about two weeks, and nothing alerted. Now long jobs report a heartbeat, have hard timeouts, and resume where they stopped.
• Measurement that lies. One system looked several times less effective than it was, because of how detections were counted. The dashboard was wrong, not the AI.
• AI judging AI. When AI helps write the notes that other AI checks, the baseline can drift. If generated notes get more uniform, the detector’s idea of “normal” moves with them. I designed monitoring for exactly that.
• Outages. Retries, backoff and resumable runs for anything that matters.

If you’re building this

• Pick work that’s high-volume, rule-based and inconsistent today. That’s where AI wins clearly.
• When a human beats your model, find the context the human had and give it to the model.
• Put plain logic in front of the model. Don’t make it decide what a rule can decide.
• Treat every overturn and disagreement as a fixable cause.
• Design for AI’s own failure modes from day one.

The point is that every case gets the full, consistent read only the lucky ones used to get, and the people who did this work get to spend their judgment on the cases that really need it.