The data layer is three layers now

I came to data sideways

I didn’t grow up in data. I worked at NASA, moved into tech, and spent three years running my own company before a data background landed me on a small startup data team. That path matters for this essay, because I didn’t arrive with ten years of habits about how a data team is supposed to work. I arrived scrappy, right as AI tools got good enough to change the answer.

Here’s the problem I walked into. Point an AI agent at a raw production database and ask it a business question. It will write a clean query, run it, and hand you a confident answer. Often that answer is wrong. It picked the table engineering stopped writing to last year. It counted test accounts. It joined on a key that fans out. Nothing looked broken.

That’s not the agent’s fault, and it’s not engineering’s fault either. Production databases are built to run an app, not to answer questions, and no engineer is perfect. Every raw schema is full of landmines, and the knowledge of where they are lives in a few analysts’ heads.

So the job of a data team changes. It’s no longer to write the answers. It’s to build the layer that lets AI get the answers right. I think that layer has three parts now.

Layer 1: the modeled data

This is the familiar part: raw tables cleaned into entities and facts, with tests, contracts, and one place each business concept lives. I use dbt.

My first version wasn’t dbt. It was a set of reusable notebook components, one for bookings and one for relationships and so on, that every dashboard pulled from instead of re-deriving the logic. It taught me what a transformation layer is for. It also showed me its limits: no tests, no contracts, no review, and every fix depends on someone remembering where the logic lives. Moving that into dbt, together with our analytics engineers, was the foundation for everything else.

Layer 2: the semantic layer

Clean tables still leave room to disagree. Two analysts and an agent can write three reasonable queries against the same table and get three different retention rates.

A semantic layer settles that. Each metric is defined once, with its grain, filters and time logic, and every consumer reads that definition: a dashboard, an analyst or an agent. I helped our analytics engineers build my group’s governed metric marts this way, so the question “what’s our rate?” has one answer no matter who or what asks it. This is the step that makes self-service safe instead of risky.

Layer 3: the context layer

Modeled, governed data still doesn’t tell an agent what the team knows: that a table has a gap last July, that one status word means three different things depending on who’s asking, that this exact question was answered six months ago and here’s how. When that knowledge lives in people’s heads, it leaves when they do, and every agent starts from zero.

So I built a context layer. Nobody asked me to. It started on October 12 as a notes page. Four days later it was a reviewed git repo with an automated reviewer. Within about two weeks it was installed for the whole data department. I built it bottom to top, moved it into production, and still own every piece of it. That speed is the thesis in action: with AI doing the building, one person with a clear design can stand up real infrastructure in days.

How it works:

A git repo written for people and AI. A page per project (what was asked, what we found, what we decided), a data guide of definitions and traps, and playbooks for recurring work.
Nobody browses it. Analysts use it through their AI assistant, which reads what the team knows at the start of a task and offers to write back what it learned at the end.
Every write is reviewed. Writebacks land as pull requests. Scripted checks run first: placement, required sections, and anything that looks like sensitive data. Then an AI reviewer checks evidence: shared facts need a source, a method and a date, and opinions can’t be stated as facts. Changes to the reviewer’s own rules always go to a person.
A steward keeps it clean. A scheduled job hunts for stale entries, duplicates and drift.
A toolkit on top. A plugin gives every analyst’s assistant the team’s methods: prepping a pull request, verifying a production query, building a dashboard, running an adversarial review, scoping a vague ticket.

AI builds the layer, and the layer keeps AI honest

This is the part I care about most. On my team, Claude Code writes the code. People shouldn’t be hand-writing models anymore. People oversee, and domain experts own the decisions the AI is making.

That only works if the agent knows what your best analyst knows. Here’s how I think about getting it there:

Find the hot zones. Look at where the team’s queries actually go. A handful of tables (bookings, almost always) get touched constantly. Those are where the knowledge matters most and where the landmines hurt most.
Write down how to navigate them. For each hot table: which rows are real, which columns lie, which joins fan out, what changed and when. That’s the tribal knowledge that used to take a new analyst six months to absorb. It goes into the context layer as field guides and traps, each with evidence.
Give it to the agent. With that context loaded, an agent can build a model close to right on the first try, because it isn’t guessing where the landmines are.
Let the layer check the agent. Tests, contracts, lint and review gates are what make AI-written code safe to ship. When I cleaned up a large backlog of lint suppressions, the AI did the rewriting and a replay across millions of rows proved nothing changed. The guardrails are what turn “AI wrote it” into “we can trust it.”

The payoff is that the knowledge stops belonging to whoever has been around longest. A new analyst’s assistant knows the traps on day one. That’s what democratizing data actually means.

What I’ve learned

Context goes stale faster than code. Project pages said “in review” weeks after the work merged. That’s why the steward exists. Context needs an owner and a review loop, just like code.
Rules beat good intentions. The riskiest rules (no patient data, nothing copied from private notes) started as guidance. Now they’re scripted checks that fail the change outright.
The hardest part is people, not code. I seeded the context layer with months of prior work from several analytics teams so an analyst’s first session would already know their history. The tool can’t force the shift, but it can make the first win come fast. (More on the human side in Give analysts their thinking time back.)

Where this goes

None of this is specific to data. Every team working with AI has the same problem: the assistant doesn’t know how the team works. The same pattern of a reviewed memory plus a toolkit of the team’s methods can grow into one context layer per team, and eventually a written record of how the whole company works, built for the people and the AI who do the work.

On the data side, the goal I’m working toward is a team that takes a new business area from first question to governed metrics and a working dashboard in under a week. AI does the building. People do the deciding, the aligning and the judgment.

If you’re building this

Start with the hot zones. Write down what your best analyst knows about the tables everyone touches.
Let AI write the code, and make tests, contracts and review the reason you can trust it.
Define metrics once, in one place, before agents answer questions.
Enforce the rules that matter with scripts, not reminders.
Make the first win fast. Adoption follows the aha moment, not the announcement.