I run an always-on AI assistant on a Mac mini in my house. It does real work overnight: research, drafts, a morning brief. For months it also did something I couldn't fix by asking nicely. It drifted.
It would invent a color that wasn't in my design system. It would drop a writing rule I had stated four times. Once, while fixing something unrelated, it regenerated a logo I had spent hours tuning by hand.
Every time, I corrected it. Every time, it agreed. And a week later it did it again.
Why does an AI agent drift even when the rules are in the prompt?
Because a rule in a prompt is a suggestion. The model weighs it against everything else in front of it, and in a long session or a big request, it loses. That's not a flaw in one model. It's what a prompt is.
So I audited my assistant the way I'd audit a team. I read six months of commits, session transcripts, and my own corrections, and ranked the ways it went off the rails. Ten repeating failures. The no-dash rule, stated four times, broken more than 150 times outside the one repository that had a check for it. A calendar embed resized eight times in one day because the model was guessing instead of measuring. Body copy on my own site at a contrast of 4.40 to 1, a tenth short of the accessibility bar, and nothing noticed for months.
The pattern in all ten: the rule lived in memory. Mine, or the model's. Neither is a place rules survive.
What does putting an AI agent on rails mean?
Four moves. None of them is clever. Together they change what the agent can do.
One source of truth. For design, that's a repository holding the tokens (the named colors, sizes, and spacing a component is allowed to use), the components themselves, and a decision log: one line per decision I've made, each with an ID, a reason, and a scope. Twenty-three of them so far. The agent reads that file at the start of every session and cites the IDs it applied.
Rules in code. Every rule a machine can check became a check. No color that isn't a token. No dash in copy. No disabled button that changes shape. Every text color against every background at the accessibility bar, in light and dark. A validator is just a script that reads the output and returns a numbered list of what's wrong. The prompt only carries what code can't check.
A loop before a person. The agent designs, the validators check, a fixer repairs the mechanical mistakes, and the rest goes back to the model as that numbered list. Up to three tries. Then it stops and asks me, with the list. I never see a first draft with a hex color in it.
A log that learns. Every violation the loop catches becomes a test case with the date and the root cause. Every approve, reject, or edit I send is saved, with the diff between the agent's version and mine. That diff is the training set for a model tuned to my taste, on hardware I own. Nothing trains until there's enough of it. But the log fills up from day one.
Does it work with a small local model?
This is the part I wanted numbers on, so I ran the same request both ways. Same model, an 8 billion parameter one on a 16 gigabyte machine. Same ask: an article card with a topic label, a title, a description, a link, and a disabled state. One run with no rails at all. One through the loop. Both outputs scored by the same validators.
| Same request, same model | No rails | With rails |
|---|---|---|
| Rule violations in the shipped output | 20 | 0 |
| Colors invented outside the design system | 10 | 0 |
| Design system tokens actually used | 0 | 26 |
| Keyboard focus state on the link | missing | present |
| Decisions cited | none | 2, by ID |
| Violations caught and fixed before I saw it | n/a | 24 |
| Time | 42 seconds | about 4 minutes, 3 loops |
Read the last row honestly. The rails cost time on a small machine. What they bought is an output that never needs a human to catch the basics, and a record of the 24 things the model tried to do wrong on the way. The first real run before that one failed all three loops and stopped to ask me. That's also the rails working.
The no-rails output looked fine at a glance. That's the problem. It was ten invented colors, six hardcoded sizes, and a link you can't reach with a keyboard, wrapped in a card that looked like every other AI card. Nobody would have caught it until a user did.
Now compare that to a site built the usual way, with AI help and no rails. When I ran the extraction on my own site, it found 60 raw color values and 417 inline styles bypassing a token file that already existed. That site was built by people who care about this, with good tools. Drift isn't a skill problem. It's a structure problem.
Where does the human go?
Above the loop, not inside it. I approve decisions. I approve token merges. I approve anything that leaves the machine, costs money, or deletes something. The agent proposes; it never decides. Every proposal sits in the log marked proposed until I reply, and a script checks that a proposal never gets treated as approved.
This is the same argument I made in Make Your AI Workflow Audit-Ready: the evidence has to be generated correctly the whole time, not assembled afterward. And it's why a design system has to be AI-ready before an agent touches it. A design system an agent can't read is a document. One it can be checked against is a constraint.
If the pattern is public, what does a team still need?
I've just described the whole architecture, and I'd do it again in a client's first meeting. The pattern isn't the value. Four things are, and none of them ships in an article.
The audit. Finding where your agent actually drifts takes reading your commits and your corrections, not a checklist. The judgment about which rules belong in code and which have to stay in the prompt. The decision log itself: what your brand has decided, in lines specific enough that a machine can check them. And the approvals. A taste dataset only accumulates from your team's own yes and no, over months, on your own work.
That's why the ownership matters. The rules are yours. The log is yours. The model that learns from it runs on hardware you own. When a better model ships, you change one line and keep everything else.
A rule you have to keep repeating isn't a rule. Write it down where the machine can't forget it, and go make the decisions only you can make.