Someone Has to Tell the Agent the Rules Changed

Collecting context before the build is half the job. Maintaining it afterwards is the half somebody has to own.

I’ve spent a lot of this newsletter arguing that the hard part of deploying AI is extraction. Getting the operational logic out of people’s heads, out of undocumented systems, out of the gap between how a process is described and how it actually runs. I still believe that. But extraction is only half the story, and it’s the half that flatters us, because it has an ending. You interview the operators, you map the process, you encode the rules, you ship.

The other half doesn’t end. Delivery days change. Regions get added and dropped. Approval thresholds move. Pricing tiers get restructured. Suppliers get replaced, exceptions become policies, policies become exceptions. Every project I’ve worked on, across every industry, has the same property: the business keeps moving after the build is done, and the system keeps acting on the version of the business that existed the day it shipped.

Context is a live thing. It is not something you collect once and store. And it raises two questions that rarely make it into a project budget. Technically, how does an agent’s knowledge get updated and, more importantly, invalidated? And operationally, who makes that call, and through what interface?

To answer either, you have to start with why the model can’t just handle this itself.

Working memory is not memory

The context window gets described as the model’s memory so often that it’s worth being precise about what it actually is. The context window is the sequence of tokens the model can attend over in a single call. That’s all. Everything the model “knows” about your task, beyond what’s baked into its weights, lives in that sequence. When the call ends, it’s gone. The weights don’t update from talking to you. Nothing persists.

So the context window isn’t memory. It’s working memory, and it comes with three constraints that shape everything downstream.

The first is cost. Attention, in its standard form, works by letting every token look at every other token, which means the compute grows quadratically with sequence length. Frontier models increasingly interleave sparse and sliding-window variants, but in practice the scaling remains effectively quadratic. Double the context, quadruple the attention work. Inference engineering softens this with the KV cache: once a token’s keys and values are computed, they don’t change, so the runtime stores them and each new token only computes its own contribution and attends against the cache. That’s why inference has two distinct phases. Prefill processes your whole prompt in one parallel pass and determines how long you wait for the first token. Decode generates one token at a time against the cache and determines tokens per second. The cache itself is enormous, scaling with layers times heads times sequence length, which makes long-context serving as much a GPU memory problem as a compute problem. This is why context is priced the way it is, and why “just put everything in the context” is not a free architecture decision. It’s a metered one.

The second constraint is degradation. Even within the window, the model doesn’t use everything equally. Retrieval quality is empirically strongest at the start and end of the context and sags in the middle, and that positional effect has held across every model generation since it was first measured. There’s a second effect on top of it, now usually called context rot, where accuracy declines as the input grows even when the evidence sits in a favourable position. Stuff a hundred documents into a long window and you have not given the model a hundred documents. You’ve given it a well-lit beginning, a well-lit end, and a dim warehouse in between.

The third constraint is the one that matters most for this essay: no persistence. Whatever you load into the window evaporates when the session ends. A deployment that has to hold knowledge across weeks, months, and personnel changes gets nothing from the window itself. Cross-session continuity is not a model feature. It has to be engineered, outside the model, in the harness.

What actually gets bolted on

In practice, agent memory is a hierarchy of systems wrapped around the model, and each layer answers a different question.

The innermost layer is the in-context scratchpad: the running trajectory of the current task. It grows with every step, and because the model re-reads the entire trajectory on every turn, it gets compacted as it grows. Older turns get summarised, recent ones stay verbatim. Compaction is a craft in itself, because a summary that drops a load-bearing detail poisons every later decision that would have needed it.

Outside the window sits long-term memory, and it splits naturally into two kinds of store, because two kinds of knowledge behave differently.

Episodic memory is what happened. Events, decisions, outcomes, each with a timestamp. This is the time-series layer: traces and logs of the agent’s own behaviour and of the business it operates in. Which orders were processed, what was decided about the mismatched invoice in April, how the exception was resolved. Episodic memory answers “what did we do last time this happened?”

Semantic memory is what’s true. Entities, rules, procedures, the SOPs. Which regions we deliver to, what the approval threshold is, how a valid purchase order is structured. Semantic memory answers “what are the rules right now?” It’s the layer that most directly encodes the operational logic extracted during the build, and it’s the layer this whole essay is about.

At each step, retrieval pulls the relevant slice of both stores into the working window. And below all of it sits durable task state, the plain record of what’s been processed and what’s pending, which belongs in a database rather than in any context, because it has to survive crashes, restarts, and model upgrades.

None of this is exotic. Storage and retrieval are, at this point, solved-ish problems. The part that isn’t solved is the part that makes the whole thing safe to rely on.

Invalidation is the hard part

Here is the failure mode that should worry you more than hallucination. The approval threshold changes in March. The semantic store still says the old number. In June, the agent retrieves that rule, with full confidence and a clean provenance trail, and acts on it. Nothing in the system flags a problem, because from the system’s point of view there isn’t one. Retrieval worked perfectly. It retrieved the truth as of last year.

Stale memory retrieved with confidence is worse than no memory. No memory makes the agent ask. Stale memory makes it certain.

The engineering answer is truth maintenance, and it has three parts. First, every memory carries a timestamp and a source, so any retrieved fact can be traced to when it was true and where it came from. Second, contradiction triggers re-verification rather than silent preference. When a fresh observation disagrees with a stored rule, the system must not quietly pick one. The disagreement is the signal, and the correct behaviour is to surface it. Third, process-type memories get expiry dates. A rule about how approvals work is not a fact about the world; it’s a fact about an organisation, and organisations change. TTLs and scheduled re-validation force the question “is this still true?” onto a calendar instead of leaving it to luck.

All three mechanisms are buildable today, and parts of them are already shipped. Temporal knowledge graph memory stores, the current research direction for agent memory, attach validity intervals to every fact and invalidate the old edge when a new one contradicts it. But watch how they resolve the conflict: they prefer the newest information. That is a recency heuristic, not a truth decision, and it is exactly the silent preference the mechanism is supposed to avoid, automated. Which exposes what all three mechanisms have in common: every one of them ends in a question that the system cannot answer for itself. Is this still true? The timestamps tell you the rule is fourteen months old. The contradiction detector tells you the last three observations disagree with it. Neither can tell you which version is correct now. That call belongs to a person.

The world moved and the agent never heard about it. More slow deployment rot traces back to that than to any model limitation I can name.

The update flows in two directions

So the question stops being technical and becomes organisational: who owns the agent’s knowledge after the engineers leave?

The flow everyone plans for runs downward. The business changes something, and somebody updates the system. This is the flow that gets a paragraph in the handover document and then quietly dies, because the person who changes the delivery schedule and the person who can edit the agent’s rules are rarely the same person, and the path between them is exactly the kind of undocumented process the agent was supposed to eliminate.

The second flow runs upward, and it’s the one deployments rarely design for. It’s also the more interesting one. An agent that operates inside a process all day is in a privileged position to notice that the process has drifted. It keeps seeing orders from a region the rules say doesn’t exist. It keeps watching a human override the same threshold. The contradiction machinery from the previous section isn’t just a safety mechanism. It makes the agent a sensor for change. The agent can be the first thing in the organisation to notice that the world moved, provided there’s somewhere for that observation to go.

Both flows converge on the same requirement: a named person on the customer side with the standing authority to confirm changes to what the agent believes. And which person that is matters more than most deployments admit. The instinct is to give the role to whoever can technically operate the system, an admin, an analyst, someone from IT. But that person confirms changes to rules they don’t live inside. They can check that the edit is well formed. They cannot check that it’s true.

The person who can verify truth is the one who holds the knowledge, the operator who plans the deliveries, approves the invoices, works the exceptions. That person is rarely technical, and in most organisations they are the last person anyone would hand system access to. Which is exactly backwards. Confirming a change to the agent’s beliefs is a judgment about the business, not about the system, so the role has to be optimised for the knowledge holder, and everything about how the change is exposed to them has to bend to that choice. If confirming a rule requires understanding the rule engine, the wrong person will end up doing the confirming, and the deployment is back to the gap it started with. In every deployment, the agent’s knowledge either has an owner who can tell true from stale, or it drifts, and drift is the default.

The interface question

Which leaves the practical question I keep turning over: what does the owner actually touch? If the right owner is a non-technical operator, the interface stops being a detail and becomes the design constraint.

My deployments so far have used structured controls, mostly tables. Rules as rows, thresholds as fields. Structured data has real virtues for this job. It can be validated before it’s accepted, diffed so a change is visible as a change, timestamped per row, and edited by a non-engineer without the risk of quietly breaking something three paragraphs away. Documents are the opposite trade. They capture nuance and exceptions that tables flatten, but they can contradict themselves, they’re hard to invalidate selectively, and an edit in prose has no natural diff.

For a while I treated this as a choice between formats. I now think that’s the wrong frame, because it assumes the human edits the storage directly. The design I keep arriving at instead splits the layers. The interface is natural language. The storage is structured. The approval is the gate between them.

It works the same in both directions. Downward: the operator tells the agent, in plain words, that deliveries no longer go out on Fridays. The agent drafts the corresponding rule change, shows the diff against the current rule, and asks for confirmation. Upward: the agent notices the standing contradiction, packages it the same way, proposed rule, evidence, diff, and asks the same question. In neither case does anyone hand-edit a table, and in neither case does an unstructured note become live behaviour on its own. The table remains the single source of truth. The conversation is just how you write to it, and the confirmation is what makes the write legal.

I want to be honest that this is a design direction, not a pattern I can point to at scale. There are open problems inside it. How the agent should behave between noticing a contradiction and receiving a ruling. How to stop an approval queue from becoming the new bottleneck. How to keep the evidence attached to the rule so that next year’s owner can see why it says what it says. I don’t have finished answers. I have a strong opinion about the shape of the answer, which is that the human approves changes to the agent’s beliefs the same way the trust ladder has them approve the agent’s actions.

This doesn’t age out

Two objections come up whenever I talk about this, and they’re really the same objection. Context windows will get big enough. The labs will solve memory at the model layer.

Maybe both happen. Neither helps. A ten-million-token window is still working memory, and it still evaporates when the session ends. A lab-built memory layer still stores what it’s told and retrieves what it stored; it doesn’t know your approval flow changed in March unless something tells it. The invalidation problem isn’t a capability gap waiting on a better model. It’s a property of deploying any system with beliefs into an organisation that keeps changing, over a horizon of months and years, not sessions. Right now, and for any future I’d plan around, it gets solved in the harness, by engineering, with a human holding the pen.

We’ve collectively become good at the first extraction. The discipline that hasn’t formed yet is the second one, and the third, and the standing arrangement that makes them routine. Somewhere in every organisation that deploys an agent, there’s a rule that’s already wrong and a system that’s still sure of it. Naming the someone is the cheapest piece of engineering in the entire deployment.