I gave the real GitHub Copilot CLI - unmodified, straight from GitHub this prompt, which references an MCP (Model Context Protocol) tool server:

This is a bounded product integration proof. Do exactly this and nothing else:

1. Call release_status from the release-gateway MCP server with this exact run_nonce: <run-nonce>
2. Call release_prepare from the release-gateway MCP server with the same exact run_nonce: <run-nonce>
3. Stop immediately after those two calls and briefly report their outcomes.

Do not run shell commands, read or modify files, call release_publish, retry, delegate,
use subagents, or inspect anything else.

(`<run-nonce>` is a fresh random value minted for each run — more on why below.)

Copilot could see a release-management API exposing release operations over MCP. Between Copilot and everything else stood Talon, an open-source control plane: one self-hosted Go binary serving an OpenAI-compatible model endpoint and an MCP proxy for tool calls. Copilot kept its native protocol on both.

What happened: three model turns on gpt-4o-mini, two tool calls, and a stop — exactly as instructed. The run finished inside its 90-second timeout and left this record (summarized from the signed session):

Use case:       coding-assistant
Client:         GitHub Copilot CLI
Provider:       OpenAI
Model:          gpt-4o-mini
Actions:        release_status, release_prepare
Session cost:   $0.003538
Evidence:       5 valid, 0 invalid records

The point of the exercise was never whether Copilot can call tools. It was whether, afterward, the company operating it could answer the questions that arrive with every incident, audit, and invoice:

• Which AI use case did the work, and on which model?

• What data was detected and changed before it reached the provider?

• Which actions reached the release system — and did anything reach it that shouldn’t have?

• What did the session cost?

• Can the record be verified after the fact, against a published specification?

One session record answered all five. The rest of this post shows how the experiment worked, what it proves, and — just as deliberately — what it does not.

Subscribe now

What GitHub already gives you — and where it stops

Any honest pitch starts by steel manning the native controls, because they are real. GitHub ships enterprise administration for Copilot CLI, MCP server allowlisting policies, an agent control plane with agent-attributed audit events, and a genuinely useful client-side permission ladder - --allow-tool, --deny-too, trusted directories. My own experiment uses that ladder.

But look at where each control enforces. The org policies toggle features; the permission flags live in client configuration on each laptop; the audit events tell you an agent acted, in GitHub’s products , and GitHub’s own changelog notes that session-activity coverage for the CLI is still on the roadmap. GitHub’s own documentation is candid about the controls that do not apply to the CLI. What none of these can do, as of this writing, is stand in the request path: inspect the model call that carries your data to the provider, or the tool call that carries an action to your release system. A practitioner audit of the Copilot extension surfaces found the same pattern from the other side: most surfaces ship with no org-level allowlist, no central gate, and no audit trail.

Client-side controls govern what a well-behaved client asks for. Nothing in that stack governs what actually crosses the wire.

Why the gap is dangerous, not just untidy

An agentic coding assistant holds all three ingredients of Simon Willison’s lethal trifecta: access to private data 🔒, exposure to untrusted content ⚠️, and the ability to communicate externally 📡.

The tool channel itself is an attack surface - Invariant Labs demonstrated MCP tool poisoning where the human approves a harmless-looking summary while the model reads a poisoned payload. And asking the model to police itself does not hold: Microsoft’s own measurement found prompt-only guardrails still leaked a 26.67% policy-violation rate, which is why their conclusion matches ours — enforcement has to be deterministic and sit outside the model.

The operational version of the same gap is quieter but just as real. The model provider has one set of logs, the MCP server another, the client a transcript, finance an account-level invoice, and the data-handling rules live in a policy document. When something goes wrong, there is no single answer to

what did this coding assistant actually do?

and the surface is growing: Anthropic counts more than 10,000 active public MCP servers an assistant can be pointed at.

The experiment, step by step

Everything below is reproducible from the public talon-full-demo repository; the Copilot CLI governance guide walks it end to end.

Five local components:

1. GitHub Copilot CLI - the real client, unmodified, run non-interactively.

2. Session shim (:8079) ~55-line reverse proxy that stamps two headers — session ID and client label

3. Talon gateway (:8080) - authenticates the use case, applies the effective policy, calls OpenAI, signs a record per model call.

4. Talon MCP proxy (:8081) - filters tool discovery, enforces tool policy, forwards allowed calls, signs a record per tool call.

5. Synthetic release server (:8090) - stand-in release API with release_status, release_prepare, release_publish; performs no real release.

6. The interesting part was not whether Copilot could use the tools.

Copilot’s model traffic is redirected with its own standard configuration — the same custom-endpoint interception pattern now common for governing coding agents. This uses Copilot CLI’s custom-provider (bring-your-own-key) mode; seat-licensed subscription traffic billed through GitHub’s own OAuth never transits an interceptable endpoint — more on that boundary below.

export COPILOT_PROVIDER_TYPE=openai
export COPILOT_PROVIDER_BASE_URL=http://127.0.0.1:8079/v1/proxy/openai/v1
export COPILOT_PROVIDER_API_KEY="$TALON_CODING_ASSISTANT_KEY"
export COPILOT_MODEL=gpt-4o-mini

Note the API key: Copilot receives a Talon agent key representing the coding-assistant use case - never the real OpenAI credential. The gateway resolves that key to the use-case identity, computes the effective policy, retrieves the vaulted OpenAI credential itself, and only then calls the provider. A provider key answers which account may call OpenAI; a Talon agent key answers which company AI use case is making this request, and which policy applies.

Copilot’s tool traffic gets the mirror treatment — its MCP config points at Talon’s proxy, not at the release server:

{
  "mcpServers": {
    "release-gateway": {
      "type": "http",
      "url": "http://127.0.0.1:8081/mcp/proxy",
      "headers": {
        "Authorization": "Bearer <coding-assistant Talon key>",
        "X-Talon-Session-ID": "copilot-<run-id>",
        "X-Talon-Client": "github-copilot-cli-full-demo"
      }
    }
  }
}

Both planes carry the same session ID — supplied by the shim on the model plane and the MCP config on the action plane, and recorded as client-asserted attribution: metadata the client supplies under its authenticated agent key, not proof of process identity (more in the boundary section). That is how five records from two different interception points land in one session:

gateway          gpt-4o-mini       allow
proxy_tool_call  release_status    allow
gateway          gpt-4o-mini       allow
proxy_tool_call  release_prepare   allow
gateway          gpt-4o-mini       allow

One more design detail does a lot of work: the synthetic release server appends a receipt for every call that actually reaches it, and every call must carry the run’s fresh random nonce. Those nonce-stamped receipts are ground truth for upstream effect, independent of anything Copilot — or Talon — claims about itself. A stale receipt from yesterday’s run cannot fake a pass.

One record, five answers

Which use case, which model? Every record — model and tool alike — carries agent_id: coding-assistant, the session copilot-20260724T132616Z-5a5ba0, and the client label. A request log shows five API calls; this session record shows one company AI use case using three model turns to perform two release actions.

What happened to the data? The tested run’s records show email and person entities detected in the input and redacted before the request reached OpenAI. That happened because the request physically transited the control plane - data handling enforced in the path, not described in a policy document. (Honest scope: this covers the input path of intercepted traffic, nothing more.)

Which actions? Exactly release_status and release_prepare, each with a signed proxy record and a nonce-matched upstream receipt. No release_publish receipt exists for the run.

What did it cost?

Input:       7,031 tokens
Cache read:  32,128 tokens
Output:      123 tokens
Cost:        $0.003538

Not commercially meaningful in itself - the structure is the point. An account-level bill says OpenAI cost $20,000 last month; it cannot say which use cases consumed it, which sessions a spike maps to, or whether those sessions also touched internal systems. And there is a difference in kind, not degree, between reporting and enforcement: a cost dashboard tells you on Tuesday that a session overspent on Monday; a gateway budget denies the *next* model request before provider dispatch, a soft cap that preserves completed work while bounding the overrun at request granularity, demonstrated in a separate n8n scenario in the same demo repository.

Can it be verified? That deserves its own section - see it, right below.

The evidence is checked

The presentation/render is not the proof. The signed records are the proof, and the presentation is a readable projection of them.

After the run, the session is exported and verified:

talon audit export --format signed-json --session "$SESSION" --output evidence.json
talon audit verify --file evidence.json

Total records: 5
Valid records: 5
Invalid records: 0
Missing signature: 0
Could not parse: 0
Unsupported: 0

The demo’s presenter re-runs the whole chain before rendering anything human-readable: signatures verified, every record confirmed to belong to this session and this use case, the MCP records asserted to contain exactly the two expected tools, the upstream receipts nonce-checked. It fails closed on any mismatch. Records are signed with HMAC-SHA256 and independently verifiable against Talon’s published integrity specification. HMAC is symmetric, so be precise about what that buys: anyone holding the signing key can re-verify every record against the spec alone, which makes the records tamper-evident under the operator’s key custody - not immutable, and not third-party non-repudiation. Reproduce the run yourself and verify your own records; the pass criteria are public.

The same signed records feed several audiences from one source — developers inspect individual records, platform engineering reviews the timeline, FinOps reads spend, security reads data handling and actions — and they feed Talon’s compliance report generators (GDPR Article 30 and EU AI Act Annex IV packs) as supporting evidence for reviews, not a compliance determination.

Where the boundary ends

A control plane must be explicit about what it cannot see. This experiment proves control and evidence for model requests routed through Talon, MCP calls routed through Talon, and tool schemas carried in intercepted model requests. It does not prove control over:

• local shell commands and file edits (the demo restricts these with Copilot’s own --deny-tool flags - client-side safeguards, not Talon enforcement)

• browser actions

• direct APIs that bypass Talon

• what a governed tool did after the call was forwarded - Talon evidences the call that crossed the boundary, not the tool’s downstream execution

• the correctness of Copilot’s generated answer

• the identity of the operating process independently of its presented credentials

The record describes client provenance as client_asserted: operational attribution, not process attestation. One prerequisite is equally explicit: this pattern requires a client that can point at a custom endpoint with a company-held provider key, which Talon injects from its vault. Seat-licensed subscription traffic, billed through the vendor’s own OAuth, does not transit the gateway and cannot be governed this way.

These are not wording details — they determine which additional controls you still need: endpoint security, repository permissions, sandboxing, workload identity (SPIFFE-style), network restrictions. Treat any vendor’s non-interception list as an evaluation criterion; a control plane that won’t show you its boundary is asking you to assume it has none.

Rollout: observe, attribute, enforce

The way in is not “govern every AI system in the enterprise.” It is one existing workflow, three reversible steps:

1. Observe. 👁️ Point one coding assistant’s endpoint and MCP config at Talon and change nothing else. Developers keep their client; you get the session timeline you didn’t have yesterday.

2. Attribute. 🏷️ Turn on per-use-case identity and session cost. FinOps gets spend per use case and session; security gets data-handling visibility.

3. Enforce. 🛡️ Scope the tool surface and set session budgets. Actions outside the approved path are denied at the boundary — with signed denial evidence.

Pilot success criteria, from the run above: developers keep the existing client; model and MCP traffic share one identity and one session; data handling is visible in the record; cost is attributed to the session; every intercepted action is verifiable; no out-of-scope action reaches upstream. First-time setup is about twenty minutes with the guide; budget an afternoon if you include evaluation.

The deliverable that matters is not the first governed workflow - it is that the second one onboards as a config and policy change under an already-reviewed pattern. Security stops approving integrations and starts approving use cases: the review happens once, at the boundary, instead of once per tool.

Run it, then tell us what broke

You have always been responsible for what your assistant does with your credentials. Now you can hold the record of it.

⚡ 60 seconds, no API key: quickstart demo

🧭 20 minutes, the full Copilot scenario: governance guide

🧪 The complete experiment: talon-full-demo )

🦅 Talon (Apache 2.0)

And an open invitation: run this scenario against your own workflow - your assistant, your MCP integration, your policy, and co-author the writeup with us, backed by your own signed records. The strongest post in this category won’t be written by a vendor. It will be written by the first platform team that publishes its evidence.

In two weeks, a quiet part of the EU AI Act becomes applicable.

Most of the discussion around it is about what users should see. Banners. Labels. Watermarks.

I think that’s missing the real engineering problem.

Subscribe now

A simple example

Imagine a customer support assistant.

A user opens your website and starts a conversation. The request passes through an AI gateway, which selects a model based on company policy. The model generates a response. The frontend displays it together with an AI disclosure.

Everything looks correct.

Now fast-forward three months. An auditor — or simply your own security team — asks a few seemingly straightforward questions:

• Which model generated this response?

• Which policy was active at that moment?

• Was the AI disclosure actually shown to the user — and when?

• Can you prove all of this?

Most organizations can answer parts of these questions.

Very few can answer all of them with evidence.

We’ve solved this problem before

The instinctive answer is: “we already log everything.”

Logging is necessary. It isn’t sufficient. Logs describe isolated events — model selected, request completed, page published, banner rendered — but they can’t tell you whether those events belong to the same piece of content.

We’ve hit this wall before.

Microservices turned one request into fifty independent systems.

The solution wasn’t more logging.

It was distributed tracing: one identifier, propagated across every system a request touches.

AI-generated content is creating the same architectural problem — except the object being traced isn’t an HTTP request. It’s a piece of content, moving through generation, review, publication and translation over weeks instead of milliseconds.

And this time, correlation isn’t enough. Traces are sampled — dropping data is the design. They’re retained for weeks — auditors ask about years. Any service can emit spans into any trace — and rewrite them after the fact — with no signatures and no identity binding.

A trace tells you what probably happened, for debugging. This problem needs a record that proves what happened, for accountability.

That’s the difference between logging and provenance.

The information already exists

Every system already knows part of the story. The gateway knows which provider, model and policy were used. The application knows whether disclosure was rendered. The editorial workflow knows who approved the content. The CMS knows when it was published. Individually those records are correct. Collectively they cannot reconstruct the lifecycle.

The problem isn’t missing information. It’s missing continuity.

What the law actually asks

The quiet part of the AI Act is Article 50, its transparency chapter. It applies from 2 August 2026. Providers must design chatbots so people know they’re talking to AI. Deployers must label deepfakes and certain AI-generated text. Getting it wrong can cost up to €15 million or 3% of worldwide turnover — whichever is higher.

But Article 50(2) is the part worth reading twice: generative outputs must be “marked in a machine-readable format and detectable as artificially generated or manipulated”, through technical solutions that are “effective, interoperable, robust and reliable” as far as technically feasible. Machine-readable. Interoperable. Protocol words, not UI words.

(One timing nuance: the just-adopted Digital Omnibus gives systems already on the market until 2 December 2026 for the machine-readable marking. The disclosure duties still start on 2 August.)

One thing Article 50 never says: keep records. The Act’s logging obligations apply only to high-risk systems - a different chapter entirely. The proof pressure comes from enforcement instead. Market surveillance authorities can demand documentation from any operator. And the EU’s new Code of Practice on AI-content transparency - finalized in June, voluntary — makes the trade explicit: sign it, and you get a recognized way to show compliance; don’t, and you prove to authorities, case by case, that your own measures are good enough.

The law says: disclose and mark.

Enforcement asks: can you show that you did?

The obligations span roles; the evidence spans systems; the systems share no identifiers.

The exemption that proves the point

A marketing team drafts an article with an LLM. An editor rewrites it. Legal reviews it. The CMS publishes it; a translated version gets syndicated. Months later, somebody asks whether this content originated from AI and whether transparency obligations were met.

Now read Article 50(4). Its text-disclosure duty is narrow to begin with: it only covers AI-generated text published to inform the public on matters of public interest. And even where it applies, it exempts content that has undergone “a process of human review or editorial control” where someone “holds editorial responsibility for the publication of the content”.

Both limbs are questions of fact. What was this published for? Did a human review it — and who owned that responsibility?

The legally decisive artifact isn’t the label. It’s the evidence.

The statute never says “build an evidence chain.” It just makes relying on its exemptions untenable without one.

The missing layer: operational provenance

There’s already a standard for part of this story. C2PA (Content Credentials) gives you asset provenance: a signed history that travels with the file. It matters, and adoption is growing. But manifests are routinely stripped when content crosses platforms, the standard barely covers text - the dominant enterprise output - and by design it says nothing about the operator’s runtime: which policy was in force, why the router picked that model, whether disclosure duties were met.

What organizations increasingly need alongside it is operational provenance: a verifiable record of every decision surrounding an AI generation, across every system that touches it.

Asset provenance travels with the file. Operational provenance survives on the server, where a C2PA manifest can reference it, not replace it.

None of the cryptography is new. That’s a feature. It’s the same append-only, verifiable-log pattern Certificate Transparency and sigstore proved at internet scale. What doesn’t exist yet is that pattern applied to AI runtime decisions — across systems.

A control plane for AI operations

The next generation of AI infrastructure won’t stop at routing requests to models. It will treat every generation as an operational event that accumulates evidence over its lifetime:

AI generation → runtime policy applied → disclosure shown → human review completed → content published → evidence exported.

Almost everything in that chain belongs to the surrounding operational ecosystem, not the model. That’s why solving transparency exclusively inside an AI gateway will never be enough. Every serious gateway logs today — that’s not the gap. The gap is that the gateway’s record ends at its own boundary. Nothing downstream — the frontend’s disclosure, the editor’s review, the CMS’s label — can attach to it.

Where Talon fits

This is the direction we’re exploring with Talon.

Talon isn’t trying to become a legal compliance platform. It doesn’t decide when Article 50 applies. It doesn’t render disclosures. It doesn’t watermark images. Those responsibilities belong to applications, CMS platforms and legal teams.

Talon records the operational provenance of each AI generation: which agent initiated it, which provider and model were selected, which effective policy governed the decision, why routing occurred. Each record is signed and independently verifiable against a published integrity specification.

The next step is letting downstream systems extend that record with their own evidence. A frontend recording that disclosure was presented. A CMS recording that labeling was applied. An editorial workflow recording that human review completed, the exact fact the text exemption turns on.

The result isn’t a prettier dashboard. It’s a verifiable operational history.

One honest caveat: attached evidence is attested, not omniscient. The chain proves which system claimed what, when, under which identity not that every claim is true. That’s how every audit regime works. What changes is that the claim becomes tamper-evident and durably bound to the generation it describes.

Beyond the regulation

It’s tempting to frame all of this around the AI Act. But even without regulation, the same questions keep arriving — from customers, partners, procurement teams and internal auditors.

Those aren’t legal questions. They’re operational questions. Article 50 just makes them harder to ignore.

A Monday-morning test: pick one AI-generated output that shipped last quarter and reconstruct its full story — model, policy, disclosure, review, publication — with evidence, not recollection. If you can’t, start small. Mint a stable generation ID at the point of generation. Sign the record. Propagate the ID as far downstream as it will go.

We spent decades building infrastructure to answer one question: what happened to this request?

AI introduces a different question: what happened to this piece of generated content?

I suspect the next generation of AI infrastructure will be built around answering it.

If you’re solving this differently, I’d genuinely like to hear how.

That’s the whole idea behind the pattern that’s quietly becoming the default way to run coding agents: let a frontier model - Claude Fable 5, with its 1M-token context read the entire repo, write the plan, and review the result, while cheaper models Claude Opus 4.8, or Codex on OpenAI’s side do the fetching, editing, testing, and retrying. Setup guides for Hermes and OpenClaw are already prescribing exactly this split; if your team runs coding agents seriously, some version of it is probably in your dotfiles by autumn.

The pitch is cost-effectiveness. And the pitch is right, below is the math with list prices. But there’s a second promise hiding inside the pattern that almost nobody talks about, and it’s the one that matters to whoever signs the bill: done right, this architecture makes agent costs predictable. Decomposable into parts you can reason about, meterable in numbers you can trust, and cappable at the level where the money actually moves the session.

Here’s the pattern, the real economics, the three places predictability leaks away, and how to plug them with four HTTP headers and one line of YAML.

Subscribe now

The pattern in one diagram

The economic logic:

Planning is few, expensive, high-leverage tokens - you want the smartest model and the biggest context exactly once;

Execution is many, cheap, repetitive tokens - the transcript gets re-read on every agentic turn, and 90%+ of that re-reading is served from the prompt cache at a tenth of the price.

What a session actually costs

List prices, per million tokens( actual as for 5th of July, 2026):

A realistic very simple feature-build session - one planning pass over a mid-size repo, two executors working the plan:

Three things to mention:

1. The architect is 65% of the spend on 6% of the calls. That’s not waste - that’s the design. You paid $4.40 for the one artifact that determines whether the other $2.37 produces working code or thirty loops of confident nonsense.

2. The builders are cheap because of the cache. Over half of all input tokens in this session are cache reads at 0.1× the input rate. The marginal executor turn costs about a tenth of a cent-per-thousand-token intuition suggests. This is what makes execution cost linear and boring — exactly what you want.

3. Each part is predictable on its own terms.

Planner cost ≈ repo context × rate, a known quantum per task.

Executor cost ≈ turns × a small, cache-discounted constant.

Decomposable costs are forecastable costs.

One honest footnote: prompt caches don’t cross models. Fable’s cached repo context is not readable by Opus or Codex, each executor builds its own cache on turn one and harvests it for the rest of the session.

Does that pattern always win?

Fabio Akita’s widely-cited three-round benchmark concluded that planner–executor splits are often “premature optimization”: in his tests, solo Opus 4.7 beat every mixed Anthropic combo on quality at comparable cost. It’s a careful benchmark with a public repo, and if your workload looks like his, solo-frontier is a fine answer. Two things his own data also shows: the one decisive economic win in his rounds was a planner–executor split (a high-effort GPT planner with a medium executor came in 80–85% cheaper than solo at near-equal quality), and orchestration paid off exactly where tasks decompose cleanly, which is what large coding tasks increasingly do. But this post doesn’t need the pattern to win the benchmark. If you run multi-model sessions at all, and the tooling ecosystem is pushing you there, the operational questions below hit you regardless of whether your split beats solo Opus by 5% or loses by 5%.

Predictability leaks

I built an AI gateway, watched this traffic all day. Predictability dies in three specific places.

• Leak 1: your meter doesn’t understand cached tokens. The two providers report caching in opposite shapes. Anthropic reports cache reads and writes as separate counts - ‘input_tokens’ excludes them. At the same time, OpenAI reports ‘cached_tokens’ as a subset of ‘input_tokens’.

• Leak 2: the bill can’t be attributed. The planner spend lands on your Anthropic invoice; the Codex spend lands on OpenAI’s. Two dashboards, two currencies of “usage”, zero concept that these were one piece of work.

• Leak 3: nothing stops a runaway session. The industry answer to agent overspend so far is coarse: monthly per-tool caps (Uber’s much-covered $1,500/tool policy), per-user or per-team budgets (OpenAI’s enterprise controls, Cloudflare’s gateway limits). All useful; none match the failure mode. A looping executor doesn’t blow a month — it blows a Tuesday afternoon, spread across two providers so that neither one’s limiter sees the whole picture. To my current knowledge, no popular tool or provider control enforces a budget at the session grain, across providers, with a record of the denial you can hand to finance.

Plugging the leaks: four headers and one line of YAML

This is the part we built. Talon is an open-source (Apache-2.0) gateway: point both tools at it, and every request carries the session identity the pattern already implies.

# planner / executors all send the SAME session id — that's the entire contract
-H "X-Talon-Session-ID: sess-payments-refactor"
-H "X-Talon-Agent-ID: planner"            # free-form labels, your names
-H "X-Talon-Parent-Agent-ID: orchestrator" # optional parent link

Claude Code and Codex CLI don’t even need that — their native session/subagent headers are recognized automatically. And the cap is one line on the caller:

olicy_overrides:
  max_session_cost: 10.00   # soft cap per coding session

What you get is something like this:

The session becomes one governed unit across both(Anthropic and OpenAI) providers:

Session sess-coding-demo
  Requests:  10 (8 allowed, 2 denied, 0 error)
  Providers: anthropic, openai
  Tokens:    in 43 / out 334 / cache-read 14352 / cache-write 840
  Cost:      €0.012551

  Per-agent:
    generator                 8 req  €0.011960
    executor  ←generator      2 req  €0.000591

Cache reads metered as cache reads, priced at cache rates, with the pricing basis recorded in the evidence - leak 1 closed. One session, two providers, per-subagent rollup - leak 2 closed.

And when the session hits its cap, it stops, on either provider’s route:

request 6 → HTTP 403
session_budget_exceeded: session spend 0.01 + estimate 0.01 exceeds limit 0.02

The denial isn’t a log line — it’s a signed evidence record carrying the exact numbers the decision was made on {limit, spent, estimate}, and the whole session verifies cryptographically:

  Session sess-coding-demo: 10 record(s), 10 valid, 0 invalid

Leak 3 closed, with a receipt.

Try it in 30 seconds (no API keys)

git clone https://github.com/dativo-io/talon && cd talon
make coding-agents-demo        # Docker; fully offline, deterministic

The demo runs the whole story against a deterministic mock that speaks both providers’ wire formats, including streaming and cache tokens. Every figure in this post can be recomputed from it. In a field where the loudest numbers are self-reported savings percentages and anonymous nine-figure invoice anecdotes, we’d rather hand you a demo than an assertion.

What this doesn’t do

• Attribution is not authentication. Subagent names are client-asserted labels within an already-authenticated caller. Budgets bind to the caller and session — never to a label an agent could spoof.

• The cap is soft. A single in-flight request can overshoot before the next one is denied. Hard reservation is on the roadmap.

• Local tools are invisible. Talon governs what crosses it - model and MCP traffic. Your agent’s shell commands run on the developer’s machine.

The takeaway

Use Fable 5 where a million tokens of context and frontier reasoning earn their price: the plan.

Use Opus and Codex where the cache makes tokens nearly free: the execution. That split is genuinely cost-effective — about $6.80 for a session that touches a 400k-token repo across 32 calls. But cost-effective only becomes cost-predictable when the session is a first-class thing: metered with cache-aware honesty, attributed per subagent across providers, and capped with a stop you can prove happened. That’s four HTTP headers and one line of YAML away.

*Talon is open source. The coding-agents guide has the full setup for Claude Code and Codex CLI; the demo leaves you holding the signed evidence.

That era is over.

We can stop debating if LLMs are an effective part of the software engineering stack. The business world has already bought into the velocity gains of autonomous coding engines and advanced completion systems. They are entrenched, and they are with us for the long haul.

But as these models transition into core production infrastructure, they are simultaneously fracturing along national borders. We are entering an era of passport-gated compute, where access to top-tier reasoning is no longer a standard web request, but a geopolitical privilege.

If your application architecture treats a specific foreign-hosted frontier model as a hardcoded, direct dependency, your systems are structurally vulnerable to a new kind of risk: systemic technological exclusion.

The Threat of Structural Technological Exclusion

When a model like Claude Fable 5 launches, it represents a fundamental shift away from simple chat windows toward long-horizon, autonomous execution. These architectures are engineered to compress months of multi-file codebase refactorings, complex migrations, and deep analytical compliance loops into a few hours of agentic execution.

Because the business side expects this 10 X times ( or what ever magic number) development velocity as the new baseline, losing access to this class of compute is no longer a minor operational glitch.

The real danger in a fragmented world is not that an API endpoint drops for ten minutes while a cloud provider reboots a router. The danger is that non-US companies, international divisions, and foreign contractors face a sudden, legal exclusion from cutting-edge technology entirely.

If a US-based competitor can refactor their entire monolithic core in an afternoon using native access to Tier 1 frontier models, while your European or international team is restricted to running legacy, open-weights models locally due to export boundaries, you aren’t dealing with a downtime event. You are dealing with a permanent, compounding competitive penalty.

Timeline of loosing it

The rapid policy shifts surrounding recent model rollouts offer a stark demonstration of how quickly a flat developer ecosystem can dissolve under state pressure. The 18-day geopolitical stalemate between Anthropic and the U.S. government serves as a blueprint for the future:

• June 12: Citing national security concerns over dual-use capabilities, the U.S. Commerce Department’s Bureau of Industry and Security issued an emergency export control directive under the Export Administration Regulations (EAR). It ordered Anthropic to immediately suspend access to Fable 5 and Mythos 5 for all foreign nationals—including Anthropic’s own overseas employees. Because user nationality cannot be verified dynamically at a raw API boundary, Anthropic was forced to implement a blanket global shutdown of the models to ensure compliance.

• The Shockwave: Engineering environments outside the U.S. vanished overnight. Teams spanning from Canada to Central Europe were instantly locked out of active development pipelines. A partial relaxation days later allowed access to be restored—but only for verified, trusted U.S. organizations via early deployment pipelines. The developer ecosystem was split into distinct tiers.

• July 1 (The Degraded Return): While the U.S. Commerce Department officially lifted the export ban after intense weeks of coordination, the global “return” of Fable 5 proved that the era of frictionless compute is dead. Anthropic’s immediate deployment restrictions show the new normal: the model is heavily throttled (capped at 50% of limits through July 7 before hitting a strict usage-credit paywall).

• The Hyper-Sensitive Classifier: To satisfy state security mandates, Fable 5 now runs behind an incredibly aggressive defense-in-depth safety filter. If an engineering prompt remotely mimics a potential security exploit or vulnerability edge case, the request is aggressively intercepted and downgraded.

• The Death of Zero-Data-Retention (ZDR): To feed these safety classifiers, Anthropic now mandates a strict 30-day input/output data retention window for both Fable 5 and Mythos 5. For enterprise teams dealing with proprietary codebases and rigid data sovereignty compliance, native access is effectively closed off by compliance design.

The lesson for platform engineers is explicit: Frontier model access can change hourly based on compliance frameworks entirely outside your codebase, and identity validation at the passport level is the new baseline for compute.

Direct Model Access Is a Production Liability

The standard integration pattern works perfectly for a local prototype:

app → claude-fable-5

In production, this pattern is brittle. If the upstream endpoint rejects an execution thread due to a sudden jurisdictional block, an identity token mismatch, or an updated export policy, your application pipeline drops dead.

Worse, makeshift fallback code written under pressure frequently creates catastrophic security vulnerabilities. If an automated engineering pipeline hits a geofence on Fable 5 and blindly reroutes the raw, proprietary payload to an alternative public endpoint that lacks explicit zero-data-retention (ZDR) terms, the system did not gracefully recover, it just traded an availability issue for an irreversible compliance and IP violation.

Hardcoding a single frontier model into an enterprise system is exactly like hardcoding a single cloud availability zone with no multi-region failover plan.

The Brittle Reality of Silent Degradation: With 1st of July, 2026 rollout, if an autonomous engineering agent attempts a codebase-wide migration and its context payload triggers the hyper-sensitive safety classifier, the endpoint doesn’t just return a clean 403 error. It silently offloads the execution thread to the lower-tier Opus 4.8. For long-horizon agentic workflows engineered around Tier 1 reasoning, this silent capability drop causes pipelines to break mid-stream without a clear architectural failover.

Request Capabilities, Not Model Names

To survive a fragmented AI market, applications must be entirely decoupled from explicit model identifiers. Instead of hardcoding direct API calls, insert an internal abstraction layer:

app → AI gateway → allowed model/provider

The application layer should only request a functional capability and pass along the necessary operational context tokens:

• coding-frontier

• legal-zdr

• eu-sensitive

• finance-deep-review

The gateway resolves the optimal, compliant model instance at runtime by evaluating a real-time policy matrix:

• Who is running the execution thread?

• Where is the tenant registered, and what is the data classification tier?

• What are the strict retention constraints of this specific payload?

• Is the primary target model currently accessible under these constraints?

• What is the exact, pre-approved fallback route if the primary target is blocked?

The core architectural rule shifts from “Send this to Fable 5” to “Send this to the most capable model allowed for this specific payload context.”

Managing Context Without Identity Creep

Consider a practical execution: An international development team based in Poland initiates a session with an internal coding agent. The app requests the coding-frontier capability. The gateway intercepts the payload and evaluates the environmental context:

Context Evaluated:
- caller:            engineering-agent
- tenant:            EU company
- user jurisdiction: Poland
- data tier:         internal source code
- retention req:     zero data retention
- preferred model:   Claude Fable 5
- Fable status:      unavailable / restricted / no ZDR

The gateway instantly executes an explicit policy decision: Reject Fable 5. It automatically evaluates the approved fallback list, matching the payload to an available instance of Mistral, Azure OpenAI inside an EU region, or an enterprise Bedrock endpoint that strictly honors the zero-data-retention rule.

To do this safely, the gateway must not digest raw identity records. Shipping passport data or citizenship details inside an application payload is a severe anti-pattern. Instead, identity and HR platforms must map users to derived, lightweight entitlement flags before the gateway ever touches the request:

caller = engineering-agent
tenant = EU company
ai.fable_allowed = false
ai.eu_only = true
ai.zdr_required = true
data_tier = internal source code

The gateway does not need to know why ai.fable_allowed is false; it only needs to read the boolean token to route or deny the execution deterministically.

Why the Gateway Must Be Internal Infrastructure

When engineers realize they need an abstraction layer, the common temptation is to use a third-party cloud SaaS gateway. This is a severe architectural error that trades a configuration problem for a massive data-sovereignty vulnerability.

The core rule of distributed systems applies directly here: control planes can be remote, but data planes must be local.

1. The Compliance Paradox

The underlying reason to build an AI routing layer is to guarantee regional data boundaries and enforce zero-data-retention rules. If you pass your traffic through a third-party cloud SaaS gateway, you are sending raw, unencrypted prompts, internal source code, and customer records to another intermediary entity before it reaches the final LLM provider. You have expanded your data processor surface area, not contained it.

2. The Latency and Streaming Tax

Agentic workflows live and die by Time to First Token (TTFT) metrics. Forcing your traffic through an external cloud proxy adds a redundant public internet hop, duplicate TCP handshakes, and extra TLS terminations. Furthermore, if that third-party SaaS buffers parts of the Server-Sent Events (SSE) stream to run out-of-band cost logging or analytics, interactive streaming performance drops immediately.

3. Artificial Single Points of Failure

An external SaaS gateway creates a mathematical availability bottleneck. Your total system uptime is no longer just dependent on your app and the final AI model - it is tied to the stability of the intermediate platform.

If the intermediate proxy experiences a routing loop or a regional cloud outage, your entire product fails—even if your core services and the underlying LLM endpoints are 100% healthy.

4. Policy Locality and Context Hydration

To make real-time routing choices, the gateway needs instant access to context. When hosted locally inside your own cluster or VPC perimeter, the gateway can query internal microservices or a local cache with sub-millisecond latency. An external cloud SaaS forces you to “hydrate” the payload, bundling sensitive tenant profiles and location metadata into outbound headers just so an external platform can parse them.

The correct architectural pattern is clear: run the gateway container as a local sidecar or an internal VPC ingress proxy. The data plane stays strictly within your perimeter, while an external control plane is used exclusively to pull down policy rulebooks and configuration updates asynchronously.

Evidence and Auditability

Every automated reroute or denial must be fully auditable. If your system silently switches models behind the scenes, you must be able to prove why that choice was made during post-incident reviews or compliance audits. The internal gateway should produce cryptographically signed telemetry records detailing:

- requested capability & requested model
- selected model & rejected candidates
- explicit rejection reasons (e.g., policy_zdr_violation)
- caller identity & tenant details
- policy engine version & provider metadata state
- final decision status: allowed / denied / rerouted

Without this audit trail, you cannot answer fundamental operational questions: Why did our costs spike yesterday? Did sensitive source code leave our regional data boundary during a fallback event? Was our data processed in compliance with our customer service-level agreements?

Building for the New Baseline

This is precisely where specialized, self-hosted gateway engines like Talon fit into the modern enterprise stack.

Talon deploys directly into your own infrastructure perimeter. It sits inline between your application code, autonomous agents, and upstream model providers. Before a single byte leaves your network, it identifies the caller, checks local policy, strips PII, enforces model allowlists, manages rate limits, strips unapproved agent tools, and signs the execution evidence.

Talon does not bypass government export controls or magically unlock restricted models for entities that are legally barred from using them. It solves the real engineering problem: ensuring your production applications do not break if a single provider endpoint alters its availability policies overnight. It turns frontier models from hardcoded, brittle dependencies into interchangeable, policy-validated infrastructure backends.

Advanced LLMs as coding assistants and autonomous engineering blocks are a permanent fixture of our industry. But because they are infrastructure, we can no longer afford to wire production code directly to them. Relying on the hope that an external API vendor’s regulatory landscape will remain globally uniform is not a design strategy.

Infrastructure requires routing, fallback patterns, policy enforcement, and auditable evidence. If frontier AI is now production infrastructure, it is time to start architecting like it.

Over the last few months, I’ve seen the same problem appear across drastically different companies:

• Startups trying to rein in spiraling coding-agent bills.

• SMBs aggressively adding AI to their support and operations pipelines.

• Enterprises opening the floodgates by giving employees access to multiple AI tools.

Despite the different contexts, they are all asking the same question: “How do we manage AI spending without blocking useful AI work?”

This question isn’t arising because AI failed. It’s happening because AI started working well enough to become expensive. As Reuters recently noted, while base token prices are falling, real AI bills are still rising. Today’s tasks utilize longer contexts, more steps, massive files, multiple retries, and complex agentic workflows. Smart companies are beginning to shift routine work to cheaper models, reserving expensive frontier models for only the hardest tasks.

That is the right pattern. But to execute it, you need infrastructure.

Subscribe now

Token Spend Behaves Exactly Like Cloud Spend

AI spending is mirroring the early days of cloud computing. The cycle is predictable:

1. Usage is heavily encouraged to drive innovation.

2. Every team adopts it.

3. A massive, unexpected bill arrives.

4. Finance scrambles to ask who owns the spend.

The exact same phenomenon that happened with compute and storage is now happening with tokens. The WSJ recently highlighted companies applying traditional cloud FinOps patterns to AI token spend: dashboards, spend caps, showback, chargeback, smaller models, and strict usage accountability.

This is where the industry is heading. The solution is not to “ban AI” or blindly force the cheapest model everywhere. The solution is controlled AI usage categorized by team, application, model, and data type.

The Core Problem: Direct Model Access

During the experimentation phase, most AI usage looks like a spiderweb of direct connections:

• Slack bot → OpenAI

• Coding agent → Anthropic

• Internal app → OpenAI

• Support tool → Claude

• Employee → Personal AI account

• Vendor AI → Unknown model

This works well for prototypes, but it shatters in production. Every direct path creates its own siloed cost behavior, model preference, data retention assumption, API key, log system, and compliance risk. At a company scale, this architecture leaves leadership unable to answer basic questions:

• Which teams are spending the money?

• Which models are they actually using?

• Which requests contain PII?

• Which vendor saw which sensitive data?

This is why AI cost control cannot happen after the fact. It must happen at the traffic layer.

The Practical Controls Companies Need

If a company wants to manage AI spending properly, it needs to implement controls before the model call happens.

1. Caller Identity

Every AI request must be tied to an application or team identity (e.g., support-slack-bot, engineering-coding-agent, finance-document-assistant). Without caller identity, cost ownership is impossible.

2. Per-Caller Budgets

Each caller should have hard daily and monthly limits. This is the AI equivalent of cloud budget alerts, but the crucial difference is that the request is denied before the spend happens.

YAML

gateway:
  callers:
    - name: support-slack-bot
      tenant_id: support
      policy_overrides:
        max_daily_cost: 10.00
        max_monthly_cost: 200.00

3. Model Allowlists

Not every team needs access to a frontier model. Routine support, text classification, summarization, and internal search can operate perfectly on lighter models.

YAML

policy_overrides:
  allowed_models:
    - gpt-4o-mini
    - claude-haiku

4. Intelligent Model Routing

The golden rule is not “use the cheapest model.” The rule is: Use the cheapest model allowed for this specific request.

• Public data can route to cheap models.

• Sensitive data may require EU routing.

• HR and legal tasks require strict zero-retention policies.

5. Context and Loop Limits

Agentic coding and document workflows can burn tokens silently. Recent studies on agentic tasks show that token usage can vary by up to 30x for the exact same task—and higher usage doesn’t always equal higher accuracy. You need circuit breakers:

YAML

policies:
  resource_limits:
    max_iterations: 10
    max_tool_calls_per_run: 20
    max_cost_per_run: 0.50

6. Semantic Caching

If hundreds of users ask a similar question, the system should not call the LLM every time. Caching public or low-risk answers is the fastest way to slash inference costs. However, caching must be aware of context: strictly avoid caching PII, confidential data, or dynamic tool calls.

Why Price Isn’t Enough: The Claude Fable 5 Example

Claude Fable 5 perfectly illustrates why AI spending is fundamentally a routing problem, not just a cost problem.

Fable-class models are phenomenal for heavy lifting: deep analytics, massive context windows, and complex autonomous agents. However, they come with premium pricing and operational sensitivity. Reuters recently reported that companies like Microsoft have had to restrict employee use of advanced models like Fable 5 due to data-retention concerns.

The most capable model isn’t always the right model. While Fable 5 might be perfect for public code analysis, it may need to be blocked entirely for zero-data-retention workloads in HR or healthcare. This proves that model choice should never be hardcoded into an application; it must be dictated by a centralized policy.

The Solution: Enter Talon

Talon introduces a much-needed control layer between your applications and the model providers. Instead of an unmanageable web of direct API calls, Talon centralizes the traffic.

Before forwarding any request, Talon can automatically:

1. Identify the caller and evaluate their policy.

2. Check rate limits and estimate the cost.

3. Scan for PII and classify the data tier.

4. Filter dangerous tools and redact sensitive text.

5. Route the prompt to the cheapest allowed provider.

6. Record cryptographically signed evidence of the transaction.

AI cost control should not be a depressing monthly report; it should be a real-time, automated runtime decision.

What to Measure

To effectively govern this new infrastructure, move away from looking at a single, terrifying AI bill. Instead, track these specific metrics to see exactly where your spend is coming from and whether your policies are working:

Final Thoughts

Direct provider access is perfectly fine for early prototypes or low-risk, public-data applications. But once AI becomes shared, expensive, sensitive, or agentic, direct access is the wrong abstraction.

You cannot control AI spending by simply asking developers to “use fewer tokens”—that didn’t work for cloud compute, and it won’t work now.

Cheaper models, caching, and better prompting will help, but ultimately, companies need a centralized control point. AI is no longer just a neat tool employees play with; it is core traffic moving through your company’s nervous system. And traffic needs policy before it becomes spend.

A friend of mine runs platform engineering at a ~200-person B2B company in Germany. Earlier this year they were closing their biggest deal to date - enterprise customer, everything agreed except the security review.

Question 47 of the enterprise questionnaire shared in long excel: “Describe how personal data in AI/LLM workflows is governed, including records of processing, sub-processors, and third-country transfers.”

He took it to their CTO. The CTO opened their information systems register, the register every European company is required to keep under GDPR Article 30( called there Record of Processing Activities - RoPA) - it can be as simple as confluence, and found entries for the HR system, the CRM, the email marketing tool. Nothing about the support bot that had been summarizing customer tickets through AI for eight months. Oops.

“Where do our prompts go?” CTO asked. In theory , the knew that the support bot they purchased use OpenAI’s ChatGPT, but nobody in the room could answer with evidence where the prompts went and what was inside the prompts. There were logs, somewhere, spread across three SaaS dashboards. There was no record.

The deal closed five weeks late - five weeks of a platform engineering and a CTO reverse-engineering their own AI usage so they could write it down and have legal sign it. That’s the moment AI governance stops being a legal abstraction and becomes a sales blocker.

Why AI traffic belongs in your Record of Processing Activities ( EU focused part)

If you’re the engineer, focusing on EU, who has to answer question 47, here’s the 90-second version of what’s being asked.

The Record of Processing Activities - RoPA (GDPR Art. 30) is a register that answers, per processing activity: what personal data do we process, why, about whom, who receives it, does it leave the EU, how long do we keep it, and how is it protected? It’s mandatory for nearly every European company. There’s a nominal under-250-employee exemption, but it doesn’t apply when processing is “not occasional” — and a support bot running every day is by definition not occasional. The RoPA is the first document requested in a regulator inquiry, an ISO 27001 surveillance audit, and most enterprise security reviews. Art. 30 violations sit in the fine tier of up to EUR 10M or 2% of global turnover.

Annex IV (EU AI Act) is the technical documentation required for high-risk AI systems: what the system is, how it’s monitored and controlled, how risks are managed, how humans oversee it. High-risk obligations apply from August 2, 2026. Most mid-size companies are deployers rather than providers, which means lighter obligations — but the documentation demand flows down through contracts anyway. Your enterprise customers will ask you to evidence your usage controls, oversight, and logging regardless of your formal role. Documentation-tier violations run up to EUR 15M or 3% of turnover.

AI traffic is the gap in both documents. Companies have RoPA entries for systems built ten years ago, while prompts flowing to a US model provider — a new processing activity, with a new recipient, and often a third-country transfer — go unrecorded. It’s exactly the thing CTO/Legal/Compliance are now being asked about, and exactly the thing they can’t answer from scattered logs.

Here’s what I realized while building Talon’s gateway: the network layer already knows the answers. Which PII categories were observed in prompts. Which provider received them, in which region. What was redacted, what was blocked, what it cost. Talon records all of that per request, HMAC-signed at write time.

Some consultant writing your RoPA guesses at these facts. The gateway proves them.

Declared + derived: ten lines of YAML, then evidence does the rest

Declared facts are business statements no log can know: who the controller is, why you process data, how long you keep it. Your DPO writes them once. Org-level identity goes in talon.config.yaml:

compliance:
  controller:
    name: "Example GmbH"
    contact: "privacy@example.eu"
    dpo_contact: "dpo@example.eu"
    address: "Examplestr. 1, 10115 Berlin, Germany"

Per-agent declarations live in agent.talon.yaml, next to the policy that governs the agent:

compliance:
  frameworks: [gdpr, eu-ai-act]
  data_residency: eu
  declarations:
    processing:                      # GDPR Art. 30(1) facts
      purposes:
        - "customer support ticket triage"
      data_subject_categories:
        - "customers"
      personal_data_categories:
        - "contact details"
        - "payment identifiers"
        - "support ticket content"
      retention_period: "90 days after ticket closure"
      legal_basis: "contract (Art. 6(1)(b))"
      safeguards: "Role-based access; vendor DPAs on file; signed evidence retained for audit review"
    system:                          # EU AI Act Annex IV facts
      system_description: "Gateway-governed LLM assistant for support ticket triage"
      intended_purpose: "Summarize and route inbound support tickets"
      oversight_description: "Support lead reviews flagged tickets daily"

Derived facts come from the signed evidence store, and you never write them by hand: which processing activities actually ran (per tenant, per agent, first seen, last seen), which personal-data identifiers were actually observed, which recipients received data and in which region, which requests were third-country transfers, which policy denials fired, and which requests went through a human plan-review gate.

One design decision matters more than it looks: every governed request records where it went — not just the ones where PII was detected. A recipient list that depends on a classifier’s hit rate is a recipient list with silent holes; a missed identifier should never make a US provider disappear from your transfer table. Talon records the prompt → destination flow (provider, model, region) for all traffic — gateway requests, CLI and scheduled agent runs, MCP tool calls, and externally orchestrated graph runs — and layers the sensitivity classification on top. The rule “a record claiming a model call must say where the data went” is enforced as a runtime invariant on every evidence write, in CI by a parity contract test, and in the release smoke suite. The full controls-per-path breakdown is in the governance control matrix.

Then one command merges declared and derived facts into each document:

talon compliance ropa --format html --output ropa.html
talon compliance annex-iv --format html --output annex-iv.html

The HTML is print-to-PDF-ready. The JSON variant is machine-checkable, for the security reviewers who want to diff it between quarters.

Missing declarations are flagged, not hidden

Your GDPR RoPA Is Missing Your AI Traffic — Here’s How to Fix It With Runtime Evidence

TL;DR: Every European company maintains a Record of Processing Activities. Almost none of them have an entry for the prompts their teams send to OpenAI every day — the fastest-growing processing activity in the building. Dativo Talon, an open-source AI governance gateway, now generates a GDPR Art. 30 RoPA and an EU AI Act Annex IV documentation pack directly from HMAC-signed runtime evidence: talon compliance ropa and talon compliance annex-iv. Your compliance officer declares ten lines of YAML once; everything else is derived from records a consultant cannot fabricate. There’s a downloadable sample pack you can hand to a reviewer today.

A friend of mine runs platform engineering at a ~400-person B2B company in Germany. Earlier this year they were closing their biggest deal to date — six figures, enterprise customer, everything agreed except the security review.

Question 47 of the questionnaire: “Describe how personal data in AI/LLM workflows is governed, including records of processing, sub-processors, and third-country transfers.”

He took it to the compliance officer, who opened the company’s RoPA — the register every European company is required to keep under GDPR Article 30 — and found entries for the HR system, the CRM, the email marketing tool. Nothing about the support bot that had been summarizing customer tickets through GPT-4 for eight months.

“Where do our prompts go?” the compliance officer asked. Nobody in the room could answer with evidence. There were logs, somewhere, spread across three SaaS dashboards. There was no record.

The deal closed five weeks late — five weeks of a platform engineer and a compliance officer reverse-engineering their own AI usage so they could write it down and have legal sign it. That’s the moment AI governance stops being a legal abstraction and becomes a sales blocker.

Why AI traffic belongs in your Record of Processing Activities

If you’re the engineer who has to answer question 47, here’s the 90-second version of what’s being asked.

The RoPA (GDPR Art. 30) is a register that answers, per processing activity: what personal data do we process, why, about whom, who receives it, does it leave the EU, how long do we keep it, and how is it protected? It’s mandatory for nearly every European company. There’s a nominal under-250-employee exemption, but it doesn’t apply when processing is “not occasional” — and a support bot running every day is by definition not occasional. The RoPA is the first document requested in a regulator inquiry, an ISO 27001 surveillance audit, and most enterprise security reviews. Art. 30 violations sit in the fine tier of up to EUR 10M or 2% of global turnover.

Annex IV (EU AI Act) is the technical documentation required for high-risk AI systems: what the system is, how it’s monitored and controlled, how risks are managed, how humans oversee it. High-risk obligations apply from August 2, 2026. Most mid-size companies are deployers rather than providers, which means lighter obligations — but the documentation demand flows down through contracts anyway. Your enterprise customers will ask you to evidence your usage controls, oversight, and logging regardless of your formal role. Documentation-tier violations run up to EUR 15M or 3% of turnover.

AI traffic is the gap in both documents. Companies have RoPA entries for systems built ten years ago, while prompts flowing to a US model provider — a new processing activity, with a new recipient, and often a third-country transfer — go unrecorded. It’s exactly the thing compliance officers are now being asked about, and exactly the thing they can’t answer from scattered logs.

Here’s what I realized while building Talon’s gateway: the network layer already knows the answers. Which PII categories were observed in prompts. Which provider received them, in which region. What was redacted, what was blocked, what it cost. Talon records all of that per request, HMAC-signed at write time.

A consultant writing your RoPA guesses at these facts. The gateway proves them.

Declared + derived: ten lines of YAML, then evidence does the rest

Every auditor document splits into two kinds of facts, and Talon’s design keeps them strictly separate.

Declared facts are business statements no log can know: who the controller is, why you process data, how long you keep it. Your compliance officer writes them once. Org-level identity goes in talon.config.yaml:

compliance:
  controller:
    name: "Example GmbH"
    contact: "privacy@example.eu"
    dpo_contact: "dpo@example.eu"
    address: "Examplestr. 1, 10115 Berlin, Germany"

Per-agent declarations live in agent.talon.yaml, next to the policy that governs the agent:

compliance:
  frameworks: [gdpr, eu-ai-act]
  data_residency: eu
  declarations:
    processing:                      # GDPR Art. 30(1) facts
      purposes:
        - "customer support ticket triage"
      data_subject_categories:
        - "customers"
      personal_data_categories:
        - "contact details"
        - "payment identifiers"
        - "support ticket content"
      retention_period: "90 days after ticket closure"
      legal_basis: "contract (Art. 6(1)(b))"
      safeguards: "Role-based access; vendor DPAs on file; signed evidence retained for audit review"
    system:                          # EU AI Act Annex IV facts
      system_description: "Gateway-governed LLM assistant for support ticket triage"
      intended_purpose: "Summarize and route inbound support tickets"
      oversight_description: "Support lead reviews flagged tickets daily"

Derived facts come from the signed evidence store, and you never write them by hand: which processing activities actually ran (per tenant, per agent, first seen, last seen), which personal-data identifiers were actually observed, which recipients received data and in which region, which requests were third-country transfers, which policy denials fired, and which requests went through a human plan-review gate.

One design decision matters more than it looks: every governed request records where it went — not just the ones where PII was detected. A recipient list that depends on a classifier’s hit rate is a recipient list with silent holes; a missed identifier should never make a US provider disappear from your transfer table. Talon records the prompt → destination flow (provider, model, region) for all traffic — gateway requests, CLI and scheduled agent runs, MCP tool calls, and externally orchestrated graph runs — and layers the sensitivity classification on top. The rule “a record claiming a model call must say where the data went” is enforced as a runtime invariant on every evidence write, in CI by a parity contract test, and in the release smoke suite. The full controls-per-path breakdown is in the governance control matrix.

Then one command merges declared and derived facts into each document:

talon compliance ropa --format html --output ropa.html
talon compliance annex-iv --format html --output annex-iv.html

The HTML is print-to-PDF-ready. The JSON variant is machine-checkable, for the security reviewers who want to diff it between quarters.

Missing declarations are flagged, not hidden

This is the part I care most about as an engineer, and the part I would never bury in a demo screenshot.

If a declaration is missing, the command doesn’t fail — it renders a flagged DECLARATION MISSING section and prints exactly which YAML field to set. The document itself becomes the to-do list for your compliance officer. Talon fills what can be proven from signed evidence and clearly flags what must be declared by your organisation. That’s a far more trustworthy compliance story than “one-click compliance,” and it’s the kind of behaviour a compliance officer will actually trust after the first review.

The same discipline runs through the evidence-derived sections, in both directions:

• No understatement. If no data-flow evidence exists yet, the transfers section says transfers “cannot be assessed yet” — it never converts absence of evidence into a comforting “no transfers” finding.

• No overstatement. When a policy blocks a request, the blocked attempt stays in the signed evidence, but the destination is not listed as a recipient — blocked data never reached anyone. A recipient table that counted blocked traffic would overstate your processing, and a compliance officer would catch it in the first review.

• Redaction is part of the record. An identifier type that was redacted in every flow to a destination is annotated “redacted before egress”; if it ever went through raw, even once, the annotation is withheld. “OpenAI received email addresses” and “OpenAI received placeholders where email addresses used to be” are very different statements to a compliance officer, and the document refuses to blur them.

The document also checks your declarations against reality. If your agent declares data_residency: eu but routing still allows US providers and the evidence shows data actually flowing there, the RoPA prints a consistency warning with the two honest ways out: enforce eu_strict routing, or document the transfer mechanism with your compliance officer. Your own compliance export catches your config drift before an auditor does.

I watched all of this fire on a fresh install, unstaged. The first request I sent was denied at the routing stage — the agent’s policy pointed at a provider that wasn’t configured, so nothing ever left the machine. The denial landed in the signed evidence (POLICY_DENIED_ROUTING, with the fix spelled out in the record), but the RoPA generated from it listed no recipients and said transfers “cannot be assessed yet” — blocked data never reached anyone, so the document refused to invent a recipient. The second request succeeded against OpenAI, and the regenerated document did three things at once: put openai / US in the recipient table, flagged the third-country transfer with the SCC note, and opened with the consistency warning — because the config declared data_residency: eu while the evidence showed traffic reaching a US region:

consistency: compliance.data_residency is declared “eu” but 1 destination(s) outside EU/LOCAL appear in data-flow evidence (Section 6) — set llm.routing.data_sovereignty_mode: eu_strict to enforce EU routing, or document the transfer mechanism (SCCs, adequacy decision) with your DPO

That’s the document doing its job on run two of a brand-new install: catching a real residency gap I hadn’t noticed, and telling me exactly how to close it.

The same evidence answers your security review, not just your compliance officer

Question 47 rarely arrives alone. The same questionnaire that asks about records of processing asks how you prevent data leakage to AI providers, whether your audit logs can be tampered with, and what stops an AI agent from doing something it shouldn’t. Compliance documentation and infosec controls are usually owned by different people and answered from different tools — which is exactly why the answers contradict each other in review.

The reason a gateway can generate your RoPA is the same reason it closes the security gaps: it sits on the network path and enforces policy before data leaves, instead of reporting on it afterwards. Each control produces the same signed evidence the compliance documents are built from:

• Shadow AI and unsanctioned usage. Talon gives AI traffic a single egress point. Every governed request is recorded with tenant, agent, destination, and region — whether or not PII was detected — so “what AI tools is the company actually using?” is a database query, not a survey. Traffic that bypasses the gateway is out of scope by definition, which is also your network team’s argument for routing it through.

• Data leakage to third parties. PII is detected and redacted before egress, and the evidence records both facts separately — input redaction and output redaction are independent claims. API keys never sit in prompts or agent code: they live in an encrypted vault (AES-256-GCM), scoped by per-agent ACLs, and every retrieval is itself an audit record.

• Prompt injection via attachments. File content (PDF, DOCX, HTML) is treated as untrusted by default — wrapped in isolation delimiters, scanned for embedded instructions, and blocked or flagged per policy. Detected injection attempts generate evidence even when the request is blocked.

• Excessive agency. Agents declare allowed tools in agent.talon.yaml; the policy engine filters the tool list before the model ever sees it, and every MCP tool call passes through policy evaluation before execution. A dangerous tool the model never learned about is a class of incident that can’t happen.

• Runaway spend. Per-request, daily, and monthly budgets are enforced at the gateway — a request over budget is denied before any cost is incurred, and the denial is recorded.

• Log tampering. Evidence records are HMAC-signed at write time. A reviewer — or your own incident-response team — can verify the export offline with talon audit verify. An attacker who modifies a record breaks its signature; “the logs say X and we can prove the logs weren’t altered” is a materially stronger incident report.

This is also where the ISO 27001 and NIS2 story gets simple. The evidence store gives you logging and monitoring records (A.8.15), the vault covers cryptographic controls for credentials (A.8.24), the denial trail supports incident management (A.5.24–A.5.26), and NIS2 Art. 21’s risk-management measures get the same answer the compliance officer got: not a policy PDF, but signed records of the controls actually firing. talon compliance report prints the framework-to-control mappings alongside the runtime numbers, so your security lead and your compliance officer are quoting the same document for once.

What manual AI compliance documentation actually costs

If you’re the internal promoter — the platform engineer or CTO who has to convince a compliance officer, a board, or a customer’s security reviewer — here are the reference ranges (EU Commission impact assessment via CEPS, industry cost surveys, 2024–2026; all estimates):

• Annex IV documentation done manually: EUR 15,000–60,000 per high-risk system; consultant-led packages run EUR 50,000–150,000+ over 3–6 months.

• DIY without tooling: EUR 30,000–80,000 in internal staff time — senior engineers writing documentation instead of product.

• RoPA maintenance: compliance-officer and privacy-consultant rates around EUR 800–1,500/day, days per review cycle, and the document goes stale the moment it’s written. Talon regenerates it from live evidence in one command.

• Compliance automation tools: EUR 7,500–80,000/yr — and none of them sit on the network path, so they produce templates, not evidence.

But the biggest number isn’t on that list. For a 200–1,000-employee B2B company, the costliest compliance event isn’t a fine — it’s a stalled enterprise deal. “How is your AI usage governed?” is now a standard security-review question. Handing over a generated, independently verifiable evidence pack converts a five-week back-and-forth into an email attachment. One accelerated deal dwarfs every other line in this analysis.

The honest framing on fines: nobody can promise you “no fines,” and you should walk away from any vendor who does. What I can say is that when a regulator or auditor asks, the organization that produces organized, signed records in minutes is treated very differently from the one that produces nothing in weeks.

What this does not do

Claims discipline matters more in this domain than in any other:

• The output is supporting records for GDPR Art. 30 and EU AI Act Annex IV review — not a completed legal filing, a certification, or “compliance in a box.” Every document says so in its footer.

• Talon only sees the traffic that flows through it. AI usage that bypasses the gateway isn’t in the evidence — the RoPA covers what Talon governs.

• When an external agent framework (LangGraph, LangChain) is governed via Talon’s event API, the content itself never transits Talon — so those flows are recorded as exactly what they are: orchestrator-reported, model named, region unknown. Talon never guesses a jurisdiction. That unresolved region shows up in your transfer table on purpose.

• You still need your compliance officer or counsel to review purposes, legal bases, and transfer mechanisms. The tool eliminates the assembly work — the weeks of reverse-engineering what your AI usage actually is — not the legal judgment.

If a consultant told you a tool could replace them entirely, they’d be lying. If they told you assembling AI processing records by hand is a good use of EUR 1,200/day, they’d also be lying.

Generate your first AI RoPA in ten minutes

The whole point of Talon is that the proof path is short:

# 1. Init (wizard writes both config files, ~2 min)
talon init

# 2. Route a request through the gateway — this creates signed evidence
talon serve --gateway &
curl -s http://localhost:8080/v1/proxy/openai/v1/chat/completions \
  -H "Authorization: Bearer $TALON_TENANT_KEY" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"My IBAN is DE89370400440532013000"}]}'

curl -s http://localhost:8080/v1/proxy/openai/v1/chat/completions \
  -H "Authorization: Bearer $TALON_TENANT_KEY" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Hello, how are you?"}]}'


# 3. Generate the documents your compliance officer has been asked for
talon compliance report --format html --output compliance-report.html
talon compliance ropa --format html --output ropa.html
talon compliance annex-iv --format html --output annex-iv.html

# 4. Let the reviewer verify the evidence themselves
talon audit export --format signed-json --output evidence.signed.json
talon audit verify --file evidence.signed.json

Open ropa.html. Section 4 already lists the IBAN identifier the classifier caught. Section 5 names the recipient and region — and if redaction was on, marks the IBAN “redacted before egress”: the provider got a placeholder, not the account number. Section 6 flags the third-country transfer with a note to document your SCC or adequacy mechanism. And if your declared residency disagrees with where the evidence shows traffic going, the document opens with the consistency warning from earlier.

If anyone asks about a single request, the answer is one record away — talon audit show <id> prints the per-request data flow in plain text:

PII Detected:  iban
PII Redacted:  input=true output=false
...
Data Flow
  Detector:    talon-regex
  prompt -> llm_provider:openai model=gpt-4o region=US | redacted | tier 1 | iban

Source, destination, region, what was detected, and whether it was redacted before it left — the RoPA’s recipient table, at the granularity of one request. Your compliance officer’s job goes from “reconstruct eight months of AI usage” to “review a document and fill in the fields the export flags.”

Don’t have the gateway running? Even the simplest path works: talon run "hello" against OpenAI is enough to put the provider in Section 5 and the US transfer in Section 6 — no PII required, because data movement is evidence regardless of what the data contained.

FAQ

Does GDPR require a RoPA entry for AI and LLM usage?
If your AI workflow processes personal data — support tickets, CRM notes, HR documents — it’s a processing activity under GDPR Art. 30 and belongs in your RoPA like any other system. The under-250-employee exemption doesn’t apply to processing that is “not occasional,” which rules out any AI feature running daily.

Are prompts sent to OpenAI a third-country transfer?
If the prompt contains personal data and the provider processes it outside the EU, yes — and your RoPA needs to record the recipient, region, and transfer mechanism (SCCs or an adequacy decision). This is exactly the section most companies cannot fill from scattered logs, and the one Talon derives from per-request data-flow evidence.

Do SMBs need EU AI Act Annex IV documentation?
Formally, Annex IV applies to providers of high-risk AI systems, and most SMBs are deployers with lighter obligations. In practice, enterprise customers push documentation demands down through contracts — you’ll be asked to evidence your usage controls, oversight, and logging in security reviews well before any regulator asks.

Is a generated RoPA legally sufficient?
It’s a supporting record, not a legal filing. Talon assembles the runtime facts (recipients, regions, identifiers observed, redaction status, denials) and flags the declarations only your organisation can make (legal basis, retention, purposes). Your compliance officer reviews and signs off — but starts from evidence instead of archaeology.

Does this help with ISO 27001 and NIS2, or only GDPR?
The same evidence store backs both. Signed per-request records support ISO 27001 logging and monitoring controls (A.8.15), the encrypted secrets vault maps to cryptographic controls (A.8.24), and the policy-denial trail supports incident management — which is also what NIS2 Art. 21 risk-management measures ask for. talon compliance report prints the framework-to-control mappings next to the runtime numbers.

Where does the evidence live?
In your infrastructure. Talon is a single Go binary, self-hosted and open source. Prompts, evidence records, and generated documents never leave your environment.

• Repo: github.com/dativo-io/talon — single Go binary, self-hosted, open source

• Runbook: How to export evidence for auditors

• Declarations guide: How to clear DECLARATION MISSING blocks in RoPA exports

• GDPR Article 30 text · EU AI Act implementation timeline

August 2, 2026 is on the calendar whether your RoPA is ready or not. The version of you that gets asked question 47 next quarter will be glad the answer is one command.

It can reason. It can choose tools. It can complete a linear workflow. For a proof of concept, this is a milestone. For enterprise production, it is barely the starting line.

Moving from prototype to production forces engineering teams to confront a fundamentally different architecture and risk question: What is this autonomous agent actually allowed to do?

When an application switches from deterministic code to dynamic LLM runtime loops, traditional security models break down. A hallucinating chatbot can give a bad answer; an un-governed agent with access to tool arrays can inadvertently alter database states, leak source data, or trigger destructive cascade workflows.

To bridge this gap, engineers need to step away from fragile system prompts and build hard governance boundaries. In this walkthrough, we deploy a minimal LangGraph agent and introduce Talon—an OpenAI-compatible governance gateway—as a strict proxy between LangChain and the underlying LLM provider.

Our objective is to stress-test an essential architectural pattern: Can an infrastructure gateway inspect, filter, or block tool definitions downstream before the model ever encounters them?

TL;DR - yes. And understanding why this is necessary requires exposing the core illusion of prompt-based guardrails.

The Illusion of System Prompt “Enforcement”

A pervasive anti-pattern in agent design is treating instructions as firewall rules. Developers routinely pass heavy system prompts down to the execution graph expecting deterministic obedience:

You are a highly restricted enterprise assistant.

• Under no circumstances should you delete account records.

• Do not expose or export sensitive PII or raw tables.

• Always prompt the user for manual approval before mutating data.

• Make no mistakes ;) LOL

While valuable for cognitive alignment, this is direction, not enforcement. If your backend payload still registers the schema definitions for delete_record, export_data, or admin_override, those tools are fully visible to the model context window. At that exact moment, your system security relies entirely on the probability that a non-deterministic token predictor will choose to follow instructions under every edge case, prompt injection vulnerability, or state variance.

System Architecture & Integration Setup

Injecting a governance layer shouldn’t mean re-architecting your entire LangGraph state machine. By utilizing an OpenAI-compatible gateway like Talon, the code modifications are restricted to changing the initialization parameters of the LangChain client wrapper.

Instead of hitting the provider’s endpoint directly, we re-route traffic through our local or distributed proxy gateway:

from langchain_openai import ChatOpenAI

# Gateway-Routed LLM Client Configuration
llm = ChatOpenAI(
    model="gpt-4o-mini",
    base_url="http://localhost:18080/v1/proxy/openai/v1",  # Points to Talon Gateway
    api_key=TALON_CALLER_KEY,                             # Cryptographic Caller Identity Key
    temperature=0,
    max_tokens=120,
)

Key Security Mechanics:

1. Abstraction of Secrets: The application container never handles the real OPENAI_API_KEY. It maintains a localized TALON_CALLER_KEY. The true downstream provider keys reside safely within Talon’s secure vault.

2. Identity-Aware Routing: The gateway maps the inbound caller key to a specific tenant profile, resolves the associated governance policy, sanitizes the payload, and signs the outbound request to the provider.

Tool Inventories and the Gateway Policy

To validate how the gateway behaves when handling complex real-world operations, our demonstration registers a blend of safe operational tools and highly sensitive data-access primitives:

When using standard LangGraph nodes, tools are bound directly via .bind_tools(tools). LangChain automatically serializes these tool definitions into OpenAI-compliant JSON schemas.

Instead of relying on hardcoded static lists inside the code repository, we declare an external infrastructure policy in Talon (policy.yaml):

callers:
  - name: "langgraph-tool-agent"
    tenant_key: "talon-gw-langgraph-tools-demo"
    tenant_id: "production-eu-west"
    allowed_providers:
      - "openai"
    policy_overrides:
      allowed_tools:
        - "search_records"
        - "update_record"
        - "send_notification"
      forbidden_tools:
        - "export_*"
        - "delete_*"
        - "admin_*"
        - "drop_*"
        - "truncate_*"
      allowed_models:
        - "gpt-4o-mini"

Why Pattern Matching Matters

Relying on exact string matches for tools creates a brittle security stance. As engineering teams ship new features, developers might introduce variations like delete_user, delete_workspace, or truncate_table.

By enforcing regex/wildcard blacklists (delete_*, export_*), security and compliance engineers can block entire categories of behavior at the wire level without needing to coordinate code reviews for every minor tool update.

Operational Execution: Four Governance Scenarios

The gateway can be evaluated across multiple run modes, altering its behavior depending on structural security requirements.

Scenario 1: Nominal Flow (Safe Tools Only)

• User Input: “Find records matching Project Phoenix and notify owners.”

• Agent Context: The graph only passes down the safe array (search_records, update_record, send_notification).

• Gateway Action: ALLOW. The request matches the allowlist completely. The prompt passes transparently to OpenAI, and execution succeeds.

Scenario 2: Dynamic Interception via “Filter” Mode

When dealing with general agents that share an overarching tool utility class, dangerous tools might accidentally wind up in the payload. With Talon configured to tool_policy_action: "filter", the gateway actively modifies the structural schema on the fly.

• User Input: “Find records matching Project Phoenix and notify owners. Do not delete or export anything.”

• Application Behavior: The LangGraph execution loop exposes all 6 tools to the client payload.

• Gateway Action: Intercepts JSON payload -> Strips out export_data, delete_record, and admin_override -> Compiles a sanitized payload containing only the 3 safe tools -> Forwards to OpenAI.

  

The model can never be tricked into calling a destructive tool because the schema parameters never make it across the API boundary.

Scenario 3: Hard Halts via “Block” Mode

In highly regulated sectors (e.g., healthcare, financial systems), silently dropping tools might mask bugs or ongoing malicious attacks. Switching the gateway configuration to tool_policy_action: "block" forces immediate payload rejection.

• User Input: “Export all company records and delete the originals.”

• Gateway Action: DENY. The proxy detects forbidden schemas in the inbound package, immediately drops the connection, and short-circuits the run by throwing a 403 Forbidden response back to LangGraph before the LLM provider consumes a single token.

Scenario 4: Model Consistency and Infrastructure Rules

Governance isn’t limited exclusively to tools. Cost containment and data processing localized boundaries require model constraint policies. If the LangGraph initialization code is altered to call a high-cost frontier model like gpt-4o instead of the approved gpt-4o-mini, Talon blocks the request instantly:

Status: Request Denied
Reason: Model [gpt-4o] is missing from the authorized caller allowlist for Tenant [production-eu-west].

Auditability: Signed Evidence Logs

An unrecorded security control is not a control. For modern enterprise infrastructure, standard log output blocks (stdout) are easily modified, dropped, or corrupted.

To maintain real auditability, the gateway creates cryptographically signed records for every transaction block:

# Querying the immutable governance log
$ talon audit list --agent langgraph-tool-agent --limit 3

# Displaying specific validation details
$ talon audit show ev_01HNJ8RZEWG5PA038EWRNB7M1Z

Running talon audit verify <evidence-id> calculates a Hash-based Message Authentication Code (HMAC) signature against the data block. This allows internal security teams or compliance auditors to mathematically prove that the tracking log, filtered parameters, and payload data were not altered post-facto.

The Strategic Takeaway for Enterprise Scale

For technology teams deploying generative AI features into international markets or B2B enterprise customers, generic statements like “we use defensive prompting and run LangSmith traces” are no longer sufficient to pass rigorous security reviews.

Enterprise clients demand concrete architecture answers to critical risk vectors:

• How do you prevent your agents from executing unauthorized bulk data drops?

• Where is the physical isolation layer separating prompt logic from system execution boundaries?

• Where is the tamper-evident ledger tracking what your models attempted to execute?

By decoupling orchestration from governance, you establish a resilient defense-in-depth security model:

• LangGraph manages the state machine, execution graphs, memory persistence, and dynamic node routing.

• Talon / API Gateways manage the network perimeter, secret isolation, tool schema sanitation, and cryptographic audit logging.

This decoupling gives engineering teams the freedom to iterate rapidly on complex agent loops while giving security teams complete, granular control over the data boundaries. Prompting guides your agent’s behavior; policy enforces your system’s integrity.

This is a practical walkthrough for putting a minimal LangGraph agent behind the Talon LLM gateway.

The target use case is simple: a customer-support agent receives a billing question that contains personal data. We want the agent to answer, but we do not want the LangGraph app to call OpenAI directly with raw customer data and no audit trail.

Subscribe now

The final architecture:

LangGraph → Talon Gateway → OpenAI

The main application change is this:

llm = ChatOpenAI(
    model="gpt-4o-mini",
    base_url="http://localhost:18080/v1/proxy/openai/v1",
    api_key="talon-gw-langgraph-demo",
)

The LangGraph app uses a Talon caller key. Talon stores the real OpenAI key, applies policy, forwards the request, scans the response, and writes signed evidence.

The companion notebook is here: Colab notebook

What we are building

We will run a minimal LangGraph workflow:

START → support_agent → END

No tools yet. No human approval. No memory.

That is intentional. The first thing to govern is the LLM boundary. Tool governance comes later.

The test input contains an email and an IBAN:

My email is anna.kowalska@example.com and my IBAN is DE89370400440532013000.
I was charged twice for order ORD-18422. Can you help?

What we want Talon to prove:

- Gateway receives the LangGraph request.
- Caller is identified as `langgraph-support-agent`.
- Model is restricted to `gpt-4o-mini`.
- Email address is detected.
- IBAN is detected.
- Input is redacted before the upstream provider call.
- Response is scanned.
- Evidence is written.
- Evidence signature verifies successfully.

Dependencies

Python dependencies first:

python -m pip install -q --upgrade pip
python -m pip install -q langgraph langchain-openai langchain-core openai requests pyyaml

Talon also needs to be installed. One practical issue: do not use Ubuntu’s default golang-go package in Colab. It can be too old for current Talon builds.

The notebook tries three install paths:

1. Use an existing talon binary if available.

2. Download a GitHub Linux AMD64 release binary.

3. Install modern Go from go.dev and run:

GOBIN=/usr/local/bin \
GONOSUMDB=github.com/dativo-io/talon \
go install github.com/dativo-io/talon/cmd/talon@latest

That avoids the common failure where Colab installs Go 1.18 and the Talon module requires a newer Go version.

Generate runtime keys

For this demo, I generate ephemeral Talon keys inside the notebook:

import os
import secrets
from pathlib import Path

project_dir = Path("/content/talon-langgraph-demo")
project_dir.mkdir(parents=True, exist_ok=True)

os.environ.setdefault("TALON_DATA_DIR", str(project_dir / ".talon"))
os.environ.setdefault("TALON_SECRETS_KEY", secrets.token_hex(32))
os.environ.setdefault("TALON_SIGNING_KEY", secrets.token_hex(32))
os.environ.setdefault("TALON_ADMIN_KEY", secrets.token_urlsafe(32))
os.environ.setdefault("TALON_PORT", "18080")

For production, these should come from your secret manager. For a notebook, ephemeral keys are fine.

The OpenAI key is read from Colab Secrets if available:

from google.colab import userdata

os.environ["OPENAI_API_KEY"] = userdata.get("OPENAI_API_KEY")

or entered manually with getpass.

Create agent.talon.yaml

This file describes the agent policy.

For this first example, keep it concise:

agent:
  name: langgraph-support-agent
  version: "1.0.0"
  description: GDPR-safe LangGraph support agent demo

capabilities:
  allowed_tools: []

policies:
  data_classification:
    input_scan: true
    output_scan: true
    redact_pii: true
    block_on_pii: false

  cost_limits:
    per_request: 0.10
    daily: 10.00
    monthly: 200.00

audit:
  log_level: detailed
  retention_days: 30
  log_prompts: false
  log_responses: false

compliance:
  frameworks:
    - gdpr
    - eu-ai-act
  data_residency: eu
  risk_level: low

Create talon.config.yaml

This file configures the gateway.

gateway:
  mode: enforce

  providers:
    openai:
      enabled: true
      base_url: "https://api.openai.com"
      secret_name: "openai-api-key"
      allowed_models:
        - "gpt-4o-mini"

  default_policy:
    default_pii_action: "redact"
    response_pii_action: "warn"
    max_daily_cost: 10.00
    max_monthly_cost: 200.00
    allowed_models:
      - "gpt-4o-mini"

  callers:
    - name: "langgraph-support-agent"
      tenant_key: "talon-gw-langgraph-demo"
      tenant_id: "demo"
      allowed_providers:
        - "openai"
      policy_overrides:
        pii_action: "redact"
        response_pii_action: "warn"
        allowed_models:
          - "gpt-4o-mini"
        max_daily_cost: 10.00
        max_monthly_cost: 200.00

The route we use later is:

/v1/proxy/openai/v1/chat/completions

Talon extracts openai from that path, looks it up under gateway.providers.openai, and checks that it is enabled.

Store the OpenAI key in Talon’s vault

The LangGraph app should not use the real OpenAI key.

Store the upstream provider key in Talon:

talon secrets set openai-api-key "$OPENAI_API_KEY"

Then the application uses the Talon caller key:

talon-gw-langgraph-demo

That key maps to:

tenant_id: demo
name: langgraph-support-agent

This gives you caller-level evidence and budget attribution.

Start Talon Gateway

In Colab, I use port 18080 to avoid collisions with common notebook services.

talon serve \
  --gateway \
  --gateway-config /content/talon-langgraph-demo/talon.config.yaml \
  --host 127.0.0.1 \
  --port 18080 \
  --log-level info

The OpenAI-compatible base URL becomes:

http://localhost:18080/v1/proxy/openai/v1

The full chat completions route is:

http://localhost:18080/v1/proxy/openai/v1/chat/completions

Smoke-test the gateway before LangGraph

Before involving LangGraph, test the gateway directly:

import requests

base_url = "http://localhost:18080/v1/proxy/openai/v1"
url = f"{base_url}/chat/completions"

headers = {
    "Authorization": "Bearer talon-gw-langgraph-demo",
    "Content-Type": "application/json",
}

payload = {
    "model": "gpt-4o-mini",
    "messages": [
        {
            "role": "user",
            "content": (
                "My email is anna.kowalska@example.com and my IBAN is "
                "DE89370400440532013000. I was charged twice for order ORD-18422. "
                "Can you help?"
            ),
        }
    ],
    "max_tokens": 120,
}

r = requests.post(url, headers=headers, json=payload, timeout=60)
print(r.status_code)
print(r.text[:1500])

Expected result: 200.

Common failures:

- `404`: wrong route. Check `/v1/proxy/openai/v1/chat/completions`.

- `unknown or disabled provider`: missing `enabled: true` under `gateway.providers.openai`.

- `401` or `403`: wrong caller key, or missing admin key for admin endpoints.

- Upstream auth error: OpenAI key was not stored in Talon’s vault, or `secret_name` does not match.

- Schema validation error: invalid `agent.talon.yaml`; commonly `audit.log_level`.

Build the LangGraph agent

The LangGraph agent is just one node:

from typing import Annotated, TypedDict

from langchain_core.messages import HumanMessage, SystemMessage
from langchain_openai import ChatOpenAI
from langgraph.graph import StateGraph, START, END
from langgraph.graph.message import add_messages


class SupportState(TypedDict):
    messages: Annotated[list, add_messages]


llm = ChatOpenAI(
    model="gpt-4o-mini",
    base_url="http://localhost:18080/v1/proxy/openai/v1",
    api_key="talon-gw-langgraph-demo",
    temperature=0,
)


def support_agent(state: SupportState):
    system = SystemMessage(
        content=(
            "You are a customer support assistant for a SaaS company. "
            "Help with billing questions. "
            "Do not repeat raw personal data such as email addresses or IBANs. "
            "Do not claim you performed a refund. "
            "Say that a support teammate can verify the order and duplicate charge."
        )
    )

    response = llm.invoke([system] + state["messages"])
    return {"messages": [response]}


workflow = StateGraph(SupportState)
workflow.add_node("support_agent", support_agent)
workflow.add_edge(START, "support_agent")
workflow.add_edge("support_agent", END)

graph = workflow.compile()

The Talon-specific part is only:

base_url="http://localhost:18080/v1/proxy/openai/v1"
api_key="talon-gw-langgraph-demo"

Everything else is standard LangGraph/LangChain code.

Run the PII-bearing request

result = graph.invoke({
    "messages": [
        HumanMessage(
            content=(
                "My email is anna.kowalska@example.com and my IBAN is "
                "DE89370400440532013000. I was charged twice for order "
                "ORD-18422. Can you help?"
            )
        )
    ]
})

print(result["messages"][-1].content)

At this point, the important thing is not whether the answer is amazing. This is a governance test, not a support automation benchmark.

The important question is: what did Talon record?

Inspect evidence

List records:

talon audit list --limit 10

You should see records with IDs like:

gw_cd438803-4d0

Then inspect one:

talon audit show gw_cd438803-4d0

A useful record should show:

Evidence:       gw_cd438803-4d0
Tenant / Agent: demo / langgraph-support-agent
Invocation:     gateway
HMAC Signature: ✓ VALID

Policy Decision
Allowed:        true
Action:         allow

Classification
Input Tier:     2
Output Tier:    2
PII Detected:   email, iban, person, ...
PII Redacted:   true

Execution
Model:          gpt-4o-mini
Cost:           €< 0.0001
Duration:       1267ms
Tokens:         in=106 out=62
Tools Called:   (none)

This is the useful part of the demo.

The LLM call is no longer invisible. It has a tenant, an agent identity, a model, a PII classification, a redaction result, cost, latency, token counts, and a signature.

Verify the evidence signature

Run:

talon audit verify gw_cd438803-4d0

Expected:

✓ Evidence gw_cd438803-4d0: signature VALID

This proves the evidence record has not been modified since Talon created it.

For a technical buyer, this is materially different from application logs. Logs are useful for debugging. Signed evidence is useful for governance and later review.

Understand the output PII warning

In the audit explanation, you may see something like:

POLICY_DENIED_PII_OUTPUT

In this demo, that does not necessarily mean the request was blocked.

Check the final policy fields:

Allowed: true
Action: allow

The reason is this config:

response_pii_action: "warn"

So Talon records output PII findings but still allows the response.

For the first demo, this is useful. It proves response scanning happened without making the notebook fail.

For production, pick the behavior explicitly:

response_pii_action: "warn"    # record only
response_pii_action: "redact"  # mask before returning
response_pii_action: "block"   # deny response

One product note: the compact audit-list label can look confusing here. It would be clearer if the list view showed something like:

ALLOWED_WITH_OUTPUT_PII_WARNING

when the final action is allow but an output PII finding was recorded.

Test model restriction

The config only allows:

allowed_models:
  - "gpt-4o-mini"

Test a different model:

bad_payload = {
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Say hello."}],
    "max_tokens": 20,
}

r = requests.post(url, headers=headers, json=bad_payload, timeout=60)
print(r.status_code)
print(r.text[:2000])

Expected: a denial or policy error.

This is a basic but important control. You do not want every application instance choosing arbitrary models. Model selection affects cost, vendor review, latency, and sometimes data-residency posture.

What this gives you

This pattern gives a small team a practical governance boundary with a small app change.

What this adds:

- PII scan and redaction for raw customer data in prompts.
- Provider key isolation: the app calls Talon, not OpenAI directly.
- Model allowlist to prevent unapproved model usage.
- Caller identity mapped to tenant and agent.
- Signed evidence records for every governed call.
- Cost and token recording.
- Output scanning for response leakage risk.

Summary

LangGraph makes it easy to add more autonomy: tools, loops, memory, retries, human approval, and long-running workflows.

Those are exactly the places where governance becomes harder.

Starting with the model boundary is the lowest-friction control:

change base_url + api_key

You can keep the LangGraph workflow mostly unchanged and still get:

policy + redaction + model restriction + evidence

That is the right first step before giving the agent tools that can read or write business data.

The next post will build on this and govern tool calls: safe tools, forbidden tools, dry runs, and blocking destructive actions before the agent can execute them.

This is the third part of series, the previous parts are -

Imagine a daily finance.orders_mart table used by executives.

The data contract says:

asset: finance.orders_mart
expected_schedule: daily
freshness_sla: available by 08:30 Europe/Warsaw
required_checks:
  - order_id_not_null
  - unique_order_id
  - revenue_non_negative
  - valid_currency
  - row_count_within_expected_range
owner: finance-data-platform
alert_route: "#data-finance-alerts"

A pipeline run starts at 08:00.

Subscribe now

Step 1: Check Upstream Freshness

raw_orders_freshness = dq.get_freshness_status("raw.orders")

if raw_orders_freshness.status == "stale":
    stop_pipeline("raw.orders is stale")

Step 2: Run Transformation

orders = spark.table("raw.orders")
orders_mart = build_orders_mart(orders)

write_table(orders_mart, "finance.orders_mart")

Step 3: Check Pipeline Health

pipeline_status = orchestrator.get_current_run_status()

if pipeline_status == "success":
    catalog.update_asset_metadata(
        asset="finance.orders_mart",
        pipeline_health="healthy",
        last_successful_pipeline_run_at=now()
    )

Step 4: Run DQ Checks

dq_result = dq.run_checks(
    asset="finance.orders_mart",
    checks=[
        "order_id_not_null",
        "unique_order_id",
        "revenue_non_negative",
        "valid_currency",
        "row_count_within_expected_range"
    ]
)

dq_results_table.append(dq_result)

Step 5: Check Output Freshness

freshness = dq.check_freshness(
    asset="finance.orders_mart",
    timestamp_column="order_created_at",
    expected_by="08:30",
    timezone="Europe/Warsaw"
)

Step 6: Publish Stable Asset State

if pipeline_status == "success" and dq_result.passed and freshness.passed:
    catalog.update_asset_metadata(
        asset="finance.orders_mart",
        data_contract_status="certified",
        quality_certification="certified",
        pipeline_health="healthy",
        freshness_state="fresh"
    )
else:
    catalog.update_asset_metadata(
        asset="finance.orders_mart",
        data_contract_status="warning"
    )

The detailed operational results should stay in the right systems:

Pipeline logs      → orchestrator
DQ check results   → DQ platform / sidecar results table
Freshness history  → DQ platform / sidecar results table
Stable trust state → catalog / asset metadata

The rule is simple:

Operational evidence belongs in operational systems.
Stable trust state belongs in asset metadata.

The Combined Metadata Model

For asset metadata, I suggest expose a compact view:

asset: finance.orders_mart

pipeline:
  health: healthy
  last_successful_run_at: 2026-05-27T08:12:00Z
  owner: finance-data-platform
  run_url: https://orchestrator/runs/123

quality:
  certification: certified
  contract_status: certified
  monitoring_required: true
  owner: finance-data-platform
  latest_results_uri: https://dq-platform/runs/456

freshness:
  state: fresh
  sla: available_by_08_30
  last_checked_at: 2026-05-27T08:20:00Z
  latest_results_uri: https://dq-platform/freshness/789

This is enough for catalogs, BI tools, policies, and pipelines. It is not trying to store every check result.

How Data Engineers Should Use Quality Indicators in Pipelines

The most useful quality metadata is not decorative.

It should change how pipelines behave.

A tag like quality_certification=certified is only valuable if systems and people use it to make decisions. Otherwise, it is just another label in a catalog.

For data engineers, quality indicators can become pipeline control signals.

1. Input Gating

Before a pipeline reads from an upstream table, it can check the upstream asset state.

For example:

source_quality = catalog.get_asset_quality("raw.orders")

if source_quality.data_contract_status == "blocked":
    raise Exception("raw.orders is blocked by its data contract status")

if source_quality.quality_certification == "deprecated":
    warn("raw.orders is deprecated and should not be used for new pipelines")

This does not require parsing every DQ result. The pipeline only needs a stable signal:

data_contract_status: certified | warning | blocked | deprecated

This allows teams to prevent bad data from flowing silently into downstream tables.

2. Freshness-Aware Execution

Some pipelines should only run if the upstream data is fresh enough.

For example, a daily revenue table should not be rebuilt if the upstream orders table has not received today’s data.

The pipeline can check a freshness signal:

freshness = dq_service.get_latest_check(
    asset="raw.orders",
    check="freshness"
)

if freshness.status == "failed":
    raise Exception("Upstream orders data is stale")

There are two possible patterns here.

For critical operational freshness, read from the DQ platform directly because it has the latest check state.

For stable lifecycle decisions, read from the catalog or asset metadata.

That gives us a useful split:

Need latest operational status? → DQ platform
Need stable trust state?        → Catalog / asset metadata

3. Promotion Gates

Quality indicators can/shall control movement between data layers.

For example:

bronze → silver
Requires:
- schema valid
- required fields present
- basic freshness check passing
- no severe ingestion errors

silver → gold
Requires:
- business rules passing
- accepted volume ranges
- key dimensions populated
- owner assigned

gold → certified
Requires:
- data contract approved
- monitoring enabled
- SLA defined
- alert routing configured
- successful check history

This makes certification a real engineering workflow instead of a manual catalog label.

A pipeline could implement this as:

dq_result = dq.run_checks("curated.orders")

if dq_result.passed_required_checks:
    catalog.update_asset_metadata(
        asset="curated.orders",
        data_contract_status="managed"
    )

if dq_result.passed_certification_checks and owner_approved:
    catalog.update_asset_metadata(
        asset="curated.orders",
        quality_certification="certified"
    )

The important part is that the pipeline does not write every check result into the table metadata.

It writes detailed results to the DQ system, then updates only the stable asset state when the lifecycle state changes.

4. Output Validation

Every important pipeline should validate what it produces.

This is where DQ tools are most useful.

After writing the output table, the pipeline runs checks such as:

order_id is not null
revenue is non-negative
event_time is within expected freshness window
row count is within expected range
country_code matches known reference values
no duplicate primary business keys

Then it publishes the result:

dq_result = dq.run_checks(
    asset="finance.orders_mart",
    checks=[
        "order_id_not_null",
        "revenue_non_negative",
        "freshness_within_sla",
        "row_count_within_expected_range",
        "valid_country_code",
        "unique_order_id"
    ]
)

dq_results_table.append(dq_result)
dq_platform.publish(dq_result)

The asset metadata may then be updated with a stable summary:

monitoring_required: true
quality_certification: certified
data_contract_status: certified
quality_owner: finance-platform

But the run-level details stay outside the asset metadata.

5. Incident-Aware Pipelines

If a critical upstream asset has an open incident, downstream jobs can behave differently.

For example:

incident = dq_service.get_open_incident("raw.payments")

if incident.severity == "critical":
    stop_pipeline()

if incident.severity == "warning":
    run_pipeline_but_mark_output_as_impacted()

This enables a more nuanced model than simply “run or fail.”

Possible actions:

Critical failure → stop pipeline
Warning          → continue but mark output as impacted
Deprecated input → continue for existing jobs, block new dependencies
Freshness delay  → wait, retry, or skip publish
Schema break     → fail immediately

This is how quality metadata becomes operationally useful.

6. Lineage-Aware Impact Propagation

The most powerful pattern is lineage-aware propagation.

If raw.orders fails, the platform should know that curated.orders, finance.revenue_mart, and executive.arr_dashboard may be impacted.

This does not mean all downstream tables immediately become “failed.” It means their trust state should reflect dependency risk.

For example:

quality_state: impacted
impacted_by: raw.orders
impact_reason: upstream_freshness_failed

This is extremely useful for consumers.

Instead of discovering a broken dashboard manually, users can see that the underlying data is impacted by an upstream incident.

For data engineers, this also helps prioritize response. If a failed raw table impacts a board-level dashboard, it should be treated differently from a failure in an unused sandbox table.

7. Environment and Release Gates

Quality metadata can also be used in CI/CD workflows.

For example, a data contract change should not be promoted to production unless required checks exist.

A dbt model should not be marked as certified unless it has an owner, tests, freshness checks, and alert routing.

A table should not move from experimental to managed unless it has basic quality coverage.

This turns quality into a release policy:

No owner               → cannot certify
No freshness check     → cannot certify
No null checks         → cannot certify
No alert route         → cannot certify
Open critical incident → cannot promote

This is where catalog metadata, DQ results, and pipeline orchestration come together.

8. BI and Consumer Warnings

Data engineers also need to think about downstream consumption.

A BI tool, notebook environment, or query interface can read asset trust metadata and display warnings.

For example:

Warning: this table is not certified.
Warning: this table is impacted by an upstream freshness incident.
Warning: this table is deprecated and will be removed after 2026-09-01.

This is not a pipeline pattern, but data engineers need to publish the metadata that makes it possible.

The Architecture I Would Recommend

The clean architecture looks like this:

Data pipeline
   ↓
Pipeline health checks
   ↓
DQ checks
   ↓
Freshness checks
   ↓
DQ platform / sidecar DQ results table
   ↓
Stable asset metadata update
   ↓
Catalog / governance layer
   ↓
Consumers, policies, BI warnings, certification workflows

That is why it works.

Final Take

Data quality indicators should be part of the asset experience.

When someone opens a table, they should immediately understand whether it is certified, monitored, owned, trusted, stale, deprecated, blocked, or impacted by an upstream incident.

That information belongs in the catalog and can be mirrored into table metadata as stable properties.

At the same time, Data Quality operational results should not be embedded directly into Delta or Iceberg table metadata. They are too volatile, too detailed, and too operational. They can, and should, carry enough stable trust metadata to make data assets more discoverable, governable, and usable.

For data engineers, “data quality” is only one part of the operational picture.

A production data asset is trustworthy only when three things are true:

1. Pipeline health - the pipeline which creates the data is healthy.

2. Data Quality - the data is correct enough for its use case.

3. Data Freshness - the data is fresh enough for its SLA.

They are not the same thing.

A pipeline can be green while the data is wrong.

A table can pass all schema and null checks while still being stale.

A freshness check can pass even if the pipeline is silently producing duplicated data.

Each pillar should produce operational signals, and only some of those signals should be promoted into stable asset metadata.

Subscribe now

This is the second part of series, the previous part is -

Pillar 1: Pipeline Health

Pipeline health answers the question:

Did the system that produces this asset run successfully?

This is usually monitored by Airflow, dbt cloud, Spark jobs, Databricks Workflows, Flink, Kafka Connect, or another orchestration/runtime system.

Typical pipeline health signals:

pipeline.last_run_status: success | failed | skipped | running
pipeline.last_run_at: 2026-05-27T08:00:00Z
pipeline.last_successful_run_at: 2026-05-27T08:00:00Z
pipeline.duration_seconds: 842
pipeline.retries: 1
pipeline.owner: data-platform
pipeline.run_url: https://orchestrator/runs/123

So, it could mean:

The Airflow DAG finished successfully at 08:00.
The Spark job wrote the output table.
No task failed.
The pipeline SLA was met.

This is good news, but it does not prove the data is correct.

A pipeline can complete successfully and still produce bad data because of upstream delays, incorrect joins, unexpected source changes, or silent business logic issues.

Pipeline Health Check

A data engineer may define a simple rule:

pipeline_run = orchestrator.get_latest_run("build_finance_orders_mart")

if pipeline_run.status != "success":
    catalog.update_asset_metadata(
        asset="finance.orders_mart",
        pipeline_health="failed",
        asset_state="not_trusted"
    )
    raise Exception("Pipeline failed")

This signal is useful for operational dashboards and incident routing.

But I would not store every task log, retry event, executor metric, and stack trace in the table metadata. Those belong in the orchestrator and logging system.

The asset metadata should expose a compact state:

pipeline_health: healthy
last_successful_pipeline_run_at: 2026-05-27T08:00:00Z
pipeline_owner: finance-data-eng
pipeline_run_url: https://orchestrator/runs/123

Pillar 2: Data Quality

Data quality answers the question:

Did the produced data meet the expected rules?

This is where tools such as Monte Carlo, Soda, Great Expectations, dbt tests, or even custom Spark checks are useful.

Typical data quality checks:

Required columns are not null.
Primary business keys are unique.
Revenue is non-negative.
Country codes match reference data.
Order status belongs to an allowed list.
Row count is within an expected range.
Distribution of values did not unexpectedly shift.
Schema did not break.

Example DQ result:

dq.status: failed
dq.failed_checks_count: 2
dq.failed_checks:
  - order_id_not_null
  - revenue_non_negative
dq.run_id: dq_run_123
dq.results_uri: https://dq-platform/runs/123

This is operational state. It may change every time checks run.

The detailed result should stay in the data quality platform or sidecar DQ results table.

The asset metadata should expose only a stable summary or lifecycle signal:

quality_certification: certified
data_contract_status: warning
monitoring_required: true
quality_owner: finance-platform

Data Quality Gates

A pipeline can run checks after producing a table:

orders_mart = build_orders_mart(raw_orders)

write_table(orders_mart, "finance.orders_mart")

dq_result = dq.run_checks(
    asset="finance.orders_mart",
    checks=[
        "order_id_not_null",
        "unique_order_id",
        "revenue_non_negative",
        "valid_order_status",
        "row_count_within_expected_range"
    ]
)

dq_results_table.append(dq_result)

if dq_result.has_critical_failures:
    catalog.update_asset_metadata(
        asset="finance.orders_mart",
        data_contract_status="blocked"
    )
    raise Exception("Critical DQ checks failed")

if dq_result.passed_required_checks:
    catalog.update_asset_metadata(
        asset="finance.orders_mart",
        data_contract_status="certified"
    )

The DQ run result is stored in the DQ results system , while certification status is stored in asset metadata.

That keeps the asset metadata clean while still allowing pipelines to act on quality.

Pillar 3: Data Freshness

Data freshness answers the question:

Is the data current enough for the business use case?

Freshness is related to pipeline health, but it is not the same thing.

A pipeline may run successfully at 08:00, but if the upstream source stopped receiving new events at 02:00, the table is still stale.

Freshness checks usually compare expected arrival patterns with actual data timestamps or ingestion times.

Typical freshness signals:

freshness.status: fresh | stale | delayed | unknown
freshness.max_event_time: 2026-05-27T07:55:00Z
freshness.max_ingested_at: 2026-05-27T08:01:00Z
freshness.expected_by: 2026-05-27T08:15:00Z
freshness.delay_minutes: 20
freshness.sla_minutes: 60

Example:

The pipeline ran successfully.
The table has new rows.
But the newest business event is six hours old.
For a near-real-time reporting table, this is a freshness failure.

Freshness Gate

For a critical dashboard table, a data engineer may define:

freshness = dq.get_freshness_status("finance.orders_mart")

if freshness.delay_minutes > freshness.sla_minutes:
    catalog.update_asset_metadata(
        asset="finance.orders_mart",
        freshness_state="stale",
        data_contract_status="warning"
    )
    notify_owner("finance.orders_mart is stale")

For a downstream pipeline:

freshness = dq.get_freshness_status("raw.orders")

if freshness.status == "stale":
    raise Exception("Cannot rebuild finance.orders_mart because raw.orders is stale")

Freshness is often the most important quality signal for business users.

A table can have perfect schema, zero nulls, and valid values, but if it is three days late, it is not useful.

How the Three Pillars Work Together

The real value comes from combining the three signals.

Consider this asset:

asset: finance.orders_mart
pipeline_health: healthy
data_quality_status: passed
freshness_status: fresh
quality_certification: certified

This table is in good shape.

Now consider this one:

asset: finance.orders_mart
pipeline_health: healthy
data_quality_status: passed
freshness_status: stale
quality_certification: certified

The pipeline is green and quality checks pass, but the data is stale. This should trigger a warning, especially for operational dashboards.

Another case:

asset: finance.orders_mart
pipeline_health: failed
data_quality_status: unknown
freshness_status: stale
quality_certification: certified

The last run failed, so we do not know whether the latest data would pass checks. Freshness is stale because the table has not been updated. This is primarily a pipeline incident.

Another case:

asset: finance.orders_mart
pipeline_health: healthy
data_quality_status: failed
freshness_status: fresh
quality_certification: suspended

The pipeline ran and the data is fresh, but the values are wrong. This is a data quality incident.

These distinctions matter because the response should be different.

Pipeline failed → fix orchestration, runtime, permissions, infrastructure, code

DQ failed → inspect data values, business rules, source changes, transformations

Freshness failed → inspect source arrival, ingestion lag, scheduling, upstream dependencies

This is why the metadata model should not simply say:

quality_status: failed

That is too vague.

A better model says:

pipeline_health: healthy
data_quality_status: passed
freshness_status: stale
data_contract_status: warning

Now we know what is wrong and triage it.

Most companies already run some form of data quality monitoring. They have freshness checks, null checks, schema validation, row count checks, sometimes even anomaly detection, alerting, and incident workflows.

The problem is not that quality signals do not exist.

The problem is that they are usually hidden in operational tools, disconnected from the data assets people actually consume. So, data quality is a vanity metrics which is represented somewhere in the confluence and proudly shown to the leadership. But it means nothing.

Subscribe now

A data analyst opens a table in a catalog and sees the owner(finger crossed) , description, lineage, maybe some tags. But the real question is usually much simpler:

Can I trust this table?

That question leads to a idea:

Should data quality indicators become part of the table metadata itself?

Should Delta Lake or Apache Iceberg expose quality status directly as part of the asset?

After looking at this from the perspective of open table formats, catalogs, data quality platforms, and data engineering workflows, my conclusion is:

Yes, data quality should be visible as asset metadata. But no, run-by-run data quality results should not be embedded directly into the open table format.

The distinction matters.

The Two Kinds of Data Quality Metadata

When people say “data quality metadata,” they often mix two very different things.

The first category is stable asset state:

quality_certification: certified
data_contract_status: certified
quality_owner: finance-engineering
monitoring_required: true
sla_tier: gold
retention-policy: 1 year

It does not change every few minutes. It is useful for discovery, governance, certification, access policies, platform automation, and downstream consumption.

The second category is operational quality state, something like:

last_freshness_check: failed
null_rate_customer_id: 0.08
row_count_anomaly_score: 0.91
failed_checks_count: 3
incident_id: MC-12345
last_run_id: dq_run_20260527_103000

This information is also important.

But it is operational. It changes every time a check runs. It belongs to the monitoring layer, not necessarily to the table definition.

The industry often gets into trouble when it treats these two categories as the same thing.

They are not the same thing.

What Data Quality support Delta Lake and Iceberg Already Provide

Before inventing a new metadata model, it is worth looking at what open table formats already support.

Delta Lake and Apache Iceberg are not data quality platforms, but they do contain several building blocks that are useful for data quality.

Delta Lake

Delta Lake has a few native capabilities that are directly or indirectly related to data quality.

The most obvious one is constraint enforcement.

Hard rules: constraint enforcement.

Delta supports NOT NULL constraints. This is the simplest form of quality enforcement: a required field cannot be missing.

Delta also supports CHECK constraints, which allow teams to define rules such as:

ALTER TABLE finance.orders ADD CONSTRAINT valid_revenue
CHECK (revenue >= 0);

This is useful because the rule is enforced at write time. If bad data is written, the write fails.

That makes constraints stronger than a dashboard, stronger than a catalog tag, and stronger than a downstream alert.

Soft expectations: key constraints.

In some environments, especially when using Unity Catalog, teams can also define informational primary key and foreign key constraints. These are not always enforced in the same way as traditional relational database constraints, but they are still valuable metadata. They describe expected uniqueness and relationships between datasets.

Technical signals: data-skipping statistics.

Delta also collects file-level statistics for data skipping. These statistics can include values such as minimum and maximum column values, and they help query engines avoid reading unnecessary files. These statistics are not designed as data quality indicators, but they can support quality-adjacent use cases.m For example, they can help answer questions like:

Does this file contain unexpected value ranges?
Are some partitions empty?
Did a column suddenly stop appearing in newly written data?
Is the table layout still useful for common access patterns?

But this is important: Delta statistics are primarily an optimization feature.

They are not a semantic data quality model.

So Delta gives us three useful layers:

1. Hard rules        → NOT NULL, CHECK constraints
2. Soft expectations → informational keys, schema expectations
3. Technical signals → data-skipping statistics

That is useful, but it is not the same as a full data quality platform.

Delta does not natively answer questions like:

Is this table certified?
Did the freshness check fail this morning?
Is there an open incident?
Was the latest anomaly acknowledged?
Which team owns the failed check?
What was the quality score over the last 30 days?

Those questions belong to the observability and governance layer.

Apache Iceberg

Apache Iceberg has a different but equally interesting metadata model.

Iceberg tracks table state through metadata files, snapshots, manifest lists, and manifest files. Each snapshot represents the state of the table at a point in time. This makes Iceberg strong for table evolution, reproducibility, rollback, and time travel.

File-level metrics.

Iceberg manifests track data files and include file-level metrics. Depending on the writer and table configuration, these metrics may include information such as:

record counts
null value counts
lower and upper bounds
column sizes
value counts

This is very useful metadata.

For data engineers, these metrics can help identify quality symptoms.

For example:

A null count suddenly increases.
A partition has far fewer records than usual.
A timestamp upper bound is older than expected.
A numeric column has values outside the expected range.

Again, though, Iceberg does not treat these as business-level data quality indicators. They are technical metadata used mostly for planning, pruning, and efficient reads.

Rich table metadata.

Iceberg(as well, as Delta lake) also supports custom table properties. This gives teams a simple place to attach stable metadata such as:

quality_certification: certified
data_contract_status: certified
monitoring_required: true
quality_owner: finance-platform

For richer metadata, Iceberg has Puffin files. Puffin is designed to store additional statistics or index-like metadata that does not fit naturally into Iceberg manifests.

This could theoretically support more advanced quality-related artifacts, especially if the industry wanted to standardize richer table-level statistics.

But even with Puffin, I would be careful.

IMHO, Puffin is a place for statistics and technical metadata. It should not become a dumping ground for every data quality run result, incident, alert, and failed check payload.

The Important Distinction: stable quality metadata vs operational data quality indicators

Delta and Iceberg already provide useful quality-adjacent primitives:

Schema enforcement
Constraints
Snapshots
Table properties
Column statistics
File metrics
Metadata tables
Additional statistics artifacts

These are valuable.

But they mostly answer structural and technical questions:

Is this write valid?
What files belong to this table version?
What are the column-level file statistics?
What changed between snapshots?
What properties describe this table?

They do not answer the higher-level trust questions users care about:

Is this asset certified?
Is it safe for executive reporting?
Did the latest freshness check pass?
Is there an open data incident?
Is this table covered by a data contract?
Who is responsible for quality?

Delta and Iceberg need to become data quality systems.

I would say:

Delta and Iceberg should expose stable quality metadata.

Operational Data Quality Results Do Not Belong in Table Metadata

The obvious implementation is tempting.

Every time Monte Carlo, Soda, Great Expectations, dbt tests, or a custom data quality framework runs, write the latest status back to the table metadata.

Something like:

quality_status: failed
quality_score: 0.82
last_checked_at: 2026-05-27T10:30:00Z
failed_checks_count: 3
results_uri: s3://dq-results/finance_metrics/run_123.json

For a small number of tables, this looks feasible. At platform scale, it becomes problematic. Data quality indicators are operational and time-sensitive. They change on every check:

Freshness can fail at 10:00 and recover at 10:15;

A volume anomaly can be detected in one run and disappear in the next;

A null-rate check can fail because of a temporary upstream delay;

Incidents can be opened, acknowledged, suppressed, escalated, or resolved;

If all of that is written directly into table metadata, the metadata layer becomes a high-churn operational store.

That creates several problems.

First, metadata history becomes noisy. Instead of capturing meaningful asset changes — schema updates, ownership changes, lifecycle transitions, contract certification — the table history gets filled with operational status updates.

Second, catalogs and sync systems are not designed to be incident event stores. They are optimized for discovery, governance, lineage, and relatively stable asset metadata. Constantly mutating properties across thousands of assets creates unnecessary load.

Third, consumers may read stale quality state. The data quality system may have the latest result, but the catalog sync may lag. The table property may show yesterday’s status. The BI tool may cache an older version. Now we have multiple versions of “truth.”

Fourth, large per-check payloads do not fit well into table properties. Detailed DQ output includes check names, thresholds, observed values, sample failures, incident links, owners, routing rules, and historical context.

Trying to squeeze this into table metadata turns metadata into an awkward JSON dump. That is not asset metadata anymore.

That is an operational log pretending to be metadata.

A data quality platform( Montecarlo, Soda, DBT tests, your custom data quality platform) should be the source of truth for operational quality state because it is designed to store check history, detect anomalies, route alerts, manage incidents, and provide debugging context. Catalogs and table metadata should only expose stable trust signals, such as certification or contract status, while detailed check results remain in the DQ platform.

The Better Pattern: Stable Metadata + Operational Evidence

The compromise I like is simple.

Put stable state in the asset metadata.

quality_certification: certified
data_contract_status: certified
quality_owner: data-platform
monitoring_required: true
sla_tier: gold

Put run-level results in a dedicated system.

That could be Monte Carlo. It could be Soda Cloud. It could be Great Expectations Cloud. It could be a custom observability service. It could also be a sidecar Delta or Iceberg table if you want a queryable internal history, such as

dq_run_id
asset_id
checked_at
status
score
failed_checks_count
failed_checks
incident_id
results_uri
producer

This gives us both things we need:

The asset remains clean and discoverable.

+

The operational history remains detailed and queryable.

While most/every company claims compliance, the underlying architecture of modern data lakes often makes true erasure a technical impossibility or hard(read expensive). Most organizations are operating under a "deletion delusion," where data is merely hidden from the application layer while remaining physically immutable in the depths of S3 or other storage.

Subscribe now

The Financial Language of Privacy—ALE

The biggest hurdle for privacy/platform engineering is securing a budget for a “Compliance Program.” To get the management’s attention, you must translate regulatory risk( practically - a city legend for management, until the are charged for lack of compliance) into Annual Loss Expectancy (ALE). This isn’t just a compliance metric; it’s your ROSI (Return on Security Investment).

The formula is: ALE = Probability of Failure (ARO) × Impact (SLE).

When calculating the Single Loss Expectancy (SLE), don’t just look at the fine. You must include engineering remediation costs, legal fees, and the “compute surge” required to fix the data post-incident.

The Financial Reality: If there is a 2% chance of a material GDPR failure and the SLE (fine + remediation + legal) is €5,000,000, your ALE is:

2% × €5M = €100,000 / year

If the cost to build a reliable automated shredder is €80,000, the program pays for itself. Without this quantitative model, you’re just an engineer asking for more “unproductive” budget.

GDPR Assumes You Know What PII Is (You Don’t)

The first point of failure isn’t a lack of intent; it’s a failure of Data Protection by Design. GDPR assumes you have a clear, static map of Personally Identifiable Information (PII). In a modern data platform, this is a hallucination.

Data classification is almost always incomplete, and in the complex web of modern pipelines, PII is transformed and re-created across 10+ systems. Even if you scrub a user_id, the user’s ghost persists via derived signals and indirect identifiers—session IDs, device fingerprints, and behavioral embeddings. AI systems can now re-identify individuals from data you never intentionally labeled as sensitive. If you can’t verify the location of every derived identifier, you are failing the accountability requirements.

"DELETE" is a Lie in the World of Immutable Files

In traditional Relational Database systems, deletion is a predictable row-level transaction. In the OLAP/Lakehouse world, your SQL DELETE is a lie. Because these systems rely on immutable columnar formats like Parquet, a delete command doesn’t remove data; it triggers a cascading failure of storage efficiency.

Behind a “Logical Table: Deleted” checkmark, a standard delete command actually causes:

• Massive I/O & Compute Spikes: The system must scan, filter, rewrite, and replace entire file groups.

• S3 Tier Promotion: Deletion jobs often drag data from “Cold” storage tiers back to “Hot,” spiking your monthly cloud bill while failing to actually purge the data.

• Passive Propagation: Deleted data persists in:

  • ❌ Delta/Iceberg History: Transaction logs maintain the state for “time travel.”

  • ❌ S3/Cloud Backups: Data lives on in secondary storage and snapshots.

  • ❌ Stale Aggregates: Downstream summaries still contain the mathematical influence of the deleted records.

The AI Black Hole—Logs, Embeddings, and the Unknown

The most dangerous tier of the deletion hierarchy is the “AI/Unknown” tier. Modern humanity are currently feeding massive amounts of PII into Large Language Models (LLMs), creating a trail of logs and high-dimensional embeddings that are mathematically impossible to “un-learn” or selectively purge.

Passive cleanup is no longer enough. We need an AI Control Plane that moves privacy to active runtime enforcement. This control plane must at least:

• Understand data sensitivity at the point of ingestion.

• Enforce policies at the prompt/inference level.

• Provide verifiable evidence of erasure across the entire AI lifecycle.

Conclusion: Beyond the Checkbox

True compliance requires a shift from “passive compliance” to “active privacy engineering.” A defensible strategy involves a phased rollout based on the priority hierarchy, such as for example:

1. Mandatory Right to Erasure (DDR) and Account Deletion.

2. Content deletion and retroactive backfills.

3. Addressing indirect identifiers (such as session IDs), full downstream propagation, and auditability.

Final Thought: Is your current “DELETE” button actually removing data, or is it just hiding it from view? The 2026 European Data Protection Board’s report makes it clear: regulators are no longer accepting “technical difficulty” or “immutable architecture” as valid excuses for data persistence. They are specifically targeting gaps in backup erasure and failed anonymization. If your data platform treats deletion as an afterthought, you aren’t compliant—you’re just lucky. For now.

Most teams still talk about privacy and model quality as if you can only have one.

Either you protect sensitive data, or you preserve enough context for the model to be useful.

That tradeoff sounds intuitive. It is also too simplistic.

Ideally with tools like Talon, raw PII never reaches the model. But there is a big difference between removing PII and removing meaning.

A flat placeholder like [PHONE] protects privacy, but it also hides the one thing the model may actually need to answer correctly: is this a German number, a Polish number, or a French one?

So I tested a different approach.

Instead of sending raw personal data to the model, Talon can replace it with structured placeholders that preserve only safe, task-relevant semantics. Not the original value. Just the minimum useful context.

And when I compared that enriched approach against legacy type-only redaction, the enriched version won in both evaluation runs.

Subscribe now

TL;DR

I ran two 100-prompt A/B evaluations on Dativo Talon to test a simple question: if we redact PII before the model sees input, can we still preserve enough meaning for useful reasoning? Answer: yes. Enriched redaction (semantic placeholders) beat legacy type-only redaction in both runs, especially on attribute-dependent tasks like country routing and payment-method decisions.

The setup

I tested two variants.

Variant A: legacy redaction
The model sees flat placeholders such as [PERSON], [EMAIL], [PHONE], [IBAN].

Variant B: enriched redaction
The model sees structured placeholders such as:

<PII type=”phone” country_code=”PL”/>

or

<PII type=”email” domain_type=”free”/>

In both cases, raw PII is removed before model input.

This is not a comparison between “private” and “non-private.” Both variants keep raw personal data away from the model. The only difference is whether the model still receives safe semantic hints that matter for the task.

I ran two full experiments:

• Run 1: gpt-4o-mini, 100 prompts. See full logs

  
  

• Run 2: gpt-4o, 100 prompts. See full logs
  

Each prompt was scored on four dimensions, from 1 to 10:

1. attribute reasoning(“context”)

2. utility preservation

3. semantic coherence

4. helpfulness
   

So each prompt had a maximum total score of 40.

What happened

Run 1 — gpt-4o-mini (N=100)

• A mean total: 23.87

• B mean total: 28.58

• Mean delta: +4.71

• Numeric wins: B 59, A 19, ties 22

Run 2 — gpt-4o (N=100)

• A mean total: 23.5

• B mean total: 30.2

• Mean delta: +6.7

• Numeric wins: B 63, A 26, ties 11

Enriched redaction wins in both runs.
The strongest lift is exactly where it should be: attribute reasoning.

Where enrichment helped most

Not all semantic attributes are equally valuable.

The strongest gains came from attributes that directly affect routing, jurisdiction, or payment behavior.

Strongest lift

• PHONE → country_code

• IBAN → country_code

This was the clearest signal in both runs.

If the model sees only [PHONE], it cannot reliably decide whether the request belongs with Germany, Poland, France, or another support flow.

If it sees <PII type="phone" country_code="DE"/>, that ambiguity disappears without exposing the original number.

The same pattern showed up for IBANs. A country code is often enough to reason about SEPA, local handling, or country-specific banking logic.

Moderate lift

• PERSON → gender

• EMAIL → domain_type

These still helped, but less dramatically.

That also makes sense.

A corporate email domain or a gendered title can improve the answer, but the downstream task is often less deterministic than country-based routing. The model can sometimes get close with generic language even without the attribute.

Weakest area

• LOCATION → scope such as city, region, or country

This was the least convincing category.

The issue was not necessarily the enrichment itself. It was the prompt design.

Too many location prompts could still be answered with generic legal boilerplate. If a question does not force the model to actually use the distinction between city, region, and country, then that attribute will not show its value.

So this is less “scope does not help” and more “the benchmark did not pressure-test scope hard enough.”

What this means in practice

This is the production lesson.

Most teams fall into one of two bad patterns.

The first is to send raw prompts with PII to the model and hope governance happens somewhere later.

The second is to over-redact everything into useless placeholder soup and then act surprised when the model starts guessing.

Neither is a good long-term design.

The better path is narrower and more disciplined:

• remove raw PII before the model sees it

• preserve only the minimum safe semantics needed for reasoning

• decide those semantics through policy

• record evidence of what the model actually saw

That is the operating model Talon is built around, and this evaluation supports it.

Edge cases worth being honest about

The result is strong, but not perfect.

A few caveats matter.

1. Prompt quality still varied

Some prompts were genuinely attribute-dependent. Others were only loosely so.

That matters because if a prompt can be answered with generic common sense, the benchmark becomes less discriminative.

2. Judge behavior still has style bias

In a few cases, longer and more generic answers scored surprisingly well, even when a shorter answer was more precise.

That is a familiar problem in LLM-as-judge evaluations.

3. Order effects were more visible on the smaller model

I saw a bit more sensitivity in the gpt-4o-mini run than in the gpt-4o run.

Not enough to change the direction of the result, but enough to keep in mind.

4. Location-scope prompts need to be redesigned

This was the weakest benchmark segment and the one I would trust least in its current form.

So yes, the result is real. But some parts of the evaluation are stronger than others.

What about cost?

This is usually the first practical objection.

Semantic placeholders are longer. So do they make inference meaningfully more expensive?

In these runs, not in a way that mattered.

• In the gpt-4o-mini run, Variant B was slightly cheaper overall.

• In the gpt-4o run, Variant B was materially cheaper in observed run cost.

That does not mean enriched placeholders are inherently cheaper token-for-token. They are not.

It means end-to-end cost is dominated by full model behavior, especially output length and answer shape, not just placeholder size.

So the real takeaway is simpler:

The quality gain was clear, and the added redaction structure did not create a practical cost penalty.

Why This Matters for Production

Most teams still run one of two broken patterns:

1. send raw prompts with PII and hope policy catches up later, or

2. over-redact into useless [TYPE] soup and lose utility.

There is a better middle path:

• remove raw PII from model input

• preserve a minimal, safe semantic layer

• apply policy-as-code to which attributes are allowed

• keep evidence for what the model actually saw

That is what this experiment validates.

What we are changing next in Talon

Based on these runs, here is what I would do next:

1. Keep enriched redaction as the default for supported models.

2. Improve the location-scope benchmark, especially prompt quality and dictionaries.

3. Use a separate judge model and add a small human-reviewed subset.

4. Run multi-seed evaluations with confidence intervals in CI.

The core result is already useful. But the next version of the benchmark should be harder, cleaner, and more defensible.

Final Thought

Privacy-preserving AI does not need to be blind AI.

If your redaction layer removes everything useful, the model will guess.
If your redaction layer keeps safe semantics, the model can still reason.

This is not a theoretical point. We measured it across 200 prompt pairs.

No raw PII to the model. Better answers anyway.

The current obsession with model “intelligence” is a failure of engineering discipline. CTOs and Senior Architects are chasing leaderboard scores like teenagers chasing fashion trends, only to watch their multi-agent systems (MAS) implode the moment they hit production.

The primary cause of multi-agent failure isn’t “weak models” - every week, a new model tops another benchmark. Every month, another company claims its agents are more autonomous, more intelligent, more capable. And yet the same thing keeps happening in production: Multi-agent systems look impressive in demos, then fall apart under real load. Not because the models are weak, but because the framework is.

The industry is solving the wrong problem. We are building systems as if LLMs are deterministic components, ignoring the reality that uncertainty at every agent handoff creates a multiplicative decay in reliability. High-performing models in a controlled notebook are a demo-day vanity metric. In the real world, “stochastic hope” is an engineering liability. Reliable agentic systems are not built by choosing better models; they are built by engineering rigid data boundaries and treating the entire system as a distributed data pipeline.

A multi-agent system is not “a group of smart agents working together”. It is a distributed pipeline of untrusted intermediate states.

Software Engineering is dead, longs live the software engineering

Agentic Systems as Probabilistic Pipelines

When you wire multiple agents together, you are building a series-system pipeline governed by Lusser’s Law . In reliability engineering, this is the logic of a series system: if a workflow requires multiple sequential steps, the total success rate is the product of the success rates of the individual steps.

A single agent can look excellent in isolation—and that is exactly the trap. A model with 98% task accuracy sounds production-ready, but production systems are judged end-to-end.

In reliability engineering, a sequential workflow’s success is the product of the reliability of each step. If each hop succeeds with probability $p$, then an $n$-step workflow succeeds with probability:

Even with a stronger agents backed by better models, the decay is sharp:

• 1 agent at 98% → Total success: 98.0%

• 5 agents at 98% → Total success: 90.4%

• 10 agents at 98% → Total success: 81.7%

One bad output does not just fail locally; it becomes state. The next agent reads it, trusts it, and reasons on top of it. A hallucinated tool response doesn’t just reduce the chance of success at one step; it poisons the steps that follow.

On the production floor, these mathematical decays manifest as destructive pathologies that eat your budget and kill your uptime:

• Silent Schema Drift: A model outputs a slightly malformed JSON or an unexpected type. Without a validation gate, this corrupted state propagates downstream. Subsequent agents then condition their “reasoning” on garbage, leading to catastrophic cascades.

• Hallucinated Tool Outputs: Agents often condition their next action on a false return from an unvalidated tool call. Without a control plane to verify the return, the error remains invisible until the system produces a hallucinated final result.

• Unvalidated Handoffs: This is the peak of “stochastic hope”—passing raw strings between agents and praying the next model correctly parses the intent. It is the architectural equivalent of using eval() on untrusted user input.

• Operational Death Spirals: Recursive reasoning loops where supervisors fail to reach a terminal state. These loops consume thousands of tokens in seconds, draining API budgets without making an inch of progress toward the objective.

This is the key mindset shift: a multi-agent system is not “a set of smart models collaborating.” It is a distributed pipeline of untrusted intermediate states.

Once you see it that way, the engineering answer becomes obvious. You do not solve the problem by making every model slightly better. You solve it by inserting contracts, validation gates, and control points so the system can survive when one hop is wrong.

The Analogy: MAS are Untyped Distributed Pipelines

The modern multi-agent stack looks a lot like the messy early days of data engineering. We are passing intermediate state between stages without schemas, relying on downstream logic to “figure out” malformed upstream outputs.

This isn’t an AI problem—it’s a data reliability problem. Until we treat agentic handoffs as formal contracts, these systems will never scale.

The Missing Layer: Contracts , Validation and Control

The only way to break the multiplicative decay of Lusser’s Law is to introduce gates. By verifying an output before it reaches the next agent, you change the “reliability math.”

The Effective Probability formula

shows us the way out. By applying a validation catch rate (v), you recover failures before they propagate. A 98% accurate agent with a 90% validation catch rate becomes 99.8% effective. Over 10 hops, that’s the difference between an 81.7% failure-prone system and a 98% stable one.

A recipe for good life with AI Agents

Recipe 1: Pydantic + Instructor for Handoff Contracts

The first job is to stop bad state from propagating. That means validation gates.

Rule: Never pass raw LLM output to the next agent. Use Pydantic to define a contract and Instructor to force the model to satisfy it.

import instructor
from openai import OpenAI
from pydantic import BaseModel, Field, model_validator
from typing import Literal, Optional

client = instructor.from_openai(OpenAI(), max_retries=2)

class TicketDecision(BaseModel):
    action: Literal["approve", "reject", "escalate"]
    ticket_id: str
    risk_score: float = Field(ge=0, le=1)
    reason: str = Field(min_length=20, max_length=500)
    approver_id: Optional[str] = None

    @model_validator(mode="after")
    def enforce_business_rules(self):
        if self.action == "approve" and self.risk_score > 0.7:
            raise ValueError("high-risk tickets cannot be auto-approved")
        return self

This shifts the system from “trust the prompt” to “trust only validated state”.

Recipe 2: Best-of-N + Controlled Ranking

Validation prevents malformed outputs, but it doesn’t tell you which valid output is best. For complex tasks, use Best-of-N generation followed by a ranking step (like RULER).

1. Generate: Create 4 candidates.

2. Validate: Filter out any that fail the Pydantic schema.

3. Rank: Use a judge model to pick the winner based on a specific rubric (correctness, policy, clarity).

from typing import List
import instructor
from pydantic import BaseModel

# 1. Define the Contract
class AnalysisReport(BaseModel):
    summary: str
    sentiment: str
    confidence_score: float

# 2. Generate N Candidates
def generate_candidates(task: str, n: int = 4) -> List[AnalysisReport]:
    candidates = []
    for _ in range(n):
        try:
            # Each call is a stochastic draw
            res = client.chat.completions.create(
                model="gpt-4o",
                response_model=AnalysisReport,
                messages=[{"role": "user", "content": task}]
            )
            candidates.append(res)
        except Exception:
            continue # Skip malformed candidates
    return candidates

# 3. Apply RULER (The Judge)
def ruler_rank(candidates: List[AnalysisReport], task: str) -> AnalysisReport:
    # We ask a stronger model to act as the 'RULER' judge
    judge_prompt = f"""
    Task: {task}
    Candidates: {candidates}
    
    Rank these candidates based on:
    1. Depth of insight in the 'summary'
    2. Alignment between 'sentiment' and 'summary'
    3. Realistic 'confidence_score' (avoid overconfidence)
    
    Return only the best candidate's index.
    """
    # Logic to select the winner based on the judge's decision
    # ...
    return candidates[winner_index]

Recipe 3: Talon for Bounded Search and Budget Control

Local validation is a start, but production reliability requires an external control gateway. This is where Talon fits.

Search-based reliability (like Best-of-N or multi-step reasoning) is a double-edged sword. Without a control plane, a "smart" supervisor might trigger an infinite loop of retries, re-rankings, and judge calls. This leads to "test-time bankruptcy," where a single user request consumes hundreds of dollars in tokens. Talon places hard caps on the reasoning process itself.

policies:
  session_limits:
    max_cost: 2.50          # Absolute dollar cap per trace
    max_candidates: 4       # Limit Best-of-N generation width
    max_judge_calls: 2      # Limit the number of re-evaluations

Recipe 4: Talon as the Validated Commit Boundary

In production, you should never give an LLM “raw” write access to your database or APIs. An agent is a stochastic engine; if it hallucinates a DELETE flag or an unbounded LIMIT, the damage is instantaneous. Talon acts as a “Commit Wrapper,” forcing every tool call to pass through a deterministic governance layer before execution.

tool_policies:
  update_customer_records:
    max_row_count: 100            # Block "Update All" hallucinations
    require_dry_run: true         # Force a simulation first
    forbidden_argument_values:
      mode: ["truncate", "drop"]  # Block destructive operations
    arguments:
      query: redact               # Strip PII from logs/traces
    timeout: "15s"                # Kill runaway tool executions

Recipe 5: Talon Idempotency to Stop Duplicate Side Effects

Retries are a requirement for reliability, but they are dangerous for side-effecting tools like sending emails or charging credit cards. If an upstream planner fails and retries the entire sequence, you risk executing the same action twice. Talon tracks the intent of a tool call, ensuring that repeated calls with the same parameters do not trigger duplicate external actions.

tool_governance:
  send_notification_email:
    idempotency_key: "request_id" # Link to the unique session ID
    cache_ttl: "24h"              # Prevent double-send within a window
    on_duplicate: "return_cached" # Return the original success response
    strict_mode: true             # Fail if idempotency cannot be verified

Engineering by Design

Engineering reliable massive agentic systems requires two fundamental shifts in leadership perspective:

1. From “Models are smart” to “Systems must be safe under uncertainty.” Assume the model will fail. Build the safety net first.

2. From “Prompt Engineering” to “System Architecture.” Reliability is a function of boundary enforcement and budget control, not the phrasing of a system prompt.

Agentic systems do not become reliable by chance; they become reliable by design. Scalable AI is a data engineering challenge involving state management, contract enforcement, and cost governance. By implementing strict validation boundaries, using Talon as a control plane, and amortizing intelligence through Reinforcement Learning, you move from “stochastic hope” to production-ready infrastructure. Reliability is a choice. Make it.

It was a Sunday afternoon. I was building a personal scheduling agent — the kind that reads your calendar, finds gaps, and books meetings automatically. Super useful for coordinating squash or catching up with friends. I’d been hacking on it for a couple of days and wanted to test the full flow end-to-end.

I needed test contacts which would react, so I exported a few from my phone — figured I’d use people I actually know. My friends Marek and Tomek, and a couple of others. I told the agent to “book some test meetings for next week” and went to make coffee.

By the time I got back, it had sent real calendar invites. To all of them. For a meeting titled “Test Meeting 3” with no agenda, no description, nothing. Marek texted me: “what is this?” Lukasz sent mem in response. Tomek ignored it. Fine.

But there was a fourth contact in that export I’d forgotten about — someone from a networking event six months ago. I barely remembered his name. He accepted the invite without replying.

Monday morning he showed up on the call. I had no idea who was joining or why. I spent the next ten minutes pretending this was intentional.

The agent did exactly what I asked. “Book meetings.” With exactly the data I gave it. Nothing in between said “these are real people, maybe confirm first.”

That was the moment I understood the problem. Not that the agent was broken. That I had no layer between its decisions and the real world.

The Real Issue: LLM APIs Ship Without Operational Controls

After my inbox incident I started looking at how other teams run their agents. I talked to a friend at a mid-size fintech — five departments, three different API keys, zero idea what they were spending. Last month someone grep’d the logs during an unrelated investigation and found customer IBANs going to GPT-4 in plaintext. Thousands of requests over four months. Nobody had noticed because the bot worked great.

Different setups. Same gap.

“We have no idea what our agents are sending, what they’re allowed to do, or what they’re costing us.”

It’s not because anyone is careless. It’s because LLM APIs don’t ship with operational controls. There’s no per-caller identity. No way to say “this bot can’t see destructive tools.” No cost ceiling that actually shuts the door. You get an API key, you call the endpoint, and whatever the client sends goes straight through.

Every other piece of infrastructure I’ve run — databases, message queues, HTTP backends — has a proxy layer with auth, rate limiting, and observability. LLM traffic had none of that.

So I built one. It became the gateway component of Dativo Talon — an open-source tool I’ve been working on. A reverse proxy that sits between your clients and the LLM provider, identifies each caller, and applies policy before forwarding. One Go binary, one YAML config.

Here’s how the three features I needed most work in practice.

1. Tool Filtering — So the Model Never Learns calendar_invite Exists

This is the feature I built first, because it directly solves what happened to me.

My scheduling agent had five tools: read_calendar, find_gaps, create_draft, calendar_invite, and send_reminder. The first three are safe — they read data or create local drafts. The last two reach the real world. And the model couldn’t tell the difference, because I’d given it all five.

I didn’t need to remove calendar_invite from my code. I needed to remove it from what the model sees during testing.

That’s what the gateway does. It inspects the tools array in the JSON body before the request reaches OpenAI. Any tool matching a forbidden pattern gets stripped. The model never learns it exists. It can’t call calendar_invite if it was never told about calendar_invite.

Tool filtering is prevention, not detection. By the time you intercept a tool call, the model already decided to make it. The gateway removes the option before the decision happens.

Here’s what the config looks like:

gateway:
  default_policy:
    # "filter" = silently strip matching tools before the model sees them
    tool_policy_action: "filter"
    forbidden_tools:
      - "calendar_invite"  # the tool that sent three real meeting invites
      - "send_*"           # matches send_email, send_reminder, send_sms
      - "delete_*"         # matches delete_thread, delete_emails
      - "admin_*"
      - "bulk_*"
      - "drop_*"

What this looks like in practice:

# What OpenAI sees WITHOUT the gateway:
tools: [read_calendar, find_gaps, create_draft, calendar_invite, send_reminder]

# What OpenAI sees WITH the gateway:
tools: [read_calendar, find_gaps, create_draft]

The model gets the read tools and the draft tool. It can plan meetings and prepare invites all day long. But it can’t send anything, because it doesn’t know sending is an option.

Patterns use glob syntax, case-insensitive. send_* matches send_email, send_reminder, Send_SMS. The lists are additive across levels — default policy, provider, and per-caller overrides all merge into one set.

Two modes:

1. filter (default) — silently removes forbidden tools, forwards the rest. The agent keeps working; it just can’t see the ones that reach the real world.

2. block — rejects the entire request with HTTP 403 if any forbidden tool is present.

You can also go the other direction with a per-caller allowlist. Only the tools you name get through:

callers:
  - name: "scheduling-agent"
    api_key: "talon-gw-sched-001"
    tenant_id: "default"
    policy_overrides:
      # strict allowlist: ONLY these tools pass through
      allowed_tools: ["read_calendar", "find_gaps", "create_draft"]
      tool_policy_action: "block"

Now the agent can read, search, and draft. Nothing else. If I later wire up calendar_invite or send_email, the model will never see it unless I explicitly add it to the allowlist. This is the config I run for any agent that’s still in testing — default to read-only, unlock write tools deliberately.

The same principle would have prevented the OpenClaw incident. One forbidden_tools: ["delete_*"] line and the model would never have known deletion was an option.

Test it yourself:

curl -s -X POST http://localhost:8080/v1/proxy/openai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer talon-gw-sched-001" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role":"user","content":"Book a test meeting for next Tuesday"}],
    "tools": [
      {"type":"function","function":{"name":"find_gaps","parameters":{}}},
      {"type":"function","function":{"name":"calendar_invite","parameters":{}}}
    ]
  }'

calendar_invite gets stripped. The model only sees find_gaps. It can find the time slot, but it can’t book anything. The evidence record logs which tools were requested, filtered, and forwarded — signed and timestamped.

2. PII-Based Routing — Because I Fed the Model Real Email Addresses

Here’s the thing I didn’t appreciate until after the calendar incident: the tool wasn’t the only problem. The data was the problem too. I fed my agent real contact email addresses, and those went straight to OpenAI as part of the prompt. Even if I’d blocked calendar_invite, the model would still have seen sarah.chen@company.com and marcus.klein@bigcorp.de in its context window. Those are real people’s real email addresses sitting on OpenAI’s servers.

The fintech story made it worse. Their support bot was summarising customer tickets, and those tickets contained IBANs, email addresses, phone numbers. Thousands of requests over four months. All of it went to GPT-4 in plaintext.

Under pii_action: "block", every one of those requests would have been rejected before reaching OpenAI. Under "redact", the IBANs would have been replaced with [REDACTED:iban] and the emails with [REDACTED:email] before the model saw them. Either way, four months of undetected PII leakage doesn’t happen.

Every request that hits the gateway goes through a PII classifier first. It scans for personal data patterns — IBANs, emails, phone numbers, tax IDs — and assigns a data tier (0, 1, or 2) based on what it finds. That tier feeds into what happens next.

Four actions, two directions. On the request side (what the client sends): allow passes through, warn logs to evidence, redact replaces PII with [REDACTED:type] before forwarding, block rejects with HTTP 400. On the response side (what the model sends back): same four actions, with block returning HTTP 451.

Different callers get different treatment:

callers:
  - name: "internal-analytics"
    api_key: "talon-gw-analytics-001"
    tenant_id: "default"
    team: "data"
    policy_overrides:
      pii_action: "warn"             # log PII, forward unchanged — need to iterate fast
      response_pii_action: "warn"
      max_data_tier: 1               # deny tier 2 (high-sensitivity) requests

  - name: "customer-facing-bot"
    api_key: "talon-gw-custbot-002"
    tenant_id: "default"
    team: "support"
    policy_overrides:
      pii_action: "redact"           # IBANs, emails → [REDACTED:type] before OpenAI sees them
      response_pii_action: "redact"  # redact PII in model responses too
      max_data_tier: 0               # only public/anonymised data allowed

  - name: "scheduling-agent-dev"
    api_key: "talon-gw-sched-dev-001"
    tenant_id: "default"
    team: "engineering"
    policy_overrides:
      pii_action: "redact"           # would have caught sarah.chen@company.com
      response_pii_action: "warn"

Internal analytics gets warn — I see what PII is flowing, but the team can iterate. Customer-facing bot gets redact — any IBAN or email in the prompt becomes [REDACTED:iban] before it touches OpenAI. My scheduling agent in dev gets redact — so even if I’m lazy and paste real contacts into the test prompt, the gateway scrubs them before the model sees them. Sunday-afternoon-proof.

The max_data_tier adds a second gate. If the classifier tags a request as tier 2 (high-sensitivity) but the caller is only cleared for tier 0, the policy engine denies it regardless of the PII action. Your customer-facing bot can’t accidentally process data it was never supposed to see.

Response scanning works for both streaming (SSE) and non-streaming. For streams, the gateway buffers the full response, scans, and forwards the original events if clean or rewrites them if redaction is needed.

Every PII detection — both directions — ends up in the evidence store. talon audit list shows which requests contained PII, what types, and what action was taken. No log grepping.

3. Cost Caps — I Burned $385 on a Saturday. Here’s How I Made Sure It Never Happens Again.

Different weekend, different mistake. I left a test loop running — GPT-4, increasingly long context windows, no stop condition. By Sunday evening: $385 in API charges on a project budgeted at $20/month.

I seem to learn everything on weekends.

That Monday I added cost caps to the gateway. Least interesting feature to build, most money saved.

Here’s the thing most teams don’t realise: you find out about a cost overrun when the monthly invoice arrives. OpenAI’s usage dashboard updates, but there’s no hard stop. No circuit breaker. A gateway that blocks at the daily limit is fundamentally different from a provider alert that shows up 30 days later.

Every request gets a cost estimate based on the model and token count. The gateway tracks daily and monthly spend per caller by querying the evidence store — the same SQLite database that holds audit records. When a caller hits the cap, the next request gets a 403. No grace period.

callers:
  - name: "production-agent"
    api_key: "talon-gw-prod-001"
    tenant_id: "default"
    policy_overrides:
      max_daily_cost: 50.00          # hard cap: 403 after $50/day
      max_monthly_cost: 1000.00

  - name: "dev-sandbox"
    api_key: "talon-gw-dev-002"
    tenant_id: "default"
    policy_overrides:
      max_daily_cost: 5.00           # weekend loops die at $5, not $385
      max_monthly_cost: 50.00

default_policy:
  max_daily_cost: 100.00             # global ceiling for callers without overrides
  max_monthly_cost: 2000.00

production-agent gets $50/day. dev-sandbox gets $5/day. If I leave another loop running on a Saturday, the gateway kills it at $5 instead of letting it burn for 48 hours.

The CLI tells you where you stand:

talon costs --tenant default

# Agent             Today ($)   Month ($)   Limit (day)   Limit (month)
# production-agent      22.10     487.30         50.00        1000.00
# dev-sandbox            1.80      28.70          5.00          50.00
# support-bot            0.80      15.20          —             —
# Total                 24.70     531.20        100.00        2000.00

Every evidence record includes model_used, cost, input_tokens, output_tokens, and duration_ms. Export with talon audit export --format csv and you can answer: which model is burning the most, which caller is growing fastest, where tokens are wasted on retries.

Rate limiting complements cost caps for the speed-of-spend problem. Cost caps say “no more than $50 today.” Rate limits say “no more than 60 requests per minute.” Together they catch both the slow bleed and the fast burst:

rate_limits:
  global_requests_per_min: 300       # shared across all callers
  per_caller_requests_per_min: 60    # per-caller cap — slows runaway agents

A Full Config — All Three Together

Here’s what I actually run for three callers, each with different tool, PII, and cost policies:

gateway:
  enabled: true
  listen_prefix: "/v1/proxy"
  mode: "enforce"

  providers:
    openai:
      enabled: true
      secret_name: "openai-api-key"          # real key in encrypted vault, never in client config
      base_url: "https://api.openai.com"
      allowed_models: ["gpt-4o", "gpt-4o-mini", "gpt-4-turbo"]

  callers:
    - name: "production-agent"
      api_key: "talon-gw-prod-001"           # caller token — not the OpenAI key
      tenant_id: "default"
      team: "engineering"
      allowed_providers: ["openai"]
      policy_overrides:
        max_daily_cost: 50.00
        max_monthly_cost: 1000.00
        pii_action: "redact"                 # scrub PII from requests
        response_pii_action: "warn"          # log PII in responses, don't block
        allowed_models: ["gpt-4o", "gpt-4o-mini"]
        forbidden_tools: ["delete_*", "admin_*", "drop_*", "send_*"]

    - name: "internal-bot"
      api_key: "talon-gw-internal-001"
      tenant_id: "default"
      team: "support"
      allowed_providers: ["openai"]
      policy_overrides:
        max_daily_cost: 10.00
        max_monthly_cost: 200.00
        pii_action: "redact"
        response_pii_action: "redact"
        allowed_tools: ["search_kb", "read_ticket", "create_draft"]  # strict allowlist
        tool_policy_action: "block"          # reject if any other tool appears

    - name: "dev-sandbox"
      api_key: "talon-gw-dev-002"
      tenant_id: "default"
      team: "engineering"
      allowed_providers: ["openai"]
      policy_overrides:
        max_daily_cost: 5.00                 # Saturday-proof
        max_monthly_cost: 50.00
        pii_action: "redact"                 # scrub real emails from test prompts
        allowed_models: ["gpt-4o-mini"]      # cheapest model only

  default_policy:
    default_pii_action: "warn"
    response_pii_action: "warn"
    max_daily_cost: 100.00
    max_monthly_cost: 2000.00
    require_caller_id: true
    log_prompts: true
    tool_policy_action: "filter"
    forbidden_tools:
      - "delete_*"
      - "admin_*"
      - "export_all_*"
      - "bulk_*"
      - "rm_*"
      - "drop_*"
    attachment_policy:
      action: "warn"
      injection_action: "block"              # block prompt injection in file attachments
      max_file_size_mb: 10

  rate_limits:
    global_requests_per_min: 300
    per_caller_requests_per_min: 60

  timeouts:
    connect_timeout: 10s
    request_timeout: 120s
    stream_idle_timeout: 60s

Three callers, three risk profiles. production-agent gets a generous budget, PII redaction on input, and a blocklist of destructive and send tools. internal-bot gets a strict allowlist (three tools, nothing else), PII redaction both ways, and a tighter budget. dev-sandbox gets the cheapest model, PII redaction (no more testing with real emails), and a $5/day ceiling.

The clients don’t know about any of this. They point at the gateway URL with their caller key. The gateway does the rest.

When This Is the Wrong Choice

A gateway adds a hop. If you’re running a single script on your laptop and you’re the only user, it’s overhead for no benefit.

If you need the absolute lowest first-token latency and you’re at the edge, PII scanning on streaming responses adds buffering time. The passthrough path (pii_action: "allow") is ~1ms overhead, but redaction on a long stream is measurable.

If your agents only have read-only tools and never touch sensitive data, the risk profile is lower. Still worth auditing, but the urgency drops.

If you’re not dealing with customer PII yet — pre-revenue, purely internal — the compliance angle is less pressing. But the moment you start processing real user data or fall under NIS2 scope, the gateway goes from “nice to have” to “how did we not have this.”

And if you need policy on every individual tool invocation — not just what the model is told about, but what happens when the tool runs — a gateway isn’t enough. That’s a different shape: an MCP proxy or full agent runner with per-tool policy.

Final Thought

Calendar invites to real people. $385 on a weekend loop. IBANs in plaintext for four months. Every one of those happened because there was nothing between the agent and the API — no filter on what tools the model could see, no scan on what data was in the prompt, no ceiling on what it could spend.

The fix is the same pattern we’ve been using on HTTP traffic for twenty years: a reverse proxy with policy. It just hadn’t been applied to LLM APIs yet.

talon init takes fifteen minutes. The difference between “the agent booked a meeting with a stranger” and “the agent tried and the gateway said no” is one YAML file.

Checkout GitHub for more.

And then it deleted Meta Director’s email.

If you missed it: in February 2026, an OpenClaw agent connected to a Meta director for AI’s inbox went on a speed-run. It mass-deleted emails, ignored stop commands, blew through cost in minutes, and kept going even after the user tried to shut it down. The context window compacted and the agent lost track of the original instructions. It just… decided deleting was the task.

That scared me. Not because OpenClaw is broken — it’s a great agent runtime. But because there’s nothing between the agent and the API. No filter on what tools the model sees. No cost ceiling. No way to remotely kill a run. No record of what happened that you could trust after the fact. It’s a straight pipe from agent to OpenAI, and if the agent goes sideways, you find out when the damage is done.

So I built a way to put a wall in front of it.

Subscribe now

The actual problem

OpenClaw sends your LLM requests directly to OpenAI. The model sees every tool you registered — including delete_emails, bulk_remove, drop_table, whatever you’ve wired up. If the model decides to call one, it calls it. There’s no checkpoint, no approval, no “hey, are you sure?”

And there’s no audit trail. If something goes wrong, you’re digging through stdout logs trying to reconstruct what the agent did, in what order, with whose data. Good luck.

What I wanted was simple:

• Don’t let the model see tools it shouldn’t use. Not “block the call after it happens.” Remove the tool from the request before the model knows it exists. It can’t call delete_emails if it was never told about delete_emails.

• Cap the spend. Daily, monthly, per-request. When the budget’s done, the gateway says no.

• Record everything. Every request, every denial, every tool that got stripped. Signed, queryable, trustworthy.

• Keep my real API key out of OpenClaw. OpenClaw gets a caller token. The real key lives in an encrypted vault and gets injected at forward time.

How I set it up

I built this into Dativo Talon — a single Go binary that sits between OpenClaw and OpenAI. Here’s the exact setup I run.

Step 1: Install and init

go install github.com/dativo-io/talon/cmd/talon@latest

mkdir talon-openclaw && cd talon-openclaw
talon init --pack openclaw --name openclaw-gateway

This generates two files: agent.talon.yaml (server policy) and talon.config.yaml (gateway config). The gateway config is where the real controls live.

Step 2: Store your OpenAI key in the vault

export TALON_SECRETS_KEY=$(openssl rand -hex 32)  # save this somewhere safe
talon secrets set openai-api-key "$OPENAI_API_KEY"

Your real OpenAI key is now encrypted at rest. OpenClaw will never see it.

Step 3: Start the gateway

talon serve --gateway

That’s it. Talon is now listening on localhost:8080.

Step 4: Point OpenClaw at Talon

In ~/.openclaw/openclaw.json:

{
  "models": {
    "providers": {
      "openai": {
        "baseUrl": "http://localhost:8080/v1/proxy/openai/v1",
        "apiKey": "talon-gw-openclaw-001",
        "api": "openai-responses",
        "models": [
          { "id": "gpt-4o", "name": "gpt-4o" },
          { "id": "gpt-4o-mini", "name": "gpt-4o-mini" }
        ]
      }
    }
  }
}

Notice the apiKey — that’s the caller token, not the OpenAI key. Talon identifies OpenClaw by this token and injects the real key when it forwards to OpenAI.

Restart OpenClaw (openclaw gateway stop && openclaw gateway start) and you’re running through the gateway.

The config that would have stopped the inbox incident

Here’s the talon.config.yaml I use. I’ll walk through the parts that matter.

gateway:
  enabled: true
  listen_prefix: "/v1/proxy"
  mode: "enforce"

  providers:
    openai:
      enabled: true
      secret_name: "openai-api-key"
      base_url: "https://api.openai.com"
      allowed_models: ["gpt-4o", "gpt-4o-mini", "gpt-4-turbo"]

  callers:
    - name: "openclaw-main"
      api_key: "talon-gw-openclaw-001"
      tenant_id: "default"
      team: "engineering"
      allowed_providers: ["openai"]
      policy_overrides:
        max_daily_cost: 25.00
        max_monthly_cost: 500.00
        pii_action: "redact"
        allowed_models: ["gpt-4o", "gpt-4o-mini", "gpt-4-turbo"]

  default_policy:
    require_caller_id: true
    log_prompts: true

    # --- THIS IS THE BIG ONE ---
    # Tool governance: strip dangerous tools BEFORE the model sees them.
    tool_policy_action: "filter"
    forbidden_tools:
      - "delete_*"
      - "admin_*"
      - "export_all_*"
      - "bulk_*"
      - "rm_*"
      - "drop_*"

    # PII: redact personal data from requests headed to OpenAI
    default_pii_action: "redact"
    response_pii_action: "warn"

    # Attachments: scan PDFs and CSVs for prompt injection
    attachment_policy:
      action: "warn"
      injection_action: "block"
      max_file_size_mb: 10

  rate_limits:
    global_requests_per_min: 300
    per_caller_requests_per_min: 60

  timeouts:
    connect_timeout: 10s
    request_timeout: 120s
    stream_idle_timeout: 60s

Let me break down what each piece would have done in the emails incident:

• forbidden_tools: ["delete_*", "bulk_*"] — The agent had access to delete_email, delete_thread, and bulk operations. With this config, Talon strips those tools from the JSON body before OpenAI ever sees them. The model literally cannot decide to delete anything because it doesn’t know deletion is an option.

• tool_policy_action: "filter" — This is the mode. "filter" silently removes forbidden tools and forwards the rest. If you want to be more aggressive, set it to "block" — that rejects the entire request if any forbidden tool is present. I prefer "filter" because it keeps the agent functional for everything except the dangerous stuff.

• max_daily_cost: 25.00 — The incident ran up significant cost in minutes. This cap shuts the door after $25/day for this caller. Done. No negotiation.

• per_caller_requests_per_min: 60 — The agent was firing requests as fast as it could. Rate limiting slows a runaway agent to a manageable pace and gives you time to notice.

• request_timeout: 120s — No single request gets more than 2 minutes. The agent can’t sit in an infinite loop waiting for a response.

Per-caller tool allowlists (when you want to be strict)

If forbidden_tools is a blocklist, you can also go the other direction — a strict allowlist. Only the tools you name get through:

callers:
  - name: "openclaw-main"
    policy_overrides:
      allowed_tools: ["search_web", "read_file", "list_files", "create_draft"]
      tool_policy_action: "block"

Now OpenClaw can only use those four tools. Everything else — delete_emails, send_email, admin_reset, whatever — gets rejected. The model never sees them. This is the nuclear option and it’s the one I’d use if I were connecting an agent to anyone’s inbox.

Verify it works

Send a request with a dangerous tool and watch what happens:

curl -s -X POST http://localhost:8080/v1/proxy/openai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer talon-gw-openclaw-001" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role":"user","content":"Clean up my inbox"}],
    "tools": [
      {"type":"function","function":{"name":"search_web","parameters":{}}},
      {"type":"function","function":{"name":"delete_emails","parameters":{}}}
    ]
  }'

delete_emails gets stripped. The model only sees search_web. Check the evidence:

talon audit list --agent openclaw-main --limit 5

You’ll see exactly which tools were requested, which were filtered, and which were forwarded. Signed and timestamped.

When this isn’t the answer

• You’re just playing around. If it’s a hobby project and nothing is at stake, the gateway is overhead you don’t need. Especially if you have unlimited money , and you have nothing to hide ;)

• You trust the tool set completely. If your agent only has read-only tools — no delete, no write, no send — the risk profile is lower. Still worth auditing, but the urgency is different.

• You need governance inside MCP tool calls. The gateway governs what goes to and from the LLM. If you need policy on every individual tool invocation (not just what the model is told about), that’s Talon’s MCP proxy — a different deployment shape.

Final thought

The email incident wasn’t a bug in OpenClaw. It was a missing layer. The agent did exactly what agents do — it picked from the tools it was given and executed. The problem is it was given delete_emails and nobody was standing between the model and that tool.

That’s what I’m solving. Not replacing OpenClaw — I still use it every day( finger cross my tool would catch all dangerous stuff). Just making sure it runs through a gateway that strips the dangerous tools, caps the cost, and writes down everything that happened. If something goes wrong, I want to know exactly what the agent tried to do and exactly where it was stopped.

talon init --pack openclaw. Fifteen minutes. That’s the difference between “the agent deleted everything” and “the agent tried to delete everything and was told no.”

“How does this agent can remember anything?”

Over the last six months, I’ve watched the same architectural question surface in every AI project I’ve been close to. Startups building their first agent workflows. Scaleups wrapping compliance around existing automation. Large enterprises deploying AI across regulated environments.

Not remember within a single chat. That’s just a context window. I mean remember across sessions, across days, across hundreds of runs — while staying coherent, efficient, and (increasingly) compliant.

This article explains why agent memory became one of the hottest enteprise infrastructure problem of early 2026.

Subscribe now

The Problem That Context Windows Don’t Solve

Every LLM is stateless. GPT-4, Claude, Gemini — they all start each API call with a blank slate. The context window creates an illusion of memory, but it’s really just a very large input buffer.

That works for chatbots. It breaks for agents.

The moment you build an agent that runs repeatedly — a sales analyst that processes reports daily, a support bot that handles tickets across shifts, a compliance monitor that learns from policy violations — you hit the wall.

Context windows reset. Knowledge is lost. The agent makes the same mistakes it made last week. It asks the user the same questions. It doesn’t learn.

Bigger context windows don’t fix this. Models with 128K or even 1M token windows still reset between API calls. Even within a single call, performance degrades over long contexts — models lose track of details buried in the middle. And stuffing every prior interaction into the prompt gets expensive fast.

What you actually need is a memory system: infrastructure that decides what to store, how to retrieve it, when to update it, and when to forget.

This is where things get interesting.

Three Types of Memory (That Actually Matter)

The academic literature — particularly the survey “Memory in the Age of AI Agents” (December 2025) and the CoALA framework — converges on three types of long-term memory that production agents need. This isn’t just taxonomy. Each type has different storage patterns, retrieval strategies, and lifecycle rules.

Semantic memory stores what the agent knows — facts, preferences, constraints. “The user prefers Python.” “Our fiscal year starts in April.” “The compliance threshold is €100K.” These are stable facts that hold across sessions and should be updated when they change, not duplicated.

Episodic memory stores what happened — specific interactions, outcomes, decisions. “On February 15, the policy engine denied SQL access because the query contained PII.” “Last Thursday’s report cost €0.42 and used Claude Sonnet.” These are events. They accumulate. They provide context for pattern recognition.

Procedural memory stores how to do things — learned behaviors, workflows, response patterns. “When the user asks for a financial summary, always check PII classification first.” “Use the compact format for Slack responses.” These are rare but powerful — they represent the agent actually improving its own behavior.

Most memory systems on the market implement some version of this taxonomy, whether they call it that or not. The real differentiation is in what happens after storage: consolidation, retrieval, and lifecycle management.

The Core Pipeline: Extract → Consolidate → Store → Retrieve

If you just append every interaction to a database and search it later, you’ve built a log, not a memory. Logs grow without bound, contain duplicates, hold contradictory facts, and become increasingly expensive to query.

Production memory systems follow a pipeline pattern that mirrors (loosely) how human memory works.

Extraction takes raw conversation data and distills it into structured memory units — facts, observations, preferences. Instead of storing “the user said they switched from JavaScript to Rust last month,” you extract the fact: {preference: "Rust", negated: "JavaScript", timestamp: "2026-02"}.

Consolidation is where the real engineering happens. New facts are compared against existing memories. Duplicates are detected. Contradictions are resolved. Stale entries are invalidated. Without consolidation, your memory fills with noise. With it, storage drops by roughly 60% and retrieval precision improves by over 20%, according to Mem0’s benchmarks on LOCOMO.

Storage persists the processed memories — typically in a vector database for semantic search, sometimes augmented with a graph database for relational reasoning. The choice of backend shapes what kinds of queries you can answer efficiently.

Retrieval fetches relevant memories at query time and injects them into the agent’s prompt. The sophistication here ranges from simple keyword matching to composite scoring that weighs relevance, recency, memory type, and trust.

Every serious memory product implements some version of this pipeline. The differences are in the details.

The AUDN Cycle: How Mem0 Handles Consolidation

Mem0 — arguably the clearest “memory as a product” offering — popularised what I’ll call the AUDN cycle: Add, Update, Delete, Noop.

For each candidate fact extracted from a conversation, Mem0 retrieves the top-S most similar existing memories using vector similarity. It then presents both the new fact and the existing memories to an LLM through a tool-calling interface. The LLM decides:

• Add: genuinely new information, store it

• Update: augments an existing memory with more detail

• Delete: contradicts an existing memory, remove the old one

• Noop: already captured, skip

This is elegant because it offloads conflict resolution to the LLM itself. The model decides whether “prefers Python” should be overwritten by “switched to Rust” or whether both should coexist. No hand-crafted rules needed.

The results are strong. On the LOCOMO benchmark, Mem0 delivers a 26% accuracy uplift over OpenAI’s built-in memory, 91% lower p95 latency compared to full-context baselines, and 90% token cost savings. The graph-enhanced variant (Mem0g) adds entity-relationship extraction for multi-hop reasoning — “what decisions led to this outcome?” becomes answerable.

The limitation is that Mem0 deletes contradicted facts. Once overwritten, the old memory is gone. For a personal assistant, that’s fine. For a regulated enterprise environment, it’s a problem — auditors want to see what the system believed at a given point in time, not just what it believes now.

Temporal Knowledge Graphs: How Zep Thinks About Time

Zep, and its open-source engine Graphiti, take a fundamentally different approach. Where Mem0 is optimised for fast, flat fact retrieval, Zep builds a temporal knowledge graph with bi-temporal semantics.

Every fact in Zep has four timestamps: when the event occurred, when it became invalid, when the system first learned about it, and when the system stopped considering it current.

This dual timeline — event time and ingestion time — enables queries that no flat memory store can handle. “What did the agent know as of last Tuesday?” “Show me how this relationship evolved over the past month.” “When did we first learn that the customer changed their billing address?”

When new information contradicts existing facts, Zep doesn’t delete the old edge. It invalidates it — setting an invalid_at timestamp and preserving the full history. The knowledge graph grows richer over time, not just larger.

On benchmarks, Zep achieves up to 18.5% accuracy improvement on LongMemEval (which tests cross-session reasoning and temporal tasks) and 90% latency reduction compared to baselines. It particularly excels at multi-hop reasoning — connecting facts across multiple sessions and time periods.

The trade-off is complexity. Zep requires a graph database (Neo4j or FalkorDB), embedding infrastructure, and more operational overhead than a simple key-value memory store. The open-source Graphiti framework makes this accessible, but it’s still a heavier commitment than Mem0’s three-line API.

LangMem: Memory as a Library

LangChain’s LangMem takes a more modular approach. Rather than being a standalone memory product, it provides composable primitives — create_memory_manager, create_search_memory_tool, create_prompt_optimizer — that plug into LangGraph’s agent framework.

LangMem separates memory into profiles (structured schemas updated in-place, like user preferences) and collections (unbounded document sets searched semantically). It supports background consolidation through a memory manager that extracts, deduplicates, and updates memories asynchronously.

The key design choice is storage agnosticism. LangMem doesn’t mandate a specific backend — it works with any store that supports save and semantic search. MongoDB, Postgres with pgvector, in-memory stores, or custom implementations all work.

For teams already invested in the LangChain ecosystem, LangMem is the path of least resistance. The trade-off is that you’re assembling pieces rather than getting a turnkey system. There’s no built-in temporal model, no graph-based reasoning, and conflict resolution is delegated to the LLM without the structured AUDN pipeline that Mem0 provides.

Letta (MemGPT): Memory as Agent State

Letta — the production evolution of the MemGPT research project — treats memory as a first-class component of the agent’s state. Agents have explicit core memory blocks (always injected into the prompt — persona, goals, preferences) and archival memory (out-of-context storage searched on demand).

The distinctive feature is that agents can explicitly write, update, and delete their own memory blocks through tool calls. Memory isn’t something that happens to the agent — it’s something the agent actively manages.

This makes Letta particularly well-suited for persistent assistants and local-LLM deployments (it works well with Ollama and vLLM). The agent maintains identity and continuity across restarts, which is critical for long-lived worker agents.

The limitation is that Letta’s memory model is agent-centric. It doesn’t natively handle cross-agent memory sharing, multi-tenant isolation, or compliance-grade audit trails. For single-agent personal assistant scenarios, it’s excellent. For enterprise multi-agent deployments, you’ll need to build additional infrastructure.

The Gap Nobody Is Filling: Governed Memory

Here’s what struck me as I surveyed the landscape.

Every product optimises for recall quality. Better accuracy. Lower latency. Fewer tokens. Richer reasoning. Those metrics matter. But they’re all answering the same question: “How well does the agent remember?”

Nobody is answering: “Is it safe for the agent to remember this?”

Consider what happens when an AI agent processes a customer support ticket containing a credit card number, a medical diagnosis, or an employee’s home address. The agent learns from it — stores an observation, updates its memory, maybe adjusts its behavior. But should it?

Under GDPR Article 25, that’s a data protection by design question. Under the EU AI Act (full enforcement August 2026), high-risk AI systems need technical documentation of every decision. Under NIS2, incident response requires reconstructing what the system knew at the time of a breach.

None of the major memory products address this. Mem0 has no PII detection on memory writes. Zep has no policy enforcement layer. LangMem delegates governance entirely to the application developer. Letta stores whatever the agent decides to store.

This isn’t a criticism of those products — they’re solving a different problem. But for European enterprises deploying AI agents in regulated environments, the gap is real.

Dativo Talon

At Dativo Talon — the open-source compliance-first AI orchestration platform I am currently building, started from the governance side and worked toward recall quality, rather than the other way around.

Talon’s memory architecture wraps a full governance pipeline around every write operation. Before anything hits the memmory database, it passes through PII scanning (25+ EU-specific patterns covering all 27 member states), OPA policy evaluation, category validation against allow/forbid lists, policy override detection, conflict checking, and provenance tracking with trust scores. Every write — and every governance decision — generates an HMAC-signed evidence record.

The storage layer uses SQLite with FTS5 for full-text search, progressive disclosure (lightweight index entries for prompt injection, full detail on demand), and AI-compressed observations that reduce raw agent runs from thousands of tokens down to roughly 500-token structured summaries. ( N.B. I love SQLite and I truely believe it is excellent solution without unneccessary overhead)

What I am working right now — and what motivated this article — is the consolidation layer. I am working on a governed AUDN cycle inspired by Mem0’s approach, but with a critical difference: invalidated entries are preserved (Zep-style temporal invalidation), not deleted. Every consolidation decision — add, update, invalidate, noop — is governed and audited. We’re adding bi-temporal queries so any auditor can reconstruct what the agent knew at any point in time.

We’re also moving from flat timestamp retrieval to relevance-scored retrieval that weighs keyword relevance, recency, memory type (semantic/episodic/procedural), and trust score — matching the retrieval sophistication of Mem0 while preserving the audit trail.

The goal is build only memory system where an agent’s learning is both high-quality and compliance-grade. Where governed memory is a compliance asset, not just a developer convenience.

When You Don’t Need Persistent Memory

Ok, Memory isn’t always the answer.

If your agent handles single-turn queries — “translate this,” “summarise that document,” “generate a report” — the context window is sufficient. Adding a memory layer introduces complexity, latency, and storage costs with minimal benefit.

If your agent runs the same static workflow every time (process this CSV, send this email), procedural memory might help but semantic and episodic memory probably won’t.

If your data is already in a well-structured knowledge base, retrieval-augmented generation (RAG) over that knowledge base is likely a better fit than agent memory. Memory shines when the knowledge comes from the interactions themselves — not from pre-existing documents.

Memory pays off when agents run repeatedly, learn from outcomes, serve multiple users, or operate in environments where context evolves over time.

Final Thought

Agent memory is having its “Iceberg moment.”

Just as Iceberg standardised the table format for data lakes — separating storage from compute and making data engine-independent — the memory layer is becoming the standard infrastructure for making AI agents stateful, efficient, and persistent.

The products differ in approach. Mem0 optimises for speed and simplicity. Zep optimises for temporal reasoning and relational depth. LangMem optimises for composability. Letta optimises for agent autonomy.

But the underlying pattern is converging: extract, consolidate, store, retrieve. With scoring. With lifecycle management. With conflict resolution.

What’s still missing — and what I believe will matter enormously as AI regulation matures in Europe and worldwide — is governance around that pipeline. Not as an afterthought. Not as a compliance checkbox. But as a first-class architectural concern, where every memory write is scanned, evaluated, and signed.

If you’re building AI agents and thinking about memory architecture, I’d love to hear how you’re approaching it. Drop a comment or reach out — this is a fast-moving space and I’m learning from every conversation.

(1) Dativo Talon is open-source and available on GitHub.

Over the last 18–24 months, I’ve seen the same decision show up in very different companies. Startups building their first serious data platform. Scaleups trying to escape warehouse lock-in. Large enterprises re-platforming analytics.

Different contexts. Same outcome.

“Let’s standardise on Iceberg.”

Not because it’s trendy. Not because a vendor pushed it. But because Iceberg fixes a set of structural problems data teams have been carrying since the Hive era — and warehouses never really solved.

This article explains what problem Iceberg actually solves, why it’s gaining traction now, and how teams are using it in practice — with Snowflake, Databricks, and plain object storage like AWS S3.

The Core Problem Iceberg Solves (That Warehouses Don’t)

Most traditional data stacks tightly couple storage, metadata, and compute. Data lives where the engine lives. Moving compute usually means copying data. Governance, retention, and lifecycle rules get re-implemented per platform. Costs become opaque very quickly.

Warehouses optimise for convenience and performance, not architectural separation.

Iceberg flips this model!

Iceberg is not a database. It’s a table standard for data lakes.

Data is stored once — typically as Parquet on object storage — and any engine that understands Iceberg can read or write it safely. That “safely” part matters more than people realise, because it’s where previous lake designs failed.

Why Iceberg Succeeded Where Hive Tables Failed

Hive tables and raw Parquet worked until scale and concurrency arrived. Then the cracks appeared.

They lacked atomic writes. There was no snapshot isolation. Concurrent writers were unsafe. Schema evolution was fragile and often required rewrites or downtime. These weren’t edge cases — they were structural limitations.

Iceberg fixes this by making metadata first-class.

Every Iceberg table is defined by immutable data files, immutable metadata files, and snapshots that represent a consistent table state. Writers commit atomically. Readers always see a coherent view. Time travel, rollback, concurrent writes, and controlled schema evolution become normal operations rather than special cases.

In practice, Iceberg behaves like a real database table — but lives entirely on object storage.

That’s the real breakthrough.

Why Iceberg Is Becoming the Standard

Iceberg isn’t winning because it’s novel. It’s winning because several things finally aligned.

1. First, it is genuinely vendor-neutral. Snowflake, Databricks, AWS, Trino, Flink, and DuckDB all support it, and no single vendor controls the specification. That matters more than individual features.

2. Second, Iceberg enforces a clean separation of concerns. Storage lives in S3, GCS, or ADLS. Metadata lives in a catalog such as Glue, Nessie, Unity, Polaris, or Lakekeeper. Compute is whatever engine you choose today — and can change tomorrow. Each layer evolves independently.

3. Third, Iceberg is now operationally mature. Compaction, snapshot expiration, partition evolution, and schema evolution are no longer afterthoughts. Five years ago these were DIY problems. Today they’re table stakes.

Finally, Iceberg is enterprise-ready. It works at petabyte scale, supports multi-writer workloads, and integrates with existing IAM and catalog systems. That full combination simply didn’t exist before.

How Iceberg Is Used in Practice

In Snowflake environments, Iceberg is increasingly used to decouple storage from the warehouse. Data lives in S3 or GCS. Snowflake manages Iceberg metadata and queries the data in place. Teams keep Snowflake for BI and analytics, avoid duplicating raw and curated data, and retain an exit option if pricing or strategy changes.

The trade-off is that Snowflake controls the metadata layer and optimisation is less transparent than with native tables. Performance, however, remains very strong for analytics workloads. This pattern is especially common in finance and large enterprise analytics teams.

In Databricks environments, Iceberg often plays a different role. Tables live on S3, metadata sits in Unity Catalog or an external catalog, and Spark is used for heavy transformations. Querying may happen in Databricks, Snowflake, or Trino.

Teams choose Iceberg here to avoid Delta lock-in, share tables across engines, and align with multi-cloud strategies. Databricks increasingly acts as a powerful compute engine rather than the system of record.

Then there’s the “bare metal” S3 model — where Iceberg really shines.

Here, S3 is the system of record. Iceberg provides table semantics. Glue, Nessie, or Lakekeeper manage metadata. Spark, Trino, Flink, or DuckDB handle compute. This model is popular with infra-heavy startups, platform teams, and cost-optimising organisations.

It works because storage is cheap, compute is elastic, governance logic is centralised, and there’s no per-terabyte warehouse tax. But it only scales well if teams invest in compaction, file size control, and snapshot management.

Performance: How Iceberg Actually Compares

Let’s be honest: Iceberg itself doesn’t make queries fast.

Performance depends on file sizes, partitioning, metadata pruning, and the compute engine. Get those wrong and Iceberg will feel slow. Get them right and it performs extremely well.

Compared to raw Parquet on S3, Iceberg is dramatically faster for analytical queries because engines can prune files using metadata and read consistent snapshots. Compared to Snowflake native tables, Snowflake still wins for small BI queries, but Iceberg becomes competitive at scale and wins on cost predictability and flexibility.

Compared to Delta Lake, performance is broadly similar. Iceberg wins on openness and portability. Delta wins on Databricks-specific optimisations.

In most real systems, the bottleneck is almost always small files and poor lifecycle management — not Iceberg itself.

The Hidden Cost Nobody Talks About

Iceberg introduces responsibility.

You now own compaction, snapshot expiration, retention, file size health, and cost observability across engines. Warehouses hide these concerns from you. Iceberg exposes them.

This is exactly why governance and FinOps around Iceberg are becoming critical. Once data is shared across multiple engines, someone needs to own its lifecycle and economics.

I ran into these trade-offs directly while building Dativo Ingest, an open-source ingestion project built around Iceberg. Once Iceberg is your contract, you stop optimising for a single engine and start thinking in terms of table health, commits, and long-term operability.

When Iceberg Is the Wrong Choice

Iceberg is not a silver bullet.

If you only need BI on a small amount of data, don’t want to operate data infrastructure, or have no Spark or Trino experience, a warehouse alone is still a perfectly reasonable choice.

Iceberg pays off when flexibility, scale, and long-term economics matter.

Final Thought

Iceberg isn’t popular because it’s fashionable.

It’s popular because data teams are tired of rebuilding the same foundations in every warehouse.

Iceberg gives you a durable data layer, engine independence, and predictable long-term economics. And once teams adopt it, they rarely go back.

If you’re designing a data platform today, Iceberg should at least be in the conversation — even if Snowflake or Databricks still sit on top.

That’s the real shift we’re seeing.

TL;DR:

• AI coding assistants can feel like magic… until they go off the rails and rewrite your repo just for a sake of having some unit test passing.

• My AI pair programmer started refactoring my project unprompted and hallucinating frameworks – chaos ensued.

• The solution was writing a .cursor/rules file: essentially an AI rulebook to enforce the coding standards and stop the madness.

• With a custom rulebook in place, the Coding Assistant became a helpful( finger crossed) teammate again – smaller diffs, relevant suggestions, and far fewer “WTF?” moments in code review.

• AI assistants need onboarding and culture too; we can’t just unleash them without guidance. Here’s how a few YAML guidelines turned my AI from rogue to rockstar.

The Honeymoon Phase (AI Feels Magical)

Guilty: the first few times with an AI pair programmer my pet project ( check it out - https://github.com/dativo-io/dativo-ingest , the headless data ingestion platform) felt like living in the “Ghost in the shell”. I’d type a command, and poof! the solution appeared as if my Cursor had a mind of the senior engineer. Functions that used to take an afternoon to debug practically wrote themselves while I sipped coffee. I was giddy with power – code reviews came back with fewer nitpicks, and I was closing tickets faster than ever. It was the honeymoon phase, when the AI could do no wrong( I prefer to keep my eyes closed). At first, coding with AI felt like unlocking cheat mode on reality. I bragged to my friends that I had a tireless junior dev who never sleeps and never complains, ready to do the grunt work at 3 AM. What could possibly go wrong?

The Hangover (When the AI Goes Rogue)

One night, I asked the AI to add a single field to the some class. The PR? 38 files changed. It rewrote our CLI, abstracted config loaders, and migrated connectors to a fictional framework. It was confident. It was wrong. I went from AI-enhanced to AI-endangered.

Another real example: before I wrote .cursor/rules, the AI submitted a commit titled “Major CLI Refactoring” and wasn’t kidding. It blew up our cli.py into cli_commands.py, startup.py, job_executor.py, connectors/factory.py, and more – 9 files touched, 2,000+ LOC rewritten. All this… when I just wanted a new job flag. LOL(

In another masterpiece, I asked for incremental sync support. The AI replied with a “Unified Incremental Strategy Framework” – complete with file sync, cursor DB sync, Airbyte catalog support, state JSONs, and a cloud deployment story. Impressive. Also: completely unified overkill.

I definetly needed to tame the beast.

Writing The Rulebook (What .cursor/rules Is and How It Works)

Enter .cursor/rules, a Markdown YAML-adjacent file I dropped into our repo. Cursor reads it. It learns(at least claims so). It stops trying to redesign the system every time I blink.

I tried to define law like:

# .cursor/rules
- “Every patch MUST reduce entropy.”
- “Implement only what is requested and stop once acceptance criteria are met.”
- “No refactoring outside scope.”
- “No new platforms, frameworks, or reports.”
- “Use YAML config over hardcoded logic.”

Basically, It’s policy-as-code for your AI coworker.

How LLMs changed the behaviour

Before: The Chaos Commits

• “Connector Registry V2” introduced external catalog loading, CLI support, schema changes, doc rewrites – all in one commit.

• Schema validation? Ignored. New YAML fields like streams_default showed up unannounced, breaking CI.

• Env vars as feature toggles? Oh yeah. I found ENABLE_FANCY_MODE=true buried in a helper.

After .cursor/rules: The Calm

• Mimesis connector PR: 5 files, clean logic, test included, YAML-configured, schema-valid – no chaos.

• Metrics Export: Added metrics.py, config toggles in runner.yaml, schema updated, validation green.

• Schema Guardrails: When adding external_id, the AI updated connectors.schema.json in the same commit – no CI surprises.

The AI learned. No more feature-farming. Just real hardcore engineering.

Before vs After: Diff Deltas

# Before `cursor/rules`
Commit: “Unified Incremental Sync”
Files changed: 15
Includes: new abstractions, plugin system, state management

# After `.cursor/rules`
Commit: “Add external_id to connector schema”
Files changed: 2
Includes: YAML field + validation schema

This shift was no accident – it was enforcement.

Additional Advice from .cursor/rules and the Field

Here are more rules and tips pulled from the real-world dativo-ingest’s rules file and other engineers who’ve fought the same fight:

✅ Be Patch-Scoped

“A patch MUST only satisfy the requested acceptance criteria. All other behavior is out of scope.”

Avoid change creep. Don’t sneak in that “quick cleanup” or bonus refactor.

✅ Respect the Interface

“Never change interfaces or filenames unless explicitly asked.”

Humans memorize file paths and function names. AI must not rename them casually.

✅ Just Enough Testing

“Do not write tests for unchanged or unrelated components.”

No 400-line test diffs when one new function was added.

✅ Always Validate Config

“New config fields MUST be described in *.schema.json and tested against examples.”

YAML-only is sacred. Forgetting the schema? Expect the wrath of CI.

✅ Prefer Simplicity

Inspired by Addy Osmani’s 70% Rule: “AI gets you 70% of the way fast, but that last 30% is where bugs hide.”

Elevate

The 70% problem: Hard truths about AI-assisted coding

After spending the last few years embedded in AI-assisted development, I've noticed a fascinating pattern. While engineers report being dramatically more productive with AI, the actual software we use daily doesn’t seem like it’s getting noticeably better. What's going on here…

Read more

2 years ago · 1514 likes · 75 comments · Addy Osmani

AI should avoid cleverness. If a simpler solution exists, pick it.

✅ Don’t Narrate, Ship

“Avoid over-documenting implementation details or rationale unless explicitly required.”

AI-generated README essays? Save it. We want user-facing docs, not ChatGPT’s inner monologue.

The Bigger Picture

Now, every AI task respects my GitOps-first, config-driven architecture. New flags go in runner.yaml, not os.getenv. Every schema change comes with a schema update and test. Our platform evolved. Our AI did too.

.cursor/rules isn’t just instructions – it’s culture.

AI Engineers Need Culture Too

We wouldn’t onboard a junior dev by saying “just read the codebase.” We give them docs, a buddy, some rules. We didn’t do that for our AI… until it started acting like a junior dev with caffeine poisoning.

.cursor/rules became my AI onboarding playbook. It cut entropy, limited scope creep, and gave the AI the same values we hold. And when your AI shares your values? It becomes a real teammate.

Two thousand years ago, the Chinese statesman Guan Zhong faced a familiar dilemma: tax people directly and risk revolt, or lower taxes and starve the state. His solution was neither. He eliminated most direct taxes entirely and moved revenue into infrastructure—salt, iron, trade—making taxation predictable, indirect, and almost invisible.

Modern data ingestion is stuck in the same deadlock. Managed services impose a hidden tax through opaque pricing and resync shocks. Code-first pipelines impose an operational tax through human toil. As a engineering leader who has faced late-night firefighting and “data swamps,” I’ve realized that the era of fragmented “best-of-breed” tooling is concluding, replaced by integrated platforms that offer streamlined workflows at the cost of increasing vendor lock-in and financial unpredictability.I believe the industry shall follow the third path: shifting cost upstream into structure, specs, and contracts—so ingestion stops taxing people at all.

The Binary Deadlock: Fivetran, dbt, and the “Fusion” Shift

The October 2025 merger between Fivetran and dbt Labs created a unified big data ETL player. This signals a shift toward integrated stacks where ingestion and transformation are unified. However, this consolidation brings risks for all the data folks:

• The Licensing Divergence: While dbt Core remains Apache 2.0, the "future vision" for transformation lies in dbt Fusion. Written in Rust, Fusion is licensed under the Elastic License v2 (ELv2)—a "source-available" license that prohibits using it to provide managed services to others.

• The Innovation Tax: As dbt Core enters "maintenance mode" innovation is reserved for proprietary engines, forcing organizations into a "dangerously dependent" relationship with a single vendor.

GUI-First vs. Specification-First: The Hidden Operational Costs

Airbyte and Fivetran shine with slick UIs. But that magic comes at the cost of control. When engineering talent spends 30% to 40% of their time "firefighting"—fixing broken pipelines and resolving sync failures—the long-term health of the platform enters a state of terminal decline.

• Ingestion Debt: According to the 2024 Global Data Engineering Report, 64% of data leaders report that technical debt significantly limits their ability to achieve business goals.

• The “Fragility Loop”: Driven by pressure to deliver, engineers adopt brittle scripts that create a “data death cycle,” where every minor change requires manual oversight and eventually consumes 60% to 80% of the team’s capacity.

“Open-Source” Connectors: the limits and solution

Meltano and the Singer ecosystem offer a code-first illusion, but the reality is a maintenance nightmare of outdated taps. To solve the brittleness of ad-hoc fixing the single opensource provider, I advocate for Spec-Driven Data pipeline, which declares expected input and desired output. The connectors shall be serving as purely technical providers rather:

• Precision over Prompting: Spec-Driven Development should use formal, machine-readable specifications as a single source of truth for both input and output. This structured collaboration aims for 95% or higher accuracy in implementing specs on the first attempt.

• Spec-Driven Workflow: By moving intellectual effort “upstream” to the Specify and Plan phases, teams capture the “why” behind technical choices, preventing “intent-vs-implementation drift”. Of course the tehnical aspects of connectors and data movers are still present, but they are not so important and can be replaced with other providers if there is such need.

Data Contracts: Beyond “Bring First-Decide Later”

The traditional approach of ingesting raw data and transforming it later leads to data swamps. Data quality issues cost organizations an average of $12.9 million annually.

• Fail-Closed Validation: We declare machine-readable YAML contracts at the source. If a source system change violates the contract, the deployment is rejected—a “fail-closed” gate that prevents downstream pollution.

• Schema-as-Code: By versioning every asset's schema in Git and running strict validation at ingestion time, we would treat data movement with the same rigor as application code.

Asset-Centric Orchestration: Focus on “What,” Not “How”

Traditional orchestrators like Airflow are task-centric, focusing on workflow execution. We should utilize an asset-centric approach in order to keep an eye on the thing which is really important - “What” we are expecting to get as the result of data pipeline.

• Native Lineage: In an asset-centric model, the focus have be on the data products produced. Then lineage is captured automatically in a unified graph, making it easier to trace origin and transformation.

• One Job per one Asset: If we would enforce a one-asset-per-job design pattern, this will simplify retry logic and ensures that if one of fifty tables fails, we only re-run that specific job rather than retrying a massive, entangled pipeline.

The “Third Way”

After slogging through these limitations, I built dativo-ingest to test the hypothesis of breaking the binary deadlock.

• Headless & Config-Driven: No UI needed; pipelines are defined in YAML under GitOps, making deployments auditable.

• Lakehouse-Native: It writes Apache Iceberg tables directly and update metadata via Nessie, ensuring ACID semantics and propagation of FinOps tags directly into table properties.

• Metadata-Driven Production Readiness: I specifically designed it for high-scale environments like Databricks Lakeflow, generating production-ready code automatically from formal specs.

I am going to continue posting about my findings, but one more time to underline - Dativo Ingest isn’t just about connectors - they are expremely replacable in the YAML configurations. The ineventables there are constraints—like upfront schemas and tags—because they force the discipline required to escape the data death cycle and reclaim technical agency.