An audit that learns from being wrong
Every time a finding gets corrected, that correction is the most valuable signal in the system. It is also the most dangerous thing to store, because it arrives wrapped in one client's data.
- The goal
- Make each audit better than the last, so that a category of wrong finding gets corrected once rather than reproduced for every client afterwards.
- What it was costing
- The learning store is shared across engagements. A correction captured carelessly carries one client's operational detail into the next client's audit — so the mechanism that makes the system smarter is also the one that could leak.
Built and operated by Pietro Sassanelli. Published with permission. No client portal, client name or portal identifier appears here or in the store described.
- Correction becomes
- A portal-agnostic improvement
- Store scope
- Global across engagements
- Redaction runs
- Before anything is written
- Promotion requires
- An evaluation that catches regression
The signal worth capturing
An automated CRM audit produces findings. Some are right, some are overstated, and some are confidently wrong in a way that is only obvious to somebody who knows the portal.
When a person corrects one, they are doing something more useful than fixing that finding. They are describing a class of mistake — a rule about how this platform actually behaves that the audit did not know. Discard it and the same wrong finding gets produced next month, for a different client, with the same confidence.
So corrections are captured deliberately rather than left in a chat log. The loop takes the correction and turns it into an improvement to how the audit reasons, not a patch to one report.
Why the output has to be portal-agnostic
A correction almost always arrives with specifics attached: this workflow, this property, this record, this portal.
Storing it in that form does two bad things. It produces a rule that only fires against one client's configuration, which is nearly useless. And it puts one client's operational detail into a store that will be read while working for another — which is not a hypothetical risk, it is the default behaviour of a shared knowledge base.
So the improvement generated from a correction is written as a platform behaviour rather than an incident: what this system does, under what conditions, and what the audit should therefore check. The specifics are what taught it; they are not what gets kept.
Redaction as a boundary, not a formatting step
Because the store is global, redaction is the mechanism that keeps engagements separate. That makes it a security boundary rather than a tidiness feature, and it belongs in the code path that writes, not in an instruction asking a model to be careful.
It runs before anything is persisted. Credentials never reach the model at all; identifiers and client-specific values are stripped on the way into the store. The rule that survives has to be true without them — and if it is not true without them, that is a strong signal it was an incident rather than a lesson.
Nothing is promoted without a test
The obvious failure mode of a self-improving system is that it improves itself into confident nonsense. A rule learned from one ambiguous case, promoted automatically, is indistinguishable from a rule learned from twenty clear ones once it is in the file.
So a new rule enters as a candidate. It is not loaded into the default operating context and it is not published until an evaluation exists that would fail if the behaviour regressed. Writing that evaluation is the actual gate: it forces someone to state precisely what the rule claims and what evidence would falsify it.
Rules that cannot be expressed that way do not get promoted, which is the intended outcome. A knowledge base with a test behind every entry stays useful; one without becomes folklore that nobody dares delete.
What this is really solving
The reason this matters is that audit quality compounds in whichever direction you point it. An audit that never learns produces the same category of wrong finding indefinitely. One that learns carelessly gets more confident about the wrong things and leaks between clients while doing it.
The loop is the narrow path between those: capture the correction, strip it to a platform truth, prove it with a test, and only then let it change how the next audit reasons.
What changed
- A correction becomes a platform-behaviour rule rather than a patch to one report, so the same class of wrong finding is not reproduced for the next client.
- Redaction runs in the write path rather than being delegated to the model, which makes separation between engagements a code boundary instead of an instruction.
- A rule that is only true with a specific client's identifiers attached fails the portability test, which is itself the signal that it was an incident and not a lesson.
- Nothing is promoted into the operating context until an evaluation exists that would fail if the behaviour regressed — so the knowledge base cannot quietly fill with confident folklore.
Where it stands
In production alongside the audit tooling. The redaction path is implemented in the write layer rather than delegated to the model, and candidate rules remain unpublished until an evaluation exists for them.
Questions people ask about this
How should an AI system learn from being corrected?
By converting the correction into a statement about how the platform behaves, not a patch to the report that was wrong. The specifics that taught you the lesson — this record, this workflow, this client — are what makes it unrepeatable, so the rule that gets kept has to be true without them.
What is the risk of a shared learning store across clients?
That one client's operational detail is read while working for another. It is not hypothetical; it is the default behaviour of a shared knowledge base. That makes redaction a security boundary rather than a formatting step, which means it belongs in the code that writes to the store and not in an instruction asking a model to be careful.
How do you stop a self-improving system learning the wrong thing?
Require an evaluation before a rule is promoted. A candidate rule is not loaded into the default context and is not published until a test exists that would fail if the behaviour regressed. Writing that test forces someone to state precisely what the rule claims and what would falsify it, and rules that cannot be expressed that way do not get promoted.
Why not just let the model remember corrections directly?
Because a rule learned once from an ambiguous case becomes indistinguishable from one learned twenty times from clear ones as soon as it is in the file. Without a test behind each entry, a knowledge base fills with confident folklore that nobody dares delete.
This is one engagement. The seven judgements that recur across all of them are what the work actually has in common.
Systems that learn from correction need a boundary and a test, or they compound their mistakes as fast as their improvements.
Operational AI Strategy