An audit that learns from being wrong

The signal worth capturing

An automated CRM audit produces findings. Some are right, some are overstated, and some are confidently wrong in a way that is only obvious to somebody who knows the portal.

When a person corrects one, they are doing something more useful than fixing that finding. They are describing a class of mistake — a rule about how this platform actually behaves that the audit did not know. Discard it and the same wrong finding gets produced next month, for a different client, with the same confidence.

So corrections are captured deliberately rather than left in a chat log. The loop takes the correction and turns it into an improvement to how the audit reasons, not a patch to one report.

Why the output has to be portal-agnostic

A correction almost always arrives with specifics attached: this workflow, this property, this record, this portal.

Storing it in that form does two bad things. It produces a rule that only fires against one client's configuration, which is nearly useless. And it puts one client's operational detail into a store that will be read while working for another — which is not a hypothetical risk, it is the default behaviour of a shared knowledge base.

So the improvement generated from a correction is written as a platform behaviour rather than an incident: what this system does, under what conditions, and what the audit should therefore check. The specifics are what taught it; they are not what gets kept.

Redaction as a boundary, not a formatting step

Because the store is global, redaction is the mechanism that keeps engagements separate. That makes it a security boundary rather than a tidiness feature, and it belongs in the code path that writes, not in an instruction asking a model to be careful.

It runs before anything is persisted. Credentials never reach the model at all; identifiers and client-specific values are stripped on the way into the store. The rule that survives has to be true without them — and if it is not true without them, that is a strong signal it was an incident rather than a lesson.

Nothing is promoted without a test

The obvious failure mode of a self-improving system is that it improves itself into confident nonsense. A rule learned from one ambiguous case, promoted automatically, is indistinguishable from a rule learned from twenty clear ones once it is in the file.

So a new rule enters as a candidate. It is not loaded into the default operating context and it is not published until an evaluation exists that would fail if the behaviour regressed. Writing that evaluation is the actual gate: it forces someone to state precisely what the rule claims and what evidence would falsify it.

Rules that cannot be expressed that way do not get promoted, which is the intended outcome. A knowledge base with a test behind every entry stays useful; one without becomes folklore that nobody dares delete.

What this is really solving

The reason this matters is that audit quality compounds in whichever direction you point it. An audit that never learns produces the same category of wrong finding indefinitely. One that learns carelessly gets more confident about the wrong things and leaks between clients while doing it.

The loop is the narrow path between those: capture the correction, strip it to a platform truth, prove it with a test, and only then let it change how the next audit reasons.