An internal alert that fired once per contact
An internal notification was arriving about four times per account. The condition that triggered it was true of the account. The workflow was enrolling the people on it.
Client anonymised — a managed IT services provider, delivered under another company. Company names, staff names and record identifiers are withheld or changed. Every figure below is real.
- Companies backfilled
- 73 companies stamped with the latch
- Eligible after enable
- 0 companies left eligible
- Same bug found again
- 2 workflows, same class, one session
- Records deleted
- None
A condition about the account, applied to people
Someone reported an internal congratulations email arriving repeatedly for the same account. Roughly four copies, each time.
The cause was a mismatch of levels. The condition that fires the alert is a property of the company — a milestone the account has reached. The workflow was enrolling contacts. Every contact on that account satisfied a condition that was never about them individually, so every contact enrolled, and every enrolment sent.
Four contacts, four emails. The count was not a bug in the sending; it was an accurate reflection of how many people were attached to the company.
The fix that should have worked
The standard remedy is a latch: keep the workflow at contact level, but stamp a marker on the parent company the first time it runs, and add an enrolment condition requiring that marker to be unset. First contact through stamps it, everyone else is excluded.
That was built, and it was reasoned about honestly at the time — with the race condition it depends on losing written down as a known risk rather than assumed away.
Three days later the race fired for real. A live account with three contacts sent three emails inside a single evaluation window. All three were evaluated concurrently, all three read the marker as unset because none of them had written it yet, and all three passed.
Why a latch cannot work here
The platform evaluates enrolment for many records at once. A latch assumes a sequence — check, write, next record checks and sees the write — and there is no such sequence to rely on. Nothing serialises N children entering the same window.
That makes the latch approach not merely fragile but structurally wrong. It appears to work whenever accounts happen to have one contact, or whenever contacts happen to be modified minutes apart, which is most of the time. It fails precisely on the busy accounts that matter.
The rule that came out of it is short enough to keep: if the truth lives on the account, enrol the account. The workflow was rebuilt at company level, where the condition already lived, and the duplication became structurally impossible rather than merely unlikely.
Then the same bug turned up again, an hour later
With the pattern named, a second reported issue — a false alert claiming a form had not been completed — was recognisable immediately. Same class: the truth lived on the deal, the workflow enrolled contacts.
It was rebuilt as a deal-level workflow, checking across all contacts associated with the deal through an association filter rather than enrolling each of them. Same bug, same fix, twice in one session, which is the usual sign that a pattern is worth writing down rather than a one-off.
That rebuild carried a guard the original lacked: a date condition preventing thirty-nine historical deals from replaying the moment it was switched on. Enabling a corrected workflow against a full history is its own outage.
Backfilled, then proved quiet
Seventy-three companies were stamped with the latch property as a backfill, each sourced in a defined order of preference — the account's own milestone date where it existed, then the run date, then the earliest contact-level stamp. The mapping was produced and audited before it was applied rather than generated on the fly.
Verification was the point of the exercise. After enabling, the number of companies still eligible was zero, and enrolments on the rebuilt workflow were exactly one — a deliberate live test, whose latch was then restored to its prior value. The second workflow correctly enrolled nobody, because it is not supposed to touch history.
Nothing was deleted at any stage, and three deletions requested during the same period were held rather than actioned, because each destroys data irreversibly and the exports that would make them safe did not exist yet.
Status
Complete. Both workflows rebuilt at the correct object level and verified quiet after enable; superseded versions turned off rather than deleted. Held deletions remain blocked on exports, by design.
Workflow forensics: finding the level a condition actually belongs to, and rebuilding so the failure mode is impossible rather than improbable.
HubSpot Engineering