The problem
I inherited a scenario that polls Twilio hourly for completed calls, transcribes the recordings, classifies them, and distributes the results to a CRM, several spreadsheets and Slack. It had grown to 127 modules across nine levels of nesting, with 25 error handlers that discarded rather than reported.
It worked, in the sense that data arrived. It was also processing a significant proportion of calls twice.
Twilio reports a call as multiple legs — a parent call and its children. The scenario treated each leg as an independent call, so a single conversation could be picked up, transcribed and written more than once. Downstream, that meant double-counted calls in reporting, duplicate notifications, and a transcription bill for work already done.
The audit
Rather than patching the symptom, I read the whole scenario and wrote up what I found. Three classes of problem:
Duplicate leg processing. The scenario had no concept of a parent call, so child legs were processed as if they were separate conversations. This was the expensive one and the one visible in reporting.
Recording URL construction. The URLs being built for recording retrieval were malformed, so some recordings could not be fetched.
Unreachable router branches. Several branches could never be entered given the conditions upstream of them. They were not causing incorrect behaviour, but they made the scenario significantly harder to reason about, and anyone reading it would reasonably assume they did something.
I also flagged a race condition in the deduplication approach, discussed below.
The fixes
Deduplication now keys on the parent call, so a conversation is processed once regardless of how many legs Twilio reports. The recording URL construction was corrected, and the unreachable branches removed.
Result: transcription volume fell by approximately half, because every call had previously been examined twice. Reporting figures became accurate for the same reason.
What I recommended, and what happened
My recommendation was to rewrite the pipeline in Python. The case was that call ingestion is a queue-and-transform problem — deduplication, retry, idempotency, ordering — and those are things a general-purpose language handles with a few well-tested functions. In a visual tool each one becomes structure on a canvas, which is why the scenario had reached 127 modules and nine levels of nesting to do work that is not conceptually complicated.
The rewrite was declined. The repairs shipped instead, and they resolved the problems that were actually costing money.
I still think the rewrite is the right long-term call, and I also think shipping the fix that removed the cost immediately was the better decision at the time. Those are not in tension. The pipeline is now correct; it remains harder to change than it needs to be.
What I would change
The deduplication should be enforced by the database, not by the scenario. It currently works by selecting on the call identifier and inserting only when nothing is found. That is a check followed by a write, with nothing between them, and the scenario is not configured to run sequentially — so two overlapping executions can both check, both find nothing, and both insert.
This has not happened. At hourly polling with the observed execution durations, the window is narrow and may never be hit in practice. But the current guarantee is that a collision is unlikely rather than that it is impossible. A unique constraint on the call identifier would make the duplicate physically unwritable and demote the check to what it should be: an optimisation that avoids a pointless insert attempt, rather than the thing the correctness depends on. It is a one-line migration.
Twenty-five error handlers discard silently. Failures are caught and execution continues, which keeps the pipeline running but means a call that fails to transcribe leaves no trace anywhere. Routing those to a single failure log would cost very little and would replace an assumption that things are working with evidence that they are.
The polling window has no overlap handling. Each run asks Twilio for calls completed in the previous 59 minutes. A delayed execution, or a call that settles on the boundary, can fall between two windows. Widening the window and relying on deduplication to absorb the overlap is the standard fix, and it becomes safe to do once deduplication is enforced by a constraint.