Why do AI coding agents fail silently? 120 of our own tasks did in 28 days
Most coding agents fail silently because nothing outside the agent is watching the run. It retries, or it quietly gives up. You find a task that never finished and a token bill. In the last 28 days, 120 coding agent tasks on our own repos ended in the dead letter queue, meaning the system gave up on them.
What does a silent failure look like in practice?
I run an autonomous pipeline of coding agents on my own repos. The numbers below come from its task log.
- The biggest group ran out of retries and never got the tests passing. That was 39 of the 120 tasks, and the 38 with a recorded cost came to $829.46 at API list price.
- 27 were an automatic fix that needed its own fix. $154.86 at list price, from the 24 that logged a cost.
- 25 couldn't be merged cleanly ($432.99 across the 24 with a cost on record).
- 9 had a single run blow past our per task spike limit. Those 9 alone were $535.85.
Why doesn't the agent just tell you it's stuck?
An agent's job is to keep going. A test fails, it edits the code, it runs the test again. From inside that loop every retry looks like progress. So it never stops to say it's stuck.
And the signal lands in the wrong place. Cost shows up on a provider dashboard days later. The dead task sits in a queue nobody reads. Nothing pages you.
Add up every failed task that logged a cost, 109 of them, and it's $2,722.04 at list price. We're on flat rate plans, so for us that's notional. On a metered key it's real money.
What's changed, and what hasn't?
In September we capped automatic fixes at one level, so a fix of a fix goes to a person.
A run over the spike limit already gets held for a person instead of retried. That guard is older, and it's what stopped those 9.
The retry failures and the merge failures? No fix shipped yet for either. Still working on those.
How do you make agent failures loud?
Something outside the agent has to count the calls. On the Claude side of our pipeline that's RelayPlane, an open source proxy I wrote that sits on the same machine as the agents.
Every job runs under a label, so its cost rolls up per run and per agent. We don't cap our runs in the proxy yet, the spike limit lives in the pipeline. If you do set a per run cap, a run that hits it gets a 429 and only that job stops.
It only sees what's routed through it. Our Codex calls aren't, so a Codex run can still fail quietly on us.
Count first
Whatever you use, count first. Add caps once you know what a normal run costs.