A circuit breaker bug that turned 3.2M calls into 40 minutes

I maintain nopanic, a small resilience toolkit for Python (retries, circuit breakers, timeouts, rate limits). Everything passed its tests, so I did what I should have done earlier: I put it under real load. The result taught me a lesson worth writing down.

The setup

A circuit breaker watches the failure rate of a dependency. If too many calls fail inside a time window, it “opens” and fails fast instead of hammering something that is already down. To know the failure rate, it has to remember recent outcomes.

My first implementation remembered them the obvious way: a list of (timestamp, ok/fail) records inside a sliding time window. On every failure it counted how many entries in that list were failures, divided by the total, and compared to the threshold.

Correct. All tests green. Shipped.

The load test

I wrote a stress test: 32 threads, 100,000 calls each, one shared breaker, about a third of the calls failing (below the trip threshold, so the breaker stays closed and keeps recording). 3.2 million calls total.

It took 2,413 seconds. Forty minutes. Single threaded, the breaker does roughly 850,000 calls per second, so those calls should have finished in about four seconds.

The diagnosis

Two problems, one root cause.

First, memory. The window kept one record per call. At high throughput that list holds every call inside the window. I measured it: one million calls in the window was about 88 MB, for a single breaker. It grows with traffic, without bound.

Second, and worse, time. On every failure the code recomputed the failure count by scanning the entire list. As the list grew, each scan grew with it. N failures, each doing O(N) work, is O(N squared). Under sustained partial failure the whole thing collapses into a quadratic crawl, and because the scan happens inside the breaker's lock, all the threads pile up behind it. That is the forty minutes.

The realization

The fix was not a faster scan. It was noticing that a circuit breaker's memory is not a log. It only needs to answer one question: what is the failure rate right now. It never needs the individual records. I was storing a detailed history to compute a single ratio.

Detailed per-request history is a real need, but it belongs in an observability stream that the user forwards to their own logging, not in the breaker's hot path. Keeping millions of records in RAM to compute one percentage was the actual bug.

The fix

Replace the per-call list with a fixed set of time buckets, each holding two integers: how many calls, how many failures. Keep running totals updated as buckets rotate out of the window. This is how Hystrix and resilience4j do it, and now I understand why.

• Recording an outcome: O(1).
• Reading the failure rate: O(1), from the running totals.
• Memory: constant, no matter the traffic. Ten buckets of two integers.

The same 3.2 million call load now finishes in about 2.4 seconds instead of 40 minutes. One million in-window calls use 0.001 MB instead of 88. The per-call cost on the success path even dropped slightly.

The trade-off: outcomes now expire in bucket-sized steps instead of at an exact per-call age. For deciding whether to trip a breaker, that precision was never worth its price.

Two lessons

First, tests that check correctness do not check behavior under load. My suite was green the entire time the library was quadratic. A partially failing dependency at high traffic, the exact situation a circuit breaker exists for, was the situation that broke it.

Second, a bonus one I tripped over while fixing this: I was testing with an editable install, and a stale copy of the package in site-packages was shadowing my source. My tests were passing against old code. Now the CI also builds the wheel, installs it into a clean environment, and runs the suite against the installed artifact. Never fully trust pip install -e alone.

The library is nopanic (pip install nopanic) if you want to see the code. Source: github.com/dagdelenemre/nopanic