Make your LLM API calls resilient in Python

If your app calls an LLM API (OpenAI, Anthropic, anything), you have already met its failure modes: 429 rate limits, occasional 500s, requests that hang, and the tail latency where one call in fifty takes ten seconds. In a demo you ignore this. In production it is most of your incidents.

Here is a compact way to handle all of it, using nopanic (pip install nopanic), a zero-dependency resilience toolkit. Everything below works the same on sync and async functions.

The problem, honestly

A naive call looks like this:

async def ask(prompt):
    return await client.chat.completions.create(...)

Every failure mode above takes down the request. Worse, the naive fix (wrap it in a retry loop) can make an outage worse: if the provider is struggling and everyone retries immediately, the retries pile on.

Retry, but politely

from nopanic import retry, backoff

@retry(attempts=4, on=RateLimitError,
       backoff=backoff.full_jitter(base=0.5, cap=30.0))
async def ask(prompt): ...

Full jitter spreads the retries out randomly so they do not arrive in a synchronized wave. If the error carries a Retry-After hint, nopanic honors it (capped, so a hostile value cannot park your client).

Stop hammering a provider that is down

from nopanic import circuit_breaker

llm = circuit_breaker(failure_threshold=0.5, reset_timeout=20.0)

@retry(...)
@llm
async def ask(prompt): ...

Once half the calls in the window fail, the breaker opens and fails fast for 20 seconds instead of sending doomed requests. Then it lets one probe through to check if the provider recovered.

Bound every attempt, and degrade instead of crashing

from nopanic import compose, fallback, timeout, CircuitOpen

resilient = compose(
    fallback(lambda e: "Sorry, please try again shortly.",
             on=(RateLimitError, CircuitOpen, TimeoutError)),
    retry(attempts=3, on=RateLimitError, backoff=backoff.full_jitter(0.5)),
    llm,
    timeout(30.0),
)

@resilient
async def ask(prompt): ...

Read it from the inside out: each attempt gets 30 seconds, outcomes feed the breaker, rate limits retry with jitter, and anything still unhandled becomes a graceful message instead of a stack trace reaching your user.

Fix the tail, not just the median

For idempotent reads, if a call is slow, race a second one:

from nopanic import hedge

@hedge(delay=0.8)
async def embed(text): ...

If the first attempt has not answered in 800 ms, a duplicate is sent and whichever returns first wins. This trades a little extra load for a much better p99.

The point

None of these patterns are new. What is annoying is wiring them together from three different libraries. Putting the whole stack in one place, with one API that reads top to bottom, is the entire idea. Full docs and source: github.com/dagdelenemre/nopanic