At 2am, with a provider down and an agent looping, nobody invents a good fix from scratch. AI design patterns are the reusable answers to the failures every production AI system eventually hits: dead providers, ballooning bills, models that quietly rot while dashboards stay green. This catalog puts them side by side, each with the problem it solves, the structure, a runnable sketch, and when not to use it.

“If the implementation is hard to explain, it’s a bad idea.”

Tim Peters, The Zen of Python (PEP 20)

Last Updated: July 2026 | Tested on: Python 3.14.6, stdlib only | Difficulty: Advanced | Reading Time: 23 minutes

Think of it like the building code an architect follows. Nobody re-derives from scratch how to keep a staircase from collapsing; there is a tested spec for that, and you adapt it to your building. Production AI has the same kind of shared spec. The tools change every few months, but the patterns underneath, retry when a call flakes, escalate only when a cheap answer is not good enough, cache what repeats, cap every loop, those outlive whichever provider or framework is fashionable this year. That is why this post teaches patterns first and names tools only as current examples.

How to Pick a Pattern

There are dozens of AI design patterns, and reading them all before you have a problem is the fastest way to over-engineer a weekend project. So flip the order. Do not ask “which pattern is best”, ask “which failure do you fear most”, and let the fear point you at the pattern. Scared the provider will go down? That is the reliability column. Scared of the bill or the latency? That is the cost column. Scared the answers are weak or the agent runs wild? That is the orchestration column. The picker below is the whole post on one page.

Tip: click the diagram to open it full screen, then use the zoom and pan controls for a closer look.

Read it top-down. Each column is one fear, and each step down the column is a pattern that goes a little further than the one above it. You almost never need the whole column at once. A small internal tool might stop at “retry with backoff” and never build a fallback chain, while a customer-facing service walks all the way to the bottom. The rest of the post is these three columns, one at a time, with code you can run for the load-bearing ones and an honest note on when each pattern is the wrong call.

Reliability Patterns

Every call to a model provider is a network call, and network calls fail. The reliability patterns are a ladder. Retry with backoff handles a call that flakes once: try again, waiting a little longer each time so you do not stampede a struggling server. Timeout caps how long you wait so one slow call cannot freeze the whole request. Circuit breaker notices when a provider is down hard and stops calling it for a while, instead of burning your retry budget on something that will not answer.

And when a provider is truly gone, the fallback chain keeps a user experience alive: try provider A, then provider B, then a cached answer, and only if all of that fails, refuse honestly instead of throwing a stack trace at the user.

Here is the fallback chain, with retry and backoff built in, running against a simulated outage where both live providers are down. Picture a support assistant an engineer named Aditi runs for a food-delivery app: even during a provider incident, it should say something sensible.

📄 fallback_chain.py: provider A to B to cache to graceful refusal

# Reliability pattern: provider A -> provider B -> cached answer -> graceful refusal.
# We simulate a full outage of BOTH live providers so the chain has to degrade
# gracefully instead of throwing an error in a user's face.
import time

class Outage(Exception):
    pass

# --- simulated providers (stand in for real LLM API calls) ---
# In production these call an SDK; here they are deterministic stubs so the
# output is reproducible. Flip these to True to see the chain stop earlier.
PRIMARY_UP = False   # pretend the primary provider is having an incident
BACKUP_UP = False    # and its backup is caught in the same regional outage

def primary_llm(prompt):
    if not PRIMARY_UP:
        raise Outage("primary: 503 service unavailable")
    return f"[primary] answer to: {prompt}"

def backup_llm(prompt):
    if not BACKUP_UP:
        raise Outage("backup: 503 service unavailable")
    return f"[backup] answer to: {prompt}"

# a tiny cache of previously good answers, keyed by the exact prompt
ANSWER_CACHE = {
    "What are your hours?": "[cached] We are open 9am to 9pm, every day.",
}

def retry(fn, prompt, attempts=2, base_delay=0.2):
    """Retry with exponential backoff. Small delays keep the demo fast."""
    for i in range(attempts):
        try:
            return fn(prompt)
        except Outage as e:
            wait = base_delay * (2 ** i)
            print(f"    {fn.__name__} attempt {i + 1} failed ({e}); waiting {wait:.1f}s")
            time.sleep(wait)
    raise Outage(f"{fn.__name__}: exhausted {attempts} attempts")

def answer(prompt):
    # 1) primary provider, with retry + backoff
    try:
        return retry(primary_llm, prompt)
    except Outage as e:
        print(f"  step 1 primary down: {e}")
    # 2) backup provider
    try:
        return retry(backup_llm, prompt)
    except Outage as e:
        print(f"  step 2 backup down: {e}")
    # 3) last known good answer from cache, if we have one
    if prompt in ANSWER_CACHE:
        print("  step 3 serving last known good answer from cache")
        return ANSWER_CACHE[prompt]
    # 4) graceful refusal: honest, never a stack trace
    print("  step 4 nothing left, refusing gracefully")
    return "Sorry, I cannot answer right now. Please try again in a few minutes."

for q in ["What are your hours?", "Do you deliver to Pune?"]:
    print(f"Q: {q}")
    print(f"A: {answer(q)}\n")

▶ Output

Q: What are your hours?
    primary_llm attempt 1 failed (primary: 503 service unavailable); waiting 0.2s
    primary_llm attempt 2 failed (primary: 503 service unavailable); waiting 0.4s
  step 1 primary down: primary_llm: exhausted 2 attempts
    backup_llm attempt 1 failed (backup: 503 service unavailable); waiting 0.2s
    backup_llm attempt 2 failed (backup: 503 service unavailable); waiting 0.4s
  step 2 backup down: backup_llm: exhausted 2 attempts
  step 3 serving last known good answer from cache
A: [cached] We are open 9am to 9pm, every day.

Q: Do you deliver to Pune?
    primary_llm attempt 1 failed (primary: 503 service unavailable); waiting 0.2s
    primary_llm attempt 2 failed (primary: 503 service unavailable); waiting 0.4s
  step 1 primary down: primary_llm: exhausted 2 attempts
    backup_llm attempt 1 failed (backup: 503 service unavailable); waiting 0.2s
    backup_llm attempt 2 failed (backup: 503 service unavailable); waiting 0.4s
  step 2 backup down: backup_llm: exhausted 2 attempts
  step 4 nothing left, refusing gracefully
A: Sorry, I cannot answer right now. Please try again in a few minutes.

What happened here: both providers were down, and the two questions still ended very differently. The first one, “What are your hours?”, had a cached answer, so after primary and backup both failed their retries, step 3 served the last known good reply and the user never saw the incident. The second question had no cache entry, so the chain walked all the way to step 4 and refused honestly instead of crashing.

That is the whole point of the pattern: the user gets the best answer still reachable, and the worst case is a polite “try again”, never a 500 error. When not to use it: a fallback chain adds real complexity, so a background job that can just fail and be retried later does not need one. Save it for the request path a human is actually waiting on.

Cost and Latency Patterns

Once a system is reliable, the next pain is the bill. The router is the highest-impact pattern in this whole post: send each request to a cheap, fast model first, and only escalate to the expensive one when the cheap answer is not confident enough. Most real traffic is easy (“what are your hours?”), so most of it never needs the frontier model at all. This is the payoff of the cost work from the Large Language Model (LLM) cost optimization post, measured on a small batch below.

📄 router.py: cheap model first, escalate only when unsure

# Cost pattern: try a cheap, fast model first. Only escalate to the expensive
# model when the cheap one is not confident enough.
# The dollar figures are illustrative per-request costs (small vs frontier model).
CHEAP_COST = 0.0002     # a small, fast model
EXPENSIVE_COST = 0.006  # a frontier model, roughly 30x pricier
CONFIDENCE_BAR = 0.75

# A stand-in "cheap model" returns (answer, self-reported confidence).
# Simple factual questions score high; open-ended or comparative ones score low.
def cheap_model(q):
    hard = any(w in q.lower() for w in ("compare", "why", "explain", "design"))
    conf = 0.55 if hard else 0.92
    return f"[small] {q}", conf

def expensive_model(q):
    return f"[large] {q}", 0.98

QUERIES = [
    "What are your opening hours?",
    "Is paneer tikka on the menu?",
    "Compare paneer tikka and paneer bhurji for a party",
    "What is the price of veg biryani?",
    "Explain why my order was delayed",
    "Do you have filter coffee?",
]

def route(q):
    ans, conf = cheap_model(q)
    if conf >= CONFIDENCE_BAR:
        return ans, CHEAP_COST, "cheap"
    ans, conf = expensive_model(q)   # not sure enough, escalate
    return ans, EXPENSIVE_COST, "escalated"

total = 0.0
escalations = 0
for q in QUERIES:
    _, cost, tier = route(q)
    total += cost
    escalations += (tier == "escalated")
    print(f"  {tier:9s} ${cost:.4f}  {q}")

baseline = len(QUERIES) * EXPENSIVE_COST   # naive: send everything to the big model
print(f"\nqueries                : {len(QUERIES)}")
print(f"escalated to big model : {escalations}/{len(QUERIES)}")
print(f"router total cost      : ${total:.4f}")
print(f"all-expensive cost     : ${baseline:.4f}")
print(f"savings                : ${baseline - total:.4f} = {(1 - total / baseline) * 100:.0f}%")

▶ Output

  cheap     $0.0002  What are your opening hours?
  cheap     $0.0002  Is paneer tikka on the menu?
  escalated $0.0060  Compare paneer tikka and paneer bhurji for a party
  cheap     $0.0002  What is the price of veg biryani?
  escalated $0.0060  Explain why my order was delayed
  cheap     $0.0002  Do you have filter coffee?

queries                : 6
escalated to big model : 2/6
router total cost      : $0.0128
all-expensive cost     : $0.0360
savings                : $0.0232 = 64%

What happened here: four of the six questions were simple, so the cheap model answered them with high confidence and the router never touched the expensive model. Only the two open-ended ones (“compare…” and “explain why…”) escalated. The result is a 64 percent cut in spend on this batch, and on real traffic where the easy-to-hard ratio is even more lopsided, the savings are usually larger. The one thing that makes or breaks a router is the confidence signal: here it is a stub, but in production you use the model’s own log-probabilities, a cheap classifier, or a self-check.

When not to use it: if every request genuinely needs your best model (say, medical or legal drafting), a router just adds a hop and a failure mode. Route only when a real slice of traffic is easy.

The router cuts cost per request. Cache-aside cuts it to zero for anything you have already answered. Before calling the model, hash the input and check a cache; on a hit, skip the call entirely. The same handful of questions (“what are your hours?”, “do you deliver?”) show up over and over, and paying for each one twice is money set on fire. In production the cache is usually Redis (current at the time of writing, with Memcached and Valkey as common alternatives); the demo below uses a plain dict with the exact same read-through logic.

📄 cache_aside.py: skip the call for anything you have seen before

# Cost + latency pattern: cache-aside. Check a cache keyed by a hash of the
# input before calling the model. On a hit you skip the call entirely. In
# production the cache is Redis; for a runnable demo we use a plain dict.
import hashlib

_store = {}                  # stands in for Redis
CALL_COST = 0.006            # dollars per real model call
CALL_LATENCY_MS = 800        # a real completion is slow
calls_made = 0

def _key(text):
    return hashlib.sha256(text.encode()).hexdigest()[:16]

def embed(text):
    """Cache-aside: return the cached value, or compute it, store it, return it."""
    global calls_made
    k = _key(text)
    if k in _store:
        return _store[k], 0.0, 0.0            # hit: no cost, no latency
    calls_made += 1
    # (a real call to the provider would happen here)
    value = f"vector({k})"
    _store[k] = value
    return value, CALL_COST, CALL_LATENCY_MS

# A realistic request stream: the same few FAQs repeat a lot.
stream = [
    "What are your hours?", "Do you deliver?", "What are your hours?",
    "Is parking free?", "What are your hours?", "Do you deliver?",
    "Do you deliver?", "Is parking free?", "What are your hours?", "Do you deliver?",
]

cost = latency = 0.0
hits = 0
for text in stream:
    _, c, ms = embed(text)
    cost += c
    latency += ms
    hits += (c == 0.0)

n = len(stream)
print(f"requests             : {n}")
print(f"unique inputs        : {len(_store)}")
print(f"cache hits           : {hits}/{n} = {hits / n:.0%}")
print(f"real model calls     : {calls_made}")
print(f"cost with cache      : ${cost:.4f}")
print(f"cost without cache   : ${n * CALL_COST:.4f}")
print(f"latency with cache   : {latency:.0f} ms of real calls")
print(f"latency without cache: {n * CALL_LATENCY_MS} ms of real calls")

▶ Output

requests             : 10
unique inputs        : 3
cache hits           : 7/10 = 70%
real model calls     : 3
cost with cache      : $0.0180
cost without cache   : $0.0600
latency with cache   : 2400 ms of real calls
latency without cache: 8000 ms of real calls

What happened here: ten requests, but only three unique inputs, so seven of them were free cache hits. Cost dropped from six cents to under two, and the slow real calls dropped from eight to three. The same read-through logic works for embeddings and for completions; you just key on the input text (or a normalized version of it). The third cost pattern, batch and queue, is the offline cousin: for work no user is waiting on, like nightly re-embedding of a document set, push jobs onto a queue and process them in bulk where per-call overhead disappears.

When not to use it: do not cache anything personalized or time-sensitive without a short expiry, or Aditi will get last week’s delivery estimate served as fresh.

ML-System Patterns

The patterns so far live at the request level. The ML-system patterns are bigger, structural habits that keep a whole model lifecycle sane. They are less about a clever function and more about where things live and how they flow, so this section is a catalog rather than a script. An engineer named Anvay keeps this table pinned above his desk.

The through-line is the same idea in four costumes: separate the thing that changes (data, model version, features) from the thing that stays stable (the pipeline, the serving code), and put a versioned boundary between them. Shadow deployment is the one people skip and regret, because it is the only pattern here that catches a model getting quietly worse on real inputs before your users are the ones who notice. It pairs naturally with the drift monitoring from the deployment at scale post.

Agent Orchestration Patterns

When one prompt to one model cannot do the job, you reach for multiple coordinated steps, and there are four shapes worth knowing. Supervisor: one lead agent breaks a task into pieces and hands each to a specialist worker, then assembles the results, like a head chef directing line cooks. Swarm: peer agents pass control to whichever one is most relevant, with no central boss, useful when the next best step is not known up front.

Plan-execute: one pass writes a plan, a second pass executes it step by step, which keeps a long task from drifting because the plan is fixed before any tool runs. Evaluator-optimizer: a generator drafts, an evaluator grades against a rubric, and the generator revises until it passes or hits a hard cap. That last one is the most broadly useful, and it is short enough to run.

📄 evaluator_optimizer.py: draft, grade, revise, with a hard cap

# Agent orchestration pattern: the evaluator-optimizer loop. A generator drafts
# an answer, an evaluator scores it against a rubric, and if it falls short the
# generator revises. The loop always has a HARD cap on rounds, so it can never
# spin forever the way an unbounded agent can.
MAX_ROUNDS = 5
PASS_SCORE = 1.0

# The rubric: a good order confirmation must cover all four of these.
REQUIRED = ["greeting", "item price", "delivery time", "thank-you"]

def evaluate(covered):
    """Score the draft and hand back the single most important missing piece."""
    have = [r for r in REQUIRED if r in covered]
    missing = [r for r in REQUIRED if r not in covered]
    score = len(have) / len(REQUIRED)
    return score, (missing[0] if missing else None)

def optimize():
    covered = set()   # what the current draft already includes
    for round_no in range(1, MAX_ROUNDS + 1):
        score, feedback = evaluate(covered)
        print(f"  round {round_no}: score={score:.2f}  covers={sorted(covered)}")
        if score >= PASS_SCORE:
            print(f"  passed the rubric on round {round_no}")
            return
        print(f"           evaluator asks for: {feedback!r}")
        covered.add(feedback)   # the generator revises to add that one piece
    print(f"  hit the {MAX_ROUNDS}-round cap, shipping best effort")

optimize()

▶ Output

  round 1: score=0.00  covers=[]
           evaluator asks for: 'greeting'
  round 2: score=0.25  covers=['greeting']
           evaluator asks for: 'item price'
  round 3: score=0.50  covers=['greeting', 'item price']
           evaluator asks for: 'delivery time'
  round 4: score=0.75  covers=['delivery time', 'greeting', 'item price']
           evaluator asks for: 'thank-you'
  round 5: score=1.00  covers=['delivery time', 'greeting', 'item price', 'thank-you']
  passed the rubric on round 5

What happened here: the generator started with nothing, the evaluator named the single most important gap each round, and the draft climbed from 0 to a full pass in five rounds. The load-bearing detail is the MAX_ROUNDS cap. Even if the draft never satisfied the rubric, the loop would stop after five rounds and ship its best effort instead of spinning forever and draining a token budget. In a real system the generator and evaluator are model calls (often the same evaluator you built in the agent evaluation post), but the control flow is exactly this.

When not to use it: if a single well-prompted call already passes the rubric, do not add a loop. The pattern earns its cost only when first drafts genuinely need revision.

Anti-Patterns to Avoid

Patterns tell you what to reach for. Anti-patterns are the ones that look smart and cost you later. Agent-for-everything: wrapping a task in a multi-step agent when a single function call or one prompt would do. An agent adds latency, cost, and a dozen new failure modes; use one only when the task genuinely needs planning and tools. Premature fine-tuning: paying to fine-tune a model before you have tried good prompting and retrieval.

Fine-tuning is slow to iterate and freezes knowledge at training time, so most teams who reach for it early would have gotten further with a better prompt and a few examples. Unbounded loops: an agent or retry with no cap on steps or attempts, which is how a stuck system quietly runs up a four-figure bill overnight. Every loop in this post has a hard ceiling for exactly this reason.

Common Mistakes

Conclusion

You now have a catalog of AI design patterns you can actually reach for under pressure. Reliability patterns keep a user’s experience alive when a provider dies: retry with backoff, timeout, circuit breaker, and a fallback chain that walks from provider A to B to cache to an honest refusal. Cost patterns keep the bill sane: a router that escalates only when a cheap answer is not good enough, cache-aside that makes repeat work free, and batching for offline jobs. ML-system patterns keep the model lifecycle reproducible, and agent patterns keep multi-step work coordinated and, above all, bounded.

The tools underneath will churn, but pick by the failure you fear most and these patterns will still be the right answer years from now.

Want the bigger picture? Browse the complete Python + AI/ML tutorial series home to see where these patterns fit among every other lesson, from your first script all the way to production AI systems.

Frequently Asked Questions

Interview Questions on AI Design Patterns

These come from real screens and onsites. Practice answering before you read each answer.

Q: Why prefer a fallback chain over a single try-except when calling a model provider?

A single try-except can only turn a failure into an error message. A fallback chain turns it into the best answer still reachable: try the primary provider, then a backup provider, then a cached answer, and only refuse if all of those fail. During a real outage that is the difference between users seeing a polite “try again in a few minutes” and users seeing a 500 error. The key discipline is to still emit a metric on every fallback, so a silent degradation does not hide the fact that your primary is down.

Q: How does a model router cut cost, and what is the one thing that makes or breaks it?

A router sends each request to a cheap model first and escalates to the expensive one only when the cheap answer is not confident enough. Since most real traffic is easy, most requests never reach the frontier model, which is where the savings come from. The make-or-break is the confidence signal you escalate on. If it is accurate, you route hard questions up and keep easy ones cheap; if it is noisy, you either escalate everything, losing the savings, or escalate nothing, hurting quality. In production you base it on log-probabilities, a small classifier, or a self-check, not a keyword guess.

Q: What is the difference between the router and cache-aside, and can you use both?

They attack cost from different angles. The router lowers the cost of answering a new question by picking the cheapest model that can handle it. Cache-aside lowers the cost of answering a repeat question to zero by skipping the call entirely on a cache hit. They compose cleanly: check the cache first, and on a miss, run the router to answer as cheaply as possible, then store the result. A production system almost always wants both, because they cover repeat traffic and novel-but-easy traffic respectively.

Q: What is shadow deployment and what does it catch that a normal rollout does not?

Shadow deployment runs a new model on live traffic in parallel with the current one, logging the new model’s answers without showing them to users. It catches quiet quality decay: a new model that looks fine on your offline eval but behaves worse on the real, messier distribution of live inputs. Because you compare the two side by side on identical traffic before promoting, you find the regression yourself instead of hearing about it from users. The cost is running two models, and it does not fit cases where the new model has side effects you cannot safely double up.

Q: Compare the supervisor, plan-execute, and evaluator-optimizer agent patterns.

Supervisor is about division of labor: one lead agent splits a task among specialist workers and assembles the results, good when the subtasks are distinct. Plan-execute is about staying on track: one pass writes the whole plan and a second executes it step by step, good for long tasks that tend to drift because the plan is fixed before any tool runs. Evaluator-optimizer is about quality: a generator drafts, an evaluator grades against a rubric, and the generator revises until it passes or hits a cap, good when first drafts genuinely need refinement. All three must bound their steps so a stuck run cannot spin forever.

Q: Scenario: your AI feature’s cloud bill tripled overnight with no traffic increase. Which patterns and anti-patterns do you check first?

A flat-traffic bill spike almost always means something is looping or escalating when it should not. First I would look for an unbounded loop: an agent or retry with no step cap that got stuck on a class of inputs and is spinning. Next I would check the router’s escalation rate, because a change to the confidence threshold or the cheap model can quietly send everything to the expensive model. Then the cache hit rate, since a key change or an expiry bug can turn free hits back into paid calls. Those three, unbounded loops, runaway escalation, and a broken cache, cover the large majority of sudden AI cost spikes.

Q: Why does this post teach patterns first and name tools only as examples?

Because the tools churn and the patterns do not. The specific provider, cache, or agent framework that is popular today will likely be replaced within a couple of years, but the ideas underneath, retry when a call flakes, escalate only when needed, cache what repeats, cap every loop, catch decay with a shadow, stay correct regardless of which product implements them. Naming a tool as a current example keeps the post concrete without tying its usefulness to that tool’s lifespan, which is exactly why every tool mention here is qualified with “at the time of writing”.

Series: Python + AI/ML Cookbook, Part 6: Machine Learning Operations (MLOps) & Deployment

Further reading: for the full reference, see the official Python documentation.

Related Posts

Previous: Airflow Tutorial: Your First Data Pipeline, the Right Way

Next: From Idea to AI MVP: The Right Way to Build

Series Home: Python + AI/ML Tutorial Series