AI features rarely die because the model was not smart enough. They die in a budget review. LLM cost optimization is how you stop that meeting from happening: know what each request costs, cache the prefix you keep resending, batch the jobs nobody is waiting on, and route easy work to cheap models. This post works all four levers in runnable Python, plus a guard that catches runaway spend.

“Optimization hinders evolution.”

Alan Perlis, Epigrams on Programming

Last Updated: July 2026 | Tested on: Python 3.14.6 (standard library only) | Difficulty: Intermediate | Reading Time: 21 minutes

Think of an LLM like a metered taxi. The mechanisms that keep the fare down do not change: share the ride when you can, avoid re-driving the same route, and do not book the airport limo for a trip to the corner shop. Providers rename models and shuffle prices constantly, but caching, batching, and tiering have stayed the winning moves for years, and they will outlast whichever model is top of the charts this quarter. That is why this post teaches the mechanisms first and treats the prices as numbers you swap in.

Token Economics: You Pay Per Token

Tip: click the diagram to open it full screen, then use the zoom and pan controls for a closer look.

The diagram above is the whole post in one picture: every request passes through the same four levers before it hits your bill. Start with the meter itself. You are billed per token, and tokens split into two buckets with very different prices. Input tokens are everything you send: your system prompt, the chat history, the user’s question. Output tokens are what the model writes back. On almost every provider, output tokens cost several times more than input tokens, and the gap between the cheapest small model and the flagship large model is enormous. Here is a dated reference table of three tiers, the kind you build once and keep updated on a freshness schedule.

These are placeholder numbers, chosen to show a clean 6x and 36x spread across tiers, which is close to what real providers charge at the time of writing. The lesson is the spread, not the exact digits. Now do the thing most teams skip: estimate the bill before you write the feature. Say Aditi is building a support bot for a grocery delivery app and expects 10,000 chats a day. A quick script turns that into a monthly number per tier.

📄 token_economics.py: estimate the monthly bill before you build

# Prices are dated placeholders (per 1M tokens), current mid-2026.
PRICING = {  # dollars per 1,000,000 tokens
    "small":  {"input": 0.10, "output": 0.40},   # nano/mini tier
    "medium": {"input": 0.60, "output": 2.40},   # 6x the small tier
    "large":  {"input": 3.60, "output": 14.40},  # 36x the small tier
}

def estimate_tokens(text: str) -> int:
    """Rough token count. Real tokenizers (tiktoken, the provider SDK) are exact;
    this 4-chars-per-token rule is close enough for a budget estimate."""
    return max(1, len(text) // 4)

system_prompt = (
    "You are a friendly support agent for a grocery delivery app. "
    "Answer in under 80 words. Be polite. If you cannot help, offer to escalate."
)
user_question = "My paneer order arrived warm and the packet was torn. What can you do?"
model_answer = (
    "So sorry about the paneer, that is not the experience we want. "
    "I have logged a full refund to your wallet, it lands within 24 hours. "
    "Want me to reorder it on the next delivery slot at no charge?"
)

in_tokens = estimate_tokens(system_prompt + user_question)
out_tokens = estimate_tokens(model_answer)
print(f"Per query:  {in_tokens} input tokens + {out_tokens} output tokens")

QUERIES_PER_DAY = 10_000
print(f"\nSupport bot at {QUERIES_PER_DAY:,} queries/day, monthly cost per tier:")
print(f"{'tier':<8}{'input $/mo':>14}{'output $/mo':>14}{'total $/mo':>14}")
for tier, rate in PRICING.items():
    day_in = QUERIES_PER_DAY * in_tokens * rate["input"] / 1_000_000
    day_out = QUERIES_PER_DAY * out_tokens * rate["output"] / 1_000_000
    mo_in, mo_out = day_in * 30, day_out * 30
    print(f"{tier:<8}{mo_in:>14.2f}{mo_out:>14.2f}{mo_in + mo_out:>14.2f}")

small, large = PRICING["small"], PRICING["large"]
print(f"\nOutput vs input price on the small tier: "
      f"{small['output'] / small['input']:.0f}x more expensive per token")
print(f"Large vs small tier, input price: {large['input'] / small['input']:.0f}x")
print(f"Large vs small tier, output price: {large['output'] / small['output']:.0f}x")

▶ Output

Per query:  51 input tokens + 48 output tokens

Support bot at 10,000 queries/day, monthly cost per tier:
tier        input $/mo   output $/mo    total $/mo
small             1.53          5.76          7.29
medium            9.18         34.56         43.74
large            55.08        207.36        262.44

Output vs input price on the small tier: 4x more expensive per token
Large vs small tier, input price: 36x
Large vs small tier, output price: 36x

What happened here: The same bot costs 7 dollars a month on the small tier and 262 dollars on the large tier, for the exact same traffic. That is the number the budget review cares about, and now Aditi has it before writing a single feature line. Notice two things. Output tokens cost 4x the input tokens here, so a chatty model that pads every answer quietly triples your output bill. The reason output costs more per token is mechanical: your whole prompt is read in one parallel pass (the prefill), where the GPU processes every prompt token at once. The answer, though, is written one token at a time, and each new token requires another forward pass through the model, attending over everything generated so far (an autoregressive decode). Because those passes happen sequentially instead of in parallel, generating output keeps the hardware busy far longer per token than reading input does, and providers price output higher to match.

And the tier spread is 36x, which is why “just use the best model everywhere” is a decision with a price tag, not a free default. The token estimate is rough on purpose; a real tokenizer like tiktoken is exact, but for a budget you only need to be in the right ballpark.

Prompt Caching: Stop Paying for the Same Prefix

Picture a call centre where every agent reads the same 3-page policy sheet aloud to the customer before answering their actual question. Absurd, right? Yet that is what an uncached LLM app does: it re-sends the same long system prompt, the same product catalog, the same rules, on every single call, and you pay full input price for all of it every time. Prompt caching fixes this. The provider stores the repeated prefix after the first call, and every later call that starts with the same prefix reads it back at a steep discount, often around 90 percent off those prefix tokens.

You only pay full price for the part that actually changes, the user’s question. Of every LLM cost optimization trick in this post, this one is the closest to free money. Here is the measured difference on a repeated-prefix workload.

📄 prompt_caching.py: what a repeated prefix costs with and without caching

# Dated placeholder prices (per 1M tokens), current mid-2026.
INPUT_RATE = 0.60         # normal input, dollars per 1M tokens
CACHED_INPUT_RATE = 0.06  # a cache HIT bills at ~10% of the normal input rate
CACHE_WRITE_RATE = 0.75   # the FIRST call that stores the prefix costs a bit extra
OUTPUT_RATE = 2.40

fixed_prefix_tokens = 2000     # the repeated part, same every request
variable_question_tokens = 40  # the user's actual question, different each time
output_tokens = 60

REQUESTS = 1000  # a busy hour on the support bot, same catalog prefix each time

# Without caching: you pay full input price for the whole prefix every single time.
no_cache = REQUESTS * (
    (fixed_prefix_tokens + variable_question_tokens) * INPUT_RATE
    + output_tokens * OUTPUT_RATE
) / 1_000_000

# With caching: the first request WRITES the prefix (small premium), every request
# after that READS it at the cheap cached rate. Only the question is full price.
first_call = (
    fixed_prefix_tokens * CACHE_WRITE_RATE
    + variable_question_tokens * INPUT_RATE
    + output_tokens * OUTPUT_RATE
) / 1_000_000
later_calls = (REQUESTS - 1) * (
    fixed_prefix_tokens * CACHED_INPUT_RATE   # cache hit, 90% off the prefix
    + variable_question_tokens * INPUT_RATE
    + output_tokens * OUTPUT_RATE
) / 1_000_000
with_cache = first_call + later_calls

print(f"{REQUESTS} requests, {fixed_prefix_tokens}-token prefix repeated each time")
print(f"  Without caching: ${no_cache:.4f}")
print(f"  With caching:    ${with_cache:.4f}")
saved = no_cache - with_cache
print(f"  Saved:           ${saved:.4f}  ({saved / no_cache * 100:.0f}% off)")

▶ Output

1000 requests, 2000-token prefix repeated each time
  Without caching: $1.3680
  With caching:    $0.2894
  Saved:           $1.0786  (79% off)

What happened here: One hour of traffic dropped from 1.37 dollars to 29 cents, a 79 percent cut, just by not re-billing the 2000-token prefix a thousand times. The bigger and more repeated your prefix, the more caching pays off, which is why it shines for support bots, coding assistants, and anything with a fat system prompt or a fixed document in context. Two honest caveats. Caches expire after a few minutes of no use, so caching helps steady traffic more than a request that fires once an hour.

And the first call writes the cache at a small premium, so a prefix used only twice barely breaks even. Order your prompt with the stable part first and the changing part last, or the cache never gets a matching prefix to reuse.

Batch APIs: Half Price When Nobody Is Waiting

Some work has a human staring at the screen, and some does not. Tagging last night’s product reviews, generating descriptions for a catalog, summarizing yesterday’s support tickets: none of these have a customer tapping their foot. For that kind of offline work, most providers offer a batch API that runs your jobs whenever they have spare capacity, usually within a few hours, in exchange for roughly half the price. You hand over a file of requests, come back later, and pick up the results. The mechanism is simple, and the savings are real. Here is the math for a nightly classification job.

📄 batch_discount.py: sync price versus batch price for offline work

# Dated placeholder prices (per 1M tokens), current mid-2026.
INPUT_RATE = 0.60
OUTPUT_RATE = 2.40
BATCH_DISCOUNT = 0.50  # batch tier bills at half the sync rate on many providers

# A nightly job: classify yesterday's 200,000 product reviews as pos/neg/neutral.
# No user is waiting, so a few hours of turnaround is fine.
JOBS = 200_000
input_tokens_each = 120   # the review text plus a short instruction
output_tokens_each = 3    # just the label word

sync_cost = JOBS * (
    input_tokens_each * INPUT_RATE + output_tokens_each * OUTPUT_RATE
) / 1_000_000
batch_cost = sync_cost * BATCH_DISCOUNT

print(f"Classifying {JOBS:,} reviews overnight")
print(f"  Sync API (answer in seconds):   ${sync_cost:.2f}")
print(f"  Batch API (answer in hours):    ${batch_cost:.2f}")
print(f"  Saved by not needing it now:    ${sync_cost - batch_cost:.2f}")
print(f"\nRule: if no human is waiting on the answer, batch it.")

▶ Output

Classifying 200,000 reviews overnight
  Sync API (answer in seconds):   $15.84
  Batch API (answer in hours):    $7.92
  Saved by not needing it now:    $7.92

What happened here: Half off, for the price of patience. The only thing you gave up is speed, and the reviews from last night do not care whether they get labelled at 2am or 6am. The instinct to reach for the fast synchronous endpoint is strong because that is what every quickstart shows you, but for scheduled jobs, backfills, and offline pipelines, batch is almost always the right call. A good habit is to ask one question of every new LLM task: is a person waiting on this answer right now? If the answer is no, it belongs in a batch.

Model Tiering: Cheap Model First, Escalate the Hard Ones

This is the lever with the biggest payoff, and it mirrors how a good support desk already works. A front-line agent handles the easy tickets, and only the tricky ones get escalated to a senior specialist. You would never route every “where is my order” question to your most expensive expert. Model tiering applies the same idea: send every request to a cheap model first, and only escalate to the expensive model when the cheap one is not confident.

The trick is knowing when to escalate. A simple, reliable signal is the cheap model’s own confidence, or a quick check on whether its answer looks complete. Below is a two-tier router running against a small labelled eval set, so we can prove the quality holds while the cost drops. The two models are mocked here so the whole thing runs with no API key; in production you swap the mocks for real calls.

📄 two_tier_router.py: cheap model first, escalate only the hard ones

# The two "models" here are mocks so the whole file runs with no API key.
# In production, replace classify_small / classify_large with real API calls.
PRICING = {  # per 1M tokens, dated placeholders (mid-2026)
    "small": {"input": 0.10, "output": 0.40},
    "large": {"input": 3.60, "output": 14.40},  # 36x the small tier
}
IN_TOK, OUT_TOK = 200, 30  # rough tokens per classification call

def call_cost(tier: str) -> float:
    r = PRICING[tier]
    return (IN_TOK * r["input"] + OUT_TOK * r["output"]) / 1_000_000

# A small eval set of support tickets, each with its correct intent (gold label).
EVAL = [
    ("Where is my order, it is 2 hours late",                 "delivery"),
    ("Refund me for the spoiled tomatoes please",             "refund"),
    ("I cannot log into my account",                          "account"),
    ("The paneer packet was torn on arrival",                 "refund"),
    ("Change my delivery address to the office",              "delivery"),
    ("My coupon SAVE20 did not apply at checkout",            "billing"),
    ("App keeps crashing when I open the cart",               "account"),
    ("Late AND the milk was warm, want money back",           "refund"),   # ambiguous
    ("Charged twice but only one order showed up",            "billing"),  # ambiguous
    ("Driver was rude and my order was also missing an item", "refund"),   # ambiguous
]

# Mock cheap model: confident and correct on clear tickets, unsure on messy ones.
CONFIDENT = {
    "Where is my order, it is 2 hours late": ("delivery", 0.95),
    "Refund me for the spoiled tomatoes please": ("refund", 0.93),
    "I cannot log into my account": ("account", 0.97),
    "The paneer packet was torn on arrival": ("refund", 0.90),
    "Change my delivery address to the office": ("delivery", 0.96),
    "My coupon SAVE20 did not apply at checkout": ("billing", 0.88),
    "App keeps crashing when I open the cart": ("account", 0.91),
}
def classify_small(ticket: str):
    # Anything not clearly known comes back low-confidence, triggering escalation.
    return CONFIDENT.get(ticket, ("other", 0.40))

# Mock strong model: gets the ambiguous tickets right (that is what you pay for).
STRONG = {
    "Late AND the milk was warm, want money back": "refund",
    "Charged twice but only one order showed up": "billing",
    "Driver was rude and my order was also missing an item": "refund",
}
def classify_large(ticket: str):
    return STRONG.get(ticket, classify_small(ticket)[0])

THRESHOLD = 0.75  # below this, we do not trust the cheap model, so we escalate

def run_all_large():
    correct = cost = 0.0
    for ticket, gold in EVAL:
        if classify_large(ticket) == gold:
            correct += 1
        cost += call_cost("large")
    return correct / len(EVAL), cost

def run_two_tier():
    correct = cost = escalations = 0.0
    for ticket, gold in EVAL:
        label, conf = classify_small(ticket)
        cost += call_cost("small")            # every ticket tries the cheap model
        if conf < THRESHOLD:                  # unsure, so pay for the strong model too
            label = classify_large(ticket)
            cost += call_cost("large")
            escalations += 1
        if label == gold:
            correct += 1
    return correct / len(EVAL), cost, escalations

acc_large, cost_large = run_all_large()
acc_tier, cost_tier, escalations = run_two_tier()

print(f"Always the large model:  accuracy {acc_large:.0%}  cost ${cost_large * 1000:.4f}/1k")
print(f"Two-tier with escalation: accuracy {acc_tier:.0%}  cost ${cost_tier * 1000:.4f}/1k")
print(f"Escalated {int(escalations)} of {len(EVAL)} tickets to the large model")
print(f"Cost cut at equal accuracy: {(1 - cost_tier / cost_large) * 100:.0f}%")

▶ Output

Always the large model:  accuracy 100%  cost $11.5200/1k
Two-tier with escalation: accuracy 100%  cost $3.7760/1k
Escalated 3 of 10 tickets to the large model
Cost cut at equal accuracy: 67%

What happened here: Same 100 percent accuracy, about two-thirds off the bill. The cheap model handled the 7 clear tickets on its own and escalated only the 3 genuinely ambiguous ones, the tickets with two complaints stuffed into one sentence. Because escalating means paying for both models on that ticket, the win depends on the cheap model catching most of the load, and here it caught 70 percent of it.

This is a preview of a router: a real system replaces the confidence lookup with the model's actual reported confidence or a lightweight quality check, and we build the full escalating pattern later in the series. The measurement discipline matters as much as the code. Never claim a tiering setup is free money without an eval set proving quality held; that eval is exactly what an interviewer wants to hear you mention.

Cost Regression Guard: Log Every Request, Alarm at Budget

Every lever above lowers your bill, but LLM cost optimization is not finished until something is watching the total. This last piece keeps the bill from blowing up when something goes wrong, and something always eventually goes wrong: a retry loop with an off-by-one bug, a prompt that quietly doubled in size, a traffic spike nobody warned you about. Think of it like the fuel gauge and low-fuel light in a car. You want the number in front of you at all times, and you want a loud beep well before the tank is empty.

The pattern is to log the cost of every single request as it happens, keep a running monthly total, and raise an alarm at a budget threshold so a bad day cannot become a bad invoice. Aviraj wires this into the same metrics system the rest of the app already reports to.

📄 cost_guard.py: log per-request cost, alarm at the monthly budget

from datetime import date

PRICING = {  # per 1M tokens, dated placeholders (mid-2026)
    "small": {"input": 0.10, "output": 0.40},
    "large": {"input": 3.60, "output": 14.40},
}

class CostGuard:
    """Add up spend per request and raise the alarm at a monthly budget."""
    def __init__(self, monthly_budget: float):
        self.monthly_budget = monthly_budget
        self.month_total = 0.0
        self.requests = 0

    def record(self, model, in_tokens, out_tokens):
        r = PRICING[model]
        cost = (in_tokens * r["input"] + out_tokens * r["output"]) / 1_000_000
        self.month_total += cost
        self.requests += 1
        pct = self.month_total / self.monthly_budget * 100
        status = "OK"
        if pct >= 100:
            status = "OVER BUDGET"
        elif pct >= 80:
            status = "WARN 80%"
        print(f"  req#{self.requests:<3} {model:<6} ${cost:.6f}  "
              f"month=${self.month_total:.4f} ({pct:.0f}% of budget) [{status}]")
        return status

guard = CostGuard(monthly_budget=0.05)  # tiny budget so the demo trips the alarm

traffic = [
    ("small", 200, 30),
    ("small", 180, 25),
    ("large", 4000, 800),   # a big summarization job
    ("small", 210, 40),
    ("large", 6000, 1200),  # another heavy one, this is what pushes us over
]
print(f"Budget this month: ${guard.monthly_budget:.2f}")
for model, in_tok, out_tok in traffic:
    status = guard.record(model, in_tok, out_tok)
    if status == "OVER BUDGET":
        print(f"  -> ALARM: paused non-critical LLM calls on {date.today().isoformat()}")
        break

▶ Output

Budget this month: $0.05
  req#1   small  $0.000032  month=$0.0000 (0% of budget) [OK]
  req#2   small  $0.000028  month=$0.0001 (0% of budget) [OK]
  req#3   large  $0.025920  month=$0.0260 (52% of budget) [OK]
  req#4   small  $0.000037  month=$0.0260 (52% of budget) [OK]
  req#5   large  $0.038880  month=$0.0649 (130% of budget) [OVER BUDGET]
  -> ALARM: paused non-critical LLM calls on 2026-07-10

What happened here: The two heavy large-model jobs, not the dozens of small ones, are what pushed spend over budget, and the guard caught it on the exact request that crossed the line. That per-request log line is the whole trick: ship it to your metrics dashboard and you get a live cost graph, a warning at 80 percent, and a hard stop at 100 percent that pauses non-critical calls before the bill runs away.

In a real service you would attach this to the response's reported token usage rather than an estimate, and you would separate a soft warn from a hard stop so genuinely important requests still get through. This is the cost side of the same monitoring habit that catches quality regressions, and it is what turns "we think it is cheap" into "we know, and we are alerted the moment it is not."

Common Mistakes

LLM Cost Optimization Best Practices

Want more? Hugging Face documentation documents everything this post could not fit.

Frequently Asked Questions

Interview Questions on LLM Cost Optimization

Try each one aloud first. The phrasing you produce under mild pressure is what interviews measure.

Q: Your LLM feature's monthly bill doubled while traffic stayed flat. How do you debug that?

Start with per-request logs of input tokens, output tokens, model, and a feature tag. Flat traffic with doubled cost usually means prompt growth, like RAG stuffing more chunks or chat history accumulating, a silent switch to a pricier model, a cache hit-rate collapse after someone edited the prompt prefix, or a retry storm. Alert on cost per request, not just the monthly total.

Q: Design the cost controls for a high-volume customer support bot without wrecking quality.

Tier the models: a small model handles the common intents and only low-confidence cases escalate to the expensive one. Cache the static system prompt and tool definitions, cap output tokens, and push non-urgent work like conversation summaries to the batch API at half price. Then prove quality held with evals before and after, because a cheap bot that answers wrong costs more than the API ever did.

Q: How do you structure prompts to actually benefit from prompt caching?

Caching matches on the exact prefix, so order content from static to volatile: system prompt, tool schemas, and reference documents first, the user's message last. One changed byte early in the prompt invalidates everything after it, which is why injecting a timestamp at the top of a system prompt silently kills your hit rate.

Q: When does fine-tuning become a cost decision rather than a quality decision?

When you are pasting the same long few-shot examples into every request, you are paying for those tokens millions of times. Fine-tuning moves that knowledge into the weights once. Compare the training plus hosting cost against tokens saved per request times volume: at high volume the curve crosses fast, at low volume prompting stays cheaper.

Q: Output tokens cost more than input tokens on most APIs. How does that change how you build?

Treat model verbosity as a budget line. Cap max_completion_tokens, request terse structured output like JSON with a fixed schema, and stop streaming once you have what you need. A long system prompt costs you once per request, but a chatty answer costs you on every single response, so brevity instructions pay for themselves immediately.

Related Posts

Previous: Python Chatbot Project: Multi-Provider Chat App with Costs

Next: How to Evaluate LLM Output: RAGAS, DeepEval, Hallucination Detection

Series Home: Python + AI/ML Tutorial Series