The model at the top of the leaderboard is often the wrong pick for your project. Working out how to choose an LLM means matching a model to the constraints you actually have: budget, latency limits, privacy rules, and the license you can legally ship. This post gives you six plain axes, a decision flowchart, and three honest worked examples so you can stop guessing.

“The best model is the cheapest one that passes your own test, not the one that wins someone else’s.”

Last Updated: July 2026 | Tested on: Python 3.14.6 | Difficulty: Intermediate | Reading Time: 25 minutes

The large language models introduction explained what an LLM is and how scale gives it new abilities. This one is the practical follow-up: given a real task, how do you actually decide between a closed Application Programming Interface (API), an open-weight model you host yourself, or a fine-tuned smaller model? We will score six axes in Python, price the tiers with a dated table, run a small bake-off harness, do the quantization math that decides what fits on your hardware, and finish with three real people making three sensible and completely different choices. Model names and prices move fast, so the framework is the product here, not any one model.

The Six Axes That Decide Every LLM Choice

Deciding how to choose an LLM is like hiring for a role. You do not just ask “who is the smartest candidate?” You ask what the job needs: can they start today, can you afford them, will they keep secrets, and are you even allowed to hire them under the rules you work with. A brilliant candidate who costs too much or cannot be trusted with your data is not the right hire. Models are the same. Six axes cover almost every real decision.

• Quality: how well it does your specific task, not its average benchmark score.
• Cost: dollars per million tokens, or hardware cost if you self-host.
• Latency: how fast the first and last token arrive, which decides if it feels live.
• Context length: how much text you can hand it in one prompt.
• Privacy: whether your data may leave your servers at all.
• License: what you are legally allowed to do with the model and its output.

The trick is that these axes are not equal for every project. A public chatbot cares about quality and cost. A hospital tool cares about privacy above all. So instead of arguing in the abstract, put weights on the axes and let the arithmetic rank your candidates. Say a developer named Aditi is choosing a model for an internal tool where privacy and license terms matter far more than raw brilliance. Here is her scorecard.

📄 axes_scorecard.py: weight the axes, let the numbers rank the models

# Aditi scores three candidate models across the six axes that decide an LLM choice.
# Every score is 1 (weak) to 5 (great). The weights say what THIS project cares about.

# Model IDs are pinned in config, not hard-coded in the logic (the point of the post).
CANDIDATES = {
    "frontier-api": {   # e.g. a top closed model, exact id in your config
        "quality": 5, "cost": 2, "latency": 3, "context": 5, "privacy": 1, "license": 3,
    },
    "capable-open": {   # e.g. a ~70B open-weight model you host yourself
        "quality": 4, "cost": 4, "latency": 3, "context": 4, "privacy": 5, "license": 4,
    },
    "small-local": {    # e.g. a ~3B model on your own laptop
        "quality": 2, "cost": 5, "latency": 5, "context": 2, "privacy": 5, "license": 5,
    },
}

# A privacy-bound internal tool: privacy and license matter most, raw quality less.
weights = {
    "quality": 2, "cost": 2, "latency": 1, "context": 1, "privacy": 4, "license": 3,
}

def weighted_score(scores, weights):
    total = sum(scores[axis] * w for axis, w in weights.items())
    max_total = sum(5 * w for w in weights.values())
    return total, round(100 * total / max_total)

print(f"Weights for this project: {weights}\n")
print(f"{'Model':<16}{'raw':>6}{'/100':>8}")
print("-" * 30)
ranked = []
for name, scores in CANDIDATES.items():
    raw, pct = weighted_score(scores, weights)
    ranked.append((pct, name, raw))
    print(f"{name:<16}{raw:>6}{pct:>7}%")

ranked.sort(reverse=True)
print("-" * 30)
winner = ranked[0]
print(f"Best fit here: {winner[1]} ({winner[0]}%)")
print("Change the weights and the winner changes. That is the whole idea.")

▶ Output

Weights for this project: {'quality': 2, 'cost': 2, 'latency': 1, 'context': 1, 'privacy': 4, 'license': 3}

Model              raw    /100
------------------------------
frontier-api        35     54%
capable-open        55     85%
small-local         56     86%
------------------------------
Best fit here: small-local (86%)
Change the weights and the winner changes. That is the whole idea.

What happened here: The frontier API is the strongest model on paper, yet it lands last at 54% because this project punishes its weak privacy and pay-per-token cost hard. Flip the weights toward quality and context, which is what a public research assistant would want, and the frontier model jumps to the top instead. That is the entire point of a framework: the “best” model is a function of your weights, not a fixed fact. Notice too that the model IDs live in config comments, not in the scoring logic, so when the lineup changes next quarter you swap names and the framework still works.

Open vs Closed, Honestly

The words “open” and “closed” get thrown around loosely, so let us be precise, because the difference decides two of your six axes at once (privacy and license). A closed model lives behind an API. You send text in, you get text out, and you never touch the weights. An open-weight model is one whose trained parameters you can download and run on your own machine. That is the split that matters day to day.

Here is the honest part most tutorials skip. “Open weights” does not mean “open source” and it does not mean “do whatever you want.” It means you can download and run the numbers. It usually does not include the training data, the training code, or an unrestricted license. Plenty of models you can download still come with strings attached, and reading those strings before you build on a model is the difference between shipping and a nasty surprise later. Think of it like a recipe someone hands you: having the finished dish in your kitchen is not the same as owning the restaurant, and the note taped to the box may say “home use only.”

These are the license traps that catch teams most often. None of them are hypothetical; all of them have shipped on real, popular models at the time of writing.

The safe move is simple. Before you commit to any model, open its license page and search for “commercial,” “acceptable use,” and any number that looks like a threshold. A genuinely permissive license such as Apache 2.0 or MIT frees you from most of this, which is exactly why those models are so popular for products. When a license is custom, read it, or have someone who reads licenses read it. This is boring for about ten minutes and saves you from rebuilding your product later.

The Tier System: Frontier, Capable, Cheap, Local

Every model on the market slots into one of four tiers, and thinking in tiers survives the constant churn of model names. It is like coffee sizes. The names on the cups change between chains, but small, medium, large, and bring-your-own-mug map cleanly everywhere. For LLMs the tiers are frontier (the smartest and priciest), capable (the sensible everyday default), cheap (fast and nearly free for simple work), and local (open weights you run yourself, where you pay for hardware instead of tokens).

The tier that fits a task is usually decided by money, and money is easy to compute. The script below prices one real feature across all four tiers. The dollar figures are rough bands per million tokens at the time of writing (mid-2026), and they sit under this series’ freshness re-check schedule, but you should still confirm live rates on the provider’s pricing page before you trust any number.

📄 tier_cost.py: what one feature costs across the four tiers

# Anvay estimates the monthly bill for one feature across four price tiers.
# Prices are dollars per MILLION tokens, rough bands at the time of writing (mid-2026).
# Always confirm live rates on the provider's pricing page before you trust a number.
TIERS = {
    #  tier name        input   output   typical use
    "frontier":  {"in": 5.00, "out": 15.00, "note": "hardest reasoning, agents"},
    "capable":   {"in": 1.00, "out":  3.00, "note": "everyday app default"},
    "cheap":     {"in": 0.15, "out":  0.60, "note": "classify, extract, summarize"},
    "local":     {"in": 0.00, "out":  0.00, "note": "self-hosted, you pay hardware"},
}

# One feature: 200,000 requests a month, ~700 tokens in and ~300 tokens out each.
requests = 200_000
tok_in = 700
tok_out = 300
mtok_in = requests * tok_in / 1_000_000   # millions of input tokens
mtok_out = requests * tok_out / 1_000_000

print(f"Workload: {requests:,} requests/month, {tok_in} in + {tok_out} out each")
print(f"= {mtok_in:.0f}M input tokens, {mtok_out:.0f}M output tokens\n")
print(f"{'Tier':<10}{'$/Mtok in':>11}{'$/Mtok out':>12}{'monthly $':>12}   use case")
print("-" * 72)
for name, p in TIERS.items():
    monthly = mtok_in * p["in"] + mtok_out * p["out"]
    print(f"{name:<10}{p['in']:>11.2f}{p['out']:>12.2f}{monthly:>12,.0f}   {p['note']}")

print("-" * 72)
frontier = mtok_in * TIERS["frontier"]["in"] + mtok_out * TIERS["frontier"]["out"]
cheap = mtok_in * TIERS["cheap"]["in"] + mtok_out * TIERS["cheap"]["out"]
print(f"Frontier costs {frontier / cheap:.0f}x the cheap tier for the same traffic.")
print("If the cheap tier passes your bake-off, the frontier tier is money set on fire.")

▶ Output

Workload: 200,000 requests/month, 700 in + 300 out each
= 140M input tokens, 60M output tokens

Tier        $/Mtok in  $/Mtok out   monthly $   use case
------------------------------------------------------------------------
frontier         5.00       15.00       1,600   hardest reasoning, agents
capable          1.00        3.00         320   everyday app default
cheap            0.15        0.60          57   classify, extract, summarize
local            0.00        0.00           0   self-hosted, you pay hardware
------------------------------------------------------------------------
Frontier costs 28x the cheap tier for the same traffic.
If the cheap tier passes your bake-off, the frontier tier is money set on fire.

What happened here: The exact same traffic costs $1,600 a month on the frontier tier and $57 on the cheap tier, a 28x gap for work that a cheap model might handle perfectly. The local tier shows $0 per token because you already paid up front in hardware, which is the trade the quantization section makes concrete. The lesson is not “always go cheap.” It is “start at the cheapest tier that passes your test and only climb when the task forces you to.” Most teams default to the frontier tier out of habit and pay that 28x multiplier for no measurable gain.

Leaderboards and Their Traps

Public leaderboards are the first thing people reach for, and they are useful for a rough sense of the field. But treating the top row as “the best model” is where good decisions go to die. A leaderboard is like a school’s ranking by average exam score. It tells you nothing about whether that school teaches the one subject your kid actually needs. Three traps in particular catch people.

• Contamination: if the benchmark’s questions leaked into a model’s training data, its score is inflated and means little. This happens more than anyone likes to admit.
• Vibes vs benchmarks: some rankings are crowd-voted popularity contests that reward a confident, chatty tone, which is not the same as being correct on your task.
• Task mismatch: a model that tops a math benchmark may be mediocre at pulling clean JSON out of messy invoices, which might be all you need.

The fix is the single most valuable habit in this whole post: run your own bake-off. Collect 20 prompts that look like your real work, write down what a good answer must contain, and score each candidate model against your grader, not against a generic benchmark. Twenty prompts sound like a lot, but it is an afternoon of work that can save you thousands of dollars and weeks of regret. The harness below shows the shape. The real API call is one function; everything else runs offline so you can build and test your grader with no key.

📄 bakeoff.py: same prompts, two models, your grader decides

"""A tiny bake-off harness: same prompts, two models, YOUR grader decides.
Aviraj runs this before picking a model instead of trusting a leaderboard.

The real API call is one function. Swap the stub below for your provider's SDK:

    # OpenAI-style (also works against a local Ollama endpoint):
    # resp = client.chat.completions.create(
    #     model=cfg["model_id"],                # pinned in config, never hard-coded
    #     messages=[{"role": "user", "content": prompt}],
    #     temperature=0,
    # )
    # return resp.choices[0].message.content

Everything else here (loading prompts, grading, tallying) runs with no key,
so you can build and test your harness offline, then drop in the real call.
"""

# 4 of your 20 real prompts. Each has a grader: a task-relative check, not "vibes".
BAKEOFF = [
    {"id": "extract-json", "prompt": "Return ONLY the year as JSON: 'Python 3.14.6 shipped in 2026.'",
     "grade": lambda r: '"year"' in r and "2026" in r},
    {"id": "refuse-bad-math", "prompt": "What is 17 * 24? Answer with the number only.",
     "grade": lambda r: "408" in r},
    {"id": "follow-format", "prompt": "List two vegetarian proteins, one per line, no bullets.",
     "grade": lambda r: len([x for x in r.strip().splitlines() if x.strip()]) == 2},
    {"id": "stay-grounded", "prompt": "If the context says nothing about pricing, reply exactly: NOT FOUND",
     "grade": lambda r: r.strip() == "NOT FOUND"},
]

# Two stand-in "models" so the harness runs offline. In real use these are API calls.
def model_careful(prompt):
    if "year as JSON" in prompt: return '{"year": 2026}'
    if "17 * 24" in prompt: return "408"
    if "vegetarian proteins" in prompt: return "paneer\nrajma"
    if "NOT FOUND" in prompt: return "NOT FOUND"
    return ""

def model_chatty(prompt):
    if "year as JSON" in prompt: return "Sure! The year is 2025."   # ignored the format
    if "17 * 24" in prompt: return "The answer is 408."             # right, extra words ok
    if "vegetarian proteins" in prompt: return "- paneer\n- rajma\n- tofu"  # 3 + bullets
    if "NOT FOUND" in prompt: return "It looks like pricing was not mentioned."  # not exact
    return ""

def run_bakeoff(name, model):
    passed = 0
    print(f"\n{name}")
    for case in BAKEOFF:
        reply = model(case["prompt"])
        ok = bool(case["grade"](reply))
        passed += ok
        print(f"  [{'PASS' if ok else 'FAIL'}] {case['id']}")
    pct = round(100 * passed / len(BAKEOFF))
    print(f"  score: {passed}/{len(BAKEOFF)}  ({pct}%)")
    return pct

print("20-prompt bake-off (showing 4 cases), grader = your task, not a leaderboard")
a = run_bakeoff("model_careful", model_careful)
b = run_bakeoff("model_chatty", model_chatty)
print("\n" + "-" * 40)
print(f"Winner for THIS task: {'model_careful' if a >= b else 'model_chatty'}")
print("The chatty model is not 'worse'. It just fails your format-strict tasks.")

▶ Output

20-prompt bake-off (showing 4 cases), grader = your task, not a leaderboard

model_careful
  [PASS] extract-json
  [PASS] refuse-bad-math
  [PASS] follow-format
  [PASS] stay-grounded
  score: 4/4  (100%)

model_chatty
  [FAIL] extract-json
  [PASS] refuse-bad-math
  [FAIL] follow-format
  [FAIL] stay-grounded
  score: 1/4  (25%)

----------------------------------------
Winner for THIS task: model_careful
The chatty model is not 'worse'. It just fails your format-strict tasks.

What happened here: Both models got the arithmetic right, but the chatty one flunked every task that demanded a strict format: it wrapped the JSON in chit-chat, added a third protein with bullets, and softened the exact “NOT FOUND” grounding signal into a sentence. On a popularity leaderboard the chatty model might rank higher because people enjoy its tone. On your grader, which is the only one that pays your bills, it scored 25%. Swap the two stub functions for real API calls to your shortlisted models and this same harness gives you an honest, task-relative ranking in an afternoon. For a deeper treatment of graders and scoring, the LLM evaluation tutorial builds this out into a full pipeline.

Quantization: The Fits-On-My-GPU Lever

If your choice leans toward a local, self-hosted model for privacy or cost, one number decides everything: will it fit in your Graphics Processing Unit (GPU) memory? Back in the Central Processing Unit (CPU) vs GPU vs Tensor Processing Unit (TPU) tutorial we saw that a GPU’s memory (VRAM) is a hard wall. A model that does not fit will not run at any speed. Quantization is the lever that moves that wall. It stores each model parameter in fewer bits, trading a small accuracy loss for a big drop in memory, the way saving a photo as a JPEG loses a little detail nobody notices but shrinks the file enough to actually carry around.

The math is simple enough to do in your head once you have seen it once. Full-ish precision (fp16) is 2 bytes per parameter, 8-bit is 1 byte, and 4-bit is half a byte. So 4-bit quantization shrinks a model to a quarter of its fp16 footprint. This script does the arithmetic for four model sizes and tells you the smallest GPU that holds each one at 4-bit.

📄 quant_fit.py: which model fits the GPU you actually have

# Anvi checks which open models she can actually run before she picks one.
# The lever: quantization stores each weight in fewer bits. Fewer bits = fits smaller GPU.
BYTES = {"fp16": 2.0, "8-bit": 1.0, "4-bit": 0.5}

def vram_gb(params_billion, precision, overhead=1.2):
    # weights + ~20% for activations and runtime overhead
    return params_billion * BYTES[precision] * overhead

models = [("small-local 3B", 3), ("capable-open 8B", 8),
          ("capable-open 70B", 70), ("frontier-open 405B", 405)]
gpus = {"laptop 8GB": 8, "gaming 16GB": 16, "gaming 24GB": 24, "datacenter 80GB": 80}

print(f"{'Model':<20}{'fp16':>8}{'8-bit':>8}{'4-bit':>8}   smallest GPU that holds 4-bit")
print("-" * 76)
for name, b in models:
    row = f"{name:<20}"
    sizes = {}
    for p in ("fp16", "8-bit", "4-bit"):
        g = vram_gb(b, p)
        sizes[p] = g
        row += f"{g:>7.0f}G"
    fits = [gpu for gpu, cap in sorted(gpus.items(), key=lambda x: x[1]) if sizes["4-bit"] <= cap]
    row += "   " + (fits[0] if fits else "none of these")
    print(row)

print("-" * 76)
print("4-bit is the lever: it shrinks a model to a quarter of its fp16 footprint.")
print("Pick the biggest model whose 4-bit size clears the GPU you actually have.")

▶ Output

Model                   fp16   8-bit   4-bit   smallest GPU that holds 4-bit
----------------------------------------------------------------------------
small-local 3B            7G      4G      2G   laptop 8GB
capable-open 8B          19G     10G      5G   laptop 8GB
capable-open 70B        168G     84G     42G   datacenter 80GB
frontier-open 405B      972G    486G    243G   none of these
----------------------------------------------------------------------------
4-bit is the lever: it shrinks a model to a quarter of its fp16 footprint.
Pick the biggest model whose 4-bit size clears the GPU you actually have.

What happened here: An 8-billion model needs 19 GB at fp16, too much for an 8 GB laptop card, but only 5 GB at 4-bit, so it fits comfortably. A 70-billion model still needs about 42 GB even compressed, which is why it lands on a datacenter card, and the 405-billion model does not fit any single GPU in the list at all. This table turns “can I self-host?” from a vague worry into a one-line answer. When you do move a chosen model into production, serving tools handle the quantization and batching for you, which the LLM serving tutorial covers in detail. Estimate the 4-bit size first, then pick the model whose number clears your hardware.

How to Choose an LLM: The Decision Flowchart

When the six axes feel like too much to hold at once, walk this flowchart instead. Four plain questions about privacy, whether a hosted model passes your bake-off, your volume, and whether your task is narrow enough to fine-tune will land you on one of five sensible answers almost every time. Follow the arrows from the top.

Tip: click the diagram to open it full screen, then use the zoom and pan controls for a closer look.

Notice that the flowchart never starts with “which model is smartest.” It starts with privacy, because that one answer can rule out every closed API before you compare anything else. Only once data is allowed to leave your walls does the question become quality, then cost, then whether your task is narrow enough that fine-tuning a smaller open model beats renting a big general one. The fine-tuning tutorial covers that last branch when a narrow, repeatable task is where you land.

Three Worked Choices

Frameworks for how to choose an LLM are easy to nod along to and hard to apply, so here are three real situations running through the same six axes and landing in three completely different places. See which one sounds most like you.

The hobby project. Say Anvi is building a personal recipe assistant for fun on evenings and weekends. Her weights are cost first (she pays out of pocket), then quality, and privacy barely registers because it is her own grocery list. Her choice is easy: a cheap-tier hosted API for anything tricky, and a small local model through Ollama for the routine calls so she pays nothing per query. She skips the frontier tier entirely. There is no world where a recipe helper justifies a $1,600 monthly bill, and the cheap tier passes her bake-off fine.

The startup MVP. Aviraj is shipping a customer-support feature and needs it live next month with a tiny budget and unknown traffic. His weights are quality and speed of shipping, with cost as a live worry once users arrive. He starts on a capable-tier hosted API, because managing his own GPUs would burn the one resource a startup cannot spare, which is time. He wires the model ID into config from day one so he can switch providers without a rewrite, and he keeps his 20-prompt bake-off in the repo so that when the bill grows he can test whether a cheaper tier or a self-hosted open model would pass. He buys flexibility now and optimizes cost later, on evidence.

The regulated enterprise. Aditi works at a hospital where patient data legally cannot leave the building. For her, privacy is not a weight among others, it is a gate. Every closed API is out before quality is even discussed. Her choice is an open-weight model with a clearly commercial license (Apache 2.0 or similar), quantized to 4-bit, running on a GPU inside her own network. She trades away some peak capability versus a frontier model and accepts a real hardware cost, and in exchange she gets compliance, which is the only axis her legal team cares about. When a task is narrow and repeats constantly, she fine-tunes that open model to lift quality back up without ever sending data outside.

Three teams, one framework, three different winners. Nobody here picked the “best” model. Each picked the model that scored highest on the axes their project actually cared about.

Common Mistakes

• Defaulting to the frontier tier for everything. The smartest model can cost tens of times more per token than a cheaper sibling that passes your test. Start cheap and climb only when the task forces you to, not out of habit.
• Trusting a leaderboard as gospel. Benchmark contamination and popularity voting make the top row a rough hint, not a verdict. The only ranking that matters is your own bake-off on prompts that look like your real work.
• Skipping the license. “Open weights” does not mean “free to ship.” A research-only or scale-triggered license can force a costly rebuild after launch. Read it before you build, not after.
• Hard-coding model names in your logic. Model IDs change every few months. Pin them in config so a swap is one line, not a scavenger hunt through your codebase.
• Choosing local before checking if it fits. A model you cannot load is not a choice. Estimate the 4-bit VRAM first, then decide whether self-hosting is even on the table.

Best Practices

• Do write down your six-axis weights before you look at any model, so the model does not talk you into caring about the wrong thing.
• Do keep a 20-prompt bake-off in your repo and re-run it whenever you consider a new model or a cheaper tier.
• Do start with a hosted API for most application work and switch to self-hosting only when privacy, volume, or fine-tuning demand it.
• Do pin every model ID in config and read the license before you commit.
• Don’t treat any price or model name in this post as permanent; re-verify current numbers, because they move every few months.
• Don’t pay for capability your task never uses. The right model is the cheapest one that passes.

Conclusion

Knowing how to choose an LLM is not about memorizing which model is on top this month. It is a matching problem: weight the six axes for your project (quality, cost, latency, context, privacy, license), price the tiers, run your own bake-off, and check what actually fits your hardware if you go local. The model names and dollar figures in this post will drift, but those axes and the decision flowchart do not. That is why the framework is the real takeaway, not any single recommendation.

So the next time someone asks “which LLM should we use?”, do not answer with a model name. Answer with the weights, the bake-off score, and the license check, and let the choice fall out of the evidence. The complete curriculum lives on the Python + AI/ML tutorial series home.

Frequently Asked Questions

Interview Questions on Choosing an LLM

If you can walk through these without peeking, you are ready for this topic in an interview.

Q: What are the main axes you weigh when choosing an LLM, and why not just pick the top benchmark model?

The core axes are quality on your specific task, cost per token or per hour, latency, context length, privacy, and license. You do not pick the top benchmark model because “best” is task-relative and constraint-bound: a model that tops a math benchmark may be worse at your JSON extraction, may cost far more than a cheaper model that passes your test, and may be ruled out entirely by privacy or license before quality is even discussed. Weighting the axes for the project and ranking candidates against those weights gives a defensible choice; a single leaderboard score does not.

Q: What is the difference between open weights and open source, and why does it matter for a product?

Open weights means you can download and run the trained parameters, but it usually excludes the training data and code and often carries a restrictive license. Open source, in the strict sense, implies a permissive license and broader freedoms. It matters because a downloadable model can still be research-only, ban certain uses, or require a paid license past a scale threshold. Shipping a product on a model without reading its license risks a forced rewrite later, so the license is a first-class decision axis, not a footnote.

Q: How would you run a fair comparison between two models for a real task?

Build a bake-off: collect about 20 prompts that mirror the real workload, and for each write a concrete grader that checks what a good answer must contain, rather than judging by feel. Run every candidate at temperature 0 for reproducibility, tally the pass rate per model, and factor in cost and latency. This gives a task-relative ranking that is far more trustworthy than a public benchmark, which can be contaminated or measure a different task than yours.

Q: How does quantization affect the choice to self-host a model?

Quantization stores each parameter in fewer bits, so 4-bit shrinks a model to a quarter of its fp16 footprint. That decides whether a model even fits your GPU memory, which is a hard wall before speed matters at all. An 8-billion model drops from about 19 GB at fp16 to roughly 5 GB at 4-bit, which is the difference between not fitting and fitting on a modest card. So the self-host decision starts with estimating the 4-bit VRAM, and only models that clear your hardware are real options.

Q: Scenario: your API bill for a summarization feature triples after a traffic spike. How do you decide whether to change models?

First check whether the feature even needs its current tier. Summarization usually runs well on a cheap-tier model, so run the existing bake-off against a cheaper model and a self-hosted open model to see if either passes. Compute the crossover: at the new volume, does per-token cost now exceed the hardware cost of self-hosting? If a cheaper tier passes the grader, switch to it, since the model ID is pinned in config and the swap is one line. If volume is high and steady, a quantized open model on owned hardware may undercut the API, which is exactly the case where self-hosting earns its complexity.

Q: A teammate named Anvay wants to use a popular downloadable model in a paid app because it tops a leaderboard. What do you check first?

Two things before anything else. First, the license: confirm it permits commercial use and does not have an acceptable-use clause or a scale trigger that would bite the product later; a research-only license kills the plan regardless of the score. Second, run our own bake-off on prompts that mirror the app, because the leaderboard rank may come from benchmark contamination or a task unlike ours. Only if the license clears and the model passes our grader does the leaderboard position become a supporting reason rather than the whole argument.

Go deeper: Hugging Face documentation covers every edge case of this topic.

Related Posts

Previous: How LLMs Are Trained: Pretraining, SFT, and RLHF Explained

Next: GenAI: Prompt Engineering, Techniques and Best Practices

Series Home: Python + AI/ML Tutorial Series