LLM observability, short for Large Language Model observability, is how you see inside a language model app after it leaves your laptop: every request broken into timed steps, with the tokens, cost, latency, and quality score attached, so a slow answer or a surprise bill becomes something you can actually read instead of guess at. Evaluation tells you whether your app is good on a fixed test set; observability tells you what real users are hitting in production right now.
The industry has quietly converged on running both, an eval framework plus an observability platform, and in this guide we build the observability half from first principles in plain Python, then map every piece onto Langfuse so the dashboard stops looking like magic.
“You cannot fix what you cannot see. An LLM app without tracing is a black box that occasionally sends you a large invoice.”
Last Updated: July 2026 | Tested on: Python 3.14.6 | Difficulty: Advanced | Reading Time: 20 minutes
- LLM tool calling and the chatbot project, since we trace that kind of app
- LLM-as-a-judge for the eval scores we attach to traces
- Python context managers and decorators, plus dictionaries
- The toy tracer here runs offline with no keys. To use real Langfuse: pip install langfuse and point it at a self-hosted or cloud instance. The SDK below was current at the time of writing; check the Langfuse docs before you ship.
Here is the mental model. Think of a restaurant kitchen on a busy night. One order ticket, say a paneer thali, is a single request. To fill it, the kitchen does several timed steps: the tandoor station, the curry station, the rice station. If a diner named Aditi waits forty minutes, the manager does not shrug; she pulls that one ticket and sees which station stalled. A trace is that ticket. The spans are the stations. Without the ticket you only know the meal was slow, not why.
Table of Contents
What LLM Observability Actually Is
A normal web app is mostly deterministic, so classic logging and metrics get you a long way. An LLM app is different in three ways that make plain logs almost useless. It is non-deterministic, so the same input can produce a different path today than yesterday. It is expensive per call, so cost is a first-class number you must watch, not an afterthought. And its quality is fuzzy, so you need scores, not just green or red status codes. Observability for LLMs is the practice of recording every request as a structured tree with those three signals baked in: how long each step took, how many tokens and dollars it burned, and how good the result was.
This is the sibling of evaluation, not a replacement for it. Evals run on a curated set before you ship. Observability runs on live traffic after you ship, and it is where you discover the inputs your eval set never imagined. Real teams wire the two together: the same judge that scores your eval set also scores a sample of production traces, so a quality dip in the wild trips the same alarm as a failing test. Get both and you can move fast without flying blind.
Traces, Spans, and Generations
Three words carry all of LLM observability, so let us pin them down once, properly. A trace is one full request from the user, start to finish. A span is any single step inside that trace, like a retrieval or a database read, with a start and end time. A generation is a special span: the one that calls a model, so on top of timing it also records the model name, the input and output token counts, and therefore the cost. Spans nest inside spans to form a tree, and the trace is the root. That is the entire data model, and it is the same whether you use Langfuse, LangSmith, or raw OpenTelemetry.
The diagram below is one Retrieval-Augmented Generation (RAG) request drawn as that tree. Read it top down: the request comes in, becomes a trace, and the trace holds a pipeline span with three children. Two of the three are generations, so they carry tokens and cost; the middle one is a plain span for the vector search. Quality scores hang off the trace itself and get attached a little later, once a judge or a user has weighed in.
Tip: click the diagram to open it full screen, then use the zoom and pan controls for a closer look.
Notice the shapes. Latency lives on every node, because every step takes time. Tokens and cost live only on the green generation nodes, because only model calls spend money. Scores live on the trace, because quality is a judgment about the whole answer, not about one internal step. Keep those three homes straight and the rest of this post falls into place.
Instrument a RAG Request: A Span Tree You Can Run
Before we touch Langfuse, we build a tracer ourselves in about eighty lines, because once you have written one you will never be confused by a tracing SDK again. The core is a Trace that holds a stack of spans and a span() context manager: entering it starts a timer and pushes a node, leaving it stops the timer and pops. A generation is just a span you tag with a model and token counts so it can compute a cost. Here is that core.
📄 obs.py: a tracer with the same data model as Langfuse
import time, uuid
from contextlib import contextmanager
# USD per 1,000,000 tokens: (input_price, output_price). Illustrative numbers
# with the right shape. Prices move, so keep them in one table you can update.
PRICES = {
"text-embedding-3-small": (0.02, 0.0),
"claude-haiku-4": (0.80, 4.00),
"claude-sonnet-4": (3.00, 15.00),
}
class Span:
def __init__(self, name, kind="span"):
self.id = uuid.uuid4().hex[:6]
self.name, self.kind = name, kind
self.children = []
self.model = None
self.tokens_in = self.tokens_out = 0
self.ms = 0.0
self._start = time.perf_counter()
def stop(self):
self.ms = round((time.perf_counter() - self._start) * 1000, 1)
@property
def cost(self):
if not self.model:
return 0.0
pin, pout = PRICES[self.model]
return self.tokens_in / 1e6 * pin + self.tokens_out / 1e6 * pout
def rollup_cost(self):
return self.cost + sum(c.rollup_cost() for c in self.children)
class Trace:
def __init__(self, name, user_id=None, session_id=None, release="v1"):
self.name, self.user_id, self.session_id, self.release = (
name, user_id, session_id, release)
self.root = Span(name, "trace")
self.scores, self._stack = {}, [self.root]
self._start = time.perf_counter()
@contextmanager
def span(self, name, kind="span"):
s = Span(name, kind)
self._stack[-1].children.append(s) # nest under the current span
self._stack.append(s)
try:
yield s
finally:
s.stop()
self._stack.pop()
def score(self, name, value, comment=""):
self.scores[name] = (value, comment)
@property
def cost(self):
return self.root.rollup_cost() # sum cost of the whole tree
What happened here: A Span times itself between construction and stop(), and computes its own cost only if it has a model. The Trace keeps a stack, so whenever you open a new span() it nests under whatever span is currently open. That stack is the one trick that turns a flat sequence of calls into a tree. The full file adds a pretty printer and a couple of helpers; it lives in the series repo as 229-obs.py. Now we wrap an actual RAG pipeline with it.
📄 run_rag.py: three steps, two of them model calls, one trace
import time
from obs import Trace, print_tree
def embed_query(text, gen):
time.sleep(0.02)
gen.model = "text-embedding-3-small"
gen.tokens_in = len(text.split()) * 2
return [0.1] * 8
def vector_search(vec):
time.sleep(0.14) # a deliberately slow index
return ["paneer butter masala recipe", "dal tadka recipe", "jeera rice recipe"]
def llm_generate(question, docs, gen):
time.sleep(0.05)
gen.model = "claude-sonnet-4"
gen.tokens_in, gen.tokens_out = 320, 90
return "Simmer the paneer in the tomato masala and serve it with jeera rice."
def answer(trace, question):
with trace.span("embed_query", "generation") as g:
vec = embed_query(question, g)
with trace.span("vector_search") as s:
docs = vector_search(vec)
with trace.span("llm_generate", "generation") as g:
return llm_generate(question, docs, g)
if __name__ == "__main__":
# say a user named Aditi asks the cooking assistant one question
trace = Trace("rag_answer", user_id="aditi", session_id="s-42")
with trace.span("rag_pipeline"):
reply = answer(trace, "How do I make paneer masala with rice?")
print_tree(trace)
print(f"\nReply: {reply}")
▶ Output
TRACE rag_answer user=aditi session=s-42 total=212.4ms cost=$0.002310
span rag_pipeline 212.4ms
GEN embed_query 20.9ms text-embedding-3-small in=16 out=0 $0.000000
span vector_search 140.6ms
GEN llm_generate 50.6ms claude-sonnet-4 in=320 out=90 $0.002310
Reply: Simmer the paneer in the tomato masala and serve it with jeera rice.
What happened here: One request produced one trace with a clean tree underneath it. The timings are real, measured by the context managers, so you can already see the story: the vector search took roughly seven times as long as the model call, and the model call is where all the money went. The embedding was basically free. Nobody told you that; the trace showed you. That single readable picture, per request, is the entire point of observability, and everything else in this post is just slicing many of these trees in useful ways.
The Same Trace with Langfuse
Now swap our toy tracer for the real thing. Langfuse is an open-source, self-hostable platform, so you can run it on your own box with Docker and keep every trace inside your network, which matters when prompts contain user data. The SDK gives you a single decorator, @observe, that turns any function into a span; mark a function as a generation and a wrapped model client fills in the tokens and cost for you. Notice that the pipeline logic is identical to what we already wrote. Only the plumbing changed.
📄 app.py: the same pipeline, traced by Langfuse instead
# pip install langfuse
# set LANGFUSE_HOST (self-hosted or cloud) + LANGFUSE_PUBLIC_KEY + LANGFUSE_SECRET_KEY
from langfuse import observe, get_client
langfuse = get_client()
@observe() # this function becomes a span
def vector_search(vec):
return ["paneer butter masala recipe", "dal tadka recipe"]
@observe(as_type="generation") # a generation: records model + tokens
def llm_generate(question, docs):
# a wrapped model client (langfuse.anthropic / langfuse.openai) reports
# tokens and cost automatically, so you do not count them by hand
return "Simmer the paneer in the masala and serve it with jeera rice."
@observe() # the top-level span is the trace
def answer(question, user_id):
langfuse.update_current_trace(user_id=user_id, session_id="s-42", release="v2")
docs = vector_search([0.1] * 8)
return llm_generate(question, docs)
answer("How do I make paneer masala?", user_id="aditi")
langfuse.flush() # push the trace before the script exits
▶ Example output (the trace as it appears in the Langfuse dashboard)
TRACE rag_answer 212ms $0.002310 user=aditi release=v2
|- answer 212ms
|- vector_search 141ms
|- llm_generate (generation) 51ms claude-sonnet-4
320 in / 90 out $0.002310
Scores: judge_relevance 0.90
What happened here: This block needs a running Langfuse instance and Application Programming Interface (API) keys, so the output above is a representative view of what the dashboard renders, not bytes from my terminal. The shape is the honest part: the same trace, the same tree, the same tokens and cost, now stored, searchable, and charted over time instead of printed once and gone. The update_current_trace call is how user_id, session_id, and the release tag get onto the trace, which is exactly what powers the per-user and regression views coming up. Everything past here uses our runnable tracer again, because the analysis logic is identical no matter who stores the traces.
Debugging From Traces
Traces earn their keep the first time production misbehaves. This is the point where LLM observability stops being theory and starts saving you money. Three failures show up again and again, and each one is a one-liner over a pile of traces. Find the slowest span, and you know which step to optimize. Find the priciest generation, and you have found the fat prompt bloating your bill.
Count how often the same generation fires inside a single trace, and you have caught a silent retry loop, the kind where a transient timeout quietly triples your cost and latency while nobody notices. We load a day of sample traffic and run all three checks.
The sample_traces.py helper you see imported below just hand-builds half a dozen traces with the Trace and Span classes from earlier (one slow search, one fat prompt, one retry loop, plus normal traffic); it lives in the series repo next to 229-obs.py, and this script and the two after it all read from it.
📄 debug_traces.py: slow span, expensive prompt, silent retry loop
from collections import Counter
from sample_traces import build_traces
traces = build_traces()
# 1. Slowest span across every trace: the step actually costing you latency.
slowest = max((c for t in traces for c in t.root.children), key=lambda s: s.ms)
print(f"Slowest span: {slowest.name} at {slowest.ms}ms")
# 2. Most expensive single generation: usually one fat prompt.
priciest = max((g for t in traces for g in t.generations()), key=lambda g: g.cost)
print(f"Priciest call: {priciest.name} at ${priciest.cost:.6f} "
f"({priciest.tokens_in} in / {priciest.tokens_out} out)")
# 3. Silent retry loop: the same generation firing many times in ONE trace.
print("\nRetry-loop check (same generation repeated in one trace):")
for t in traces:
for name, n in Counter(g.name for g in t.generations()).items():
if n >= 3:
print(f" trace {t._name} (user {t.user_id}): "
f"{name} ran {n} times, {t.ms}ms total")
▶ Output
Slowest span: vector_search at 610.0ms Priciest call: llm_generate at $0.007800 (1900 in / 140 out) Retry-loop check (same generation repeated in one trace): trace t3 (user aviraj): llm_generate ran 3 times, 1180.0ms total
What happened here: Three real problems, found by reading structure instead of scrolling logs. The slowest span is a vector search that ballooned to 610ms on one trace, a sign the index needs attention. The priciest call is a generation that ate 1900 input tokens, almost certainly a prompt that stuffed in too much context. And a user named Aviraj hit a trace where the model call ran three times in a row, the classic silent retry that quietly billed triple. In Langfuse each of these is a saved filter or a sort on a column; the logic you just read is what those buttons run under the hood.
Cost per User and Catching Regressions
Two questions land on every AI team within the first month of real traffic. Who is costing us money, and did that last deploy make things worse? Both are trivial once traces carry a user_id and a release tag. Cost per user is a group-and-sum. Regression detection is comparing the average judge score before and after a release, so a prompt tweak that quietly hurt quality gets caught in numbers instead of in angry emails a week later.
📄 analytics.py: cost per user, and a score regression on redeploy
from collections import defaultdict
from statistics import mean
from sample_traces import build_traces
traces = build_traces()
by_user = defaultdict(float)
for t in traces:
by_user[t.user_id] += t.cost
print("Cost per user:")
for user, cost in sorted(by_user.items(), key=lambda kv: -kv[1]):
print(f" {user:<8} ${cost:.6f}")
print(f"\nTotal traces: {len(traces)} "
f"Total cost: ${sum(t.cost for t in traces):.6f} "
f"Avg cost/trace: ${mean(t.cost for t in traces):.6f}")
# A prompt change shipped as release v2. Did the judge score move?
def avg_score(release):
return mean(t.scores["judge_relevance"][0] for t in traces if t.release == release)
before, after = avg_score("v1"), avg_score("v2")
print(f"\nRegression check on judge_relevance:")
print(f" v1 avg score: {before:.3f}")
print(f" v2 avg score: {after:.3f}")
drop = before - after
print(f" ALERT: score dropped {drop:.3f} after v2. Roll back." if drop > 0.1
else f" OK: no meaningful drop ({drop:.3f}).")
▶ Output
Cost per user: anvay $0.012421 aviraj $0.006930 aditi $0.004621 Total traces: 6 Total cost: $0.023972 Avg cost/trace: $0.003995 Regression check on judge_relevance: v1 avg score: 0.850 v2 avg score: 0.550 ALERT: score dropped 0.300 after the v2 deploy. Roll back.
What happened here: The cost view instantly ranks users, so a single heavy account stops being a mystery line on the invoice. The regression check is the one to frame on your wall: someone shortened the system prompt and shipped it as v2, the average relevance score fell from 0.85 to 0.55, and the script flags a rollback. That is a quality regression caught by data, not by a customer complaint. Run this same comparison automatically on every deploy and you have a canary that watches production quality the way your tests watch code.
Sessions, Feedback, and Eval Scores on a Trace
A trace is not finished when the response is sent. Quality signals arrive afterward, from three directions, and all of them attach to the same trace object. A session ties many traces from one conversation together, so you can follow a whole chat, not just one turn. User feedback is the thumbs up or down a person clicks. And an eval score is the automated judgment from the LLM-as-a-judge pipeline. Land all three on the trace and LLM observability gives you the one view that matters most in production: the review queue of answers a human should actually look at.
📄 scores.py: attach feedback and judge scores, then build a review queue
from sample_traces import build_traces
traces = build_traces()
by_name = {t._name: t for t in traces}
# 1. A user thumbs-down (explicit human feedback) on one trace.
by_name["t6"].score("user_feedback", 0.0, comment="answer felt generic")
# 2. An automated judge score from the LLM-as-a-judge pipeline (previous lesson) is
# already on every trace as judge_relevance.
# 3. A cheap heuristic score you can compute in-line: was the answer length sane?
for t in traces:
out = sum(g.tokens_out for g in t.generations())
t.score("length_ok", 1.0 if 40 <= out <= 400 else 0.0)
# The payoff: the traces a human should review, low judge score OR a thumbs-down.
print("Review queue (judge < 0.65 or a thumbs-down):")
for t in traces:
judge = t.scores["judge_relevance"][0]
fb = t.scores.get("user_feedback", (None,))[0]
if judge < 0.65 or fb == 0.0:
note = t.scores.get("user_feedback", ("", ""))[1]
print(f" {t._name} user={t.user_id:<7} judge={judge:.2f}"
f" feedback={fb} {note}")
▶ Output
Review queue (judge < 0.65 or a thumbs-down): t4 user=aditi judge=0.55 feedback=None t5 user=anvay judge=0.50 feedback=None t6 user=anvay judge=0.60 feedback=0.0 answer felt generic
What happened here: Three score sources, one trace, one queue. The judge caught two weak answers on their own; the thumbs-down flagged a third where the judge was borderline but the human disagreed. That last row is why you keep both signals: automated scores scale, human feedback grounds them. In Langfuse this queue becomes an annotation view your team works through, and those human labels feed straight back into a better eval set. Observability, evaluation, and feedback stop being separate tools and become one loop.
Observability Without Lock-In: OpenTelemetry
The tools in this space are young and they move fast, so bet on the concept, not the vendor. The concept is the trace-span-generation tree you just built, and it is standardized under OpenTelemetry, the same open tracing standard that regular backend services have used for years. Langfuse is OpenTelemetry-native, which is the important detail: your instrumentation is not really Langfuse instrumentation, it is OpenTelemetry instrumentation that Langfuse happens to read. Point the same spans at a different backend and they still work.
At the time of writing, Langfuse is the popular open-source, self-hostable choice, which is why we used it here. It is not the only good option, and you should pick by fit. LangSmith from the LangChain team is polished and tightly integrated if you already live in LangChain, though it is a hosted commercial product. Arize Phoenix is open source, OpenTelemetry-first, and strong on embeddings and evaluation. MLflow 3 added GenAI tracing, which is handy if your team already runs MLflow for classic model tracking.
They all record the same tree. Because your app emits OpenTelemetry spans, switching between them is a config change, not a rewrite, and that portability is the real insurance against any one of these LLM observability tools fading.
Common Mistakes
- Logging text instead of tracing structure: a wall of print statements cannot tell you which step was slow or how the tree branched. Record spans, not sentences.
- No cost or token capture: a trace without tokens and dollars answers the latency question but leaves the bill a mystery. Capture cost on every generation from day one.
- Forgetting to flush: most SDKs send traces in the background, so a script that exits immediately can drop its last traces. Call flush before a short-lived process ends.
- No user or release tag: without user_id and a release version on the trace, you cannot do per-user cost or catch a regression after a deploy. Tag every trace.
- Putting raw secrets or full user data in spans: traces are readable by your team, so scrub API keys and sensitive fields before they land in a span, or self-host so the data never leaves your network.
Best Practices
- Trace at the right grain: one span per meaningful step, retrieval, rerank, generation, tool call. Too coarse hides the problem, too fine drowns you in noise.
- Tag traces with user, session, and release: these three fields unlock per-user cost, full-conversation views, and regression detection, and they cost one line each.
- Score a sample of production, not all of it: running a judge on every live trace gets expensive. Sample a slice, and always score the ones that got a thumbs-down.
- Gate deploys on the regression check: compare the average score before and after a release automatically, and block the rollout if quality drops past a threshold.
- Stay on OpenTelemetry: emit standard spans so you can change observability vendors with a config edit instead of a rewrite. Bet on the standard, not the logo.
Conclusion
You built LLM observability from the ground up: a tracer that turns one request into a tree of spans and generations, real latency and cost on every node, then debugging, per-user cost, regression detection, and a review queue fed by judge scores and human feedback. That is the exact set of views Langfuse, LangSmith, Phoenix, and MLflow give you, and now you know what each button runs underneath. Pair this with an eval framework and you can ship LLM features fast without flying blind, because a slow step, a fat prompt, or a quality dip becomes a number you can see the same day it happens.
For the full path from Python basics to production AI systems, head back to the Python + AI/ML tutorial series home and pick your next stop.
Frequently Asked Questions
What is LLM observability in simple terms?
LLM observability is recording every request to a language model app as a structured, timed tree instead of a flat log. Each request is a trace, each step inside it is a span, and each model call is a generation that also stores tokens and cost. On top you attach quality scores. That structure lets you answer which step was slow, which prompt was expensive, and whether quality dropped after a deploy, none of which a plain log can tell you.
What is the difference between a trace, a span, and a generation?
A trace is one full user request from start to finish. A span is any single step inside that trace, with a start and end time, like a retrieval or a database read. A generation is a special span for a model call, so besides timing it records the model name, input and output token counts, and therefore cost. Spans nest to form a tree whose root is the trace. This data model is the same across Langfuse, LangSmith, and raw OpenTelemetry.
How is observability different from evaluation?
Evaluation runs on a curated test set before you ship and tells you whether the app is good on known inputs. Observability runs on live traffic after you ship and tells you what real users are actually hitting. They work best together: the same judge that scores your eval set also scores a sample of production traces, so a quality dip in the wild trips the same alarm as a failing test. Teams run both, not one or the other.
Is Langfuse free and can I self-host it?
Langfuse is open source and self-hostable, so you can run it with Docker on your own infrastructure and keep every trace inside your network, which matters when prompts contain user data. There is also a managed cloud option if you would rather not operate it. Because it is OpenTelemetry-native, the instrumentation you write is portable, so you are not locked in even if you start on the free self-hosted path.
How do I track cost per user or per feature?
Tag each trace with a user_id, and optionally a session_id and a release version, when it starts. Cost then becomes a group-and-sum over traces: sum the cost of every trace belonging to a user to get their spend, or group by feature to see which feature drives the bill. The same tags let you compare average quality scores across releases to catch a regression after a deploy.
Interview Questions on LLM Observability
Try each one aloud first. The phrasing you produce under mild pressure is what interviews measure.
Q: Define trace, span, and generation, and explain how they relate.
A trace is one complete user request. A span is any timed step inside it, and spans nest to form a tree whose root is the trace. A generation is a special span for a model call: on top of timing it records the model, input and output tokens, and cost. Latency lives on every node, cost lives only on generations, and quality scores live on the trace. That separation is the whole data model and it is standardized by OpenTelemetry, so it carries across every observability tool.
Q: Why is plain logging not enough for an LLM application?
LLM apps are non-deterministic, expensive per call, and fuzzy in quality. Flat logs cannot show how a request branched, cannot roll up token cost per request or per user, and cannot express a quality score. Tracing records the branching structure, attaches cost to each model call, and lets you hang scores on the trace. So instead of grepping text you sort a column to find the slowest step or the fattest prompt, and you can compare quality across releases.
Q: A deploy went out and users say answers got worse, but nothing errored. How do you confirm it?
Tag traces with a release version, then compare the average judge score before and after the deploy. If the mean relevance score dropped past a threshold, that is your confirmation, and you roll back. Nothing throws an exception in a quality regression, so error rates stay green while the product gets worse. Only a score attached to production traces catches it, ideally automatically as a canary check on every release.
Q: How would you detect a silent retry loop that is inflating cost?
Count how many times the same generation runs inside a single trace. One expected model call per step is normal; the same generation firing three or five times in one trace usually means a transient error was retried without anyone logging it, which triples cost and latency invisibly. Tracing makes it a one-line check over the tree. A flat log would just show more model calls with no signal that they all belonged to one stuck request.
Q: Why does OpenTelemetry matter when choosing an observability tool?
OpenTelemetry is the open standard for traces and spans, so if your app emits OpenTelemetry-native instrumentation you are not tied to one vendor. Langfuse, Arize Phoenix, and others read the same standard, so switching backends is a config change rather than a rewrite. Given how fast this tooling moves, betting on the standard instead of a single product is the practical way to keep your instrumentation alive across tool changes.
Further reading: for the full reference, see Hugging Face documentation.
Related Posts
Previous: LLM-as-a-Judge: Online Evals, CI Gates, and Canary Sets
Next: RAG Tutorial in Python: LlamaIndex & Retrieval-Augmented Generation
Series Home: Python + AI/ML Tutorial Series
No comment