Your model works. Your RAG (Retrieval-Augmented Generation) pipeline is evaluated. Your guardrails are in place. Now comes the real ML deployment work: serving live users, handling traffic spikes, restarting after failures, and scaling without you babysitting it. This guide covers the production stack that gets you there: FastAPI for serving, Docker for packaging, and cloud patterns that hold up whether you are answering 100 or 100,000 requests per day.

“A model that nobody can call is just an expensive file on a disk.”

Every ML engineer, eventually

Last Updated: July 2026 | Tested on: Python 3.14.6, FastAPI 0.138.0, uvicorn 0.49.0, Docker 27.x | Difficulty: Expert | Reading Time: 13 minutes

Running a model on your laptop is easy. Getting it to answer thousands of requests with steady latency, scale up when traffic spikes, and stay up when one container dies, that is the real Machine Learning (ML) engineering work. Think of it like the difference between cooking one dish for yourself and running a kitchen during the dinner rush. Same recipe, completely different game. This post walks through the patterns that make the second one work: wrapping your model in a FastAPI service, packaging it in a Docker container, and picking the right scaling shape for your traffic.

None of this is tied to one specific tool. It is a small set of habits you can reuse on any cloud. By the end you will have a serving Application Programming Interface (API) you can actually curl, a container you can ship anywhere, and a clear mental model of when to add a second worker, a second container, or a GPU (Graphics Processing Unit).

FastAPI Model Serving

Tip: click the diagram to open it full screen, then use the zoom and pan controls for a closer look.

Here is the map for this python ml production deployment. One trained model can be served three ways: a live REST API for instant predictions, batch jobs for scheduled bulk scoring, or edge deployment for phones and IoT (Internet of Things) devices. The scaling column shows how you grow (a load balancer in front of several replicas, model caching and GPU pooling, auto-scaling with Kubernetes HPA, the Horizontal Pod Autoscaler), and the monitoring column shows what you watch (latency and throughput, data drift, slow decay in model quality).

Monitoring matters most, because a model rots quietly as real-world data drifts away from the data it learned on. Skip it and your ML deployment only tells you something is wrong when users start complaining.

We start with the REST API, since that is what most teams ship first. The one rule that beats every other tip here: load the model once when the app starts, not on every request. Picture a chef who reads the entire recipe from scratch for every single order versus one who memorizes it before service begins. Loading a model can take a few seconds; a prediction takes a few milliseconds. FastAPI gives us a lifespan hook that runs once at startup, so an ML engineer named Viraj loads the model there and keeps it in memory for every request that follows.

📄 serve_model.py: a production ML API with FastAPI

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
from contextlib import asynccontextmanager
import joblib
import numpy as np
import time

# Viraj builds a production ML API

# Global model (loaded once at startup)
model = None

@asynccontextmanager
async def lifespan(app: FastAPI):
    """Load model at startup, cleanup at shutdown."""
    global model
    model = joblib.load("model.joblib")
    print(f"Model loaded: {type(model).__name__}")
    yield
    print("Shutting down, cleaning up...")

app = FastAPI(
    title="ML Prediction API",
    version="1.0.0",
    lifespan=lifespan,
)

class PredictionRequest(BaseModel):
    features: list[float] = Field(..., min_length=4, max_length=4)

class PredictionResponse(BaseModel):
    prediction: int
    probability: float
    latency_ms: float
    model_version: str = "1.0.0"

@app.post("/predict", response_model=PredictionResponse)
async def predict(request: PredictionRequest):
    """Make a prediction with the loaded model."""
    start = time.perf_counter()

    try:
        X = np.array(request.features).reshape(1, -1)
        pred = model.predict(X)[0]
        proba = model.predict_proba(X)[0].max()
    except Exception as e:
        raise HTTPException(status_code=500, detail=f"Prediction failed: {e}")

    latency = (time.perf_counter() - start) * 1000

    return PredictionResponse(
        prediction=int(pred),
        probability=float(proba),
        latency_ms=round(latency, 2),
    )

@app.get("/health")
async def health():
    return {"status": "healthy", "model_loaded": model is not None}

# Run: uvicorn serve_model:app --host 0.0.0.0 --port 8000 --workers 4

▶ Output (curl test)

$ curl -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"features": [5.1, 3.5, 1.4, 0.2]}'

{
  "prediction": 0,
  "probability": 0.9815737196632449,
  "latency_ms": 0.82,
  "model_version": "1.0.0"
}

$ curl http://localhost:8000/health
{"status": "healthy", "model_loaded": true}

What happened here: we sent four iris flower measurements and got back class 0 with about 98 percent confidence in under a millisecond. Those numbers are the real output of a small scikit-learn LogisticRegression model called through FastAPI’s test client, not made-up values. Pydantic checked that the request had exactly four floats before the model ever ran, and the /health endpoint reports that the model is loaded. The probability is the full float the model returns, so do not be surprised that it is not a tidy 0.98. Your latency_ms will differ from machine to machine, which is exactly why we measure it per request instead of hard-coding a number.

Docker Containerization

The API runs on Viraj’s laptop. Now a teammate named Niranjan needs it to run the same way on a cloud server he has never touched. That is exactly what Docker solves. A container is like a shipping container at a port: it does not care whether it sits on a truck, a ship, or a train, because everything it needs is sealed inside. We pin the Python version, copy in the code and the saved model, drop root for safety, and add a health check so the orchestrator can tell whether the service is alive.

📄 Dockerfile: a production container for the ML service

FROM python:3.14-slim

WORKDIR /app

# Install dependencies first (Docker layer caching)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application code and model
COPY serve_model.py .
COPY model.joblib .

# Non-root user for security
RUN useradd -m appuser
USER appuser

# Health check
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
  CMD curl -f http://localhost:8000/health || exit 1

EXPOSE 8000

CMD ["uvicorn", "serve_model:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "4"]

One small catch worth knowing: the slim base image does not ship curl, so the health check above needs it installed first (add RUN apt-get update && apt-get install -y --no-install-recommends curl before the HEALTHCHECK line), or swap the check to a tiny Python one-liner with python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')". The order of the lines is deliberate. Dependencies are copied and installed before the app code, so Docker caches that slow layer and only rebuilds it when requirements.txt actually changes. Edit your Python file and the rebuild is nearly instant.

📄 Build and deploy commands

# Build the image
docker build -t ml-api:1.0.0 .

# Run locally
docker run -d -p 8000:8000 --name ml-api ml-api:1.0.0

# Push to a registry
docker tag ml-api:1.0.0 registry.example.com/ml-api:1.0.0
docker push registry.example.com/ml-api:1.0.0

# Deploy to cloud (example: AWS ECS, GCP Cloud Run, Azure Container Apps)
# Cloud Run (simplest):
gcloud run deploy ml-api \
  --image registry.example.com/ml-api:1.0.0 \
  --port 8000 \
  --memory 2Gi \
  --cpu 2 \
  --min-instances 1 \
  --max-instances 10

Scaling Patterns

Scaling is like opening checkout lanes at a supermarket. One lane is fine at 7 in the morning. At 6 in the evening you open more, and when the rush ends you close them again. You do not build a 40-lane megastore for a shop that gets busy twice a day. The same logic applies to ML deployment: match the pattern to your real traffic and budget, not to the biggest system you have read about. An engineer named Prathamesh keeps this cheat sheet for his team so nobody over-engineers a weekend project or under-provisions a launch.

📄 scaling_patterns.py: when and how to scale ML services

# Prathamesh documents scaling patterns for his team
scaling_patterns = {
    "Single Server + Multiple Workers": {
        "when": "< 1,000 RPM (requests per minute)",
        "how": "uvicorn --workers 4 (one worker per CPU core)",
        "cost": "$20-50/month (single VM or container)",
        "example": "Small internal tool, MVP, low-traffic API",
    },
    "Horizontal Scaling (multiple containers)": {
        "when": "1,000 - 50,000 RPM",
        "how": "Cloud Run / ECS auto-scaling (2-20 instances)",
        "cost": "$100-500/month (auto-scales to zero when idle)",
        "example": "Production API, customer-facing chatbot",
    },
    "GPU Serving (model inference)": {
        "when": "Running local LLMs, PyTorch models, or YOLO",
        "how": "vLLM (standard), SGLang, or NVIDIA Triton",
        "cost": "$500-2000/month per GPU instance",
        "example": "Self-hosted LLM, real-time computer vision",
    },
    "Serverless (event-driven)": {
        "when": "Sporadic traffic, batch processing",
        "how": "AWS Lambda, Google Cloud Functions (with container)",
        "cost": "Pay per invocation (~$0.001 per request)",
        "example": "Webhook handlers, async processing pipelines",
    },
}

print("ML Deployment Scaling Patterns\n")
for pattern, info in scaling_patterns.items():
    print(f"  {pattern}")
    for key, val in info.items():
        print(f"    {key:>8}: {val}")
    print()

▶ Output

ML Deployment Scaling Patterns

  Single Server + Multiple Workers
        when: < 1,000 RPM (requests per minute)
         how: uvicorn --workers 4 (one worker per CPU core)
        cost: $20-50/month (single VM or container)
     example: Small internal tool, MVP, low-traffic API

  Horizontal Scaling (multiple containers)
        when: 1,000 - 50,000 RPM
         how: Cloud Run / ECS auto-scaling (2-20 instances)
        cost: $100-500/month (auto-scales to zero when idle)
     example: Production API, customer-facing chatbot

  GPU Serving (model inference)
        when: Running local LLMs, PyTorch models, or YOLO
         how: vLLM (standard), SGLang, or NVIDIA Triton
        cost: $500-2000/month per GPU instance
     example: Self-hosted LLM, real-time computer vision

  Serverless (event-driven)
        when: Sporadic traffic, batch processing
         how: AWS Lambda, Google Cloud Functions (with container)
        cost: Pay per invocation (~$0.001 per request)
     example: Webhook handlers, async processing pipelines

A quick word on that GPU serving row, because it is the fastest-moving box on the whole diagram and the one most likely to date a tutorial. The pattern is what matters and it is stable: put your model behind a fast batching server that exposes an OpenAI-compatible endpoint, then scale that server like any other container. The engine is the swappable part. At the time of writing (mid-2026), vLLM is the default production engine for self-hosted Large Language Models (LLMs): it does continuous batching and PagedAttention (paged Key-Value cache memory), speaks the OpenAI API so your client code does not change, and is now a core project in the PyTorch ecosystem.

The closest credible alternative is SGLang, which many teams prefer for multi-turn and Retrieval-Augmented Generation (RAG) heavy traffic, and NVIDIA Triton (now packaged as Dynamo-Triton) still leads when you need one server to host models from several frameworks at once. Pin whichever you pick to a specific version and check the vLLM docs for the current release, since this space moves every few weeks.

Two names you will still meet in older guides are worth skipping for new work. Hugging Face Text Generation Inference (TGI) went into maintenance in December 2025 and its repository was archived read-only in March 2026, with Hugging Face itself now pointing users to vLLM or SGLang. TorchServe, the old PyTorch-native server, was archived in August 2025. Neither gets new features or security fixes, so treat them as historical references, not starting points.

If you are serving on Kubernetes and want scale-to-zero, canary rollouts, and traffic splitting handled for you, KServe (a Cloud Native Computing Foundation project) wraps a vLLM or Triton runtime behind a single InferenceService, so you keep the same serving pattern and let the platform do the scaling.

Common Mistakes

Practice Exercises

1. Exercise 1: Train any small scikit-learn model, save it with joblib, and wrap it in the FastAPI app from this post. Confirm /predict and /health both return real responses with the test client.
2. Exercise 2: Add a /predict-batch endpoint that scores a list of inputs in one call, and put the model version in the response so callers always know which model answered them.
3. Exercise 3: Build the Docker image, run it locally, and pick a scaling pattern from the cheat sheet for an imaginary launch with 10,000 requests per minute. Write down which one you chose and why.

More in this series:

• LLM Serving: Ollama Locally, vLLM in Production
• From Idea to AI MVP: The Right Way to Build
• Choosing an AI Tech Stack: 3 Real Case Studies

Frequently Asked Questions

Interview Questions on ML Deployment at Scale

Scenario questions, not trivia: this is the form this topic takes in a real interview.

Q: Why should the model be loaded once at startup instead of inside the request handler?

Loading a model from disk can take seconds because it involves reading the file and rebuilding the object in memory, while a single prediction usually takes milliseconds. If you load it inside the endpoint, every request pays that startup cost and your latency becomes unusable under load. FastAPI's lifespan hook runs once when the app boots, so you load the model there, keep it in a module-level variable, and reuse it for every request.

Q: What does uvicorn --workers 4 actually do, and what is the memory tradeoff?

It forks four separate worker processes, each running its own copy of your app, so they can serve requests in parallel across Central Processing Unit (CPU) cores (Python's Global Interpreter Lock (GIL) means one process cannot use multiple cores for CPU-bound work). The catch is that every worker loads its own copy of the model into memory, so four workers with a 500 MB model use roughly 2 GB. Size your container memory for workers times model size, not just one copy.

Q: Your /predict API returns correct results, but over a few hours latency creeps up and memory keeps climbing until the container is killed. What do you check first?

This is the classic slow memory leak. First confirm the model is loaded once at startup and not re-loaded or re-fitted per request. Then look for unbounded state that grows with each call: appending to a global list, an in-memory cache with no eviction, or request objects held by references that never get released. Add memory metrics per request, reproduce with a load test, and compare a fresh container against one that has served traffic for an hour to isolate what grows.

Q: The container image builds and runs fine on your laptop, but the orchestrator keeps restarting it in production. Where do you look?

Start with the health check, because a failing /health is the most common reason an orchestrator kills a container. Remember the python:3.14-slim base image does not ship curl, so a HEALTHCHECK that calls curl silently fails and marks the container unhealthy. Also check that the app binds to 0.0.0.0 and not 127.0.0.1, that the exposed port matches, and that the memory limit is high enough for all your uvicorn workers plus the model.

Q: How do you decide between a single server with multiple workers, horizontal scaling, and serverless for an ML service?

Match the pattern to real traffic and budget. Under about 1,000 requests per minute, one VM or container with a few uvicorn workers is the cheapest and simplest option. From roughly 1,000 to 50,000 requests per minute, move to horizontal scaling with something like Cloud Run or ECS that adds and removes containers automatically. For sporadic or event-driven traffic, serverless functions let you pay per invocation and scale to zero when idle. Do not reach for the biggest pattern by default, because idle capacity is wasted money.

Q: What is data drift, and why is monitoring for it more important than latency dashboards?

Data drift is when the live input data slowly moves away from the distribution the model was trained on, for example user behavior or prices shifting over months. Unlike a crash or a latency spike, drift causes no error and no alert: the API stays fast and returns confident answers that are quietly getting less accurate. That is why you track prediction distributions and input statistics over time, so you catch quality decay before users do and can trigger a retrain.

Q: Why put a model_version field in the response, and how does it help during a rollout?

Tagging every response with the model version means callers and logs always know exactly which model produced a given prediction, which is essential for debugging and reproducing issues. During a blue-green or canary rollout you can run two versions side by side, compare their outputs on live traffic, and roll back instantly if the new one behaves worse. Without a version stamp you are guessing which model answered when something goes wrong.

What’s Next?

You now know how to take a trained model out of a notebook and put it in front of real users: wrap it in a FastAPI service that loads the model once, package that service in a Docker container that runs the same everywhere, and pick a scaling shape that matches your actual traffic instead of your fears. You also saw why monitoring is not optional, because a model quietly rots as real-world data drifts. That is ML deployment in one sentence: boring, repeatable steps that keep answering requests at 3 a.m. without you.

Next, in the AI capstone project, we bring together everything from the entire 247-post series into one grand finale project: a production-ready AI application built from scratch.

Want the full learning path from first line of Python to shipped AI systems? Start at the Python + AI/ML tutorial series home and work through it in order.

Related Posts

Previous: LLM Serving: Ollama Locally, vLLM in Production

Next: Airflow Tutorial: Your First Data Pipeline, the Right Way

Series Home: Python + AI/ML Tutorial Series