This Python log parser project takes five separate skills you have met one at a time, regular expressions, dataclasses, the Counter, datetime, and argparse, and welds them into a single command line tool you would actually reach for at work. Server logs are where a lot of real debugging starts, and by the end of this post you will have a tool that reads a ten thousand line access log and answers real questions about it in under a second.

“Data is a precious thing and will last longer than the systems themselves.”

Tim Berners-Lee

Last Updated: July 2026 | Tested on: Python 3.14.6 | Difficulty: Intermediate | Reading Time: 19 minutes

Picture the receptionist at a busy clinic who writes one line in a register for every visitor: who came in, at what time, what they asked for, and whether they were helped or turned away. By the end of the week that register has thousands of lines. If someone asks “which hour was the busiest?” or “how many people were turned away?”, flipping pages by hand is hopeless. A web server keeps exactly this kind of register, called an access log, and our job is to teach Python to read it and answer those questions in one command.

What We Are Building

Before writing any code, we decide what “done” looks like. It is the same habit as writing a shopping list before you enter the shop, so you know when you are finished instead of wandering. Here is the acceptance criteria for the Python log parser we are naming logparse. Keep it beside you and tick each box as we go.

• Parse an Apache-style access log into structured records, skipping any line that does not fit
• Report the overall error rate and a breakdown by status code
• List the top N busiest IP addresses
• Find the minutes with the worst spikes of server errors
• Expose all of it through a real command line, like logparse errors access.log --top 10

The diagram below shows how a single log line travels through the tool, from raw text to a ranked report. We will build the pipeline in exactly that order, one step per section.

Our test log is a generated file of ten thousand lines in the common Apache format. One line looks like this, and every parser in this post is built to read exactly this shape.

📄 a single line from access.log

203.0.113.7 - - [09/Jul/2026:14:05:31 +0000] "POST /api/login HTTP/1.1" 500 128 "-" "curl/8.5.0"

Step 1: A Regex That Reads One Line

A log line has a fixed grammar: an IP, then a timestamp in square brackets, then a quoted request, then a status code and a byte count. That regular grammar is exactly what a regular expression is for. The trick that keeps a big pattern readable is named groups, written (?P<name>...). Instead of pulling fields out by number like match.group(3), you ask for match.group("status"), which reads like plain English. If regex still feels shaky, the regular expressions tutorial and its worked examples are the place to warm up.

Think of named groups like labelled compartments in a lunch box. You do not reach for “the third compartment”, you reach for “the rice”. Here is the pattern, split across lines with a comment on each piece so you can see what every part matches.

📄 b1_regex.py: one pattern with named groups

import re

LOG_PATTERN = re.compile(
    r'(?P<ip>\d+\.\d+\.\d+\.\d+) '      # client IP
    r'\S+ \S+ '                          # identity and user (unused, "-")
    r'\[(?P<time>[^\]]+)\] '             # [09/Jul/2026:14:05:31 +0000]
    r'"(?P<method>[A-Z]+) (?P<path>\S+) [^"]+" '  # "GET /api/orders HTTP/1.1"
    r'(?P<status>\d{3}) '                # 200, 404, 500
    r'(?P<size>\d+)'                     # bytes sent
)

sample = '203.0.113.7 - - [09/Jul/2026:14:05:31 +0000] "POST /api/login HTTP/1.1" 500 128 "-" "curl/8.5.0"'

m = LOG_PATTERN.match(sample)
if m:
    for field, value in m.groupdict().items():
        print(f"{field:>8} : {value}")

▶ Output

      ip : 203.0.113.7
    time : 09/Jul/2026:14:05:31 +0000
  method : POST
    path : /api/login
  status : 500
    size : 128

What happened here: the six adjacent strings are automatically joined by Python into one pattern, which lets us comment each field without breaking the regex. re.compile builds the pattern once so it is fast to reuse across ten thousand lines. The star of the show is m.groupdict(), which hands back every named group as a plain dictionary. That dictionary is the bridge to the next step, where we turn it into a proper record.

Step 2: From Match to Record

A dictionary of strings is a fine start, but everything in it is text, including the timestamp and the status code. We want a real datetime we can compare, and a real int we can do maths on. The clean way to hold a fixed set of typed fields is a dataclass: you declare the field names and their types once, and Python writes the boring constructor for you. If dataclasses are new, the dataclasses tutorial covers them end to end.

A dataclass is like a labelled form template. Every log line fills in the same blanks, so once you have the template, each record is just that form filled out. Here we parse the whole file into a list of these records.

📄 b2_parse.py: turn each match into a typed LogEntry

import re
from dataclasses import dataclass
from datetime import datetime

LOG_PATTERN = re.compile(
    r'(?P<ip>\d+\.\d+\.\d+\.\d+) \S+ \S+ '
    r'\[(?P<time>[^\]]+)\] '
    r'"(?P<method>[A-Z]+) (?P<path>\S+) [^"]+" '
    r'(?P<status>\d{3}) (?P<size>\d+)'
)


@dataclass
class LogEntry:
    ip: str
    when: datetime
    method: str
    path: str
    status: int
    size: int


def parse_line(line):
    m = LOG_PATTERN.match(line)
    if m is None:
        return None
    g = m.groupdict()
    return LogEntry(
        ip=g["ip"],
        when=datetime.strptime(g["time"], "%d/%b/%Y:%H:%M:%S %z"),
        method=g["method"],
        path=g["path"],
        status=int(g["status"]),
        size=int(g["size"]),
    )


def parse_file(path):
    entries, skipped = [], 0
    with open(path, encoding="utf-8") as f:
        for line in f:
            entry = parse_line(line)
            if entry is None:
                skipped += 1
            else:
                entries.append(entry)
    return entries, skipped


if __name__ == "__main__":
    entries, skipped = parse_file("access.log")
    print(f"Parsed {len(entries)} entries, skipped {skipped} malformed lines.")
    first = entries[0]
    print(f"First entry: {first.ip} hit {first.path} -> {first.status} at {first.when:%Y-%m-%d %H:%M:%S}")
    print(f"Type of .when: {type(first.when).__name__}")

▶ Output

Parsed 10000 entries, skipped 0 malformed lines.
First entry: 192.168.1.4 hit /api/orders -> 403 at 2026-07-09 08:00:01
Type of .when: datetime

What happened here: parse_line returns None for anything that does not match, so parse_file can quietly count skipped junk instead of crashing on it. All ten thousand of our lines are well formed, so nothing was skipped. The important line is datetime.strptime(g["time"], "%d/%b/%Y:%H:%M:%S %z"), which turns the timestamp text into a real datetime, proven by the last line of output. The %z at the end reads the +0000 zone offset, so our records are timezone aware and safe to compare later.

Step 3: Counting with Counter

Now that every line is a tidy record, the questions get easy. “How many of each status code?” and “which IPs are noisiest?” are both just counting, and the standard library has a tool built for precisely that: collections.Counter. You feed it any sequence and it hands back a tally, and most_common(n) gives you the top n already sorted. The collections module tutorial goes deeper on it.

A Counter is like the tally sheet a shopkeeper keeps by the till, one stroke per item sold. At the end of the day the strokes are already grouped and counted. Here we count status codes and IPs, then compute the error rate.

📄 b3_counter.py: tallies and an error rate

from collections import Counter
from b2_parse import parse_file

entries, _ = parse_file("access.log")

status_counts = Counter(e.status for e in entries)
ip_counts = Counter(e.ip for e in entries)

total = len(entries)
errors = sum(c for status, c in status_counts.items() if status >= 400)
error_rate = errors / total * 100

print("Status code breakdown")
for status, count in sorted(status_counts.items()):
    bar = "#" * (count * 40 // total)
    print(f"  {status}: {count:>5}  {bar}")

print(f"\nError rate (>=400): {errors}/{total} = {error_rate:.1f}%")

print("\nTop 5 noisiest IPs")
for ip, count in ip_counts.most_common(5):
    print(f"  {ip:<15} {count:>4} requests")

▶ Output

Status code breakdown
  200:  6940  ###########################
  301:   577  ##
  302:   195
  403:   394  #
  404:  1193  ####
  500:   701  ##

Error rate (>=400): 2288/10000 = 22.9%

Top 5 noisiest IPs
  10.0.0.17        182 requests
  192.168.1.36     173 requests
  192.168.1.13     170 requests
  192.168.1.6      166 requests
  10.0.0.19        163 requests

What happened here: Counter(e.status for e in entries) walks the records once and tallies every status code, no manual dictionary bookkeeping. The little text bars are just "#" * (count * 40 // total), a cheap way to see the shape of the data without a charting library. We call anything >= 400 an error, sum those up, and divide to get the rate. The most_common(5) call does the sorting for us, so the five busiest IPs come out ranked and ready to print.

Step 4: Finding Error Spikes with datetime

An overall error rate hides the story. Twenty percent errors spread evenly all day is annoying, but twenty percent because everything broke for four minutes at 2pm is an outage. To find that, we group the server errors by the minute they happened. Because each record already has a real datetime, we can format it down to the minute and count. The datetime tutorial has the full formatting reference.

This is like sorting a pile of dated receipts into one envelope per day, then seeing which envelope is fattest. Our “envelope” is a minute label like 2026-07-09 14:11, and the fattest one is the spike.

📄 b4_spikes.py: bucket 5xx errors by the minute

from collections import Counter
from b2_parse import parse_file

entries, _ = parse_file("access.log")

# Bucket every 5xx error into its clock minute, then show the busiest buckets.
error_buckets = Counter()
for e in entries:
    if e.status >= 500:
        bucket = e.when.strftime("%Y-%m-%d %H:%M")
        error_buckets[bucket] += 1

print("Minutes with the most 5xx errors")
for minute, count in error_buckets.most_common(6):
    print(f"  {minute}   {count} errors  {'!' * count}")

▶ Output

Minutes with the most 5xx errors
  2026-07-09 14:11   12 errors  !!!!!!!!!!!!
  2026-07-09 14:13   10 errors  !!!!!!!!!!
  2026-07-09 14:12   9 errors  !!!!!!!!!
  2026-07-09 14:14   9 errors  !!!!!!!!!
  2026-07-09 14:16   9 errors  !!!!!!!!!
  2026-07-09 14:00   7 errors  !!!!!!!

What happened here: the whole idea is e.when.strftime("%Y-%m-%d %H:%M"), which chops the timestamp down to a per-minute label and throws away the seconds. Every error in the same minute lands in the same bucket, and the Counter tallies them. The output tells a clear story: the errors cluster hard around 14:11 to 14:16, which is exactly the kind of window you would go dig into. Without the datetime grouping, that spike would be invisible inside the flat 22.9% overall rate.

Step 5: A Real CLI with argparse

Right now each question needs its own script. A real tool takes the question as an argument, like logparse errors access.log or logparse top access.log --top 10. The standard library module for this is argparse, which parses the words after your program name into tidy options, generates a help screen for free, and rejects bad input with a clear message. For the fuller tour, including the third party click and typer alternatives, see the Command-Line Interface (CLI) tools tutorial.

Think of argparse as the order form at a canteen counter. You tick the dish (the command), fill in a couple of options (how many, what format), and the counter either serves you or points out that you asked for something not on the menu. Here is the finished tool, which we will then drive with four different orders.

First, the help screen argparse builds automatically. You never write this text, it comes from your argument definitions.

▶ Output of: python logparse.py –help

usage: logparse [-h] [--since SINCE] [--top TOP] [--json]
                {errors,top,spikes} logfile

Analyze Apache-style access logs.

positional arguments:
  {errors,top,spikes}  what to report
  logfile              path to the access log (.log or .gz)

options:
  -h, --help           show this help message and exit
  --since SINCE        only entries newer than e.g. 30m, 1h, 2d
  --top TOP            how many rows to show
  --json               emit JSON instead of text

Now four real runs, one per flag combination, so you can see how the same tool answers different questions.

▶ Output of four commands

$ python logparse.py errors access.log
Errors: 2288/10000 requests (22.9%)
  403: 394
  404: 1193
  500: 701

$ python logparse.py top access.log --top 10
Top 10 IPs by request count
  10.0.0.17       182
  192.168.1.36    173
  192.168.1.13    170
  192.168.1.6     166
  10.0.0.19       163
  10.0.0.23       163
  10.0.0.24       163
  192.168.1.4     162
  192.168.1.7     162
  192.168.1.9     161

$ python logparse.py spikes access.log --top 3
Busiest minutes for 5xx errors
  2026-07-09 14:11  12 errors
  2026-07-09 14:13  10 errors
  2026-07-09 14:12  9 errors

$ python logparse.py errors access.log --json
{
  "total": 10000,
  "errors": 2288,
  "error_rate_pct": 22.88,
  "by_status": {
    "403": 394,
    "404": 1193,
    "500": 701
  }
}

What happened here: one program, four questions, no code change between runs. The errors, top, and spikes words pick the report; --top controls how many rows come back; and --json flips the whole output from human text to machine readable JSON, which is what you would pipe into another tool. The --top 10 versus --top 3 difference shows the flag actually threading through to the report function.

The Full Tool

Here is the complete Python log parser assembled into one file, about 110 lines. Save it as logparse.py next to your access.log and run it with python logparse.py errors access.log. Every command from the runs above is served by this single file.

📄 logparse.py: the complete log analysis CLI

"""logparse: a tiny log analysis CLI built from regex, dataclasses, Counter, and argparse."""
import argparse
import gzip
import json
import re
from collections import Counter
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone

LOG_PATTERN = re.compile(
    r'(?P<ip>\d+\.\d+\.\d+\.\d+) \S+ \S+ '
    r'\[(?P<time>[^\]]+)\] '
    r'"(?P<method>[A-Z]+) (?P<path>\S+) [^"]+" '
    r'(?P<status>\d{3}) (?P<size>\d+)'
)


@dataclass
class LogEntry:
    ip: str
    when: datetime
    method: str
    path: str
    status: int
    size: int


def parse_line(line):
    m = LOG_PATTERN.match(line)
    if m is None:
        return None
    g = m.groupdict()
    return LogEntry(
        ip=g["ip"],
        when=datetime.strptime(g["time"], "%d/%b/%Y:%H:%M:%S %z"),
        method=g["method"],
        path=g["path"],
        status=int(g["status"]),
        size=int(g["size"]),
    )


def open_log(path):
    """Open plain or .gz logs transparently."""
    if path.endswith(".gz"):
        return gzip.open(path, "rt", encoding="utf-8")
    return open(path, encoding="utf-8")


def load_entries(path, since=None):
    entries = []
    with open_log(path) as f:
        for line in f:
            entry = parse_line(line)
            if entry is None:
                continue
            if since is not None and entry.when < since:
                continue
            entries.append(entry)
    return entries


def parse_since(text, now):
    """Turn '1h', '30m', '2d' into a cutoff datetime relative to now."""
    unit = text[-1]
    amount = int(text[:-1])
    spans = {"m": "minutes", "h": "hours", "d": "days"}
    if unit not in spans:
        raise argparse.ArgumentTypeError(f"bad --since '{text}', use 30m, 1h or 2d")
    return now - timedelta(**{spans[unit]: amount})


def cmd_errors(entries, args):
    errs = [e for e in entries if e.status >= 400]
    by_status = Counter(e.status for e in errs)
    rate = len(errs) / len(entries) * 100 if entries else 0
    if args.json:
        return {"total": len(entries), "errors": len(errs),
                "error_rate_pct": round(rate, 2), "by_status": dict(by_status)}
    lines = [f"Errors: {len(errs)}/{len(entries)} requests ({rate:.1f}%)"]
    for status, count in sorted(by_status.items()):
        lines.append(f"  {status}: {count}")
    return "\n".join(lines)


def cmd_top(entries, args):
    counts = Counter(e.ip for e in entries).most_common(args.top)
    if args.json:
        return {"top_ips": [{"ip": ip, "requests": n} for ip, n in counts]}
    lines = [f"Top {args.top} IPs by request count"]
    for ip, n in counts:
        lines.append(f"  {ip:<15} {n}")
    return "\n".join(lines)


def cmd_spikes(entries, args):
    buckets = Counter(e.when.strftime("%Y-%m-%d %H:%M")
                      for e in entries if e.status >= 500)
    top = buckets.most_common(args.top)
    if args.json:
        return {"spikes": [{"minute": m, "errors": n} for m, n in top]}
    lines = ["Busiest minutes for 5xx errors"]
    for minute, n in top:
        lines.append(f"  {minute}  {n} errors")
    return "\n".join(lines)


COMMANDS = {"errors": cmd_errors, "top": cmd_top, "spikes": cmd_spikes}


def build_parser():
    parser = argparse.ArgumentParser(prog="logparse", description="Analyze Apache-style access logs.")
    parser.add_argument("command", choices=COMMANDS, help="what to report")
    parser.add_argument("logfile", help="path to the access log (.log or .gz)")
    parser.add_argument("--since", help="only entries newer than e.g. 30m, 1h, 2d")
    parser.add_argument("--top", type=int, default=5, help="how many rows to show")
    parser.add_argument("--json", action="store_true", help="emit JSON instead of text")
    return parser


def main(argv=None):
    parser = build_parser()
    args = parser.parse_args(argv)
    now = datetime.now(timezone.utc)
    try:
        since = parse_since(args.since, now) if args.since else None
        entries = load_entries(args.logfile, since=since)
    except (ValueError, argparse.ArgumentTypeError) as e:
        parser.error(str(e))
    except FileNotFoundError:
        parser.error(f"no such log file: {args.logfile}")
    result = COMMANDS[args.command](entries, args)
    if args.json:
        print(json.dumps(result, indent=2))
    else:
        print(result)


if __name__ == "__main__":
    main()

What happened here: notice the shape. Parsing lives in one place, each report is its own small function, and COMMANDS is a dictionary that maps the command word straight to the function that handles it. Adding a new report later means writing one function and adding one dictionary entry, nothing else changes. The main function catches a missing file or a bad --since value and turns it into a clean one line error through parser.error, instead of dumping a scary traceback on the user.

Stretch Goals and Git History

The Python log parser works, and two of the classic stretch features are already baked in so you can see how they fit. The --json flag gives machine readable output for piping into other tools, and open_log reads gzipped .gz logs transparently, which matters because rotated server logs are almost always compressed. Here is proof the gzip path works on a compressed copy of the same file.

▶ Output of: python logparse.py errors access.log.gz

Errors: 2288/10000 requests (22.9%)
  403: 394
  404: 1193
  500: 701

From here, pick a couple more upgrades to make it your own. Each one reuses a skill from earlier in the series.

• A paths report: add a cmd_paths that ranks the most requested URLs, using the same Counter pattern on e.path.
• Bandwidth totals: sum e.size per IP to find who is pulling the most data, not just making the most requests.
• Multiple files: accept several log paths and merge them, so a week of rotated logs reports as one.
• Unit tests: write a test that feeds one known line to parse_line and checks every field, so future edits cannot silently break parsing.

One acceptance criterion we have not ticked yet is version control. A real project grows through a history of small, meaningful commits, not one giant dump at the end. As you build this tool, commit after each working step, so your history reads like the sections of this post. A good first history looks like this:

📄 a clean commit history for the project

git init
git add logparse.py && git commit -m "Add regex and LogEntry parser for access logs"
git commit -am "Add errors report with status breakdown and rate"
git commit -am "Add top IPs report using Counter.most_common"
git commit -am "Add 5xx spike detection bucketed by minute"
git commit -am "Wire up argparse CLI with --since, --top, --json"
git commit -am "Support gzip logs and clean error messages"

Six commits, each one a working step you could check out and run. If Git is still new to you, the Git basics tutorial walks through init, add, and commit from scratch.

Common Mistakes

Mistake 1: Recompiling the regex inside the loop

Calling re.compile or re.match with the raw pattern string on every one of ten thousand lines makes Python re-parse the pattern each time. Compile it once at module level, as we do with LOG_PATTERN, and reuse the compiled object. On a big log the difference is easy to feel.

Mistake 2: Assuming every line matches

Real logs contain surprises: blank lines, half written entries from a crash, or a different format mixed in. If you index into match.group(...) without checking that the match succeeded, one odd line raises AttributeError and kills the run. Returning None for non matches and counting them, as parse_line does, keeps the tool alive and honest about what it skipped.

Mistake 3: Comparing naive and aware datetimes

Our timestamps carry a +0000 offset, so strptime with %z produces timezone aware datetimes. If your --since cutoff were a naive datetime with no zone, comparing the two raises TypeError. That is why we build the cutoff with datetime.now(timezone.utc), keeping both sides aware. Mixing the two is one of the most common datetime bugs there is.

Best Practices

• DO compile regex patterns once and give every group a descriptive name
• DO parse text into typed records early, so the rest of the code works with real dates and ints
• DO keep one small function per report and map commands to them with a dictionary
• DO let argparse generate your help and validate input, rather than parsing sys.argv by hand
• DON’T assume input is clean, count and report the lines you could not parse
• DON’T print reports straight from deep inside logic, return strings or data and print once at the edge, which makes the code testable

Conclusion

You just built a genuinely useful Python log parser, and every piece of it was a skill you had already met alone: regex to read a line, a dataclass to hold it, Counter to tally, datetime to find spikes, and argparse to wrap it all in a real command line. That is the whole lesson of a project post. The building blocks do not change, you just learn to stack them with intent. A tool that reports error rates, noisy IPs, and outage windows from a raw log is something you can drop into a real job on Monday.

Next, try the stretch goals, then point the tool at a log from your own machine or a small server and see what it tells you. And if you want the full path from first steps through AI and machine learning, browse the Python + AI/ML tutorial series home.

Frequently Asked Questions

Interview Questions on This Project

If you can walk through these without peeking, you are ready for this topic in an interview.

Q: Why turn each log line into a dataclass instead of just keeping the groupdict?

The groupdict is all strings, so the timestamp and status code cannot be compared or summed correctly. The dataclass parses them into a real datetime and int once, at the boundary, so every later step works with proper types. It also gives the record a name and a fixed set of fields, which makes the code self documenting and catches typos in field names early.

Q: What does Counter.most_common(n) save you compared to sorting a plain dict?

Counter builds the tally in one pass and most_common(n) returns the top n already sorted by count, descending. With a plain dict you would tally manually, then call sorted with a key on items() and slice. Counter is both less code and clearer intent, and it is tuned for exactly this counting job.

Q: How would you extend the tool to add a new report, say the most requested paths?

Write one function, cmd_paths(entries, args), that counts e.path with a Counter and returns text or a dict, then add "paths": cmd_paths to the COMMANDS dictionary. Because argparse reads its command choices from that same dictionary, the new command appears in the help and validation automatically. Nothing else changes, which is the payoff of the dispatch table design.

Q: Why does the tool return report strings from the command functions instead of printing inside them?

Returning the result and printing once in main keeps the report functions pure and testable: a test can call cmd_errors and assert on the returned value with no output capturing. It also keeps the text and JSON paths in one place. Printing from deep inside logic ties the function to the console and makes it awkward to reuse.

Q: What breaks if a log line has an unexpected format, and how does the tool cope?

LOG_PATTERN.match returns None for a line that does not fit, and parse_line returns None in that case rather than indexing into a non existent match. The loader simply skips those lines. In the standalone parser we count them so the user knows how many were ignored. The tool never crashes on one bad line, which is essential for messy real world logs.

Further reading: the official Python documentation is the authoritative source on this.

Related Posts

Previous: Coding Interview Patterns in Python: Two Pointers, Sliding Window, DP

Next: Python Design Patterns That Still Matter in 2026

Series Home: Python + AI/ML Tutorial Series