The data you need is sitting right there on a web page: product prices, job listings, book catalogs. No API, no download button, just HTML. Python BeautifulSoup is how you take it anyway: fetch the page with requests, parse the HTML into a searchable tree, and pull out exactly the elements you want. This guide runs the full pipeline, from your first parse to pagination and saving results.
“Information is power. But like all power, there are those who want to keep it for themselves.”
Aaron Swartz
Last Updated: July 2026 | Tested on: Python 3.14.6, beautifulsoup4 4.15.0, requests 2.34.2 | Difficulty: Intermediate | Reading Time: 18 minutes
Here is the easiest way to picture it. A web page is a printed magazine. Your eyes already know how to skim it for the bit you care about, the price in the corner, the title on the cover. A computer just sees one long blob of text. Python BeautifulSoup is the pair of reading glasses that turns that blob into something with structure: headings, paragraphs, links, prices. Once it has structure, you can point at any piece and say “give me that one”.
So scraping is two steps. First you fetch the page with requests.get(), which is the same HTTP (HyperText Transfer Protocol) request your browser fires when you type a URL. Second you hand that HTML to BeautifulSoup, which builds a tree you can search by tag name, class, ID, or CSS selector. After that you pull out text and attributes and store the results however you like.
One honest warning before the fun starts. Scraping is powerful, and that power comes with manners. Check the site’s robots.txt, slow your requests down, set a real User-Agent so the site knows who you are, and never grab personal data without consent. If the site offers an API, use the API. Scraping is Plan B, not Plan A.
Tip: click the diagram to open it full screen, then use the zoom and pan controls for a closer look.
The diagram walks the whole pipeline. You fetch the HTML from a URL with requests, check the status code, parse the HTML into a navigable tree with BeautifulSoup, pick out the data you want with selectors, and write the results to a file or database. Each stage has its own way of going wrong: the network drops at fetch, the markup is messy at parse, an element is missing at extract. A good scraper expects those bumps and keeps going instead of crashing. The code below builds each stage in turn.
Table of Contents
Install and Verify
You need two packages: requests to download pages and beautifulsoup4 to parse them. The import name is bs4, which trips up newcomers, so keep that in mind: you install beautifulsoup4 but you import bs4.
📄 Terminal: install requests and BeautifulSoup
pip install requests beautifulsoup4
📄 Terminal: confirm the install worked
python -c "import bs4; print(bs4.__version__)"
▶ Output
4.15.0
If you see a version number, you are ready. If you see ModuleNotFoundError: No module named 'bs4', the install landed in a different Python than the one you just ran. Make sure you installed into the same virtual environment you are using.
The Quick Win
Before touching the network, get a feel for Python BeautifulSoup on a tiny piece of HTML you control. It is like practicing knots on a short piece of rope before you go climbing: same skill, zero risk. Hand it a string and start asking questions.
📄 quick_win.py: parse a string and pull out pieces
from bs4 import BeautifulSoup
html = "<html><body><h1>Books to Scrape</h1><p class='tag'>fiction</p></body></html>"
soup = BeautifulSoup(html, "html.parser")
print(soup.h1.text) # the first <h1>
print(soup.find("p").text) # the first <p>
print(soup.find("p")["class"]) # its class attribute
▶ Output
Books to Scrape fiction ['tag']
What happened here: BeautifulSoup(html, "html.parser") built the tree. After that, soup.h1 reaches straight for the first <h1> tag like it was an attribute, and .text gives you what is inside it. The ["class"] bit reads an attribute off the tag, and notice it comes back as a list, not a string, because an element can carry several classes. That is the whole mental model: a tag is an object, its text is one property, and its attributes are another.
Your First Real Python BeautifulSoup Scrape
Now point it at a real site. We will use books.toscrape.com, a sandbox built specifically for practicing scraping. Think of it as a driving school’s practice track: real controls, no traffic, so nobody’s server gets hurt while you learn.
📄 first_scrape.py: fetch a page and read it
import requests
from bs4 import BeautifulSoup
# Fetch the page
url = "https://books.toscrape.com/"
headers = {"User-Agent": "Mozilla/5.0 (Python Tutorial Bot)"}
response = requests.get(url, headers=headers)
response.raise_for_status() # Raise an error if the status is not 200
# Parse the HTML into a searchable tree
soup = BeautifulSoup(response.text, "html.parser")
# Grab the page title (.strip() removes the newlines around it)
print(f"Title: {soup.title.string.strip()}")
# Find the first 5 book titles on the page
books = soup.select("article.product_pod h3 a")
for i, book in enumerate(books[:5], 1):
print(f"{i}. {book['title']}")
▶ Output
Title: All products | Books to Scrape - Sandbox 1. A Light in the Attic 2. Tipping the Velvet 3. Soumission 4. Sharp Objects 5. Sapiens: A Brief History of Humankind
What happened here: requests.get() downloaded the HTML and raise_for_status() guards against quietly carrying on after a 404 or 500. Then BeautifulSoup() parsed the page. The selector article.product_pod h3 a reads left to right like a sentence: find every <a> inside an <h3> inside an <article class="product_pod">. We pulled the title attribute off each link. One small but real detail: soup.title.string arrives wrapped in newlines because the page author indented the <title> tag, so .strip() tidies it up.
Finding Elements: Three Approaches
BeautifulSoup gives you three ways to locate elements, and they overlap a lot. find() grabs the first match. find_all() grabs every match. select() takes a CSS selector, the exact same syntax you use in browser DevTools. Picture a library. find("h3") is walking up to a librarian and saying “bring me the first book with a title like this”. select("div.product a") is handing over the full shelf address: “the link inside any product box”. Both fetch a book. One asks in plain words, the other gives precise directions.
📄 find_elements.py: find(), find_all(), and select()
from bs4 import BeautifulSoup
html = """
<div class="product-list">
<div class="product" id="p1">
<h3>Python Cookbook</h3>
<span class="price">$39.99</span>
<a href="/products/python-cookbook">Details</a>
</div>
<div class="product" id="p2">
<h3>Django for Professionals</h3>
<span class="price">$44.99</span>
<a href="/products/django-pro">Details</a>
</div>
</div>
"""
soup = BeautifulSoup(html, "html.parser")
# Method 1: find() returns the first match only
first_title = soup.find("h3")
print(f"First: {first_title.string}")
# Method 2: find_all() returns every match in a list
all_prices = soup.find_all("span", class_="price")
for price in all_prices:
print(f"Price: {price.string}")
# Method 3: select() takes a CSS selector (the most flexible option)
links = soup.select("div.product a")
for link in links:
print(f"Link: {link['href']} -> {link.string}")
# Grab one element by ID with select_one()
product = soup.select_one("#p2 h3")
print(f"Product 2: {product.string}")
▶ Output
First: Python Cookbook Price: $39.99 Price: $44.99 Link: /products/python-cookbook -> Details Link: /products/django-pro -> Details Product 2: Django for Professionals
What happened here: Notice that find_all("span", class_="price") uses class_ with a trailing underscore, because class is a reserved word in Python. The CSS version, select("div.product a"), says the same thing more compactly: any <a> sitting inside a <div class="product">. Once your queries get more involved, like “the third list item that has a data attribute”, select() stays readable while the find_all() keyword style starts to strain. Pick whichever reads clearest for the job in front of you.
Extracting Data: Text, Attributes, and Nested Elements
Finding a tag is half the job, like locating the right parcel in a warehouse: you still have to open it and take out what is inside. The other half is pulling the actual value out of it, and there are a few methods that do slightly different things. The one that bites people is the difference between .string and .get_text(): .string only works when a tag holds a single piece of text, while .get_text() gathers text from the tag and everything nested inside it.
📄 extract.py: text, href, class, and stripped text
from bs4 import BeautifulSoup
html = """
<div class="product" id="p1">
<h3>Python Cookbook</h3>
<span class="price"> $39.99 </span>
<a href="/products/python-cookbook">Details</a>
</div>
"""
soup = BeautifulSoup(html, "html.parser")
# .string direct text of a single tag
# .get_text() all text, including text inside nested tags
# .get("attr") safely read an attribute (returns None if missing)
# .get_text(strip=True) text with surrounding whitespace removed
product = soup.find("div", class_="product")
title = product.find("h3").string
price = product.find("span", class_="price").get_text(strip=True)
link = product.find("a").get("href")
classes = product.get("class", [])
print(f"Title: {title}")
print(f"Price: {price}")
print(f"Link: {link}")
print(f"Classes: {classes}")
▶ Output
Title: Python Cookbook Price: $39.99 Link: /products/python-cookbook Classes: ['product']
What happened here: The price in the HTML had messy spaces around it ( $39.99 ), and get_text(strip=True) cleaned them off so you get $39.99, not the padded version. Reaching for an attribute with .get("href") rather than ["href"] is the safe habit: if the attribute is missing, .get() hands back None instead of raising KeyError and stopping your scraper cold. The same trick works on .get("class", []), where the empty list is a sensible fallback.
Handling Pagination
Real sites split their data across many pages, like a catalog that shows twenty books per page. The trick is almost always the same: the URL carries a page number, so you loop over the numbers and request each one. Just remember to pause between requests. Think of it like knocking on someone’s door. Knock once and wait, and they answer. Hammer the door a hundred times a second and they call security. Rapid-fire calls are the fastest way to get your IP blocked.
📄 paginate.py: scrape several pages politely
import requests
from bs4 import BeautifulSoup
import time
all_books = []
base_url = "https://books.toscrape.com/catalogue/page-{}.html"
for page in range(1, 4): # The first 3 pages
response = requests.get(base_url.format(page))
if response.status_code != 200:
break
# This site sends UTF-8 but no charset header, so requests guesses
# ISO-8859-1. Fix it before reading .text, or the pound sign breaks.
response.encoding = response.apparent_encoding
soup = BeautifulSoup(response.text, "html.parser")
books = soup.select("article.product_pod")
for book in books:
title = book.select_one("h3 a")["title"]
price = book.select_one(".price_color").text
all_books.append({"title": title, "price": price})
print(f"Page {page}: {len(books)} books")
time.sleep(1) # Be polite, pause between requests
print(f"\nTotal: {len(all_books)} books scraped")
▶ Output
Page 1: 20 books Page 2: 20 books Page 3: 20 books Total: 60 books scraped
What happened here: The loop built each URL with base_url.format(page), fetched it, and stopped early if a page returned anything other than 200. The line that earns its keep is response.encoding = response.apparent_encoding. This site serves UTF-8 text but forgets to say so in the headers, and requests then defaults to ISO-8859-1, which mangles the pound sign in the prices. Setting the encoding from what the bytes actually look like fixes it. The time.sleep(1) at the end of each loop is not optional in real work: it is the difference between a welcome guest and a banned bot.
Storing Results
Scraped data that lives only in a variable is gone the moment your script ends, like notes you never wrote down. Save it. CSV opens in any spreadsheet, JSON keeps nested structure, and SQLite sits right there in the standard library when you want to query later. Here are the two file formats you will reach for most.
📄 store.py: save to CSV and JSON
import csv
import json
# all_books is the list of dicts we built in paginate.py
# Save to CSV
with open("books.csv", "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=["title", "price"])
writer.writeheader()
writer.writerows(all_books)
# Save to JSON
with open("books.json", "w", encoding="utf-8") as f:
json.dump(all_books, f, indent=2, ensure_ascii=False)
print(f"Saved {len(all_books)} books to CSV and JSON")
print("First entry:", all_books[0])
▶ Output
Saved 60 books to CSV and JSON
First entry: {'title': 'A Light in the Attic', 'price': '£51.77'}
What happened here: csv.DictWriter turned the list of dictionaries into a tidy CSV with a header row. On the JSON side, two arguments matter. indent=2 keeps the file readable instead of one giant line, and ensure_ascii=False writes the real £ character rather than an ugly escape like \u00a3. Because we fixed the encoding back in the pagination step, that pound sign lands in both files correct and clean.
Common Configuration
Two settings come up on almost every real scraping job: choosing a faster parser, and keeping a session alive so cookies and headers stick across requests. A Session is like staying logged in to a website in your browser. Say a reader named Aditi signs in to her online library account: she types her password once, then browses fifty pages without being asked again, because the site remembers her. That memory is exactly what a Session gives your scraper.
📄 session.py: reuse headers and cookies across requests
import requests
# A Session reuses one connection and remembers cookies and headers
# across requests, which is exactly what login-protected scraping needs.
session = requests.Session()
session.headers.update({"User-Agent": "Mozilla/5.0 (Python Tutorial Bot)"})
# httpbin echoes back what it receives, so we can prove the header stuck.
r1 = session.get("https://httpbin.org/headers")
r2 = session.get("https://httpbin.org/cookies/set/visited/yes")
print("User-Agent sent:", r1.json()["headers"]["User-Agent"])
print("Cookies remembered:", session.cookies.get_dict())
▶ Output
User-Agent sent: Mozilla/5.0 (Python Tutorial Bot)
Cookies remembered: {'visited': 'yes'}
What happened here: We set the User-Agent once on the session, and every request through that session carried it. The second request asked httpbin to set a cookie, and the session held onto it, which you can see in session.cookies.get_dict(). This is the pattern behind scraping pages that need a login: POST your credentials once, and the session sends the auth cookie along with every page you fetch afterward.
The other knob worth knowing is the parser. html.parser ships with Python and needs no install, which is why every example here uses it. For big jobs, pip install lxml and pass "lxml" instead for a noticeable speed boost. For genuinely broken HTML, html5lib is the most forgiving, at the cost of being the slowest.
Scraping Ethics and Best Practices
Before you scrape a site, ask whether you are allowed to. A site’s robots.txt is the sign on a shop door: it tells visitors which aisles are open and which are staff only. Python ships with urllib.robotparser, which reads that file and tells you which paths are off limits. Use it.
📄 check_robots.py: ask permission before you fetch
from urllib.robotparser import RobotFileParser
# Python ships with a robots.txt parser in the standard library.
rp = RobotFileParser()
rp.set_url("https://www.python.org/robots.txt")
rp.read()
bot = "Mozilla/5.0 (Python Tutorial Bot)"
# Ask before you fetch: am I allowed to scrape this path?
print("Can fetch /downloads/ ?", rp.can_fetch(bot, "https://www.python.org/downloads/"))
print("Can fetch /webstats/ ?", rp.can_fetch(bot, "https://www.python.org/webstats/"))
▶ Output
Can fetch /downloads/ ? True Can fetch /webstats/ ? False
What happened here: python.org’s robots.txt allows general bots to read /downloads/ but blocks /webstats/, and the parser reflects that with a clean True and False. Checking this in code, instead of by eye, means your scraper can skip disallowed paths automatically. The rest of the etiquette is just as important.
- Check
robots.txt: read it withurllib.robotparserbefore you start, like the code above - Set a User-Agent: identify your bot honestly, do not pretend to be Chrome
- Rate limit:
time.sleep(1)between requests at the very least - Use APIs when available: scraping is Plan B, not Plan A
- Cache responses: do not re-fetch pages you already have on disk
- Handle errors gracefully: sites change their structure, so your scraper should bend, not crash
The Ecosystem
Python BeautifulSoup is one piece of a larger toolbox. A few neighbors worth knowing as your scraping grows:
- lxml: a faster parser you can plug straight into BeautifulSoup with
BeautifulSoup(html, "lxml") - Selenium (covered in the next post): drives a real browser, so it can scrape pages that build themselves with JavaScript, which BeautifulSoup alone cannot
- httpx: a modern alternative to
requestswith built-in async support, handy when you want to fetch many pages at once - Scrapy: a full scraping framework for large crawls, with built-in queuing, retries, and pipelines
Common Mistakes
❌ Mistake 1: no rate limiting
# BAD: hammering the server with rapid-fire requests
for url in urls:
response = requests.get(url) # 100 requests in 2 seconds, then banned
# GOOD: respectful rate limiting
import time
for url in urls:
response = requests.get(url)
time.sleep(1.5) # 1 to 2 seconds between requests
Why: a tight loop with no pause can fire hundreds of requests a second, which looks exactly like an attack. Many sites will rate-limit or ban an IP that behaves like that. One short time.sleep() per request keeps you under the radar and is just good manners.
❌ Mistake 2: not handling missing elements
# BAD: crashes if the element does not exist
title = soup.find("h3").text # AttributeError if h3 is not found
# GOOD: check before you reach in
h3 = soup.find("h3")
title = h3.text if h3 else "No title"
# Or in one line with getattr
title = getattr(soup.select_one("h3"), "text", "No title")
Why: when find() finds nothing it returns None, and None.text raises AttributeError: 'NoneType' object has no attribute 'text'. On a page with thousands of items, one missing element will kill the whole run unless you guard it. Check the result first, or use getattr with a default, and your scraper rolls past the gap instead of falling over.
Wrap Up
You now have the full scraping pipeline in your hands: fetch a page with requests, parse it with Python BeautifulSoup, locate elements with find(), find_all(), or CSS selectors, walk through paginated results with polite delays, and save everything to CSV or JSON. You also know the manners that keep you welcome: check robots.txt, set an honest User-Agent, and reach for an API before you reach for a scraper. The one wall BeautifulSoup cannot climb is JavaScript-rendered content, and that is exactly where the Selenium tutorial picks up: it drives a real browser so you can scrape pages that build themselves on the fly.
Want the full roadmap from Python basics to AI and ML? Browse every post at the Python + AI/ML tutorial series home.
Frequently Asked Questions
What is BeautifulSoup in Python web scraping?
Python BeautifulSoup (the bs4 package) is an HTML and XML parsing library. It turns an HTML string into a navigable tree of Python objects that you can search with find(), find_all(), and CSS selectors (select()). It handles broken HTML gracefully, which is why it is a staple of Python web scraping.
What is the difference between BeautifulSoup and Selenium?
BeautifulSoup parses static HTML and cannot run JavaScript. Selenium drives a real browser, so it can handle JavaScript-rendered pages, click buttons, and fill forms. Use BeautifulSoup for static sites, Selenium for dynamic ones.
Is web scraping legal?
It depends on the jurisdiction, the website’s terms of service, and what data you scrape. Publicly available data is generally fair game, but scraping personal data, getting around access controls, or breaking the terms of service can be illegal. Always check robots.txt and the terms of service first.
What parser should I use with BeautifulSoup?
Use html.parser (built in, no extra install) for most cases. Use lxml when speed matters on big documents (it is noticeably faster, needs pip install lxml). Use html5lib for the most lenient parsing of broken HTML.
How do I scrape a page that requires login?
Use a requests.Session() to keep cookies. POST to the login endpoint with your credentials, then GET the protected page. The session automatically sends the authentication cookies with every later request.
What is the difference between find() and select() in BeautifulSoup?
find() and find_all() use BeautifulSoup’s own API with keyword arguments (tag, class_, id). select() and select_one() use CSS selectors, the same syntax as browser DevTools. CSS selectors are usually more concise and powerful for complex queries.
Try It Yourself
Scrape the top 10 trending repositories from https://github.com/trending. Pull out the repository name, description, programming language, and star count, then save the results to a JSON file. Handle missing descriptions gracefully (some repos have none) using the safe-navigation pattern from the Common Mistakes section. Add a time.sleep() if you fetch more than one page.
Interview Questions on BeautifulSoup
Try each one aloud first. The phrasing you produce under mild pressure is what interviews measure.
Q: You scrape prices from a UK bookstore and every pound sign comes out mangled as two garbage characters in your CSV. What went wrong and how do you fix it?
The server sent UTF-8 bytes but did not declare a charset in its Content-Type header, so requests fell back to ISO-8859-1 when decoding response.text, which mangles multi-byte characters like the pound sign. Set response.encoding = response.apparent_encoding (or explicitly "utf-8") before touching response.text. Also open your output files with encoding="utf-8" so the fix survives the trip to disk.
Q: What is the difference between .string and .get_text() on a tag?
.string only returns a value when the tag contains exactly one piece of text and nothing else; if the tag has nested children, it returns None. .get_text() walks the whole subtree and concatenates every bit of text inside it, and .get_text(strip=True) also trims surrounding whitespace. The None behavior of .string is a classic source of surprise crashes, so prefer .get_text() when the markup might contain nested tags.
Q: Your scraper ran fine every night for a month, then one morning it died halfway through with AttributeError: ‘NoneType’ object has no attribute ‘text’. What happened and what do you change?
The site changed its markup, so a find() or select_one() that used to match now returns None, and chaining .text off None raises the error. The fix is defensive extraction: check the result before reading it, or use getattr(tag, "text", default), and log and skip bad records instead of crashing the whole run. Longer term, add a sanity check that alerts you when the number of extracted items drops suddenly, because that usually means your selectors have gone stale.
Q: Why does find_all() use class_ with a trailing underscore, and what are the alternatives?
Because class is a reserved keyword in Python, it cannot be used as a keyword argument name, so BeautifulSoup exposes it as class_. The alternatives are passing a dictionary, find_all("span", attrs={"class": "price"}), or sidestepping the issue entirely with a CSS selector like soup.select("span.price"). All three return the same elements; the CSS form is usually the most readable.
Q: Fifty pages into a five-hundred-page crawl, the site suddenly starts answering every request with 403 Forbidden. What do you check first?
You have almost certainly tripped the site’s rate limiting or bot detection. Check whether you are pausing between requests at all, whether your User-Agent identifies you honestly, and whether robots.txt even allows the paths you are hitting. The right response is to back off: add longer delays with exponential backoff, cache pages you already fetched so you never re-request them, and if the site offers an API, switch to it rather than trying to sneak around the block.
Q: You need to parse thousands of large HTML documents and profiling shows parsing is the bottleneck. How do you speed it up?
Swap html.parser for lxml, a C-based parser that is noticeably faster on big documents. If you only need one region of each page, pass a SoupStrainer through the parse_only argument so BeautifulSoup builds a tree for just that slice instead of the whole document. Beyond that, cache parsed results so you never parse the same page twice, and keep fetching separate from parsing so slow network calls do not hide where the real time goes.
Further reading: the official Beautiful Soup documentation is the authoritative source on this.
Related Posts
Previous: Python: REST APIs with requests, Authentication, Pagination
Next: Python: Web Scraping with Selenium
Series Home: Python + AI/ML Tutorial Series
No comment