Tutorials Python Python Project — Web Scraper
Ch 55 / 58
Python Project 25 min build Chapter 55 of 58

Python Project: Web Scraper

Build a complete web scraping application in Python! This project teaches you to extract real data from websites using requests + BeautifulSoup, navigate HTML structure with CSS selectors, clean and validate extracted data, handle pagination and multiple pages, export to CSV and JSON, and build a reusable scraper framework. You'll also learn the ethical and legal rules of scraping — an essential professional responsibility.

What you'll build in this chapter
v1 — Fetch & parse HTML with BeautifulSoup
v2 — CSS selector targeting & data extraction
v3 — Multi-page scraping with pagination
v4 — Data cleaning, validation & export to CSV/JSON
v5 — Reusable scraper class with rate limiting
Bonus — Scraping ethics, robots.txt & best practices
Advertisement

Project Overview — The Scraping Pipeline

Every web scraper follows the same fundamental pipeline. Understanding this flow helps you debug when things go wrong and design better scrapers:

Web scraping pipeline — step by step
1. HTTP Request — requests.get(url, headers=...) with timeout & retry
▼
raw HTML response
2. HTML Parsing — BeautifulSoup(html, "html.parser") builds a parse tree
▼
navigable tree
3. Element Selection — soup.select("CSS selectors") or .find() / .find_all()
▼
raw extracted strings
4. Data Cleaning — .strip(), .text, regex, type conversion, deduplication
▼
clean structured data
5. Export — CSV (csv module), JSON (json module) or database (sqlite3)
Scraper build roadmap — 5 progressive versions
1
v1 — BeautifulSoup Basics
requests.get(), BeautifulSoup, .find(), .find_all(), .text, .get(), href extraction.
~25 lines
2
v2 — CSS Selector Targeting
soup.select(), attribute selectors, nested selectors, extracting multiple fields per item.
~45 lines
3
v3 — Pagination & Multi-Page
Detecting next-page links, looping over pages, delay between requests, page limit.
~65 lines
4
v4 — Data Cleaning & Export
Strip whitespace, regex validation, deduplication, CSV writer, JSON dump.
~90 lines
5
v5 — Reusable Scraper Class
OOP Scraper class, session management, rate limiting, logging, configurable headers.
~130 lines
requests
HTTP client (Ch 52)
BeautifulSoup4
HTML/XML parsing
CSS Selectors
Element targeting
csv / json
Data export
time.sleep()
Rate limiting
logging
Debug & audit trail

Version 1 — BeautifulSoup Basics

Install: pip install requests beautifulsoup4. We'll scrape books to Scrape (books.toscrape.com) — a legal, purpose-built scraping practice site.

Python — scraper_v1.py
import requests
from bs4 import BeautifulSoup

URL = "https://books.toscrape.com/"

# ── Fetch HTML ────────────────────────────────────────
response = requests.get(URL, timeout=10)
response.raise_for_status()            # raise on 4xx/5xx

# ── Parse HTML ────────────────────────────────────────
soup = BeautifulSoup(response.text, "html.parser")

# ── Explore the tree ──────────────────────────────────
print(soup.title.text)                 # All products | Books to Scrape...

# ── find() — first matching element ──────────────────
first_book = soup.find("article", class_="product_pod")
print(first_book.find("h3").find("a")["title"])   # book title
print(first_book.find("p", class_="price_color").text)  # price

# ── find_all() — all matching elements ───────────────
books = soup.find_all("article", class_="product_pod")
print(f"Found {len(books)} books on this page")   # 20

for book in books[:3]:
    title = book.find("h3").find("a")["title"]
    price = book.find("p", class_="price_color").text.strip()
    rating_class = book.find("p", class_="star-rating")["class"][1]
    print(f"📚 {title[:40]:<40} {price}  ⭐ {rating_class}")

# ── Key BeautifulSoup methods ─────────────────────────
# soup.find("tag")                  → first matching tag
# soup.find("tag", class_="name")   → with class attribute
# soup.find_all("tag")              → list of all matches
# element.text                      → inner text (all descendant text)
# element.get_text(strip=True)      → text with stripping
# element["href"]                   → attribute value
# element.get("href")               → safe attribute access (no KeyError)
# element.parent                    → parent element
# element.next_sibling              → next sibling element

Version 2 — CSS Selector Targeting

soup.select() uses CSS selector syntax — the same selectors you use in stylesheets. This is often more concise and powerful than find()/find_all() for complex queries.

Python — scraper_v2_selectors.py
import requests
from bs4 import BeautifulSoup

def fetch_soup(url: str) -> BeautifulSoup:
    headers = {"User-Agent": "Mozilla/5.0 (Python Scraper; educational)"}
    r = requests.get(url, headers=headers, timeout=10)
    r.raise_for_status()
    return BeautifulSoup(r.text, "html.parser")

soup = fetch_soup("https://books.toscrape.com/")

# ── CSS selector examples ────────────────────────────
# soup.select("tag")                    tag selector
# soup.select(".class")                 class selector
# soup.select("#id")                    id selector
# soup.select("parent > child")         direct child
# soup.select("ancestor descendant")    any descendant
# soup.select("[attribute=value]")      attribute match
# soup.select("tag.class1.class2")      multiple classes
# soup.select_one("selector")           first match only

# ── Extract all book data with CSS selectors ─────────
RATING_MAP = {"One":1, "Two":2, "Three":3, "Four":4, "Five":5}

books = []
for article in soup.select("article.product_pod"):
    title   = article.select_one("h3 a")["title"]
    price   = article.select_one("p.price_color").text.strip()
    rating_class = article.select_one("p.star-rating")["class"][1]
    rating  = RATING_MAP.get(rating_class, 0)
    avail   = article.select_one("p.availability").get_text(strip=True)
    link    = "https://books.toscrape.com/catalogue/" + article.select_one("h3 a")["href"]
    books.append({
        "title": title,
        "price": price,
        "rating": rating,
        "availability": avail,
        "url": link,
    })

for b in books[:3]:
    print(f"📚 {b['title'][:35]:<35} {b['price']}  {'⭐'*b['rating']}")
print(f"\nScraped {len(books)} books")
Advertisement
K2infocom Python Masterclass & Backend Development
Master Python, Django & Backend APIs with AI
Go from absolute beginner to building production-ready REST APIs, web applications, and ML pipelines. Real projects, live mentoring in Vadodara, and placement support included.
2,400+
Students Enrolled
4.9 ⭐
Google Rating
6 Months
Full Program
100%
Placement Support
Python 3 Web Scraping REST APIs Django & DRF Data Science Real Projects Portfolio Building AI Tools

Version 3 — Pagination & Multi-Page Scraping

Most websites split data across multiple pages. Detect the "next" link, follow it until there are no more pages, and collect all data with polite delays between requests.

Python — scraper_v3_pagination.py
import requests, time
from bs4 import BeautifulSoup
from urllib.parse import urljoin

BASE_URL = "https://books.toscrape.com/"
DELAY    = 1.0        # seconds between requests (be polite!)
MAX_PAGES = 5         # limit for demo — site has 50 pages

HEADERS = {"User-Agent": "Mozilla/5.0 (Python Scraper; educational)"}

def scrape_page(url: str) -> tuple[list[dict], str | None]:
    """Scrape one page. Returns (books, next_page_url | None)."""
    r = requests.get(url, headers=HEADERS, timeout=10)
    r.raise_for_status()
    soup = BeautifulSoup(r.text, "html.parser")

    # Extract books from current page
    RATING_MAP = {"One":1,"Two":2,"Three":3,"Four":4,"Five":5}
    books = []
    for article in soup.select("article.product_pod"):
        books.append({
            "title":  article.select_one("h3 a")["title"],
            "price":  article.select_one("p.price_color").text.strip(),
            "rating": RATING_MAP.get(article.select_one("p.star-rating")["class"][1], 0),
        })

    # Find "next" button link
    next_btn = soup.select_one("li.next a")
    next_url = urljoin(url, next_btn["href"]) if next_btn else None
    return books, next_url

def scrape_all_pages(start_url: str, max_pages: int = MAX_PAGES) -> list[dict]:
    """Scrape all pages up to max_pages limit."""
    all_books = []
    url       = start_url
    page_num  = 1

    while url and page_num <= max_pages:
        print(f"🔄 Scraping page {page_num}... {url}")
        books, next_url = scrape_page(url)
        all_books.extend(books)
        print(f"   ✅ Got {len(books)} books (total: {len(all_books)})")
        url = next_url
        page_num += 1
        if url:
            time.sleep(DELAY)     # ⚠️ always sleep between requests!

    print(f"\n🎉 Scraped {len(all_books)} books across {page_num-1} pages")
    return all_books

books = scrape_all_pages(BASE_URL, max_pages=3)

Version 4 — Data Cleaning & Export to CSV / JSON

Raw scraped data is messy — whitespace, HTML entities, price symbols, duplicates. Clean it, validate it, then export to CSV for spreadsheets or JSON for APIs and further processing.

Python — scraper_v4_export.py
import csv, json, re
from datetime import datetime

# ── Data cleaning functions ───────────────────────────
def clean_price(price_str: str) -> float:
    """'£12.99' → 12.99"""
    cleaned = re.sub(rr"[£$€\s]", "", price_str)
    try: return float(cleaned)
    except ValueError: return 0.0

def clean_text(text: str) -> str:
    """Strip whitespace and normalise HTML entities."""
    return (text
        .strip()
        .replace("&",  "&")
        .replace("<",   "<")
        .replace(">",   ">")
        .replace(""", '"')
    )

def clean_books(raw_books: list[dict]) -> list[dict]:
    """Clean all fields and remove duplicates."""
    seen   = set()
    cleaned = []
    for b in raw_books:
        title = clean_text(b["title"])
        if title in seen: continue        # deduplicate
        seen.add(title)
        cleaned.append({
            "title":     title,
            "price_gbp": clean_price(b["price"]),
            "rating":    b.get("rating", 0),
            "available": "In stock" in b.get("availability", ""),
        })
    return cleaned

# ── Export to CSV ─────────────────────────────────────
def export_csv(books: list[dict], path: str = "books.csv") -> None:
    if not books: return
    with open(path, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=books[0].keys())
        writer.writeheader()
        writer.writerows(books)
    print(f"✅ Saved {len(books)} records to {path}")

# ── Export to JSON ────────────────────────────────────
def export_json(books: list[dict], path: str = "books.json") -> None:
    export_data = {
        "scraped_at": datetime.now().isoformat(),
        "count":      len(books),
        "books":      books,
    }
    with open(path, "w", encoding="utf-8") as f:
        json.dump(export_data, f, indent=2, ensure_ascii=False)
    print(f"✅ Saved {len(books)} records to {path}")

# ── Quick analysis after scraping ────────────────────
def analyse(books: list[dict]) -> None:
    prices = [b["price_gbp"] for b in books if b["price_gbp"] > 0]
    print(f"\n📊 Analysis ({len(books)} books):")
    print(f"   Avg price : £{sum(prices)/len(prices):.2f}")
    print(f"   Min price : £{min(prices):.2f}")
    print(f"   Max price : £{max(prices):.2f}")
    print(f"   5-star    : {sum(1 for b in books if b['rating']==5)} books")

Version 5 — Reusable Scraper Class

Wrap everything in a clean OOP class with session management, configurable rate limiting, logging, and a simple interface for building domain-specific scrapers by subclassing.

Python — scraper_v5_class.py
import requests, time, logging
from bs4 import BeautifulSoup
from urllib.parse import urljoin
from abc import ABC, abstractmethod

logging.basicConfig(level=logging.INFO,
                    format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("scraper")

class BaseScraper(ABC):
    """Abstract base class for web scrapers."""

    DEFAULT_HEADERS = {
        "User-Agent": "Mozilla/5.0 (Python Scraper; educational)",
        "Accept":     "text/html,application/xhtml+xml",
        "Accept-Language": "en-US,en;q=0.9",
    }

    def __init__(self, delay: float = 1.0,
                       max_retries: int = 3,
                       timeout: int = 15):
        self.delay       = delay
        self.max_retries  = max_retries
        self.timeout      = timeout
        self._session     = requests.Session()
        self._session.headers.update(self.DEFAULT_HEADERS)
        self._request_count = 0

    def fetch(self, url: str) -> BeautifulSoup | None:
        """Fetch URL with retry logic and rate limiting."""
        for attempt in range(1, self.max_retries + 1):
            try:
                logger.info(f"Fetching: {url}")
                r = self._session.get(url, timeout=self.timeout)
                r.raise_for_status()
                self._request_count += 1
                time.sleep(self.delay)       # rate limit AFTER each request
                return BeautifulSoup(r.text, "html.parser")
            except requests.RequestException as e:
                wait = 2 ** attempt
                logger.warning(f"Attempt {attempt} failed: {e}. Retry in {wait}s")
                time.sleep(wait)
        logger.error(f"All retries failed for {url}")
        return None

    @abstractmethod
    def parse_page(self, soup: BeautifulSoup, url: str) -> list[dict]: ...

    @abstractmethod
    def get_next_page_url(self, soup: BeautifulSoup, current_url: str) -> str | None: ...

    def scrape(self, start_url: str, max_pages: int = 10) -> list[dict]:
        """Orchestrate multi-page scraping."""
        all_data = []
        url  = start_url
        page = 1
        while url and page <= max_pages:
            soup = self.fetch(url)
            if soup is None: break
            items = self.parse_page(soup, url)
            all_data.extend(items)
            logger.info(f"Page {page}: {len(items)} items (total {len(all_data)})")
            url  = self.get_next_page_url(soup, url)
            page += 1
        logger.info(f"Done. {len(all_data)} items in {self._request_count} requests")
        return all_data

# ── Concrete scraper for books.toscrape.com ───────────
class BooksScraper(BaseScraper):
    RATING_MAP = {"One":1,"Two":2,"Three":3,"Four":4,"Five":5}

    def parse_page(self, soup, url) -> list[dict]:
        return [{
            "title":  a.select_one("h3 a")["title"],
            "price":  a.select_one("p.price_color").text.strip(),
            "rating": self.RATING_MAP.get(a.select_one("p.star-rating")["class"][1], 0),
        } for a in soup.select("article.product_pod")]

    def get_next_page_url(self, soup, current_url) -> str | None:
        nxt = soup.select_one("li.next a")
        return urljoin(current_url, nxt["href"]) if nxt else None

# ── Usage ─────────────────────────────────────────────
scraper = BooksScraper(delay=1.0)
books   = scraper.scrape("https://books.toscrape.com/", max_pages=2)
export_csv(books, "books.csv")
export_json(books, "books.json")

Ethics, robots.txt & Best Practices

Web scraping must be done responsibly. Breaking these rules can cause harm to servers, violate laws, and get your IP banned. Always follow these guidelines:

Python — scraping_ethics.py
# ── 1. Always check robots.txt first ─────────────────
from urllib.robotparser import RobotFileParser

def can_scrape(url: str, user_agent: str = "*") -> bool:
    """Check robots.txt before scraping."""
    from urllib.parse import urlparse
    parsed  = urlparse(url)
    robots_url = f"{parsed.scheme}://{parsed.netloc}/robots.txt"
    rp = RobotFileParser()
    rp.set_url(robots_url)
    rp.read()
    return rp.can_fetch(user_agent, url)

print(can_scrape("https://books.toscrape.com/"))  # True
print(can_scrape("https://example.com/private/"))  # depends on robots.txt

# ── 2. Scraping ethics checklist ─────────────────────
# ✅ Check robots.txt and Terms of Service
# ✅ Add delays (1+ seconds) between requests
# ✅ Set a descriptive User-Agent identifying your bot
# ✅ Scrape during off-peak hours for popular sites
# ✅ Cache pages locally if you need to re-process
# ✅ Use the site's public API if one exists
# ❌ Never scrape personal or private user data
# ❌ Never bypass login walls or paywalls
# ❌ Never overload a server (DDoS)
# ❌ Never scrape copyrighted content for redistribution

# ── 3. Good User-Agent ───────────────────────────────
GOOD_UA = "MyResearchBot/1.0 (educational; github.com/alice/myproject)"
# Include: name, version, purpose, contact or repo URL

# ── 4. Respect Retry-After headers ───────────────────
def handle_rate_limit(response):
    if response.status_code == 429:
        wait = int(response.headers.get("Retry-After", 60))
        print(f"Rate limited — waiting {wait}s")
        time.sleep(wait)

# ── 5. Cache pages to avoid re-downloading ───────────
from pathlib import Path

def fetch_cached(url: str, cache_dir: str = ".cache") -> str:
    import hashlib
    cache  = Path(cache_dir)
    cache.mkdir(exist_ok=True)
    key    = hashlib.md5(url.encode()).hexdigest()
    cached = cache / f"{key}.html"
    if cached.exists():
        return cached.read_text(encoding="utf-8")
    html = requests.get(url, timeout=10).text
    cached.write_text(html, encoding="utf-8")
    return html

Practice Python Online — Free Compilers

Test scraper code in your browser. Google Colab and Replit have internet access so you can actually make HTTP requests. Use books.toscrape.com — it's a purpose-built scraping practice site.

▶
Online Python 3 Compiler
Write, run and debug Python scraping code in your browser. No installation required.
Open Compiler
Replit
Full IDE online
Google Colab
Free GPU notebooks
JupyterLite
Notebooks in browser
Programiz
Beginner-friendly
Chapter Summary
  • Install with pip install requests beautifulsoup4. Fetch HTML with requests.get(url), parse with BeautifulSoup(r.text, "html.parser").
  • Navigate with .find("tag", class_="name"), .find_all(), or CSS selectors via .select("selector") / .select_one("selector").
  • Extract text with .text or .get_text(strip=True). Extract attributes with element["href"] or element.get("href").
  • Multi-page: find the "next" link with soup.select_one("li.next a"), use urljoin() to build absolute URLs, loop until None.
  • Clean data: .strip(), regex to remove symbols, type conversion, deduplication with a set. Export with csv.DictWriter or json.dump().
  • Production scraper: requests.Session() for connection reuse, exponential backoff retries, logging for audit trails, abstract base class for extensibility.
  • Always check robots.txt with RobotFileParser. Add delays (time.sleep(1)). Set a descriptive User-Agent. Never scrape personal data or bypass auth.
  • For JavaScript-heavy sites (SPAs), use playwright or selenium instead of requests + BeautifulSoup. For complex scraping at scale, consider scrapy framework.
⚡
Free Resource
Download our Python Web Scraper Cheat Sheet
BeautifulSoup methods, CSS selectors, pagination, data cleaning, CSV/JSON export and scraping ethics — one printable page.
Get Sheet
Advertisement
Quick Check — Test Your Understanding
Question & Answer
✓ Answer

soup.find(tag, attrs) uses BeautifulSoup's own method syntax — pass the tag name and attributes as keyword args. Returns the first match.

soup.select("CSS selector") uses standard CSS selector syntax — the same selectors used in stylesheets and JavaScript's querySelectorAll(). Returns a list of all matches.

soup.find("article", class_="product_pod")  # first match
soup.select("article.product_pod")          # all matches
soup.select_one("h3 > a")               # first match with CSS
✓ Answer

Without delays, your scraper will make requests as fast as Python can execute — potentially hundreds per second. This can:

  • Overload the target server (effectively a DDoS attack)
  • Trigger rate limiting — the server returns 429 and blocks you
  • Get your IP banned permanently
  • Violate the site's Terms of Service (potentially illegal)

A 1-2 second delay per request is a common minimum. Always sleep AFTER each request, not before.

✓ Answer

robots.txt is a file at the root of every website (e.g. https://example.com/robots.txt) that tells bots and scrapers which pages they are allowed to access and which are off-limits.

Python's urllib.robotparser.RobotFileParser reads and parses robots.txt:

rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()
rp.can_fetch("*", "https://example.com/page")  # True/False
✓ Answer

Pagination links in HTML are often relative — e.g. href="catalogue/page-2" instead of the full URL. urljoin() resolves a relative URL against a base URL:

from urllib.parse import urljoin
urljoin("https://books.toscrape.com/", "catalogue/page-2.html")
# → "https://books.toscrape.com/catalogue/page-2.html"

urljoin("https://example.com/a/b/", "../c")
# → "https://example.com/a/c"
✓ Answer

Choose the right tool for the job:

  • requests + BeautifulSoup — static HTML sites, simple projects, learning, small data. What we built in this chapter.
  • Playwright or Selenium — JavaScript-heavy SPAs (React, Vue, Angular) where content loads dynamically after page load. BeautifulSoup only sees the raw HTML, not rendered JS.
  • Scrapy — large-scale production scraping: built-in concurrency, middleware, pipelines, item validation, distributed crawling. The professional choice for big jobs.
MCQ 1 of 3
Which method returns ALL matching elements as a list using CSS selector syntax?
MCQ 2 of 3
What does element.get_text(strip=True) do differently from element.text?
MCQ 3 of 3
Which Python standard library module is used to write scraped data to a spreadsheet-compatible file?

Rate Us on Google

4.8 / 5
Enjoying this tutorial?
Your 5-star review helps other learners discover our free tutorials and encourages us to keep creating quality content.
80+ reviews
Write a Google Review
Join WhatsApp Channel