Python Project: Web Scraper
Build a complete web scraping application in Python! This project
teaches you to extract real data from websites using requests + BeautifulSoup, navigate HTML structure with CSS selectors, clean and validate
extracted data, handle pagination and multiple pages, export to CSV and JSON, and build a reusable scraper
framework. You'll also learn the ethical and legal rules of scraping — an essential professional
responsibility.
Project Overview — The Scraping Pipeline
Every web scraper follows the same fundamental pipeline. Understanding this flow helps you debug when things go wrong and design better scrapers:
Version 1 — BeautifulSoup Basics
Install: pip install requests beautifulsoup4. We'll
scrape books to Scrape (books.toscrape.com) — a legal,
purpose-built scraping practice site.
import requests
from bs4 import BeautifulSoup
URL = "https://books.toscrape.com/"
# ── Fetch HTML ────────────────────────────────────────
response = requests.get(URL, timeout=10)
response.raise_for_status() # raise on 4xx/5xx
# ── Parse HTML ────────────────────────────────────────
soup = BeautifulSoup(response.text, "html.parser")
# ── Explore the tree ──────────────────────────────────
print(soup.title.text) # All products | Books to Scrape...
# ── find() — first matching element ──────────────────
first_book = soup.find("article", class_="product_pod")
print(first_book.find("h3").find("a")["title"]) # book title
print(first_book.find("p", class_="price_color").text) # price
# ── find_all() — all matching elements ───────────────
books = soup.find_all("article", class_="product_pod")
print(f"Found {len(books)} books on this page") # 20
for book in books[:3]:
title = book.find("h3").find("a")["title"]
price = book.find("p", class_="price_color").text.strip()
rating_class = book.find("p", class_="star-rating")["class"][1]
print(f"📚 {title[:40]:<40} {price} ⭐ {rating_class}")
# ── Key BeautifulSoup methods ─────────────────────────
# soup.find("tag") → first matching tag
# soup.find("tag", class_="name") → with class attribute
# soup.find_all("tag") → list of all matches
# element.text → inner text (all descendant text)
# element.get_text(strip=True) → text with stripping
# element["href"] → attribute value
# element.get("href") → safe attribute access (no KeyError)
# element.parent → parent element
# element.next_sibling → next sibling element
Version 2 — CSS Selector Targeting
soup.select() uses CSS selector syntax — the same
selectors you use in stylesheets. This is often more concise and powerful than find()/find_all() for complex queries.
import requests
from bs4 import BeautifulSoup
def fetch_soup(url: str) -> BeautifulSoup:
headers = {"User-Agent": "Mozilla/5.0 (Python Scraper; educational)"}
r = requests.get(url, headers=headers, timeout=10)
r.raise_for_status()
return BeautifulSoup(r.text, "html.parser")
soup = fetch_soup("https://books.toscrape.com/")
# ── CSS selector examples ────────────────────────────
# soup.select("tag") tag selector
# soup.select(".class") class selector
# soup.select("#id") id selector
# soup.select("parent > child") direct child
# soup.select("ancestor descendant") any descendant
# soup.select("[attribute=value]") attribute match
# soup.select("tag.class1.class2") multiple classes
# soup.select_one("selector") first match only
# ── Extract all book data with CSS selectors ─────────
RATING_MAP = {"One":1, "Two":2, "Three":3, "Four":4, "Five":5}
books = []
for article in soup.select("article.product_pod"):
title = article.select_one("h3 a")["title"]
price = article.select_one("p.price_color").text.strip()
rating_class = article.select_one("p.star-rating")["class"][1]
rating = RATING_MAP.get(rating_class, 0)
avail = article.select_one("p.availability").get_text(strip=True)
link = "https://books.toscrape.com/catalogue/" + article.select_one("h3 a")["href"]
books.append({
"title": title,
"price": price,
"rating": rating,
"availability": avail,
"url": link,
})
for b in books[:3]:
print(f"📚 {b['title'][:35]:<35} {b['price']} {'⭐'*b['rating']}")
print(f"\nScraped {len(books)} books")
Version 3 — Pagination & Multi-Page Scraping
Most websites split data across multiple pages. Detect the "next" link, follow it until there are no more pages, and collect all data with polite delays between requests.
import requests, time
from bs4 import BeautifulSoup
from urllib.parse import urljoin
BASE_URL = "https://books.toscrape.com/"
DELAY = 1.0 # seconds between requests (be polite!)
MAX_PAGES = 5 # limit for demo — site has 50 pages
HEADERS = {"User-Agent": "Mozilla/5.0 (Python Scraper; educational)"}
def scrape_page(url: str) -> tuple[list[dict], str | None]:
"""Scrape one page. Returns (books, next_page_url | None)."""
r = requests.get(url, headers=HEADERS, timeout=10)
r.raise_for_status()
soup = BeautifulSoup(r.text, "html.parser")
# Extract books from current page
RATING_MAP = {"One":1,"Two":2,"Three":3,"Four":4,"Five":5}
books = []
for article in soup.select("article.product_pod"):
books.append({
"title": article.select_one("h3 a")["title"],
"price": article.select_one("p.price_color").text.strip(),
"rating": RATING_MAP.get(article.select_one("p.star-rating")["class"][1], 0),
})
# Find "next" button link
next_btn = soup.select_one("li.next a")
next_url = urljoin(url, next_btn["href"]) if next_btn else None
return books, next_url
def scrape_all_pages(start_url: str, max_pages: int = MAX_PAGES) -> list[dict]:
"""Scrape all pages up to max_pages limit."""
all_books = []
url = start_url
page_num = 1
while url and page_num <= max_pages:
print(f"🔄 Scraping page {page_num}... {url}")
books, next_url = scrape_page(url)
all_books.extend(books)
print(f" ✅ Got {len(books)} books (total: {len(all_books)})")
url = next_url
page_num += 1
if url:
time.sleep(DELAY) # ⚠️ always sleep between requests!
print(f"\n🎉 Scraped {len(all_books)} books across {page_num-1} pages")
return all_books
books = scrape_all_pages(BASE_URL, max_pages=3)
Version 4 — Data Cleaning & Export to CSV / JSON
Raw scraped data is messy — whitespace, HTML entities, price symbols, duplicates. Clean it, validate it, then export to CSV for spreadsheets or JSON for APIs and further processing.
import csv, json, re
from datetime import datetime
# ── Data cleaning functions ───────────────────────────
def clean_price(price_str: str) -> float:
"""'£12.99' → 12.99"""
cleaned = re.sub(rr"[£$€\s]", "", price_str)
try: return float(cleaned)
except ValueError: return 0.0
def clean_text(text: str) -> str:
"""Strip whitespace and normalise HTML entities."""
return (text
.strip()
.replace("&", "&")
.replace("<", "<")
.replace(">", ">")
.replace(""", '"')
)
def clean_books(raw_books: list[dict]) -> list[dict]:
"""Clean all fields and remove duplicates."""
seen = set()
cleaned = []
for b in raw_books:
title = clean_text(b["title"])
if title in seen: continue # deduplicate
seen.add(title)
cleaned.append({
"title": title,
"price_gbp": clean_price(b["price"]),
"rating": b.get("rating", 0),
"available": "In stock" in b.get("availability", ""),
})
return cleaned
# ── Export to CSV ─────────────────────────────────────
def export_csv(books: list[dict], path: str = "books.csv") -> None:
if not books: return
with open(path, "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=books[0].keys())
writer.writeheader()
writer.writerows(books)
print(f"✅ Saved {len(books)} records to {path}")
# ── Export to JSON ────────────────────────────────────
def export_json(books: list[dict], path: str = "books.json") -> None:
export_data = {
"scraped_at": datetime.now().isoformat(),
"count": len(books),
"books": books,
}
with open(path, "w", encoding="utf-8") as f:
json.dump(export_data, f, indent=2, ensure_ascii=False)
print(f"✅ Saved {len(books)} records to {path}")
# ── Quick analysis after scraping ────────────────────
def analyse(books: list[dict]) -> None:
prices = [b["price_gbp"] for b in books if b["price_gbp"] > 0]
print(f"\n📊 Analysis ({len(books)} books):")
print(f" Avg price : £{sum(prices)/len(prices):.2f}")
print(f" Min price : £{min(prices):.2f}")
print(f" Max price : £{max(prices):.2f}")
print(f" 5-star : {sum(1 for b in books if b['rating']==5)} books")
Version 5 — Reusable Scraper Class
Wrap everything in a clean OOP class with session management, configurable rate limiting, logging, and a simple interface for building domain-specific scrapers by subclassing.
import requests, time, logging
from bs4 import BeautifulSoup
from urllib.parse import urljoin
from abc import ABC, abstractmethod
logging.basicConfig(level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("scraper")
class BaseScraper(ABC):
"""Abstract base class for web scrapers."""
DEFAULT_HEADERS = {
"User-Agent": "Mozilla/5.0 (Python Scraper; educational)",
"Accept": "text/html,application/xhtml+xml",
"Accept-Language": "en-US,en;q=0.9",
}
def __init__(self, delay: float = 1.0,
max_retries: int = 3,
timeout: int = 15):
self.delay = delay
self.max_retries = max_retries
self.timeout = timeout
self._session = requests.Session()
self._session.headers.update(self.DEFAULT_HEADERS)
self._request_count = 0
def fetch(self, url: str) -> BeautifulSoup | None:
"""Fetch URL with retry logic and rate limiting."""
for attempt in range(1, self.max_retries + 1):
try:
logger.info(f"Fetching: {url}")
r = self._session.get(url, timeout=self.timeout)
r.raise_for_status()
self._request_count += 1
time.sleep(self.delay) # rate limit AFTER each request
return BeautifulSoup(r.text, "html.parser")
except requests.RequestException as e:
wait = 2 ** attempt
logger.warning(f"Attempt {attempt} failed: {e}. Retry in {wait}s")
time.sleep(wait)
logger.error(f"All retries failed for {url}")
return None
@abstractmethod
def parse_page(self, soup: BeautifulSoup, url: str) -> list[dict]: ...
@abstractmethod
def get_next_page_url(self, soup: BeautifulSoup, current_url: str) -> str | None: ...
def scrape(self, start_url: str, max_pages: int = 10) -> list[dict]:
"""Orchestrate multi-page scraping."""
all_data = []
url = start_url
page = 1
while url and page <= max_pages:
soup = self.fetch(url)
if soup is None: break
items = self.parse_page(soup, url)
all_data.extend(items)
logger.info(f"Page {page}: {len(items)} items (total {len(all_data)})")
url = self.get_next_page_url(soup, url)
page += 1
logger.info(f"Done. {len(all_data)} items in {self._request_count} requests")
return all_data
# ── Concrete scraper for books.toscrape.com ───────────
class BooksScraper(BaseScraper):
RATING_MAP = {"One":1,"Two":2,"Three":3,"Four":4,"Five":5}
def parse_page(self, soup, url) -> list[dict]:
return [{
"title": a.select_one("h3 a")["title"],
"price": a.select_one("p.price_color").text.strip(),
"rating": self.RATING_MAP.get(a.select_one("p.star-rating")["class"][1], 0),
} for a in soup.select("article.product_pod")]
def get_next_page_url(self, soup, current_url) -> str | None:
nxt = soup.select_one("li.next a")
return urljoin(current_url, nxt["href"]) if nxt else None
# ── Usage ─────────────────────────────────────────────
scraper = BooksScraper(delay=1.0)
books = scraper.scrape("https://books.toscrape.com/", max_pages=2)
export_csv(books, "books.csv")
export_json(books, "books.json")
Ethics, robots.txt & Best Practices
Web scraping must be done responsibly. Breaking these rules can cause harm to servers, violate laws, and get your IP banned. Always follow these guidelines:
# ── 1. Always check robots.txt first ─────────────────
from urllib.robotparser import RobotFileParser
def can_scrape(url: str, user_agent: str = "*") -> bool:
"""Check robots.txt before scraping."""
from urllib.parse import urlparse
parsed = urlparse(url)
robots_url = f"{parsed.scheme}://{parsed.netloc}/robots.txt"
rp = RobotFileParser()
rp.set_url(robots_url)
rp.read()
return rp.can_fetch(user_agent, url)
print(can_scrape("https://books.toscrape.com/")) # True
print(can_scrape("https://example.com/private/")) # depends on robots.txt
# ── 2. Scraping ethics checklist ─────────────────────
# ✅ Check robots.txt and Terms of Service
# ✅ Add delays (1+ seconds) between requests
# ✅ Set a descriptive User-Agent identifying your bot
# ✅ Scrape during off-peak hours for popular sites
# ✅ Cache pages locally if you need to re-process
# ✅ Use the site's public API if one exists
# ❌ Never scrape personal or private user data
# ❌ Never bypass login walls or paywalls
# ❌ Never overload a server (DDoS)
# ❌ Never scrape copyrighted content for redistribution
# ── 3. Good User-Agent ───────────────────────────────
GOOD_UA = "MyResearchBot/1.0 (educational; github.com/alice/myproject)"
# Include: name, version, purpose, contact or repo URL
# ── 4. Respect Retry-After headers ───────────────────
def handle_rate_limit(response):
if response.status_code == 429:
wait = int(response.headers.get("Retry-After", 60))
print(f"Rate limited — waiting {wait}s")
time.sleep(wait)
# ── 5. Cache pages to avoid re-downloading ───────────
from pathlib import Path
def fetch_cached(url: str, cache_dir: str = ".cache") -> str:
import hashlib
cache = Path(cache_dir)
cache.mkdir(exist_ok=True)
key = hashlib.md5(url.encode()).hexdigest()
cached = cache / f"{key}.html"
if cached.exists():
return cached.read_text(encoding="utf-8")
html = requests.get(url, timeout=10).text
cached.write_text(html, encoding="utf-8")
return html
Practice Python Online — Free Compilers
Test scraper code in your browser. Google Colab and Replit have internet access so you
can actually make HTTP requests. Use books.toscrape.com — it's a
purpose-built scraping practice site.
- Install with
pip install requests beautifulsoup4. Fetch HTML withrequests.get(url), parse withBeautifulSoup(r.text, "html.parser"). - Navigate with
.find("tag", class_="name"),.find_all(), or CSS selectors via.select("selector")/.select_one("selector"). - Extract text with
.textor.get_text(strip=True). Extract attributes withelement["href"]orelement.get("href"). - Multi-page: find the "next" link with
soup.select_one("li.next a"), useurljoin()to build absolute URLs, loop untilNone. - Clean data:
.strip(), regex to remove symbols, type conversion, deduplication with a set. Export withcsv.DictWriterorjson.dump(). - Production scraper:
requests.Session()for connection reuse, exponential backoff retries,loggingfor audit trails, abstract base class for extensibility. - Always check
robots.txtwithRobotFileParser. Add delays (time.sleep(1)). Set a descriptive User-Agent. Never scrape personal data or bypass auth. - For JavaScript-heavy sites (SPAs), use
playwrightorseleniuminstead of requests + BeautifulSoup. For complex scraping at scale, considerscrapyframework.
soup.find(tag, attrs) uses BeautifulSoup's own method syntax —
pass the tag name and attributes as keyword args. Returns the first match.
soup.select("CSS selector") uses standard
CSS selector syntax — the same selectors used in stylesheets and JavaScript's querySelectorAll(). Returns a list of all matches.
soup.find("article", class_="product_pod") # first match
soup.select("article.product_pod") # all matches
soup.select_one("h3 > a") # first match with CSS
Without delays, your scraper will make requests as fast as Python can execute — potentially hundreds per second. This can:
- Overload the target server (effectively a DDoS attack)
- Trigger rate limiting — the server returns 429 and blocks you
- Get your IP banned permanently
- Violate the site's Terms of Service (potentially illegal)
A 1-2 second delay per request is a common minimum. Always sleep AFTER each request, not before.
robots.txt is a file at the root of every website (e.g. https://example.com/robots.txt) that tells bots and scrapers which
pages they are allowed to access and which are off-limits.
Python's urllib.robotparser.RobotFileParser reads and parses robots.txt:
rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()
rp.can_fetch("*", "https://example.com/page") # True/False
Pagination links in HTML are often relative — e.g. href="catalogue/page-2" instead of the full URL. urljoin() resolves a relative URL against a base URL:
from urllib.parse import urljoin
urljoin("https://books.toscrape.com/", "catalogue/page-2.html")
# → "https://books.toscrape.com/catalogue/page-2.html"
urljoin("https://example.com/a/b/", "../c")
# → "https://example.com/a/c"
Choose the right tool for the job:
- requests + BeautifulSoup — static HTML sites, simple projects, learning, small data. What we built in this chapter.
- Playwright or Selenium — JavaScript-heavy SPAs (React, Vue, Angular) where content loads dynamically after page load. BeautifulSoup only sees the raw HTML, not rendered JS.
- Scrapy — large-scale production scraping: built-in concurrency, middleware, pipelines, item validation, distributed crawling. The professional choice for big jobs.
element.get_text(strip=True) do
differently from element.text?
Rate Us on Google
Join Our Learning Community
Connect with thousands of learners, get daily tips, ask questions and stay updated with new tutorials.