Tutorials Python Python Requests Library
Ch 52 / 58
Python Intermediate 14 min read Chapter 52 of 58

Python Requests Library

The Requests library is the de-facto standard for making HTTP requests in Python — it turns complex, low-level HTTP calls into simple, human-friendly Python code. With Requests you can call REST APIs, scrape websites, authenticate against services, upload files, manage sessions, and handle all HTTP methods. It's used in virtually every Python project that communicates with the web — from weather apps to machine learning pipelines.

What you'll learn in this chapter
HTTP methods — GET, POST, PUT, PATCH, DELETE
Response object — status, JSON, headers, text
Query params, request body, JSON payload
Headers — custom, auth tokens, content type
Authentication — Basic, Bearer, API Key
Sessions, timeouts, retries, error handling
Advertisement

HTTP Overview — Methods, Status Codes & the Request Lifecycle

Before diving into Requests, a quick overview of the HTTP protocol that powers every web interaction — and the flow your Python code follows when making a request:

HTTP request-response lifecycle
1. Python: requests.get("https://api.example.com/data")
▼
DNS lookup → TCP/TLS connection
2. HTTP Request sent — method, URL, headers, body
▼
travels over the internet
3. Server processes the request — reads params, calls DB, etc.
▼
response travels back
4. HTTP Response — status code, headers, body (JSON/HTML)
▼
5. Python: response.json() / .text / .content
GET
requests.get()
Read/fetch data. No body. Safe & idempotent.
POST
requests.post()
Create new resource. Has body. Not idempotent.
PUT
requests.put()
Replace entire resource. Idempotent.
PATCH
requests.patch()
Partial update — change specific fields.
DELETE
requests.delete()
Remove a resource. Idempotent.
HEAD
requests.head()
Like GET but no response body — check if URL exists.
Common HTTP status codes
200
OK
Request succeeded — data returned
201
Created
New resource created (POST)
204
No Content
Success but no body (DELETE)
301
Moved
Permanent redirect (auto-followed)
400
Bad Request
Invalid request data
401
Unauthorised
Authentication required
404
Not Found
Resource doesn't exist
429
Too Many Requests
Rate limit exceeded
500
Server Error
Bug or crash on server

GET Requests — Reading Data from APIs

GET is the most common HTTP method — it retrieves data. Install with pip install requests. The Response object gives you everything you need:

Python — get_requests.py
import requests

# ── Basic GET ─────────────────────────────────────────
r = requests.get("https://api.github.com")

# ── Response attributes ───────────────────────────────
print(r.status_code)         # 200
print(r.ok)                  # True (2xx status)
print(r.url)                 # final URL (after redirects)
print(r.headers)             # dict of response headers
print(r.encoding)            # utf-8
print(r.elapsed)             # timedelta of request duration
print(r.text[:200])          # response body as string
print(r.content[:100])       # response body as bytes
print(r.json())              # parse JSON body as dict/list

# ── Query parameters ──────────────────────────────────
params = {
    "q":        "python requests",
    "sort":     "stars",
    "per_page": 5,
}
r = requests.get("https://api.github.com/search/repositories",
                   params=params)
# URL becomes: .../search/repositories?q=python+requests&sort=stars&per_page=5
print(r.url)

data = r.json()
for repo in data["items"]:
    print(f"{repo['full_name']:40} ⭐ {repo['stargazers_count']}")

# ── Inspect response headers ──────────────────────────
print(r.headers["Content-Type"])    # application/json
print(r.headers.get("X-RateLimit-Remaining", "N/A"))

POST, PUT & PATCH — Sending Data

POST creates resources, PUT replaces them, PATCH updates specific fields. All three can send data in the request body — as form data, JSON payload, or files.

Python — post_put_delete.py
import requests

BASE = "https://jsonplaceholder.typicode.com"    # free fake REST API

# ── POST — create a new resource ──────────────────────
new_post = {
    "title":  "Python Requests Tutorial",
    "body":   "Learning requests with K2infocom",
    "userId": 1,
}
r = requests.post(f"{BASE}/posts", json=new_post)  # json= sets Content-Type automatically
print(r.status_code)          # 201 Created
print(r.json())               # {"id": 101, "title": "...", ...}

# ── POST with form data (x-www-form-urlencoded) ───────
form_data = {"username": "alice", "password": "secret"}
r = requests.post("https://httpbin.org/post", data=form_data)  # data= for form

# ── PUT — replace entire resource ─────────────────────
updated = {"id": 1, "title": "Updated Title", "body": "New body", "userId": 1}
r = requests.put(f"{BASE}/posts/1", json=updated)
print(r.status_code)          # 200

# ── PATCH — partial update ────────────────────────────
r = requests.patch(f"{BASE}/posts/1", json={"title": "Patched Title"})
print(r.json()["title"])     # Patched Title

# ── DELETE ────────────────────────────────────────────
r = requests.delete(f"{BASE}/posts/1")
print(r.status_code)          # 200

# ── Upload file ───────────────────────────────────────
with open("report.pdf", "rb") as f:
    r = requests.post("https://httpbin.org/post",
                       files={"file": ("report.pdf", f, "application/pdf")})
Advertisement
K2infocom Python Masterclass & Backend Development
Master Python, Django & Backend APIs with AI
Go from absolute beginner to building production-ready REST APIs, web applications, and ML pipelines. Real projects, live mentoring in Vadodara, and placement support included.
2,400+
Students Enrolled
4.9 ⭐
Google Rating
6 Months
Full Program
100%
Placement Support
Python 3 REST APIs Django & DRF FastAPI PostgreSQL Docker & AWS Real Projects AI Tools

Headers & Authentication

HTTP headers carry metadata about the request or response. Authentication headers prove your identity to the API server. Requests supports multiple auth patterns out of the box:

Python — headers_auth.py
import requests
from requests.auth import HTTPBasicAuth

# ── Custom headers ────────────────────────────────────
headers = {
    "User-Agent":    "MyApp/1.0 (k2infocom.com)",
    "Accept":         "application/json",
    "Accept-Language": "en-US",
    "X-Request-ID":   "abc-123",           # custom header
}
r = requests.get("https://httpbin.org/headers", headers=headers)
print(r.json())

# ── Bearer token (JWT) ────────────────────────────────
TOKEN = "eyJhbGci..."                        # your JWT
r = requests.get("https://api.example.com/me",
                   headers={"Authorization": f"Bearer {TOKEN}"})

# ── API Key in header ─────────────────────────────────
r = requests.get("https://api.openweathermap.org/data/2.5/weather",
                   params={"q": "Vadodara", "appid": "YOUR_KEY"})

# ── API Key in header ─────────────────────────────────
r = requests.get("https://api.newsapi.org/v2/top-headlines",
                   headers={"X-Api-Key": "YOUR_API_KEY"},
                   params={"country": "in"})

# ── HTTP Basic Auth ───────────────────────────────────
r = requests.get("https://httpbin.org/basic-auth/user/pass",
                   auth=("user", "pass"))       # tuple shorthand
# or:
r = requests.get("...", auth=HTTPBasicAuth("user", "pass"))

# ── GitHub personal access token ─────────────────────
r = requests.get("https://api.github.com/user",
                   headers={"Authorization": "token ghp_xxxx..."})
print(r.json()["login"])

# ── Inspect request headers sent ─────────────────────
prepared = r.request
print(prepared.headers)        # what was actually sent

Error Handling, Timeouts & Retries

Network requests can fail in many ways — bad status codes, timeouts, connection errors. Robust code handles all of these explicitly:

Python — error_handling.py
import requests
from requests.exceptions import (
    RequestException, Timeout, ConnectionError,
    HTTPError, TooManyRedirects
)

# ── raise_for_status() — check 4xx/5xx ────────────────
r = requests.get("https://api.github.com/nonexistent")
print(r.status_code)         # 404
r.raise_for_status()         # raises HTTPError for 4xx/5xx

# ── Timeout ───────────────────────────────────────────
try:
    r = requests.get("https://httpbin.org/delay/5",
                       timeout=2)                   # 2 sec connect+read
    # or: timeout=(3.05, 27)  ← (connect, read)
except Timeout:
    print("Request timed out!")

# ── Full error handling pattern ───────────────────────
def safe_get(url: str, **kwargs) -> dict | None:
    """GET request with comprehensive error handling."""
    try:
        r = requests.get(url, timeout=10, **kwargs)
        r.raise_for_status()          # raises HTTPError for bad codes
        return r.json()
    except Timeout:
        print(f"Timeout: {url}")
    except ConnectionError:
        print("No internet / DNS failure")
    except HTTPError as e:
        print(f"HTTP {e.response.status_code}: {url}")
    except TooManyRedirects:
        print("Too many redirects")
    except RequestException as e:
        print(f"Unexpected error: {e}")
    return None

# ── Retry with urllib3 ────────────────────────────────
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()
retry = Retry(total=3, backoff_factor=1,
               status_forcelist=[429, 500, 502, 503])
adapter = HTTPAdapter(max_retries=retry)
session.mount("https://", adapter)
session.mount("http://",  adapter)

Sessions — Persistent Connections & Cookies

A requests.Session persists settings (headers, auth, cookies) across multiple requests and reuses the underlying TCP connection for better performance.

Python — sessions.py
import requests

# ── Basic session ─────────────────────────────────────
with requests.Session() as session:
    # Set headers/auth once — applied to all requests
    session.headers.update({
        "Authorization": "Bearer TOKEN_HERE",
        "User-Agent":     "MyApp/2.0",
    })

    r1 = session.get("https://api.example.com/users")
    r2 = session.get("https://api.example.com/products")
    r3 = session.post("https://api.example.com/orders",
                       json={"product_id": 1})
# session closed automatically

# ── Session with login (cookies) ─────────────────────
session = requests.Session()

# Login — server sets cookie
r = session.post("https://example.com/login",
                   data={"username": "alice", "password": "s3cr3t"})

# Subsequent requests automatically include the session cookie
profile = session.get("https://example.com/profile")
orders  = session.get("https://example.com/orders")
print(session.cookies.get_dict())          # view cookies

# ── Session vs individual request performance ─────────
# Session reuses TCP connection → much faster for many requests
# Individual requests open new connection each time

# ── Download large file in chunks ────────────────────
with requests.get("https://example.com/large.zip",
                    stream=True) as r:
    r.raise_for_status()
    with open("large.zip", "wb") as f:
        for chunk in r.iter_content(chunk_size=8192):
            f.write(chunk)
    print("Download complete!")

Real-World API Integration Patterns

Here are production-ready patterns for consuming REST APIs — pagination, rate limiting, API client classes, and environment-variable configuration:

Python — real_world_api.py
import requests, os, time

# ── API client class ──────────────────────────────────
class GitHubClient:
    """Minimal GitHub API client."""
    BASE = "https://api.github.com"

    def __init__(self, token: str):
        self._session = requests.Session()
        self._session.headers.update({
            "Authorization":        f"Bearer {token}",
            "Accept":               "application/vnd.github+json",
            "X-GitHub-Api-Version": "2022-11-28",
        })

    def get_user(self) -> dict:
        r = self._session.get(f"{self.BASE}/user")
        r.raise_for_status()
        return r.json()

    def get_repos(self, per_page=30) -> list[dict]:
        r = self._session.get(f"{self.BASE}/user/repos",
                                params={"per_page": per_page})
        r.raise_for_status()
        return r.json()

# Load API key from environment variable (never hardcode!
TOKEN = os.getenv("GITHUB_TOKEN")
client = GitHubClient(TOKEN)
print(client.get_user()["login"])

# ── Pagination ────────────────────────────────────────
def get_all_pages(url: str, params: dict = None) -> list:
    """Fetch all pages from a paginated REST API."""
    params = params or {}
    params["page"] = 1
    results = []
    while True:
        r = requests.get(url, params=params, timeout=10)
        r.raise_for_status()
        page = r.json()
        if not page: break
        results.extend(page)
        params["page"] += 1
        time.sleep(0.2)        # be respectful to the server
    return results

# ── Handle rate limits (429) ──────────────────────────
r = requests.get("https://api.example.com/data")
if r.status_code == 429:
    retry_after = int(r.headers.get("Retry-After", 60))
    print(f"Rate limited — retrying in {retry_after}s")
    time.sleep(retry_after)

Practice Python Online — Free Compilers

Don't want to install anything yet? Practice Requests examples from this chapter directly in your browser. Google Colab has Requests pre-installed. Use https://httpbin.org as a free test API.

▶
Online Python 3 Compiler
Write, run and debug Python code in your browser. No installation required. Test every Requests example instantly.
Open Compiler
Replit
Full IDE online
Google Colab
Free GPU notebooks
JupyterLite
Notebooks in browser
Programiz
Beginner-friendly
Chapter Summary
  • Install with pip install requests. Import as import requests. Every function returns a Response object with .status_code, .json(), .text, .headers.
  • Core methods: get(), post(), put(), patch(), delete(). Send query params with params={}, JSON body with json={}, form data with data={}.
  • Always call r.raise_for_status() to automatically raise an HTTPError for 4xx/5xx responses. Or check r.ok or r.status_code manually.
  • Set custom headers with headers={}. Auth patterns: Bearer token, API key in headers/params, HTTP Basic Auth via auth=(user, pass).
  • Always set a timeout (e.g. timeout=10) — without it, a slow server will hang your program indefinitely.
  • requests.Session() persists headers, auth, and cookies across requests and reuses TCP connections. Use as a context manager with with.
  • For large downloads, use stream=True and iter_content(chunk_size=8192) to avoid loading everything into memory.
  • Never hardcode API keys — store in environment variables and read with os.getenv("KEY_NAME"). Respect rate limits — add time.sleep() between requests.
⚡
Free Resource
Download our Python Requests Library Cheat Sheet
HTTP methods, response attributes, headers, auth, error handling, sessions & real-world patterns — one printable page.
Get Sheet
Advertisement
Quick Check — Test Your Understanding
Question & Answer
✓ Answer

json= serialises a Python dict/list to JSON, sets Content-Type: application/json automatically, and sends it as the request body. Use this for REST APIs that expect JSON.

data= encodes the dict as URL-encoded form data (application/x-www-form-urlencoded) — like an HTML form submission. Use this for traditional web forms or APIs that expect form data.

✓ Answer

Without a timeout, requests.get() will wait indefinitely for a response — if the server is down or very slow, your program hangs forever with no way to recover.

With timeout=10, a Timeout exception is raised after 10 seconds, which you can catch and handle. You can also specify timeout=(3.05, 27) — connect timeout and read timeout separately.

✓ Answer

r.raise_for_status() raises an HTTPError exception if the response status code indicates a client error (4xx) or server error (5xx). For 2xx or 3xx responses, it does nothing.

It's the cleanest way to check for HTTP errors — instead of manually checking if r.status_code == 404 for every possible error code, one call handles them all. Always wrap it in a try/except:

try:
    r.raise_for_status()
except requests.HTTPError as e:
    print(f"Error {e.response.status_code}")
✓ Answer

Use a Session when:

  • You're making multiple requests to the same host — Session reuses the TCP connection for better performance.
  • You have shared headers or auth that should apply to every request — set once on the session, not on each call.
  • The API uses cookies/sessions — login once, subsequent requests automatically send the cookie.
  • You want to add retry logic via HTTPAdapter — mount it once on the session.
✓ Answer

Never hardcode API keys in source code — they'll end up in version control and be exposed. The correct pattern:

  • Store keys in environment variables: set in your terminal, CI/CD secrets, or a .env file (add to .gitignore!)
  • Read with os.getenv("MY_API_KEY")
  • Use the python-dotenv package to auto-load .env files in development
import os
from dotenv import load_dotenv
load_dotenv()
KEY = os.getenv("MY_API_KEY")
MCQ 1 of 3
Which HTTP method is used to retrieve data from a server without modifying anything?
MCQ 2 of 3
What does r.json() do with a Response object?
MCQ 3 of 3
Which HTTP status code range indicates a successful response?

Rate Us on Google

4.8 / 5
Enjoying this tutorial?
Your 5-star review helps other learners discover our free tutorials and encourages us to keep creating quality content.
80+ reviews
Write a Google Review
Join WhatsApp Channel