Python Requests Library
The Requests library is the de-facto standard for making HTTP requests in Python — it turns complex, low-level HTTP calls into simple, human-friendly Python code. With Requests you can call REST APIs, scrape websites, authenticate against services, upload files, manage sessions, and handle all HTTP methods. It's used in virtually every Python project that communicates with the web — from weather apps to machine learning pipelines.
HTTP Overview — Methods, Status Codes & the Request Lifecycle
Before diving into Requests, a quick overview of the HTTP protocol that powers every web interaction — and the flow your Python code follows when making a request:
GET Requests — Reading Data from APIs
GET is the most common HTTP method — it retrieves data. Install with pip install requests. The Response object
gives you everything you need:
import requests
# ── Basic GET ─────────────────────────────────────────
r = requests.get("https://api.github.com")
# ── Response attributes ───────────────────────────────
print(r.status_code) # 200
print(r.ok) # True (2xx status)
print(r.url) # final URL (after redirects)
print(r.headers) # dict of response headers
print(r.encoding) # utf-8
print(r.elapsed) # timedelta of request duration
print(r.text[:200]) # response body as string
print(r.content[:100]) # response body as bytes
print(r.json()) # parse JSON body as dict/list
# ── Query parameters ──────────────────────────────────
params = {
"q": "python requests",
"sort": "stars",
"per_page": 5,
}
r = requests.get("https://api.github.com/search/repositories",
params=params)
# URL becomes: .../search/repositories?q=python+requests&sort=stars&per_page=5
print(r.url)
data = r.json()
for repo in data["items"]:
print(f"{repo['full_name']:40} ⭐ {repo['stargazers_count']}")
# ── Inspect response headers ──────────────────────────
print(r.headers["Content-Type"]) # application/json
print(r.headers.get("X-RateLimit-Remaining", "N/A"))
POST, PUT & PATCH — Sending Data
POST creates resources, PUT replaces them, PATCH updates specific fields. All three can send data in the request body — as form data, JSON payload, or files.
import requests
BASE = "https://jsonplaceholder.typicode.com" # free fake REST API
# ── POST — create a new resource ──────────────────────
new_post = {
"title": "Python Requests Tutorial",
"body": "Learning requests with K2infocom",
"userId": 1,
}
r = requests.post(f"{BASE}/posts", json=new_post) # json= sets Content-Type automatically
print(r.status_code) # 201 Created
print(r.json()) # {"id": 101, "title": "...", ...}
# ── POST with form data (x-www-form-urlencoded) ───────
form_data = {"username": "alice", "password": "secret"}
r = requests.post("https://httpbin.org/post", data=form_data) # data= for form
# ── PUT — replace entire resource ─────────────────────
updated = {"id": 1, "title": "Updated Title", "body": "New body", "userId": 1}
r = requests.put(f"{BASE}/posts/1", json=updated)
print(r.status_code) # 200
# ── PATCH — partial update ────────────────────────────
r = requests.patch(f"{BASE}/posts/1", json={"title": "Patched Title"})
print(r.json()["title"]) # Patched Title
# ── DELETE ────────────────────────────────────────────
r = requests.delete(f"{BASE}/posts/1")
print(r.status_code) # 200
# ── Upload file ───────────────────────────────────────
with open("report.pdf", "rb") as f:
r = requests.post("https://httpbin.org/post",
files={"file": ("report.pdf", f, "application/pdf")})
Headers & Authentication
HTTP headers carry metadata about the request or response. Authentication headers prove your identity to the API server. Requests supports multiple auth patterns out of the box:
import requests
from requests.auth import HTTPBasicAuth
# ── Custom headers ────────────────────────────────────
headers = {
"User-Agent": "MyApp/1.0 (k2infocom.com)",
"Accept": "application/json",
"Accept-Language": "en-US",
"X-Request-ID": "abc-123", # custom header
}
r = requests.get("https://httpbin.org/headers", headers=headers)
print(r.json())
# ── Bearer token (JWT) ────────────────────────────────
TOKEN = "eyJhbGci..." # your JWT
r = requests.get("https://api.example.com/me",
headers={"Authorization": f"Bearer {TOKEN}"})
# ── API Key in header ─────────────────────────────────
r = requests.get("https://api.openweathermap.org/data/2.5/weather",
params={"q": "Vadodara", "appid": "YOUR_KEY"})
# ── API Key in header ─────────────────────────────────
r = requests.get("https://api.newsapi.org/v2/top-headlines",
headers={"X-Api-Key": "YOUR_API_KEY"},
params={"country": "in"})
# ── HTTP Basic Auth ───────────────────────────────────
r = requests.get("https://httpbin.org/basic-auth/user/pass",
auth=("user", "pass")) # tuple shorthand
# or:
r = requests.get("...", auth=HTTPBasicAuth("user", "pass"))
# ── GitHub personal access token ─────────────────────
r = requests.get("https://api.github.com/user",
headers={"Authorization": "token ghp_xxxx..."})
print(r.json()["login"])
# ── Inspect request headers sent ─────────────────────
prepared = r.request
print(prepared.headers) # what was actually sent
Error Handling, Timeouts & Retries
Network requests can fail in many ways — bad status codes, timeouts, connection errors. Robust code handles all of these explicitly:
import requests
from requests.exceptions import (
RequestException, Timeout, ConnectionError,
HTTPError, TooManyRedirects
)
# ── raise_for_status() — check 4xx/5xx ────────────────
r = requests.get("https://api.github.com/nonexistent")
print(r.status_code) # 404
r.raise_for_status() # raises HTTPError for 4xx/5xx
# ── Timeout ───────────────────────────────────────────
try:
r = requests.get("https://httpbin.org/delay/5",
timeout=2) # 2 sec connect+read
# or: timeout=(3.05, 27) ← (connect, read)
except Timeout:
print("Request timed out!")
# ── Full error handling pattern ───────────────────────
def safe_get(url: str, **kwargs) -> dict | None:
"""GET request with comprehensive error handling."""
try:
r = requests.get(url, timeout=10, **kwargs)
r.raise_for_status() # raises HTTPError for bad codes
return r.json()
except Timeout:
print(f"Timeout: {url}")
except ConnectionError:
print("No internet / DNS failure")
except HTTPError as e:
print(f"HTTP {e.response.status_code}: {url}")
except TooManyRedirects:
print("Too many redirects")
except RequestException as e:
print(f"Unexpected error: {e}")
return None
# ── Retry with urllib3 ────────────────────────────────
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
session = requests.Session()
retry = Retry(total=3, backoff_factor=1,
status_forcelist=[429, 500, 502, 503])
adapter = HTTPAdapter(max_retries=retry)
session.mount("https://", adapter)
session.mount("http://", adapter)
Sessions — Persistent Connections & Cookies
A requests.Session persists settings (headers, auth,
cookies) across multiple requests and reuses the underlying TCP connection for better performance.
import requests
# ── Basic session ─────────────────────────────────────
with requests.Session() as session:
# Set headers/auth once — applied to all requests
session.headers.update({
"Authorization": "Bearer TOKEN_HERE",
"User-Agent": "MyApp/2.0",
})
r1 = session.get("https://api.example.com/users")
r2 = session.get("https://api.example.com/products")
r3 = session.post("https://api.example.com/orders",
json={"product_id": 1})
# session closed automatically
# ── Session with login (cookies) ─────────────────────
session = requests.Session()
# Login — server sets cookie
r = session.post("https://example.com/login",
data={"username": "alice", "password": "s3cr3t"})
# Subsequent requests automatically include the session cookie
profile = session.get("https://example.com/profile")
orders = session.get("https://example.com/orders")
print(session.cookies.get_dict()) # view cookies
# ── Session vs individual request performance ─────────
# Session reuses TCP connection → much faster for many requests
# Individual requests open new connection each time
# ── Download large file in chunks ────────────────────
with requests.get("https://example.com/large.zip",
stream=True) as r:
r.raise_for_status()
with open("large.zip", "wb") as f:
for chunk in r.iter_content(chunk_size=8192):
f.write(chunk)
print("Download complete!")
Real-World API Integration Patterns
Here are production-ready patterns for consuming REST APIs — pagination, rate limiting, API client classes, and environment-variable configuration:
import requests, os, time
# ── API client class ──────────────────────────────────
class GitHubClient:
"""Minimal GitHub API client."""
BASE = "https://api.github.com"
def __init__(self, token: str):
self._session = requests.Session()
self._session.headers.update({
"Authorization": f"Bearer {token}",
"Accept": "application/vnd.github+json",
"X-GitHub-Api-Version": "2022-11-28",
})
def get_user(self) -> dict:
r = self._session.get(f"{self.BASE}/user")
r.raise_for_status()
return r.json()
def get_repos(self, per_page=30) -> list[dict]:
r = self._session.get(f"{self.BASE}/user/repos",
params={"per_page": per_page})
r.raise_for_status()
return r.json()
# Load API key from environment variable (never hardcode!
TOKEN = os.getenv("GITHUB_TOKEN")
client = GitHubClient(TOKEN)
print(client.get_user()["login"])
# ── Pagination ────────────────────────────────────────
def get_all_pages(url: str, params: dict = None) -> list:
"""Fetch all pages from a paginated REST API."""
params = params or {}
params["page"] = 1
results = []
while True:
r = requests.get(url, params=params, timeout=10)
r.raise_for_status()
page = r.json()
if not page: break
results.extend(page)
params["page"] += 1
time.sleep(0.2) # be respectful to the server
return results
# ── Handle rate limits (429) ──────────────────────────
r = requests.get("https://api.example.com/data")
if r.status_code == 429:
retry_after = int(r.headers.get("Retry-After", 60))
print(f"Rate limited — retrying in {retry_after}s")
time.sleep(retry_after)
Practice Python Online — Free Compilers
Don't want to install anything yet? Practice Requests examples from this chapter
directly in your browser. Google Colab has Requests pre-installed. Use https://httpbin.org as a free test API.
- Install with
pip install requests. Import asimport requests. Every function returns a Response object with.status_code,.json(),.text,.headers. - Core methods:
get(),post(),put(),patch(),delete(). Send query params withparams={}, JSON body withjson={}, form data withdata={}. - Always call
r.raise_for_status()to automatically raise anHTTPErrorfor 4xx/5xx responses. Or checkr.okorr.status_codemanually. - Set custom headers with
headers={}. Auth patterns: Bearer token, API key in headers/params, HTTP Basic Auth viaauth=(user, pass). - Always set a
timeout(e.g.timeout=10) — without it, a slow server will hang your program indefinitely. requests.Session()persists headers, auth, and cookies across requests and reuses TCP connections. Use as a context manager withwith.- For large downloads, use
stream=Trueanditer_content(chunk_size=8192)to avoid loading everything into memory. - Never hardcode API keys — store in environment variables and read with
os.getenv("KEY_NAME"). Respect rate limits — addtime.sleep()between requests.
json= serialises a Python dict/list to JSON, sets Content-Type: application/json automatically, and sends it as the
request body. Use this for REST APIs that expect JSON.
data= encodes the dict as URL-encoded
form data (application/x-www-form-urlencoded) — like an HTML form
submission. Use this for traditional web forms or APIs that expect form data.
Without a timeout, requests.get() will wait
indefinitely for a response — if the server is down or very slow, your program hangs
forever with no way to recover.
With timeout=10, a Timeout exception is raised after 10 seconds, which you can catch and
handle. You can also specify timeout=(3.05, 27) — connect timeout
and read timeout separately.
r.raise_for_status() raises an HTTPError exception if the response status code indicates a client
error (4xx) or server error (5xx). For 2xx or 3xx responses, it does nothing.
It's the cleanest way to check for HTTP errors — instead of manually checking
if r.status_code == 404 for every possible error code, one call
handles them all. Always wrap it in a try/except:
try:
r.raise_for_status()
except requests.HTTPError as e:
print(f"Error {e.response.status_code}")
Use a Session when:
- You're making multiple requests to the same host — Session reuses the TCP connection for better performance.
- You have shared headers or auth that should apply to every request — set once on the session, not on each call.
- The API uses cookies/sessions — login once, subsequent requests automatically send the cookie.
- You want to add retry logic via HTTPAdapter — mount it once on the session.
Never hardcode API keys in source code — they'll end up in version control and be exposed. The correct pattern:
- Store keys in environment variables: set in your terminal, CI/CD secrets, or a
.envfile (add to.gitignore!) - Read with
os.getenv("MY_API_KEY") - Use the
python-dotenvpackage to auto-load.envfiles in development
import os
from dotenv import load_dotenv
load_dotenv()
KEY = os.getenv("MY_API_KEY")
r.json() do with a Response object?
Rate Us on Google
Join Our Learning Community
Connect with thousands of learners, get daily tips, ask questions and stay updated with new tutorials.