Python Pandas Basics
Pandas is Python's premier data manipulation library. Built on top of
NumPy, it provides two high-level data structures — Series (1D labelled
array) and DataFrame (2D labelled table) — that make loading, cleaning,
filtering, grouping, and transforming real-world data intuitive and fast. From reading CSV files to performing
complex aggregations, Pandas is the tool data analysts, data scientists, and backend engineers reach for every
day.
Series & DataFrame — The Two Core Structures
Pandas has two primary data structures. Understanding when to use each and how they relate is the foundation of all Pandas work:
import pandas as pd
import numpy as np
# ── Create a Series ───────────────────────────────────
s1 = pd.Series([10, 20, 30, 40]) # default index 0,1,2,3
s2 = pd.Series([10, 20, 30],
index=["a", "b", "c"]) # custom index
s3 = pd.Series({"x": 1, "y": 2, "z": 3}) # from dict
print(s1)
# 0 10
# 1 20
# 2 30
# 3 40
# dtype: int64
# ── Series attributes ─────────────────────────────────
print(s2.values) # [10 20 30] — numpy array
print(s2.index) # Index(['a', 'b', 'c'], dtype='object')
print(s2.dtype) # int64
print(s2.name) # None (can assign: s2.name = "score")
print(len(s2)) # 3
# ── Index-based access ────────────────────────────────
print(s2["b"]) # 20
print(s2["a":"b"]) # inclusive slice by label
# ── Vectorised operations ─────────────────────────────
print(s1 * 2) # [20 40 60 80]
print(s1 + s1) # aligned by index
print(s1.mean()) # 25.0
print(s1.describe()) # count, mean, std, min, 25%, 50%, 75%, max
DataFrame — Creating & Exploring
The DataFrame is the workhorse of Pandas. You can
create DataFrames from dicts, lists, CSV files, databases, APIs, and NumPy arrays. Key exploration methods let
you understand your data instantly.
import pandas as pd
# ── Create from dict ──────────────────────────────────
df = pd.DataFrame({
"name": ["Alice", "Bob", "Carol", "Dave"],
"dept": ["Eng", "HR", "Eng", "HR"],
"salary": [80000, 60000, 90000, 55000],
"years": [5, 3, 8, 2],
})
print(df)
print(df.dtypes) # name: object, salary: int64...
print(df.shape) # (4, 4)
print(df.describe()) # stats for numeric columns
print(df.info()) # types + non-null counts
# ── From CSV / Excel ──────────────────────────────────
# df = pd.read_csv("employees.csv")
# df = pd.read_excel("data.xlsx", sheet_name="Sheet1")
# df = pd.read_json("data.json")
# df = pd.read_sql("SELECT * FROM users", conn)
# ── Column access ─────────────────────────────────────
print(df["salary"]) # Series — single column
print(df[["name", "salary"]]) # DataFrame — multiple columns
print(df.salary) # attribute access (only if valid Python name)
# ── Add/modify columns ────────────────────────────────
df["bonus"] = df["salary"] * 0.1
df["senior"] = df["years"] >= 5
df["full_name"] = df["name"].str.upper()
# ── Remove column ─────────────────────────────────────
df = df.drop(columns=["full_name"]) # returns new df
# df.drop(columns=["col"], inplace=True) ← modify in place
# ── Save to CSV ───────────────────────────────────────
# df.to_csv("output.csv", index=False)
Indexing — loc, iloc and Column Selection
Pandas has two primary indexers: loc (label-based)
and iloc (integer-position-based). Understanding which to use prevents the
most common source of bugs in Pandas code.
import pandas as pd
df = pd.DataFrame({
"name": ["Alice", "Bob", "Carol", "Dave"],
"dept": ["Eng", "HR", "Eng", "HR"],
"salary": [80000, 60000, 90000, 55000],
}, index=["r0", "r1", "r2", "r3"]) # custom row labels
# ── loc — by label (inclusive slices) ────────────────
print(df.loc["r1"]) # row r1 as Series
print(df.loc["r0":"r2", "name":"dept"]) # rows r0-r2, cols name-dept
print(df.loc[:, "salary"]) # all rows, salary column
# ── iloc — by position (exclusive slices) ────────────
print(df.iloc[0]) # first row
print(df.iloc[0:2, 0:2]) # rows 0-1, cols 0-1
print(df.iloc[-1]) # last row
print(df.iloc[:, [0, 2]]) # all rows, columns 0 and 2
# ── Setting values ────────────────────────────────────
df.loc["r1", "salary"] = 65000 # update single value
df.iloc[0, 2] = 85000 # by position
# ── Reset / set index ─────────────────────────────────
df2 = df.reset_index(drop=True) # 0,1,2,3 integer index
df3 = df.set_index("name") # name becomes the index
print(df3.loc["Alice"]) # access by name
Filtering — Boolean Masks & Query
Filtering rows is one of the most common Pandas operations. Boolean masks return rows
where the condition is True. The query() method provides a more readable SQL-like syntax.
import pandas as pd
df = pd.DataFrame({
"name": ["Alice", "Bob", "Carol", "Dave", "Eve"],
"dept": ["Eng", "HR", "Eng", "HR", "Eng"],
"salary": [80000, 60000, 90000, 55000, 75000],
"years": [5, 3, 8, 2, 4],
})
# ── Single condition ──────────────────────────────────
eng = df[df["dept"] == "Eng"]
print(eng) # Alice, Carol, Eve
high_salary = df[df["salary"] > 70000]
print(high_salary) # Alice, Carol, Eve
# ── Multiple conditions — use & | ~ (NOT python and/or)
mask = (df["dept"] == "Eng") & (df["salary"] > 75000)
print(df[mask]) # Alice, Carol
hr_or_senior = df[(df["dept"] == "HR") | (df["years"] >= 5)]
print(hr_or_senior)
# ── isin() — filter by list of values ────────────────
eng_or_hr = df[df["dept"].isin(["Eng", "HR"])]
# ── str methods ───────────────────────────────────────
a_names = df[df["name"].str.startswith("A")]
print(a_names) # Alice
# ── between() ────────────────────────────────────────
mid_range = df[df["salary"].between(60000, 80000)]
# ── query() — SQL-like syntax ─────────────────────────
print(df.query("dept == 'Eng' and salary > 75000"))
min_salary = 70000
print(df.query("salary > @min_salary")) # @var for Python vars
# ── sort_values ───────────────────────────────────────
sorted_df = df.sort_values("salary", ascending=False)
print(sorted_df[["name", "salary"]])
Handling Missing Data — NaN, dropna, fillna
Real-world datasets always have missing values. Pandas represents them as NaN (Not a Number) for numeric or None for
objects. Proper handling is critical for correct analysis.
import pandas as pd
import numpy as np
df = pd.DataFrame({
"name": ["Alice", "Bob", "Carol", None],
"salary": [80000, np.nan, 90000, 55000],
"years": [5, 3, np.nan, 2],
})
# ── Detect missing values ─────────────────────────────
print(df.isnull()) # bool DataFrame — True where NaN
print(df.isnull().sum()) # count NaN per column
print(df.isnull().any()) # True if any NaN in column
print(df["salary"].notna()) # True where NOT NaN
# ── Drop missing ──────────────────────────────────────
clean = df.dropna() # drop rows with ANY NaN
no_sal = df.dropna(subset=["salary"]) # drop only if salary is NaN
all_nan = df.dropna(how="all") # drop only if ALL columns NaN
# ── Fill missing ──────────────────────────────────────
df_filled = df.fillna({
"name": "Unknown",
"salary": df["salary"].mean(), # fill with mean
"years": 0,
})
print(df_filled)
# ── Forward / backward fill ───────────────────────────
df["salary"] = df["salary"].ffill() # fill with previous value
df["salary"] = df["salary"].bfill() # fill with next value
# ── Interpolate ───────────────────────────────────────
ts = pd.Series([1.0, np.nan, np.nan, 4.0])
print(ts.interpolate()) # [1.0, 2.0, 3.0, 4.0]
GroupBy — Split, Apply, Combine
groupby() implements the split-apply-combine
pattern: split the DataFrame by a key column, apply an aggregation function to each group, combine results.
This is Pandas' equivalent of SQL's GROUP BY.
import pandas as pd
df = pd.DataFrame({
"name": ["Alice","Bob","Carol","Dave","Eve"],
"dept": ["Eng","HR","Eng","HR","Eng"],
"salary": [80000,60000,90000,55000,75000],
"years": [5,3,8,2,4],
})
# ── Basic groupby + aggregation ───────────────────────
grp = df.groupby("dept")["salary"]
print(grp.mean()) # avg salary per dept
print(grp.sum()) # total salary per dept
print(grp.count()) # headcount per dept
print(grp.max()) # highest salary per dept
# ── Multiple aggregations — agg() ────────────────────
stats = df.groupby("dept")["salary"].agg(
avg="mean", total="sum", headcount="count", top="max"
)
print(stats)
# ── Group multiple columns ────────────────────────────
multi = df.groupby("dept")[["salary", "years"]].mean()
print(multi)
# ── transform() — adds group result back to original df
df["dept_avg_salary"] = df.groupby("dept")["salary"].transform("mean")
df["pct_of_dept_avg"] = df["salary"] / df["dept_avg_salary"] * 100
# ── apply() — custom function per group ──────────────
def top_earner(group):
return group.nlargest(1, "salary")["name"].values[0]
print(df.groupby("dept").apply(top_earner))
# dept
# Eng Carol
# HR Bob
Merge, Join & Concatenate
Combining DataFrames is essential for real data work. Pandas provides SQL-like merge() for joins, join() for index-based
combination, and concat() for stacking DataFrames.
import pandas as pd
employees = pd.DataFrame({
"id": [1, 2, 3, 4],
"name": ["Alice", "Bob", "Carol", "Dave"],
"dept_id": [10, 20, 10, 30],
})
departments = pd.DataFrame({
"dept_id": [10, 20, 40],
"dept_name": ["Eng", "HR", "Finance"],
})
# ── INNER JOIN (default) ──────────────────────────────
inner = pd.merge(employees, departments, on="dept_id")
print(inner) # only matched rows: Alice, Bob, Carol
# ── LEFT JOIN — keep all employees ───────────────────
left = pd.merge(employees, departments, on="dept_id", how="left")
print(left) # Dave: dept_name = NaN (no dept 30)
# ── RIGHT / OUTER joins ───────────────────────────────
right = pd.merge(employees, departments, on="dept_id", how="right")
outer = pd.merge(employees, departments, on="dept_id", how="outer")
# ── Different column names ────────────────────────────
dept2 = departments.rename(columns={"dept_id": "id"})
joined = pd.merge(employees, dept2,
left_on="dept_id", right_on="id",
suffixes=("_emp", "_dept"))
# ── concat — stack DataFrames ─────────────────────────
q1 = pd.DataFrame({"sales": [100, 200]})
q2 = pd.DataFrame({"sales": [300, 400]})
full = pd.concat([q1, q2], ignore_index=True)
print(full) # sales: 100, 200, 300, 400 (reset index)
Practice Python Online — Free Compilers
Don't want to install anything yet? Practice all Pandas examples from this chapter directly in your browser. Google Colab and JupyterLite have Pandas pre-installed.
- Pandas has two core structures: Series (1D labelled array — like a dict with a NumPy
array backend) and DataFrame (2D table of Series, like a spreadsheet). Install with
pip install pandas, import aspd. - Create DataFrames from dicts, lists, CSV (
pd.read_csv()), Excel, JSON, SQL. Explore withdf.head(),df.info(),df.describe(),df.shape. df.loc[label, col]selects by label (inclusive slices).df.iloc[pos, pos]selects by integer position (exclusive slices like Python lists).- Boolean filtering:
df[condition]. Combine with&(and),|(or),~(not) — not Python'sand/or. Usedf.query("...")for readable SQL-like syntax. - Missing data:
isnull()detects NaN,dropna()removes rows/cols with missing values,fillna(value)fills them,ffill()/bfill()fills from adjacent values. groupby("col")splits the DataFrame into groups. Chain aggregations:mean(),sum(),agg(). Usetransform()to add group results as a column.pd.merge()is SQL-style JOIN — supports inner, left, right, outer.pd.concat()stacks DataFrames row-wise or column-wise.ignore_index=Trueresets the index after concat.- Never modify a slice directly (
SettingWithCopyWarning) — always useloc/ilocto assign values or work on a.copy().
.loc selects by label (the actual index value,
like "Alice" or 2025-01-01). Slices with loc are inclusive on both ends.
.iloc selects by integer
position (0, 1, 2...). Slices are exclusive on the right end like Python
lists — df.iloc[0:3] gives rows 0, 1, 2.
Use loc when you know the label; use
iloc when you know the position. If your index is 0,1,2... they
look similar but behave differently with slices.
Python's and/or only work
with single boolean values — they don't know how to operate on entire Series of booleans. You must use
bitwise operators which work element-wise:
&for AND ·|for OR ·~for NOT- Always wrap each condition in parentheses:
(cond1) & (cond2)
# ❌ Wrong:
df[df["dept"] == "Eng" and df["salary"] > 75000]
# ✅ Correct:
df[(df["dept"] == "Eng") & (df["salary"] > 75000)]
agg() returns a summary DataFrame with one row per group
— the shape is different from the original.
transform() returns a Series with the
same shape as the original DataFrame — the group result is broadcast back to each original
row:
# agg() — one row per group
df.groupby("dept")["salary"].agg("mean")
# transform() — same rows as original
df["dept_avg"] = df.groupby("dept")["salary"].transform("mean")
pd.merge() is a relational join — combines rows
from two DataFrames based on matching values in key columns (like SQL JOIN). Returns only matching
rows by default (inner join).
pd.concat() stacks
DataFrames along an axis — rows (axis=0, default) or columns (axis=1). No key matching required — just
appends one after another.
Rule of thumb: use merge() when you want
to combine related data; use concat() when you want to add more
rows (same schema) or more columns.
Three-step process:
- Detect:
df.isnull()ordf.isnull().sum()to count NaN per column.df.isnull().any()to know which columns have any NaN. - Remove:
df.dropna()(all rows with NaN),df.dropna(subset=["col"])(only if specific column is NaN),df.dropna(how="all")(only if ALL cols are NaN). - Fill:
df.fillna(value)ordf.fillna({"col1": val1, "col2": val2}). Useffill()/bfill()for time-series or ordered data.
df["salary"] return?pd.merge()?
Rate Us on Google
Join Our Learning Community
Connect with thousands of learners, get daily tips, ask questions and stay updated with new tutorials.