Tutorials Python Python Pandas Basics
Ch 50 / 58
Python Intermediate 15 min read Chapter 50 of 58

Python Pandas Basics

Pandas is Python's premier data manipulation library. Built on top of NumPy, it provides two high-level data structures — Series (1D labelled array) and DataFrame (2D labelled table) — that make loading, cleaning, filtering, grouping, and transforming real-world data intuitive and fast. From reading CSV files to performing complex aggregations, Pandas is the tool data analysts, data scientists, and backend engineers reach for every day.

What you'll learn in this chapter
Series — 1D labelled data structure
DataFrame — 2D tables, creation methods
Indexing — loc, iloc, boolean filtering
Handling missing data — NaN, dropna, fillna
GroupBy — split, apply, combine
Merging, joining and concatenating DataFrames
Advertisement

Series & DataFrame — The Two Core Structures

Pandas has two primary data structures. Understanding when to use each and how they relate is the foundation of all Pandas work:

Pandas core structures — Series and DataFrame
pd.Series 1D labelled array — like a column
A one-dimensional array with an index (labels). Like a Python dict where keys become the index. Each element has the same dtype. A DataFrame column IS a Series.
Index: 0 1 2 3 | Values: 10 20 30 40 dtype: int64
pd.DataFrame 2D labelled table — rows × columns
A two-dimensional table with labelled rows (index) and columns. Like a spreadsheet or SQL table. Each column is a Series with its own dtype. Most data analysis uses DataFrames.
Columns: name, age, salary | Rows: 0,1,2... | Each col = Series
Python — pandas_series.py
import pandas as pd
import numpy as np

# ── Create a Series ───────────────────────────────────
s1 = pd.Series([10, 20, 30, 40])         # default index 0,1,2,3
s2 = pd.Series([10, 20, 30],
               index=["a", "b", "c"])       # custom index
s3 = pd.Series({"x": 1, "y": 2, "z": 3})  # from dict

print(s1)
# 0    10
# 1    20
# 2    30
# 3    40
# dtype: int64

# ── Series attributes ─────────────────────────────────
print(s2.values)    # [10 20 30] — numpy array
print(s2.index)     # Index(['a', 'b', 'c'], dtype='object')
print(s2.dtype)     # int64
print(s2.name)      # None (can assign: s2.name = "score")
print(len(s2))      # 3

# ── Index-based access ────────────────────────────────
print(s2["b"])      # 20
print(s2["a":"b"])  # inclusive slice by label

# ── Vectorised operations ─────────────────────────────
print(s1 * 2)       # [20 40 60 80]
print(s1 + s1)      # aligned by index
print(s1.mean())    # 25.0
print(s1.describe())  # count, mean, std, min, 25%, 50%, 75%, max

DataFrame — Creating & Exploring

The DataFrame is the workhorse of Pandas. You can create DataFrames from dicts, lists, CSV files, databases, APIs, and NumPy arrays. Key exploration methods let you understand your data instantly.

df.head(n)
First n rows (default 5)
df.tail(n)
Last n rows
df.info()
Column types & non-null counts
df.describe()
Stats summary (numeric cols)
df.shape
Tuple (rows, columns)
df.dtypes
Data type of each column
df.columns
Column names Index
df.value_counts()
Frequency of each value
Python — pandas_dataframe.py
import pandas as pd

# ── Create from dict ──────────────────────────────────
df = pd.DataFrame({
    "name":   ["Alice", "Bob", "Carol", "Dave"],
    "dept":   ["Eng",   "HR",  "Eng",   "HR"],
    "salary": [80000,  60000, 90000,  55000],
    "years":  [5,      3,     8,      2],
})
print(df)
print(df.dtypes)               # name: object, salary: int64...
print(df.shape)                 # (4, 4)
print(df.describe())            # stats for numeric columns
print(df.info())                # types + non-null counts

# ── From CSV / Excel ──────────────────────────────────
# df = pd.read_csv("employees.csv")
# df = pd.read_excel("data.xlsx", sheet_name="Sheet1")
# df = pd.read_json("data.json")
# df = pd.read_sql("SELECT * FROM users", conn)

# ── Column access ─────────────────────────────────────
print(df["salary"])             # Series — single column
print(df[["name", "salary"]])  # DataFrame — multiple columns
print(df.salary)                 # attribute access (only if valid Python name)

# ── Add/modify columns ────────────────────────────────
df["bonus"]    = df["salary"] * 0.1
df["senior"]   = df["years"] >= 5
df["full_name"] = df["name"].str.upper()

# ── Remove column ─────────────────────────────────────
df = df.drop(columns=["full_name"])  # returns new df
# df.drop(columns=["col"], inplace=True)  ← modify in place

# ── Save to CSV ───────────────────────────────────────
# df.to_csv("output.csv", index=False)

Indexing — loc, iloc and Column Selection

Pandas has two primary indexers: loc (label-based) and iloc (integer-position-based). Understanding which to use prevents the most common source of bugs in Pandas code.

loc vs iloc — label vs position based selection
df.loc[row_label, col_label]
Select by label (index name, column name). The stop in a slice is inclusive. Works with string/datetime/custom indices.
df.loc[0, "salary"] ← row label 0, column "salary"
df.loc[1:3, ["name","dept"]] ← rows 1,2,3 (inclusive!)
df.iloc[row_pos, col_pos]
Select by integer position (0-based). The stop in a slice is exclusive like Python lists. Always works regardless of index type.
df.iloc[0, 2] ← row 0, column position 2
df.iloc[1:3, 0:2] ← rows 1-2 (exclusive), cols 0-1
Python — pandas_indexing.py
import pandas as pd

df = pd.DataFrame({
    "name":   ["Alice", "Bob", "Carol", "Dave"],
    "dept":   ["Eng",   "HR",  "Eng",   "HR"],
    "salary": [80000,  60000, 90000,  55000],
}, index=["r0", "r1", "r2", "r3"])  # custom row labels

# ── loc — by label (inclusive slices) ────────────────
print(df.loc["r1"])                    # row r1 as Series
print(df.loc["r0":"r2", "name":"dept"])  # rows r0-r2, cols name-dept
print(df.loc[:, "salary"])              # all rows, salary column

# ── iloc — by position (exclusive slices) ────────────
print(df.iloc[0])                        # first row
print(df.iloc[0:2, 0:2])                # rows 0-1, cols 0-1
print(df.iloc[-1])                       # last row
print(df.iloc[:, [0, 2]])               # all rows, columns 0 and 2

# ── Setting values ────────────────────────────────────
df.loc["r1", "salary"] = 65000          # update single value
df.iloc[0, 2] = 85000                    # by position

# ── Reset / set index ─────────────────────────────────
df2 = df.reset_index(drop=True)          # 0,1,2,3 integer index
df3 = df.set_index("name")               # name becomes the index
print(df3.loc["Alice"])                  # access by name
Advertisement
K2infocom Python Masterclass & Backend Development
Master Python, Django & Backend APIs with AI
Go from absolute beginner to building production-ready REST APIs, web applications, and ML pipelines. Real projects, live mentoring in Vadodara, and placement support included.
2,400+
Students Enrolled
4.9 ⭐
Google Rating
6 Months
Full Program
100%
Placement Support
Python 3 NumPy & Pandas Data Science Machine Learning Django & DRF Docker & AWS Real Projects AI Tools

Filtering — Boolean Masks & Query

Filtering rows is one of the most common Pandas operations. Boolean masks return rows where the condition is True. The query() method provides a more readable SQL-like syntax.

Python — pandas_filtering.py
import pandas as pd

df = pd.DataFrame({
    "name":   ["Alice", "Bob", "Carol", "Dave", "Eve"],
    "dept":   ["Eng", "HR", "Eng", "HR", "Eng"],
    "salary": [80000, 60000, 90000, 55000, 75000],
    "years":  [5, 3, 8, 2, 4],
})

# ── Single condition ──────────────────────────────────
eng = df[df["dept"] == "Eng"]
print(eng)                           # Alice, Carol, Eve

high_salary = df[df["salary"] > 70000]
print(high_salary)                   # Alice, Carol, Eve

# ── Multiple conditions — use & | ~ (NOT python and/or)
mask = (df["dept"] == "Eng") & (df["salary"] > 75000)
print(df[mask])                      # Alice, Carol

hr_or_senior = df[(df["dept"] == "HR") | (df["years"] >= 5)]
print(hr_or_senior)

# ── isin() — filter by list of values ────────────────
eng_or_hr = df[df["dept"].isin(["Eng", "HR"])]

# ── str methods ───────────────────────────────────────
a_names = df[df["name"].str.startswith("A")]
print(a_names)                       # Alice

# ── between() ────────────────────────────────────────
mid_range = df[df["salary"].between(60000, 80000)]

# ── query() — SQL-like syntax ─────────────────────────
print(df.query("dept == 'Eng' and salary > 75000"))
min_salary = 70000
print(df.query("salary > @min_salary"))  # @var for Python vars

# ── sort_values ───────────────────────────────────────
sorted_df = df.sort_values("salary", ascending=False)
print(sorted_df[["name", "salary"]])

Handling Missing Data — NaN, dropna, fillna

Real-world datasets always have missing values. Pandas represents them as NaN (Not a Number) for numeric or None for objects. Proper handling is critical for correct analysis.

Python — pandas_missing_data.py
import pandas as pd
import numpy as np

df = pd.DataFrame({
    "name":   ["Alice", "Bob",  "Carol", None],
    "salary": [80000,  np.nan, 90000,  55000],
    "years":  [5,      3,     np.nan,  2],
})

# ── Detect missing values ─────────────────────────────
print(df.isnull())              # bool DataFrame — True where NaN
print(df.isnull().sum())       # count NaN per column
print(df.isnull().any())       # True if any NaN in column
print(df["salary"].notna())   # True where NOT NaN

# ── Drop missing ──────────────────────────────────────
clean   = df.dropna()                     # drop rows with ANY NaN
no_sal  = df.dropna(subset=["salary"])    # drop only if salary is NaN
all_nan = df.dropna(how="all")           # drop only if ALL columns NaN

# ── Fill missing ──────────────────────────────────────
df_filled = df.fillna({
    "name":   "Unknown",
    "salary": df["salary"].mean(),   # fill with mean
    "years":  0,
})
print(df_filled)

# ── Forward / backward fill ───────────────────────────
df["salary"] = df["salary"].ffill()     # fill with previous value
df["salary"] = df["salary"].bfill()     # fill with next value

# ── Interpolate ───────────────────────────────────────
ts = pd.Series([1.0, np.nan, np.nan, 4.0])
print(ts.interpolate())          # [1.0, 2.0, 3.0, 4.0]

GroupBy — Split, Apply, Combine

groupby() implements the split-apply-combine pattern: split the DataFrame by a key column, apply an aggregation function to each group, combine results. This is Pandas' equivalent of SQL's GROUP BY.

Python — pandas_groupby.py
import pandas as pd

df = pd.DataFrame({
    "name":   ["Alice","Bob","Carol","Dave","Eve"],
    "dept":   ["Eng","HR","Eng","HR","Eng"],
    "salary": [80000,60000,90000,55000,75000],
    "years":  [5,3,8,2,4],
})

# ── Basic groupby + aggregation ───────────────────────
grp = df.groupby("dept")["salary"]
print(grp.mean())           # avg salary per dept
print(grp.sum())            # total salary per dept
print(grp.count())          # headcount per dept
print(grp.max())            # highest salary per dept

# ── Multiple aggregations — agg() ────────────────────
stats = df.groupby("dept")["salary"].agg(
    avg="mean", total="sum", headcount="count", top="max"
)
print(stats)

# ── Group multiple columns ────────────────────────────
multi = df.groupby("dept")[["salary", "years"]].mean()
print(multi)

# ── transform() — adds group result back to original df
df["dept_avg_salary"] = df.groupby("dept")["salary"].transform("mean")
df["pct_of_dept_avg"]  = df["salary"] / df["dept_avg_salary"] * 100

# ── apply() — custom function per group ──────────────
def top_earner(group):
    return group.nlargest(1, "salary")["name"].values[0]

print(df.groupby("dept").apply(top_earner))
# dept
# Eng    Carol
# HR     Bob

Merge, Join & Concatenate

Combining DataFrames is essential for real data work. Pandas provides SQL-like merge() for joins, join() for index-based combination, and concat() for stacking DataFrames.

Python — pandas_merge.py
import pandas as pd

employees = pd.DataFrame({
    "id":   [1, 2, 3, 4],
    "name": ["Alice", "Bob", "Carol", "Dave"],
    "dept_id": [10, 20, 10, 30],
})

departments = pd.DataFrame({
    "dept_id": [10, 20, 40],
    "dept_name": ["Eng", "HR", "Finance"],
})

# ── INNER JOIN (default) ──────────────────────────────
inner = pd.merge(employees, departments, on="dept_id")
print(inner)    # only matched rows: Alice, Bob, Carol

# ── LEFT JOIN — keep all employees ───────────────────
left = pd.merge(employees, departments, on="dept_id", how="left")
print(left)     # Dave: dept_name = NaN (no dept 30)

# ── RIGHT / OUTER joins ───────────────────────────────
right = pd.merge(employees, departments, on="dept_id", how="right")
outer = pd.merge(employees, departments, on="dept_id", how="outer")

# ── Different column names ────────────────────────────
dept2 = departments.rename(columns={"dept_id": "id"})
joined = pd.merge(employees, dept2,
                    left_on="dept_id", right_on="id",
                    suffixes=("_emp", "_dept"))

# ── concat — stack DataFrames ─────────────────────────
q1 = pd.DataFrame({"sales": [100, 200]})
q2 = pd.DataFrame({"sales": [300, 400]})
full = pd.concat([q1, q2], ignore_index=True)
print(full)       # sales: 100, 200, 300, 400 (reset index)

Practice Python Online — Free Compilers

Don't want to install anything yet? Practice all Pandas examples from this chapter directly in your browser. Google Colab and JupyterLite have Pandas pre-installed.

▶
Online Python 3 Compiler
Write, run and debug Python code in your browser. No installation required. Test every Pandas example instantly.
Open Compiler
Replit
Full IDE online
Google Colab
Free GPU notebooks
JupyterLite
Notebooks in browser
Programiz
Beginner-friendly
Chapter Summary
  • Pandas has two core structures: Series (1D labelled array — like a dict with a NumPy array backend) and DataFrame (2D table of Series, like a spreadsheet). Install with pip install pandas, import as pd.
  • Create DataFrames from dicts, lists, CSV (pd.read_csv()), Excel, JSON, SQL. Explore with df.head(), df.info(), df.describe(), df.shape.
  • df.loc[label, col] selects by label (inclusive slices). df.iloc[pos, pos] selects by integer position (exclusive slices like Python lists).
  • Boolean filtering: df[condition]. Combine with & (and), | (or), ~ (not) — not Python's and/or. Use df.query("...") for readable SQL-like syntax.
  • Missing data: isnull() detects NaN, dropna() removes rows/cols with missing values, fillna(value) fills them, ffill()/bfill() fills from adjacent values.
  • groupby("col") splits the DataFrame into groups. Chain aggregations: mean(), sum(), agg(). Use transform() to add group results as a column.
  • pd.merge() is SQL-style JOIN — supports inner, left, right, outer. pd.concat() stacks DataFrames row-wise or column-wise. ignore_index=True resets the index after concat.
  • Never modify a slice directly (SettingWithCopyWarning) — always use loc/iloc to assign values or work on a .copy().
⚡
Free Resource
Download our Python Pandas Basics Cheat Sheet
Series, DataFrame, loc/iloc, filtering, missing data, groupby, merge & concat — one printable page.
Get Sheet
Advertisement
Quick Check — Test Your Understanding
Question & Answer
✓ Answer

.loc selects by label (the actual index value, like "Alice" or 2025-01-01). Slices with loc are inclusive on both ends.

.iloc selects by integer position (0, 1, 2...). Slices are exclusive on the right end like Python lists — df.iloc[0:3] gives rows 0, 1, 2.

Use loc when you know the label; use iloc when you know the position. If your index is 0,1,2... they look similar but behave differently with slices.

✓ Answer

Python's and/or only work with single boolean values — they don't know how to operate on entire Series of booleans. You must use bitwise operators which work element-wise:

  • & for AND  ·  | for OR  ·  ~ for NOT
  • Always wrap each condition in parentheses: (cond1) & (cond2)
# ❌ Wrong:
df[df["dept"] == "Eng" and df["salary"] > 75000]
# ✅ Correct:
df[(df["dept"] == "Eng") & (df["salary"] > 75000)]
✓ Answer

agg() returns a summary DataFrame with one row per group — the shape is different from the original.

transform() returns a Series with the same shape as the original DataFrame — the group result is broadcast back to each original row:

# agg() — one row per group
df.groupby("dept")["salary"].agg("mean")

# transform() — same rows as original
df["dept_avg"] = df.groupby("dept")["salary"].transform("mean")
✓ Answer

pd.merge() is a relational join — combines rows from two DataFrames based on matching values in key columns (like SQL JOIN). Returns only matching rows by default (inner join).

pd.concat() stacks DataFrames along an axis — rows (axis=0, default) or columns (axis=1). No key matching required — just appends one after another.

Rule of thumb: use merge() when you want to combine related data; use concat() when you want to add more rows (same schema) or more columns.

✓ Answer

Three-step process:

  • Detect: df.isnull() or df.isnull().sum() to count NaN per column. df.isnull().any() to know which columns have any NaN.
  • Remove: df.dropna() (all rows with NaN), df.dropna(subset=["col"]) (only if specific column is NaN), df.dropna(how="all") (only if ALL cols are NaN).
  • Fill: df.fillna(value) or df.fillna({"col1": val1, "col2": val2}). Use ffill()/bfill() for time-series or ordered data.
MCQ 1 of 3
What does df["salary"] return?
MCQ 2 of 3
Which method drops rows that have any missing (NaN) values?
MCQ 3 of 3
What is the default join type in pd.merge()?

Rate Us on Google

4.8 / 5
Enjoying this tutorial?
Your 5-star review helps other learners discover our free tutorials and encourages us to keep creating quality content.
80+ reviews
Write a Google Review
Join WhatsApp Channel