Pandas Series, DataFrames & CSV
Pandas (Python Data Analysis Library) is the world's most popular tool for data wrangling, cleaning, inspection, and exploratory data analysis (EDA).
The 2 Primary Pandas Data Structures:
- Series (1D): A one-dimensional labeled array capable of holding any data type. It consists of two components: the Index labels and the Data values.
- DataFrame (2D): A two-dimensional tabular spreadsheet-like structure with labeled rows (Index) and labeled columns. You can think of a DataFrame as a dictionary of Series sharing a common Index!
# Conceptual Pandas DataFrame Tabular Inspection:
tabular_dataset = [
{"emp_id": 101, "name": "Balaji Dev", "dept": "AI Engineering", "salary": 95000.0, "experience_yrs": 5},
{"emp_id": 102, "name": "Alex Smith", "dept": "Cloud DevOps", "salary": 82000.0, "experience_yrs": 3},
{"emp_id": 103, "name": "Chloe Davis", "dept": "Data Science", "salary": 88000.0, "experience_yrs": 4},
{"emp_id": 104, "name": "David Miller","dept": "AI Engineering", "salary": 65000.0, "experience_yrs": 2}
]
print("--- 📊 Dataset Overview (4 Records x 5 Columns) ---")
header = f"{'ID':<6} {'Name':<16} {'Department':<18} {'Salary':<12} {'Exp (Yrs)':<10}"
print(header)
print("-" * len(header))
for row in tabular_dataset:
print(f"{row['emp_id']:<6} {row['name']:<16} {row['dept']:<18} ₹{row['salary']:<11,.2f} {row['experience_yrs']:<10}")df.head(n=5): Returns the first $n$ rows of the dataset.df.info(): Displays memory usage, column names, and count of non-null values.df.describe(): Computes statistical summary (count, mean, std, min, 25%, 50%, 75%, max) for numeric columns.
One of the most important concepts in Pandas is knowing when to use .loc versus .iloc:
| Indexer | Syntax | How It Works | Slicing Stop Behavior |
|---|---|---|---|
.loc[] | df.loc[row_label, col_label] | Selects by explicit Name/Label | Inclusive (stops at and includes the end label) |
.iloc[] | df.iloc[row_pos, col_pos] | Selects by 0-based Integer Index | Exclusive (standard Python behavior; excludes end index) |
# Demonstrating loc vs iloc selection logic:
employees = {
101: {"name": "Balaji", "role": "Lead", "salary": 95000},
102: {"name": "Alex", "role": "DevOps", "salary": 82000},
103: {"name": "Chloe", "role": "Data", "salary": 88000}
}
# 1. Label-Based Selection (.loc equivalent):
# Accessing row with explicit ID key 101:
loc_sample = employees[101]["salary"]
print(f"loc selection (ID 101 -> 'salary'): ₹{loc_sample:,}")
# 2. Integer-Based Selection (.iloc equivalent):
# Accessing 1st row (index 0) in order:
row_keys = list(employees.keys())
iloc_sample = employees[row_keys[0]]["name"]
print(f"iloc selection (Row 0 -> 'name'): {iloc_sample}")
# 3. Multi-Condition Filtering (Employees in AI or Data with salary >= 85,000):
high_earners = [e for e in employees.values() if e["salary"] >= 85000]
print(f"\nFiltered High Earners (salary >= 85k): {len(high_earners)} employees matched.")In real Pandas, write: df[(df['salary'] >= 85000) & (df['dept'] == 'AI Engineering')]. Always wrap individual conditions in parentheses when combining with bitwise & (AND) or | (OR).
In Python, "and" and "or" evaluate truthiness of the entire object, which raises a "ValueError: The truth value of a Series is ambiguous". In Pandas, always use element-wise bitwise operators & and | with parentheses: df[(df["a"] > 1) & (df["b"] < 5)].
Simulate Pandas sorting: Write a Python script to sort a list of product dictionaries first by category ascending, then by price descending.
products = [
{"name": "Mouse", "cat": "Tech", "price": 800},
{"name": "Desk", "cat": "Furniture", "price": 4500},
{"name": "Laptop", "cat": "Tech", "price": 55000},
{"name": "Chair", "cat": "Furniture", "price": 3200}
]
# Equivalent to df.sort_values(by=["cat", "price"], ascending=[True, False])
sorted_prods = sorted(products, key=lambda x: (x["cat"], -x["price"]))
print("Sorted Products (Category ASC, Price DESC):")
for p in sorted_prods:
print(f"• [{p['cat']:9}] {p['name']:10} - ₹{p['price']:,}")Q What is the difference between inplace=True and assigning back in Pandas?
df.dropna(inplace=True) modifies the DataFrame directly in memory without returning a new object. Modern Pandas best practices recommend assigning back (df = df.dropna()) to support method chaining.
Q How does Pandas handle missing data under the hood?
Pandas historically represented missing numeric data with IEEE NaN (floating-point Not-a-Number) and objects with None. Modern Pandas (v2.0+) includes native nullable data types (Int64, boolean, string) using pd.NA.
Q What is the difference between df.apply() and vectorized operations?
Vectorized operations (df["a"] + df["b"]) execute compiled C code at maximum speed. df.apply(func) iterates row-by-row in Python, which is significantly slower for large datasets.