Data Cleaning and Exploratory Analysis
Real data is messy. Cleaning and exploring it usually takes most of the time in a project.
Typical problems
| Problem | Typical fix |
|---|---|
| Missing values | Drop, or impute (mean, median, mode, model-based) |
| Duplicates | drop_duplicates() |
| Wrong types | Convert (to_datetime, astype) |
| Inconsistent categories | Normalise case, trim spaces, map synonyms |
| Outliers | Investigate; cap, transform or remove with care |
import pandas as pd
df = pd.read_csv("sales.csv")
print(df.isna().sum()) # missing per column
df["price"] = df["price"].fillna(df["price"].median())
df = df.drop_duplicates()
df["date"] = pd.to_datetime(df["date"], errors="coerce")
df["city"] = df["city"].str.strip().str.title()
Exploratory data analysis (EDA)
Look before you model.
print(df.shape)
print(df.dtypes)
print(df.describe(include="all"))
print(df["city"].value_counts().head())
print(df.corr(numeric_only=True)) # relationships
import matplotlib.pyplot as plt
df["price"].hist(bins=30) # distribution
plt.show()
df.plot.scatter(x="area", y="price") # relationship
plt.show()
Feature engineering
| Technique | Example |
|---|---|
| Scaling | Standardise numeric features (StandardScaler) |
| Encoding | One-hot encode categories (get_dummies, OneHotEncoder) |
| Date features | Day of week, month, is_weekend |
| Interactions / ratios | price_per_sqft = price / area |
| Binning | Age groups |
Avoid data leakage
Fit scalers, imputers and encoders only on the training split, then apply them to validation and test data. Using information from the test set during training gives falsely good results.
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
model = make_pipeline(StandardScaler(), LogisticRegression())
# model.fit(X_train, y_train) learns the scaler from training data only
Loading Comments...