Complete EDA Workflow

You’ve cleaned a few columns, made a couple of plots, maybe even computed a correlation matrix. But when the dataset is messy and the questions are vague, ad-hoc exploration turns into chaos — you forget why you made a transformation, miss a critical outlier, or redo the same step twice. Without a complete EDA workflow in Python, you’re not exploring; you’re stumbling. In this lesson, you’ll build a repeatable pipeline that takes you from raw CSV to actionable insights — so every analysis starts with structure, not guesswork.

The problem this lesson solves

Most data science projects don’t fail because of missing machines or exotic algorithms. They fail in the messy middle: the exploratory data analysis (EDA) stage. Here’s the pain you’ve probably felt:

• You load a CSV, run .head(), and stare at columns like customer_id, purchase_amount, signup_date — then what?
• You write throwaway snippets that mutate the DataFrame in ways you can’t reproduce later.
• You plot 20 histograms, but nothing tells a story — you’re just burning notebook cells.
• You discover a column with 40% missing values after you built a model, and now your results are meaningless.

Without a structure, EDA becomes a one-way ticket to confusion and rework. A complete EDA workflow fixes this by giving you a checklist — a sequence of steps that are repeatable, auditable, and focused on insight.

Core concept / mental model

Think of a complete EDA workflow as a detective’s investigation. You don’t walk into a crime scene and start guessing. You follow a loose order:

1. Get your bearings — what’s the scene? (Data dictionary, shape, types)
2. Look for evidence — what’s missing or broken? (Missing values, duplicates, outliers)
3. Interview the witnesses — what do the variables say individually? (Univariate stats and distributions)
4. Map the relationships — who’s connected to whom? (Bivariate/multivariate trends, correlations)
5. Write the report — what really matters? (Summarize findings and questions to chase)

In Python, this maps to a pipeline of functions and libraries:

Pro tip: A complete EDA workflow doesn’t mean doing every possible analysis — it means covering the essential bases in a logical order, so you never miss something critical.

How it works step by step

Here’s the step-by-step process to build a complete EDA workflow in a reusable way. You’ll write a set of functions you can reuse across projects.

1. Start with a data dictionary

Before you touch the data, define what each column means and its expected type. This prevents later misinterpretation (e.g., treating id as a number to aggregate). This isn’t a pandas step — it’s a project-level step that guides everything else.

2. Load and inspect the structure

Use pandas to load your file, then immediately check .shape, .columns, .dtypes, and .head(). This is your first pass at spotting obvious problems like wrong dtypes (e.g., dates as strings) or extra unnamed columns.

3. Clean systematically

Clean data in a logical order — don’t mix missing-value handling with outlier removal. Track every change you make:

• Missing values: count per column; decide between dropping or imputing based on domain.
• Duplicates: remove full-row duplicates.
• Type conversions: convert dates to datetime, categories to category dtype.
• Outliers: detect (e.g., with IQR) and decide — don’t blindly drop; sometimes outliers are the story.

4. Explore univariate distributions

For each numeric column, look at describe() and histograms. For categorical columns, use value_counts() to see frequencies. This reveals skewness, rare categories, and potential data quality issues (e.g., a value of -1 for age).

5. Explore bivariate relationships

Use scatter plots for numeric pairs, boxplots for numeric-by-categorical, and a correlation matrix (for numeric only) to spot potential predictive features. This is where you start answering your actual questions.

6. Synthesize and document

Write a markdown cell (or a separate notebook) summarizing: key findings, open questions, assumptions, and next steps. This is the artifact your team or future-you will thank you for.

Pro tip: Keep your EDA self-contained — use functions that take a DataFrame and return summaries/plots, so you can rerun the whole analysis with one cell after you update the data.

Hands-on walkthrough

Now let’s build a complete EDA workflow for a sample dataset — the classic Titanic dataset to illustrate the process. You’ll write reusable functions and see the expected output. Run this in a Jupyter notebook.

Step 1 – Load and inspect

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Load data (use a local copy if offline)
df = pd.read_csv('https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv')

# 1. Structure inspection
print("Shape:", df.shape)
print("Columns:", df.columns.tolist())
print("\nData types:\n", df.dtypes)
print("\nFirst 5 rows:\n", df.head())

Expected output (truncated):

Shape: (891, 12)
Columns: ['PassengerId', 'Survived', 'Pclass', 'Name', 'Sex', 'Age', 'SibSp', 'Parch', 'Ticket', 'Fare', 'Cabin', 'Embarked']

Data types:
 PassengerId     int64
 ...
 Survived        int64
 Age            float64
 Fare           float64
 ...

Step 2 – Systematic cleaning

# 2. Cleaning
def clean_titanic(df):
    """Return a cleaned copy of the Titanic DataFrame."""
    df = df.copy()
    # Drop duplicates
    df = df.drop_duplicates()
    # Fill missing Age with median (or use a better imputation later)
    df['Age'] = df['Age'].fillna(df['Age'].median())
    # Fill missing Embarked with mode
    df['Embarked'] = df['Embarked'].fillna(df['Embarked'].mode()[0])
    # Drop columns with >50% missing
    df = df.drop(columns=['Cabin'])
    return df

# Check missing before/after
print("Missing before:\n", df.isnull().sum())
df_clean = clean_titanic(df)
print("\nMissing after:\n", df_clean.isnull().sum())

Expected output:

Missing before:
 PassengerId    0
 ...
 Age          177
 Cabin        687
 Embarked       2
 ...

Missing after:
 PassengerId    0
 ...
 Age            0
 Embarked       0

Step 3 – Univariate exploration

# 3. Univariate analysis
# Numeric distributions
numeric_cols = df_clean.select_dtypes(include=np.number).columns
df_clean[numeric_cols].hist(bins=30, figsize=(12, 8))
plt.suptitle('Numeric Distributions')
plt.show()

# Categorical frequencies
for col in ['Sex', 'Embarked', 'Pclass']:
    print(f"\n{col} counts:\n", df_clean[col].value_counts())

Expected output: you’ll see histograms for Age, Fare, etc. and printed counts for Sex (male/female), Embarked (S/C/Q), and Pclass (1/2/3).

Step 4 – Bivariate exploration

# 4. Bivariate: survival by sex and class
print(pd.crosstab(df_clean['Sex'], df_clean['Survived']))
print("\nSurvival rate by class:\n", df_clean.groupby('Pclass')['Survived'].mean())

# Correlation matrix
corr = df_clean[['Survived', 'Pclass', 'Age', 'Fare']].corr()
sns.heatmap(corr, annot=True, cmap='coolwarm')
plt.title('Correlation with Survival')
plt.show()

Expected output: a crosstab showing females survived more, survival rate drops with Pclass (1st ≈ 0.63, 3rd ≈ 0.24), and heatmap showing Fare positively correlates with survival, Pclass negatively.

Compare options / when to choose what

Your complete EDA workflow can be built with different tools and styles. Here’s a quick comparison to help you choose:

Rule of thumb: Start in a notebook for initial discovery; once your workflow stabilizes, convert the cleaning and summary functions into a script or a pandas pipeline class. For a quick check during a competition or before modeling, a profiling report can save you minutes.

Troubleshooting & edge cases

• You see object dtype for dates: Convert with pd.to_datetime(column, errors='coerce'). If you don’t, min() and max() will compare strings lexicographically — silent errors.
• Histograms look empty or useless: Check if your numeric column is actually stored as strings. Use df[col] = pd.to_numeric(df[col], errors='coerce') and then inspect missing values.
• Correlation matrix shows NaN: Some columns are non-numeric or have missing values. Use df.corr(numeric_only=True) (pandas 2.1+) or drop non-numeric columns first.
• value_counts() returns too many categories: This is a signal that your categorical column is dirty (e.g., “Cat”, “ cat ”, “CAT”). Standardize text (lowercase, strip) before analysis.
• Outliers skew your plots and stats: Detect with IQR = Q3 - Q1 and lower/upper = Q1 - 1.5*IQR, Q3 + 1.5*IQR, but don’t remove them automatically — cap or flag them instead.
• MemoryError on large datasets: Use read_csv(usecols=...) to load only needed columns, or dtype to specify category for low-cardinality strings.

Pro tip: Always version your dataset and cleaning code. When you discover a bug in your EDA, you want to know exactly which version of the data produced your insights.

What you learned & what's next

You now know how to build a complete EDA workflow in Python: you can load data, inspect its structure, clean it systematically, explore univariate and bivariate patterns, and document your findings — all in a repeatable order. You’ve also learned when to choose different tooling (notebook vs script vs profiling) and how to troubleshoot common pitfalls like wrong dtypes, missing-data surprises, and outlier effects.

Key learning objectives met: - You can explain the core idea: the workflow is a detective’s checklist, not a random set of plots. - You completed a hands-on exercise (Titanic) that walks through every step.

Your next step in the Data Science with Python track is Feature Engineering — you’ll take the clean, explored data and create new features that improve model performance. The solid EDA foundation you built here is the exact foundation that makes feature engineering meaningful.

Keep this workflow as a template in your toolbox. Every new dataset — sales, sensor logs, customer surveys — will benefit from this same structure.