Scatter Plots & Correlations

You’ve cleaned your data, grouped it, and maybe even built a pivot table. But numbers alone can lie: a correlation coefficient of 0.85 might look impressive, yet the underlying relationship could be curved, clustered, or ruined by a single outlier. Scatter plots expose those hidden patterns before you commit to a model or a conclusion. In data science, a scatter plot is your lie detector, and understanding correlations is how you read the verdict. This lesson shows you how to visualize scatter plots and correlations in Python, interpret what you see, and avoid the traps that trip up even experienced analysts.

The problem this lesson solves

Imagine you’re analyzing customer data: hours spent on your app versus monthly spend. You compute a Pearson correlation of 0.7 and think, “Great, more hours means more money.” But you’re flying blind. That single number hides whether the relationship is linear, whether one extreme spender is dragging the correlation up, or whether the data splits into two distinct groups with opposite behaviors.

Scatter plots solve this exact problem. They reveal: - The direction of the relationship (positive, negative, or none) - The shape (linear, curved, exponential) - The strength (tight cluster vs. fuzzy cloud) - Outliers that distort your statistics - Clusters that suggest hidden subgroups

Without a scatter plot, you can’t trust your correlation coefficient. With it, you can make confident, data-driven decisions.

Core concept / mental model

Think of a scatter plot as a two-dimensional map where each point is a person, product, or event. The x-axis is one variable (“app hours”), the y-axis is another (“spend”), and each dot is one observation. A correlation coefficient (like Pearson’s r) is a single score summarizing the linear alignment of those dots: it ranges from -1 to +1.

• +1 means perfect positive linear relationship (dots climb a straight line)
• -1 means perfect negative linear relationship (dots fall along a line)
• 0 means no linear relationship (dots scattered randomly)

But here’s the key: correlation is not the whole story. The coefficient assumes a straight line. If your data curves like a U, r might be near 0—even though a strong nonlinear relationship exists. The scatter plot shows you the truth; the coefficient gives you a shorthand summary.

Definitions you’ll use

• Scatter plot: A graph that plots each observation as a point with coordinates (x, y).
• Pearson’s r: Measures linear correlation. Sensitive to outliers and assumes normality.
• Spearman’s rho: Rank-based correlation, robust to outliers and monotonic—not necessarily linear—relationships.

Why visualize first?

As statistician Francis Anscombe demonstrated with his famous quartet, four very different datasets can have nearly identical correlation coefficients and regression lines. Only scatter plots reveal the truth. Visualize first, quantify second.

How it works step by step

Creating a scatter plot in Python with matplotlib is straightforward. Let’s break it down logically.

1. Load your data

Start with your DataFrame—pandas makes it easy to handle real-world data.

2. Extract the two variables

You need two numerical columns to compare. If you plan to compute correlations, drop any missing values (NaN) because correlation functions ignore them or return NaN.

3. Create the scatter plot

Use plt.scatter(x, y) for basic plots or df.plot.scatter(x=..., y=...) for a pandas one-liner. Add labels and a title always—confusing axes are a beginner trap.

4. Compute the correlation

Call df['x'].corr(df['y']) for Pearson, or specify method='spearman' for rank-based correlation.

5. Annotate the plot

A good scatter plot shows the correlation coefficient right on the chart, so viewers can quickly assess strength and direction.

6. Interpret, don’t just stare

Ask: Is the relationship linear? Are there outliers? Do clusters exist? Answer those before you draw conclusions.

Hands-on walkthrough

Let’s practice with a realistic dataset: advertising spend versus sales. We’ll generate synthetic data, but the workflow is identical for real-world CSVs.

import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

# Generate synthetic data: advertising spend vs sales
np.random.seed(42)
ad_spend = np.linspace(0, 100, 50)
sales = 2.5 * ad_spend + np.random.normal(0, 30, 50)

df = pd.DataFrame({'ad_spend': ad_spend, 'sales': sales})

# Create a scatter plot
plt.figure(figsize=(8, 5))
plt.scatter(df['ad_spend'], df['sales'], alpha=0.7, color='steelblue')
plt.xlabel('Advertising Spend ($)')
plt.ylabel('Sales ($)')
plt.title('Scatter Plot: Ad Spend vs. Sales')
plt.grid(True, linestyle='--', alpha=0.6)
plt.show()

# Compute Pearson correlation
corr_pearson = df['ad_spend'].corr(df['sales'])
print(f"Pearson correlation: {corr_pearson:.2f}")

Expected output:

Pearson correlation: 0.90

You see a clear positive trend—higher ad spend, higher sales. The correlation coefficient confirms this. But what if there’s an outlier? Let’s see how a single bad data point affects the story.

# Add an extreme outlier
df.loc[49, 'sales'] = 500  # one weird data point

corr_with_outlier = df['ad_spend'].corr(df['sales'])
print(f"Pearson correlation with outlier: {corr_with_outlier:.2f}")

# Use Spearman as a robust alternative
corr_spearman = df['ad_spend'].corr(df['sales'], method='spearman')
print(f"Spearman correlation with outlier: {corr_spearman:.2f}")

# Plot to see the outlier's effect
plt.figure(figsize=(8, 5))
plt.scatter(df['ad_spend'], df['sales'], alpha=0.7, color='steelblue')
plt.annotate('Outlier', xy=(100, 500), xytext=(60, 450),
             arrowprops=dict(arrowstyle='->', color='red'), fontsize=12, color='red')
plt.xlabel('Advertising Spend ($)')
plt.ylabel('Sales ($)')
plt.title('Scatter Plot with Outlier')
plt.grid(True, linestyle='--', alpha=0.6)
plt.show()

Expected output:

Pearson correlation with outlier: 0.09
Spearman correlation with outlier: 0.93

Notice how one outlier tanks the Pearson correlation from 0.90 to 0.09—a dramatic drop. Spearman, using ranks, barely changes (0.93). The scatter plot shows the outlier pulling the trend line toward it. This is why visualizing scatter plots and correlations together is non-negotiable.

Now, let’s iterate: maybe the relationship is nonlinear. Here’s a curved relationship and its misleading Pearson r.

# Nonlinear relationship: quadratic
time = np.linspace(0, 10, 100)
revenue = 50 + 10 * time - 4 * time**2 + np.random.normal(0, 5, 100)

df_nonlinear = pd.DataFrame({'time': time, 'revenue': revenue})

corr_lin = df_nonlinear['time'].corr(df_nonlinear['revenue'])
print(f"Pearson correlation: {corr_lin:.2f}")

plt.figure(figsize=(8, 5))
plt.scatter(df_nonlinear['time'], df_nonlinear['revenue'], alpha=0.7, color='darkorange')
plt.xlabel('Time (months)')
plt.ylabel('Revenue')
plt.title('Nonlinear Relationship: Pearson r is misleading')
plt.axhline(df_nonlinear['revenue'].mean(), color='red', linestyle='--', label='Mean revenue')
plt.legend()
plt.grid(True, linestyle='--', alpha=0.6)
plt.show()

Expected output:

Pearson correlation: 0.65

But the scatter plot reveals a clear upside-down U: revenue rises then falls with time. The 0.65 correlation overstates the strength of a linear fit. Always plot first.

Compare options / when to choose what

Two visualization libraries dominate: matplotlib and seaborn. Both are built on the same base, but each has strengths.

For correlation interpretation, seaborn’s lmplot adds a regression line plus a confidence band, turning a simple scatter into a richer visualization. Choose matplotlib when you need pixel-level control, seaborn when you want speed and aesthetics.

Alternative correlation methods

• Pearson: Default, but sensitive to outliers.
• Spearman: Uses ranks—robust to outliers and captures monotonic relationships.
• Kendall’s tau: Another rank-based method, useful for small samples with many ties.

Use Spearman or Kendall when your data has extreme outliers or you suspect a monotonic—but not necessarily linear—relationship.

Troubleshooting & edge cases

1. Correlation is NaN

This typically happens when a column contains missing values. Fix: drop NaN rows before computing.

# Drop missing values
df_clean = df[['ad_spend', 'sales']].dropna()
corr = df_clean['ad_spend'].corr(df_clean['sales'])

2. Overlapping points hide density

When thousands of points pile up, the plot becomes a black blob. Fix: add transparency (alpha=0.3) or use seaborn’s kdeplot/scatter with a density estimate.

3. Outliers dominate the scale

A single extreme point squishes the rest of the data into a corner. Fix: use a log scale on the affected axis (plt.xscale('log')) or zoom in by setting axis limits. Alternatively, use Spearman correlation to reduce outlier influence.

4. Categorical variable accidentally used

If one of your variables is a string or category, matplotlib may plot arbitrary numbers. Fix: ensure both columns are numeric—convert with pd.to_numeric and check with df.dtypes.

5. “I see a clear pattern, but correlation is ~0”

This usually means the relationship is nonlinear (e.g., a U-shape). Don’t force a Pearson coefficient; instead, describe it as nonlinear, or try Spearman which captures monotonic trends.

6. Correlation is high, but scatter shows no pattern

That’s likely due to a hidden subgroup or a third variable. Facet the plot by a categorical column, or compute correlation within groups.

What you learned & what's next

This lesson taught you the core idea behind visualizing scatter plots and correlations: scatter plots reveal truth, correlation coefficients summarize it—but only for linear relationships. You practiced creating scatter plots with matplotlib and pandas, computing Pearson and Spearman correlations, and diagnosing outliers and nonlinearity. You also compared matplotlib and seaborn, and learned how to avoid common pitfalls like NaN errors and misinterpretation.

Now you’re equipped to explore relationships in your own datasets. Next, you’ll move to building linear regression models—using the patterns you visualize here to quantify and predict. Scatter plots and correlations are the foundation; regression is the next step on that path.

Pro tip: Always pair a scatter plot with your correlation coefficient. If they disagree, trust the plot.