How to Implement SCD Type 1 Overwrite in Python with SQLite

Implement SCD Type 1 dimension updates in Python using SQLite — overwrite existing rows with new data while preserving keys.

Medium Python 3.9+ Aug 9, 2026 Data pipelines & processing 14 views 0 copies

Python code

49 lines
Python 3.9+
import sqlite3

# Simulate a dimension table with SCD Type 1 (overwrite)
conn = sqlite3.connect(":memory:")
cursor = conn.cursor()

# Create dimension table
cursor.execute("""
    CREATE TABLE customer_dim (
        customer_id INTEGER PRIMARY KEY,
        customer_name TEXT,
        city TEXT,
        updated_at TEXT
    )
""")

# Initial load
initial_data = [
    (1, "Alice Smith", "New York", "2024-01-01"),
    (2, "Bob Johnson", "Chicago", "2024-01-01"),
    (3, "Carol White", "Boston", "2024-01-01"),
]
cursor.executemany(
    "INSERT INTO customer_dim VALUES (?, ?, ?, ?)", initial_data
)

# Incoming source data — Alice moved and Bob's name changed
incoming_data = [
    (1, "Alice Smith", "Los Angeles", "2024-06-01"),
    (2, "Robert Johnson", "Chicago", "2024-06-01"),
]

# SCD Type 1: overwrite existing rows with new values
for customer_id, name, city, updated_at in incoming_data:
    cursor.execute(
        """
        UPDATE customer_dim
        SET customer_name = ?, city = ?, updated_at = ?
        WHERE customer_id = ?
        """,
        (name, city, updated_at, customer_id)
    )

# Show result
cursor.execute("SELECT * FROM customer_dim ORDER BY customer_id")
for row in cursor.fetchall():
    print(row)

conn.close()

Output

stdout
(1, 'Alice Smith', 'Los Angeles', '2024-06-01')
(2, 'Robert Johnson', 'Chicago', '2024-06-01')
(3, 'Carol White', 'Boston', '2024-01-01')

How it works

This example uses SQLite's UPDATE statement inside a loop to apply SCD Type 1 logic. The WHERE customer_id = ? clause ensures only existing rows are overwritten, keeping the natural key unchanged. Rows not present in the incoming data (like Carol White) remain untouched, so the dimension table reflects the latest state without historical tracking. Using a primary key on customer_id guarantees uniqueness and fast lookup during updates. The in-memory SQLite database makes the pattern easy to test before porting to a real warehouse.

Common mistakes

  • Forgetting to use `WHERE` in the UPDATE, overwriting all rows instead of just the matching key
  • Not committing transactions in a production database, leaving updates unpersisted
  • Assuming new records are handled — SCD Type 1 only updates existing keys; inserts need separate logic

Variations

  1. Use `INSERT ... ON CONFLICT(customer_id) DO UPDATE` for an upsert that handles both new and existing keys
  2. Use pandas `merge` with `how='left'` then update in bulk for large datasets before writing back

Real-world use cases

  • Updating customer master data in a warehouse nightly from a CRM extract where records are overwritten with fresh values.
  • Refreshing product dimension attributes (price, weight) in an analytics database without keeping change history.
  • Syncing employee department changes to a BI reporting cube where only the current value matters for dashboards.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Data pipelines & processing

Related tutorials and quizzes for this topic.