How to Implement SCD Type 1 Overwrite in Python with SQLite
Implement SCD Type 1 dimension updates in Python using SQLite — overwrite existing rows with new data while preserving keys.
Python code
49 linesimport sqlite3
# Simulate a dimension table with SCD Type 1 (overwrite)
conn = sqlite3.connect(":memory:")
cursor = conn.cursor()
# Create dimension table
cursor.execute("""
CREATE TABLE customer_dim (
customer_id INTEGER PRIMARY KEY,
customer_name TEXT,
city TEXT,
updated_at TEXT
)
""")
# Initial load
initial_data = [
(1, "Alice Smith", "New York", "2024-01-01"),
(2, "Bob Johnson", "Chicago", "2024-01-01"),
(3, "Carol White", "Boston", "2024-01-01"),
]
cursor.executemany(
"INSERT INTO customer_dim VALUES (?, ?, ?, ?)", initial_data
)
# Incoming source data — Alice moved and Bob's name changed
incoming_data = [
(1, "Alice Smith", "Los Angeles", "2024-06-01"),
(2, "Robert Johnson", "Chicago", "2024-06-01"),
]
# SCD Type 1: overwrite existing rows with new values
for customer_id, name, city, updated_at in incoming_data:
cursor.execute(
"""
UPDATE customer_dim
SET customer_name = ?, city = ?, updated_at = ?
WHERE customer_id = ?
""",
(name, city, updated_at, customer_id)
)
# Show result
cursor.execute("SELECT * FROM customer_dim ORDER BY customer_id")
for row in cursor.fetchall():
print(row)
conn.close()
Output
(1, 'Alice Smith', 'Los Angeles', '2024-06-01')
(2, 'Robert Johnson', 'Chicago', '2024-06-01')
(3, 'Carol White', 'Boston', '2024-01-01')
How it works
This example uses SQLite's UPDATE statement inside a loop to apply SCD Type 1 logic. The WHERE customer_id = ? clause ensures only existing rows are overwritten, keeping the natural key unchanged. Rows not present in the incoming data (like Carol White) remain untouched, so the dimension table reflects the latest state without historical tracking. Using a primary key on customer_id guarantees uniqueness and fast lookup during updates. The in-memory SQLite database makes the pattern easy to test before porting to a real warehouse.
Common mistakes
- Forgetting to use `WHERE` in the UPDATE, overwriting all rows instead of just the matching key
- Not committing transactions in a production database, leaving updates unpersisted
- Assuming new records are handled — SCD Type 1 only updates existing keys; inserts need separate logic
Variations
- Use `INSERT ... ON CONFLICT(customer_id) DO UPDATE` for an upsert that handles both new and existing keys
- Use pandas `merge` with `how='left'` then update in bulk for large datasets before writing back
Real-world use cases
- Updating customer master data in a warehouse nightly from a CRM extract where records are overwritten with fresh values.
- Refreshing product dimension attributes (price, weight) in an analytics database without keeping change history.
- Syncing employee department changes to a BI reporting cube where only the current value matters for dashboards.
Sponsored
More from Data pipelines & processing
- Add a UUID Surrogate Key to Each Row in a CSV with Python easy
- Attach Source File Metadata to Records in Python easy
- Build a Python Utility That Detects Duplicate Records Across Multiple Excel Sheets medium
- Check Null Rate Threshold in PySpark DataFrame medium
- Count Records Processed per Category in Python easy
- Create Data Helper Functions in Python for Beginners easy
Keep learning
Related tutorials and quizzes for this topic.