Template Method Workflow Steps Base Class in Python
Define a reusable workflow skeleton in a base class and let subclasses fill in each step with the Template Method design pattern.
Python code
70 linesfrom abc import ABC, abstractmethod
class DataPipeline(ABC):
"""Template Method pattern: defines a workflow skeleton."""
def run(self):
"""Template method - defines the algorithm's structure."""
result = {"extracted": False, "transformed": False, "loaded": False}
raw_data = self._extract()
result["extracted"] = True
transformed = self._transform(raw_data)
result["transformed"] = True
self._load(transformed)
result["loaded"] = True
return result
@abstractmethod
def _extract(self):
pass
@abstractmethod
def _transform(self, data):
pass
@abstractmethod
def _load(self, data):
pass
class CSVToDatabasePipeline(DataPipeline):
"""Concrete implementation for CSV to DB workflow."""
def _extract(self):
print("Extracting data from CSV...")
return ["alice@example.com", "bob@example.com"]
def _transform(self, data):
print("Transforming emails...")
return [email.strip().lower() for email in data]
def _load(self, data):
print(f"Loading {len(data)} rows into database...")
for email in data:
print(f" - INSERT {email}")
class APIValidationPipeline(DataPipeline):
"""Concrete implementation for API validation workflow."""
def _extract(self):
print("Fetching API responses...")
return {"status": "ok", "code": 200}
def _transform(self, data):
print("Validating response...")
return data["status"] == "ok"
def _load(self, data):
print(f"Saving validation result: {data}")
if __name__ == "__main__":
print("=== CSV Pipeline ===")
csv_result = CSVToDatabasePipeline().run()
print(f"Result: {csv_result}")
print("\n=== API Pipeline ===")
api_result = APIValidationPipeline().run()
print(f"Result: {api_result}")
Output
=== CSV Pipeline ===
Extracting data from CSV...
Transforming emails...
Loading 2 rows into database...
- INSERT alice@example.com
- INSERT bob@example.com
Result: {'extracted': True, 'transformed': True, 'loaded': True}
=== API Pipeline ===
Fetching API responses...
Validating response...
Saving validation result: True
Result: {'extracted': True, 'transformed': True, 'loaded': True}
How it works
The DataPipeline base class defines a run() method that lays out the fixed algorithm: extract, transform, load. Subclasses implement only the abstract step methods, so the overall flow stays consistent while each step varies. ABC and @abstractmethod enforce that every concrete pipeline implements all required steps, preventing silent omissions. The template method centralizes shared logic like progress tracking and result reporting, so subclasses avoid duplicating orchestration code. This keeps the algorithm invariant while allowing steps to be customized per pipeline.
Common mistakes
- Forgetting to call `super().run()` when overriding the template method in a subclass
- Trying to instantiate the abstract base class directly instead of a concrete subclass
- Adding new required steps without updating the abstract method list and all subclasses
- Putting concrete step logic in the base class instead of leaving steps abstract
Variations
- Add a `hook()` method in the base class with a default no-op body that subclasses can optionally override
- Wrap each step in try/except blocks within the template method for centralized error handling
Real-world use cases
- Defining a consistent ETL pipeline across different data sources and sinks in a data engineering project.
- Standardizing API request lifecycle with auth, retry, and logging steps while allowing endpoint-specific parsing.
- Creating a multi-stage model training workflow where preprocessing, training, and evaluation stages are customized per experiment.
Sponsored
More from System design patterns
- Build a BFF (Backend for Frontend) Mock Aggregator in Python medium
- Builder pattern for mocking complex objects in Python easy
- Circuit Breaker Pattern in Python: Closed, Open, and Half-Open States medium
- Create a Data Helper Class in Python easy
- Domain Driven Design Aggregate Root Example in Python medium
- Facade Pattern in Python with Mock Simplification medium
Keep learning
Related tutorials and quizzes for this topic.