How to Truncate Lineage Back to a Checkpoint in Python
Walks a linked list of lineage nodes upward to find the nearest checkpoint and returns that node, truncating the lineage.
Python code
31 linesclass LineageNode:
def __init__(self, name, parent=None, checkpoint=None):
self.name = name
self.parent = parent
self.checkpoint = checkpoint
def truncate_at_checkpoint(self):
"""Truncate lineage back to the last checkpoint."""
current = self
while current.checkpoint is None and current.parent is not None:
current = current.parent
return current if current.checkpoint is not None else None
def build_mock_lineage():
"""Create a mock pipeline with checkpoints at intermediate stages."""
node_a = LineageNode("A")
node_b = LineageNode("B", parent=node_a)
node_c = LineageNode("C", parent=node_b)
node_d = LineageNode("D", parent=node_c, checkpoint=True)
node_e = LineageNode("E", parent=node_d)
node_f = LineageNode("F", parent=node_e)
return node_f
if __name__ == "__main__":
leaf = build_mock_lineage()
truncated = leaf.truncate_at_checkpoint()
print(f"Leaf: {leaf.name}")
print(f"Truncated at: {truncated.name if truncated else 'None'}")
print(f"Full lineage: {leaf.name} -> {leaf.parent.name} -> {leaf.parent.parent.name} -> {leaf.parent.parent.parent.name}")
Output
Leaf: F
Truncated at: D
Full lineage: F -> E -> D -> C
How it works
The truncate_at_checkpoint method traverses the parent chain starting from the current node. It keeps moving upward while the current node has no checkpoint and a parent exists. The loop stops when a node with a checkpoint is found or the root is reached. Returning the checkpoint node allows callers to discard any lineage before it, simulating a truncation. This pattern is common in data pipelines where you want to recover to a known good state.
Common mistakes
- Not handling the case where no checkpoint exists, causing a None return and potential AttributeError
- Assuming the leaf always has a parent without checking None
- Confusing the checkpoint node with the node before it — truncation should include the checkpoint
Variations
- Use a while loop to collect all nodes into a list and then slice from the checkpoint.
- Implement the check as a separate function that takes a root node and returns the truncated list.
Real-world use cases
- In data pipelines like Airflow, restoring lineage to the last successful checkpoint after a failure.
- In Spark applications, truncating the RDD lineage graph to avoid long recomputation chains after a cache or checkpoint.
- In event sourcing systems, replaying events from the nearest snapshot to rebuild state efficiently.
Sponsored
More from Big data & Spark
Keep learning
Related tutorials and quizzes for this topic.