How to Truncate Lineage Back to a Checkpoint in Python

Walks a linked list of lineage nodes upward to find the nearest checkpoint and returns that node, truncating the lineage.

Easy Python 3.9+ Aug 9, 2026 Big data & Spark 16 views 0 copies

Python code

31 lines
Python 3.9+
class LineageNode:
    def __init__(self, name, parent=None, checkpoint=None):
        self.name = name
        self.parent = parent
        self.checkpoint = checkpoint

    def truncate_at_checkpoint(self):
        """Truncate lineage back to the last checkpoint."""
        current = self
        while current.checkpoint is None and current.parent is not None:
            current = current.parent
        return current if current.checkpoint is not None else None


def build_mock_lineage():
    """Create a mock pipeline with checkpoints at intermediate stages."""
    node_a = LineageNode("A")
    node_b = LineageNode("B", parent=node_a)
    node_c = LineageNode("C", parent=node_b)
    node_d = LineageNode("D", parent=node_c, checkpoint=True)
    node_e = LineageNode("E", parent=node_d)
    node_f = LineageNode("F", parent=node_e)
    return node_f


if __name__ == "__main__":
    leaf = build_mock_lineage()
    truncated = leaf.truncate_at_checkpoint()
    print(f"Leaf: {leaf.name}")
    print(f"Truncated at: {truncated.name if truncated else 'None'}")
    print(f"Full lineage: {leaf.name} -> {leaf.parent.name} -> {leaf.parent.parent.name} -> {leaf.parent.parent.parent.name}")

Output

stdout
Leaf: F
Truncated at: D
Full lineage: F -> E -> D -> C

How it works

The truncate_at_checkpoint method traverses the parent chain starting from the current node. It keeps moving upward while the current node has no checkpoint and a parent exists. The loop stops when a node with a checkpoint is found or the root is reached. Returning the checkpoint node allows callers to discard any lineage before it, simulating a truncation. This pattern is common in data pipelines where you want to recover to a known good state.

Common mistakes

  • Not handling the case where no checkpoint exists, causing a None return and potential AttributeError
  • Assuming the leaf always has a parent without checking None
  • Confusing the checkpoint node with the node before it — truncation should include the checkpoint

Variations

  1. Use a while loop to collect all nodes into a list and then slice from the checkpoint.
  2. Implement the check as a separate function that takes a root node and returns the truncated list.

Real-world use cases

  • In data pipelines like Airflow, restoring lineage to the last successful checkpoint after a failure.
  • In Spark applications, truncating the RDD lineage graph to avoid long recomputation chains after a cache or checkpoint.
  • In event sourcing systems, replaying events from the nearest snapshot to rebuild state efficiently.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Big data & Spark

Related tutorials and quizzes for this topic.