How to Build a DAG Execution Stage Calculator in Python

Computes the execution stages of a directed acyclic graph (DAG) by grouping nodes that become ready simultaneously using topological sorting with Kahn's algorithm.

Medium Python 3.9+ Aug 9, 2026 Big data & Spark 15 views 0 copies

Python code

44 lines
Python 3.9+
from collections import defaultdict, deque


def get_stages(edges):
    """Return list of stages, where each stage is a list of nodes
    that become ready at the same time in a DAG."""
    graph = defaultdict(list)
    in_degree = defaultdict(int)
    nodes = set()

    for src, dst in edges:
        graph[src].append(dst)
        in_degree[dst] += 1
        if src not in in_degree:
            in_degree[src] = 0
        nodes.add(src)
        nodes.add(dst)

    queue = deque([node for node in nodes if in_degree[node] == 0])
    stages = []

    while queue:
        stage = list(queue)
        stages.append(sorted(stage))
        for _ in range(len(queue)):
            node = queue.popleft()
            for neighbor in graph[node]:
                in_degree[neighbor] -= 1
                if in_degree[neighbor] == 0:
                    queue.append(neighbor)

    return stages


if __name__ == "__main__":
    dag_edges = [
        ("parse", "validate"),
        ("validate", "transform"),
        ("transform", "load"),
        ("validate", "aggregate"),
        ("aggregate", "load"),
        ("fetch", "parse"),
    ]
    print(get_stages(dag_edges))

Output

stdout
[['fetch'], ['parse'], ['validate'], ['aggregate', 'transform'], ['load']]

How it works

The code builds a graph using adjacency lists and tracks in-degrees for each node. Nodes with zero in-degree are initially ready and placed in a queue. At each stage, all currently queued nodes are captured as a stage, then their outgoing edges are processed to decrement neighbor in-degrees. Newly zero in-degree nodes become ready for the next stage. This is essentially Kahn's algorithm for topological sort, but grouping nodes by their parallel readiness level.

Common mistakes

  • Forgetting to initialize in-degree for source nodes to 0, causing LookupError.
  • Assuming the input graph is connected; isolated nodes need handling (though they appear as stages).
  • Not sorting stages, leading to non-deterministic output order.

Variations

  1. Use networkx's topological_generations for a ready-made solution.
  2. Implement with recursive DFS and track a 'level' counter for each node.

Real-world use cases

  • Planning parallel execution of data pipeline tasks in a workflow engine like Airflow.
  • Determining minimal layers for distributed Spark job execution where dependencies exist.
  • Scheduling build steps in a CI/CD system to maximize parallelism.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Big data & Spark

Related tutorials and quizzes for this topic.