How to Build a DAG Execution Stage Calculator in Python
Computes the execution stages of a directed acyclic graph (DAG) by grouping nodes that become ready simultaneously using topological sorting with Kahn's algorithm.
Python code
44 linesfrom collections import defaultdict, deque
def get_stages(edges):
"""Return list of stages, where each stage is a list of nodes
that become ready at the same time in a DAG."""
graph = defaultdict(list)
in_degree = defaultdict(int)
nodes = set()
for src, dst in edges:
graph[src].append(dst)
in_degree[dst] += 1
if src not in in_degree:
in_degree[src] = 0
nodes.add(src)
nodes.add(dst)
queue = deque([node for node in nodes if in_degree[node] == 0])
stages = []
while queue:
stage = list(queue)
stages.append(sorted(stage))
for _ in range(len(queue)):
node = queue.popleft()
for neighbor in graph[node]:
in_degree[neighbor] -= 1
if in_degree[neighbor] == 0:
queue.append(neighbor)
return stages
if __name__ == "__main__":
dag_edges = [
("parse", "validate"),
("validate", "transform"),
("transform", "load"),
("validate", "aggregate"),
("aggregate", "load"),
("fetch", "parse"),
]
print(get_stages(dag_edges))
Output
[['fetch'], ['parse'], ['validate'], ['aggregate', 'transform'], ['load']]
How it works
The code builds a graph using adjacency lists and tracks in-degrees for each node. Nodes with zero in-degree are initially ready and placed in a queue. At each stage, all currently queued nodes are captured as a stage, then their outgoing edges are processed to decrement neighbor in-degrees. Newly zero in-degree nodes become ready for the next stage. This is essentially Kahn's algorithm for topological sort, but grouping nodes by their parallel readiness level.
Common mistakes
- Forgetting to initialize in-degree for source nodes to 0, causing LookupError.
- Assuming the input graph is connected; isolated nodes need handling (though they appear as stages).
- Not sorting stages, leading to non-deterministic output order.
Variations
- Use networkx's topological_generations for a ready-made solution.
- Implement with recursive DFS and track a 'level' counter for each node.
Real-world use cases
- Planning parallel execution of data pipeline tasks in a workflow engine like Airflow.
- Determining minimal layers for distributed Spark job execution where dependencies exist.
- Scheduling build steps in a CI/CD system to maximize parallelism.
Sponsored
More from Big data & Spark
Keep learning
Related tutorials and quizzes for this topic.