How to Implement collect_list in Python
Group rows by a key and collect all corresponding values into a list — a pure-Python mock of Spark's collect_list aggregation.
Python code
19 linesfrom collections import defaultdict
def collect_list(rows, key_field, value_field):
grouped = defaultdict(list)
for row in rows:
grouped[row[key_field]].append(row[value_field])
return dict(grouped)
if __name__ == "__main__":
data = [
{"dept": "sales", "emp": "alice"},
{"dept": "sales", "emp": "bob"},
{"dept": "eng", "emp": "carol"},
{"dept": "eng", "emp": "dave"},
{"dept": "eng", "emp": "erin"},
]
result = collect_list(data, "dept", "emp")
for dept in sorted(result):
print(f"{dept}: {result[dept]}")Output
eng: ['carol', 'dave', 'erin']
sales: ['alice', 'bob']How it works
This function mimics the behavior of Spark's collect_list by iterating over a list of dictionaries and appending each value to a grouped list using a defaultdict. The defaultdict(list) automatically creates a new list for each new key, avoiding manual key-existence checks. Converting the result to a regular dict keeps the output clean and JSON-serializable. Sorting the keys before printing ensures deterministic output. This pattern is useful for in-memory prototyping before scaling to a Spark DataFrame.
Common mistakes
- Forgetting to convert defaultdict back to dict, which can affect serialization
- Assuming the input rows are sorted by key — sort at display time instead
- Using a plain dict without checking key existence, causing KeyError
- Mixing up key_field and value_field parameter order
Variations
- Use pandas groupby + apply(list) for tabular data
- Use itertools.groupby after sorting rows by the key field
Real-world use cases
- Prototyping aggregation logic locally before running the same operation on a Spark cluster.
- Grouping log entries by service name to collect all error messages per service for quick triage.
- Preparing a mapping of user IDs to their transaction lists for a recommendation dashboard.
Sponsored
More from Big data & Spark
Keep learning
Related tutorials and quizzes for this topic.