How to Explode an Array Column in Python
This code demonstrates a mock explode operation that converts an array column into multiple rows, similar to Spark's explode function.
Python code
21 linesimport json
def explode_array_column(data, column):
"""Mock explode: split array column into multiple rows."""
exploded = []
for row in data:
values = row.get(column, [])
for value in values:
new_row = dict(row)
new_row[column] = value
exploded.append(new_row)
return exploded
if __name__ == "__main__":
data = [
{"id": 1, "tags": ["a", "b"]},
{"id": 2, "tags": ["c"]},
{"id": 3, "tags": []}
]
result = explode_array_column(data, "tags")
print(json.dumps(result, indent=2))Output
[
{
"id": 1,
"tags": "a"
},
{
"id": 2,
"tags": "b"
},
{
"id": 3,
"tags": "c"
}
]How it works
The explode_array_column function iterates over each row in the data list and retrieves the array column using row.get(column, []) to safely handle missing keys. For each value in the array, it creates a shallow copy of the row with dict(row) to avoid mutating the original data, then replaces the array column with the single value and appends it to the result list. Rows with empty arrays (like the third row) produce no output, matching the behavior of explode in PySpark. This mock is useful for understanding the concept before applying it to large datasets.
Common mistakes
- Forgetting to copy the row with `dict(row)` and unintentionally mutating the original data
- Assuming the array column always exists without using `.get()` to handle missing keys
- Confusing this mock with a true distributed explode that handles large datasets efficiently
Variations
- Use a nested list comprehension for a more concise version: `[dict(row, **{column: v}) for row in data for v in row.get(column, [])]`
- For PySpark, use the built-in `explode` function on a DataFrame column
Real-world use cases
- Preparing nested JSON arrays from an API for loading into a relational database as separate rows.
- Transforming event logs where a single record contains multiple user tags or attributes needing individual analysis.
- Simulating distributed explode logic locally for unit tests before deploying to a Spark cluster.
Sponsored
More from Big data & Spark
Keep learning
Related tutorials and quizzes for this topic.