Big data & Spark
PySpark jobs, partitioning, batch processing, and large-dataset transform patterns.
How to Explode an Array Column in Python
This code demonstrates a mock explode operation that converts an array column into multiple rows, similar to Spark's explode function.
import json
def explode_array_column(data, column):
"""Mock explode: split array column into multiple rows."""
exploded = []
for row in data:
values = row.get(column, [])
for value in values:
new_row = dict(row)
new_row[column] = value
exploded.append(n…How to Mock Hive Support in PySpark with unittest.mock
This code demonstrates how to mock Hive support in a PySpark environment using unittest.mock to simulate SQL queries returning fixed data.
from unittest.mock import Mock, patch
def get_hive_tables(spark):
"""Mock Hive support by returning a fixed list of tables."""
return spark.sql("SHOW TABLES").collect()
class HiveTable:
"""Simple class that mimics a Hive table row."""
def __init__(self, database, tableName):
self.database =…Z-Order Optimization in Python
A mock concept demonstrating z-order layout optimization by reassigning z-indices based on areas size.
class ZOrderLayout:
"""
Minimal mock for z-order layout optimization using a stacking score.
Elements overlap; higher z_index is drawn on top.
"""
def __init__(self):
self.elements = []
def add_element(self, name, area, z_index):
self.elements.append({"name": name, "area": area…Browse by section
Each section groups closely related Python snippets.
Big data & Spark — Python code examples
What you will find here
This page collects big data & spark snippets — short, copy-ready Python you can paste into our free online IDE and run without installing anything. Each sample includes a plain-English explanation and the full source code.
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.