Python Code
Samples
Easy snippets you can copy, study, and run in the browser editor.
Compaction Small Files Mock in Python
Simulates a small-files compaction job by creating small mock files and merging them into a single output file using Python's standard library.
from pathlib import Path
import tempfile
import os
def create_small_files(directory: Path, file_count: int = 5, lines_per_file: int = 3):
"""Create several small mock files with sample content."""
directory.mkdir(exist_ok=True)
for i in range(file_count):
file_path = directory / f"part-{i:04d}.tx…How to Create a Mock Kafka Producer in Python
Build a Kafka producer that generates mock streaming records with JSON serialization and error handling for local testing.
import json
import time
from kafka import KafkaProducer
from kafka.errors import KafkaError
def create_mock_producer(bootstrap_servers="localhost:9092", topic="input-topic"):
"""Create a Kafka producer that generates mock streaming data."""
producer = KafkaProducer(
bootstrap_servers=bootstrap_servers…How to Explode an Array Column in Python
This code demonstrates a mock explode operation that converts an array column into multiple rows, similar to Spark's explode function.
import json
def explode_array_column(data, column):
"""Mock explode: split array column into multiple rows."""
exploded = []
for row in data:
values = row.get(column, [])
for value in values:
new_row = dict(row)
new_row[column] = value
exploded.append(n…How to Filter and Project Spark DataFrames with PySpark SQL
Simulate a SQL SELECT with WHERE using PySpark DataFrame select and filter to project columns and apply conditions.
from pyspark.sql import SparkSession
from pyspark.sql.functions import col
spark = SparkSession.builder.appName("QueryFilterMock").master("local[2]").getOrCreate()
data = [
("Alice", 28, "Engineering"),
("Bob", 35, "Sales"),
("Carol", 32, "Engineering"),
("David", 25, "Marketing"),
("Eve", 29, "E…How to Implement MapReduce Word Count in Python Using a Dict
Simulate a MapReduce word count pipeline in Python with a mock dict, splitting text into words, shuffling, and reducing to frequency counts.
def map_reduce_word_count(text: str) -> dict:
"""Simulate a MapReduce pipeline to count word frequencies."""
# MAP phase: split into words and emit (word, 1) pairs
mapped = []
for word in text.lower().split():
# Clean word of punctuation
clean_word = ''.join(char for char in word if cha…How to Mock DataFrame Schema Columns in Python
Create an empty pandas DataFrame with only the specified column names to mock a schema before any data is loaded.
import pandas as pd
def mock_schema(columns):
return pd.DataFrame(columns=columns)
if __name__ == "__main__":
cols = ["name", "age", "city"]
df = mock_schema(cols)
print(df)
print(f"Columns: {list(df.columns)}, Shape: {df.shape}")How to Mock Hive Support in PySpark with unittest.mock
This code demonstrates how to mock Hive support in a PySpark environment using unittest.mock to simulate SQL queries returning fixed data.
from unittest.mock import Mock, patch
def get_hive_tables(spark):
"""Mock Hive support by returning a fixed list of tables."""
return spark.sql("SHOW TABLES").collect()
class HiveTable:
"""Simple class that mimics a Hive table row."""
def __init__(self, database, tableName):
self.database =…How to Mock Partition Pruning in Python
A dataclass-based mock that filters partitions by year and month to emulate Spark's partition pruning logic.
from dataclasses import dataclass
from typing import List
@dataclass(frozen=True)
class Partition:
id: int
year: int
month: int
class PartitionPruner:
"""Mock partition pruning: only keep partitions that match the filter."""
def __init__(self, partitions: List[Partition]):
self._partiti…How to Mock a File Source Watch Directory in Python
Poll a directory for new files and log changes, simulating a watch directory for data ingestion patterns.
import os
import time
from pathlib import Path
def watch_directory(dir_path: str, poll_interval: float = 1.0, max_iterations: int = 5):
"""
Mock a file-source watch directory by polling for changes.
Returns new files detected during each poll cycle.
"""
directory = Path(dir_path)
directory.mk…How to Mock a Hash Join on Large and Small Tables in Python
This code efficiently joins a large dataset (1000 rows) with a small lookup table (20 rows) by building a dictionary hash lookup, mimicking a hash join strategy used in big data systems.
import random
from pprint import pprint
# Large table: 1000 rows (id, group_id, value)
large = [{"id": i, "group_id": random.randint(1, 20), "value": random.random() * 100} for i in range(1000)]
# Small table: 20 rows (group_id, label)
small = [{"group_id": g, "label": f"Group-{g}"} for g in range(1, 21)]
# Mock a …How to Mock a User-Defined Function (UDF) in Python
Wrap a real UDF implementation with call logging to simulate and track invocations in a data pipeline.
from typing import Any, Callable
# Mock a user-defined function (UDF) that was previously complex or external
def mock_udf(name: str, implementation: Callable[..., Any], *, calls: list[Any]) -> Callable[..., Any]:
"""Wrap a real implementation with call logging to simulate a UDF."""
def wrapper(*args: Any, *…How to select specific columns in Python with SQLite
A reusable function that connects to a SQLite database and returns only the requested columns from a given table.
import sqlite3
def select_pruned_columns(db_path, table, columns):
with sqlite3.connect(db_path) as conn:
cursor = conn.cursor()
col_list = ", ".join(columns)
query = f"SELECT {col_list} FROM {table}"
return cursor.execute(query).fetchall()
if __name__ == "__main__":
conn = sq…Hudi Upsert Mock Copy on Write in Python
Simulates Apache Hudi's Copy-on-Write upsert behavior by merging update records into a deep copy of base records, replacing matches or appending new ones.
import copy
from typing import Dict, List, Any
def upsert_copy_on_write(base_records: List[Dict[str, Any]], updates: List[Dict[str, Any]], key_field: str = "id") -> List[Dict[str, Any]]:
"""Simulate Hudi Copy-on-Write upsert: merge updates into a copy of base records."""
result = copy.deepcopy(base_records)
…Modeling a Hive Metastore Table Schema in Python
A dataclass that mimics a Hive metastore table schema—columns, partition keys, storage format, and location—with helper methods for description and mutation.
from dataclasses import dataclass, field
from typing import Dict, List, Optional
@dataclass
class HiveTable:
"""Simple mock of a Hive metastore table schema."""
name: str
database: str = "default"
columns: List[Dict[str, str]] = field(default_factory=list)
partition_keys: List[Dict[str, str]] = f…Partition Data by Hash Key Mod N in Python
Returns a partition index for a string key by hashing it with MD5 and taking modulo N, then groups sample keys into partitions.
import hashlib
def partition_key(key: str, num_partitions: int) -> int:
"""Return partition index for key using MD5 hash mod N."""
digest = hashlib.md5(key.encode()).hexdigest()
return int(digest, 16) % num_partitions
if __name__ == "__main__":
keys = ["alice", "bob", "carol", "dave", "eve"]
nu…Sliding Window Streaming Mock in Python
A simple Python class that maintains a sliding window of recent streaming values and computes the running average.
import time
import random
class StreamingMock:
"""Produces a stream of numbers using a sliding window."""
def __init__(self, window_size=5):
self.window = []
self.window_size = window_size
def push(self, value):
"""Add a value, sliding the window forward."""
s…Build a Data Helper Class in Python for ML Pipelines
A beginner-friendly Python class that summarizes, filters, and exports ML dataset rows as JSON.
from typing import List, Dict, Any
import json
class DataHelper:
"""Beginner-friendly helpers for ML data pipelines."""
def __init__(self, data: List[Dict[str, Any]]):
self.data = data
self.keys = list(data[0].keys()) if data else []
def summary(self) -> Dict[str, Any]:
"…Create a Minimal Great Expectations Suite Mock in Python
Build a small Python class that mimics a Great Expectations suite, storing and serializing column expectations as JSON.
import json
class GreatExpectationsSuite:
"""A minimal mock of a Great Expectations suite."""
def __init__(self, suite_name, expectations=None):
self.suite_name = suite_name
self.expectations = expectations or []
def add_expectation(self, expectation_type, column=None, kwargs=None):
…How to Build a Data Validation Schema in Python
Create a lightweight validation schema using dataclasses and lambda validators to check fields in a dictionary.
import re
from dataclasses import dataclass, field
from typing import Any, Callable
@dataclass
class Field:
name: str
validator: Callable[[Any], bool]
required: bool = True
def validate(self, value: Any) -> bool:
if not self.required and value is None:
return True
return …How to Build a Mock TFX Pipeline in Python
Simulate a TFX-style ML pipeline with simple Python functions to understand component orchestration, data flow, and artifact passing.
# Mock TFX pipeline to illustrate component orchestration
def CsvExampleGen(data_path):
"""Mock component: Simulates reading CSV data."""
print(f"ExampleGen: Reading from {data_path}")
return {"records": 100, "name": "examples"}
def StatisticsGen(example_artifact):
"""Mock component: Simulates genera…How to Build a Simple ML Pipeline with ZenML in Python
Build a mock machine learning pipeline with ZenML steps for data loading, training, and evaluation, and run it to print the final accuracy.
from zenml import pipeline, step
@step
def load_data() -> dict:
"""Simulate loading data from a source."""
return {"accuracy": 0.0, "loss": 1.0}
@step
def train_model(data: dict) -> dict:
"""Simulate training a model."""
data["accuracy"] = 0.95
data["loss"] = 0.1
return data
@step
def eva…How to Create a Mock Metaflow Flow in Python
Build a minimal Metaflow flow with two sequential steps that pass data between them using instance attributes.
from metaflow import FlowSpec, step, current
class MockFlow(FlowSpec):
"""A minimal Metaflow flow to demonstrate basic steps and branching."""
@step
def start(self):
self.category = "mock"
print(f"Start step for {self.category} flow")
self.next(self.process)
@step
def pr…How to Impute Missing Values with Mean in Python
Replace None values in a list with the mean of the existing values using Python's statistics module.
import statistics
from statistics import mean
def impute_mean(values):
"""Replace None with the mean of the non-None values."""
# Filter out None to compute the mean of existing values
valid = [v for v in values if v is not None]
if not valid:
return values # nothing to impute if all are Non…How to Load CSV Training Data in Python Without Pandas
Load CSV training data using Python's standard library and mock it with io.StringIO for testing, returning headers and rows as dictionaries.
import csv
from pathlib import Path
def load_csv_training_data(file_path: str | Path) -> tuple[list[str], list[dict[str, str]]]:
"""Load CSV training data and return headers plus rows as dictionaries."""
with open(file_path, mode="r", newline="", encoding="utf-8") as csv_file:
reader = csv.DictReader…Browse by section
Each section groups closely related Python snippets.
Guide: free Python code samples library
Copy-ready Python snippets for learners and developers
PythonSkillset code samples are short, focused examples organised by topic and difficulty. Every snippet is server-rendered HTML — readable by search engines and easy to copy. Open any sample, read the notes, copy the code, then press Try in editor to run it in the browser with Pyodide.
How to use this library
- Pick a topic section — strings, lists, files, functions, and more
- Open a sample, read How it works, and copy the code block
- Run it in the IDE, tweak values, then take a related quiz or tutorial lesson
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.