How to Load, Save, and Split JSON Data in Python
Provides helper functions to load, save, and split JSON dictionary data for simple ML pipeline preprocessing.
Python code
42 linesimport json
from pathlib import Path
def load_json_data(file_path):
"""Load JSON data from a file, returning an empty dict if missing."""
path = Path(file_path)
if path.exists():
with path.open("r", encoding="utf-8") as f:
return json.load(f)
return {}
def save_json_data(data, file_path):
"""Save data to a JSON file with pretty formatting."""
path = Path(file_path)
with path.open("w", encoding="utf-8") as f:
json.dump(data, f, indent=2)
def split_data(data, train_ratio=0.8):
"""Split dictionary data into train and test subsets by ratio."""
items = list(data.items())
split_index = int(len(items) * train_ratio)
train_items = items[:split_index]
test_items = items[split_index:]
return dict(train_items), dict(test_items)
if __name__ == "__main__":
sample = {"user1": 25, "user2": 30, "user3": 22, "user4": 35, "user5": 28}
save_json_data(sample, "temp_data.json")
loaded = load_json_data("temp_data.json")
train, test = split_data(loaded, train_ratio=0.6)
print("Loaded data:", loaded)
print("Train set:", train)
print("Test set:", test)
# Cleanup temp file
Path("temp_data.json").unlink()
Output
Loaded data: {'user1': 25, 'user2': 30, 'user3': 22, 'user4': 35, 'user5': 28}
Train set: {'user1': 25, 'user2': 30, 'user3': 22}
Test set: {'user4': 35, 'user5': 28}
How it works
The load_json_data function checks if a file exists before reading, preventing exceptions for missing files. save_json_data uses json.dump with indent=2 for readable output. split_data slices the dictionary items into train and test subsets based on the given ratio, preserving order. The __main__ block demonstrates the full workflow with a sample dataset and cleans up the temporary file.
Common mistakes
- Using `json.load` instead of `json.loads` for string data, or vice versa.
- Assuming the file exists without checking, leading to FileNotFoundError.
- Not specifying encoding='utf-8' can cause Unicode issues on some platforms.
Variations
- Use `pandas.read_json` and `train_test_split` for larger datasets.
- Add a seed parameter to `split_data` for reproducible shuffling.
Real-world use cases
- Loading raw feature data from JSON files before splitting into training and validation sets in a batch ML job.
- Persisting intermediate pipeline outputs (e.g., cleaned features) to JSON for reproducibility.
- Splitting a small labeled dataset (stored as a dictionary) into train/test for quick model prototyping.
Sponsored
More from ML engineering pipelines
- Bayesian Optimization in Python: A Simplified Mock Implementation medium
- Build a Data Helper Class in Python for ML Pipelines easy
- Build a Mock Random Forest Classifier in Python easy
- Champion Challenger Deployment Mock in Python easy
- Compare Model A vs Model B Metrics in Python easy
- Create a Minimal Great Expectations Suite Mock in Python easy
Keep learning
Related tutorials and quizzes for this topic.