Parse JSON and YAML in Python

You know the feeling: a config file that looked perfect opens into a wall of errors, or a JSON payload from an API refuses to cooperate even though it printed fine on screen. When every cloud resource, CI pipeline, and Kubernetes manifest lives in JSON or YAML, fumbling these formats costs you hours and breaks automation in production. This lesson gives you a battle-tested mental model and hands-on patterns for working with JSON and YAML data in Python — so you can parse, transform, and write configs with confidence, not guesswork.

The problem this lesson solves

In DevOps, JSON and YAML are everywhere: API responses, Terraform state files, Docker Compose services, Kubernetes deployments, CI/CD pipeline definitions. Yet Python treats them differently: JSON is built into the standard library, while YAML needs a third-party package. Even seasoned devs hit traps — like failing to handle nested structures, confusing dictionaries with objects, or silently corrupting a manifest during a write-back.

The pain is real: you fetch a Kubernetes pod list, try to grab the namespace field, and your script crashes with a KeyError. Or you load a docker-compose.yml and realize comments and anchors are lost when you re-save. This lesson solves that by giving you a repeatable approach: load → navigate → modify → dump, with edge-case handling baked in.

Core concept / mental model

Think of JSON and YAML as two dialects of the same data language. Both map onto Python's core data structures: dictionaries for objects, lists for arrays, strings, numbers, booleans, and null/None. The difference? JSON is strict and minimal — no comments, no trailing commas. YAML is expressive and forgiving — comments, anchors, multi-line strings, and multiple document streams.

Here's the mental model in plain words:

• JSON = the strict sibling. It's what APIs speak. You parse it with json.loads() and produce it with json.dumps().
• YAML = the human-friendly sibling. It's what config files speak. You parse it with yaml.safe_load() and produce it with yaml.safe_dump().

Both convert to the same Python data structures, so once you're inside, you're working with familiar dicts and lists. The skill isn't remembering syntax — it's knowing which loader to use and how to walk the tree.

🧠 Pro tip: Always use yaml.safe_load() and yaml.safe_dump() in real scripts. The full yaml.load() can execute arbitrary code — a security hole you do not want in your CI pipeline.

How it works step by step

Step 1: Parse the raw text

Start with the raw bytes or string. For JSON, use the json module. For YAML, use PyYAML (or ruamel.yaml for round-trip fidelity).

import json
import yaml

json_text = '{"service": "api", "port": 8080}'
yaml_text = """\
service: api
port: 8080
"""

json_dict = json.loads(json_text)
yaml_dict = yaml.safe_load(yaml_text)

print(json_dict)  # {'service': 'api', 'port': 8080}
print(yaml_dict)  # {'service': 'api', 'port': 8080}

Step 2: Navigate the structure

Both formats produce nested dictionaries and lists. Use square bracket chaining or the .get() method for safety.

config = {
    "services": {
        "web": {"image": "nginx", "replicas": 2},
        "db": {"image": "postgres", "replicas": 1}
    }
}

# Direct access — risky if key missing
print(config["services"]["web"]["image"])  # nginx

# Safe access — returns None if missing
print(config.get("services", {}).get("db", {}).get("replicas"))  # 1

Step 3: Modify and write back

Changes are just dictionary/list operations. Then serialize back to the original format.

# Increase replicas
config["services"]["db"]["replicas"] = 3

# Write JSON
with open("config.json", "w") as f:
    json.dump(config, f, indent=2, sort_keys=True)

# Write YAML
with open("config.yml", "w") as f:
    yaml.safe_dump(config, f, default_flow_style=False, sort_keys=False)

Step 4: Read from files

Use json.load() and yaml.safe_load() for file objects.

with open("config.json") as f:
    data = json.load(f)

with open("config.yml") as f:
    data = yaml.safe_load(f)

Hands-on walkthrough

Example 1: Parse a Kubernetes Pod list (JSON)

Fetch a pod list from the Kubernetes API (simulated here) and extract pod names.

import json

pod_list = json.loads("""
{
  "items": [
    {"metadata": {"name": "api-1"}},
    {"metadata": {"name": "web-2"}}
  ]
}
""")

pod_names = [pod["metadata"]["name"] for pod in pod_list["items"]]
print(pod_names)  # ['api-1', 'web-2']

Example 2: Read and transform a Docker Compose file (YAML)

Imagine you need to bump all image tags to latest in a compose file.

import yaml

with open("docker-compose.yml") as f:
    compose = yaml.safe_load(f)

for service in compose["services"].values():
    service["image"] = service["image"].split(":")[0] + ":latest"

with open("docker-compose-latest.yml", "w") as f:
    yaml.safe_dump(compose, f, sort_keys=False)

Assume docker-compose.yml contains:

services:
  web:
    image: nginx:1.21
  db:
    image: postgres:13

Output docker-compose-latest.yml:

services:
  web:
    image: nginx:latest
  db:
    image: postgres:latest

Example 3: Convert between YAML and JSON

A common DevOps task — translate a YAML config to JSON for an API request.

import json
import yaml

with open("config.yml") as f:
    data = yaml.safe_load(f)

json_output = json.dumps(data, indent=2)
print(json_output)

Example 4: Write a structured file cleanly

When generating configs, always use consistent formatting. For JSON, indent=2 is conventional; for YAML, default_flow_style=False gives block style.

import json
import yaml

base = {"version": "3", "services": {"web": {"image": "nginx", "ports": ["80:80"]}}}

json.dump(base, open("base.json", "w"), indent=2)
yaml.safe_dump(base, open("base.yml", "w"), default_flow_style=False)

These produce human-readable files that pass strict linters.

Compare options / when to choose what

• Use JSON when the data goes over the wire, or when you need strict structure.
• Use YAML when humans write and read configs, and when comments are valuable.
• For round-tripping YAML (preserving comments and formatting), ruamel.yaml is the choice.

⚠️ Caution: PyYAML's safe_dump will convert tuples to lists and drop comments. If you need pristine round-trip, use ruamel.yaml.

Troubleshooting & edge cases

My YAML file uses anchors, and safe_load flattens them

Anchors are a YAML feature meant to avoid repetition. safe_load expands them — the output is the full merged object. That's usually desired, but if you need to preserve anchors, ruamel.yaml can do it.

json.loads fails on trailing commas or comments

That's by design. Strip comments with regex or use YAML for human-authored files. For JSON from sources that shouldn't have comments, you likely have a bug upstream.

KeyError in nested lookup

Use .get() with defaults. A helper function can make deep access painless:

def deep_get(d, path, default=None):
    for key in path:
        if isinstance(d, dict):
            d = d.get(key)
        else:
            return default
        if d is None:
            return default
    return d if d is not None else default

yaml.safe_dump produces ugly single-line output

Force block style with default_flow_style=False. To avoid alphabetical reordering, pass sort_keys=False.

Boolean gotchas in YAML

YAML 1.1 treats yes, no, on, off as booleans. If you have strings like on in your config, quote them to keep them strings.

What you learned & what's next

You now understand the core idea behind working with JSON and YAML data — both map to Python dicts/lists, and you can parse, navigate, modify, and serialize them with confidence. You completed practical exercises covering file reading, transformation, and format conversion. You also know when to choose JSON over YAML and how to dodge common edge cases.

Your next step in the Python for DevOps automation track is to build on this foundation with environment variable management or configuration validation — putting these parsing skills to work in real infrastructure code.

Now go ahead: write a small script that reads a docker-compose.yml, updates a service's image tag, and writes a new file. Then challenge yourself to convert that file to JSON and back again.