How to stream parse JSON arrays in Python

This code demonstrates two generators: one that streams a JSON array as individual chunks, and another that incrementally parses those chunks into Python objects using json.JSONDecoder.

Medium Python 3.9+ Aug 9, 2026 Comprehensions & generators 14 views 0 copies

json generator streaming parsing

Python code

50 lines

Python 3.9+

import json


def json_array_stream(items):
    """Generator that yields JSON-encoded values one at a time."""
    yield "["
    for i, item in enumerate(items):
        if i > 0:
            yield ","
        yield json.dumps(item)
    yield "]"


def parse_json_stream(stream):
    """Consumes a stream of JSON fragments and yields parsed objects."""
    buffer = ""
    for chunk in stream:
        buffer += chunk
        decoder = json.JSONDecoder()
        idx = 0
        while idx < len(buffer):
            try:
                obj, end = decoder.raw_decode(buffer[idx:])
                yield obj
                idx += end
            except json.JSONDecodeError:
                break
        buffer = buffer[idx:]


if __name__ == "__main__":
    data = [
        {"name": "Alice", "age": 30},
        {"name": "Bob", "age": 25},
        {"name": "Carol", "age": 35},
    ]

    # Stream the JSON array one chunk at a time
    chunks = json_array_stream(data)
    print("Streaming chunks:")
    for chunk in chunks:
        print(f"  {chunk!r}")

    # Re-assemble and parse the stream incrementally
    print("\nParsed objects:")
    parsed = list(parse_json_stream(json_array_stream(data)))
    for obj in parsed:
        print(f"  {obj}")

    print(f"\nTotal parsed: {len(parsed)}")

Output

stdout

Streaming chunks:
  '['
  '{"name": "Alice", "age": 30}'
  ','
  '{"name": "Bob", "age": 25}'
  ','
  '{"name": "Carol", "age": 35}'
  ']'

Parsed objects:
  {'name': 'Alice', 'age': 30}
  {'name': 'Bob', 'age': 25}
  {'name': 'Carol', 'age': 35}

Total parsed: 3

How it works

The json_array_stream generator yields each JSON fragment separately, enabling streaming of large arrays without building the entire string in memory. The parse_json_stream uses JSONDecoder.raw_decode to parse as many complete objects as possible from the buffer, while handling incomplete chunks gracefully. By yielding parsed objects immediately, processing can begin before the entire array is received.

Common mistakes

Using `json.loads` on each chunk instead of `raw_decode`, which fails on partial JSON.
Forgetting to update the buffer after parsing, causing data loss or infinite loops.
Assuming all chunks arrive in order and complete without checking for errors.

Variations

Use `ijson` library for very large JSON streams with better performance.
Create a generator that yields a list of all objects at once using `json.loads` for simplicity.

Real-world use cases

Processing huge JSON log files line by line without loading everything into RAM.
Streaming JSON responses from a network socket and parsing records as they arrive.
Building a real-time data pipeline that ingests JSON records from a message queue.

How to stream parse JSON arrays in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Comprehensions & generators

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Comprehensions & generators

Keep learning

Tutorials

Quizzes