Read Streaming Data with Structured Streaming

You've built batch pipelines that process data on a schedule, but what happens when your data arrives continuously — every second, every minute, from sensors, clickstreams, or transactions? Waiting for the next batch run means stale insights, delayed alerts, and a bottleneck in your data pipeline. In this lesson, you'll learn how to read streaming data with Structured Streaming in Databricks, turning your Spark skills toward real-time data processing without abandoning the familiar DataFrame API.

The problem this lesson solves

Traditional batch processing treats data as a finite collection. You load it, transform it, and write it out. But real-world data is often unbounded — it never stops arriving. IoT devices emit telemetry around the clock. Web applications generate clickstreams with every user interaction. Financial systems process transactions continuously. If you only ever process data in daily or hourly batches, you're always looking at the past.

Structured Streaming solves this by letting you treat a live data stream like an infinite table. Instead of querying a static DataFrame, you define a query that runs continuously, ingesting new data as it arrives and updating results in near real time. This lesson gives you the foundation to read streaming data with Structured Streaming — the first step toward building streaming ETL pipelines, real-time dashboards, and event-driven applications on Databricks.

Core concept / mental model

Think of a stream as a table that grows indefinitely. With Structured Streaming, Spark reads new records as they arrive and treats them as new rows appended to this ever-expanding table. Your query — the transformations you apply — runs over these new rows each time a batch of data lands, and the results are written out to a sink.

Here's the key mental model: Structured Streaming uses micro-batches. You set a trigger interval (for example, 1 second). Every interval, Spark checks for new data, processes it as a small batch, and updates the output. This gives you exactly-once semantics and fault tolerance through checkpointing, all while using the same DataFrame operations you already know.

A streaming DataFrame is the entry point. You create it from a source — a directory of files, a Kafka topic, a socket, or an Auto Loader-managed cloud storage path. In Databricks, the most common source is a directory with JSON, CSV, or Parquet files landing continuously.

Pro tip: You can think of a streaming DataFrame as a batch DataFrame that never ends. The same select(), filter(), and groupBy() operations work. The main difference is the write — you use writeStream and start the query.

How it works step by step

Here is the step-by-step flow to read streaming data in Databricks:

1. Identify the source: Decide where your streaming data lives — cloud object storage (S3, ADLS, GCS), Kafka, or a local directory for testing.

2. Create a streaming DataFrame: Use spark.readStream with the appropriate format (JSON, CSV, Parquet, or cloudFiles for Auto Loader).

3. Apply transformations: Use standard DataFrame API methods like select(), filter(), and withColumn() to shape the data.

4. Define the output: Choose a sink — a Delta table, console, memory, or a streaming sink like Kafka.

5. Start the query: Use writeStream with output mode, trigger, checkpoint location, and then .start().

6. Monitor and manage: Use spark.streams.active to list running queries and stop them when needed.

Each step is straightforward, but the ordering matters — you must define the read, transform, and write before starting the stream.

Hands-on walkthrough

Let's put this into practice in a Databricks notebook. We'll simulate a stream of JSON files landing in a directory and read them with Structured Streaming.

Step 1: Set up the source directory

First, create a directory and a few sample JSON files. In Databricks, you can use dbutils.fs to interact with DBFS:

# Create a directory for streaming data
source_dir = "/FileStore/streaming/input"
dbutils.fs.mkdirs(source_dir)

# Write a sample JSON file
sample_data = '''{"id": 1, "event": "purchase", "amount": 125.50}
{"id": 2, "event": "click", "amount": 0.00}
'''
dbutils.fs.put(source_dir + "/batch1.json", sample_data, overwrite=True)

Step 2: Create a streaming DataFrame

Now, define a streaming read that watches for new files in that directory:

from pyspark.sql.types import StructType, StructField, IntegerType, StringType, DoubleType

# Define schema for better performance
schema = StructType([
    StructField("id", IntegerType(), True),
    StructField("event", StringType(), True),
    StructField("amount", DoubleType(), True)
])

# Read as a streaming DataFrame
streaming_df = (
    spark.readStream
    .schema(schema)
    .json(source_dir)
)

# The DataFrame is now streaming — verify it
print(streaming_df.isStreaming)  # Output: True

Step 3: Apply transformations

Just like a batch DataFrame, you can transform the stream. Let's add a column that flags high-value transactions:

from pyspark.sql.functions import col, when

# Add a flag for transactions > 100
transformed_df = streaming_df.withColumn(
    "high_value", when(col("amount") > 100, True).otherwise(False)
)

Step 4: Write to a sink

For learning, write to the console in append mode. In production, you'd write to a Delta table.

query = (
    transformed_df.writeStream
    .outputMode("append")
    .trigger(processingTime="2 seconds")
    .format("console")
    .option("truncate", "false")
    .start()
)

# Let it run for a few seconds, then stop
import time
time.sleep(10)
query.stop()

Expected console output (simplified):

+---+-------+------+----------+
| id|  event|amount|high_value|
+---+-------+------+----------+
|  1|purchase|125.5 |      true|
|  2|   click|   0.0|     false|
+---+-------+------+----------+

Pro tip: Always specify a schema when reading JSON streaming data. This avoids schema inference overhead and potential type mismatches.

Step 5: Reading with Auto Loader (Databricks recommendation)

For production streaming on cloud storage, use Auto Loader — it incrementally and efficiently processes new files:

autoloader_df = (
    spark.readStream
    .format("cloudFiles")
    .option("cloudFiles.format", "json")
    .schema(schema)
    .load(source_dir)
)

Auto Loader tracks which files have already been processed, eliminating the need for manual file management.

Compare options / when to choose what

When to use what:

• Learning and prototyping → file stream with readStream.
• Production data lake ingestion → Auto Loader.
• Real-time microservices/event-driven → Kafka.
• Just to see output in notebook → console sink.

Troubleshooting & edge cases

"No output" in console

If you don't see output, the stream may not have picked up existing files. By default, file streams only process new files arriving after the stream starts. To include existing files, use option("includeExisting", "true").

streaming_df = (
    spark.readStream
    .option("includeExisting", "true")
    .schema(schema)
    .json(source_dir)
)

Schema mismatch errors

If your JSON files have missing or extra fields, you may get nulls or runtime errors. Always define a schema and consider option("cloudFiles.schemaLocation", ...) for schema evolution.

Streaming queries left running

If you stop a notebook but the stream continues, your cluster may be overwhelmed. Always stop queries in a finally block or use spark.streams.active to find and stop them.

for s in spark.streams.active:
    s.stop()

Checkpoint location conflicts

If you reuse the same checkpoint directory, the query may fail with "Already exists" errors. Use unique paths per stream, especially in tests.

What you learned & what's next

You've learned how to read streaming data with Structured Streaming in Databricks. You can now:

• Create a streaming DataFrame from a file source.
• Apply DataFrame transformations to streaming data.
• Write to a sink using writeStream.
• Choose between file streams, Auto Loader, and Kafka.
• Debug common streaming issues.

This is the first step toward building production-grade streaming pipelines. In the next lesson, you'll explore how to write streaming data efficiently and reliably, covering sinks like Delta Lake and exactly-once semantics. Master this read-side foundation, and you'll be ready to handle the write side with confidence.

Final pro tip: Streaming is not a replacement for batch — it's a complement. Use streaming when data arrives continuously and freshness matters; use batch when you can tolerate delay. Knowing when to use each is a mark of a senior data engineer.