Trigger Jobs on File Arrival

You've built your ETL pipeline, your tables are shiny, and your notebooks run like clockwork — until someone asks, "Can you make this run automatically the moment a file lands in S3?" Manual triggers are the silent killer of data engineering SLAs: files arrive at 3 AM, dashboards go stale, and your pager goes off because no one re-ran the job. In this lesson, you'll learn how to trigger jobs on file arrival via Auto Loader so your pipelines react instantly to new data — no cron hacks, no polling scripts, no missed updates.

The problem this lesson solves

Most ingestion pipelines run on fixed schedules: every hour, every night, every Sunday. Schedules assume data arrives on time — but reality disagrees. A partner uploads at 2:14 PM instead of 2:00 PM; a webhook retries at 4:00 AM; a data lake sync takes 40 minutes longer than expected.

The result? Stale data, redundant processing, and sprint-level debugging when you finally notice the last_updated column hasn't moved in 18 hours.

Polling is the naive fix — a job that checks "is there anything new?" every five minutes. That wastes cluster time, burns Databricks DBUs, and still introduces latency. A better approach is event-driven ingestion: a notification fires the moment a file arrives, and your pipeline starts immediately.

Auto Loader gives you that capability natively. It watches cloud storage, increments on new files, and — critically — can trigger downstream jobs the instant it detects data. Combined with Databricks Jobs' file arrival trigger, you get a fully event-driven pipeline with no glue code.

By the end of this lesson, you'll understand not just the mechanics, but why Auto Loader is the right tool, when to choose it over polling or triggers on tables, and how to avoid the common pitfalls that trip up first-time users.

Core concept / mental model

Think of Auto Loader as a mailroom clerk with a guaranteed delivery slip — not a receptionist who checks the front desk every few minutes.

Traditional polling: your code periodically asks "any mail?" — the S3 ListObjects API call. That's expensive and slow. Auto Loader instead registers for object notifications (like S3 event notifications or Azure Event Grid) and processes files only when they actually arrive. The clerk only rings your doorbell when there's a package.

Here's the key mental model in three parts:

1. Incremental ingestion core — Auto Loader maintains a checkpoint (in a schema called _checkpoint_location) recording which files it has already processed. Every new run compares the current file listing or event notifications against that checkpoint and processes only the new files.
2. Trigger mechanism — Databricks Jobs has a File arrived trigger type. Instead of "Run every hour," you configure it to watch a cloud storage path, and the job starts only when new files appear there. This pairs naturally with Auto Loader's incremental logic.
3. Two-step handshake — Auto Loader processes the new files into a Delta table. A downstream job or Delta Live Tables pipeline can then be triggered after that ingest job completes, forming a file-arrival cascade.

The beauty is that Auto Loader uses structured streaming under the hood: it reads data in micro-batches, is fault-tolerant via checkpoints, and can run incrementally both in streaming mode and in trigger-once mode (a batch run on all available new files).

How it works step by step

To trigger jobs on file arrival via Auto Loader, you follow a logical sequence:

1. Enable file notifications on your cloud storage. For S3, that means enabling s3:ObjectCreated:* events to an SQS queue. For Azure, configure Event Grid. Databricks makes this easier with cloudFiles.inferColumnTypes only if needed — the permissions are what matter.
2. Write your ingestion notebook or script. Use spark.readStream.format("cloudFiles") with options like cloudFiles.format = "json" (or csv, parquet, etc.) and cloudFiles.includeExistingFiles = "true" if you want to backfill existing files.
3. Choose a checkpoint location. This is a directory in cloud storage that stores auto-loader state. It must be unique per source + target combination — don't share it between different tables.
4. Write to a Delta table. Use writeStream.outputMode("append").format("delta").outputMode("append"). The trigger can be once (.trigger(once=True)) for manual retries or default for streaming.
5. Create a Databricks Job with the notebook as a task, and set the trigger type to File arrived. This requires a cluster with the needed IAM roles — the same cloud credentials that can read from the storage location.
6. Optionally chain downstream jobs. Add a second task that depends on the first task (via the task dependency feature) — e.g., a job that runs a SQL update or a dashboard refresh.

Cause → effect: when a file is uploaded, the cloud storage event triggers Databricks Jobs → the job starts the cluster, runs the Auto Loader notebook → the streaming query ingests the new file(s) and appends to the Delta table → the trigger(once=True) finishes → the next job in the chain starts.

Hands-on walkthrough

Let's put this into practice. You need access to a Databricks workspace with a cluster that can read from S3/Azure/ADLS. I'll use AWS S3 in the examples, but the pattern is identical for Azure.

Example 1: Basic Auto Loader streaming read into Delta

Create a notebook and attach it to a cluster, then run:

from pyspark.sql.types import StructType, StructField, StringType, IntegerType

schema = StructType([
    StructField("id", IntegerType()),
    StructField("name", StringType()),
    StructField("ts", TimestampType())
])

input_path = "s3://your-bucket/incoming/"
checkpoint = "s3://your-bucket/checkpoints/my_table/"
table_path = "s3://your-bucket/tables/my_table/"

df = (
    spark.readStream
    .format("cloudFiles")
    .option("cloudFiles.format", "json")
    .option("cloudFiles.includeExistingFiles", "true")
    .option("cloudFiles.schemaLocation", f"{checkpoint}/schema")
    .schema(schema)
    .load(input_path)
)

stream = (
    df.writeStream
    .format("delta")
    .outputMode("append")
    .option("checkpointLocation", f"{checkpoint}/stream")
    .trigger(once=True)
    .start(table_path)
)

stream.awaitTermination()

If you run this notebook manually, it will process all files currently in the bucket and then stop. The trigger(once=True) is crucial — without it, the stream runs forever, waiting for new files.

Example 2: Full streaming mode for continuous ingestion

If you prefer a long-running stream (Spark structured streaming micro-batches every 10 seconds), remove the trigger line:

# Remove .trigger(once=True) to stream continuously
stream = (
    df.writeStream
    .format("delta")
    .outputMode("append")
    .option("checkpointLocation", f"{checkpoint}/stream")
    .start(table_path)
)

# Do NOT call awaitTermination() in a notebook — it will block
# Instead, run it as a job with continuous or default trigger

In a notebook, you should avoid awaitTermination() because it blocks until the stream stops — the notebook will hang. Use this mode in a job with a continuous trigger, or just keep the notebook cell active.

Example 3: Chain with a downstream job

Now create a second notebook that does some transformation or sends an alert:

delta_df = spark.read.table("my_db.my_table")
count = delta_df.count()
print(f"Total rows: {count}")
# Imagine a dashboard refresh or an Airflow call here

Then in the Databricks Jobs UI:

1. Create a new job called ingest_and_process.
2. Add the ingestion notebook as task 1.
3. Add the downstream notebook as task 2 and set Depends on task 1.
4. Under Trigger, select File arrived.
5. Provide the input_path (e.g., s3://your-bucket/incoming/).
6. Save and enable.

Now upload a test file to that S3 path. Within seconds, the job starts, runs the Auto Loader ingestion, then triggers the downstream task.

Compare options / when to choose what

You might wonder: why not just trigger a Job when a table is updated? Or why use Auto Loader at all when I can poll? Here's a comparison to help you decide:

When to choose what:

• Use Auto Loader with file-arrival trigger as your default for batch-oriented ingestion that must be timely. It's the sweet spot between cost and latency.
• Use polling only for non-production or development environments where simplicity beats efficiency.
• Use continuous streaming when you need true sub-second latency and can afford a always-on cluster.
• Use Delta Live Tables if you already use DLT and want a simpler orchestration layer — but note it still needs a trigger (file arrival or scheduled).

Consider variations:

• Directory listing mode: Auto Loader can also work without notifications (polls S3 for new files) — use cloudFiles.useNotifications = false — this is simpler but slower and more expensive.
• Event-based with SQS: If you need more control over the queue, you can set up the SQS queue manually and configure Auto Loader to read from it.
• Azure Event Grid / Google Pub/Sub: The same pattern applies — Auto Loader is cloud-agnostic.

Troubleshooting & edge cases

Here are the most common gotchas when triggering jobs on file arrival via Auto Loader:

1. Job doesn't start when file arrives

• The event notification isn't set up. For S3, you must configure S3 bucket events to publish to SQS, and the Databricks instance profile must have permission to read from that SQS queue and list the bucket.
• Check the Job run history — if it says "skipped" or "pending," inspect the job's trigger configuration.

2. Schema inference fails on the first file

• Solution: Provide an explicit schema, or use cloudFiles.schemaLocation with cloudFiles.inferColumnTypes = true. For nested JSON, schema inference can be fragile — always provide a schema for production.

3. Files are processed twice

• Cause: sharing the same checkpointLocation across multiple query instances, or re-deploying with a different code path.
• Fix: Use a unique checkpoint per table per source. Never reuse checkpointLocation across different tables.

4. trigger(once=True) seems to hang

• It waits for the stream to stop naturally — if you have no new files, it will finish quickly. But if you have a long micro-batch (e.g., huge file), it may take time. Set a generous job timeout.
• In a notebook, the morning is fine — the cell returns when done. In a job, it's fine too.

5. IAM permissions for SQS / Event Grid

• Auto Loader needs s3:GetObject, s3:ListBucket, and sqs:ReceiveMessage (for SQS mode). For Azure, you need Event Grid listener permissions and storage blob read. The error message typically says "Access Denied" — but which permission is missing?
• Tip: Simulate a read using spark.read.format("cloudFiles") in a notebook to verify permissions before wiring the job.

6. Existing files not picked up

• Add cloudFiles.includeExistingFiles = "true" (default is false for new streams). If you already ran the stream, new files added to the source afterward are not ingested — you must either re-create the checkpoint or use a different directory.

What you learned & what's next

You now understand how to trigger jobs on file arrival via Auto Loader — you've seen the mental model, the step-by-step wiring, hands-on examples, and the pitfalls that could have cost you hours. Specifically, you can:

• Explain how Auto Loader uses cloud notifications and checkpoints to incrementally process new files.
• Set up a Databricks Job with a File arrived trigger and chain downstream tasks.
• Decide when to use Auto Loader over polling or continuous streaming.

This is a cornerstone of event-driven data engineering on the Lakehouse. Your next lesson in the Databricks track will cover orchestrating multi-step pipelines with Databricks Jobs — you'll learn how to parameterize tasks, handle retries, and monitor run status across complex workflows. With file-arrival triggers under your belt, you'll be ready to build production-grade pipelines that react instantly to your data's heartbeat.

Now, open your workspace and create a test job with a 2-minute workflow: a file-arrival trigger, an Auto Loader ingestion, and a simple print notification. Make it fail on purpose — then fix it. That's how you'll make this stick.