Work with DBFS

Every Databricks user eventually hits the same wall: you create a DataFrame, write it to a path, and then… where did it go? Files land in the Databricks File System (DBFS), a distributed storage layer that feels like a local drive but behaves very differently under the hood. In this lesson, you'll stop guessing and start working with the Databricks file system (DBFS) like a pro — reading, writing, listing, and deleting files with confidence.

The problem this lesson solves

Data engineering is full of file chaos. You need to stage raw CSVs, store intermediate results, share data with teammates, and keep notebooks organized. On Databricks, all of that storage happens in DBFS — but it's not your laptop's filesystem. Paths like /FileStore and /mnt can confuse newcomers, and the old habit of writing files straight to the cluster's local disk is a recipe for lost data the moment your cluster stops or terminates.

Here's the painful truth: if you don't understand DBFS, you'll find yourself asking Where did my file go? after every cluster restart. This lesson fixes that pain by giving you a clear mental model and hands-on commands to manage files reliably.

Core concept / mental model

Think of DBFS as a virtual file system mounted on top of cloud object storage (like Amazon S3, Azure Data Lake Storage, or Google Cloud Storage). It provides a familiar path-based interface—/FileStore/my-file.csv—so your Python and SQL code can read and write without knowing the underlying cloud provider's API.

But remember two critical truths:

1. DBFS is not a local disk. Files stored in DBFS live in the cloud, not on the cluster's ephemeral storage.
2. DBFS is not meant for long-term data lakes. It's perfect for temporary files, libraries, and small datasets, but large production data belongs in external tables stored on cloud storage directly.

Key terms

• DBFS root: The default storage location for your workspace, typically /FileStore.
• dbutils.fs: A Python/ Scala utility for file operations (like cp, mv, rm) — your Swiss Army knife.
• Mount point: A symbolic link from /mnt/ to an external cloud storage path.

A diagram in words

Imagine DBFS as a glass window to your cloud bucket. When you do dbutils.fs.put("/FileStore/hello.txt", "hi"), you're actually writing to an S3 bucket—but you never have to touch the AWS console. The window lets you see and manipulate files with simple paths.

How it works step by step

Working with DBFS boils down to a few core operations. Here's the logical sequence:

1. Check what's already there — list files in a directory with dbutils.fs.ls("/FileStore").
2. Write new data — use dbutils.fs.put() for text, or df.write() to save a DataFrame.
3. Verify the write — list again or use dbutils.fs.head() to peek at file contents.
4. Read it back — with spark.read or dbutils.fs.head().
5. Modify or delete — cp, mv, rm when you need cleanup.
6. Manage access — use DBFS permissions (or RBAC) to control who can see what.

The dbutils.fs command cheat sheet

Hands-on walkthrough

Let's get our hands dirty. In this exercise, you'll create a CSV file, inspect it, and then work with it using Spark.

Step 1: Write a file with dbutils.fs.put

# Create a small CSV file in DBFS
dbutils.fs.put("/FileStore/tutorial/data.csv", """name,age\nAlice,30\nBob,25\n""")
print("File written successfully!")

Expected output:

File written successfully!

Step 2: List files to verify

# List the contents of the tutorial directory
files = dbutils.fs.ls("/FileStore/tutorial")
for f in files:
    print(f.name, f.size)

Expected output:

data.csv 27

Step 3: Read the file back with Spark

df = spark.read.csv("/FileStore/tutorial/data.csv", header=True, inferSchema=True)
df.show()

Expected output:

+-----+---+
| name|age|
+-----+---+
|Alice| 30|
|  Bob| 25|
+-----+---+

Step 4: Save a DataFrame to DBFS

# Write a DataFrame as JSON
df.write.json("/FileStore/tutorial/output_json")

# Check the output
dbutils.fs.ls("/FileStore/tutorial/output_json")

Expected output shows a part-00000-...json file and a _SUCCESS file.

Step 5: Delete the temporary files

# Clean up
dbutils.fs.rm("/FileStore/tutorial/output_json", recurse=True)
print("Deleted!")

Pro tip: Always use recurse=True when removing directories—otherwise you'll get an error.

Compare options / when to choose what

Not all storage is the same on Databricks. Here's how DBFS stacks up against alternatives:

When to use what

• Use DBFS for quick experiments, sample files, and sharing small artifacts with your team.
• Use external locations for your data lake tables—keep large volumes out of DBFS to avoid cost surprises.
• Use local disk for intermediate shuffle files or cache, not for anything you need to keep.

Variations

• SQL approach: You can also access DBFS via SQL with CREATE TABLE pointing to DBFS paths, but dbutils.fs is more flexible for generic file ops.
• Mount points: If you need to work with external storage frequently, create a mount point (/mnt/mybucket) for simpler paths.
• Databricks CLI: For automation outside notebooks, use the Databricks CLI to upload/download files to DBFS.

Troubleshooting & edge cases

Here are the most common DBFS frustrations and how to fix them:

Error: java.io.FileNotFoundException

Cause: The path doesn't exist, or you mistyped the directory.

Fix: Run dbutils.fs.ls on the parent directory to confirm the exact name. Remember—DBFS paths are case-sensitive.

Error: File already exists when writing

Cause: Spark's df.write refuses to overwrite by default.

Fix: Use mode="overwrite" if you intentionally want to replace data.

df.write.mode("overwrite").json("/FileStore/tutorial/output_json")

Error: Cannot delete a file when using rm

Cause: You tried to remove a non-empty directory without recurse=True.

Fix: Add recurse=True to your rm call.

Data disappears after cluster restart

Cause: You wrote to the local disk (e.g., /tmp) instead of DBFS.

Fix: Always write to DBFS paths (starting with /FileStore) or external mounts.

Performance issues with large files in DBFS

Cause: DBFS isn't designed for heavy read/write workloads.

Fix: Move large datasets to external storage and use spark.read directly on cloud paths.

What you learned & what's next

You now have a solid grasp of working with the Databricks file system (DBFS). Let's recap what you've mastered:

• You can explain what DBFS is and why it matters for data engineering.
• You know the key dbutils.fs commands for listing, writing, reading, copying, and deleting files.
• You can write a DataFrame to DBFS and read it back.
• You can decide when to use DBFS versus external storage.
• You can troubleshoot the most common DBFS errors.

What's next? Now that you can move files around, it's time to learn how to ingest data from various sources into your Databricks environment. In the next lesson, you'll build on this foundation to read data from cloud storage, APIs, and databases.

Pro tip: Practice by creating a scratch directory in /FileStore, writing a few files, and cleaning them up. The more you use dbutils.fs, the faster it becomes muscle memory.