Build a File Index by Relative Path Hash Map in Python

Recursively walk a directory and map normalized relative paths to absolute file paths using a defaultdict hash map.

Easy Python 3.9+ Aug 9, 2026 Files & data 18 views 0 copies

os.walk file-index defaultdict filesystem hash-map stdlib

Python code

30 lines

Python 3.9+

import os
from collections import defaultdict


def build_file_index(root_dir):
    index = defaultdict(list)

    for dirpath, dirnames, filenames in os.walk(root_dir):
        for filename in filenames:
            full_path = os.path.join(dirpath, filename)
            relative_path = os.path.relpath(full_path, root_dir)
            normalized_path = os.path.normpath(relative_path)
            index[normalized_path].append(full_path)

    return dict(index)


if __name__ == "__main__":
    test_dir = "sample_files"
    os.makedirs(test_dir, exist_ok=True)
    os.makedirs(os.path.join(test_dir, "subdir"), exist_ok=True)

    with open(os.path.join(test_dir, "a.txt"), "w") as f:
        f.write("hello")
    with open(os.path.join(test_dir, "subdir", "b.py"), "w") as f:
        f.write("print('hi')")

    index = build_file_index(test_dir)
    for rel_path, full_paths in sorted(index.items()):
        print(f"{rel_path} -> {full_paths}")

Output

stdout

a.txt -> ['sample_files/a.txt']
subdir/b.py -> ['sample_files/subdir/b.py']

How it works

os.walk recursively traverses the filesystem starting from a root directory, yielding tuples of dirpath, dirnames, and filenames. For each file, we join the directory path with the filename to form the absolute path. os.path.relpath converts the absolute path into a path relative to the root, and os.path.normpath normalizes separators and redundant components, making keys consistent across platforms. Using defaultdict(list) means every unique relative path automatically gets an empty list we can append to, creating a file index hash map where keys are relative paths and values are lists of absolute file paths.

Common mistakes

Forgetting `os.path.normpath`, leading to inconsistent keys on Windows with backslashes vs. forward slashes
Using `os.path.abspath` on relative paths that are already absolute, creating duplicated prefix noise

Variations

Use `pathlib.Path` with `rglob('*')` to filter and collect matching files
Filter by extension and use a dictionary of sets to dedupe duplicates across hard links

Real-world use cases

Building a manifest of deployed assets to verify every expected file was uploaded.
Creating a searchable map of user-uploaded documents keyed by their storage-relative path.
Powering incremental backups by comparing relative-path indices across snapshots.

Build a File Index by Relative Path Hash Map in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Files & data

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Files & data

Keep learning

Tutorials

Quizzes