How to Hash Duplicate Photos and Delete Copies in Python

This script hashes image files in a directory using SHA-256 and deletes duplicate copies while keeping the first occurrence, ideal for cleaning up photo libraries.

Easy Python 3.9+ Aug 9, 2026 Automation & scripting 14 views 0 copies

Python code

40 lines
Python 3.9+
from pathlib import Path
import hashlib

def file_hash(path, chunk_size=8192):
    hasher = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(chunk_size), b""):
            hasher.update(chunk)
    return hasher.hexdigest()

def delete_duplicate_photos(directory):
    directory = Path(directory)
    seen_hashes = {}
    deleted = []
    
    for photo in directory.glob("*"):
        if not photo.is_file() or photo.suffix.lower() not in {".jpg", ".jpeg", ".png", ".gif", ".bmp"}:
            continue
        digest = file_hash(photo)
        if digest in seen_hashes:
            print(f"Deleting duplicate: {photo.name} (identical to {seen_hashes[digest].name})")
            photo.unlink()
            deleted.append(photo.name)
        else:
            seen_hashes[digest] = photo
    
    return deleted

if __name__ == "__main__":
    # Create test files to demonstrate
    test_dir = Path("test_photos")
    test_dir.mkdir(exist_ok=True)
    original = test_dir / "original.jpg"
    duplicate = test_dir / "copy.jpg"
    original.write_bytes(b"same image data")
    duplicate.write_bytes(b"same image data")
    
    removed = delete_duplicate_photos(test_dir)
    print(f"Removed {len(removed)} duplicate(s): {removed}")
    print(f"Remaining files: {[f.name for f in test_dir.iterdir()]}")

Output

stdout
Deleting duplicate: copy.jpg (identical to original.jpg)
Removed 1 duplicate(s): ['copy.jpg']
Remaining files: ['original.jpg']

How it works

The file_hash function reads each file in binary chunks and computes a SHA-256 digest, ensuring memory efficiency even for large images. delete_duplicate_photos scans only image extensions, hashes each file, and uses a dictionary to track previously seen hashes. If a hash repeats, the script deletes the duplicate file with unlink(), preserving the first occurrence. The code demonstrates a practical use of hashlib and pathlib for filesystem automation.

Common mistakes

  • Hashing the whole file at once can exhaust memory for large photos—use chunking.
  • Not filtering by file extension may accidentally process non-image files.
  • Deleting files without extra confirmation can cause accidental data loss.
  • Assuming file names reflect content; hashing reveals true duplicates regardless of names.

Variations

  1. Use `md5` instead of `sha256` if hash collisions are less concerning and speed matters.
  2. Move duplicates to a trash folder instead of permanently deleting them for safety.

Real-world use cases

  • Cleaning up a user's photo library where sync tools created identical copies.
  • Freeing space on a media server by removing duplicate images before archiving.
  • Automating duplicate detection in CI pipelines for image assets in repositories.

Sponsored

Run this sample

Open the browser IDE to tweak the example and see results without installing anything.

Open editor

More from Automation & scripting

Related tutorials and quizzes for this topic.