How to Hash Duplicate Photos and Delete Copies in Python
This script hashes image files in a directory using SHA-256 and deletes duplicate copies while keeping the first occurrence, ideal for cleaning up photo libraries.
Python code
40 linesfrom pathlib import Path
import hashlib
def file_hash(path, chunk_size=8192):
hasher = hashlib.sha256()
with open(path, "rb") as f:
for chunk in iter(lambda: f.read(chunk_size), b""):
hasher.update(chunk)
return hasher.hexdigest()
def delete_duplicate_photos(directory):
directory = Path(directory)
seen_hashes = {}
deleted = []
for photo in directory.glob("*"):
if not photo.is_file() or photo.suffix.lower() not in {".jpg", ".jpeg", ".png", ".gif", ".bmp"}:
continue
digest = file_hash(photo)
if digest in seen_hashes:
print(f"Deleting duplicate: {photo.name} (identical to {seen_hashes[digest].name})")
photo.unlink()
deleted.append(photo.name)
else:
seen_hashes[digest] = photo
return deleted
if __name__ == "__main__":
# Create test files to demonstrate
test_dir = Path("test_photos")
test_dir.mkdir(exist_ok=True)
original = test_dir / "original.jpg"
duplicate = test_dir / "copy.jpg"
original.write_bytes(b"same image data")
duplicate.write_bytes(b"same image data")
removed = delete_duplicate_photos(test_dir)
print(f"Removed {len(removed)} duplicate(s): {removed}")
print(f"Remaining files: {[f.name for f in test_dir.iterdir()]}")
Output
Deleting duplicate: copy.jpg (identical to original.jpg)
Removed 1 duplicate(s): ['copy.jpg']
Remaining files: ['original.jpg']
How it works
The file_hash function reads each file in binary chunks and computes a SHA-256 digest, ensuring memory efficiency even for large images. delete_duplicate_photos scans only image extensions, hashes each file, and uses a dictionary to track previously seen hashes. If a hash repeats, the script deletes the duplicate file with unlink(), preserving the first occurrence. The code demonstrates a practical use of hashlib and pathlib for filesystem automation.
Common mistakes
- Hashing the whole file at once can exhaust memory for large photos—use chunking.
- Not filtering by file extension may accidentally process non-image files.
- Deleting files without extra confirmation can cause accidental data loss.
- Assuming file names reflect content; hashing reveals true duplicates regardless of names.
Variations
- Use `md5` instead of `sha256` if hash collisions are less concerning and speed matters.
- Move duplicates to a trash folder instead of permanently deleting them for safety.
Real-world use cases
- Cleaning up a user's photo library where sync tools created identical copies.
- Freeing space on a media server by removing duplicate images before archiving.
- Automating duplicate detection in CI pipelines for image assets in repositories.
Sponsored
More from Automation & scripting
- Aggregate Log Errors Count by Hour in Python easy
- Automate Tweeting New Blog Posts in Python easy
- Automatically Clean Temporary Files from Applications Using Python medium
- Automatically Download the Latest Software Release from GitHub with Python medium
- Automatically Generate Charts from CSV Files with One Command medium
- Automatically Generate Hardware Inventory Reports in Python easy
Keep learning
Related tutorials and quizzes for this topic.