Python Code
Samples
Copy-ready Python snippets by topic and difficulty — short, focused, and runnable in the browser editor.
How to Get the Union of Two Lists Without Duplicates in Python
Merge two lists and remove duplicate values using a set, then convert back to a list.
def union_without_duplicates(list1, list2):
return list(set(list1 + list2))
if __name__ == "__main__":
list_a = [1, 2, 3, 4]
list_b = [3, 4, 5, 6]
result = union_without_duplicates(list_a, list_b)
print(f"Union of {list_a} and {list_b}: {result}")How to Find Duplicate Files by Size and Hash in Python
Recursively scan a directory, group files by size, then hash candidates to identify exact duplicate files.
import hashlib
from pathlib import Path
def hash_file(path, chunk_size=8192):
hasher = hashlib.md5()
with open(path, 'rb') as f:
while chunk := f.read(chunk_size):
hasher.update(chunk)
return hasher.hexdigest()
def find_duplicates(directory):
size_map = {}
for path in Path(dir…How to Use defaultdict(set) in Python to Group Unique Values
Group key-value pairs into a dictionary of sets, automatically creating a new set for each key using defaultdict.
from collections import defaultdict
def track_groups(pairs):
groups = defaultdict(set)
for key, value in pairs:
groups[key].add(value)
return groups
if __name__ == "__main__":
data = [
("fruit", "apple"),
("fruit", "banana"),
("fruit", "apple"),
("veg", "carrot…How to merge dictionaries and sets in Python
Merges multiple dictionaries with the ** unpacking operator and combines sets using union operations into a single structure.
def merge_dictionaries_and_sets(school_dict, teacher_dict, course_dict, student_sets):
"""
Merges multiple dictionaries and sets into a single combined structure.
Demonstrates dict unpacking and set union operations.
"""
# Merge all dictionaries using the unpacking operator (Python 3.9+)
merged…How to Remove Duplicates in Python Preserving Order
Removes duplicate items from a list while keeping the first occurrence order intact using a set for fast membership checks.
def remove_duplicates_preserving_order(items):
seen = set()
result = []
for item in items:
if item not in seen:
seen.add(item)
result.append(item)
return result
if __name__ == "__main__":
sample = [3, 1, 2, 1, 3, 4, 2, 5]
unique_items = remove_duplicates_preserv…Find and Delete Duplicate Files Using Hashing in Python
Walk a directory tree, compute SHA256 hashes for every file, and delete duplicates that share the same hash.
import hashlib
import os
from pathlib import Path
def file_hash(path, block_size=65536):
"""Return SHA256 hash of file content."""
hasher = hashlib.sha256()
with open(path, 'rb') as f:
while chunk := f.read(block_size):
hasher.update(chunk)
return hasher.hexdigest()
def find_and_d…How to Hash Duplicate Photos and Delete Copies in Python
This script hashes image files in a directory using SHA-256 and deletes duplicate copies while keeping the first occurrence, ideal for cleaning up photo libraries.
from pathlib import Path
import hashlib
def file_hash(path, chunk_size=8192):
hasher = hashlib.sha256()
with open(path, "rb") as f:
for chunk in iter(lambda: f.read(chunk_size), b""):
hasher.update(chunk)
return hasher.hexdigest()
def delete_duplicate_photos(directory):
directory …Deduplicate events by ID within a window in Python
Deduplicate event streams by ID within sliding time windows, keeping the newest occurrence per window using heaps and sets.
import heapq
from collections import defaultdict
def deduplicate_events(events, window_size):
"""Return events deduplicated by id, keeping newest within each sliding window."""
# Index events by (timestamp, id) for deterministic ordering
events_by_id = defaultdict(list)
for ts, eid, *payload in events…Generate a Deterministic Hash for Deduplication in Python
Create a stable SHA-256 fingerprint from nested data and file contents to deduplicate records in a data pipeline.
import hashlib
import json
from pathlib import Path
def natural_key_hash(data, salt=""):
"""
Generate a deterministic fingerprint from raw data (dict/list/str).
Uses JSON canonical-ish serialization with sorted keys and SHA-256.
"""
canonical = json.dumps(data, sort_keys=True, separators=(",", ":"…How to Deduplicate Events with At-Least-Once Delivery in Python
Implements an exactly-once processing pattern for at-least-once event delivery by tracking seen event IDs in a set, skipping duplicates.
seen_ids = set()
def process_event(event_id: str, payload: dict) -> dict:
"""Process an event exactly once, ignoring duplicates."""
if event_id in seen_ids:
return {"status": "duplicate", "event_id": event_id}
seen_ids.add(event_id)
return {"status": "processed", "event_id": event_id, **payloa…Idempotent Pipeline Dedupe by Record ID Set in Python
Filters records against a persistent set of seen IDs, returning only new ones and the updated set for idempotent pipeline processing.
def dedupe_records(records, seen_ids=None):
"""Return records whose id has not been seen before."""
if seen_ids is None:
seen_ids = set()
unique = []
for record in records:
record_id = record.get("id")
if record_id not in seen_ids:
seen_ids.add(record_id)
…Implement Exactly-Once Transaction Log in Python
A mock transaction log that deduplicates transaction IDs so each is recorded only once, with a dataclass for records and simple in-memory storage.
from dataclasses import dataclass
from typing import Dict, Optional
@dataclass
class TxnRecord:
txn_id: str
status: str
class ExactlyOnceTxnLog:
def __init__(self) -> None:
self._log: Dict[str, TxnRecord] = {}
self._processed_ids: set = set()
def record(self, txn_id: str, status: s…Inbox pattern consumer dedupe mock in Python
Implements a mock inbox consumer that deduplicates incoming messages by ID, with automatic eviction of old seen IDs to prevent unbounded memory growth.
import json
from collections import deque
from dataclasses import dataclass, field
from hashlib import sha256
from typing import Any
@dataclass
class InboxConsumer:
max_seen: int = 1000
seen_ids: set = field(default_factory=set)
seen_history: deque = field(default_factory=deque)
def _mark_seen(self,…Dedupe processed message IDs in Python
Filters an inbox of messages by removing items whose IDs have already been processed, using a set for fast lookups.
from pathlib import Path
import json
def dedupe_processed_ids(inbox_file: Path, processed_file: Path) -> list:
processed = set(json.loads(processed_file.read_text()))
inbox = json.loads(inbox_file.read_text())
deduped = [item for item in inbox if item["id"] not in processed]
return deduped
if __nam…Exactly Once Idempotent Consumer Store in Python
A mock key-value store that guarantees exactly-once processing by rejecting duplicate message keys in a message or event stream.
from collections import defaultdict
class ExactlyOnceStore:
def __init__(self):
self.processed = defaultdict(set)
self.data = {}
def consume(self, key, value):
if key in self.data:
return False
self.data[key] = value
return True
def get_processed_count…How to deduplicate messages by ID in Python
Track seen message IDs in a set to skip duplicate messages and store unique content in a dict, with exact output showing which messages were added or skipped.
import time
class MessageStore:
def __init__(self):
self.seen_ids = set()
self.messages = {}
def add(self, message_id, content, timestamp=None):
timestamp = timestamp or time.time()
if message_id in self.seen_ids:
return False
self.seen_ids.add(message_…Exactly Once Processing Dedupe Mock in Python
Implements a streaming deduplicator using a set and queue to guarantee each item is processed exactly once while preserving insertion order.
from collections import deque
class DedupeStream:
def __init__(self):
self.seen = set()
self.queue = deque()
def add(self, item):
if item not in self.seen:
self.seen.add(item)
self.queue.append(item)
print(f"Processed: {item} (exactly once)")
…How to Deduplicate Messages in Python by ID
This code consumes a mock inbox of JSON messages and deduplicates them by message ID, keeping either the first or last occurrence.
import json
from collections import OrderedDict
mock_inbox = [
{"id": 1, "message": "hello", "timestamp": "2024-01-01T10:00:00Z"},
{"id": 2, "message": "world", "timestamp": "2024-01-01T10:01:00Z"},
{"id": 1, "message": "hello", "timestamp": "2024-01-01T10:00:00Z"},
{"id": 3, "message": "test", "times…How to Deduplicate Events in Python with SHA256 Hashing
Build an event deduplicator that identifies duplicate inbox messages using SHA256 hashes and tracks duplicate counts per event type.
```python
import hashlib
import json
from collections import defaultdict
class EventDeduplicator:
def __init__(self):
self.seen_hashes = set()
self.duplicate_counts = defaultdict(int)
def process_event(self, event):
event_key = f"{event['event_id']}:{event['timestamp']}"
even…How to Implement an Exactly-Once Deduplication Store in Python
Implement a Python class that deduplicates keys exactly once, tracking first-seen timestamps and duplicate counts.
from datetime import datetime
from typing import Any, Hashable
class ExactlyOnceStore:
def __init__(self) -> None:
self._seen: set[Hashable] = set()
self._first_seen: dict[Hashable, datetime] = {}
self._counts: dict[Hashable, int] = {}
def add(self, key: Hashable, value: Any = None) …Browse by section
Each section groups closely related Python snippets.
Guide: free Python code samples library
Copy-ready Python snippets for learners and developers
PythonSkillset code samples are short, focused examples organised by topic and difficulty. Every snippet is server-rendered HTML — readable by search engines and easy to copy. Open any sample, read the notes, copy the code, then press Try in editor to run it in the browser with Pyodide.
How to use this library
- Pick a topic section — strings, lists, files, functions, and more
- Open a sample, read How it works, and copy the code block
- Run it in the IDE, tweak values, then take a related quiz or tutorial lesson
Samples vs tutorials and challenges
Samples are quick reference — one concept per page. For step-by-step teaching, use our Python tutorials. To test yourself, try quizzes or coding challenges. Clean up style with the Python formatter.