How to Strip BOM When Reading UTF-8 Files in Python
Read a UTF-8 text file with Python's pathlib while automatically stripping the Byte Order Mark (BOM) so the first character isn't a hidden glyph.
Python code
16 linesfrom pathlib import Path
def read_text_without_bom(file_path):
"""Read a UTF-8 text file, stripping the BOM if present."""
return Path(file_path).read_text(encoding='utf-8-sig')
if __name__ == "__main__":
# Create a sample file with BOM for demonstration
sample_path = Path("sample_with_bom.txt")
with open(sample_path, "wb") as f:
f.write(b'\xef\xbb\xbfHello, world!\nThis is line two.')
content = read_text_without_bom(sample_path)
print(repr(content))
print(content)
sample_path.unlink()Output
'Hello, world!\nThis is line two.'
Hello, world!
This is line two.How it works
The utf-8-sig codec tells Python to decode the file and remove a leading UTF-8 BOM (\xef\xbb\xbf) if present. Path.read_text opens the file, decodes it, and returns the content in one step. Without this codec, a \ufeff character would appear at the start of the string, breaking equality checks and string operations. The code also works with files that have no BOM, since the codec handles the absence gracefully. This pattern is especially useful when reading files from Windows tools or editors that add a BOM by default.
Common mistakes
- Using `utf-8` instead of `utf-8-sig`, which leaves a hidden `\ufeff` at the start.
- Forgetting that `Path.read_text` only works on `pathlib.Path` objects, not plain strings.
- Manually splitting the BOM with string slicing, which is fragile and only works for UTF-8.
- Assuming the file exists and missing the `FileNotFoundError`—wrap in a try/except or check with `.is_file()`.
Variations
- Use `open(file, 'r', encoding='utf-8-sig')` for classic file objects instead of pathlib.
- Read bytes with `Path.read_bytes()` and strip `.lstrip(b'\xef\xbb\xbf')` for binary control.
Real-world use cases
- Processing config files generated by Windows Notepad or PowerShell that add a UTF-8 BOM.
- Parsing CSV or log files exported from Excel that include a BOM, preventing header mismatch.
- Cleaning data files in an automated ETL job before sending records to a database.
Sponsored
More from Files & data
- Append a Line to a Log File in Python easy
- Audit File Permissions Across a Project in Python easy
- Automatically Detect Corrupted Files Using SHA-256 Checksums in Python easy
- Automatically Highlight Data Validation Errors Inside Excel Files in Python easy
- Build a Command-Line To-Do List Application with Data Persistence in Python easy
- Build a File Index by Relative Path Hash Map in Python easy
Keep learning
Related tutorials and quizzes for this topic.