How to Detect File Encoding: UTF-8 vs Latin-1 in Python

Detect whether a file is UTF-8 or Latin-1 encoded by attempting a UTF-8 decode and falling back to Latin-1.

Easy Python 3.9+ Aug 9, 2026 Files & data 13 views 0 copies

file-encoding utf-8 latin-1 encoding-detection binary-file

Python code

15 lines

Python 3.9+

import sys

def detect_encoding(file_path):
    with open(file_path, 'rb') as f:
        raw = f.read()
    
    try:
        raw.decode('utf-8')
        return 'UTF-8'
    except UnicodeDecodeError:
        return 'latin1'

if __name__ == "__main__":
    file_path = sys.argv[1] if len(sys.argv) > 1 else 'sample.txt'
    print(f"{file_path}: {detect_encoding(file_path)}")

Output

stdout

sample.txt: UTF-8

How it works

The function reads the file as raw bytes and tries to decode them as UTF-8. UTF-8 is a superset of ASCII and has specific byte patterns, so any valid UTF-8 file decodes successfully. If decoding fails with a UnicodeDecodeError, the file likely uses Latin-1, which maps every byte to a character and never fails. This approach is simple and works well for text files with these two common encodings.

Common mistakes

Opening the file in text mode instead of binary mode, which applies the system's default encoding.
Assuming any non-UTF-8 file is Latin-1, when it could be another encoding like cp1252 or Shift-JIS.
Not handling empty files, which decode successfully as both encodings.

Variations

Use `chardet` or `charset-normalizer` for automatic detection across many encodings.
Check for a BOM (byte order mark) to identify UTF-8 with BOM or UTF-16.

Real-world use cases

Determining encoding before parsing CSV or log files from legacy systems.
Choosing the right decoder when ingesting user-uploaded files in a web app.
Migrating old data files to a consistent UTF-8 standard in a data pipeline.

How to Detect File Encoding: UTF-8 vs Latin-1 in Python

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Files & data

Tutorials

Quizzes

Python code

Output

How it works

Common mistakes

Variations

Real-world use cases

More from Files & data

Keep learning

Tutorials

Quizzes