Configure Logging to Avoid Data Leaks

Your logs are a goldmine — not just for debugging, but for attackers. Every innocent-looking logger.info(f"User {user.email} logged in") line can silently leak personally identifiable information (PII), tokens, or secrets into your log aggregation system, where they sit in plaintext for anyone with read access. In this lesson, you'll learn how to configure logging to avoid data leaks — a critical skill for secure development. We'll cover the mental model, hands-on configuration, tooling choices, and real-world troubleshooting so you can ship logs that help you debug without exposing your users.

The problem this lesson solves

Unstructured logging is the default in most codebases, and it's a security time bomb. Here's what happens when you don't think about logging configuration:

• PII exposure: Names, emails, phone numbers, and addresses end up in log files that may be backed up, replicated, or shared with third-party support tools.
• Token leaks: OAuth tokens, API keys, and session IDs appear in URLs, headers, or stack traces when exceptions are logged with full context.
• Compliance violations: GDPR, HIPAA, PCI-DSS, and SOC 2 all require that you limit exposure of sensitive data — even in logs. Fines can reach millions.
• Debugging becomes dangerous: The more you log to diagnose issues, the more you risk leaking. But logging everything is often how you find the bug.

A real-world example: a developer logs f"Request: {request.body}" to debug an API issue. The body contains a credit card number. Within a week, the logs are indexed by an analytics tool, and a support rep searches for a customer and finds the card number in plaintext. That's a reportable breach.

To avoid these leaks, you need to configure logging deliberately — with levels, formatting, filters, and redaction — not just sprinkle print() calls.

Core concept / mental model

Think of logging as a security boundary. Your application has internal data (variables, tokens, PII) and external data (user input). Logging is a channel that moves data from your app to the outside world (log files, metrics, dashboards). The goal is to control what crosses that boundary.

A useful analogy: logging is like a window into your server. You want to let in enough light (diagnostic information) but never leave the window open wide enough for someone to climb in and grab the crown jewels (secrets).

The key components of logging configuration are:

• Levels (DEBUG, INFO, WARNING, ERROR, CRITICAL): control verbosity and what gets captured.
• Formatters: define the structure of each log line.
• Handlers: decide where logs go (file, console, network).
• Filters: intercept and modify or discard log records before they reach the handler.
• Context: the data associated with a log record (e.g., request ID, user ID).

Redaction is the process of removing or masking sensitive values before they are written. It's your last line of defense — even if a developer accidentally logs a token, the filter should strip it.

Default Python logging is insecure because it logs whatever string you pass. There is no automatic redaction. So you must layer protections.

How it works step by step

Here's the mental flow from raw log call to safe output:

1. Decide what should never be logged — Create a denylist of field names (password, secret, token, authorization, credit_card, ssn).
2. Configure a custom formatter — Use a structured format (JSON or key=value) so redaction filters can target fields.
3. Add a global filter — This filter inspects every log record and replaces sensitive values with [REDACTED].
4. Set appropriate levels — In production, suppress DEBUG to avoid verbose data dumps.
5. Use logging best practices in code — Never log request bodies or full exception tracebacks with sensitive locals.
6. Test your configuration — Write unit tests that verify no sensitive data appears in the output.

Cause and effect: if you skip the filter, a single careless logger.info(request.headers) will leak the Authorization header. With the filter in place, the same call becomes Authorization: [REDACTED].

Hands-on walkthrough

Let's build a small Python application that uses the standard logging module with a redaction filter.

Step 1: Basic logging setup

Start with a simple configuration that uses JSON formatting — this makes redaction easier later.

import logging
import json
import sys

class JsonFormatter(logging.Formatter):
    def format(self, record):
        log_entry = {
            "timestamp": self.formatTime(record),
            "level": record.levelname,
            "message": record.getMessage(),
            "module": record.module,
        }
        if hasattr(record, 'user_id'):
            log_entry["user_id"] = record.user_id
        return json.dumps(log_entry)

handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JsonFormatter())
logger = logging.getLogger("app")
logger.addHandler(handler)
logger.setLevel(logging.INFO)

logger.info("User login", extra={"user_id": 123})

Expected output:

{"timestamp": "2025-01-01 12:00:00,000", "level": "INFO", "message": "User login", "module": "__main__", "user_id": 123}

Step 2: Add a redaction filter

Now add a filter that masks sensitive fields at the point of formatting. We'll use a regex-based approach to catch common patterns.

import re
import logging

class RedactionFilter(logging.Filter):
    SENSITIVE_PATTERNS = [
        re.compile(r'password\s*=\s*[^\s&]+', re.IGNORECASE),
        re.compile(r'authorization\s*:\s*Bearer\s+[\w\.-]+', re.IGNORECASE),
        re.compile(r'credit_card\s*=\s*\d{12,16}', re.IGNORECASE),
    ]

    def filter(self, record):
        if isinstance(record.msg, str):
            msg = record.msg
            for pattern in self.SENSITIVE_PATTERNS:
                msg = pattern.sub(lambda m: m.group(0).split(':')[0] + ': [REDACTED]', msg)
            record.msg = msg
        return True

logger.addFilter(RedactionFilter())

logger.info("Password set for user", extra={"user_id": 123})

Now call:

logger.info("User with password=supersecret logged in")

Expected output:

{"timestamp": "...", "level": "INFO", "message": "User with password=[REDACTED] logged in", "module": "__main__"}

The filter runs on every log record. If a developer logs a raw token, the filter catches it if the pattern matches.

Step 3: Use a structured logging library (structlog)

For production, consider structlog — it gives you automatic context, JSON output, and easy redaction.

import structlog

log = structlog.get_logger()

def log_event(event, **kwargs):
    # kwargs may contain sensitive data; we'll redact known keys
    redacted = {k: ('[REDACTED]' if 'token' in k or 'password' in k else v) for k, v in kwargs.items()}
    log.info(event, **redacted)

log_event("user.login", user_id=123, access_token="abc123")

With a JSON renderer configured, output will be structured and safe.

Step 4: Secure error logging

Never log exception objects directly without scrubbing. Always use exc_info=True carefully and ensure local variables aren't leaked.

try:
    risky_operation()
except Exception as e:
    # format a safe message - do not log the full exception as it may contain args
    logger.error("Operation failed", exc_info=False)
    # If you need traceback, ensure it is sanitized separately

Compare options / when to choose what

Pro tip: No matter which approach you choose, configure log rotation (e.g., RotatingFileHandler) and restrict file permissions so logs aren't world-readable.

Troubleshooting & edge cases

• Regex redaction misses multiline values — Use re.DOTALL if needed, or use key-based redaction instead.
• Logging library filters don't apply to logger.exception — Ensure your filter is set on the root logger, not just a single handler, because exception uses the root logger's handlers.
• Sensitive data in exception args — Even with exc_info=False, the message might contain user input. Always validate that you aren't interpolating untrusted values into log strings.
• Pre-production logs in dev — Developers often set DEBUG in development and forget to change it in production. Use environment variables to control levels.
• Metrics and third-party tools scrape logs — If you send logs to a SaaS, ensure the transport is encrypted (TLS) and that the log retention policy is as short as possible.
• Multiprocessing — When using logging.handlers.QueueHandler and a listener, ensure the filter is applied in both processes.

What you learned & what's next

You now understand how to configure logging to avoid data leaks: from levels and formatters, to redaction filters and structured logging. You applied the core idea in a hands-on exercise and know how to choose between options. This protects your users and your compliance posture.

Next lesson in the Secure development track continues with [Next Secure Development Topic] — where you'll apply similar boundary thinking to other areas like input validation or secret management.