How to Ship Logs to an Aggregator Endpoint in Python

Ship batched log entries to a mock HTTP aggregator endpoint with proper error handling and response status.

Easy Python 3.9+ Aug 9, 2026 Observability & SRE 13 views 0 copies

Requires third-party packages — install first
pip install requests

Python code

33 lines
Python 3.9+
import json
import requests
from datetime import datetime, timezone

LOG_ENTRIES = [
    {"timestamp": "2024-01-15T10:00:00Z", "level": "INFO", "message": "Server started"},
    {"timestamp": "2024-01-15T10:00:05Z", "level": "WARN", "message": "High memory usage"},
    {"timestamp": "2024-01-15T10:00:10Z", "level": "ERROR", "message": "Database connection lost"},
]

def ship_logs(entries: list[dict]) -> int:
    """Send log entries to an aggregator endpoint and return HTTP status."""
    try:
        response = requests.post(
            "https://httpbin.org/post",
            json={"source": "web-app", "logs": entries},
            timeout=5
        )
        response.raise_for_status()
        return response.status_code
    except requests.RequestException as exc:
        print(f"Shipping failed: {exc}")
        return -1

if __name__ == "__main__":
    # Simulate shipping at a given timestamp
    batch = {
        "received_at": datetime.now(timezone.utc).isoformat(),
        "logs": LOG_ENTRIES
    }
    
    status = ship_logs([batch])
    print(f"HTTP {status} - Shipped {len(LOG_ENTRIES)} log entries")

Output

stdout
HTTP 200 - Shipped 3 log entries

How it works

The ship_logs function serializes log entries into JSON and sends them via a POST request using the requests library. raise_for_status() ensures non-2xx responses raise an exception, which is caught by RequestException to gracefully handle failures. The batch wrapper adds a UTC timestamp for traceability. This pattern decouples log generation from transport, making it easy to retry or redirect logs in production.

Common mistakes

  • Forgetting to set a timeout, which can hang the app indefinitely
  • Not catching `requests.RequestException` for network failures
  • Sending logs one-by-one instead of batching for efficiency
  • Using local timestamps without timezone info

Variations

  1. Use `asyncio.to_thread` to ship logs asynchronously without blocking the main thread
  2. Add retries with exponential backoff using `requests` + `time.sleep`

Real-world use cases

  • Forwarding application logs from a web server to a central logging service like ELK or Datadog.
  • Batching transaction logs from a payment gateway to an audit trail API.
  • Sending structured logs from microservices to a Kafka-based log aggregator via HTTP bridge.

Sponsored

Run locally

This sample needs third-party packages, so it cannot run in the browser IDE. Copy the code above, install the packages shown at the top, then run it in your own Python environment.

More from Observability & SRE

Related tutorials and quizzes for this topic.