How to Ship Logs to an Aggregator Endpoint in Python
Ship batched log entries to a mock HTTP aggregator endpoint with proper error handling and response status.
pip install requests
Python code
33 linesimport json
import requests
from datetime import datetime, timezone
LOG_ENTRIES = [
{"timestamp": "2024-01-15T10:00:00Z", "level": "INFO", "message": "Server started"},
{"timestamp": "2024-01-15T10:00:05Z", "level": "WARN", "message": "High memory usage"},
{"timestamp": "2024-01-15T10:00:10Z", "level": "ERROR", "message": "Database connection lost"},
]
def ship_logs(entries: list[dict]) -> int:
"""Send log entries to an aggregator endpoint and return HTTP status."""
try:
response = requests.post(
"https://httpbin.org/post",
json={"source": "web-app", "logs": entries},
timeout=5
)
response.raise_for_status()
return response.status_code
except requests.RequestException as exc:
print(f"Shipping failed: {exc}")
return -1
if __name__ == "__main__":
# Simulate shipping at a given timestamp
batch = {
"received_at": datetime.now(timezone.utc).isoformat(),
"logs": LOG_ENTRIES
}
status = ship_logs([batch])
print(f"HTTP {status} - Shipped {len(LOG_ENTRIES)} log entries")
Output
HTTP 200 - Shipped 3 log entries
How it works
The ship_logs function serializes log entries into JSON and sends them via a POST request using the requests library. raise_for_status() ensures non-2xx responses raise an exception, which is caught by RequestException to gracefully handle failures. The batch wrapper adds a UTC timestamp for traceability. This pattern decouples log generation from transport, making it easy to retry or redirect logs in production.
Common mistakes
- Forgetting to set a timeout, which can hang the app indefinitely
- Not catching `requests.RequestException` for network failures
- Sending logs one-by-one instead of batching for efficiency
- Using local timestamps without timezone info
Variations
- Use `asyncio.to_thread` to ship logs asynchronously without blocking the main thread
- Add retries with exponential backoff using `requests` + `time.sleep`
Real-world use cases
- Forwarding application logs from a web server to a central logging service like ELK or Datadog.
- Batching transaction logs from a payment gateway to an audit trail API.
- Sending structured logs from microservices to a Kafka-based log aggregator via HTTP bridge.
Sponsored
More from Observability & SRE
- Adding a Correlation ID to Log Context in Python medium
- Calculate Error Rate from Log Stream in Python easy
- Check if a Timestamp Falls in a Daily Maintenance Window in Python easy
- Export Metrics with OTLP Mock in Python medium
- Generate Mock CPU and Memory Metrics in Python easy
- Generate Prometheus Text Exposition Format in Python easy
Keep learning
Related tutorials and quizzes for this topic.