Collect and Query Logs with Loki

Logging is the backbone of debugging and observability in Kubernetes, but traditional tools like the ELK stack can be expensive and complex to manage. When you're running hundreds of pods, collecting every log line becomes a cost and performance nightmare. Loki offers a smarter way: it only indexes metadata (labels) and stores logs in compressed chunks, making log storage affordable at scale. In this lesson, you will learn to collect and query logs with Loki — a Prometheus-inspired, horizontally scalable, multi-tenant log aggregation system designed for cost-effective observability in Kubernetes.

The problem this lesson solves

Kubernetes pods can restart, scale up, or be evicted at any moment. Without a centralized log system, you'd have to chase logs across multiple nodes using kubectl logs — which is slow, non-persistent, and impossible for historical analysis. Traditional log solutions (like Elasticsearch) index every log line, leading to massive storage costs and complex operations. Loki solves this by storing logs in a compressed format and indexing only the labels — dramatically reducing storage and query costs while integrating seamlessly with Prometheus and Grafana.

Pain points addressed

• Cost: Storing terabytes of logs in Elasticsearch is expensive. Loki's compression and minimal indexing cut costs by up to 80%.
• Speed: Queries run against compressed chunks, but metadata filtering makes them fast — not as fast as full-text search, but good enough for most debugging.
• Simplicity: Loki uses Prometheus-style labels (service, namespace, pod) to organize logs, so you can reuse existing alerting and dashboards.

Core concept / mental model

Think of Loki as Prometheus for logs. Instead of storing every log line as a searchable document (like Elasticsearch), Loki stores chunks of log data on object storage (S3, GCS, MinIO) and indexes only the labels. When you query, Loki pulls only the chunks that match the label filter, then scans the raw log lines inside those chunks.

Key terminology

• Chunk: A compressed block of log data (usually 512 KB). Multiple chunks make up a log stream.
• Stream: A unique combination of labels (e.g., {app="nginx", namespace="default"}). Each stream gets its own set of chunks.
• Ingester: Receives logs and builds chunks in memory, then flushes them to storage.
• Querier: Reads chunks from storage and runs LogQL queries.
• Distributor: Receives logs from Promtail or other agents and forwards them to the correct ingester.

Pro Tip: Because Loki doesn't index the log content, it's perfect for high-volume logs (like Kubernetes audit logs). Avoid using it for full-text search over all logs — that's Elasticsearch's domain.

How it works step by step

1. Log collection: Deploy Promtail as a DaemonSet on each Kubernetes node. Promtail reads log files (usually /var/log/pods/*.log), attaches labels (namespace, pod name, container name), and sends them to Loki's distributor.
2. Ingestion: The distributor hashes the labels to determine which ingester should own the stream. The ingester appends logs to an in-memory chunk and periodically flushes it to object storage (e.g., 1 MB or 1 hour max).
3. Storage: Chunks are stored in a compressed format (often with gzip or snappy) in object storage. A boltdb-shipper index keeps track of which labels belong to which chunks.
4. Querying: When you run a LogQL query, the querier first looks up the index to find matching chunks, then decompresses and scans the actual log lines. Results are returned to Grafana or the command line.

LogQL in a nutshell

LogQL is like PromQL but for logs. You can filter, aggregate, and extract patterns.

# Select all logs from app=nginx namespace=default
{app="nginx", namespace="default"}

# Filter lines containing 'error'
{app="nginx"} |= "error"

# Count log lines per minute by status code
rate({app="nginx"}[5m])

Hands-on walkthrough

We'll deploy Loki together with Promtail and Grafana on a Kubernetes cluster (minikube or any cluster). This is a production-ready setup using the official Helm chart.

Step 1: Add Helm repos

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Step 2: Deploy Loki (single binary mode for simplicity)

We'll use a minimal configuration for this walkthrough. Create a file named loki-values.yaml:

# loki-values.yaml
auth_enabled: false

ingester:
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  max_chunk_age: 1h

storage_config:
  boltdb_shipper:
    active_index_directory: /tmp/loki/boltdb-shipper-active
    cache_location: /tmp/loki/boltdb-shipper-cache
    cache_ttl: 24h
    shared_store: filesystem
  filesystem:
    directory: /tmp/loki/chunks

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

Deploy Loki:

helm upgrade --install loki grafana/loki -f loki-values.yaml --namespace=loki --create-namespace

Step 3: Deploy Promtail

Create promtail-values.yaml:

# promtail-values.yaml
config:
  clients:
    - url: http://loki:3100/loki/api/v1/push

helm upgrade --install promtail grafana/promtail -f promtail-values.yaml --namespace=loki

Step 4: View logs in Grafana with a Data Source

Assuming you have Grafana deployed, add a Loki data source pointing to http://loki:3100. Then explore logs:

# Show all logs from the default namespace
{namespace="default"}

# Count log lines per second
rate({namespace="default"}[5m])

Step 5: Query from command line (optional)

Use logcli, the CLI tool for Loki:

# Install logcli (on host)
brew install logcli   # macOS
# Or download from releases

# Query logs
logcli query '{namespace="default"}' --addr http://localhost:3100 --output raw

Expected output: You should see raw log lines from pods in the default namespace.

Compare options / when to choose what

When to use Loki: You need to collect massive amounts of logs from Kubernetes, cloud services, or microservices — where storage costs matter more than instant full-text search. Perfect for incident response, compliance archives, and trend analysis.

Troubleshooting & edge cases

Common mistakes

• Missing logs: Promtail cannot access /var/log/pods/*.log if the DaemonSet runs as non-root. Ensure securityContext: privileged: true or mount the host log directory correctly.
• High memory usage: If ingester memory spikes, reduce chunk_idle_period or increase max_chunk_age. The default of 1h is usually fine.
• No data in Grafana: Double-check the Loki URL in your data source. For example, if Loki runs in the loki namespace, use http://loki.loki.svc.cluster.local:3100.
• Rate limiting: Loki's default ingester limits (e.g., ingester_client.max_recv_message_size: 67108864) may drop large log lines (~64 MB). Adjust via Helm values.

Edge case: High-cardinality labels (e.g., pod IP)

If you add high-cardinality labels (like pod_ip or request_id), Loki creates too many streams, causing memory and performance issues. Never use high-cardinality labels as stream labels. Instead, use structured metadata (available in Loki 2.6+) or parse log content.

Pro Tip: Keep your label cardinality under 10k distinct label values. For example, use namespace, app, component — not pod, container_id, or request_id.

What you learned & what's next

You now understand the core concept behind collecting and querying logs with Loki: minimal indexing via labels, chunk-based storage, and LogQL for queries. You have deployed Loki with Promtail on Kubernetes and queried logs from Grafana. This pattern replaces expensive log systems with a cost-effective, scalable solution.

Next steps: In the next lesson, you will learn to configure alerting rules based on log patterns using Loki's Ruler component, or dive deeper into LogQL aggregations to monitor error rates and latency from logs.

Key takeaways

• Loki stores logs in compressed chunks and indexes only labels, drastically reducing costs.
• Promtail (or any Grafana agent) collects logs from nodes and sends them to Loki.
• LogQL is like PromQL but for logs — use |= "error" to filter, rate() to count.
• Avoid high-cardinality labels to keep performance stable.
• Loki integrates natively with Grafana for visualization and alerting.

Practice recap

Deploy Loki with Promtail on a minikube cluster using the Helm commands above. Then, create a test pod that generates logs (e.g., kubectl run test --image=nginx). Query all logs from that pod using LogQL: {pod="test"}. Then, parse the output to extract HTTP status codes using pattern parser.