Read System Logs with journalctl

You've deployed a service, traffic is flowing, and then — something breaks. The logs are scattered across cryptic files, and you're left guessing which one holds the answer. That's the pain this lesson solves: journalctl gives you one unified, queryable view of every system log on a modern Linux box, turning a frantic hunt into a precise, seconds-long search.

The problem this lesson solves

Traditional syslog setups dump logs into /var/log/ — syslog, auth.log, kern.log. Each service writes to its own file, with its own format, rotation policy, and timezone. When an incident hits, you have to grep across multiple files, correlate timestamps manually, and hope you're looking in the right place. It's slow, error-prone, and a nightmare to teach to newcomers.

The real pain: you can't afford to spend minutes locating a log when your service is down. And even when you find the file, filtering by time range or service requires complex awk pipelines. The industry moved to structured, centralized logging precisely because this old model doesn't scale.

Core concept / mental model

Think of journald as the kernel's memory and disk-based log broker. It's a systemd component that captures log messages from the kernel, systemd services, and any process that writes to standard output or standard error. journalctl is the read-only client that queries this journal.

Analogy: If traditional logs are a pile of papers scattered on desks (files), the journal is a single, indexed ledger. journalctl is your assistant who can instantly flip to any page, filter by date, author (service), or keyword, and even show you the original handwriting (raw output) when needed.

Key definitions

• Journal: The binary database where journald stores log entries, typically at /var/log/journal/.
• Entry: Each log line with structured metadata — timestamp, priority, unit, message, and more.
• Unit: A systemd service, socket, or timer that generates logs.
• Priority: Syslog severity levels from 0 (emerg) to 7 (debug).

How it works step by step

1. Log capture: When a service (like nginx.service) writes to stdout/stderr, systemd's StandardOutput=journal (default) captures and sends it to journald.
2. Structured storage: journald stores each entry as a binary record with fields — for example, _PID, _SYSTEMD_UNIT, MESSAGE, PRIORITY. The MESSAGE field holds the human-readable text.
3. Persistence: By default, logs live in memory. If you create /var/log/journal, the journal becomes persistent — surviving reboots.
4. Query via journalctl: The tool reads these binary records and renders them as text, supporting filters, formatting, and live tailing.

The flow in practice

• Service writes log → systemd captures → journald stores → you run journalctl -u my-service → you see filtered, timestamped output.

Hands-on walkthrough

This is where you turn theory into mastery. Run these commands on any systemd-based Linux (Ubuntu, Debian, Fedora, Arch). You'll need sudo for some queries.

Example 1: View recent logs

Open a terminal and run:

# Show all logs from the current boot
journalctl -b

# Show only the last 50 lines
journalctl -n 50

# Follow new logs in real time (like tail -f)
sudo journalctl -f

Expected output: a stream of lines with format Mmm DD HH:MM:SS hostname systemd[1]: message, and -f will keep printing as new logs arrive.

Example 2: Filter by service unit

Let's inspect logs from a hypothetical nginx.service:

# Logs for the unit, regardless of boot
journalctl -u nginx.service

# With specific boot and prioritize errors
journalctl -b -u nginx.service -p err

Expected output: only lines from that unit, and with -p err only errors. Notice how clean it is — no cross-service noise.

Example 3: Filter by time range and keyword

# Since yesterday, errors only
journalctl --since "yesterday" -p err

# Since a specific time, search for 'timeout'
journalctl --since "2025-01-15 09:00:00" --until "2025-01-15 10:00:00" | grep -i timeout

# Full verbose output with all metadata fields
sudo journalctl -u nginx.service -o verbose | head -20

Expected output: entries with human-readable dates, and -o verbose shows fields like _PID, _UID, _EXE, and the MESSAGE.

Example 4: Persistent journal and disk usage

# Enable persistent logging (creates /var/log/journal)
sudo mkdir -p /var/log/journal
# Restart journald to apply
sudo systemctl restart systemd-journald

# Check journal disk usage
journalctl --disk-usage

# Rotate logs older than 7 days
sudo journalctl --vacuum-time=7d

Expected output: --disk-usage shows like 9.6M, and vacuum deletes older entries, freeing space.

Pro tip: -u can take a glob, like journalctl -u nginx* to match multiple units. Combine with --since today for quick daily reviews.

Compare options / when to choose what

Both journalctl and traditional syslog have merits. Here's a quick comparison:

When to choose what: Use journalctl when you want quick, contextual queries on the local system — it's the default for operational debugging. Use syslog forwarding when you need a central log aggregator (e.g., ELK stack) — journald can forward messages to syslog or another log manager via ForwardToSyslog=yes in /etc/systemd/journald.conf.

Variations

• journalctl -p: Priority filter — -p warning shows warnings and above (0-4).
• journalctl -k and -f: Kernel logs only, or continuous follow.
• journalctl -o json-pretty: Output as JSON for scripting or integration.

Troubleshooting & edge cases

Error: No journal files were found. — This happens when the journal isn't persistent and the system was rebooted. Fix: create /var/log/journal and restart systemd-journald to start persisting.

Wrong output: Logs are missing from a service. — The service might be set to StandardOutput=null or logging to a file directly. Check the unit file and journal settings. If it logs to a file, journalctl won't capture it — you may need StandardOutput=append:/var/log/service.log (not ideal) or forward to syslog.

Edge case: Large disk usage. — If /var/log/journal grows unbounded, use journalctl --vacuum-size=500M to trim to a maximum size.

Wrong timezone or timestamps? — journalctl uses the system timezone by default. Use --utc for UTC, or set TZ environment variable.

Pro tip: If you see Failed to get journal: Operation not permitted, you likely need sudo — many journal fields are restricted to root.

What you learned & what's next

You've now mastered the core of reading system logs with journalctl. You can explain the mental model (a binary journal with structured fields), and you've completed hands-on exercises covering filtering by unit, time, priority, and persistence. You've also compared journalctl to traditional syslog and know how to troubleshoot common pitfalls.

Next in this track, you'll look at systemd service analysis — understanding how units start, fail, and restart. That ties directly into log reading: you'll use journalctl -u to debug service failures and correlate logs with systemd states.

Now, put this into your own practice: when a service misbehaves, your first instinct should be journalctl -b -u <unit> -p err — you'll find the answer in seconds.