Collect metrics with node_exporter

You’ve deployed a service, and now you need to know if it’s healthy under load. SSH-ing into every box to run top or free doesn’t scale, and by the time you notice a problem, it’s already affected users. The Linux kernel exposes a wealth of performance and health data, but there’s no built-in way to collect it centrally. Collecting metrics with node_exporter gives you a standardized, lightweight way to export system metrics over HTTP, ready for Prometheus to scrape and alert on — without writing custom scripts or instrumenting your application.

The problem this lesson solves

When you manage more than a handful of Linux servers, you lose visibility. Manual checks are error-prone, slow, and don’t capture historical trends. You need a way to answer questions like:

• Is CPU usage spiking on a specific host?
• Are we running out of disk space?
• Is memory pressure causing swap storms?
• Has the network dropped packets?

Built-in tools like top, vmstat, or /proc files give you raw data, but they are not designed for central collection or alerting. Writing custom exporters for every metric is unsustainable. node_exporter solves this by exposing dozens of system metrics on a single HTTP endpoint, using the industry-standard Prometheus text format — which can be consumed by Prometheus, Grafana, and other tools. It’s the foundation of infrastructure telemetry.

By the end of this lesson, you’ll be able to set up node_exporter on a Linux host, verify it’s running, and query the metrics it exposes — and you’ll know how it fits into a broader monitoring pipeline.

Core concept / mental model

Think of node_exporter as a translator between the Linux kernel and Prometheus. The kernel keeps detailed, low-level statistics in /proc and /sys — things like process CPU usage, memory allocation, disk I/O, and network packet counters. But these files are not in a format Prometheus understands, and they’re scattered across dozens of pseudo-files.

node_exporter acts as a metric gateway: it reads those kernel interfaces, organizes the data into meaningful metrics (like node_cpu_seconds_total), and exposes them on an HTTP endpoint (default port 9100) in the Prometheus text exposition format — which is just plain text with a well-defined structure:

# HELP node_cpu_seconds_total Seconds the user/system CPU spent in each mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="idle"} 123456.78

Here’s an analogy: the kernel is like a factory floor full of sensors; node_exporter is the display panel that aggregates all those sensors into a single dashboard; and Prometheus is the observer that periodically takes a snapshot of that dashboard and stores it in a time-series database. Each snapshot is called a scrape.

Important terms: - Exporter: a program that exposes metrics in Prometheus format (node_exporter is one). - Scrape: Prometheus’s HTTP GET request to collector an exporter’s metrics. - Metric family: a set of metrics with the same name but different labels (e.g., CPU metrics per core and mode). - Collector: a subsystem inside node_exporter that gathers a specific set of metrics (e.g., cpu, meminfo, filesystem).

How it works step by step

To collect metrics with node_exporter, you follow this logical sequence:

1. Install the node_exporter binary on each Linux host you want to monitor.
2. Run it as a service (ideally under systemd) so it starts on boot and restarts on failure.
3. Verify the HTTP endpoint is serving metrics on port 9100.
4. Configure Prometheus to discover and scrape those endpoints.
5. Query and visualize the metrics to gain insight.

Cause → effect: Each step builds on the previous one. If you skip the service setup, node_exporter won’t restart after a reboot, and you’ll lose telemetry. If you don’t configure Prometheus to scrape, the data sits there unused. Getting the foundation right is critical.

Let’s explore each step in detail.

Step 1: Download and install node_exporter

node_exporter is a single static binary — no dependencies, no runtime, no package manager required (though you can use package repositories). Download the latest release from the official GitHub releases page. Example for version 1.8.2 (check for the latest):

# Download and extract
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
tar xzf node_exporter-1.8.2.linux-amd64.tar.gz

# Move the binary to a standard location
sudo cp node_exporter-1.8.2.linux-amd64/node_exporter /usr/local/bin/

# Verify it runs
node_exporter --version

Expected output (excerpt):

node_exporter, version 1.8.2 (branch: HEAD, revision: ...)

Pro tip: Always verify the checksum (SHA256) of the downloaded archive to avoid supply-chain attacks. The release page lists checksums for every artifact.

Step 2: Run node_exporter as a systemd service

Running it in the foreground is fine for testing, but for production you want a resilient service. Create a systemd unit file:

sudo tee /etc/systemd/system/node_exporter.service <<EOF
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=nobody
Group=nogroup
Type=simple
ExecStart=/usr/local/bin/node_exporter
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter

Check status:

sudo systemctl status node_exporter

Expected output (excerpt):

● node_exporter.service - Node Exporter
   Loaded: loaded (/etc/systemd/system/node_exporter.service; enabled)
   Active: active (running) since ...

Security note: Running as nobody limits privileges, but node_exporter still needs read access to /proc and /sys, which is generally allowed. For extra hardening, you can use a dedicated user with minimal permissions.

Step 3: Verify the metrics endpoint

Now test the HTTP endpoint on port 9100:

curl http://localhost:9100/metrics | head -n 20

Expected output (excerpt):

# HELP node_boot_time_seconds Node boot time, in seconds since epoch.
# TYPE node_boot_time_seconds gauge
node_boot_time_seconds 1.7152e+09
# HELP node_cpu_seconds_total Seconds the cpus spent in each mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="idle"} 5.4372e+06
node_cpu_seconds_total{cpu="0",mode="iowait"} 1.234e+05

If you see metric lines like this, you’re successfully collecting metrics with node_exporter.

Step 4: Configure Prometheus to scrape

Add a job to your Prometheus configuration (prometheus.yml):

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

Replace localhost with the actual IP if Prometheus is on another machine. Reload Prometheus config with kill -HUP $(pidof prometheus) or via the API, and verify the target is up in the Prometheus UI (Status → Targets).

Step 5: Query and visualize

Once Prometheus is scraping, you can run PromQL queries like:

100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

That gives you CPU usage percentage per host. Graph it in Grafana with a dashboard like node_exporter-full — but that’s beyond this lesson.

Hands-on walkthrough

Let’s put it all together in a working setup on an Ubuntu server (tested on 22.04). This section assumes you have sudo privileges and internet access.

Exercise: Stand up node_exporter in 5 minutes

Goal: Have node_exporter running as a systemd service and verify metric collection from the command line.

1. Download and install node_exporter using the commands from the previous section.
2. Create the systemd unit file exactly as shown.
3. Start and enable the service.
4. Verify the endpoint with curl.
5. Check that specific metrics are being collected — for example, wrap the curl output to grep for memory metrics:

curl -s http://localhost:9100/metrics | grep -E "^node_memory_Mem(Total|Free|Available)_bytes"

Expected output:

node_memory_MemAvailable_bytes 1.27e+09
node_memory_MemFree_bytes 1.8e+08
node_memory_MemTotal_bytes 2.09e+09

1. (Optional) If Prometheus is installed, add the job and see the targets appear.

Sample: Querying custom labels

node_exporter exposes a lot of metrics with labels. Try listing all filesystem metrics:

curl -s http://localhost:9100/metrics | grep "^node_filesystem_avail_bytes"

You’ll see multiple lines, one per mounted filesystem, labeled with device, fstype, and mountpoint.

Expected pitfalls in the exercise

• Port already in use: If something else uses 9100, node_exporter will fail to bind. Change the port with --web.listen-address=":9101".
• Permission denied: If you run node_exporter as a user without access to /proc or /sys, metrics may be missing or the process may error. Check logs: journalctl -u node_exporter.
• Firewall blocking: On cloud instances, ensure security groups allow inbound traffic on 9100 from Prometheus’s IP.

Compare options / when to choose what

node_exporter is the de facto standard, but it’s not the only way to collect system metrics. Here’s a comparison.

Recommendation: For 95% of Linux server monitoring with Prometheus, use node_exporter. It’s minimalistic, reliable, and you’ll find plenty of online support. If you need application-level custom metrics, write a small exporter using the Prometheus client library instead of modifying node_exporter.

Troubleshooting & edge cases

No exporter setup goes smoothly every time. Here are the issues you’re most likely to hit:

Service fails to start

• Symptom: systemctl status says failed, and logs show bind: address already in use.
• Fix: Find the process using port 9100 (sudo ss -tlnp | grep 9100) and either stop it or change node_exporter’s port.

Metrics missing for some subsystems

• Symptom: You see node_cpu_seconds_total but no node_disk_* metrics.
• Cause: The diskstats collector may be disabled by default in some kernels, or the --disabled-collectors flag was set.
• Fix: List all collectors with node_exporter --collector.disable-defaults --help (not fully accurate) or check /proc/diskstats. Enable specific collectors with --collector.diskstats if necessary (they’re usually on by default).

Textfile collector isn’t picking up custom metrics

• Symptom: You created a .prom file in the specified directory, but it’s not showing up in the output.
• Fix: Ensure the file extension is .prom, the path passed via --collector.textfile.directory is correct, and node_exporter has read permission for the nobody user. Example: sudo chown node_exporter:node_exporter /var/lib/node_exporter/textfile and set --collector.textfile.directory=/var/lib/node_exporter/textfile.

High cardinality causing memory blowup

• Symptom: Prometheus runs out of memory with many labels.
• Cause: node_exporter’s default collectors expose many time series per host (e.g., per-CPU). With hundreds of hosts, it adds up.
• Mitigation: Partition your Prometheus server or use recording rules to aggregate. You can also disable unused collectors with --no-collector.<name> to reduce load.

Scraping fails with 404

• Symptom: Prometheus target is down, and curl returns 404.
• Fix: Ensure node_exporter is serving on the correct path — default is /metrics. If you changed the listener with --web.telemetry-path, update the scrape config accordingly.

What you learned & what's next

In this lesson, you learned the core concept of collecting metrics with node_exporter — a lightweight exporter that translates kernel statistics into Prometheus-compatible metrics. You completed a hands-on exercise: installing, running as a systemd service, verifying the endpoint, and even configuring Prometheus to scrape it. You now know why node_exporter is the standard choice for system-level monitoring and how it compares with alternatives like Telegraf or custom exporters.

You also picked up troubleshooting skills for common pitfalls — port conflicts, missing metrics, textfile collector issues, and cardinality concerns.

Next lesson: Now that you have raw system metrics flowing into Prometheus, the next step in your telemetry journey is querying metrics with PromQL — learning how to filter, aggregate, and compute rates to turn raw numbers into actionable insights. That’s where the power of telemetry truly shines.

Keep your node_exporter running — you’ll need it for the PromQL exercises!