Monitor Disk I/O with iostat

Your database is slow, your API latency is spiking, and top shows the CPU quiet as a mouse. You suspect the disk, but you don't know how to prove it. iostat is the Linux tool that turns vague suspicion into hard data, showing you exactly how your storage is performing, one I/O at a time.

The Problem: Why You Can’t Ignore Disk I/O

When a server slows down, most developers immediately blame the CPU or memory. But in modern systems, disk I/O is often the silent bottleneck. A misconfigured database, a log file that's grown out of control, or a failing SSD can cause terrible performance without ever touching your CPU usage. Without the ability to monitor disk I/O with iostat, you're flying blind. You might be adding more memory or scaling out your application when the real issue is that your disks are saturated with read and write requests. This lesson gives you the exact command-line skills to identify, measure, and resolve disk-related performance issues before they turn into outages.

Core Concept / Mental Model: The Disk as a Factory

Think of your storage device as a factory with a loading dock. The factory is the hard disk or SSD. The loading dock is the disk controller and the I/O queue. Your applications are trucks constantly arriving to drop off (write) or pick up (read) goods — data blocks. The speed of the factory is limited by two things: how many trucks can be processed per second (IOPS), and how quickly each truck can be loaded or unloaded (data throughput).

iostat is your factory supervisor. It reports two key numbers:

• tps (transfers per second): The total number of read and write I/O requests that are being sent to the device per second. This is your throughput in terms of operations.
• KB_read/s, KB_wrtn/s: The amount of data read and written per second, in kilobytes. This is your data throughput.

The factory also has a queue. If too many trucks arrive at once, they line up on the street. iostat tells you about this with await (average time in the queue for a request) and %util (percentage of time the device was busy).

Core Definition: iostat is a command-line utility that reports CPU statistics and input/output statistics for devices and partitions. It is part of the sysstat package.

Key Terms You'll See

• TPS: Transfers per second (tiny operations).
• KB/s: Kilobytes of data transferred per second.
• await: Average time (in milliseconds) an I/O request takes from the moment it enters the queue until it's completed.
• svctm: Average service time (how long the device itself takes, ignoring queue time).
• %util: The percentage of CPU time during which the device was busy processing I/O. High %util doesn't always mean saturation, but over 90% is a red flag.

How It Works Step by Step

iostat pulls data from the kernel's proc filesystem, specifically /proc/diskstats. Here's the logical flow:

1. The kernel tracks I/O. Every time a block device is read from or written to, the kernel updates counters — the number of reads/writes, the number of milliseconds spent doing I/O, and the number of sectors transferred.
2. You run iostat. The command reads these counters.
3. It calculates the difference. To get a snapshot, iostat reads the counters, waits a specified interval, and then reads them again. The difference, divided by the time, gives you rates.
4. It displays formatted output. The results are shown as a table with columns for each metric.

The Command Structure

• iostat — Shows a single snapshot from system boot.
• iostat <interval> — Shows a report every <interval> seconds indefinitely.
• iostat <interval> <count> — Shows <count> reports, each separated by <interval> seconds.
• iostat -x — Shows extended statistics (await, %util, etc.).
• iostat -d — Shows only device utilization, not CPU.

Hands-On Walkthrough

Step 1: Install iostat

If you don't have sysstat installed, you'll need to add it. On Debian/Ubuntu:

sudo apt update
sudo apt install sysstat

On RHEL/CentOS/Fedora:

sudo yum install sysstat

Step 2: Look at the Basic Report

Run iostat to see a summary since boot:

iostat

Expected Output (simplified):

Linux 5.15.0-91-generic (hostname)  02/18/2025  _x86_64_    (4 CPU)

average-cpu:  %user   %nice    %sys %iowait   %idle
               2.5      0.1      1.0      2.0     94.4

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               20.5        50.0       100.0      50000     100000

This tells you the average CPU and disk activity. %iowait is the percentage of time the CPU was waiting for I/O to complete — high values here hint at disk bottlenecks.

Step 3: Monitor Continuously with Extended Stats

The real power of iostat comes from watching it live with extended metrics. Use -x for the extended report, and pass an interval of 2 seconds and 5 reports:

iostat -x 2 5

Key columns in extended mode:

• rrqm/s and wrqm/s: Merged read/write requests per second (queue merges are good).
• r/s and w/s: Actual read/write requests per second.
• rkB/s and wkB/s: Kilobytes read/written per second.
• await: Average service time + wait time per request (milliseconds).
• %util: Device busy percentage.

Step 4: Generate Some Load and Watch the Results

Let's create some I/O so you can see the metrics change. Run this in one terminal:

# Create a large file that writes continuously for 10 seconds
dd if=/dev/zero of=/tmp/test_io bs=1M count=2000 conv=fdatasync

While that's writing, run iostat in a second terminal:

iostat -x 1 3

Expected Output (excerpt):

Device:         rrqm/s   wrqm/s     r/s     w/s   rMB/s   wMB/s  avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00  1000.0     0.00   500.0    0.0   200.0    128.0     1.00    2.00    0.00    2.00  1.50   75.0

You'll see %util jump to a high value (often 100% on spinning disks) and wMB/s spike. This confirms that the dd command is hammering the disk.

Step 5: Watch Specific Devices

If you have multiple disks (like /dev/sda for OS and /dev/nvme0n1 for data), you can filter:

iostat -x /dev/sda 2

Step 6: Save and Review Sessions

For later analysis, log your output:

iostat -x 5 > /tmp/io_perf.log

Compare Options: When to Use What

iostat is powerful, but it's not the only tool. Here's a comparison to help you choose the right tool for the job:

Rule of thumb: Use iostat to confirm there's a disk problem, then use iotop to find who is causing it. For sustainable monitoring, set up sar to record data regularly.

Pro Tip: For micro-benchmarking, use fio — it gives precise read/write workloads and reports IOPS and latency with much finer control than dd.

Troubleshooting & Edge Cases

1. "Command 'iostat' not found"

• Problem: Not installed.
• Fix: Install the sysstat package as shown earlier. On some minimal containers, you may need apt-get install sysstat or apk add sysstat.

2. High %util but Low IOPS

The disk is busy but not transferring much data. This typically indicates many small random I/O operations (e.g., a database doing random reads). The disk is constantly seeking, so even a single request takes time. Check await — if it's high (over 20-30ms), the disk is struggling.

3. Low %util but High Latency

You see high await but low %util. This could mean the I/O queue is being delayed elsewhere — perhaps by a crowded same disk or by other processes. Check avgqu-sz (average queue size). If it's > 1, there are too many pending requests.

4. RAID and Virtualized Environments

On a RAID array, the individual disks may show unusual utilization. Your physical disk is one array; a high %util on one disk doesn't always mean the entire array is slow. In virtual machines, your host's disk might be the bottleneck — inside the VM, you'll see high await but low %util if the host is throttling I/O.

5. Zero Stats on Some Devices

Some virtual devices (like virtual disks in cloud VMs) may not expose full stats to iostat. If you see all zeros, verify the device name and try -p ALL to list all devices.

6. Interpreting %util on SSDs

SSDs can report 100% %util even with low queue depth because they handle commands in parallel. For SSDs, watch await — if it's under 1ms consistently, the disk is fine even at 100% util.

What You Learned & What's Next

You now know how to monitor disk I/O with iostat — from installing the tool to understanding the key metrics like tps, await, and %util. You've run live monitoring, generated load to see real-time changes, and learned how to troubleshoot common I/O issues. This is a foundational skill for any engineer managing servers.

You successfully explained the core idea behind iostat and completed a practical exercise — your two learning objectives.

Next up in the track: Now that you can spot disk pressure, learn to prevent it with filesystem benchmarking (using fio) and log rotation strategies. Moving forward, you'll apply these monitoring skills to databases and distributed systems, where disk I/O becomes even more critical for performance.

Stay tuned for the next lesson on fio — the tool for simulating I/O workloads and stress-testing your storage.