eBPF for Linux Observability: What It Is and How to Get Started

eBPF is quietly rewriting the rules of Linux observability. If you’ve been hearing about it but aren’t sure how it fits into your toolkit, here’s the short version: eBPF lets you run sandboxed programs inside the Linux kernel without changing kernel code or loading unsafe modules. That’s a superpower for debugging, performance monitoring, and security.

What eBPF Actually Is

eBPF stands for "extended Berkeley Packet Filter," but the name undersells it. Originally a packet filtering innovation, it’s evolved into a general-purpose execution engine inside the kernel. You write small programs in a restricted C-like language, verify them for safety, and attach them to kernel events—syscalls, network packets, tracepoints, or even arbitrary functions.

The key difference from traditional kernel modules: eBPF programs are verified for termination and memory safety before they run. The kernel will reject any program that loops indefinitely or accesses unsafe memory. That means you can experiment with live production systems without crashing them.

Where eBPF Shines in Observability

Classic observability tools (strace, top, perf) often rely on fixed counters or breakpoints. eBPF gives you dynamic instrumentation—you can attach to almost any kernel function and selectively collect data.

Example scenarios: - Network performance: Instead of dumping all packets, eBPF can count packets by protocol, source IP, or latency bucket with minimal overhead. - File system profiling: Trace which files are opened by which process, and how long each read() takes. - Application latency: Hook into the kernel’s scheduler to measure how long a process waits for CPU, not just its total runtime.

The overhead is remarkably low. Because eBPF programs run in a JIT-compiled, in-kernel bytecode, you can aggregate data there instead of shipping every event to userspace.

How It Works Under the Hood

Your eBPF program gets compiled to bytecode, verified, then attached to a hook point (like kprobe, tracepoint, or a network socket). The program runs in a restricted context—no loops, no dynamic memory allocation. It can read kernel memory, but only if the verifier confirms it’s safe.

Data is passed back to userspace via maps (key-value stores) or perf ring buffers. This split design keeps the kernel side lean—your program just updates counters or records timestamps. Userspace pulls summary data, not raw events.

Practical Examples

Count syscalls per process:

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __type(key, u64);
    __type(value, u64);
    __uint(max_entries, 1024);
} syscall_count SEC(".maps");

SEC("tracepoint/syscalls/sys_enter_openat")
int count_openat(struct trace_event_raw_sys_enter *ctx) {
    u64 pid = bpf_get_current_pid_tgid() >> 32;
    u64 *val = bpf_map_lookup_elem(&syscall_count, &pid);
    if (val) {
        __sync_fetch_and_add(val, 1);
    } else {
        u64 one = 1;
        bpf_map_update_elem(&syscall_count, &pid, &one, BPF_NOEXIST);
    }
    return 0;
}

Then read the map in Python with bcc or bpftool.

Track HTTP request latency without modifying the app: You can attach a probe to the kernel’s tcp_sendmsg or the application’s accept() syscall, timestamp the event, and then log only when the response exits. No agent sidecar needed.

The Ecosystem

The most accessible tools today are bcc (BPF Compiler Collection) and bpftrace. Bcc provides Python bindings—you write the eBPF logic in C, the Python part handles loading and data collection. Bpftrace is an awk-like one-liner language for quick ad-hoc tracing.

For deeper analysis, Cilium uses eBPF for Kubernetes networking and security. Falco uses it for runtime security detection. And Pixie (now under New Relic) offers eBPF-based auto-instrumentation for microservices.

When Not to Use eBPF

It’s not a silver bullet. eBPF programs can’t call arbitrary kernel functions or block indefinitely. For high-frequency tracing, you still need to keep your probes short. And the tooling is evolving—you’ll encounter compatibility quirks between kernel versions and distribution-specific eBPF features.

But for most observability use cases, eBPF replaces the old model of “instrument everything and dump to logs.” You capture exactly what you need, process it in-kernel, and leave the rest untouched.

Getting Started

Install bcc on any modern Linux (4.1+ kernel) and try:

sudo opensnoop

That lists every file open across the system—no changes to processes or filesystems.

For more productive use, learn bpftrace syntax. It’s wickedly simple:

sudo bpftrace -e 'kprobe:vfs_read { @[comm] = count() }'

Counts how many times each process calls vfs_read. The output is live and seconds matter.

The Bottom Line

eBPF isn’t just another tool—it’s a new layer in the Linux stack. It separates the what of observability from the how. You decide what to trace, the kernel handles the heavy lifting safely. If you’re still grepping /proc or installing heavyweight agents, you’re working too hard.