Work with pipes & processes

You've learned to navigate files and directories, but real Linux power comes when you combine commands and manage the processes they spawn. Staring at a terminal where you can't run the next command until the current one finishes, or capturing output from one tool to feed another without copy-paste, is a productivity killer. This lesson shows you how to chain commands with pipes and take control of processes — essential skills for any backend developer who needs to inspect logs, monitor services, or prepare for telemetry.

The problem this lesson solves

Imagine you're debugging a slow API endpoint. You need to see which processes are consuming CPU, then check if any are related to your service. Without pipes, you run ps aux and get a wall of text, then scan it manually. Without process control, you start a long-running backup job, and now you can't use that terminal for anything else. Pipes and process management solve both problems: they let you compose small, focused commands into powerful pipelines and run, pause, and background tasks with ease.

Pro tip: On a server, your terminal is your cockpit. Mastering pipes and processes is like learning the instrument panel — you can't fly blind.

Core concept / mental model

Think of a pipe (|) as a conveyor belt between two commands. The output of the command on the left becomes the input for the command on the right. Each command runs as a separate process — a running instance of a program. The shell (usually Bash) is the manager that coordinates these processes.

• Process: An executing program with its own PID (Process ID).
• Pipe: A unidirectional data channel connecting the stdout (standard output) of one command to the stdin (standard input) of another.
• Job control: The shell's ability to start, stop, pause, or resume processes.

Here's a visual in words:

[command A] --stdout--> | pipe | --stdin--> [command B]

You can chain multiple pipes: A | B | C. Each command runs concurrently, but data flows left to right. This is a unix philosophy — do one thing well, and connect them with pipes.

How it works step by step

1. Start a process: When you type a command and press Enter, the shell forks a new child process.
2. Connect stdout to stdin: When you use |, the shell creates a pipe (a buffer in memory) and redirects the left command's stdout to the pipe's write end, and the right command's stdin from the read end.
3. Run concurrently: Both commands start at the same time. The left command produces data, the right one consumes it. The shell waits for both to finish.
4. Manage processes: You can run processes in the foreground (blocking the terminal) or background (by appending &). You can pause/resume with Ctrl+Z and bg/fg, and list active jobs with jobs.

Let's see some real commands in action.

Hands-on walkthrough

Chaining commands with pipes

Start with a simple pipeline to list files by size:

# List files in current dir, sort by size (largest first)
ls -l | sort -k5 -nr

Expected output: A sorted list of files, largest at the top.

Now filter for something specific — find largest log files:

ls -l *.log | sort -k5 -nr | head -5

Expected output: The five largest .log files.

Counting occurrences with grep and wc

# How many times does "ERROR" appear in app.log?
grep -c "ERROR" app.log

# Alternative using pipe
cat app.log | grep "ERROR" | wc -l

Expected output: A number (the count). The first is more efficient, but the pipeline shows the pattern.

Background processes and job control

# Run a long process in background
sleep 100 &

# List background jobs
jobs

# Bring it to foreground (press Ctrl+Z to pause, then bg to resume)
fg %1

Expected output: The sleep process runs in background; jobs shows [1]+ Running sleep 100.

Using process substitution with pipes

Sometimes you need a file-like input from a command. Use <(...) for process substitution:

diff <(sort list1.txt) <(sort list2.txt)

This compares the sorted contents of two files without writing temp files.

Compare options / when to choose what

When to choose what: Use pipes for most composition. Use redirection when you need to persist output. Use process substitution when a tool's interface demands a file path.

Troubleshooting & edge cases

• Broken pipe error: If a command aborts early (e.g., head stops reading), you may see "broken pipe". This is normal — the downstream command finished.
• Mixing stderr: Pipes only pass stdout. To include stderr, redirect it: command 2>&1 | grep ....
• Zombie processes: If a child process terminates but the parent hasn't called wait(), it becomes a zombie. They're harmless but indicate a bug — use ps aux | grep Z to spot them.
• Piping with commands that buffer: Some tools (like grep --line-buffered) buffer output. Add --line-buffered to get real-time output in logs.
• Background process with input: If a background process tries to read from the terminal, it may hang. Redirect stdin from /dev/null.

What you learned & what's next

You can now chain commands with pipes, manage background jobs, and troubleshoot common process issues. You've seen practical examples and compared options. This foundation is critical for the next lesson: telemetry — where you'll pipe logs and metrics into monitoring tools. With pipes, you'll feed data from system commands into collectors; with process control, you'll manage those collectors as daemons.

Now, try the practice exercise: build a pipeline to show the top 3 CPU-consuming processes, then start a background job and practice stopping it. This hands-on effort will cement your skills and prepare you for the next step.