Interact with REST APIs using requests

Have you ever found yourself manually clicking through a web UI to check if a service is healthy, or copying curl commands from a README into your terminal over and over again? Every moment you spend doing that in a DevOps role is a moment you're not automating something more valuable. Interacting with REST APIs using requests is the single most common pattern in infrastructure automation — whether you're querying Kubernetes metrics, triggering a CI/CD pipeline, or rotating an API key, the requests library is your Swiss Army knife. This lesson will take you from your first GET to handling authentication, timeouts, and graceful error handling, so you can write automation that's both reliable and production-ready.

The problem this lesson solves

Imagine you need to check the status of 50 microservices across 5 environments every morning. Doing that manually means opening a browser, typing in URLs, and copy-pasting JSON into a document — at least 30 minutes of boring, error-prone work. Now imagine you need to do it every hour because your on-call rotation depends on it. That's the pain point: manual API interaction doesn't scale, and it's a breeding ground for human error.

In DevOps, automation is the whole point. You need to programmatically interact with REST APIs to:

• Check service health — hit /healthz or /status endpoints and parse the response.
• Trigger deployments — call a CI/CD API (like Jenkins or GitHub Actions) to start a build.
• Provision or clean up resources — create and delete cloud resources via REST APIs (e.g., DigitalOcean, AWS API Gateway, Azure).
• Rotate secrets — update API keys or tokens by calling a secrets management API.

Without a solid understanding of how to query REST APIs in Python, you'll fall back to shell scripts with curl and jq, which are brittle and hard to debug. The requests library gives you a clean, Pythonic interface that handles the details for you — and it's the de facto standard for API interaction in the Python ecosystem.

By the end of this lesson, you'll be able to write a Python script that authenticates, performs a CRUD operation, and handles errors gracefully — everything you need to replace those manual browser sessions.

Core concept / mental model

Think of a REST API as a restaurant menu. The menu has a list of items (resources) you can order (request), each with a specific method (how you want it prepared: GET = inspect, POST = order, PUT = modify, DELETE = remove). The requests library is your waiter — it takes your order (the URL and parameters), talks to the kitchen (the server), and brings back the result (the response).

The key concepts to grasp:

• HTTP methods (verbs): GET, POST, PUT, DELETE, PATCH. Each maps to an action on a resource.
• Endpoints: The specific URL where a resource lives, e.g., https://api.github.com/repos/{owner}/{repo}.
• Headers: Metadata you send with a request — content type, authentication tokens, user-agent, etc.
• Query parameters (params) and request body (data or json): The ingredients of your request.
• Status codes: The server's response — 200 OK, 201 Created, 404 Not Found, 429 Too Many Requests, 500 Internal Server Error.

Here's a simple visualization:

[Your Python script] -> (requests.get(url, params={...}, headers={...})) -> HTTP Request
        ^                                                                 |
        |                                                                 v
        +------------------------ (response.status_code, response.json()) <- HTTP Response

The beauty of requests is that it abstracts away the low-level socket handling, SSL/TLS, and connection pooling. You just write requests.get(url, params={}) and you're done. Under the hood, it's using urllib3 for connection management and handles redirects, cookies, and gzip decompression automatically.

The mental model to lock in: every API call is a request-response cycle, and requests makes that cycle as simple as a function call. The response object gives you the status, headers, and body — which you can then inspect and parse.

How it works step by step

Let's break down the anatomy of a requests call. Every API interaction boils down to five steps:

1. Import the library — import requests.
2. Build the request — choose the HTTP method and specify the URL, params, headers, and/or body.
3. Send the request — call the appropriate requests.get(), requests.post(), etc.
4. Inspect the response — check response.status_code, response.headers, response.json().
5. Handle errors — use response.raise_for_status() or check the status code manually.

Step 1: The URL

The URL is the address of the resource. It can include path parameters (e.g., /users/<id>) and query parameters (e.g., ?name=bob&page=2).

url = "https://jsonplaceholder.typicode.com/todos/1"

Step 2: Params vs. Data

• params — used for GET requests to pass query string parameters. The library will URL-encode them for you.
• data — used for POST/PUT to send form-encoded data.
• json — used for POST/PUT to send JSON serialized data, automatically setting Content-Type: application/json.

Step 3: Headers and Authentication

Most real-world APIs require headers for authentication. You can set them per request or use a session object to persist them.

headers = {
    "Authorization": "Bearer your_token_here",
    "Accept": "application/json",
}

Step 4: Timeouts and Retries

Always set a timeout so your automation doesn't hang forever. You can also use the HTTPAdapter to configure retries based on status codes.

response = requests.get(url, timeout=5)  # 5 seconds

Step 5: Parsing the Response

The response object has .status_code, .headers, .text, and (if JSON) .json(). Never assume the response is valid JSON — wrap it in a try/except.

Step 6: Error Handling

response.raise_for_status() raises an HTTPError if the status code is 4xx or 5xx. This is a clean way to catch failures and bail out early.

Hands-on walkthrough

Now for the fun part — let's write some real code. We'll build a script that fetches user data from two JSONPlaceholder endpoints (a free fake API for testing) and prints a summary.

Example 1: Basic GET request

import requests

url = "https://jsonplaceholder.typicode.com/users/1"
try:
    response = requests.get(url, timeout=5)
    response.raise_for_status()
    user = response.json()
    print(f"User: {user['name']}, email: {user['email']}")
    print(f"Address: {user['address']['street']}, {user['address']['city']}")
except requests.exceptions.RequestException as e:
    print(f"Failed to fetch user: {e}")

Expected output:

User: Leanne Graham, email: Sincere@april.biz
Address: Kulas Light, Gwenborough

Example 2: POST with JSON data

Now let's create a new post (a common pattern for creating resources).

import requests

url = "https://jsonplaceholder.typicode.com/posts"
payload = {
    "title": "My first automated post",
    "body": "This was created by a Python script!",
    "userId": 1
}
try:
    response = requests.post(url, json=payload, timeout=5)
    response.raise_for_status()
    created = response.json()
    print(f"Created post with ID: {created['id']}")
    print(f"Title: {created['title']}")
except requests.exceptions.RequestException as e:
    print(f"Failed to create post: {e}")

Expected output:

Created post with ID: 101
Title: My first automated post

Example 3: Using query parameters

Filtering resources is done with query params. For example, fetch all todos for a specific user.

import requests

url = "https://jsonplaceholder.typicode.com/todos"
params = {"userId": 1, "completed": "false"}
try:
    response = requests.get(url, params=params, timeout=5)
    response.raise_for_status()
    todos = response.json()
    print(f"Found {len(todos)} incomplete todos for user 1.")
    for todo in todos:
        print(f"  - {todo['title']}")
except requests.exceptions.RequestException as e:
    print(f"Failed to fetch todos: {e}")

Expected output:

Found 4 incomplete todos for user 1.
  - laboriosam mollitia et enim quasi adipisci quia provident illum
  - ...

Example 4: Authentication via session

When you need to make multiple calls with the same auth token, use a requests.Session() to persist headers across requests. This is more efficient because it reuses the underlying TCP connection.

import requests

session = requests.Session()
session.headers.update({
    "Authorization": "Bearer abc123xyz",
    "Accept": "application/json",
})

# First call
resp1 = session.get("https://api.example.com/v1/health", timeout=5)
print("Health check status:", resp1.status_code)

# Second call (same session, so token is still sent)
resp2 = session.get("https://api.example.com/v1/users/me", timeout=5)
print("Me endpoint status:", resp2.status_code)

Using a session also allows you to set per-request overrides, and it handles cookies and connection pooling automatically.

Pro tip: Always wrap your API calls in a try/except for requests.exceptions.RequestException, which catches connection errors, timeouts, and HTTP errors. Then use response.raise_for_status() to catch non-2xx responses. This makes your scripts robust in production.

Compare options / when to choose what

While requests is the go-to for simple, synchronous API calls, you have other options depending on your needs. Here's a quick comparison:

When to choose what: - Stick with requests if you're writing a one-off script or a simple automation that makes a few calls. - Choose httpx if you need async without abandoning the requests-style API. - Use aiohttp if you have to make thousands of parallel requests (e.g., health-checking a large cluster). - Avoid urllib.request unless you have no choice — its ergonomics are terrible for automation.

Troubleshooting & edge cases

Even with requests, things go wrong. Here are the most common pitfalls and how to handle them:

• ConnectionError: Max retries exceeded — The server is down or the URL is wrong. Double-check the URL and whether you're behind a proxy. Use response.raise_for_status() to catch it.
• Timeout — The server didn't respond within your timeout. Always set timeout — never leave it unlimited, or your script can hang indefinitely.
• JSONDecodeError — The response body isn't valid JSON. This often happens when you request an endpoint that returns HTML (like a 404 page). Check response.text and the Content-Type header.
• Too many redirects — Some APIs redirect to a login page. Set allow_redirects=False for POST requests if you want to avoid silent redirects.
• Rate limiting (429) — The API is throttling you. Respect the Retry-After header and implement exponential backoff.
• SSL certificate errors — If you're hitting a self-signed cert (common in test environments), you can pass verify=False, but never do this in production — use a proper CA bundle instead.

Edge case example: A common mistake is not checking whether the response contains valid JSON before calling .json(). Always wrap it in a try/except or use the r.raise_for_status() first.

What you learned & what's next

Congratulations! You now understand the core idea behind interacting with REST APIs using requests — that every API call is a request-response cycle, and you've seen how to execute that cycle in Python. You've completed a practical exercise where you made a GET, a POST, used query parameters, and handled authentication with sessions. You're well on your way to automating real-world infrastructure tasks.

Now you're ready to take the next step in the Python for DevOps automation track: processing and transforming the JSON data you fetch. In the next lesson, we'll dive into manipulating JSON payloads — filtering, sorting, and reshaping data so you can turn raw API responses into actionable reports and configuration files.

Keep your requests cheat sheet handy — you'll use it in every subsequent lesson, from cloud resource management to Kubernetes operations.