Aggregate Data in PyMongo

If you've ever tried to compute totals, averages, or grouped metrics directly from MongoDB using plain find() queries, you already know the pain: you pull thousands of documents into Python, loop over them, and write fragile code that buries the actual logic under boilerplate. That approach wastes memory, slows down your application, and makes your code harder to maintain. The solution is right there in the driver: the collection.aggregate() method, which lets you run MongoDB's powerful aggregation pipeline directly from PyMongo. This lesson shows you how to master it.

The problem this lesson solves

Imagine you run an e-commerce store and need a daily sales report. With find(), you fetch every order document and aggregate in Python:

import pymongo
from datetime import datetime, timedelta

client = pymongo.MongoClient("mongodb://localhost:27017/")
db = client["shop"]
orders = db["orders"]

yesterday = datetime.now() - timedelta(days=1)

# Pull all orders from the last day into memory
all_orders = list(orders.find({"created_at": {"$gte": yesterday}}))

# Now aggregate in Python (slow, memory-hungry, and error-prone)
totals = {}
for order in all_orders:
    totals[order["product"]] = totals.get(order["product"], 0) + order["amount"]

print(totals)

This works for a small dataset, but scale it to millions of documents and you'll hit memory limits, network bottlenecks, and painfully slow reports. Worse, every logic change forces a full round-trip of data. Aggregation in MongoDB solves exactly this: it pushes the computation into the database, returning only the results you need.

Core concept / mental model

Think of the aggregation pipeline as an assembly line for data processing. Each stage is a machine on that line that takes a stream of documents, transforms or filters them, and passes the result to the next machine. The pipeline is declared as a list, and PyMongo's collection.aggregate() executes it server-side, returning a cursor with the final output.

In MongoDB jargon: - Pipeline — the list of stages. - Stage — one transformation, like $match, $group, $project, $sort... - Operator — keyword inside a stage, like $gt for greater-than, $sum for accumulating.

For example, [{"$match": {"status": "active"}}, {"$group": {"_id": "$category", "total": {"$sum": 1}}}] tells MongoDB: first filter to active documents, then count per category. The order matters — grouping before filtering would waste effort.

This mental model makes it easy to design pipelines in your head: start with the data you have, apply filters and transformations, then group and shape the results.

How it works step by step

Let's break down the anatomy of a PyMongo aggregation call:

1. Connect to MongoDB and get your collection (as always).
2. Build the pipeline as a list of dictionaries. Each dictionary represents one stage.
3. Call collection.aggregate(pipeline) — the driver sends the pipeline to the server.
4. Iterate over the resulting cursor, which yields result documents.

Key pipeline stages you'll use daily

• $match — filters documents by conditions. Always put it early to reduce data flow.
• $group — groups documents by a key and applies accumulators like $sum, $avg, $max, $min, $push.
• $project — reshapes documents: include, exclude, or compute fields.
• $sort — orders results by a field (1 ascending, -1 descending).
• $limit / $skip — paginate results.
• $unwind — deconstructs an array field into multiple documents (useful for nested data).

But aggregate() isn't just for grouping. You can also use it as a more flexible find() with computations — for instance, converting field types or adding calculated fields with $addFields.

Hands-on walkthrough

Setting up sample data

Let's use the shop database and an orders collection with these documents (you can insert them with insert_many):

import pymongo

client = pymongo.MongoClient("mongodb://localhost:27017/")
db = client["shop"]
orders = db["orders"]

orders.insert_many([
    {"product": "laptop", "amount": 900, "status": "completed", "customer": "Alice"},
    {"product": "mouse", "amount": 25, "status": "completed", "customer": "Bob"},
    {"product": "laptop", "amount": 850, "status": "pending", "customer": "Charlie"},
    {"product": "keyboard", "amount": 45, "status": "completed", "customer": "Alice"},
    {"product": "mouse", "amount": 20, "status": "completed", "customer": "David"},
])

Example 1: Simple grouping and sum

Compute total sales per product, only for completed orders:

pipeline = [
    {"$match": {"status": "completed"}},
    {"$group": {"_id": "$product", "total_sales": {"$sum": "$amount"}}}
]

results = orders.aggregate(pipeline)
for doc in results:
    print(doc)

Expected output (order may vary):

{'_id': 'laptop', 'total_sales': 900}
{'_id': 'mouse', 'total_sales': 45}
{'_id': 'keyboard', 'total_sales': 45}

Notice how the _id field is the grouping key, and we compute the sum using $sum on the amount field.

Example 2: Filter, group, and sort

Now add a sort to get the best-selling products first:

pipeline = [
    {"$match": {"status": "completed"}},
    {"$group": {"_id": "$product", "total_sales": {"$sum": "$amount"}}},
    {"$sort": {"total_sales": -1}}
]

for doc in orders.aggregate(pipeline):
    print(doc)

Expected output:

{'_id': 'laptop', 'total_sales': 900}
{'_id': 'keyboard', 'total_sales': 45}
{'_id': 'mouse', 'total_sales': 45}

Example 3: Using $project to shape output

If you want a cleaner result without the _id field as product name, use $project:

pipeline = [
    {"$match": {"status": "completed"}},
    {"$group": {"_id": "$product", "total_sales": {"$sum": "$amount"}}},
    {"$project": {"_id": 0, "product": "$_id", "total_sales": 1}},
    {"$sort": {"total_sales": -1}}
]

for doc in orders.aggregate(pipeline):
    print(doc)

Output:

{'product': 'laptop', 'total_sales': 900}
{'product': 'keyboard', 'total_sales': 45}
{'product': 'mouse', 'total_sales': 45}

Example 4: Averages and counts

What if you need the count of orders and the average amount per product?

pipeline = [
    {"$group": {
        "_id": "$product",
        "order_count": {"$sum": 1},
        "average_amount": {"$avg": "$amount"}
    }}
]

for doc in orders.aggregate(pipeline):
    print(doc)

Output:

{'_id': 'keyboard', 'order_count': 1, 'average_amount': 45.0}
{'_id': 'mouse', 'order_count': 2, 'average_amount': 22.5}
{'_id': 'laptop', 'order_count': 2, 'average_amount': 875.0}

Pro tip: The pipeline stages are simple dictionaries. You can even build them programmatically, using Python variables for conditions, just keep in mind that MongoDB operators are strings starting with $.

Compare options / when to choose what

If aggregate() is so great, why would you ever use find()? Good question. Here's a quick comparison:

In short, choose find() when you just need to retrieve documents without any server-side computation. Choose aggregate() whenever you need groupings, totals, averages, or in-database transformations. It's the modern, scalable choice for data analysis in MongoDB.

Variations of aggregation

• $facet — perform multiple pipelines on the same input in one go (multi-dimensional analytics).
• $lookup — join documents from another collection, a kind of SQL join in MongoDB.
• $unwind + $group — process arrays flatten them first for accurate grouping.

These are advanced tools you'll meet later, but collection.aggregate() is the base for all of them.

Troubleshooting & edge cases

Common mistakes

• Forgetting that _id in $group is mandatory. If you don't set _id, you'll get an error.
• Using Python operators like == inside the pipeline. MongoDB uses $eq etc., not Python syntax.
• No index on fields you $match or $sort on. Without an index, performance degrades on large collections.

Real error examples

1. Filtering before grouping works, but double-check field names.

# Wrong: field name typo
pipeline = [{"$match": {"status": "completed"}}, {"$group": {"_id": "$product", "total": {"$sum": "$ammount"}}}]
# Result: total is 0 for all groups because $ammount doesn't exist

Fix: ensure spelling matches the document.

2. Type mismatch in $sort

# If some documents have amount as string and others as number
pipeline = [{"$sort": {"amount": -1}}]
# MongoDB may sort inconsistently; use $convert in $project first.

3. Cursor is not a list

aggregate() returns a cursor, not a list. Wrap with list() if you need to reuse it, otherwise it's exhausted after one iteration.

What you learned & what's next

You've now seen how to aggregate data with PyMongo and the collection.aggregate() method: from building a pipeline, to filtering, grouping, projecting, and sorting. You know the mental model of a document assembly line, and you can choose between find() and aggregate() based on your use case. You've also learned common pitfalls to avoid.

Next up: in the following lesson, you'll dive deeper into more advanced aggregation stages like $lookup for cross-collection joins, $unwind for array manipulation, and $facet for multi-faceted analytics — all still leveraging the same collection.aggregate() foundation you've mastered here. Keep practicing, and you'll soon be a MongoDB data-crunching pro.

Pro tip: Every aggregation pipeline you write becomes a reusable function. Wrap it in a Python function that accepts parameters (like date ranges or product categories) to build clean, testable data-analytics code.