Set Up a Simple Aggregation Pipeline

You've mastered CRUD operations — inserting, querying, updating, and deleting documents. But what happens when you need to answer questions like "What's the average order value per customer?" or "Which products are trending this week?" Doing this with raw queries would mean pulling thousands of documents into your application and doing the math yourself — slow, painful, and wasteful. That's the problem this lesson solves: setting up a simple aggregation pipeline in MongoDB so you can transform and analyze your data directly in the database, with power and elegance.

The problem this lesson solves

When your dataset grows beyond a handful of documents, you inevitably need to aggregate data — group records, calculate totals, filter on computed values, or reshape documents for reports. Without aggregation, you'd write multiple queries, loop through results in your application, and manage temporary variables. Not only is this slow (especially over a network), but it's also error-prone and hard to maintain.

Consider a common task: find the total sales per product category for the last month. With a naive approach, you'd fetch every order in that timeframe, iterate through them in your application, sum up values by category — and then do it all again next week with slightly different logic. This approach breaks down as data grows and questions become more complex.

Aggregation pipelines solve this by letting you define a multi-stage transformation — each stage takes the output of the previous one, filters, groups, or reshapes it, and passes the result along. It's like an assembly line for your data, and it runs entirely inside MongoDB, close to the data.

By the end of this lesson, you'll be able to explain what an aggregation pipeline is, set up a simple one using the $match, $group, $sort, and $project stages, and run it with both the mongosh shell and the PyMongo driver.

Core concept / mental model

Think of a pipeline like a factory assembly line for documents. Each stage is a workstation. Documents enter the line, get processed at each station, and come out the other end transformed. You can have as many stations as you need, and each one does one specific job.

The aggregation framework in MongoDB uses a declarative syntax: you tell MongoDB what you want, not how to do it. Each stage is a document that starts with a $ operator, for example:

• $match — filters documents (like find, but inside the pipeline).
• $group — groups documents by a key and applies accumulator expressions (like $sum, $avg, $max).
• $sort — orders the resulting documents.
• $project — shapes the output, including adding computed fields or removing ones.
• $limit / $skip — pagination.

A simple pipeline looks like this in words: "Match orders from last month, group them by category, calculate the total sales, then sort by that total, and finally output only the category and total."

Pro tip: In MongoDB, a pipeline is an array of stages. The order of stages matters — the output of one becomes the input to the next. A well-ordered pipeline can dramatically affect performance and readability.

Think of it as a chain: $match first to reduce data (great for performance), $group to compress, $sort to order, and $project to shape the final output. That's your mental model for most simple pipelines.

How it works step by step

Setting up a simple aggregation pipeline is a logical sequence of decisions, each building on the previous. Here's the cause → effect path you'll follow:

1. Decide what question you want to answer. State it in plain language. Example: "What are the total sales per product category for orders placed in 2024?"

2. Identify the input collection. Which collection holds the source documents? In our example, it's the orders collection.

3. Choose the first stage (usually $match). Filter the documents as early as possible to limit the data that subsequent stages process. This is both logical and a performance best practice.

4. Pick the grouping stage ($group). Decide the grouping key (e.g., $category) and the accumulator expressions (e.g., $sum on $total). Each $group produces one document per distinct key.

5. Add sorting ($sort). To get the highest-total categories first, sort by the computed field in descending order.

6. Shape the output ($project). Keep only the fields you need; rename or compute new ones.

7. Run the pipeline using db.collection.aggregate([...]) in mongosh or the driver equivalent.

The beauty is that each stage is independent — you can reason about it in isolation, and you can add or remove stages without rewriting the whole thing.

Hands-on walkthrough

Let's put this into practice. First, ensure you have MongoDB running and mongosh available. We'll insert some sample order documents, then build a pipeline step by step.

Step 1: Prepare sample data

// In mongosh
db.orders.insertMany([
  { _id: 1, item: "Laptop", category: "Electronics", qty: 1, price: 1200, date: ISODate("2024-06-01T10:00:00Z") },
  { _id: 2, item: "Mouse", category: "Electronics", qty: 2, price: 25, date: ISODate("2024-06-03T12:30:00Z") },
  { _id: 3, item: "Desk Chair", category: "Furniture", qty: 1, price: 350, date: ISODate("2024-06-05T09:00:00Z") },
  { _id: 4, item: "Notebook", category: "Office", qty: 5, price: 3, date: ISODate("2024-06-07T14:00:00Z") },
  { _id: 5, item: "Monitor", category: "Electronics", qty: 1, price: 300, date: ISODate("2024-06-08T11:00:00Z") },
  { _id: 6, item: "Desk", category: "Furniture", qty: 1, price: 500, date: ISODate("2024-06-10T16:00:00Z") }
])

Step 2: Start with $match to filter

db.orders.aggregate([
  { $match: { date: { $gte: ISODate("2024-06-01T00:00:00Z"), $lt: ISODate("2024-07-01T00:00:00Z") } } }
])

This returns all orders in June 2024. In this case, all six documents match.

Step 3: Add $group to calculate totals per category

db.orders.aggregate([
  { $match: { date: { $gte: ISODate("2024-06-01T00:00:00Z"), $lt: ISODate("2024-07-01T00:00:00Z") } } },
  { $group: { _id: "$category", totalSales: { $sum: { $multiply: ["$qty", "$price"] } }, count: { $sum: 1 } } }
])

Expected output (order may vary):

[ { _id: 'Electronics', totalSales: 1250, count: 3 },
  { _id: 'Furniture', totalSales: 850, count: 2 },
  { _id: 'Office', totalSales: 15, count: 1 } ]

Here, _id becomes the grouping key (category), totalSales uses $sum over qty * price, and count counts documents per group.

Step 4: Sort and project

db.orders.aggregate([
  { $match: { date: { $gte: ISODate("2024-06-01T00:00:00Z"), $lt: ISODate("2024-07-01T00:00:00Z") } } },
  { $group: { _id: "$category", totalSales: { $sum: { $multiply: ["$qty", "$price"] } } } },
  { $sort: { totalSales: -1 } },
  { $project: { category: "$_id", totalSales: 1, _id: 0 } }
])

Expected output:

[ { category: 'Electronics', totalSales: 1250 },
  { category: 'Furniture', totalSales: 850 },
  { category: 'Office', totalSales: 15 } ]

The $project stage renames _id to category and removes _id from the output.

Using PyMongo (Python)

If you're working in Python, the same pipeline runs via the aggregate() method on a collection object:

from pymongo import MongoClient
from datetime import datetime

client = MongoClient("mongodb://localhost:27017")
db = client["shop"]
orders = db["orders"]

pipeline = [
    {"$match": {"date": {"$gte": datetime(2024, 6, 1), "$lt": datetime(2024, 7, 1)}}},
    {"$group": {"_id": "$category", "totalSales": {"$sum": {"$multiply": ["$qty", "$price"]}}}},
    {"$sort": {"totalSales": -1}},
    {"$project": {"category": "$_id", "totalSales": 1, "_id": 0}}
]

results = list(orders.aggregate(pipeline))
for r in results:
    print(r)

Expected output:

{'category': 'Electronics', 'totalSales': 1250}
{'category': 'Furniture', 'totalSales': 850}
{'category': 'Office', 'totalSales': 15}

Pro tip: In PyMongo, aggregate() returns a cursor — you must iterate it (e.g., with list()) to see the results. Always convert to a list or loop over it.

Compare options / when to choose what

Aggregation is powerful, but it's not always the right tool. Here's a quick comparison:

In practice, you'll use find() for simple lookups and aggregate() when you need grouping, sorting on computed values, or reshaping. Since MongoDB 5.0, MapReduce is deprecated in favor of $group and $accumulator — stick with aggregation.

Troubleshooting & edge cases

Let's tackle common pitfalls when setting up a simple aggregation pipeline:

1. Forgetting $ before field paths

// Wrong — "category" is treated as a literal string
{ $group: { _id: "category", ... } }

// Correct — "$category" refers to the field value
{ $group: { _id: "$category", ... } }

If you use "category" without the dollar sign, all documents get grouped into a single group with _id equal to the string "category". Check your output — if you see that, you've missed the $.

2. $group output field names must start with a non-$

In $group, the output fields (like totalSales) must not start with $. The _id field is the exception and must be the grouping key. Mistaking this causes a syntax error.

3. Using $where or JavaScript in $match — avoid it

$where runs JavaScript and kills performance. Use aggregation operators like $expr instead:

// Instead of $where, use:
{ $match: { $expr: { $gt: ["$price", 100] } } }

4. Expected output is empty

If your pipeline returns nothing, check your $match dates. ISODate in mongosh and datetime in Python must fall within the range — make sure your filter is inclusive/exclusive as intended.

5. Grouping by multiple fields

To group by multiple fields, use a document as _id: { _id: { category: "$category", year: { $year: "$date" } } }. The output _id will be a subdocument. This is common and works fine.

6. Memory limits

If you're using $sort after $group without an index, MongoDB may hit the 100 MB RAM limit for sort operations. In production, either add an index or use allowDiskUse: true (though try to avoid it for performance).

What you learned & what's next

You now understand the core idea of an aggregation pipeline: a sequence of stages that transforms documents inside MongoDB. You can set up a simple pipeline with $match, $group, $sort, and $project, run it in both mongosh and Python, and you know how to troubleshoot the typical mistakes.

Specifically, you've learned:

• To explain what a pipeline is and why it beats manual app-side aggregation.
• To complete a practical exercise: filtering, grouping, sorting, and projecting to answer a business question.
• To avoid common pitfalls like missing $ prefixes and wrong data types.

Next in the track: You'll dive into more advanced stages like $unwind, $lookup (for joins), and $addFields to handle even richer aggregation scenarios. Building on today's foundation, you'll be able to combine stages creatively to solve complex reporting problems.

Pro tip: Before moving on, practice building a pipeline that answers "Which product had the highest total revenue in June 2024?" Try adding a $limit stage after sorting to get just the top result. This will cement the stage-order mental model.