MongoDB _id and ObjectId

Every MongoDB document you insert — every user, order, or log entry — carries a hidden field that silently guarantees uniqueness across your entire cluster. That field is _id, and when you let MongoDB assign it automatically, you get an ObjectId that stores far more than just a random number. If you've ever wondered why _id looks like a 24-character hex string, why it's safe to generate on the client side, or how to avoid _id collisions in your own applications, this lesson is for you. By the end, you'll not only understand MongoDB's _id and ObjectId in depth, but you'll know exactly when to keep the default, when to supply your own _id, and how to debug the common issues that trip up developers.

The problem this lesson solves

MongoDB is a document database built for scale, and every document needs a reliable way to be identified — whether you're updating it, deleting it, or joining it with another collection. The naive approach is to use an auto-incrementing integer, like you might in a relational database. But auto-increment IDs are a terrible fit for distributed systems: if multiple application servers or MongoDB shards insert documents at the same time, they could easily generate the same 1, 2, or 3, causing duplicate errors and forcing you to coordinate globally.

Another common problem is the misconception that MongoID's _id is just a string you can ignore. Developers often skip defining _id altogether, insert a document, and later discover that the default ObjectId encodes a timestamp they could have used for sorting or debugging — but didn't know how to read it. Others try to replace _id with UUIDs or custom values without understanding the trade-offs, leading to performance issues or fragile code.

Without a deep understanding of _id and ObjectId, you'll make decisions about schema design, index keys, and sharding that are hard to reverse. This lesson solves that by demystifying the default identifier, showing you exactly how ObjectId is built and why it's safe to generate client-side, and giving you a clear mental model for when to use the default versus when to bring your own.

Core concept / mental model

Think of _id as the primary key of every MongoDB document. It's a required field that must be unique within a collection, and it's automatically indexed on the _id field (creating a unique index) so that lookups by _id are blazingly fast. If you don't provide an _id, MongoDB generates one for you — by default, a 12-byte ObjectId.

Here's the mental model: an ObjectId is like a timestamp + machine + process + counter packed into a compact binary value. When you see a hex string like 507f1f77bcf86cd799439011, you're actually looking at a 24-character hexadecimal representation of those components:

• 4-byte timestamp (seconds since the Unix epoch) — lets you know when the document was created
• 5-byte random value — unique per machine and process, so even two servers generating IDs at the same second won't collide
• 3-byte incrementing counter — starts at a random value and increments per process, ensuring uniqueness for multiple IDs generated in the same second

This design is brilliant for distributed systems because it allows any application server to generate a unique ID without talking to a central server — no coordination needed. It's the same reason UUIDs are popular, but ObjectIds are smaller (12 bytes vs. 16 bytes for a UUID) and include a timestamp, which is a nice extra.

Key definitions:

• _id: the mandatory unique identifier field in every document.
• ObjectId: the default 12-byte BSON type used for _id, generated by MongoDB or the driver.
• _id index: the automatic unique index that speeds up queries on _id.

How it works step by step

Let's walk through what happens when you insert a document without an _id:

1. Your application (or the MongoDB shell) builds a document, e.g., { "name": "Ada", "age": 36 }.
2. The driver checks the document — no _id is present.
3. The driver (or MongoDB server, depending on version) generates an ObjectId — using the current timestamp, a random machine identifier, and an incrementing counter.
4. The driver adds _id to the document, so it becomes { _id: ObjectId("507f1f77bcf86cd799439011"), name: "Ada", age: 36 }.
5. MongoDB inserts the document, and the unique index on _id ensures no duplicate _id values exist in the collection.
6. On subsequent queries, you can use _id to fetch the document: db.users.find({ _id: ObjectId("...") }).

When you provide your own _id:

1. You set _id yourself — it can be any BSON type except an array.
2. MongoDB uses that value as-is, and still creates the unique index.
3. If you try to insert another document with the same _id, MongoDB throws a duplicate key error.

Important nuance: In older MongoDB versions (pre-3.4), the client generated ObjectIds; in later versions, the server can also generate them if the client doesn't. But the end result is the same — you get a valid ObjectId.

Why client-side generation is safe: Because ObjectId includes a random component plus an incrementing counter, the probability of two processes generating the same ID is astronomically low — roughly 1 in 16^15. That's why you can safely generate ObjectIds in your application code without hitting the database, which is great for building offline-first apps or batch processing.

Hands-on walkthrough

Let's get your hands dirty. We'll use the mongosh shell to see ObjectIds in action. Make sure you have MongoDB running locally.

1. Insert a document without an _id

use test_db

db.users.insertOne({ name: "Ada", age: 36 })

Expected output:

{
  "acknowledged": true,
  "insertedId": ObjectId("65f0c1a2b3c4d5e6f7a8b9c0")
}

Notice how MongoDB returns the generated ObjectId. When you query, you'll see the _id field added automatically.

2. Inspect the ObjectId structure

MongoDB's ObjectId has built-in methods to extract its parts:

db.users.insertOne({ name: "Bob" })
const doc = db.users.findOne({ name: "Bob" })
print("Timestamp (seconds):", doc._id.getTimestamp())
print("Hex string:", doc._id.toString())
print("JSON:", JSON.stringify(doc._id))

Expected output (similar):

Timestamp (seconds): 2024-03-14T12:00:00.000Z
Hex string: 65f0c1a2b3c4d5e6f7a8b9c0
JSON: "65f0c1a2b3c4d5e6f7a8b9c0"

The getTimestamp() returns the creation time, which is super useful for sorting documents by insertion time without storing a separate field.

3. Generate an ObjectId in Python (PyMongo)

If you're using Python, the bson.objectid.ObjectId class lets you generate your own:

from bson.objectid import ObjectId
import datetime

# Generate a new ObjectId
new_id = ObjectId()
print("Generated:", new_id)

# Extract timestamp
ts = new_id.generation_time
print("Creation time:", ts)

# Insert with a custom _id
from pymongo import MongoClient

client = MongoClient("mongodb://localhost:27017")
db = client.test_db
result = db.items.insert_one({ "_id": new_id, "name": "Widget" })
print("Inserted with _id:", result.inserted_id)

# Verify uniqueness
# Try inserting again with the same _id -> should raise DuplicateKeyError
try:
    db.items.insert_one({ "_id": new_id, "name": "Widget2" })
except Exception as e:
    print("Duplicate error:", e)

Expected output:

Generated: 65f0c1a2b3c4d5e6f7a8b9c0
Creation time: 2024-03-14 12:00:00+00:00
Inserted with _id: 65f0c1a2b3c4d5e6f7a8b9c0
Duplicate error: E11000 duplicate key error collection: test_db.items index: _id_ dup key: { _id: ObjectId('65f0c1a2b3c4d5e6f7a8b9c0') }

That error proves the unique index is working.

Compare options / when to choose what

Not every _id needs to be an ObjectId. MongoDB lets you set _id to any BSON type (except array), which opens the door for custom identifiers. Here's a comparison of common choices:

When to use ObjectId:

• You need a globally unique ID without extra logic.
• You want to sort by creation time for free.
• You want to generate IDs offline or in a distributed environment.

When to avoid ObjectId:

• You need sequential or human-readable IDs (e.g., invoice numbers).
• You already have a natural unique key and want to avoid redundancy.
• You're using MongoDB as a cache and need compact keys.

Variation: Using a UUID as _id via the UUID type in BSON (in MongoDB 4.0+) or storing it as a string. This can be good for interoperability, but remember that UUIDs are 16 bytes, so indexes will be larger and slightly slower.

Troubleshooting & edge cases

1. Duplicate key error on _id

If you see E11000 duplicate key error on _id, it means you tried to insert two documents with the same _id. This can happen if:

• You re-insert a document after a retry without clearing the _id field.
• You generate ObjectIds using a bad custom method that reuses values.
• You import data that already has _ids and you didn't delete them.

Fix: Check your import/insert logic, and either remove _id or use updateOne with upsert if you want to insert if not exists.

2. Sorting by _id doesn't reflect actual insertion order

Because ObjectId has a random component, sorting by _id with .sort({ _id: 1 }) does not guarantee insertion order across processes. It only gives you an approximate ordering within the same second (due to the counter). If you need exact insertion order, add a createdAt field with a timestamp.

3. _id is immutable by default

You cannot update _id on an existing document without deleting and re-inserting it. If you try updateOne({ _id: old }, { $set: { _id: new } }), MongoDB will throw an error like Mod on _id not allowed. Design your schemas so _id never needs to change.

4. ObjectId generated in Python vs. MongoDB server mismatch

If you're using PyMongo, the driver generates ObjectIds client-side by default, but if you connect to MongoDB 3.4+, the server can also generate them. In practice, it doesn't matter — both produce standard ObjectIds. But if you ever see an ObjectId that looks different (e.g., all zeros), it might come from a deprecated driver.

5. Custom _id that's an array — not allowed

You cannot use an array as _id. If you try, MongoDB rejects it. Use a subdocument with a unique field inside instead.

6. Performance concerns with large _id values

If you choose a long string as _id, it increases the index size and makes queries slower. Keep _id as small as possible.

What you learned & what's next

You've now mastered MongoDB's _id and ObjectId in depth. You understand:

• _id is the required unique identifier, automatically indexed.
• ObjectId is a 12-byte structure with timestamp, machine, and counter components.
• You can generate ObjectIds client-side, which is safe for distributed apps.
• You can customize _id to be any BSON type except array, but ObjectId is the best default for most cases.
• You know how to avoid duplicate key errors and can interpret ObjectId timestamps.

Next step: Now that you understand _id, you're ready to dive into MongoDB indexes — how to create, manage, and optimize index performance. Indexes are the natural next topic because _id is the first index you've seen, and understanding it will help you reason about compound and secondary indexes.

You've built a solid foundation — go index something!