When working with MongoDB sharded clusters, developers often notice that db.collection.count() returns enconsistent or incorrect results. While this is largely addressed in version 4.0 and later for queries with predicates, understanding the underlying causes is critical for maintaining data integrity in legacy systems (like MongoDB 3.6) and high-concurrency environments.

Primary Causes of Inaccurate Counts

In a sharded environment, MongoDB typically retrieves the count from the metadata of individual shards rather than performing a full scan. Discrepancies usually arise from two specific scenarios: the presence of orphaned documents and active chunk migrations.

1. Orphaned Documents

An orphaned document is a record that exists on a shard but technically does not belong there according to the cluster's metadata. This occurs when a moveChunk operation fails or when the source shard fails to delete data after a successful migration to a destination shard.

The migration process follows these general stages:

1. The balancer initiates a moveChunk command.
2. The destination shard begins requesting documents from the source shard.
3. Once data transfer is complete, the destination shard syncs incremental changes that occurred during the transfer.
4. The metadata is updated in the Config Server to point to the new location.
5. The source shard asynchronously deletes the documents in the migrated chunk.

If the mongod process on the source shard crashes after the metadata update but before the asynchronous deletion completes, the documents remain on the source shard as orphans. Since count() without a predicate often sums up the metadata reported by shards, these duplicates result in a count higher than the actual number of documents.

Simulating Orphaned Documents

You can observe this behavior by forcing a shard rebalance and interrupting the process. In a cluster with orphaned documents, you might see results like this:

// Count without filters (reads from metadata)
mongos> db.inventory.count()
51028273

// Count with a dummy predicate (older versions might still be inaccurate)
mongos> db.inventory.count({ _id: { $exists: true } })
43937296

// Accurate count using Aggregation Framework
mongos> db.inventory.aggregate([{ $count: "total" }])
{ "total" : 43937296 }

2. Active Chunk Migrations

During a moveChunk operation, documents temporarily exist on both the source and target shards. MongoDB 3.6 and earlier versions do not always filter out these "double-counted" records when executing a basic count(). Even with _waitForDelete enabled (which forces the source to wait for deletion before proceeding), there is a window where the count fluctuates as data is added to the destination and subsequently removed from the source.

Version-Specific Behavior

The reliability of the count command changed significantly between versions:

• MongoDB 3.6: Both db.collection.count() and db.collection.count(query) can return inaccurate results if orphaned documents exist or migration are in progres.
• MongoDB 4.0+: db.collection.count() without a query predicate remains potentially inaccurate on sharded clusters. However, db.collection.count(query) was improved to behave like a standard read operation, yielding accurate results by filtering orphans based on metadata.

Remediation Strategies

Cleaning Up Orphaned Documents

If a cluster has accumulated orphans, you can manually trigger a cleanup on each shard. The following script iterates through a collection to remove data that does not belong to the shard:

var currentKey = {};
var opResult;

while (currentKey != null) {
    opResult = db.adminCommand({
        cleanupOrphaned: "app_db.inventory",
        startingFromKey: currentKey
    });

    if (opResult.ok !== 1) {
        print("Error encountered during cleanup: " + opResult.errmsg);
    }

    printjson(opResult);
    currentKey = opResult.stoppedAtKey;
}

Enforcing Synchronous Deletion

To reduce the likelihood of orphaned documents during rebalancing, you can configure the balancer to wait for the source shard to delete migrated data before starting the next chunk migration. Note that this may slow down the rebalancing process.

use config
db.settings.update(
   { _id: "balancer" },
   { $set: { _waitForDelete: true } },
   { upsert: true }
)

Best Practices for Accuracy

• Use Aggregation: For 100% accuracy in sharded environments, use db.collection.aggregate([{ $match: ... }, { $count: "count" }]). This forces the engine to follow the same logic as a standard query, respecting shard versioning and metadata.
• Maintenance Windows: Schedule the balancer to run during off-peak hours to minimize the impact of migrations on reporting queries.
• Upgrade: Move to MongoDB 4.0 or higher to ensure that counts with predicates are handled correctly.