Large Collection Problems

On a laptop dataset, everything is in RAM and COLLSCAN looks fine. At millions of orders the working set — documents and indexes you actually touch — must fit in the WiredTiger cache or every read becomes disk. That is when skip, unbounded find, and 'just $lookup users' fall over together.

• Pagination. skip(n) walks n keys every time — Module 5. Cursor + _id tiebreaker, with an index that matches the sort.
• Counts. countDocuments on a fat filter is another heavy query. estimatedDocumentCount is metadata. APIs that need 'page 4 of 80,221' pay for it; feeds should not.
• Materialising. .toArray() on an unbounded cursor in Node pulls the collection into the process. Stream, or limit.
• Indexes bigger than RAM. A 20-index orders collection that does not fit is slower than fewer indexes that do. $indexStats, drop.
• Joins. $lookup per document against a huge foreign collection needs an index on the foreign field and a small left side ($match/$limit first).

Sharding is how you split a collection you cannot host on one replica set — Production, not a first fix. First fixes: smaller working set, cursor pagination, fewer indexes, no collection scan on the hot path. Prove each with explain on production-shaped data, not on 42 users.