Animall is a marketplace for rural India’s cattle trade: e-commerce, but the listings are cows and buffaloes, and the buyers are farmers scrolling on their phones. The discovery feed is where most sessions begin, so its speed matters more than almost anything else in the app.

When I took it on, the feed took around 2,000 ms to load. I brought it under 200 ms. Most of the gain came from moving one piece of work: the scoring that ranks the feed moved out of our Node.js service and into MongoDB’s aggregation pipeline.

What the feed had to rank

A good feed for a farmer depends on many signals at once:

  • The animal: breed, age, and milking capacity
  • The deal: the quoted price
  • The distance: how far the listing is from the viewer, because cattle are transported by road
  • Freshness: when the listing was uploaded
  • and a long tail of other variables

Every request had to weigh these across candidate sets in the region of 100K listings. The exact formula was proprietary, but the shape of the problem is common to every marketplace feed.

Before: the app fetched candidates and ranked them itself. After: MongoDB filtered, scored, sorted and returned one page.
Before: the app fetched candidates and ranked them itself. After: MongoDB filtered, scored, sorted and returned one page.

Where the time went

The slow version followed a common pattern:

  1. Query the database for the candidate listings.
  2. Send them all over the network to the application server.
  3. Compute a score for each one in JavaScript.
  4. Sort them in memory and return the top page.

Every step scales with the size of the candidate set, not with the page the user sees. With around 100K candidates, the service paid for network transfer, document deserialization, a scoring loop and an in-memory sort, and then threw almost all of the documents away to show one page.

The fix: rank where the data lives

MongoDB’s aggregation pipeline can filter, compute, sort and trim inside the database engine. Its arithmetic operators ($add, $multiply, $divide, $subtract) are enough to express a weighted ranking score, so the whole ranking step can run next to the data. Only the final page crosses the network.

A simplified version of the shape. The real signals, weights and normalizations were proprietary and far more numerous:

db.listings.aggregate([
  // 1. Filter first, on indexed fields, so later stages touch fewer documents
  { $match: { status: "active", breed: { $in: preferredBreeds },
              price: { $lte: maxPrice }, region: { $in: nearbyRegions } } },

  // 2. Compute the ranking score inside the engine
  { $addFields: {
      listingAgeHours: { $divide: [{ $subtract: ["$$NOW", "$uploadedAt"] }, 3600000] }
  } },
  { $addFields: {
      score: { $add: [
        { $multiply: ["$milkYieldScore", wMilk] },
        { $multiply: ["$breedScore", wBreed] },
        { $multiply: ["$animalAgeScore", wAnimalAge] },
        { $multiply: ["$priceScore", wPrice] },
        { $multiply: ["$distanceScore", wDistance] },
        { $multiply: [{ $divide: [1, { $add: ["$listingAgeHours", 2] }] }, wFresh] }
      ] }
  } },

  // 3. Sort and keep only one page
  { $sort: { score: -1, _id: -1 } },
  { $skip: page * pageSize },
  { $limit: pageSize },

  // 4. Return only the fields the feed card renders
  { $project: { breed: 1, price: 1, media: 1, location: 1, score: 1 } }
])

What made it fast

  • Filter before computing, on indexes. $match comes first. A mix of single-field and compound indexes, designed around the feed’s real filter combinations, let MongoDB discard most of the ~100K candidates before any arithmetic ran.
  • Return one page, not the candidate set. When $sort is followed by $limit, MongoDB keeps only the top results while sorting instead of sorting everything.
  • Project small. The response carries only what the card shows, which keeps both the network payload and the parsing cost low.

What to watch

  • Memory for large sorts. A blocking $sort has a memory limit per stage. Know your candidate-set size, and check whether you need allowDiskUse.
  • Deep pagination. $skip gets slower the deeper a user scrolls. For long feeds, switch to keyset pagination on (score, _id).
  • Ranking logic now lives in a query. Keep the pipeline in version control next to the code, and test it with fixture documents so a weight change can’t silently reorder the feed.

Takeaway

Before optimizing an algorithm, look at where the data crosses a boundary. The ranking idea stayed the same. Moving where it ran made the feed roughly ten times faster.