10,000 Requests
a Second

It is 11:40 PM on sale night. The war room at Utsav Mart is calmer than you would think. Utsav Mart is a made-up online shop, but the numbers in this story are close to what a mid-sized Indian e-commerce site sees, so use them as a guide and not as a benchmark. Twelve engineers sit in front of two big screens. Filter coffee is going cold on every desk. The home page shows a countdown, and the site is getting 1,200 requests a second. At 12:00:00 a phone that normally costs ₹79,999 goes on sale for ₹49,999. Within four seconds, traffic jumps to 10,000 requests a second.

Nobody is writing code tonight. Everything that decides whether the site survives was done in the weeks before: what to keep in cache, which requests to let in, which work to do now and which to do later, and how many servers to keep ready. This article goes through those decisions in the same order a request travels, from the internet edge down to the database. At the end, there is a table that maps each idea to other kinds of traffic you will face outside a sale.

10,000 requests a second sounds like a lot. If it ran all day, it would be about 86 crore requests — but it never runs all day. The real problem is the shape of the traffic. It stays flat for hours and then shoots straight up. A system that can handle 10,000 requests a second when traffic grows slowly can still fall over when all of them arrive in the same four seconds.

01 · THREE WEEKS BEFOREDo the maths first

The first meeting about sale night had no diagrams. It had one spreadsheet. Every decision later in this article depends on three simple numbers. Teams that skip this step usually spend money on the wrong layer.

What kind of requests are coming?

10,000 requests a second is never 10,000 of the same request. In Utsav Mart's last sale, about 90% were reads: people opening the home page, category pages and product pages, or checking price and stock. About 8% were cart actions. Only about 2% were checkout and payment. Those 2% bring in the money — and they are also the hardest to scale, because they write data. The 90% reads are easier, because the same answer can be copied, cached and served from somewhere closer to the user.

So the first rule is simple: treat reads and writes as two separate problems. At this volume, they behave very differently. Section 05 turns this idea into CQRS.

How many requests are open at the same time?

There is an easy formula here. Requests open at any moment = requests per second × time each request takes. At 10,000 requests a second and 200 ms (0.2 seconds) per request, you have 2,000 requests open at once. Each open request holds a thread, a network connection and maybe a database connection. If you bring the time down to 50 ms, the same traffic needs only 500 open requests.

That is why making the code faster and handling more traffic are the same job. Every millisecond you save means fewer things held open. One slow call to another service makes that user wait, and it also adds load to the whole system.

How much can one server handle?

The third number comes from testing one pod (one running copy of the service). Utsav Mart's catalogue API, on a pod with 2 CPU cores, could handle about 800 requests a second when the product came from cache. When it had to go to the Azure SQL database, the same pod managed only about 140. That is almost six times less. This one number tells you something important: how often you hit the cache decides your bill, much more than how many pods you run.

Try the funnel below. It follows 10,000 requests a second through all the layers in this article and shows how many actually reach your servers and your database.

Now press "Redis goes down" and look at the database line. Without the cache, a database built for a few hundred reads a second suddenly gets several thousand. This exact problem took Utsav Mart down two years ago — and it shaped half of this article.

Practise before the real day

Three weeks before the sale, the team ran a full practice using Azure Load Testing. They replayed the real mix of traffic and pushed it from 1,000 to 20,000 requests a second in five seconds. That is double the expected peak, because sale forecasts are almost always too low. The first run failed at 6,200 requests a second. CPU was fine. The servers had run out of outgoing connections to the payment gateway. Section 10 explains this. You find such problems in practice — or you find them at midnight.

When you test, do not look only at the average response time. An average of 90 ms can hide a small group of users who wait four seconds. Those users keep pressing refresh, and that doubles your traffic. Always check how long the slowest requests take, how many requests fail, and how close each layer is to its limit.

02 · 22:00 · THE EDGEAnswer the request before it reaches you

The cheapest request is the one your servers never see. Utsav Mart puts Azure Front Door in front of everything. Cloudflare, Akamai or CloudFront work the same way. Front Door has servers in many cities — Mumbai, Chennai, Delhi and more — so the user connects to a nearby one, and it can send back a saved copy of the response from there.

Static files are the easy part: images, JavaScript, fonts and CSS. Put a version number or hash in each file name and tell the CDN to keep them for a year. After that, they are not your problem. On sale night, product images alone were about 70% of all data sent — and none of it touched a pod.

The less obvious part is saving API responses at the edge. A category listing, a product description or the sale banner settings are the same for every user for at least a few seconds. If your API marks them as cacheable, the CDN handles the repeat requests:

Cache-Control: public, max-age=10, s-maxage=30, stale-while-revalidate=30
Vary: Accept-Encoding
ETag: "cat-phones-p1-v8812"

s-maxage=30 tells the CDN to keep the response for 30 seconds, while max-age=10 tells the browser to keep it for 10. Suppose 3,000 users a second open the phones category. With a 30-second CDN cache, your server gets about one request per CDN location every 30 seconds, instead of 90,000. Even a five-second cache on a busy listing cuts load massively, and no shopper notices a product list that is five seconds old. stale-while-revalidate lets the CDN (where supported) keep serving the old copy while it fetches a fresh one, so the moment of expiry does not create a sudden rush.

Keep the cache key clean

The cache key is what the CDN uses to decide whether two requests are "the same". If it includes too much, caching stops working and nothing warns you. A marketing tag like ?utm_source=whatsapp, a login cookie sent along to the server, or a Vary: Cookie header can turn one cacheable page into a million different ones. Set up the CDN to ignore marketing tags and cookies on public pages.

One more rule: anything personal must never go into the shared cache. The price is the same for everyone. "You have 2 items in your cart" is not. So split the page. The common part comes from the CDN, and the personal part comes from a small separate API call. This one change made about 85% of Utsav Mart's product page cacheable, up from zero.

The edge also protects you

Sale nights bring bots: people using scripts to grab stock, scrapers copying prices, and sometimes traffic that looks like an attack. The web application firewall (WAF) and bot rules in Front Door block this traffic before it reaches a pod. On sale night, about 6% of all requests were blocked here. That is the "blocked at gate" slider in the funnel above.

03 · 22:30 · THE GATELet in, slow down, or ask to wait

After the edge comes the gateway. Think of it as the security guard at a housing society gate. Utsav Mart uses Azure API Management for partner APIs and YARP inside the cluster for the shopping site. Both do the same job: decide which requests come in, how fast, and what happens to the rest.

Rate limits at two levels

An overall limit protects the whole system. It must sit somewhere that sees all the traffic, which means the edge or the gateway. In API Management, it is one policy:

<rate-limit-by-key calls="600" renewal-period="60"
    counter-key="@(context.Subscription?.Key ?? context.Request.IpAddress)" />

A per-endpoint limit stops any one user from hammering a costly operation. ASP.NET Core has built-in rate limiting (since .NET 7). The "token bucket" type works well for shoppers. Each user gets a small bucket of tokens. A quick burst, like opening five tabs, is allowed. After that, the bucket refills at a fixed speed:

builder.Services.AddRateLimiter(o =>
{
    o.RejectionStatusCode = StatusCodes.Status429TooManyRequests;

    o.AddPolicy("per-user", ctx =>
        RateLimitPartition.GetTokenBucketLimiter(
            partitionKey: ctx.User.FindFirst("sub")?.Value
                          ?? ctx.Connection.RemoteIpAddress?.ToString() ?? "anon",
            factory: _ => new TokenBucketRateLimiterOptions
            {
                TokenLimit          = 20,                     // burst
                TokensPerPeriod     = 10,                     // steady rate...
                ReplenishmentPeriod = TimeSpan.FromSeconds(1), // ...per second
                QueueLimit          = 0,
                AutoReplenishment   = true
            }));

    o.OnRejected = (ctx, _) =>
    {
        ctx.HttpContext.Response.Headers.RetryAfter = "2";
        return ValueTask.CompletedTask;
    };
});

app.UseRateLimiter();
app.MapPost("/api/cart/items", AddToCart).RequireRateLimiting("per-user");

Be careful about one thing. This limit is kept in each pod's own memory. With 24 pods, a user whose requests are spread across pods gets 24 buckets. That is fine for protecting a single pod from overload. If you need one strict limit per user across all pods, keep the count in Redis or apply it at the gateway.

When you reject a request, return status 429 with a Retry-After header. Make sure your own mobile app respects it and waits a little longer before each retry, with some randomness added. An app that retries immediately turns a small problem into a big one. An old version of Utsav Mart's app did exactly that. During one incident, 30% of all traffic was just retries — load the team created for itself.

Drop some load to save the rest

Rate limiting is about being fair to each user. Load shedding is about staying alive. When a pod already has too much work waiting, it is better to say "no" quickly than to accept the request and fail slowly. For example, a pod can allow 200 requests at a time and return 503 at once for anything more. This keeps it fast for the requests it does accept. Drop the less important traffic first. Recommendations and "customers also viewed" can fail. Add to cart and checkout should not.

A waiting room, like a railway queue

For the midnight deal itself — 2 lakh people, 5,000 phones — Utsav Mart puts checkout behind a virtual waiting room. It works like the Tatkal booking queue. Each user gets a token with a place in the line. The gateway lets people through at a speed checkout can handle, say 800 a second. Everyone else sees a waiting screen that comes straight from the CDN. It is less exciting than a free-for-all, but it turns an uncontrolled rush of orders into a steady flow. And customers prefer an honest queue to an error page.

04 · 22:45 · THE CACHEThree levels of cache

Requests that get past the edge and the gate reach the pods. Here, the goal is to answer from memory as often as possible, without going to the database. Utsav Mart uses three levels. Each one is closer to the code and smaller than the one below it.

• Output cache in ASP.NET Core saves the complete HTTP response. Your code does not run at all. It is useful for public endpoints that the CDN cannot cache, for example a partner API that needs a login header.
• Level 1: memory inside the pod. Extremely fast, but each pod has its own copy. It keeps a few thousand popular items for 10 to 60 seconds.
• Level 2: shared cache in Azure Cache for Redis. About one millisecond away, and shared by all pods. It keeps many more items, for minutes.

From .NET 9, HybridCache gives you both levels through one simple API. It also solves a problem that matters a lot at this volume. When a cached item expires and 500 requests ask for it at the same moment, only one of them goes to the database. The other 499 wait for that one answer and share it.

// Program.cs
builder.Services.AddStackExchangeRedisCache(o => o.Configuration = redisConnection);
builder.Services.AddHybridCache(o =>
{
    o.DefaultEntryOptions = new HybridCacheEntryOptions
    {
        Expiration           = TimeSpan.FromMinutes(5),  // L2 (Redis)
        LocalCacheExpiration = TimeSpan.FromSeconds(30)  // L1 (this pod)
    };
});

// ProductQueries.cs
public ValueTask<ProductDto> GetAsync(int id, CancellationToken ct) =>
    _cache.GetOrCreateAsync(
        $"product:{id}",
        async token => await _db.Products
            .AsNoTracking()
            .Where(p => p.Id == id)
            .Select(p => new ProductDto(p.Id, p.Name, p.Price, p.Rating))
            .FirstAsync(token),
        tags: [$"product:{id}", "catalog"],
        cancellationToken: ct);

// When the price changes (from an event handler):
await _cache.RemoveByTagAsync($"product:{id}");

See the difference below. There are 500 requests, all arriving the moment a popular product's cache entry expires.

Plan the expiry times

Two habits stop the cache from causing trouble. First, do not give every item the same expiry time. If you load 50,000 products into the cache at 11 PM with a 60-minute expiry, all 50,000 expire together at midnight — exactly when you need them most. Use a random time between 55 and 65 minutes for each item. Second, fill the cache before the sale starts. At 11:30 PM, Utsav Mart runs a job that loads every sale product into Redis. So the very first request at midnight finds the answer ready.

The one very popular item

At midnight, one product gets a huge share of all traffic: the ₹49,999 phone. In Redis, each item is stored on one server (called a shard), so one item means one shard does all the work — no matter how many shards you pay for. The fix is simple. Let the in-pod memory (level 1) take the load. With 30 seconds of in-pod caching across 24 pods, Redis sees about one request per pod every 30 seconds for that phone. If you still need it in Redis, keep a few copies under different names (product:88121:r0 to r7) and let each pod read a random one.

When the data changes

"How does the cache know that the price changed?" Your answer to this question decides how long you can cache. At Utsav Mart, the pricing service sends a PriceChanged event whenever a price changes. It uses the outbox pattern, so the event can never get lost. A listener then removes that product from the cache. The expiry time becomes only a backup in case an event is missed, so it can be minutes instead of seconds.

Stock is different. Showing "In stock" for a few seconds after the last piece is sold is acceptable, as long as checkout does the final check. Showing an old price is not acceptable. That is why price changes clear the cache immediately, while stock counts are allowed to be a little behind.

05 · 23:00 · CQRSKeep readers and writers apart

Remember the traffic mix: 90% reads, 10% writes, and the money comes from the writes. When both go through the same code and the same database, they get in each other's way. Reads want wide tables with everything in one place and many indexes. Writes want small, simple tables with few indexes, because every index slows down an insert. A product page wants the product, price, rating, seller name and delivery date in one go. The order service wants to add one order row and reduce the stock count in a short transaction.

CQRS (Command Query Responsibility Segregation) is a long name for a simple idea: give reading and writing their own separate models. Commands — PlaceOrder, AddToCart, UpdatePrice — go to the write side. It checks the business rules and saves to the main database. Then it sends out events. Small background services called projectors listen to these events and build "read models": ready-made copies of the data, shaped exactly the way each screen needs it.

What this gives you at 10,000 requests a second:

• Each side scales on its own. The read API runs 24 pods and the order service runs 8. Neither takes capacity from the other.
• The right storage for each kind of read. Product search and filters (brand, price range, RAM) go to Azure AI Search or Elasticsearch, which are made for this. Running such searches as LIKE queries with six optional filters on Azure SQL is a common way to kill a database. Order history, split by customer, goes to Cosmos DB.
• No joins while reading. The product card copy already has the seller name and rating inside it. One read is one simple lookup.
• A simple write path. Short transactions, few indexes, and only one place in the whole system that reduces stock.

There is a cost. The read copy is always slightly behind the main database. Usually the gap is a few milliseconds, but under heavy load it can be a few seconds. Design your screens with this in mind. For example, after a user places an order, show the confirmation using the order details you already have, instead of reading them back from the read copy. The order history page can catch up a second later.

You do not need a separate database on day one. An Azure SQL read replica (connect with ApplicationIntent=ReadOnly in the connection string), plus separate classes for reading and writing in your code, is a good first step. Full CQRS with events and projectors is worth the extra parts only when reads and writes are very different in volume and shape. For a small internal API with a hundred users, it is extra work you will regret.

06 · 23:15 · ASYNCSay yes now, do the work later

When a customer places an order at Utsav Mart, seven things need to happen: check the cart, reserve the stock, create the order, call the payment gateway, send an SMS, add loyalty points and inform the warehouse. If all of this happens inside one HTTP request, every order takes as long as the slowest step. And if any one step fails, the whole checkout fails.

Only two steps must finish before you reply to the customer: reserve the stock and save the order. Everything else can happen a few seconds later. So the checkout API does only these two, puts a message in a queue, and replies with 202 Accepted. This means "we have your order and we are working on it". The app can then check the order status using the link it gets back.

app.MapPost("/api/orders", async (
    PlaceOrderRequest req,
    [FromHeader(Name = "Idempotency-Key")] string idemKey,
    IDatabase redis, OrderWriter orders, CancellationToken ct) =>
{
    // A retry after a timeout must not create a second order.
    var orderId = Guid.NewGuid();
    bool first = await redis.StringSetAsync(
        $"idem:{idemKey}", orderId.ToString(), TimeSpan.FromHours(24), When.NotExists);

    if (!first)
    {
        var existing = await redis.StringGetAsync($"idem:{idemKey}");
        return Results.Accepted($"/api/orders/{existing}");
    }

    // Reserve stock atomically, then write order + outbox row in one transaction.
    if (!await orders.TryReserveAndRecordAsync(orderId, req, ct))
        return Results.Conflict(new { reason = "SoldOut" });

    return Results.Accepted($"/api/orders/{orderId}", new { orderId, status = "Processing" });
});

Next, the outbox sends an OrderPlaced message to Azure Service Bus. Separate background workers handle payment, SMS, loyalty points and the warehouse, each at its own speed. The queue works like a shock absorber. Orders arrive at 800 a second at midnight. If the payment workers can handle only 500, the queue grows for a few minutes and then empties. Nothing times out and nothing is lost.

Two safety rules here. The same message can sometimes arrive twice, so every worker must be safe to repeat: processing a message a second time should change nothing. And the customer's app may resend the order if the network drops. That is why the code above uses an Idempotency-Key, a unique ID the app sends with each order. If the same key comes again, the server returns the first order instead of creating a second one. Turn on duplicate detection in Service Bus as well.

Reducing stock without a long line

5,000 phones and 2 lakh buyers means everyone is fighting over one number. If you run UPDATE Stock SET Qty = Qty - 1 WHERE Sku = @sku AND Qty > 0 on one database row, every buyer stands in a single line behind that row's lock. Utsav Mart keeps the stock count of sale items in Redis instead, and reduces it with a small Lua script. Redis runs the whole script in one step, so two buyers can never get the same last piece:

-- KEYS[1] = stock:{sku}   ARGV[1] = qty
local left = tonumber(redis.call('GET', KEYS[1]) or '0')
if left >= tonumber(ARGV[1]) then
  return redis.call('DECRBY', KEYS[1], ARGV[1])
end
return -1

If payment does not finish within ten minutes, the reserved piece goes back to stock. A job runs every minute to compare the Redis numbers with the SQL database. Redis handles the rush; SQL stays the final record.

07 · 23:30 · THE CODEThe code review before the sale

Design decides how many requests reach your code. Your code decides how many pods you need to serve them. In the practice test, one API alone needed 22 pods. The code review that followed found the usual mistakes, all in one method:

Here is what was wrong with each line, in simple words. All five are common in .NET code.

• new HttpClient() on every call. Each call opens a new network connection and leaves the old one hanging for a while. At a few hundred calls a second, the pod runs out of connections. Use IHttpClientFactory with a typed client. It reuses connections safely.
• .Result and .Wait(). These make a thread sit and wait while data comes over the network. Under load, all threads end up waiting, new threads are added only slowly, and every request on that pod becomes slow, even requests that never touch this code. The typical sign: the pod is slow but CPU is only at 30%. The fix is to use async and await everywhere. Doing it only halfway does not work.
• Loading data you do not need. Include(p => p.Reviews) loaded thousands of review rows just to show a star rating. Select only the columns the response needs, use AsNoTracking() for read-only queries, and watch out for loops that fire one database query per item in a list.
• Blocking database calls like First instead of FirstAsync cause the same thread problem, only in the database layer.
• Logging on every request. Joining strings for a log message uses memory even when that log level is switched off. Source-generated [LoggerMessage] methods avoid this. Also, one log line per request at 10,000 requests a second is 86 crore log lines a day, and somebody pays to store them.

For calls to other services, the Microsoft.Extensions.Http.Resilience package (.NET 8 and later) adds timeouts, retries and a circuit breaker to a typed client in a few lines. A circuit breaker works like the MCB in your home: when too many calls fail, it stops calling that service for a while instead of making things worse.

builder.Services.AddHttpClient<PricingClient>(c => c.BaseAddress = new("http://pricing-api"))
    .AddStandardResilienceHandler(o =>
    {
        o.AttemptTimeout.Timeout      = TimeSpan.FromMilliseconds(300);
        o.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(1);
        o.Retry.MaxRetryAttempts      = 2;     // with exponential backoff + jitter
        o.CircuitBreaker.FailureRatio = 0.2;   // open at 20% failures
    });

These fixes took that one API from 140 to 780 requests a second per pod. But one method is only a start. The next section goes through the code-level techniques the team now uses everywhere, each with a small example.

08 · 23:35 · INSIDE THE CODEFourteen code-level techniques, with examples

The CDN, cache and Kubernetes decide how many requests reach your code. After that, everything depends on what your code does with each request. A pod that wastes 5 ms of CPU per request needs twice the pods of one that wastes 2.5 ms. Multiply that by 24 pods and a sale night — small code habits turn into a big Azure bill.

Each technique below has a short "slow" version and a "better" version. None of them needs a new framework. They are all standard .NET 8 to .NET 10 features.

1. Make independent calls at the same time

A product page needs four things: product details, price, stock and a delivery estimate for the user's PIN code. None of these depends on the other. If you call them one after another, the waiting times add up.

// Each call waits for the previous one: 40 + 60 + 35 + 50 = 185 ms
var product  = await _catalogue.GetAsync(id, ct);
var price    = await _pricing.GetAsync(id, ct);
var stock    = await _stock.GetAsync(id, ct);
var delivery = await _delivery.EstimateAsync(id, pin, ct);

// Start all four, then wait once: total time = slowest call = 60 ms
var productTask  = _catalogue.GetAsync(id, ct);
var priceTask    = _pricing.GetAsync(id, ct);
var stockTask    = _stock.GetAsync(id, ct);
var deliveryTask = _delivery.EstimateAsync(id, pin, ct);

await Task.WhenAll(productTask, priceTask, stockTask, deliveryTask);

return new ProductPage(productTask.Result, priceTask.Result,   // safe: already completed
                       stockTask.Result, deliveryTask.Result);

One warning: DbContext is not thread-safe. Do not run two queries on the same DbContext in parallel. Parallel calls are for different services, or for queries that each get their own context from IDbContextFactory.

2. Stop the "one query per row" loop

This mistake is easy to miss — the code looks innocent. You load 20 orders, then loop and load the items of each order. That is 1 + 20 = 21 trips to the database. For a customer with 200 orders, it is 201 trips — for one page.

var orders = await _db.Orders
    .Where(o => o.CustomerId == customerId)
    .ToListAsync(ct);

foreach (var order in orders)                        // 1 query + 1 per order
    order.Items = await _db.OrderItems
        .Where(i => i.OrderId == order.Id)
        .ToListAsync(ct);

// One query. Only the columns the screen shows. No change tracking.
var orders = await _db.Orders
    .AsNoTracking()
    .Where(o => o.CustomerId == customerId)
    .OrderByDescending(o => o.CreatedAt)
    .Take(20)
    .Select(o => new OrderDto(
        o.Id, o.CreatedAt, o.Total,
        o.Items.Select(i => new OrderItemDto(i.Sku, i.Name, i.Qty)).ToList()))
    .ToListAsync(ct);

The Select into a small DTO does two jobs. It brings back only the columns you need, and it lets EF Core build one SQL query with a join. Turn on EF Core's SQL logging in development and count the queries per request. If the number grows with the data, you have this problem.

3. Page by "last seen", not by page number

Skip and Take look cheap, but the database still has to read and throw away every skipped row. Page 1 is fast. Page 1,000 reads 20,000 rows to return 20. Bots and scrapers love deep pages, so on sale night this hurts.

var page = await _db.Products
    .OrderBy(p => p.Id)
    .Skip(pageNumber * 20)            // reads and discards all earlier rows
    .Take(20)
    .ToListAsync(ct);

var page = await _db.Products
    .AsNoTracking()
    .Where(p => p.Id > lastSeenId)    // jumps straight there using the index
    .OrderBy(p => p.Id)
    .Take(20)
    .ToListAsync(ct);
// Send lastSeenId back to the app as the "next page" token.

4. Reuse the DbContext and the query plan

Creating a DbContext and turning LINQ into SQL both take time. For queries that run thousands of times a second, you can skip both steps. DbContext pooling keeps ready contexts in a pool. A compiled query converts LINQ to SQL once, at startup.

builder.Services.AddDbContextPool<ShopDb>(o => o.UseSqlServer(connectionString), poolSize: 256);

public static class ProductQueries
{
    // LINQ is translated once and reused for every call
    public static readonly Func<ShopDb, int, Task<ProductDto?>> GetById =
        EF.CompileAsyncQuery((ShopDb db, int id) =>
            db.Products.AsNoTracking()
              .Where(p => p.Id == id)
              .Select(p => new ProductDto(p.Id, p.Name, p.Price, p.Rating))
              .FirstOrDefault());
}

// Usage
var product = await ProductQueries.GetById(_db, id);

Use compiled queries only for the few queries that run most often. For everything else, EF Core's own query cache is good enough.

5. Update many rows without loading them

At 11 PM, the team applies the sale price to 5,000 products. Loading 5,000 entities, changing them in memory and saving them sends thousands of UPDATE statements. From EF Core 7, ExecuteUpdateAsync sends a single SQL statement.

var items = await _db.Products.Where(p => p.SaleId == saleId).ToListAsync(ct);
foreach (var p in items) p.Price = p.Mrp * 0.6m;
await _db.SaveChangesAsync(ct);                     // 5,000 rows loaded, 5,000 updates sent

await _db.Products
    .Where(p => p.SaleId == saleId)
    .ExecuteUpdateAsync(s => s.SetProperty(p => p.Price, p => p.Mrp * 0.6m), ct);   // one UPDATE

ExecuteDeleteAsync does the same for deletes. Remember that these skip the change tracker, so clear the cache (or publish a PriceChanged event) after running them.

6. Return cached answers without extra work

When most calls are answered from memory, even creating a Task object on each call adds up. ValueTask lets a method return a ready value with no extra allocation, and fall back to real async work only on a cache miss.

public ValueTask<decimal> GetPriceAsync(int productId, CancellationToken ct)
{
    if (_memory.TryGetValue(productId, out decimal price))
        return ValueTask.FromResult(price);                // cache hit: nothing allocated

    return new ValueTask<decimal>(LoadAndCacheAsync(productId, ct)); // cache miss: normal async
}

A ValueTask must be awaited only once. If you are not sure, keep using Task. Use this only on hot paths where a profiler shows allocations as a problem.

7. Read strings without cutting them into pieces

The cart API receives items as "88121:2" (product ID and quantity). Split creates a new array and two new strings every time. At thousands of calls a second, this is garbage the GC has to clean up. ReadOnlySpan<char> reads the same text in place, with no new objects.

var parts = input.Split(':');                // new array + 2 new strings
int productId = int.Parse(parts[0]);
int qty       = int.Parse(parts[1]);

ReadOnlySpan<char> s = input;
int colon     = s.IndexOf(':');
int productId = int.Parse(s[..colon]);       // no new strings created
int qty       = int.Parse(s[(colon + 1)..]);

8. Do not use exceptions for normal cases

Throwing and catching an exception is expensive. It is meant for real errors, not for "the user typed a wrong number". A bad quantity in a URL is an everyday thing, so check it the cheap way.

int qty;
try { qty = int.Parse(qtyText); }
catch (FormatException) { qty = 1; }         // exception on every bad input

if (!int.TryParse(qtyText, out int qty) || qty < 1)
    qty = 1;                                 // no exception, same result

The same idea applies to your own code. For "product not found" or "sold out", return a result (Results.NotFound(), a bool, or a result object) instead of throwing.

9. Make fixed lookup tables faster

Some data almost never changes while the app runs: delivery zones by PIN code prefix, GST slabs by category, feature settings. .NET 8 added FrozenDictionary. You build it once at startup, and after that it is read-only and faster to look up than a normal Dictionary.

public sealed class DeliveryZones
{
    private readonly FrozenDictionary<string, int> _daysByPinPrefix;

    public DeliveryZones(Dictionary<string, int> fromDatabase) =>
        _daysByPinPrefix = fromDatabase.ToFrozenDictionary();   // built once at startup

    public int EstimateDays(string pin) =>
        _daysByPinPrefix.TryGetValue(pin[..3], out int days) ? days : 5;
}

Register this class as a singleton. Loading the same table from the database on every request is one of the most common hidden costs in APIs.

10. Make JSON cheaper

Every response is turned into JSON, so small savings here repeat 10,000 times a second. Two changes help. First, System.Text.Json source generation creates the serialisation code at build time, instead of working it out with reflection at runtime. Second, for large results, stream the response with IAsyncEnumerable, so the server sends rows as they come instead of holding the whole list in memory.

[JsonSerializable(typeof(ProductDto))]
[JsonSerializable(typeof(List<ProductDto>))]
internal partial class AppJsonContext : JsonSerializerContext { }

builder.Services.ConfigureHttpJsonOptions(o =>
    o.SerializerOptions.TypeInfoResolverChain.Insert(0, AppJsonContext.Default));

// Stream a big export row by row instead of building one huge list
app.MapGet("/api/orders/export", (ShopDb db) =>
    db.Orders.AsNoTracking()
      .Select(o => new OrderDto(o.Id, o.CreatedAt, o.Total, null))
      .AsAsyncEnumerable());

Also send only what the screen needs. A product card with 8 fields instead of the full 60-field entity is smaller to build, smaller to send, and faster on a 4G phone.

11. Give every call a deadline, and pass the token along

When a user closes the app, ASP.NET Core cancels the request's CancellationToken. If you pass that token to EF Core and HttpClient, the work stops too. If you forget it, the server keeps working — for someone who has already left. You can also add your own short deadline for optional parts of the page.

// Reviews are nice to have. Give them 300 ms, then show the page without them.
using var cts = CancellationTokenSource.CreateLinkedTokenSource(ct);
cts.CancelAfter(TimeSpan.FromMilliseconds(300));

ReviewSummary? reviews = null;
try
{
    reviews = await _reviews.GetSummaryAsync(productId, cts.Token);
}
catch (OperationCanceledException) when (!ct.IsCancellationRequested)
{
    // Took too long. Page still loads, just without reviews.
}

12. Collect small writes and save them in batches

Every product view creates an analytics event. Writing each one to the database inside the request adds a database call to the busiest path in the system. Instead, put the event in an in-memory Channel and return at once. A background service reads the channel and saves 500 events at a time.

// Registered as a singleton. Holds at most 10,000 events; drops the oldest if full.
public sealed class ClickTracker
{
    private readonly Channel<ClickEvent> _channel = Channel.CreateBounded<ClickEvent>(
        new BoundedChannelOptions(10_000) { FullMode = BoundedChannelFullMode.DropOldest });

    public void Track(ClickEvent e) => _channel.Writer.TryWrite(e);   // takes microseconds
    public ChannelReader<ClickEvent> Reader => _channel.Reader;
}

public sealed class ClickWriter(ClickTracker tracker, IDbContextFactory<ShopDb> dbFactory) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        var batch = new List<ClickEvent>(500);
        while (await tracker.Reader.WaitToReadAsync(ct))
        {
            while (batch.Count < 500 && tracker.Reader.TryRead(out var e)) batch.Add(e);

            await using var db = await dbFactory.CreateDbContextAsync(ct);
            db.Clicks.AddRange(batch);
            await db.SaveChangesAsync(ct);          // one round trip for up to 500 rows
            batch.Clear();
        }
    }
}

Use this only for data you can afford to lose if a pod crashes, like analytics. Orders and payments must go through the outbox and a durable queue, as shown in section 06.

13. Log cheaply on busy paths

Logging looks free — but string joining and boxing happen even when the log level is off. The [LoggerMessage] source generator creates logging code that checks the level first and does not allocate. Keep per-request logs at Debug and use metrics for counts.

_logger.LogInformation("Served product " + product.Id + " in " + ms + " ms");   // builds a string every time

public static partial class Log
{
    [LoggerMessage(Level = LogLevel.Debug, Message = "Served product {ProductId} in {Ms} ms")]
    public static partial void ProductServed(this ILogger logger, int productId, long ms);
}

_logger.ProductServed(product.Id, ms);      // no work at all when Debug is off

14. Prove it with a benchmark

Do not trust a technique just because an article (including this one) says so. BenchmarkDotNet measures the time and memory of two versions of the same code side by side. Run it before you change a hot path, and keep the benchmark in the repo so the next person can run it too.

[MemoryDiagnoser]                               // also reports memory allocated per call
public class ParseBench
{
    private const string Input = "88121:2";

    [Benchmark(Baseline = true)]
    public int WithSplit()
    {
        var parts = Input.Split(':');
        return int.Parse(parts[0]) + int.Parse(parts[1]);
    }

    [Benchmark]
    public int WithSpan()
    {
        ReadOnlySpan<char> s = Input;
        int i = s.IndexOf(':');
        return int.Parse(s[..i]) + int.Parse(s[(i + 1)..]);
    }
}
// Run in Release mode: dotnet run -c Release

The report shows mean time and bytes allocated for each method. In this case the span version allocates nothing, while Split allocates an array and two strings on every call. For real endpoints, pair this with dotnet-counters (live GC and thread numbers) and dotnet-trace during the practice load test.

Quick reference

Not every technique is worth it everywhere. Use this table to decide where to start:

Taken together, these are the difference between an API that needs 22 pods and one that needs 4 for the same traffic. None of them is clever. They are small habits, applied on the paths that run 10,000 times a second.

09 · 23:40 · THE DATABASEThe last bottleneck

You can add pods in a minute. You cannot do that with a relational database. Everything before this section exists to keep the database's share of traffic small. What still reaches it needs care.

• Indexes for the queries you actually run. After the practice test, take the top 20 queries by CPU from Query Store and check that each one has the right index. One missing index on Orders(CustomerId, CreatedAt) was using 40% of Utsav Mart's database CPU.
• Read replicas for queries that can be a little behind. Azure SQL Premium, Business Critical and Hyperscale support this. Connect with ApplicationIntent=ReadOnly.
• Count your connections. Forty pods across all services, each with a default connection pool of 100, can open 4,000 database connections. Check this against the limit of your database tier before sale night checks it for you. A small pool with fast queries is better than a big pool with slow ones.
• Keep transactions short. Never call an outside service inside a transaction. If you call the payment gateway inside one, the rows stay locked for as long as the gateway takes to reply.
• Write in batches where the business allows. Loyalty points and analytics can be saved 500 at a time by a background worker, instead of one insert per request.
• Split the data when one database is not enough. In Cosmos DB, the partition key decides everything. /customerId spreads order history evenly. /saleId would put every midnight order in one place and overload it. In SQL, split by region or by customer group before trying anything clever.

10 · 23:50 · THE PLATFORMAdd capacity before the rush, not during it

Utsav Mart runs on AKS (Azure Kubernetes Service), with two clusters, one in Central India and one in South India, both behind Front Door. Autoscaling sounds like the easy part. At midnight, it is the part most likely to fail you. The reason is simple: traffic arrives in four seconds, but a new machine takes three to five minutes to start.

Scale by the clock, then by the traffic

The normal Kubernetes autoscaler watches CPU over a few minutes. So it starts adding pods only after the rush has already hurt you. New machines are added only after pods have nowhere to run. For an event whose time you already know, schedule the capacity in advance. KEDA's cron trigger raises the minimum number of pods before midnight, and a second trigger adds more if traffic goes beyond the forecast:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: catalog-api
spec:
  scaleTargetRef:
    name: catalog-api
  minReplicaCount: 6
  maxReplicaCount: 80
  triggers:
    - type: cron                    # floor of 24 pods from 23:00 to 02:00 IST
      metadata:
        timezone: Asia/Kolkata
        start: "0 23 * * *"
        end: "0 2 * * *"
        desiredReplicas: "24"
    - type: prometheus              # beyond the forecast: ~400 req/s per pod
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: sum(rate(http_server_request_duration_seconds_count{app="catalog-api"}[1m]))
        threshold: "400"

The order workers scale on queue length using KEDA's azure-servicebus trigger. They grow when orders pile up and go back to two when the queue is empty. Machines are also added in advance: the node pool's minimum size is raised at 10 PM so the extra pods have somewhere to run.

Pods that are really ready

• "Ready" must mean ready. If a pod says it is ready before its connections and caches are warmed up, it gets full traffic while still cold, and its first thousand requests are slow. Use a startup probe while the app boots, and let the readiness probe pass only after a warm-up step has run the main APIs once. ReadyToRun compilation also makes startup faster.
• Be careful with CPU limits. A CPU limit can make Kubernetes pause the container in short bursts. It shows up as sudden slow requests on a pod that looks idle on average. Many teams set CPU requests properly and leave CPU limits off, or set them high. Keep memory limits; .NET respects them.
• Check garbage collection. Look at GC pause times in dotnet-counters during the practice test, instead of changing GC settings blindly.
• Spread out and stay safe. Spread pods across availability zones, add a PodDisruptionBudget so a node upgrade cannot remove half the pods at 11:58 PM, and stop all deployments from 8 PM.

The problem that was not about CPU

Remember the practice test that failed at 6,200 requests a second? Calls from AKS to the payment gateway were going out through the cluster's load balancer. That load balancer gives each machine only a fixed number of outgoing ports (Azure calls this SNAT). Thousands of short connections used them all up, and new connections just waited until they timed out. The team made two changes. They added a NAT Gateway on the subnet, which gives 64,512 ports per public IP, and you can attach more IPs. And they reused connections through IHttpClientFactory, so the same ports carried many requests. If your pods call anything outside Azure at high volume, check this before sale night. The sign is random connection timeouts while CPU looks healthy everywhere.

Two regions

Front Door sends each user to the nearest healthy cluster. Reads are served from both regions, because Redis and the read copies exist in both. Writes go to the main database in one region, with a copy ready in the other. Writing in both regions at once is possible, but then you have to handle conflicts — and sale night is not the time to learn that. The second region earns its cost by taking half the read traffic, and by being ready if a whole zone goes down.

11 · 23:55 · PLAN BDecide in advance how you will fail

Something will break at midnight. The real question is what the customer sees when it does. Utsav Mart wrote down a step-by-step fallback plan in advance. Each step has a switch (a feature flag in Azure App Configuration), so the person in charge can turn things off in seconds, without a new deployment:

• Step 1: switch off personal recommendations and show a fixed "Top deals" list from the CDN.
• Step 2: hide review text and show only the star rating.
• Step 3: serve category pages from a ready-made static copy created at 11 PM.
• Step 4: let fewer people into the waiting room, so checkout stays fast for those already inside.

Circuit breakers do a similar job automatically. When the reviews service fails 20% of the time, its circuit breaker opens and the product page loads without reviews, instead of waiting for a dead service. Also give each outside service its own limited set of connections. Then one slow service cannot use up every thread and connection in the pod. The golden rule: checkout must never depend on anything that is not needed to take the money and reserve the stock.

12 · 00:00 · MIDNIGHTWhat the war room watches

At midnight, the dashboards matter more than the code. Utsav Mart's big screen shows four things for every service: how many requests are coming in, how many are failing, how long the slowest requests take, and how close each part is to its limit (threads waiting, Redis load, SQL CPU, and how many messages are waiting in Service Bus). All of it comes from OpenTelemetry into Azure Monitor and Grafana. When requests slow down, traces in Application Insights show which service in the chain is slow. I wrote about following one such trace in The Case of the Vanishing Request.

Alerts are set on what the customer feels. The checkout target is simple: 999 out of every 1,000 checkout requests must finish in under 800 ms. If checkout starts missing that target much faster than normal, someone's phone rings. Everything else can wait till morning.

13 · THE NEXT MORNINGWhich solution for which kind of traffic

A festive sale uses almost every trick at once. Most real systems need only some of them, and picking the wrong ones costs money without adding capacity. Here is a simple guide for the next time someone says "we need to handle 10K requests a second":

See how often the answer is "do less work per request" or "do the work somewhere cheaper", rather than "add more servers". Adding pods should be the last step, because every new pod also adds database connections, Redis connections and outgoing ports. Scale out only after the layers above have done their share.

14 · BEFORE YOUR OWN MIDNIGHTThe pre-sale checklist

Utsav Mart's team goes through this list one week before every big event. Tick it off for your own system. Your ticks stay only in this browser tab.

The key takeaway

Handling 10,000 requests a second is mostly about taking work away. The CDN removes the repeat reads. The gateway removes bots and extra requests. The cache removes most of what is left. CQRS keeps the remaining reads away from the writes, and the queue turns a sudden rush of orders into a steady flow. By the time a request reaches the database, it is one of a few hundred — and the database can give it full attention.

Good code decides how cheaply each layer does its job. Good platform setup makes sure the capacity is already running when the rush comes. None of this is exotic. But all of it has to be done before the countdown starts, because at 12:00:00 the war room can do only one thing — watch.