How Rate Limiting Handles Traffic Bursts: Windows, Leaky Buckets, and Token Buckets

October 31, 2026 · 10 min read

If you have any questions, feel free to comment below. Click the block can copy the code.
And if you think it's helpful to you, just click on the ads which can support this site. Thanks!

Rate limiting controls the number or rate of requests admitted into a system, preventing traffic bursts from exhausting threads, connections, and database processing capacity.

1. What Is Rate Limiting? #

Suppose an endpoint can reliably handle 1000 requests per second, but suddenly receives 5000 requests per second. If it accepts all of them, the excess requests will occupy connections and wait in queues, eventually slowing down normal requests as well.

Rate limiting takes appropriate action when a configured threshold is exceeded:

• Reject immediately: Do not proceed to business processing; for example, return HTTP 429 Too Many Requests^[1].
• Queue and wait: First enter a bounded queue, then process requests at the permitted rate; reject requests when the queue is full or the wait times out.
• Apply separate limits to different entities: For example, 10 requests per second per user, 1000 requests per minute per merchant, or 500 requests per second for the entire endpoint.

“How many requests are allowed per second” and “how many requests may execute at the same time” are also different:

At the same rate of 100 requests per second, if processing time per request increases from 10ms to 1s, concurrent resource usage will rise significantly. Protecting slow endpoints may require limiting the number of requests executing concurrently in addition to the request rate.

Thresholds can be determined through load testing and monitoring of the actual endpoint. After applying rate limits, processing time, error rates, and downstream load should also be monitored.

2. Rate Limiting Methods #

2.1. Counter: Fixed Window #

Implementation #

Divide time into fixed-length windows^[3] and maintain a counter for each window. For example, allow at most 60 requests per minute:

[00:00, 01:00) → At most 60 requests
[01:00, 02:00) → At most 60 requests

When a request arrives, first determine which window it belongs to, then check whether the count can be incremented. When switching to a new window, the count starts from zero:

Atomic operation:
  current_window = floor(current_time / window_length)
  If current_window != saved_window:
    Save current_window
    admitted_count = 0
  If admitted_count >= limit:
    Reject
  admitted_count += 1
  Admit

The first request in a new window must also be counted; it must not be admitted after a reset without incrementing the count. In a concurrent implementation, switching windows, checking the limit, and incrementing the count must be protected together. Using AtomicInteger for the counter alone is not enough to ensure the correctness of the entire operation.

Bursts at Window Boundaries #

Assume there were no earlier requests:

1. At 59.9 seconds, 60 requests arrive. They all belong to the first window and are admitted.
2. At 60.1 seconds, another 60 requests arrive. The counter has been reset, so they are all admitted as well.
3. Neither window exceeds 60 requests per minute, but the two batches, only 0.2 seconds apart, admit a total of 120 requests.

Figure: The two batches are positioned proportionally to their actual times. A fixed window limits the total within each aligned window, not within any continuous one-minute interval.

2.2. Sliding Window #

Implementation #

A sliding window does not wait until the next whole minute to reset. Instead, its counting interval moves forward with the current time. For example, when evaluating a request at 60.1 seconds, it counts requests admitted during the preceding 60 seconds, within (0.1, 60.1].

The 60 requests admitted at 59.9 seconds therefore remain within the interval, so the new batch arriving at 60.1 seconds cannot all be admitted.

There are two common implementations:

Per-request logs can handle boundaries precisely, but storage and cleanup costs increase with request volume. Bucketed counters use fewer counters in exchange for approximate statistics, with a possible error of up to one bucket width at the window edges.

Figure: The six cells represent a counting granularity of 10 seconds per cell. The batch near the boundary is not immediately cleared just because the minute changes. Per-request logs use timestamps for decisions, while bucketed counters approximate the boundaries using buckets.

The sliding-log decision can be written as:

Atomic operation:
  Remove admitted records with timestamp <= current_time - window_length
  If record_count >= limit:
    Reject
  Add the timestamp of the current request
  Admit

Cleanup, counting, and insertion must execute together. Once the limit is reached, the request should be rejected, not merely logged as limit exceeded and then admitted anyway. A single-machine implementation can use a lock, while a distributed implementation can use a Redis Lua script^[4] for the atomic decision.

2.3. Leaky Bucket #

Implementation #

Treat requests as water entering a bucket. The bucket has a maximum capacity, and its outlet drains at a fixed rate:

• When a request arrives, accept it if the bucket is not full.
• Reject it if the bucket is full; requests cannot queue indefinitely.
• Release accepted requests to the backend at a fixed pace to smooth traffic bursts.

For example, let the capacity be 4 and the outlet release 2 requests per second. At time 0, six requests arrive simultaneously: four enter the queue and two are rejected. If dequeuing starts at 0.5 seconds, the four requests are released at 0.5, 1, 1.5, and 2 seconds, respectively.

Figure: The bucket holds requests waiting to be processed. Its capacity controls how many can wait, while its drain rate controls the pace at which they are released to the backend.

For a leaky-bucket limiter that maintains only the current water level, the amount already drained can be calculated from the elapsed time:

water_level = max(0, previous_water_level - drain_rate × elapsed_time)
If water_level + current_request_amount > bucket_capacity:
  Reject
Else:
  Increase water_level
  Accept

This water-level calculation only determines whether to accept a request; it does not automatically delay its execution. Achieving the steady dequeuing shown in the figure also requires a bounded queue and an actual scheduler. A leaky bucket suits backends that need a steady flow, but even accepted burst requests may have to wait.

2.4. Token Bucket #

Implementation #

Tokens are added to the bucket at a fixed rate and stop accumulating when the bucket is full. When a request arrives, it must first obtain tokens before it is allowed to execute:

• Refill rate r: The number of tokens added per second.
• Bucket capacity B: The maximum number of tokens that can accumulate, determining how much burst allowance can be saved.
• Request cost c: The number of tokens consumed by one request; it can be set to 1 for a regular endpoint.

Atomic operation:
  tokens = min(B, previous_tokens + r × elapsed_time)
  Update the timestamp
  If tokens < c:
    Reject
  tokens -= c
  Admit

There is no need to run a task every second to add tokens. The refill amount can be calculated from the elapsed time when a request arrives. Time units must be consistent. On a single machine, a monotonic clock can be used to calculate elapsed time so that system clock adjustments do not affect the refill amount.

Why It Allows Bursts #

Again, consider a capacity of 4 and a refill rate of 2 tokens per second, assuming the bucket is initially full:

1. At time 0, six requests arrive simultaneously. The first four immediately obtain tokens and are admitted; the remaining two are rejected.
2. After 0.5 seconds, one token has been replenished, allowing one newly arriving request to be admitted.
3. After enough idle time, the bucket recovers to at most four tokens; they do not accumulate indefinitely.

Figure: The bucket holds permits, not queued requests. A token bucket allows short bursts using saved allowances, while a leaky-bucket queue controls the output pace.

In a model with continuous replenishment and one token consumed per request, any interval lasting t seconds can use at most the B tokens available at its start plus the r × t tokens replenished during it. This limits the long-term rate without requiring exactly the same number of requests to be admitted every second.

3. Distributed Rate Limiting Across Multiple Instances #

If each gateway independently maintains a rate limit quota^[5] of 10 requests per second, deploying three gateways may give the same user a combined quota of 30 requests per second. Using the same Redis token bucket^[6] and consistent key rules allows multiple gateway instances to check the same bucket. A Redis script performs reading, replenishment, checking, and deduction together, preventing multiple instances from each reading the balance and admitting requests more than once against the same allowance.

When Redis is unavailable, decide in advance whether to reject requests, temporarily use a local quota, or admit requests and raise an alert. A local fallback quota is not the same as a shared quota. The number of 429 responses and Redis response times also need to be monitored so that the rate limit check itself does not slow down requests.

The diagrams in this article were created with logic2image^[7].

Related readings

• The Method to Manage Traffic