Design a Rate Limiter — System Design Interview Walkthrough

A rate limiter protects a service from being overwhelmed — by abusive clients, buggy retries, or thundering herds — by capping how many requests a caller can make in a window. "Design a rate limiter" is a compact interview favorite because it packs algorithms, distributed state, and failure trade-offs into one problem. Here's the full walkthrough.

Step 1 — Clarify requirements

Functional

Limit requests per client (by user ID, API key, or IP) — e.g., 100 requests/minute.
Support different limits per endpoint or tier (free vs. paid).
When the limit is exceeded, reject with HTTP 429 Too Many Requests (ideally with a Retry-After header).

Non-functional

Low latency — the limiter sits on every request, so it must add minimal overhead.
Accurate enough — small over/under-counting at the boundary is usually acceptable; perfectly exact limiting is more expensive.
Distributed — many API servers must share one view of a client's usage.
Fault-tolerant — decide what happens if the limiter's datastore is down.

Step 2 — The algorithms (this is the core)

Interviewers want you to compare a few and pick with reasons.

Fixed window counter. Count requests per fixed clock window (e.g., per minute) using a counter that resets each window. Simple and cheap (INCR + EXPIRE). Flaw: boundary bursts — a client can send the full limit at 0:59 and again at 1:00, doubling the intended rate across the boundary.

Sliding window log. Store a timestamp for every request; count how many fall in the trailing window. Exact, no boundary problem — but memory-heavy (one entry per request) and costlier to compute. Good when precision matters and volume is modest.

Sliding window counter. A practical hybrid: keep per-window counts and weight the previous window by how much of it overlaps the trailing period. Approximates the sliding log with O(1) memory. This is the common production choice — smooths the boundary burst without storing every request.

Token bucket. A bucket holds up to N tokens and refills at a steady rate; each request consumes a token, and an empty bucket means reject. Allows controlled bursts (up to the bucket size) while enforcing a long-run average — which is often what you actually want. Very popular.

Leaky bucket. Requests enter a fixed-size queue drained at a constant rate; overflow is dropped. Smooths output to a steady rate (good for shaping traffic to a downstream), but adds queueing and doesn't allow bursts.

A strong answer: "I'd use token bucket for a general API limiter — it allows short bursts while capping the average — or a sliding window counter if I want strict per-window limits without the boundary burst." Naming why beats reciting all five.

Step 3 — Where does the state live?

On a single server, an in-memory counter works. But real APIs run many servers behind a load balancer, and a client's requests hit different servers — so per-server counters would let a client multiply their limit by the server count. You need shared state.

The standard answer: a centralized in-memory store like Redis, keyed by client + window. Every API server reads/updates the same counter. Redis is fast enough to keep the added latency small, and it has TTLs built in for window expiry.

Step 4 — The atomicity problem

Naively, "read counter, check limit, increment" is a race: two servers can both read 99, both allow, and both write 100 — over-admitting. Fixes:

Use atomic operations: INCR returns the new value atomically, paired with EXPIRE to reset the window.
For multi-step logic (token bucket refill + consume), run it as a Lua script in Redis so the whole check-and-decrement is atomic.

Calling out this race and solving it with atomic ops / a Lua script is a senior signal.

Step 5 — Distributed challenges and performance

Latency: a network round-trip to Redis on every request adds up. Mitigate with a local per-server allowance (approximate token bucket) synced periodically, accepting slightly looser limits for lower latency.
Hot keys: a single very active client hammers one Redis key. Usually fine, but extreme cases may need sharding the counter.
Clock skew: windows should be computed from a consistent time source; don't rely on each server's local clock for boundaries.

Step 6 — Failure handling: fail-open vs. fail-closed

What if the rate-limiter store (Redis) is unavailable?

Fail-open: allow requests through when the limiter is down — prioritizes availability, risks overload during an outage.
Fail-closed: reject requests when the limiter is down — protects the backend but can turn a limiter outage into a full outage.

There's no universal right answer — it depends on whether the limiter is protecting a fragile backend (lean fail-closed) or is a nice-to-have guardrail (lean fail-open). Stating the trade-off and picking for the context is the point.

Step 7 — Follow-ups interviewers love

"How do you communicate the limit to clients?" Return 429 with Retry-After and X-RateLimit-Remaining/Reset headers.
"Different limits per tier?" Look up the client's plan and apply the matching limit config; keep limits as data, not code.
"Where does the limiter run?" Often at the API gateway/edge (before requests hit your services), sometimes as a shared library/sidecar.
"How do you avoid punishing legitimate bursts?" Token bucket's burst allowance, or a separate higher short-term limit.

Common mistakes

Only knowing fixed-window and missing the boundary-burst flaw.
Ignoring shared state — per-server counters silently multiply the limit.
Missing the race condition — non-atomic read-modify-write over-admits.
Not addressing the datastore-down case — fail-open vs. fail-closed is a required trade-off.

Practice this out loud

The rate limiter is deceptively deep — the algorithm comparison and the atomicity/failure trade-offs are where it gets interesting, and an interviewer will push on both. On Whitepad, a distributed rate limiter is a preset problem: a senior AI interviewer runs it by voice, watches your whiteboard, and probes your algorithm choice, shared state, and fail-open/closed reasoning — then scores you. Talk it through a few times and the trade-offs become reflexes.