TokenBucket
A bucket holds tokens, refilled at a steady rate; each request consumes a token, and requests are rejected/queued when the bucket is empty — allows short bursts up to the bucket size while enforcing a long-run average rate.
Why we need this / what value this brings
Lets legitimate bursty usage (a user quickly retrying a few times) through without being overly strict, while still bounding the long-run average rate.
When to use this
General-purpose API rate limiting where some burst tolerance is desirable — a common default choice.
How to use or implement this
Track tokens and last-refill-time per client key (user ID, API key, IP); refill proportionally to elapsed time on each check rather than running a separate timer.
Research questions
- Token bucket is the standard choice for APIs that should tolerate brief bursts (a user clicking retry a few times) without being that strict — is that the right fit for Localz's public API?
- At fleet scale (1000s of devices' diffs hitting an Enterprise LLM Gateway), fan-out without a rate limiter turns into a self-inflicted DoS against the gateway/model — the concurrency cap referenced from DataFlowPatterns/FanOut is essentially a token bucket in front of the LLM calls.
Empty folder — drop notes, links, and findings here as you research.