Implementing Request Throttling
1. Configuring Concurrent Request Limits
| Setting | Purpose |
|---|---|
max_concurrent | In-flight cap per key |
queue_size | Pending allowed |
queue_timeout | Reject if waiting too long |
overload_action | 429 or shed |
2. Setting Up Queue-Based Throttling
Example: Bounded queue with timeout
Semaphore inflight = new Semaphore(100);
boolean acquired = inflight.tryAcquire(50, TimeUnit.MILLISECONDS);
if (!acquired) {
response.status(429).header("Retry-After", "1").end();
return;
}
try { proxy(request, response); } finally { inflight.release(); }3. Implementing Priority-Based Throttling
| Priority | Drop Order |
|---|---|
| Critical (paid) | Last to drop |
| Normal | Drop on heavy load |
| Bulk (batch) | Drop first |
| Health/admin | Reserved capacity |
4. Using Adaptive Throttling
| Signal | Action |
|---|---|
| p99 latency > SLO | Reduce concurrency |
| Error rate > 5% | Apply backoff |
| CPU > 80% | Shed lowest priority |
| Healthy | Gradually raise limits |
5. Configuring Backpressure Mechanisms
Client → [Gateway in:1000 rps]
↓ slows accept
[Worker pool full]
↓ TCP window shrinks
[Client OS sees slow ACK]
↓ Reactive client throttles
6. Setting Throttle Response Codes (429)
Example: 429 response
HTTP/1.1 429 Too Many Requests
Retry-After: 30
RateLimit-Limit: 60
RateLimit-Remaining: 0
RateLimit-Reset: 30
Content-Type: application/problem+json
{ "type":"about:blank", "title":"Rate limit exceeded", "status":429 }7. Implementing Quota Management
| Quota Type | Window |
|---|---|
| Daily | 24h rolling |
| Monthly | Calendar or 30d |
| Per-endpoint | Different by route |
| Soft vs hard | Warn vs block |
8. Configuring Throttle Retry Headers
| Header | Format |
|---|---|
Retry-After | Seconds or HTTP date |
X-RateLimit-Reset | Unix epoch |
X-Backoff | Recommended jitter ms |
9. Setting Up Client-Specific Limits
Example: Per-consumer override
consumers:
- username: enterprise-corp
plugins:
- name: rate-limiting
config: { minute: 100000 }
- username: free-tier-user
plugins:
- name: rate-limiting
config: { minute: 60 }10. Using Dynamic Throttling Rules
| Trigger | Rule Change |
|---|---|
| Incident | Lower all limits 50% |
| Black Friday | Raise premium limits 2x |
| Abuse pattern | Per-IP penalty box |
| Backend degradation | Auto-shed by tier |