Internal API serves feature lookups to several ML services, one of which occasionally sends bursts that degrade latency for everyone else. Where do you actually put the rate limiter, API gateway, application middleware, or the database layer?
Accepted answer
As close to the edge as possible, API gateway or a dedicated middleware layer, using a token bucket per consumer (API key or service identity), so the abusive burst gets rejected before it consumes application or database resources at all.
Plain English: the system prefers to guess a good-enough plan than the perfect plan, because figuring out the perfect plan would take longer than just running the good-enough one.
Adding a per-service token bucket at the gateway, sized to each service's normal traffic plus headroom. Database connection limits stay as the safety net.
In our case the root cause was an implicit cast preventing pushdown.
Tried that, partial improvement but lag still spikes on redeploy.
In our case the root cause was an implicit cast preventing pushdown.
In simple terms: your job is asking for way more data than it actually needs, and that extra data has to travel across the network or between machines, which is slow. Ask for less, or ask smarter.
Worth measuring the serialized size before choosing broadcast.
Sign in to reply.
© 2026 Lakebench, operated by Hunnurji Rao. Bengaluru, Karnataka, India.
No cluster. No install. Just the tab.
