A 429 means "you are going too fast, wait". The right handling is to wait for as long as the server asks, retry a limited number of times with growing delays, and never retry things that will not get better. Then fail loudly if it still does not work.
Respect Retry-After first
Many APIs send a Retry-After header with the number of seconds to wait. Use it. Guessing a shorter delay just earns more 429s, and some APIs penalise that.
Exponential backoff with jitter
If there is no header, wait 1 second, then 2, 4, 8, up to a maximum, and add a random amount so that many workers do not all retry at the same instant.
import random, time, requests
def get_with_retry(url, max_tries=6):
for attempt in range(max_tries):
resp = requests.get(url, timeout=30)
if resp.status_code == 429 or resp.status_code >= 500:
wait = float(resp.headers.get("Retry-After", 2 ** attempt))
time.sleep(wait + random.uniform(0, 1))
continue
resp.raise_for_status() # 400, 401, 404 raise at once
return resp.json()
raise RuntimeError(f"gave up after {max_tries} tries: {url}")What to retry, and what not
Retry the transient failures: 429, 5xx errors and timeouts. Do not retry 400 (bad request), 401 or 403 (credentials), or 404, because repeating them gives the same answer and just burns your quota. Only retry requests that are safe to repeat. A GET is safe. A POST that creates something needs an idempotency key, or you may create it twice.
Cap it
Limit the number of tries and the total waiting time. An endless retry loop hides a real outage and can keep a job running for hours.
Be polite from the start
Better than recovering from 429s is not triggering them. Add a client-side throttle, such as a token bucket that allows N calls per second across all your workers, and reduce concurrency when you start to see 429s.
Libraries
tenacity gives you decorators for retry, backoff and stop rules. requests can use urllib3's Retry through an HTTPAdapter, with status_forcelist=[429, 500, 502, 503, 504] and backoff_factor. Using a tested library beats writing your own loop in every script.
When retries run out
Log which request failed and why, and make the task fail so the orchestrator marks it and alerts someone. Do not swallow the error and continue with a partial extraction, because downstream tables would silently miss data.