fix API v3 rate limiting which claims to be per API key
The API v3 rate limiting documentation (https://haveibeenpwned.com/API/v3#RateLimiting) initially claims that the API is rate-limited on a per-API key basis. Reading the fine print, it indicates that the rate limit is actually applied to the IP address. This disconnect leads to immense challenges in working with the API at scale. For example, I bought 7 API key licenses today so that I could work through a very large data set more quickly. However, all of my API keys are working from the same source IP address. So every time your API gets busy, you start blocking me by my IP address. With seven API keys all trying to do work, this invariably leads pretty quickly to a condition in which all of them are being told to Retry repeatedly. I'm currently waiting 2 seconds between my initial requests. And I'm tripling your "retry-after" suggestion before trying again. Doesn't matter. It always ends up with the same congestion. Whether by offering a "premium" tier or by letting me actually use the API keys that I've already paid for, I really need some help to make this thing scale.
Justin Lute
shared this idea
The API is rate limited per key at the Azure API Management level. There are no rate limits per IP address. Usually when I hear a report like this, it’s because someone is inadvertently making too many requests so I’d normally suggest changing the API key (you can do that on the page you registered on), then testing the new key totally independently of your code, for example in Postman.
Closing this “idea” as it’s not an idea, contact me directly if you still have problems: https://www.troyhunt.com/contact/
-
Justin Lute
commented
Thanks for the response, Troy. FYI, your documentation implies that even if it was your _API Key_ that was found to be violating the rate limit, it's your _IP address_ which will be denied access:
"...from any given HIBP API key... response header expressing the number of seconds remaining before the IP address can make a successful API call..."