Registry and DDoS attacks, a follow up

Notifications
#1

As a result of the DDoS attacks we have had to put in place rate limits on the registry. We apologise for any service disruption caused, we have been working with the vendor to try to come to a workable solution.

As we don’t yet have space for static pages on the registry site to present a change log we are posting the details here:

Current Configuration of IATI Registry NGINX Bot Blocker

For Normal Users Rate Limits (per IP):
20 requests per second
Queuing system for additional requests – 200 requests
If the request is more than this gives 503 error

Code:
limit_req_zone $binary_remote_addr zone=flood:50m rate=20r/s;
limit_conn_zone $binary_remote_addr zone=addr:50m;
limit_req_zone $server_name zone=perserver:1m rate=10r/s
limit_conn addr 200;
limit_req zone=flood burst=200 nodelay; ‘’’

For Bots:
Bots are identified as known bots or those requests associated with agents below:

    "~*\bMozilla/4\.0\b"            4;
    "~*\bOpera/9\.80\b"             4;
    "~*\bFirefox/4\.0\.1\b"           4;
    "~*\bSafari/534\.50\b"           4;

2 requests per second
If the request is more than this gives 444 error
Code:
limit_conn_zone $bot_iplimit zone=bot2_connlimit:16m;
limit_req_zone $bot_iplimit zone=bot2_reqlimitip:16m rate=2r/s;

An agreement has been made to add a changelog to the Registry in order to keep track of these changes in the future.
Please get in touch with us if you are experiencing issues or need further support or clarifications.

3 Likes
Datastore down (14th April)? Now up again, now down again
#2

This is really informative – thanks so much for posting details of the new rate limits! Rate limiting seems like a sensible feature to add to the registry API, so thanks & well done for doing that!

One comment: Since the service here is an API, “normal users” are probably all bots. (The non-bots are just bots posing as a different user-agent.) So I guess that means the 2 requests/s limit is the important one.

One suggestion: Again, given this service is an API, it might be a bit friendlier to respond with a real status code, rather than 444. In many cases, excessive requests will be an accident rather than malicious, so it’s useful to respond with e.g. a 429 status code and perhaps a message explaining the problem.

I think there previously was a static page on the registry site about the API? That was replaced by this page (which can be reached by clicking the “API” button in the navigation menu on iatiregistry.org.)

Given that, I think I’d expect to find details of rate limits under “Using the Registry metadata API” section of that page.

As a bit of an aside: I’m a little confused about the timeline here, because I was hitting registry API rate limits as early as Wednesday last week (24/04), but the DDOS attack was subsequent to that (26/04?) I don’t know if it’s possible to elucidate on that… Or maybe post-mortem the DDOS event?

1 Like
#3

Thanks also for this update :slight_smile:

Timeline-wise, I first reported this on the 14th of April - and guess I first saw it a week earlier.

When the new datastore goes live, perhaps several of the users (who I agree probably currently appear as bots) might transition to that? And maybe this is a longer term solution as the new datastore can already return the links to the activity and org files and some of the associated metadata. I am not sure what metadata is therefore left that would require someone to use the registry?

#4

We can discuss with Derilinx changing the codes used for error messages at some point in the future, but we’d need to keep things obfuscated. This is a low priority.

Thank you for pointing out about updating the website copy with this information about the API. we have added this task to our backlog.

The work for having a changelog on the Registry is now pending with Derilinx.

In terms of timeline, a large spike in requests had been noticed a week before than the attacks, and Derilinx started to blacklist IP addresses and user agents to assess the scale and type of situation. As the situation progressed it became clear it was the same bot hammering the server so the throttling was introduced instead.

To Matt’s point the 14th of April was Datastore down time and not related to this incident.

2 Likes
#5

Thanks for the response – that’s super helpful.

One more thing has occurred to me…

So I think this indirectly encourages API users to set the User-Agent header to something browser-like (e.g. Firefox; Safari), so their bots look like a “normal user”.

I think it would be preferable to encourage API users to set the User-Agent to something that describes the purpose of their bot, e.g. “Somali AIMS” or “IATI Data Dump”. This would be super helpful in making it possible to monitor who is using the API.