Current Datastore API use

Community Zone Using IATI Data
#1

We have reviewed the existing data store logs over the past 10 days:

  • All API calls containing the string “recipient-country=” were analysed.
  • Bots and csv download requests were excluded
  • The content of the specific calls (often including reporting-org filters) was analysed
  • Follow-up queries to the possible originators of API calls were made

We find that:

  • Only two developing countries and one donor (Sweden) are using the datastore in their production systems
  • Somalia automatically retrieves all activities a number of times each day
  • Bangladesh makes manually initiated calls for selected publishers on a random basis.
  • Rwanda is making a single invalid call every second day
  • Haiti and Malawi are making test calls from a staging server
  • There are a substantial number of non-country specific calls emanating from IP addresses in Belgium, South Korea and Russia

Please contact us if you believe we have missed something.

We are planning to continue to monitor them for the next couple of weeks to inform our planning so that we can offer users the smoothest transition possible.

4 Likes
#2

Thanks for sharing this. I’m afraid that a 10-day period is unlikely to be the best judgement for both actual and potential use of the DataStore, or the impact of failing to preserve backward-compatibility. For example, Madagascar production AMP has the latest version of the IATI-AIMS import tool deployed, which allows them to use the (current) datastore directly – I’m not aware if they’ve done this yet, but they had previously been importing IATI so it is likely that they will, if they’ve not already (outside of that 10-day evaluation period). Other AMPs also have the tool deployed, but have not engaged with IATI. If those countries chose to do so in the future, they would first need to have any new changes pushed to their servers.

2 Likes
#3

@bill_anderson great to get some data on the table and thanks for sharing. A small note to say that the South Korea calls are probably a developer who works for me - we have a tool that I use to figure out email addresses across multiple activities which uses the v1 datastore CSVs for their aggregations plus a few other calls - we should be done soon but it is throttled pretty heavily at the moment so it is taking us some time to iterate through the bugfixes and then all the publishers.

#4

It’s great to see this research! Thanks to @IATI-techteam for digging into this. A few points:

  1. Agree with @Joshua_Powell - 10 days is unlikely to provide a representative sample of current users, even aside from underestimating the number of systems that are able to use IATI data but not currently doing so.

  2. On Bangladesh: “random” is not correct. As I explained to @bill_anderson, the IATI import module is supposed to check nightly for new data. This process needs to be started again. However in the meantime, donors can (and are) still choosing to import their data. Three donors have imported their data in the last month or so: the Netherlands, USAID and World Bank.

  3. I also discovered a system I built a couple of years ago for the Liberian Ministry of Finance that is still importing data for Liberia and Uganda from the Datastore, and generating nice spreadsheets of data according to fiscal years.

  4. It would be great to work with those countries (Rwanda, Haiti and Malawi) that have nascent IATI import systems to get them into production mode. Perhaps also Bangladesh to address the current fairly minor issue there.

  5. Finally: I think this shows how hard it is to get these systems up and running sustainably at country level. We should be supporting and strengthening these existing systems. However, if the existing IATI Datastore API is discontinued, many of these systems will break. That will significantly set back efforts to promote data use at country level. It will also raise serious questions as to whether countries should invest in IATI in future.

What is the actual cost of providing backwards compatibility to these existing systems? Based on Andy’s work, it does seem that the investment on the Datastore side should be fairly modest, and certainly less than the cost of all providers needing to both update their tools and deploy those updates (to the extent this is even possible given ongoing contracts or relationships). Would that not be worth the cost to avoid losing any of these countries?

1 Like
#5

I also thought that we might need to add aidhedge to this list as they were using the DSv1 according to their website: https://story.aidhedge.org/integrating-iati-a-tech-overview-3c921aa9ded9

#6

As mentioned in the first post, we are continuing to monitor the logs so decisions are not going to be based on 10 days only.

To expand a bit on our plans, we have listened to your requests and concerns and, in conjunction with the new Datastore’s dev team, are devising a plan to cater to all the existing user’s needs during the months that will see both Datastores running in parallel so that no action will be needed on the existing user’s side.

How this will be implemented still needs to be finalised, but once that happens it will be promptly communicated. We just need to be given some thinking space and proper time to consider all edge cases, drawbacks and release a robust solution for all.

Hope this helps a bit.

5 Likes
#7

Great – thank you for the clarification, @samuele-mattiuzzo! That is great to hear.

#8

@samuele-mattiuzzo - great to hear - if there is anything (e.g. edge cases, calls that could be tweaked to make a specific solution possible etc) that you would find helpful to get input on, then please do just ask.

#9

Thank you @matmaxgeds appreciated, will do once things move further along!

#10

Sounds great, thanks for the clarification!

#11

Great to hear, thanks @samuele-mattiuzzo

To echo @markbrough @matmaxgeds @Joshua_Powell and others in the Community That Can Give Technical Advice & Support In A Collaborative Way (CTCGTASIACW - I agree it’s not that catchy), do reach out early and often for our help.

1 Like
#12

@Markbrough: interesting that you say that the Netherlands is using the old datastore. As far as I know, we do not use the old IATI datastore maintained by the IATI tech team. Zimmerman & Zimmerman are, as far as I know, making use of OIPA though to feed the ‘openaid.nl’ website.

Maybe I am a bit confused here, but to what datastore are you referring here (OIPA or the old IATI datastore maintained by the techteam)?

#13

@Herman I believe @markbrough means that donor staff from the Netherlands embassy imported the Netherlands IATI data using the tool built into the Bangladesh AIMS (which relies on the current datastore maintained by the IATI tech team).

1 Like
#14

Yes that’s right, thanks @matmaxgeds! @Herman sorry for my slow reply. The Bangladesh AIMS IATI importer uses the current IATI Datastore maintained by the IATI tech team (I know the Netherlands uses OIPA for e.g. openaid.nl)

#15

@markbrough and @matmaxgeds: thanks, perfectly clear now!

1 Like