Datastore update?

Community Zone Using IATI Data
#6

Hi @matmaxgeds, I meant check with comms. We are updating our timelines on delivery for the new IATI Datastore. From our perspective the workload should/could be wrapped up in May. But there will be a link to the new Validator service, so you would have to ask the @IATI-techteam on the complete timeline of all services bundled coming online. Makes sense?

Actual specifications as per contract.

and something we call a Master document reflecting the progress as per contract item can be found here.

Github has a label on each of the phases:

Datastore phase I and

Datastore phase II

The items left are on our list of to-do. The reason you may not see an issue in progress may very well be due to another non Datastore (OIPA that is) specific ticket being handled outside of the Datastore Project. Current ETA on staff days is estimated at 20-30 days, hence the May call for delivery.

Hope this helps!

2 Likes
#7

great - thanks for the transparent and comprehensive response @siemvaessen - looking forward to seeing it.

@IATI-techteam - it would be great to hear from you a) on the timeline for the changeover, and b) for confirmation that the datastore endpoints/routes will not change for systems currently using the existing datastore, and c) to confirm that the datastore will return the original publisher XML when requested, not just the ‘standardised’ version.

1 Like
#8

+1 @matmaxgeds and thanks @siemvaessen!

@IATI-techteam answers to these points would be very useful for planning around use of data at country level (especially “confirmation that the datastore endpoints/routes will not change for systems currently using the existing datastore”), thanks!

Also: it would be great to know what the strikethrough on “D1.3: Results should be deliverable in full” means?

#9

We took that out, due to forced pagination is in place rather than potentially endless API hammering in full.

#10

Hi everyone,
Sorry for the delay. In terms of timeline the datastore will be using the IATI validator tool so the two things need to be launched together. We have a meeting next week in Holland where we are doing some face to face planning with @siemvaessen and @rolfkleef. After that I’ll be able to share more information!

Expect more details from the @IATI-techteam after Easter.

Thanks for your patience.

#11

Thanks for the update @KateHughes - just to confirm, Easter is around the 20th of April I think, so it will be approximately three weeks before you will know if the new datastore will have different API routes to the current one?

#12

Hi Matt,
I’m keen that when we give information we give full and clear information which is why we are going to wait to give you the update.
Please bare with us,

Kate

#13

hi @KateHughes @IATI-techteam

I am not sure that we can bare with you. Given that Siem is estimating a May launch, this could mean that the community gets just a few weeks notice that systems that currently use the datastore might stop working. I am particularly thinking about those without in-house programming support who could not respond on that kind of time-frame.

Several of these questions were inserted into the ToR a year ago and I have been asking for confirmation since last October so it doesn’t seem unreasonable to know by now for the two questions on the API endpoints, and returning the original/unstandardised XML?

@siemvaessen RE D1.3 - thanks for the info - I am guessing that there will be an equivalent to stream=true, or that the API will allow parameters e.g. start=100 or tokens or similar so that the full return can be accessed?

If the answer is ‘no’ then I guess there is some sort of a process planned i.e. identifying the affected users, discussing a timeline for the changeover? I guess there would also have been a discussion of the pro’s and con’s when the decision was taken which may include some alternative solutions for existing users that we can publicise as guidelines? Are you able to share more about that?

2 Likes
#14

@KateHughes thanks a lot for the reply! While I appreciate you may not have any more information for us right now, I do want to second @matmaxgeds concern – this is something we have been raising for over one year and have never actually had an answer to.

My assumption was that that radio silence meant that the new Datastore would ensure that existing systems using IATI data via the Datastore would not break. For example, the Bangladesh AIMS has been importing IATI data for about 3-4 years now using the IATI Datastore. I’m not involved directly with it anymore, but I know that it is still working and frequently importing/updating data via IATI. This is great, and something that we should be encouraging, rather than breaking systems at country level and forcing them to spend money and conduct procurement to get the system working again. Probably in some cases, the country will just give up.

So I really do want to emphasise the importance of getting this right and maintaining compatibility with existing endpoints to ensure we are not quite significantly harming efforts to use the data at country level. We really do need to have stable and reliable central infrastructure if IATI is going to work.

@siemvaessen, I think forced pagination will also be a problem, because that is not the way the existing Datastore works.

UPDATE: I also just remembered that this was included as a recommendation in the technical audit:

Where possible, maintain consistency in API endpoints between the old Datastore and the new one, to avoid breaking existing applications.

2 Likes
#15

If there could be some guarantee that the existing datastore will remain live for some minimum period alongside the new datastore, that would probably make life easier for everyone (users of the existing datastore; the new validator/datastore contractors; the secretariat). Then there’s no deadline day for the “big switch”, but instead a period of time in which systems using the existing datastore can make the switch to the new one.

At the risk of getting a bit technical… Current datastore routes live at:
datastore.iatistandard.org/api/1/ (where the “1” refers to version 1 of the API)

The new datastore could therefore live alongside it, at:
datastore.iatistandard.org/api/2/

It could use semantic versioning (i.e. 2.0.0), but the crucial bit is: it should co-exist with version 1, at least for some predefined period. That’s the usual way for technical services like this to work. When a new version is released, it doesn’t just replace the old version. It sits alongside it for a period of time, in order to give users time to migrate.

Versioning the API like this would also set a good precedent for future datastore upgrades.

2 Likes
#16

Yes, as Andy says we can run the two side by side for a period.

As Siem mentioned the datastore has a dependancy on the IATI validator tool, next week we are all getting together to discuss timelines for both tools so we can have a launch plan. It will not be launched in May because the validator won’t be built by then.

The validation contract was only signed three weeks ago, so we’d appreciate some time to meet face to face with the vendors of these core products so that we can get all the details in order., as Andy says we can run the two side by side for a period.

As Siem mentioned the datastore has a dependancy on the IATI validator tool, next week we are all getting together to discuss timelines for both tools so we can have a launch plan. It will not be launched in May because the validator won’t be built by then.

1 Like
#17

Hi @KateHughes, that sounds like a sensible plan.

RE technical questions, looking forward to hearing after Easter then. In the absence of any information, we will proceed on that basis that the new datastore will fully meet the specifications as outlined in the ToR. UNDP are funding our programmer in Somalia until the end of July, so any changes to the datastore (api base path, api routes, pagination, delivery of unstandardised data) that we are not aware of by the end of May means that the IATI aspects of the Somali AIMS will break when the full changeover happens, no matter how long the transition period.

#18

Hi @KateHughes

Thanks for your feedback.

Somalia is currently developing Aid Information Management System which is having an IATI data import feature. The UNDP support for the government in development of AIMS will end in July 2019. we hope that IATI datastore changes will not cause any fault to our system.

My colleague Matt mentioned that Somali AIMS is coping the old IATI registry and dataset and parsing all files to make them Somalia related information. he repeatedly said to me that we need to get in touch with @IATI-techteam for further discussions about the possible change of current API.

It would great that we are clear with everything ahead of time so that we keep our development smoothly.

Looking forward to hear from IATI Technical Team after Easter.

Gele,

#19

Hi @markbrough @matmaxgeds @GeleMohamed (& others?) could you supply a list of API calls in use on current datastore and current output, usage etc.

That way we actually have some numbers to crunch. Would be helpful in taking this forward. I took the liberty of providing a basic Google sheet, so we can see what kind of calls are being used. You can find it here.

Thanks!

3 Likes
#20

@siemvaessen did you mean to post a link to a Google Sheet in that link?

#21

Yup @markbrough - changed it, used that exampe DS link for reference in the actual doc.

#22

@siemvaessen sounds like a plan - will get our dev to input shortly.

@IATI-techteam I would also second the call here for some kind of monitoring so we can start to work out who all the other users might be - is this kind of thing part of the data use strategy i.e. there must be a document somewhere that breaks down the data use a bit?

3 Likes
#23

@siemvaessen – great, thanks! I will add into the document the API calls used by the Bangladesh AIMS IATI importer.

@IATI-techteam – +1 on @matmaxgeds’ suggestion. While it would make me very happy if we didn’t break things that I use / have worked on, I am also a bit concerned about breaking the tools for other users we are not aware of, or for users who are less present on these kind of forums. So I think it would be really great if we could use analytics or monitoring tools to see which other routes are popular.

2 Likes
#24

Also: make sure to claim your own tab in that sheet :fire:

1 Like
#25

Hi all,

Please note that an update on the new datastore API has been posted in the Notifications category - Update on IATI Datastore API