Datastore & Query Builder: Project Update

Notifications
#1

Earlier this year, in August and October respectively, the IATI Technical Team and Zimmerman & Zimmerman released the new IATI Datastore and Query Builder for testing.

We would like to thank members of the IATI community who engaged in testing both the API and Query Builder. Your feedback has proved invaluable in discovering bugs and raising potential improvements for the products.

A number of bug fixes and functionality improvements have already been implemented based on the public feedback, including:

  • Ensuring that the default XML output from the Activity endpoint complies with the IATI Standard Schema (#1638)
  • Transactions and Results have been added to the Activities endpoint output (#1628, #1677)
  • Allow for full, unpaginated results via the Solr function (#1520)
  • Moving documentation to internal server to streamline loading (#1676)
  • Flatten CSV output so that there so only data is returned (no JSON within cells) (#1817)
  • Ensure that “0.00” values in published data are not replaced by “ ” in DS results (#1707)

For a full list of issues that have been raised, please see the GitHub project board.

What’s next?

The project team are now evaluating which outstanding bug fixes and functionality enhancements are required to be implemented before the product can be fully launched.

Once work on these items has been completed, a launch date will be set. The following requirements are included:

  • Ensure that parameter names are consistent with their corresponding IATI Standard element (#1643)
  • Ensure that equivalent data is simultaneously available between the standard API and Solr functions (#1841)
  • Ensure that data is parsed successfully when a vocabulary is not present, in instances where a vocabulary is assumed if not provided (#1713)
  • Implement redirects from queries to ‘old’ DS to equivalent queries on the new DS (#1962)

We continue to welcome further input on the project at this time, as there are also a number of backlogged items that the team will work on beyond the date of the official launch.

Old Datastore API and Query Builder

As has been previously mentioned , the ‘old’ Datastore API and Query Builder will be phased out six months after the full launch of the new Datastore API and Query Builder. We do not recommend that users migrate any tools which currently utilise the old Datastore until the new Datastore has been officially launched.

Please continue to check IATI Discuss for future updates in regard to a date for the official, full launch of the new IATI Datastore.

9 Likes
#2

Hello @IATI-techteam

I am getting questions about the new datastore. Do we have an approximate launch date?

Also, the Datastore page on the IATI website is extremely out of date - it still says the tendering process for the datastore is underway. An update would be useful.

#3

Hi Yohanna,

The Technical Team has been working on implementing the required improvement and fixes to the Datastore functionality that were raised during the testing period. The following priority items have already been completed (among others):

  • Ensure that equivalent data is simultaneously available between the standard API and Solr functions (#1841)
  • Ensure that the default columns from the activities endpoint match the order of the IATI Standard Schema (#1840)
  • Ensure that the Solr XML output is schema-valid (#1763)
  • Ensure that data is parsed successfully when a vocabulary is not present, in instances where a vocabulary is assumed if not provided (#1713)

The following items still need to be completed before a full launch:

  • Ensure that the default columns from the transactions endpoint follow the order of the IATI Standard Schema (#2105)
  • Update parameter names so that they follow the labelling of IATI Standard elements (#1643)
  • Implement redirects from equivalent queries to the ‘old’ Datastore to the new tool (#2036)

Once work on these final launch blockers has been completed, we will be in a position to announce a full launch date. We anticipate that this will be in the next few months.

With regard to the out of date information on the Datastore page on the website, this will be fixed as part of the wider restructuring that is taking place on the website. This project is due to be completed at the end of February.

If you have any questions, do let us know!

Many thanks,

IATI Tech Team

1 Like
#4

Thank you for the update @IATI-techteam!

That is quite vague. Are we looking at spring, summer, fall (hopefully not the latter…)? What was the planned delivery date in the contract?

2 Likes
#5

Thanks for the update @IATI-techteam!

Please can you confirm that:

  1. the existing Datastore will continue to be maintained until the new Datastore goes live? (It looks like it is working well at the moment, but I’m not sure exactly what the plans are here.)
  2. you’ll put in place permanent redirects from the existing to the new Datastore once it goes live?

If both of these points are confirmed, it seems to me that delaying the launch of the new Datastore doesn’t necessarily cause problems for those wanting to access it as developers (I can appreciate that it would be helpful for other users to have a clearer timetable). We can all continue developing applications on the existing Datastore, which will continue to work the same way once the new Datastore is launched.

Many thanks!

#6

@YohannaLoucheur

We should have an update on a full launch date very soon, as the Zimmerman team continues to work on the remaining requirements. Please note that the Datastore also has a dependency on system integration with the Validator. Once the Validator is ready, both tools will be officially launched.

@markbrough

  1. As per the original post on this thread, the existing DS will be maintained for six months following a full launch of the DS.

  2. That’s correct. As per our recent update, implementing the redirects is one of the few remaining items to be completed prior to a full launch. We will be asking for some community input in regard to testing the redirects so please keep an eye out for a separate thread being created about this in the coming weeks.

Many thanks,
IATI Tech Team

1 Like
#7

Great, thanks a lot for the confirmation @IATI-techteam