Datastore Project Update: August 2020

Notifications
#1

Zimmerman and the IATI Secretariat are pleased to provide an update on the progress of the IATI Datastore project.

Over the last few months we have worked intensively to make a huge number of improvements to the tool, in response to community feedback during the testing phase, and to satisfy remaining requirements from the TOR. We’ve fixed bugs and added the required enhancements that enable the tool to be as robust and dependable as possible. We are working towards a release date of 22nd September.

What work has been completed?

A non-exhaustive list of these items include:

Bug fixes:

  • 0.00 values not being output
  • ‘Default’ elements being farmed to transactions
  • Location output not matching published data
  • Multiple description types not output

Functional Additions:

  • Validator Integration
  • Schema-valid XML output
  • Several additional filters
  • Streamlined parsing process
  • Output CSV columns in Schema order

Remaining Requirements and official launch

As existing users will know, all items mentioned above have been implemented and are available to use now.

The only remaining requirement for the product to be fully launched is for sector and recipient-country expansion functionality to be added to the csv output. Sector and recipient-country expansion allows, if required by the user, the ability to repeat each activity or transaction (depending on the row-output choice) over multiple rows, for each sector or recipient-country that is reported.

The core functionality is already available on the tool, but the Zimmerman team is now working to make this available for all filters, and ensuring that all fields can be returned, as required.

Once the work on the sector and recipient-country expansion feature is completed, the tool will be ready for launch. The purpose of the launch is to actively promote the use of the IATI Datastore both to existing and new IATI stakeholders within the international development and humanitarian sectors. A comprehensive User Guide for the Query Builder will also be published at the time of launch.

Redirects

As mentioned on a previous Discuss post, an important requirement is that queries that are made to the ‘old’ IATI Datastore API are redirected to the new one, with the equivalent query parameters used. This will ensure that any regular queries that utilise the existing tool can continue to return corresponding data from the new tool.

The timeline for deprecating the old Datastore and implementing the redirects is as follows:

From New Datastore communications Launch until 6 months after:

  • Old Datastore remains live, as is.
  • “Redirect version” of Old Datastore is also available for use. The “Redirect version” of the Old Datastore will provide users with the corresponding query to the New Datastore. This should help with any migrations before the Old Datastore is deprecated

6 months after Datastore official launch

  • Old Datastore is deprecated and no longer available
  • “Redirect version” of Old Datastore is deprecated and no longer available
  • Any queries to the Old Datastore automatically redirect to a corresponding query on the new Datastore.

Post Launch Requirements

Following the launch, we will continue to work on implementing the following requirements previously identified and keep the Community updated as to when these will be released .

  • Allow elements that can be returned at either transaction or activity level to be returned within the same query - #2081
  • Add aggregated transaction data to the activity output - #2025
  • Allow the user to choose between basic, complete, or custom csv output - #2295
  • Normalise transaction values to USD - #2219
  • Enable return of Validation results from the Datastore - #2296

As with all of our services, the IATI Secretariat will continue to monitor and respond to user needs with any necessary updates in the future, and will undertake consultations with the wider IATI Community about potential changes to the datastore and validator integration, including on topics currently under review on Discuss.

Get in touch

As ever, if you have further questions for the IATI Secretariat and Zimmerman, do post them here or email support@iatistandard.org.

Best wishes,
IATI Secretariat & Zimmerman Team

4 Likes
#2

Thanks for this update, @IATI-techteam, this is super helpful, also to see the enormous amount of work that has been done over recent months – congratulations to everyone involved in getting to this point.

I just wanted to check one thing – here it is suggested that sector and recipient-country expansion would be added to the CSV output for activities and transactions:

However, this issue suggests that actually this will only be provided for activities, and not for transactions or budgets, so I wanted to check which was correct?

Providing sector and recipient-country expansion for transactions and budgets would be really helpful (and is in the original ToR). Based on the really useful presentation in today’s DUTF, I think it would also significantly simplify access to the data for users that don’t have programming skills.

#3

Hi Mark,

Thanks for your feedback and apologies if our original post led to any confusion on this point. Adding a transaction and budget explosion is one of our post launch priorities. The current plan is to launch the validator with the country and sector explosion only when querying one activity per row.

From the datastore perspective the current explosion only deals with activity level data. It doesn’t include countries and sectors reported at transaction level. Going forwards we want to be able to combine data from both levels, and be able to use this in the three different output formats / across the different endpoints. The dependency is on querying transaction and activity level data simultaneously which is another priority post launch feature (2081).

This complication is not something we anticipated when writing the ToR.

For now, we want to make it clear in the UI that expansion options are only available when selecting ‘each unique activity’ per row.

1 Like
#4

OK understood, thanks for the clarification and explanation, @IATI-techteam!