IATI's Datastore and Validator Tools launched

Notifications
#1

IATI is delighted to announce the launch today of its new Datastore and Validator, fulfilling a commitment to members to provide robust tools to improve the quality and use of development and humanitarian data published to IATI.

  1. Why use the new IATI Datastore?
  2. Why use the new IATI Validator?
  3. Get support: online guidance and webinars
  4. What’s changed since the testing phase?
  5. Deprecation of old Datastore and Public Validator - 22 March 2021
  6. Upcoming work

Developing and building these new tools has been the result of a close collaboration between the IATI Secretariat and suppliers Zimmerman and Data4Development, and thanks are also due to our dedicated community for testing the new tools and helping us to improve them.

The following information is provided to enable our community of publishers and users to easily transition to using the new tools and get help through webinars and detailed guidance. It also outlines planning for subsequent phases of development to continually improve our suite of IATI tools.

Why use the new IATI Datastore?

The new IATI Datastore allows for much more detailed queries and searches of IATI data than previously possible. Use the Query Builder to search data across 39 different fields (elements and attributes) of the IATI Standard and choose to download data in three different formats (CSV, XML and JSON). For the more technical user, the Datastore API enables searches of IATI data across all elements of the IATI Standard, enabling more complex data analysis.

IATI Datastore Query Builder launch.png|autoxauto
IATI Datastore Query Builder launch.png|autoxauto700×517

Use the Query Builder to search for development and humanitarian activities responding to COVID-19

The Query Builder and API also provide users with data that complies with the most up-to-date version of the IATI Standard (version 2 and beyond). The IATI Datastore complements existing online tools for accessing IATI data such as d-portal.

Why use the new IATI Validator?

Organisations are strongly encouraged to use the new IATI Validator to check if their data is accessible and of good quality, increasing its utility for partner countries, civil society and other data users. Organisations have the option of checking for errors in their data before publishing their files, or after data has been published (using the Public Data Viewer).

IATI Validator launch.png|autoxauto
IATI Validator launch.png|autoxauto700×369

Use the new IATI Validator to check the quality of IATI data.

Most organisations will find at least some errors in their data and we know this may take some time for them to correct. IATI’s Secretariat is providing support in the following ways to help organisations make these changes to improve the quality of their data.

Get support: online guidance and webinars

The IATI Secretariat has provided instructions and useful information on both tools through online guidance: IATI Datastore Query Builder User Guide, IATI Datastore API documentation and the IATI Validator Q&A. The Secretariat will also be proactively contacting IATI publishers in the coming months to offer direct support to improve their datafiles using the IATI Validator.

You can learn how to make the most of both tools by joining upcoming webinars held by IATI’s Technical Team:

  • Webinar: Using IATI’s new Validator, Wednesday 7 October Register
  • Webinar: Using IATI’s new Datastore, Wednesday 28 October Register

As always, the IATI Technical Team welcomes your questions and feedback at: support@iatistandard.org.

What’s changed since the testing phase?

The new Datastore and Validator were pre-released for extensive testing. Both have benefitted from a huge number of improvements in response to feedback from IATI’s community of data users and publishers. Here are just a few examples:

Datastore

  • Additional filters have been added to the Datastore Query Builder
  • Users can now output CSV columns in the order of the IATI XML Schema
  • The Query Builder has a new multi-sector and multi-country expansion option
  • The Datastore is now integrated with the Validator. This means that the Validator will check that data is fed into the Datastore and ensure only data that follows the IATI schema is available to users. The IATI schema provides the exact order and structure for publishing the XML. Data that does not follow the IATI XML schema will not be outputted through the Datastore. For further information see IATI Validator Q&A (page 5-6).

Validator

  • All bugs raised by users following the testing phase have been fixed
  • There is a new feature to allow the bulk-uploading of files
  • Users can now receive validation reports via email.

The IATI Technical Team has worked with suppliers Zimmerman (who continue to host the new IATI Datastore) and Data4Development (developers of the new IATI Validator, now hosted by the IATI Secretariat) to implement changes to both products.

Important: Deprecation of old Datastore and Public Validator - 22 March 2021

As previously announced, the new IATI Datastore and Validator replace the old Datastore CSV Query Builder, API and Public Validator. These older versions will be deprecated on Monday 22 March 2021 and after this date, users will no longer be able to access these services.

  • Old Datastore API: Between now and 22 March 2021 users can choose to either use existing API calls to return data from the old Datastore, or use a Redirect version from the Old Datastore to the new Datastore. Redirects can be tested by replacing “datastore.iatistandard.org” in previous queries with “datastoreredirect.iatistandard.org.” By using this version, existing API calls will be redirected and return data from the new Datastore. After 22 March 2021, the old Datastore will be deprecated and all queries to the Old Datastore will automatically be redirected to corresponding queries on the new Datastore.

For any questions or support to transition to using the new Datastore and Validator, please email the IATI Secretariat: support@iatistandard.org.

Upcoming work

IATI’s Secretariat is committed to maintaining and updating all IATI digital products to meet users’ needs. Some examples of planned work include:

Datastore:

  • Enable transaction and activity level data to be output from a single query
  • Enable output of normalised USD values for transactions, regardless of published currency
  • Enable output of validation results.

Validator:

  • Key priority post-launch is working proactively with publishers in fixing data errors and improving their data quality.

To provide feedback on either of these two projects, please raise an issue on Github: IATI Datastore Validator or email support@iatistandard.org

6 Likes
#2

Great news, thank you all for your hard work!

#3

Great - is there a statement somewhere on which data (if any) is excluded from the datastore on the basis of failing validation, or is that part not implemented yet (or was never planned to be)? And linked, how would I tell in the datastore (e.g. when using the query builder) that some data is missing due to failing validation - if this is possible?

1 Like
#4

Hi Matt,

Both the IATI website Datastore page and the Datastore’s about page say that schema invalid data isn’t included.

I’ll check about ways of finding a list of files that aren’t included in the datastore because they fail schema validation. @siemvaessen is this included in the api already or something we could add?

#5

Thanks Amy, I think this was the bit I was missing e.g. that only the schema affects what is in the datastore and therefore adhering to the guidance (which is on top of the schema) will not affect what data is in the datastore, is that correct? Will the Validator say when something is a Schema error vs when it is a guidance error?

1 Like
IATI Publishing Guidance Pages- How to Suggest Changes - input by 12 October
#6

Hi Matt, yes you’re (mostly) correct.

The Validator will flag schema issues (Critical), standard Must breaks (Errors) and standard Should breaks (Warnings) separately.

image

The Validator only includes checks that are in the Standard. It doesn’t check against the additional guidance pages.

#7

Ace, thanks - so:

  1. Schema error = Validator ‘critical’ = data removed from datastore
  2. Guidance ‘must’ error = Validator ‘error’ = still in datastore
  3. Guidance ‘should’ = Validator ‘warning’ = still in datastore
  4. Guidance ‘can/may/is preferred’…what happens in these cases - is it the same as ‘should’, or is it just not included?
  5. Additional guidance = Validator ‘not checked’ = still in datastore

Please can we update the validator message so that it actually tells users who get these error messages what the impact is on their data RE the datastore…seems like that is a critical thing to know, especially in the schema error message, especially as it applies to their entire file at this stage?

I asked this before as well - can we just rename the error types in the validator so that they match the guidance words for them therefore we do not have to maintain a weird linguistic translation table in our heads? Or update the guidance to use the validator terms?

2 Likes
#8

I am also confused why something that is a ‘must’, when not met, is still allowed in the datastore? Maybe there is a plan to phase enforcing this at a later stage? If it is a ‘must’ in the rules, this is surely because not doing it has big implications for the usability of the data?

PS - I think the video is great - I can really imagine using these to help people to understand rather than wading through discuss/online docs (but noting that the videos might best be linked to a specific version of a document, so when that changes, the video is updated accordingly): https://www.youtube.com/watch?v=Bre_hflB2Sw