Our recent post on IATI Systems Design & Architecture outlined the long-term plan to migrate core IATI products to a more resilient and sustainable technology setup. A key part of this new architecture is the development of a Python library - this library will help the IATI Technical Team, as well as tool builders and software developers to more easily work with IATI data and build end-user tools.
Purpose
The Python library will contain IATI functionality that is currently replicated across many of our software projects - as shown by the number of standalone tools in our current architecture map. Alongside our tools, there are an numerous third-party tools (including OIPA, IATI Studio, AidSight and many others) which also reproduce very similar functionality (e.g. fetching and loading files, data validation, extracting data, etc).
The IATI Technical Team will use this new library, with core IATI services refactored over time to use the library as a base for common tasks. For others, it is intended that the library will make it easier to use IATI data by removing the need to reinvent the wheel before moving onto implementing innovative functionality, performing analysis of data or simply accessing data for data visualisations.
The library will be distributed publicly using the PyPi python package index, therefore allowing software developers to install using the pip package manager.
Long-term functionality
Current plans for library components include:
-
iati.core - will allow programmatic access to SSOT constructs - i.e. the schema, codelists, rulesets
-
iati.validate - an API to validate IATI XML against the schema, codelists and rulesets. The new validator API and web user-interface will heavily use this functionality
-
iati.fetch - functionality will be included to download individual raw IATI XML datasets (i.e as a wrapper around the IATI Registry API) links to queryable data (i.e. as a wrapper around the Datastore API)
-
iati.convert - (under exploration) conversion between versions of the IATI Standard or from XML to CSV
Current development
The IATI Technical Team been working to develop the first stages of this library for a few weeks, focussing on iati.core.
Following overwhelming interest and discussion from those we have spoken to directly, we are today making this project open source: https://github.com/IATI/iati.core
Current usage examples are provided in the project readme. However, please note, at present the contents of this library is at best unstable, and much is experimental. As such, it must be expected that its contents and API will change over the short-to-medium term future. Warning sections in code docstrings help flag up some particular known stability concerns. Todo sections describe known missing or incorrect features.
At present the library (core) represents the contents of the IATI Single Source of Truth (SSOT). It is able to handle a single version of the Standard (2.02 by default, although it is possible to load any IATI XML dataset).
It is planned that different sections of the library, such as validate are split into their own repositories. They exist within this repository at present to help speed up the iteration process.
This code is provided for illustrative purposes, as we are aware that some developers are about to imminently begin work on new applications to work with IATI data. We hope that an insight into our current development will inform technology choices, encourage future adoption of the library and prevent developers from again needing to reproduce common functionality, therefore saving time.
Developer feedback and contributions welcome
Feedback, suggestions, use-case descriptions, code reviews, bug reports and so on are much appreciated - it’s far better to know of issues earlier in the development cycle. Please use the code repository’s Issues page for this.
Furthermore, if the direction of the library suggests development time can be saved on your projects, we would encourage you to use some of the time saved to be used to contribute to future development - please post below if you are at all interested in helping this library, and we can follow up direct.