Update on the Roadmaps for IATI Technical Team Products (August-December 2017)

Just before the March 2017 TAG, the IATI Technical Team posted an outline of our Product Roadmap This included short-term plans up to the end of Q2 2017. Now that we are in Q3, we are providing an update on progress in these areas, plus information about plans for the next few months.

Review of the past few months

There are five key areas that we highlighted as areas to focus on in our previous post.

1. Maintenance

We have undertaken a range of maintenance activities on existing products. A small subset of the changes we have made are:

  • Updates and management of Non-Embedded Codelists

  • Started migration of products to HTTPS. Some, such as Discuss, have been migrated, others are at various stages in the pipeline.

  • Resolved a range of issues with products including the Dashboard, Standard Website, Query Builder and Registry Community input from @andylolz on investigating some issues has been particularly noted and appreciated :slight_smile:

  • Major review of internal processes and coding standards which has generated efficiencies, giving us more time to work on things that arenā€™t maintenance.

2. Time-sensitive Work

We have worked on a number of time-sensitive projects:

3. Architecture Design and Development

In January, we took time to assess the current state of IATI architecture and we posted our plans to improve the situation.

4. Python Library Development: Core and Validation

We have worked to begin implementing the base architecture, formed initially around an IATI Python Library*.

Our primary work has centered around the core module to act as a reference implementation of the letter (rather than the undocumented ā€˜spiritā€™ (that varies from person to person)) of the ā€˜Standardā€™. We have initially used version 2.02 of the Standard for this, and will soon extend to other versions.

We have also undertaken some work to perform dataset validation within the Schema. A proof-of-concept implementation for validating a dataset against IATI Codelists and Rulesets has been demonstrated.

Alongside this, work has recently begun work on additional parts of the Python Library, such as iati.fetch, which aims to offer functionality to download data and updates to Single Source Of Truth content using a network connection.

* The Python library is a collection of reusable software components that the IATI Technical Team are building to help us to improve the underlying structure of existing tools. The library will act as a base for common tasks and is being built in an extensible way. Therefore, it is intended that the library will make it easier to use IATI data by removing the need to reinvent the wheel before moving onto implementing innovative functionality, performing analysis of data or simply accessing data for data visualisations. More information here.

5. Datastore Development

We have, unfortunately, not had the capacity to properly assess future plans for the Datastore However, we are definitely aware that there are problems with the current product and, as such, we will ensure a proper review is undertaken at the point in our migration to a sustainable architecture when we require the functionality that a Datastore may provide.

Direction for the next few months

Alongside maintenance, there are three main projects that we plan to work on between now and the end of 2017:

  • Validator: We will aim to release developer functionality (with accompanying user guidance) that will enable datasets to be validated against more of the IATI Standard than just Schemas ā€“ for example, that the correct values from a related codelist have been used.

  • 2.03 Standard Upgrade: Once proposals have been finalised and agreed upon, we will implement them in the Schema, and update tools to support the new functionality as appropriate.

  • Architecture Design and Development / Web Project: We will continue work on the Python Library to provide the base of a sustainable architecture. Alongside this, we will use it in practice during work to significantly improve the experience of those using IATIā€™s websites. The scope of this project will be confirmed over the coming weeks before we start implementation in the Autumn.

Super helpful to have this kind of update from the Technical Team. I understand you guys have a lot of competing demands on your time and that resources are constrained.

In view of the existing constraints and the calls to focus more on data use, I am not really sure how the Python Library development responds to the user needs identified at the TAG. From my perspective, providing reliable, stable, queryable data through the IATI Datastore (or an alternative solution guaranteed by the Secretariat) is essential to being able to effectively use this data at country level. The technical roadmap also states that it is seen as ā€œcore to IATIā€.

However, according to this update there are no plans to address the Datastoreā€™s shortcomings even in the medium term.

Can this be reconsidered please? Thanks! :wink:

3 Likes

@markbrough our medium term is a little longer than yours. @dalepotter has outlined priorities for the next three months. The Datastore is top of the priority list thereafter.

If you can bear with me for a couple of weeks, I will (in a piece accompanying the launch of the Python Library) make the case for investing in rock-hard foundations and sustainable infrastructures, putting our work in this field into historical perspective.

3 Likes

Alongside what is presented here, there is a separate Data Use workplan due to run from September 2017 through to August 2018. This will look at the various dimensions of Data Use including both technical and political interventions (plus a variety of things that donā€™t fit neatly into these categories). The current prep work is being undertaken by other members of the Tech Team, so I am unsure of the full detail of what it will entail and where the balance of work will lie, though there is reference to work around improving / developing Datastore-esque tools in there.

2 Likes

Excellent, thanks @bill_anderson and @hayfield, and sorry for my impatience. I am placated and reassured :blush:

1 Like