Updated information about IATI data APIs currently available/in development

Community Zone APIs
#1

Hi, if anyone is working on an API please would you add information here. Also if anyone know of issues with existing APIs. Will save a lot of heartache for users! Thanks.

#2

Hi Sarah,

We are working with the datastore in Bangladesh and it would be great for a joint push on some of the current issues: https://github.com/iati/iati-datastore/issues.

I would also find it helpful to understand who is maintaining these APIs (as well as core IATI tools like the datastore and d-portal) and what are their plans for them going forwards?

Thanks

Matt

#3

Hi Sarah, all,

We have been working on a native IATI API for some years now: we call it OIPA and is has been deployed as a BaaS (Backend as a Service) powering the DFID Devtracker and serves as a SaaS like solution to MFA as openaidNL (Dutch Government), UNESCO, RVO (Netherlands Enterprise Agency) and UN-Habitat amongst others. The good people at Catalpa built the Myanmar AIMS on top of OIPA: Mohinga.

OIPA was constructed back in 2011 as part of a pilot with MFA. The focus of OIPA (which stands for Openaid IATI Parser and API) has always been to provide IATI data producers and consumers with a standarised API. OIPA (semi)-automatically extracts all IATI publishers and data from the registry (or any other location) and parses XML for data storage in PostgreSQL. Making use of the Django REST framework anyone can access data in OIPA. In short OIPA is an IATI in, IATI out data-engine.

All the examples I provided earlier are examples of that: the API powers all the data visuailsation on many different levels (activities, countries, aggregation etc). OIPA has many different data endpoints:

"endpoints": {
    "regions": "http://dev.oipa.nl/api/regions/",
    "activities": "http://dev.oipa.nl/api/activities/",
    "datasets": "http://dev.oipa.nl/api/datasets/",
    "sectors": "http://dev.oipa.nl/api/sectors/",
    "countries": "http://dev.oipa.nl/api/countries/",
    "publishers": "http://dev.oipa.nl/api/publishers/",
    "cities": "http://dev.oipa.nl/api/cities/",
    "organisations": "http://dev.oipa.nl/api/organisations/",
    "transactions": "http://dev.oipa.nl/api/transactions/"
}

}

OIPA is being maintained by Zimmerman & Zimmerman. We are based in Amsterdam. OIPA is licensed under AGPL so feel free to fork, push, commit etc. Talks are in progress to move away from the ‘Dog on a Leash’ license, which I feel AGPL to be and move it to either GPL, Apache or MIT license.

We have started production work on IATI Studio, an initiative by Zimmerman & Zimmerman, Oxfam Novib Netherlands and Leiden University. IATI Studio will be a service layer on top of OIPA, enabling data producers and consumers to either manage and publish IATI data or access any of the data available in the OIPA datastore and perform analysis and visualisations. It will also feature something we call the micro-portal creator: build openaid like portals with native IATI data under a minute.

Anyway, that’s my take. If you have any questions, shout out here, ping me on twitter or contact me via the ZZ contacts page.

#4

@Wendy @rory_scott Are you able to answer Matt’s questions?

#5

Give them a chance!

The team will provide relevant info shortly.

#6

The IATI Datastore is an online service that gathers all data published to the IATI standard into a single queryable source. This can deliver selections of IATI data in JSON or XML formats, or CSV (spreadsheet) for less-technical users. A large number of API endpoints are available, with data queryable by:

  • iati-identifier
  • recipient-country
  • recipient-region
  • reporting-org
  • reporting-org.type
  • sector
  • policy-marker
  • participating-org
  • participating-org.role
  • related-activity
  • transaction
  • transaction_provider-org
  • transaction_provider-org.provider-activity-id
  • transaction_receiver-org
  • start-date
  • end-date
  • last-change
  • last-updated-datetime
  • registry-dataset

An online CSV query builder is also available as a point-and-click interface to select CSV datasets to download.

The Datastore was initially built by Open Knowledge Foundation and is maintained by the IATI Secretariat. The code written in Python, with data stored in a PostgreSQL database. The project is open-source and hosted on GitHub, and we welcome comments and code contributions in order to improve the product and make it as useful as possible.

Advantages

  • XML and JSON outputs contains the display full IATI XML, as it was originally published
  • Well-documented, with further documentation to be added shortly featuring example queries for each API endpoint

Disadvantages

  • In recent months, there have been an issue with missing activities in the past few months, but recent work appears to have minimised this issue
  • Displaying IATI XML as it was published means there are inconsistencies between IATI versions (for example codelist values reported as textual strings in IATI version 1.x, v.s. numeric values in IATI version 2.x)
  • Only a selection of fields are available when outputting in CSV format
  • Datastore doesn’t handle hierarchies of activities
  • Doesn’t allow filtering on all fields it outputs (although this appears to be in common with other APIs)

Roadmap for the future

The IATI Secretariat are putting together a roadmap for how the Datastore will look in future. Current options include building a new version of the Datastore to address many of the disadvantages above, or further work towards addressing some of these issues based on the current code.

#7

Hi
Just a quick note on d-portal too in response to Matt’s question - it’s maintained by the IATI team with a fairly small budget for maintenance and ad hoc development, usually for addressing specific user requirements (if you have any suggestions, do contact the support desk or add to the GitHub issues)
Thanks
Matt

#8

As mentioned yesterday, the IATI Technical team have now further updated the documentation for the API calls themselves, including examples for each of the filters available: http://datastore.iatistandard.org/docs/api/

We welcome user feedback and hope this updated page now gives further clarity on the type of data that is available through the IATI Datastore.

#9

Siem,
at the end of your message above you say “It will also feature something we call the micro-portal creator: build openaid like portals with native IATI data under a minute.”

Could this be used to create country-level portals, that would bring together all the IATI data about a given country? For example, having a portal for Kenya, where all projects coded to Kenya, all the forward budgets for Kenya etc would be available, with a map of the country, and where people could easily filter data to look at a specific sub-sector (eg road construction), or a specific region (south-west), or all projects involving NGOs, etc.

This would be a tremendous tool to be used for aid coordination; in doing so, it would give much more visibility to IATI on the ground (especially with local donor offices) and would give them a strong incentive to improve data quality.

As far as I know, the openaid portal is not able yet to bring together data from multiple publishers. Is the OIPA getting better at it, so that what I describe above would become possible? I’m starting to see demand for this kind of things in our country offices. If would be great to have a mock-up or demo, as currently most of them don’t realise what they could do with the data.

#10

Hi Yohanna,

The OIPA API is used to power Devtracker, UNESCO, MFA etc. and manages all the IATI data that is reported to the IATI registry. The API allows for country based filtering, aggregation etc.etc.

So yes, this API support a country-level portal - something we intend to offer in IATI Studio as an “IATI theme”. The main idea behind the IATI Studio is to offer different data-themes supporting a variety of business cases.

I will message you offline, seeing how this is becoming somewhat off-topic.

Siem

#11

Hi all,

Dale/Matt - thanks for the answers on the datastore/Dportal. We start using the API next week so will be able to give proper feedback very soon. Much of the below presumably also applies to D-portal.

Siem, It would be great participate in the discussion on how OIPA creates country portals so we can come up with common solutions to some of the common issues? Following Copenhagen (and taking up Bill’s suggestion) we were trying to start a group including DG, Synergy and Catalpa and others and it would be great to have all your experience added. For example, in Bangladesh we are finding that:

  1. DPs report different Units of Aid and this means it is hard to understand what you see e.g. pulling up IATI data and a) finding several hundred entries relating to project sub-components, or b) finding a donor reports based on high level programmes covering multiple projects and sectors. Engagement at the country level depends heavily on having project level units. This is linked to how to interpret hierarchies where they are in use. Even more difficult are DPs who report thematically against their country objectives in IATI, rather than on the basis of recognisable projects or programmes.

  2. To make a country portal with aggregate numbers, all the double reporting/counting needs to be removed, we think there are some programmatic ways to this and are keen to collaborate on new tricks e.g. via the aid type where reported but this only gets us so far, and not enough use is currently made of the tools built into IATI to show flows between different reporters. This issue is also linked to the correct identification of pooled funds, trust funds, as well as the ability for the portal to make use of IATI data from implementing partners - and one day, that reported by the recipients!

  3. Translation!

  4. The need for clarity on the status of IATI data. For which DPs can it be considered their official data, and therefore be expected to match what is reported by country offices? If not, IATI data is very quickly dismissed. Linked is that country portals stimulate the need for far better contact data than a generic info@donorHQ.org email address.

It would be great to join forces for some lobbying of both DPs for small adjustments to their reporting, and to the standard for similarly small changes, would you like to chat off-forum with Mark and I sometime? We are documenting what we find (and all code is open source) at http://http://bd-iati.github.io/ e.g. related to the above: http://bd-iati.github.io/documentation/project-types/ and http://bd-iati.github.io/data/dfid/

Thanks, Matt

#12

Is anyone interested in sharing the cost of Z &Z developing a reporting-org endpoint for OIPA?

#13

Hi Sarah,

Here’s a basic version of an organisation endpoint. Containing organisations as reported in the organisation standard files or as reporting-org in the activity standard files.

Just identifiers and names for now, I had no time yet to implement the full organisation standard (for organisations that do report in the organisation standard).

There’s a lot of US-GG-* organisations in there. This is because Global Giving is reporting 6000+ organisations in their organisation file; http://www.globalgiving.org/iati/organisations.xml I’m not sure on the organisation identifiers they use…

#14

Hi Matt,

Late response here, totally agree with your comments on different reporting strategies and double counts on aggregations and it would be great to chat! I’ll make sure I’m in the #iati chatroom on freenode this week (no one seems to use that anymore, or I’m just in the wrong room)

#15

Hi Vincent,

It would be great to have a chat and the timing is quite good. We are nearing the end of the current phase in Bangladesh, and Mark and I are looking to put together the lessons that would be most useful for the community. Hopping onto freenode now but you can also skype me: ‘matmaxgeds’ or email me direct: work@mattgeddes.co.uk.

Matt