Making codelists more accessible

Community Zone Using IATI Data
#1

See below for the latest short post, cross-posted from the Code for IATI blog – this time to launch our new replicated codelists website.

Making codelists more accessible

When using IATI data, there are a number of reference codelists that are needed to make sense of the data. Some of these codelists are controlled by IATI directly, but others are controlled by a number of other organisations: for example, the Country codelist (the ISO-managed list of countries) or the Sector codelist (the OECD DAC list of Purpose Codes).

IATI maintains copies of these third-party codelists – called replicated codelists – but there are a number of reasons why, as users of IATI data, we would often have to construct our own versions directly from the source. We built the the replicated codelists interface to share this work with others who could stand to benefit from it.

Aims and additional functionality

We had three main aims in building this new interface. We wanted to ensure that replicated codelists are:

  • as up to date as possible;
  • provided in formats that are most useful for users of IATI data (including via an intuitive API);
  • accessible in multiple languages - initially in English and French, with scope to expand the range of supported languages depending on availability of translated codelists.

We’ve gone through most of the replicated codelists to ensure they are fully up to date and consistent with the source, including adding in historical codes that have subsequently been withdrawn, and adding in official French translations. The entire interface can be switched from English to French (with thanks to @YohannaLoucheur!) and codelists can be downloaded in a range of formats (including Excel) in either language.

We added in a codelist for Humanitarian Global Cluster codes, and a link to a separate set of work around exchange rates data - vital reference data when working with IATI data. We made all codelists accessible through a simple API for developers. Finally, we have a workflow in place to ensure that everything is kept up to date and consistent whenever the source codelists change.

Next steps

We already sent our proposed updates to the underlying codelists to IATI, who can decide whether they would like to incorporate these changes.

Let us know if you think this is useful for accessing codelists, and what else you think should be included! We have logged a number of issues we plan to work on here.

6 Likes
Region codelist is out-of-date (from OECD DAC source)
#2

Dear Andy,

I’m not a technical person but I do find this list and easy access useful. May I ask if the UN(M49) list could be put here as well. As I’m sure you know, it is a standard code list maintained by the UN which, unlike the OECD list, contains all countries mapped to all regions which is useful for many who have benefiting regions not on the OECD list.

5 Likes
#3

Hi Michelle,

I am keeping a list of ones people would find useful and whenever I get time adding them, so I will put this on the list - I should be able to get it done towards the end of the month.

Are you just after the regions, or the country codes as well, as I couldn’t work our whether the country codes would be valid as my quick search only found M49 codes here: http://reference.iatistandard.org/203/codelists/RegionVocabulary/ - but I am sure having the country codes on the region list would make picking the right region easier, if for nothing else?

Matt

2 Likes
#4

Matt,

Agree that having the full list of countries helps everyone or at least those of us who work outside of the recipient country list of OECD. We also work with UN regions not the OECD regions for consistency with other reporting requirements we have.

I’ve previously asked the IATI Secretariat, the UN data Cube people and the UN Finance & Budget network teams to see if they can get the UN Stats team (the ones who maintain the M49 list) to include the 2-digit ISO country references so there would be one source for all. Sadly, I’ve had no luck so the UN list only really gets us regions and official country names.

One day I hope someone with better knowledge of the UN systems, people, processes can make this happen. Perhaps it is because I don’t have a first born to sacrifice to the cause. :slight_smile:

Michelle

#5

Hi @Michelle_IOM,

I think you are looking for a list that maps the Country codelist (containing ISO-2 country codes, e.g. KE for Kenya) to the UN M49 codes. I found this list of UN M49 codes, which includes ISO-3 country codes (e.g. KEN for Kenya), as well as UN M49 country codes, and UN M49 regions, so I guess a mapping between the two codelists should be relatively straightforward.

I also noticed this DAC proposal to map UN M49 regions to DAC regions which may be of interest. It’s in a PDF at the moment, I guess it could be useful in Excel?

Have I understood correctly – is this what you’re looking for?

#6

Mark,

The UN M49 list is what I would hope would get added to the replicated codelist list. But I was also pointing out that rather than the UN M49 list use the 3 digit ISO codes, they could switch or at least add the 2 digit code so it would be all together and, while easy to map, wouldn’t be necessary for the cross-reference. The UN has indicated they are trying to harmonize to existing standards and IATI in particular when it makes sense. Using 2-digit rather than 3-digit ISO codes was just one of the minor things discussed. I just don’t know how to get the UN Stats team to make the change as keepers of the list.

I hope that makes more sense.

Kind Regards,
Michelle

1 Like
#7

Great, yes makes sense! I’m sure we could add the UN M49 codes to the replicated codelists site. And I also think it would be relatively straightforward to also include the 2-digit ISO codes on there (in addition to the 3-digit ISO codes), given that the mapping already exists.

1 Like
#8

Is this repository of any use? http://joinedupdata.org/sup/en/page/?uri=http://joinedupdata.org/Supranational/geographic_regions

3 Likes
#9

Nice work, Andy—I think it will be useful to a lot of people. Do you transform on the fly, or work off static copies updated via a cronjob?

1 Like
#10

Bill,

I like the what one can compare and contrasts lists in your link.
However, the UN list in this link is from 2015 and doesn’t have the correct names of some countries so it wouldn’t be helpful for the UN agencies. We have to be very careful with the country names we use. It also doesn’t contain the code behind the names from what I can see. But maybe that is visible behind the scenes in the code.

#11

I think this is super useful, thanks @bill_anderson!

@Michelle_IOM the country codes (both 2-digit and 3-digit) are also visible behind the scenes. I think once we have this into a spreadsheet format it will be a bit easier to see.

#12

Many thanks, @David_Megginson! I should clarify that this is the work of several people, including @markbrough, @matmaxgeds and @YohannaLoucheur.

Yes exactly – static copies, updated via a cronjob.

1 Like
#13

Unsustained funding >> unsustained project >> unsustainable data.

(Sigh)

#14

Hi @Michelle_IOM, just to let you know that we’re currently working on this. @markbrough has merged M49 with the 2 digit ISO country codes. The next step is to include it on the replicated codelists site. This should be sorted in the next week or so.

Thanks for your patience!

2 Likes
#15

@Michelle_IOM the M49 codelist is now available on the replicated codelists site: https://codelists.codeforiati.org/RegionM49/

This includes mappings to ISO 3166-1 alpha-2 codes.

The data is available in HTML, XML, JSON, CSV and XLSX, and in English and French.

Credit to @markbrough for sorting this!

4 Likes
#16

@andylolz Fantastic work. Thank you for your persistence and the notification.

Stay Safe,
Michelle

1 Like
#17

@andylolz

Ok I’m either going blind or misunderstood but when I pulled up the link in IATI to the UNM49 code list it brought me to
https://unstats.un.org/unsd/methodology/m49/
and that only has the ISO-alpha3 code. Did I misunderstand what what was changing?

#18

Hey @Michelle_IOM

Yes - the original source of the UN M49 codelist only has the 3-digit ISO code. To make it easier to use, we added the 2-digit ISO code in our replicated version of the codelists: https://codelists.codeforiati.org/RegionM49/

You can download the data directly from the Code for IATI codelists interface in a range of different formats. The data in that interface is kept up to date by automatically checking for changes with the source every night, so it should be fine to use our version if you wish.

3 Likes
#19

Got it. Was really hoping the UN would change the source since it is used for other reporting. Glad at least we got it for IATI.

#20

@Michelle_IOM – it appears the UN M49 codelist has also now been updated to include the 2-digit ISO code:
https://unstats.un.org/unsd/methodology/m49/overview/

We’ll continue to make this codelist available in the replicated codelists interface (we are automatically tracking any changes to the source).