Moving IATI Standard content from IATI reference site to main IATI website

Notifications
#1

This post provides an update on the work to migrate IATI Standard content from IATI’s old Reference site (now available here) to iatistandard.org as announced in our previous posts. Many thanks to all the feedback from the community on this project.

What changes have been made?

IATI’s website has been updated to include information on the rules and guidance of the IATI Standard. Content previously held on IATI’s old Reference site has been added to iatistandard.org, ensuring all information about the IATI Standard can be easily accessed in one place.

All IATI Standard information now on iatistandard.org

  • Technical documentation on all components of the IATI Standard (elements, attributes, rulesets, upgrades etc.) can be found in the IATI Standard section of this website. Do note, as the IATI Standard version 1 was deprecated in 2019, all version 1 pages are kept as an archive within this section.

  • All guidance material (including setting up IATI accounts, how to publish data, Standard guidance and developer documentation) can be found in the Guidance and Support section of the website.

Old IATI Reference site

  • The old IATI Reference site has been archived and will be turned off in 6 months time on 3 February 2020.
  • However, from today users who access old links to the IATI Reference site, will be redirected to the new pages on iatistandard.org.

Users should note that no information on the IATI Standard that has been migrated from the old IATI Reference site has changed, however it has been reordered and re-structured on iatistandard.org. Updates have been made to content in the Guidance section of iatistandard.org, following consultations with the community. These changes will make it easier and more streamlined for publishers and data users to find and understand information about the IATI Standard.

Additional Technical changes: IATI - GitHub

There have also been a few behind-the-scenes changes on IATI-GitHub to provide more flexibility in how the Secretariat can use and display IATI Standard content.

Changes:

  • From IATI/Extra-Documentation we have moved:

    • The updated overview pages to the IATI/Guidance repository
    • The codelist-api.rst page now sits within IATI/Developer-Documentation
    • The codelist-management.rst page sits within IATI/Upgrades

  • From IATI/Guidance we have moved:

    • The old datastore documentation to the IATI/Developer Documentation repository
    • The old introduction pages have been archived as these are now contained within Wagtail CMS as per the first website launch
    • The IATI upgrade pages to the IATI/Upgrades repository.

These changes provide the Technical Team with the required level of version control for each page type. For example, we only need one copy of each IATI standard upgrade page. These pages do not differ between the IATI versions.

Next steps

In the coming months we will continue to undertake formatting fixes and optimisations on iatistandard.org. These will happen both on the website and behind the scenes.

We’re committed to listening to any feedback so please email any questions about these changes to support@iatistandard.org.

#2

It looks as though the codelist API has moved as a result of this change. Most of the API URLs listed on that page now return a 404.

For example: previously, this URL returned the sector codelist as JSON:
https://iatistandard.org/203/codelists/downloads/clv2/json/en/Sector.json

It looks as though this has now moved here:
https://iatistandard.org/downloads/203/codelists/downloads/clv2/json/en/Sector.json

Would it be possible to add the relevant redirects? This is currently a breaking change for services that rely on the codelist API.

Also worth noting that the link to the codelist API is now missing from the codelist page (I suspect this means all links to developer docs have been removed?) In any case, it would be good to restore this link.

Thanks,

IATI datastore not updating and previewer down?
Show IATI: IATI Canary
#3

Hi Andy, absolutely! We will look into this today.

Please do let us know if you find any more redirects or suggestions for areas we could better link the content due to the new structure.

1 Like
#4

Very keen to get an update on this as we have been using these codelists links in our live system in Somalia

#5

Hi Matt, we’re looking into it as a priority today and will update this thread as soon as possible. Thanks for your patience!

1 Like
#6

Yes, sure.

A couple I’ve noticed are:

1 Like
#7

Thanks Andy. The old datastore documentation is on Github here

There are links to it from the new IATI Datastore page.

#8

Is this the page you’re referring to? It includes this sentence:

For technical details about how to use the existing Query Builder while it is still being maintained, please visit the Github repository.

That text appears to be about the query builder, and that link is broken. I don’t see another link to the old datastore documentation on that page.

1 Like
#9

Update about the Codelist API links from one of our devs:

Hi all,

Apologies for any interruption of service you may have experienced. If you need access urgently, please use the backup domain we set up at archivereference.iatistandard.org .When we made the migration from static HTML to Wagtail, all files referenced in the content were programmatically downloaded, uploaded to the Wagtail CMS, and made available with redirects from their old paths.

For e.g. http://iatistandard.org/201/codelists/downloads/clv1/codelist/ActivityDateType.csv Where this seems to have fallen flat is that not all needed links are actively referenced in the content (most of 1.04 and 1.05 download files for e.g.)

Additionally, what wasn’t anticipated was the need for users to see Apache2 style download folders of files for download. We will need to remove those files from the Wagtail CMS, and instead manually host them to imitate this previous behaviour.

1 Like
#10

Good spot. I’ve updated the links on both these pages:

#11

Right, yes – I see. That approach won’t work for an API, where most URLs aren’t referenced explicitly. Although… I think the codelist API had already been moved from reference.iatistandard.org to iatistandard.org?

I think there are some other problems here, too, but they’re hopefully not too painful to iron out. Here’s an example redirect path for a codelist:

I think redirect 3 is new, and it looks like it’s the problematic one – it incorrectly strips the “en” language code from the URL.

Additionally, I’m not sure where the codelists.json and mappings.json files live now?

#12

Great – thanks. It’s still non-trivial to reach the API docs, since the links to the individual API pages point back to datastore page again. I guess they should now be relative or absolute links to github instead.

#13

Hi all,

Implementing a fix right now. You might experience some additional 500 or 404 errors while the fix is going live.

Once the fix is live, I’ll leave another update. Unfortunately some of the redirects are 301 permanent so you may have incorrect endpoints cached even after the fix is live.

1 Like
#14

Hi @andylolz and @matmaxgeds

I’ve reworked the SSOT import to work around the Wagtail Document interface.

Indexed all downloads should be available here: https://iatistandard.org/reference_downloads/

And all redirects should properly point to the correct file now.

I think there are still some additional links that @amys and I will need to work together to fix throughout the RST content. But hopefully these fixes should allow your services that depend on the SSOT to continue functioning.

Please let me know if you spot anything else that needs fixing.

2 Likes
#15

Thanks, @alex_miller

We rely on the redirects as part of d-portal’s nightly update. Should we change to the new urls; ie. are these going to be stable?

This downtime has affected us and we haven’t been able to do a full import for the last 5 days. We’re currently checking to see if everything is still working; ie. redirects are all working.

We notice that the json downloads are much slower than they used to be before this change. Is this intentional?

We also notice that version 1.04 and below is not listed on https://iatistandard.org/en/iati-standard/ under ‘Previous versions’. Is this as designed?

Thanks!

1 Like
#16

Codelist API redirects are all working for me – thanks for resolving this.

#17

Hi @shi,

I believe not listing 1.04 and below is a content decision from the IATI BAs, but they will be able to confirm tomorrow morning.

https://iatistandard.org/reference_downloads/ will be stable, and I’ll try my best to keep redirects from old paths up to date.

As for performance, I could look into some static file caching on the NGINX side, but we could run into a cache-clearing issue for updates. reference.iatistandard.org was proxied via CloudFlare, and I’ve just turned on the proxy for iatistandard.org as well. Hopefully this provides you with a bit of a faster response in the meantime.

#18

Hi Shi,

We decided to slightly hide version 1.04 and below. Most of the IATI users and publishers don’t need it.

We could change the version 1.05 link to a generic ‘version 1’ and link here: https://iatistandard.org/en/iati-standard/upgrades/how-we-manage-the-standard/versions/ ?

#19

If the trailing slash is omitted from a codelist page URL, the server now responds with a 403 error. For example:
https://iatistandard.org/en/iati-standard/203/codelists/sector

I think that’s a bug. Other pages on the site don’t appear to behave that way.

Is there a better place to flag such bugs? Presumably https://github.com/IATI/IATI-Standard-Website/issues is now the best place to flag reference site issues?

#20

That would be great, yes please Andy

1 Like