TECHNICAL NOTICE: Moving IATI Standard content from IATI reference site to main IATI website

#1

The IATI Technical Team will shortly be moving the content of the IATI Standard Reference site onto IATI’s main website: iatistandard.org/. This work is being delivered in line with the IATI Workplan Y6 (Sept 2018 - Dec 2019) and IATI Workplan Y7 2020.

This move means that IATI Standard reference information will be accessible on one website, alongside other useful publisher and user content. It will also streamline the Technical Team’s process of updating content, saving valuable time.

What is changing?

Other content to be moved to from GitHub to Wagtail include:

Changes to the format of the IATI-Extra-Documentation GitHub repository - please provide your views

The IATI-Extra-Documentation repository will remain on GitHub however the Technical Team proposes to change the way the pages are written (as described below). Currently the Technical Team uses reStructuredText (rst) format, with additional Sphinx markup. However the additional Sphinx markup added to the files can be a nuisance to read as code and can sometimes be confusing for human reading as well.

The Technical Team propose changing the format of these files into one of two options:

  1. Change the files into JSON format with some embedded Jinja which would link to the example XML files. This would mean it would be easier for developers to obtain the Extra-Documentation content from the repository, and it would correspond with the format that is used by the IATI-Standard-Website.

  2. Change the Extra-Documentation repository into XML formatted files. This would enable developers to easily read and convert the documentation to the required formats, whilst also remaining in a familiar and readable format, allowing for easy generation or amendment of content as necessary.

Feedback required: Please provide your preferred option or any other comments about this proposal by responding to this post or emailing: support@iatistandard.org.

Do note: that the Technical Team will keep sphinx branches in the IATI-Extra-Documentation GitHub repository after the format switch is made. If you wish to keep using them in that format, only a branch switch will be required. However if you wish to use the new format, your work will have to adapt to include them.

What is not changing?

The following content will be accessible on iatistandard.org but will remain on Github and not be moved to Wagtail:

Who will be affected by these changes? Should I do anything to prepare?

Developers who pull guidance content from Github repositories into their own platforms may be affected. If a developer has concerns about the content in the Github repositories that will be moved to Wagtail, please contact the IATI Technical Team: support@iatistandard.org.

If you are not constantly creating guidance pages from the content of the IATI-Guidance or IATI-Extra-Documentation repositories, there’s nothing to prepare for.

Should I be aware of any potential difficulties with these changes?

After IATI Standard guidance moves from reference.iatistandard.org/ to iatistandard.org/, users will no longer be able to access new changes made to the content. Only changelogs will keep a record of new changes, and users will find the changelogs in the IATI-Standard-SSOT GitHub repository.

Will historic GitHub content be accessible to users?

Yes, historic content will still be available to users, as Github keeps control of commits over time.

Will there be French translations of the content moving from the IATI reference site to iatistandard.org?

The following content will be translated into French:

The following content will be automatically generated from the IATI single source of truth, and will remain in English only:

When will these changes happen?

The proposed changes are scheduled to be implemented in early March. Github repositories deletion will only occur once the guidance is moved and deployed to iatistandard.org

Get in touch

If you have any questions about these changes please email support@iatistandard.org or post a comment below.

Many thanks,
IATI Technical Team

1 Like
Changes to Github SSOT repository & developer documentation- please comment by 27 March
Invitation: IATI Standard guidance review webinar 3
Invitation: IATI Standard guidance review webinar 3
#2

Thanks for the clear breakdown.

Its probably better to archive, rather than delete GitHub repos.

5 Likes
#3

Thanks for sharing that. Would it be possible to make HTTP 301 (Permanently moved) redirects from the old URLs to the new URLs? That would have two major benefits:

  1. People with old bookmarks, links from social media/email, etc, will still be able to get to the content they’re looking for.

  2. Search engines like Google will recognise the permanent relocation and transfer over the pagerank from the old pages, so they won’t lose their search-engine rankings.

4 Likes
#4

Thanks for the update

As someone who wrote that restructuted text, I follow your quest for a another format!

However, it should be noted that we moved all guidance and documentation content to an openly accessible source (github) precisely because the previous attempt (embedded with a wordpress instance) had reached its limit for a scalable and open standard. I am therefore cautious in terms of the benefits of this proposal

  • Have other approaches been considered? There is a range of possibilities with (for example) static site generators such as readTheDocs. (both Open Contracting and 360Giving use this).

  • Is it possible to provide a preview of any site before any potential switch over?

  • I am unclear on the differences between:

and, then later:

What does this mean to the user that access the schema, schema documentation and codelists on a daily basis?

In terms of version control of our shared resources:

  • How will guidance documentation be version controlled? If content will no longer exist in GitHub, how can issues be raised, pull requests be submitted, and fixes be reviewed and merged, etc?

  • Where does this fit with the version control of the whole standard? The SSOT is meant to encompass schema, codelists, rulesets and (alas) guidance/narrative documents. How do we now govern this?

  • In terms of the last two points, I would encouraged the secretariat to look more closely at the policies developed elsewhere for normative and non-normative content. These can really help us, if we are to have a discussion about the differences between the standard to adhere to and guidance to help implement it. It’s a thorny subject - and not, imho - solved by a switch of backend

Thanks

2 Likes
#5

Its probably better to archive, rather than delete GitHub repos.

Yes, the intention would be to move these (and several old tools), to an “Archived” repository. However, this removal + move will happen further down the line.

2 Likes
#6

Would it be possible to make HTTP 301 (Permanently moved) redirects from the old URLs to the new URLs?

Yes absolutely, the intention is to make the move as seamless as possible. Redirects will be put in place from reference.iatistandard.org to iatistandard.org for both guidance and documentation pages.

2 Likes
#7

Thanks for the opportunity to feed into this consultation and for all the work already done on this. In principle it will be great for the reference pages to get a refresh and to have a consistent theme with the rest of the IATI site, as well as to improve the navigation between the two.

I agree with several of the other comments here, and would just emphasise that I think it is important that all of the IATI Standard remains version controlled (I think this should definitely include the guidance materials) and that there is transparency around changes and which version is shown at a particular time.

For these reasons, I do like the approach that Open Contracting (mentioned above) uses. I am a little bit nervous about the idea of putting all of the Standard into the database behind the Wagtail system (I assume this is what is planned?) because it could make it much harder to track changes over time.

But I guess using Wagtail it could also be possible to achieve version control and transparency, depending on the implementation – can you give us some more details on how Wagtail would determine which version of the SSOT to show - and how users would know which version was being displayed? Would this be automated, e.g. the latest version of a particular branch on Github? (I understand from all this that the SSOT would continue to be managed on Github.)

Many thanks!

1 Like
#8

Since OCDS was mentioned a few times, I’ll add reflections from our experience.

Normative and non-normative content

Our policy around normative and non-normative content made it much clearer to the community:

  • what content is prescriptive (in short: the reference pages, schema files and codelist files)
  • what content isn’t prescriptive (in short: the rest)
  • what the rules and processes are for changing each type of content

From an internal perspective, the policy also enables us to change the non-normative content more frequently and responsively (a first big update is coming soon).

360Giving is also preparing (or has already prepared) a similar policy.

Version control

It is important for us to be able to trace the history of every change (even to non-normative content) and to be able to relate those changes to discussions about them (which occur in GitHub issues in our case, or at minimum are linked from GitHub issues).

Why is it important? One simple example: We (or stakeholders) sometimes disagree with or don’t understand some old guidance, but if we’re going to change it, we need to be sure we understand how it got into the documentation in the first place.

Our most robust way to do that is to go through the change history (git blame), read commit messages, and find linked issues and pull requests (all changes are made through pull requests). Our usual way is to simply search the GitHub repository.

There is certainly more than one way to solve the problems of tracking changes and linking discussions. My understanding of earlier comments is that IATI’s solutions to these problems are not clear under the new proposal.

Technical options

For OCDS, we use Sphinx (with a Markdown extension and custom extensions) to generate a static website.

So far, we’re satisfied with Sphinx. Our documentation pages are mostly plain Markdown, except for a few Sphinx directives to render JSON snippets with syntax highlighting, CSV files as tables, and JSON Schema files as tables – and to list tables of contents and add images and admonitions.

The two proposals above are (1) JSON with embedded Jinja and (2) XML. It’s hard to evaluate these against the status quo, since both would be bespoke. Could a sample/skeleton JSON or XML file be shared to give more clarity?

Human readability

It’s not clear to me that Jinja’s templating language will be less confusing to a human reader. Also, wouldn’t the switch to Jinja require adding a lot of HTML markup to the documentation pages? I think HTML is harder for a human to read than reStructured Text (which in our usage has been mostly backticks and asterisks around words, and hyphens under headers). Nevermind that this will be embedded in JSON or XML, which aren’t human-friendly either.

Machine readability

the additional Sphinx markup added to the files can be a nuisance to read as code

What are the use cases for reading the documentation pages as code?

Without knowing the specific use cases, I think it’d be simpler for third parties to parse the generated HTML, than to try to parse the source files (which sometimes depend on other files).

Translation support

We use Sphinx to extract text from our Markdown files to be translated in Transifex. Sphinx supports this for reStructured Text out of the box. (We use Babel to do the same with JSON and CSV files.)

Under the new proposals, would new tooling need to be authored to support translation of the text in the JSON and XML files?

Summary

I hope these reflections are useful to the discussion. In short, it seems like some next steps are:

  1. Clarify how to track changes and link to discussions about changes under the new proposal.
  2. Clarify which changes are normative/non-normative and what the processes are for changing each.
    • The solutions to (1) might be different for each type of content. This is where that should be clarified. (Presumably normative content would follow an existing governance process.)
  3. Agree on a format that is sufficiently human-readable and machine-readable, and that has translation support.
    • The use cases for machine-readability need to be clarified.
4 Likes
#9

Thanks @James_McKinney - really helpful

360Giving is also preparing (or has already prepared) a similar policy.

Yes, this will be released shortly (it’s based on the OCDS policy)

We use Sphinx to extract text from our Markdown files to be translated in Transifex.

…reminds me - I posted about OCDS use of transifex a while back.

#10

Thank you all for your comments. Your feedback has been really useful and constructive and have made us revisit how we manage the content for the IATI guidance pages. Thanks to @markbrough, @stevieflow and @James_McKinney for the detailed responses. I have provided a brief response summarising our proposed change (based on your feedback) and I would contact @James_McKinney directly to go in more detail on the practices Open Contracting is following !

Single Source of Truth (SSOT)

Currently, in the SSOT repository on Github we have:

  • Codeslists
  • Extra Documentation
  • Rulesets
  • Schemas

Together, as you say, these make up the IATI Standard. We agree that these need to be accessible and version controlled. The concerns raised in response to this post were mainly around the Guidance Overview Pages that currently sit inside the Extra Documentation.

Guidance Documentation:

Over the last year the IATI tech team has been drafting and consulting on new guidance for both publishers and data users. We intend for this to replace the need and expand upon the Guidance Overview Pages, which currently live under ‘Extra Documentation’. Reflecting on your feedback we propose the following:

  1. All Guidance Overview Pages and new redrafted guidance will be managed on Github. This would allow for version control using each major version of the IATI standard. Content will then be pulled through to Wagtail. We will not manage content directly in the CMS (Wagtail), which addresses most of the concerns raised in this post.
  2. We would like to move the Guidance Overview Page and new redrafted guidance content to a new Github Guidance Documentation repo (outside the SSOT). Similar to the Guidance repo. We do need to do a bit of work on how that will be managed, which links to having more detailed conversation with @James_McKinney on normative and non-normative content . We can then update the community.
  3. We propose that the new Github Guidance and Extra-Documentation repositories include XML formatted files, following html tags. This would enable developers to easily read and convert the documentation to the required formats, whilst also remaining in a familiar and readable format, allowing for easy generation, additional customisation without the need of bespoke markup comprehension code or amendment of content as necessary. We would welcome your input on this.

We will need to change the way that markdown and sphinx works for this repository that allows it to be pulled into Wagtail. For now only one branch of V2 is needed. This does allow for a full changelog and issues to be raised on particular sections of the guidance documentation. We can then implement the same changelog, as suggested above, on Wagtail so that users know what has substantially changed since the last time they reviewed the guidance.

It will be useful to have a conversation on that with Open Contracting as well. We welcome views from others on the format questions as well.

  1. Updates to IATI Guidance pages: We also recognise there’s a need for consultation about how the Guidance Documentation pages should be updated over time. Keeping the content on Github would allow for tracking any updates in the public and allowing for changelogs.

We thank everyone for the useful responses and engagement, which did help us revisit how the Guidance pages content will be managed. We will not move on with the work until we have had the exchange on sharing practices with Open Contracting as well as a decision on the format we will be using. We will update you as this process progresses, and thank you all once again for your valuable contributions.

4 Likes
Changes to Github SSOT repository & developer documentation- please comment by 27 March
#11

Many thanks for this comprehensive and considered update @IATI-techteam!

This all seems most welcome.

A couple of points:

I’d suggest that the answer is not to go into offline discussions with one person, but an open call/webinar/show& tell discussion with other standards (I know 360Giving have recently adopted this policy, and are due to publish it) and interested community members. Perhaps this can then lead to a Working Group?

I would also understand any decisions and processes agreed in terms of the SSOT will be rolled out to include future updates to guidance material, as broadcast in this recent post. However, right now, I can’t quite see these linkages being explicitly made…

#12

Hi @stevieflow, from my perspective, it’s more feasible for me to have a quick bilateral chat to share background information, than to participate in a longer multilateral call. I have no doubt that @IATI-techteam will share any updates, which I’m sure will incorporate many inputs, not only mine. :slight_smile:

Separately, regarding my earlier comments on format, I think I got too deep in the weeds, because the choice of format seems to be primarily about solving technical challenges raised by the community-informed decision to merge reference.iatistandard.org into the main website. In that respect, OCDS has little to contribute to the conversation, as we haven’t faced those technical challenges, and we haven’t had to balance them against other considerations. (OCP uses WordPress for its main website, and Sphinx for the OCDS documentation, and they don’t integrate in any way, besides hyperlinks.)

#13

The 360Giving data standard team have now published their version of the normative vs non-normative policy…

#14

Update and further changes

Over the last month we’ve been working on converting the guidance pages into .rst files and making sure they’re displayed properly on Wagtail (a bigger job than expected). To keep things streamlined and robust there are a couple more changes we’d like to make.

Changelogs and upgrade processes

  • We had proposed to move the upgrade and changelog documentation from the IATI-Guidance repository to the IATI-Standard-SSOT repository. Doing this means we would create a copy for each page, for each version of the standard both in GitHub, and on the wagtail website. Our c.20 pages become c.200 pages on both sites. This adds work to keep them all up to date and display in a timely way.

    Instead, we propose that we create a new ‘IATI-Upgrades’ repository. This would contain details of all standard upgrades, codelist changes and related processes.

    The benefits are that we only need one version of each page on both GitHub and Wagtail (note that these pages do not need to be version controlled). This makes it easier for us to manage/ update and helps the website loading time.

Developer documentation

  • We had proposed to content manage the IATI-Developer-Documentation in Wagtail. Whilst this works, there are a few pages, such as the codelist api (currently in the IATI-Extra-Documentation repository) and old datastore guidance (currently in the IATI-Guidance repository) which we think best sit in developer documentation and ideally need to be kept in GitHub. We don’t want to keep these pages in the guidance repository as this introduces complexities when pulling the content through to wagtail.

    We propose to keep the developer documentation managed in Github, and add the codelist-api and the old datastore documentation to this repository.

    We will be taking this opportunity to update the developer documentation. The other content in the changelogs and upgrade processes will not be changing, only moving location.

We would like to check:

  • Whether there are any objections to creating a new ‘IATI-Upgrades’ repository. We would specifically like to hear from users that make use of the current Github repositories
  • That there are no large issues with increasing the content in IATI-Developer-Documentation and keeping this in GitHub We would specifically welcome views from the developer community.

Please do post your comments in the next seven days. We will then review them and take this work forward if there are no objections. Thank you for your continued input.

1 Like
Moving IATI Standard content from IATI reference site to main IATI website