 |
 |
There are currently no policies or guidelines relating to the continuing validity of old versions of the standard. There is a clear need for the earliest versions to now be deprecated. We welcome proposals as to how this should be approached and implemented.
This paper is part of the Agenda for the IATI TAG 2016 Technical Consultation Workshop
#IATI #TAG2016
The proposal for the deprecation of old versions is detailed below. We welcome comments on this from the community ahead of its scheduled discussion as part of the Agenda for the IATI TAG 2016 Technical Consultation Workshop.
Since the initial launch of the IATI standard in November 2010, 7 versions of the Standard have been developed and launched. Retaining a large number of versions requires some maintenance overhead and adds to the complexity of understanding the standard, particularly for users.
The IATI Technical Team would like to reduce the number of supported versions. As more concurrent versions have come to existence, the greater the difficulty for publishers and data users – especially for those new to the standard. Additionally from a maintenance point of view, offering multiple versions of the standard increases the resource and knowledge required.
IATI is a living and evolving standard. Therefore, there is also an argument that publishers should be encouraged and, where possible, incentivised to use newer versions which offer enhancements to the quality, clarity and usability of their published data.
An overview of all versions of the IATI Standard and usage is shown below:
+------------------------------------------------------------------------+----------------+------------------------------+------------------------------------------------------------+
| Version number | Launch date | Status of documentation site | Usage: Percentage of activities published to this version* |
+------------------------------------------------------------------------+----------------+------------------------------+------------------------------------------------------------+
| 1.01 | November 2010 | Dormant & archived site | 2.1% |
+------------------------------------------------------------------------+----------------+------------------------------+------------------------------------------------------------+
| 1.02 | December 2012 | Dormant site | 0.4% |
+------------------------------------------------------------------------+----------------+------------------------------+------------------------------------------------------------+
| 1.03 | September 2013 | Dormant site | 16.0% |
+------------------------------------------------------------------------+----------------+------------------------------+------------------------------------------------------------+
| 1.04 | May 2014 | Actively updated on changes | 11.5% |
+------------------------------------------------------------------------+----------------+------------------------------+------------------------------------------------------------+
| 1.05 | October 2014 | Actively updated on changes | 3.2% |
+------------------------------------------------------------------------+----------------+------------------------------+------------------------------------------------------------+
| 2.01 | January 2015 | Actively updated on changes | 61.7% |
+------------------------------------------------------------------------+----------------+------------------------------+------------------------------------------------------------+
| 2.02 | December 2015 | Actively updated on changes | 3.8% |
+------------------------------------------------------------------------+----------------+------------------------------+------------------------------------------------------------+
| Activities published with no version specified, or an invalid version* | | | 1.3% |
+------------------------------------------------------------------------+----------------+------------------------------+------------------------------------------------------------+
Data correct at 9 September 2016
The above table shows that the most popular versions are 2.01 (which comprises 62% of all activities published), followed by 1.03 (16%) and 1.04 (12%). Almost a fifth (18%) of activities are published to versions with dormant documentation sites. This raises concerns about how these publishers access information about codelists, since documentation websites for versions below 1.04 are not updated, for example upon additions to non-embedded codelists.
Option 1 (recommended): Support ends 2 years after the release of a higher integer
An integer and all its versions are deprecated not less than 2 years following the introduction of a new integer. This gives publishers a time window to scope and adjust any internal systems in order to benefit from enhancements to the standard.
Option 2 (the next best alternative): Support the 3 most recent versions only
The IATI Technical team will support the most recent 3 versions of the standard. However, this shall always include the latest version of the previous major version in order to give grace to publishers who are in the process of upgrading to the latest major version. The depreciation process for outdated versions would begin two months after the successful go-live of a new version.
With the present latest version being v2.02, this proposal implies support will continue for versions 1.05, 2.01 and 2.02 only. Versions 1.04 and below would be subject to the deprecation process proposed below, to commence within 4 weeks (from the date this proposal is accepted) leading to the end of formal support.
Looking to the future, and as an example of the above principle, at the time when version 2.03 is launched, the proposal implies support will remain for version 1.05, as it will continue to be the latest version of the previous major version.
Deprecation process
The process to deprecate applicable a version would be as follows. A version will become formally deprecated when it is removed from the Version codelist.
+-------------------------------------------------------------------------------------------+--------------------------------+
| Action | Time until formal depreciation |
+-------------------------------------------------------------------------------------------+--------------------------------+
| A notification will be posted on IATI communication channels | 3 months |
+-------------------------------------------------------------------------------------------+--------------------------------+
| IATI Registry publishers using soon-to-be deprecated versions will be contacted directly. | 3 months |
+-------------------------------------------------------------------------------------------+--------------------------------+
| Reminder communications will be sent to affected IATI Registry publishers | 1 month |
+-------------------------------------------------------------------------------------------+--------------------------------+
| The deprecated version will be removed from the Version codelist. | 0 |
+-------------------------------------------------------------------------------------------+--------------------------------+
Implications of using a deprecated version
At the point when a version is deprecated, publishers may choose to continue publishing at deprecated versions at their discretion albeit with impacts on their Dashboard comprehensiveness scoring and support - as detailed below.
IATI tools
The use of deprecated versions within official tools maintained by the IATI Technical Team will be phased out. Priority will be given to removing any functionality which outputs IATI XML in deprecated versions. Support for functionality which uses IATI XML in deprecated versions as an input will be removed when practical.
Helpdesk support
The IATI Technical Team will prioritise helpdesk support for data in current versions. This means no guarantees can be made to give any help or advice to publishers and data users who are seeking to publish or interpret data in deprecated versions.
Archiving of documentation
Where available and practical, achieved deprecated versions will be available in source format - for example on the IATI-Standard-SSOT Github repository. Snapshot versions of documentation websites will be taken and hosted (as plain HTML) on an archive.iatistandard.org subdomain.
Anyone publishing using a depreciated version should be aware that there are placing significant barriers to usage of their data. Users of raw data and automated IATI data warehousing products are unlikely to be support non-current versions.
Adjustment of publisher statistics
Deprecated versions of the IATI Standard will not appear on the Version codelist, therefore publishers using deprecated versions will be regarded to be publishing invalid data when scores are calculated for the Dashboard Comprehensiveness (Core) measure of Version.
Please note, upon deprecation of the last versions of an integer series (for example 1.05), publishers continuing to use deprecated versions are likely to suffer reduced Publisher statistics scores as publication using deprecated styles (for example many v1.x codelists) will count as invalid data.
Upon consensus, the above policies for deprecation will need to be approved by the IATI Members Assembly (next scheduled for Summer 2017). After this, the proposal will take effect in January 2018.
If agreed, this implies that versions 1.01 to 1.05 would be subject to the agreed deprecation process proposed above, commencing at this point.
While it is reasonable to formally encourage or prevent publishers to stop publishing new data on old versions of the standard, care needs to be taken with regards to data which has already been published in a given version of the standard.
For example, say a set of activities has been published using version 2.01 of the standard. Upon the release of version 2.04 or 3.01, support for this version of the standard would be deprecated. Functionality in IATI tools and helpdesk support for utilising this data would then be removed or reduced. This is of even more immediate concern to the ~28% of activities currently published in versions of the standard that would be deprecated upon approval of this policy.
With this in mind, consideration needs to be had with regards to how:
With regards to removing support for deprecated versions in tools that utilise IATI data…
With regards to adjusting publisher statistics…
I would support option 1 for a two year support window after a new integer version is released.
I would not support removing past versions from the version codelist, because, as Hayden notes, this would cause historic and archived data to cease to validate. Instead, I would suggest:
I would also encourage investment in conversion tools when new versions are released - as in a number of cases, where changes are structural, but not semantic, it is possible to programmatically map from old versions to newer versions.
I tend to agree with @hayfield and @TimDavies. There is another consideration regarding the use of old standard versions: wouldn’t it be be better to slow down on introducing new integer upgrades of the standard? Given the fact that there are a lot of quality issues with the existing publications , shouldn’t publishers concentrate on getting their data right, instead of concentrating on conversion of their data to a new version of the standard?
That would mean slowing down on standard development, which would also mean that older versions of the standard can be longer supported. Lets focus for a while on the use of what we have instead of converting from one version to another.
@Herman I agree
Based on the discussion above the suggestion for Standards Day will be presented as:
Proposal
Proposed Introduction.
I would propose that we update the deprecation process, from saying:
To saying
I agree that the codes should not be removed, though in what way would the deprecated code be marked as deprecated?
status and withdrawal-date attributes to Codes.active and withdrawn as Codelist statuses.status attribute is defined as a string.As such…
status="withdrawn" / withdrawal-date="xxx")?status may be any string OK?status attribute on a Code (v2.03 change as Guideline / v3.01 as Rule)?xsd:string to become particular values (Embedded Codelist as part of the Schema - v3.01 change)?(though it may be worth splitting this into a separate discussion)
I know I should remember, but… Was this discussed at Standards Day? I don’t see anything in the notes. If it was, what was the outcome?
If the proposal described here was agreed upon, then that would mean v1.0x is now deprecated. Is that the case, @IATI-techteam?
There was not consensus in this area.
In particular, there were concerns around the typical 5-7 year lifetime of systems within larger organisations and the impact that deprecating old versions under the proposed timeframes (over a period of around 2 years) would have.