Really useful bit of analysis â thanks @IATI-techteam.
The above post doesnât say whether v1.0x datasets will be included in the v2 (ânewâ) IATI Datastore. Can I take this opportunity to again flag IATI transformer (described in this discuss post) which I think could be of use here. One option would be for the v2 (ânewâ) IATI Datastore to run any v1.0x IATI data through IATI Transformer as a pre-processing step. This is really trivial to do â you can even do it on the fly. So for instance, instead of using:
Apologies if it was not clear from the above post; we can confirm that v1.0x datasets will not be consumed by the new DataStore.
It is important to bear in mind that the new Datastore is not a database; rather it is a curation and display of files that are hosted on the IATI Registry. As such, unless publishers themselves host the transformed file after using your transformer tool and link back to the Registry, the dataset will not be picked up by the DataStore.
It is for this reason that we cannot ourselves run all v1 data through the transformer tool. We do however appreciate its availability as a public tool that can be used by publishers who are undergoing the v2 transition and we have certainly raised it as a potential transition option for the publishers that we are in touch with.
Apologies, youâre absolutely right. It was clear â Iâm afraid I didnât read it carefully enough.
Okay, great! This is a good decision in my opinion. I donât see how the one-off upload would have worked (in terms of data ownership) so Iâm relieved to hear this has changed.
Iâm not sure I understand what this means. The datastore performs an ETL (Extract, Transform, Load) cycle. I think the âLâ bit means it is a database. The âTâ bit means it already performs a transformation/normalisation step. Iâm just suggesting adding to that transformation/normalisation step.
Is there an easy (or even a known) way to check how many of those v1 files contain active activities, i.e. activities that have not yet reached their predicted end date, as those are the things that would be most key for me to keep in the new datastore - if you could share a list of the urls for the from the great analysis above then I might be able to figure this out myself?
Hi @matmaxgeds, if I was to do the above (someone more technical than me probs has a better approach) Iâd use the current IATI datastore.
download the activities for all publishers we know are using v1, and that have activities with an end date greater than today.
filter out all v2 activities (ones with numeric codes).
That should provide you with a list of v1 activities that are still active. Or, use the activity-status attribute as a filter as thereâs some debate whether itâs updated more regularly than the activity dates.
Hah, @andylolz, I âhope it helpsâ is underselling it somewhat
Based on this, I feel that not having v1 data in the new datastore would be a substantial loss until we can get this figure down substantially.
I am aware that DI are working hard with many of the key publishers still using v1
Suggested ways forward that I am aware of in the interim (until the publishers update their systems):
Not implement the v1 aspects of validation yet
Keep the existing datastore online
Have the new DS use the iati-transform tool to convert them to v2 data
Republish them all using iati-transform and a secondary publisher (although they still wouldnât be available in d-portal and several other tools)
Explore how many of these activities are actually closed but where the data has not been updated
In light of my recent commitments to being a good discuss citizen - how about we have a skype/similar meeting for anyone who is interested - we could discuss:
What are the strategic/political decisions that are affecting this?
How this fits in/alters current Secretariat/DI workplans - and what scope there is for tweaking them?
What is a reasonable number to aim for in terms of the abandoned data - or whether we should just drop this data?
What the community can contribute to making this work - and how best to do that in a coordinated way?
Whether the different technical options really work?
So - if keen, please let me know on this thread, and I will setup a doodle to find a time that works for everyone - or maybe this process is better managed by DI/UNDP - please say if so.
Thanks for your thoughts; our response is below. Weâre happy to set up a call if that would be helpful.
A few specific responses to your points:
The way the system has been built is that the Validator checks if a file is 2.0x compliant; if not then it doesnât parse it onto the DS. This means there is no way to âturn it offâ.
We would have to turn off all validation to allow v1 data into the DS or weâd have to delay the launch of the validator and the DS in order to build a v1 validation service, which would add considerable cost.
There is an agreement to keep the existing DS in place until the end of 2019.
We discussed with our board focal points the problems of having data that doesnât have an owner; this gets into the core of how IATI is designed.
The way the whole IATI data system functions is that a publisher owns and hosts their own data. If we (as the tech team) start to transform and host peopleâs data this is a fundamental change to how IATI publishing works - a big change of mandate for the IATI secretariat and something that was agreed with IATI Board focal points wasnât the right direction for us to take.
One example of where it would get complicated is when a publisher chooses to update their data. We would then be left with two versions in the system: the data transformed and held under a fake account as well as the data owned by the publisher that they have later handled.
The IATI-transform tool will work in some cases but not in all. Not all mandatory elements and attributes in version 2 of the IATI Standard are required in version 1. The IATI-transform tool can turn existing version 1 data into version 2, but it cannot create data.
For example, it cannot provide an activity status, or an iso-date for an activity where none was originally given. That information needs to be added by the publisher.
Hopefully the more detailed explanation above makes it a bit clearer as to why the IATI secretariat canât re-publish data that we donât own and how there isnât something we can âturn offâ.
In terms of what we are doing behind the scenes; we have been in touch with all the publishers that this pertains to offering our assistance for them to upgrade.
From our analysis only 61 publishers in v1 are still active. We have reached out to all of these organisations and have been pleased with the level of engagement so far. The vast majority of these publishers only have their Organisation file in v1 and/or use AidStream to publish their data, meaning that the transition process is not particularly strenuous.
Getting the organisations to engage is the biggest barrier! We would really welcome your thoughts and ideas on how we can get better engagement.
Also, itâs important to note that version 1 data will also still be available from the Registry.
As above, weâre happy to have a call with you and any other community members if you think it would be helpful.
Agreed that neither of the approaches described here are desirable. One alternative approach would be: if the dataset is v1.0x, attempt to transform it (i.e. pass the dataset through an XSLT.) Then continue as before i.e. pass the transformed version through the validator.
Personally I am pretty happy with this change. The analysis indicates that all major (âtargetedâ) organisations will hopefully be migrating to v2 so we will not miss their data. There are two exceptions: Scottish Government and JICA. I think it is OK that JICAâs data wonât be migrated as my understanding is that it is CRS data anyway â so it would be preferable to move them to a better publication process instead. That leaves only the Scottish Govt.
On @matmaxgedsâ point, I think the number of activities overstates a bit the issue here â thanks to @andylolzâ IATI Kit, I did a quick analysis based on the great analysis spreadsheet @IATI-techteam shared at the top of this thread â see here for details (here for code), but in summary â looking only at activities:
84% of activities are from targeted publishers
of the 62,056 activities from targeted publishers, all but 2,886 (for Japan and Scotland) should become available in v2.
there are only 12 untargeted V1 publishers with over 100 activities
I guess there could be other effects in terms of enabling traceability if the data from some of these smaller organisations is no longer so easily accessible, but I think that is OK if the data is normally quite stale.
Maybe I have got something wrong in my analysis, but I think this indicates that the vast majority of activities, and almost all from targeted organisations, should be covered by organisations that will convert soon.
So maybe need to keep an eye on the remaining targeted publishers, but in the meantime, I think this looks good @IATI-techteam!
(As an aside: I think it might be good to have a conversation about what we do in the future in similar cases, where there are a large number of organisations with stale data.)
We should remind ourselves of how productive it is to drill into the issue we first started to discuss in January, whilst also minded of the need for this key infrastructure to be delivered and available to us as planned.
Thanks to this latest discussion, I think we now have further evidence to advise the @IATI-techteam that this decision is productive. As @markbrough also highlights, there are further issues to discuss and deliberate on - but letâs make sure we consider those in away from this decision.
Thanks @JoshStanley great to get the update - are you able to update progress against the targetted organisations on your useful google doc so we can see who we are still at risk of losing?
Of our initial list of target publishers, the following have already fully transitioned:
Asian Development Bank
Catholic Agency for Overseas Development (CAFOD)
Climate Investment Funds
European Commission - Humanitarian Aid & Civil Protection
The Global Fund to Fight AIDS, Tuberculosis and Malaria
European Commission - Service for Foreign Policy Instruments
Several more are in the process of transitioning and some have advised that they will upgrade alongside their next publication.
We will provide another update closer to the deprecation date (before the end of June); if you are interested in the progress of any specific publishers then do feel free to get in touch with the publisher directly (Iâd be happy to link you with the contacts that we have).
The following is an update on publishersâ transition to version 2 of the Standard, now that the v1 deprecation date has passed:
âTargeted publishersâ
The following is a list of âtargetedâ publishers which have successfully transitioned all of their data to v2 of the Standard:
Asian Development Bank
Catholic Agency for Overseas Development (CAFOD)
CDC Group plc
Climate Investment Funds
European Commission â Humanitarian Aid & Civil Protection
European Commission â Service for Foreign Policy Instruments
The Global Fund to Fight AIDS, Tuberculosis and Malaria
UK â Home Office
UNITAID
We are also still actively engaging with most of the other âtargetedâ publishers as they continue to go through the process. Some of these require just a few more files to be updated before they re-publish at v2, while others have completed the transition process and plan to publish at v2 in their next publication.
AidData are a slight anomaly in that there were limitations to their data (as geo-location data was their primary focus when they first published) which meant that some activities were missing mandatory fields that were required to update to v2. In this instance, we worked with them to ensure that the activities that do have the mandatory information were transformed, while the rest were at version 1.
âNon-targeted publishersâ
In the lead up to the deprecation date (30 June 2019), we sent a number of bulk emails to both the âactiveâ non-targeted publishers and the ânon-activeâ non-targeted publishers. The âactiveâ non-priority publishers received emails that were tailored to certain features of their data (e.g. only Org file at v1 vs. Org and Activity file at v1; AidStream users vs. internally output XML etc.) while the ânon-activeâ non-targeted publishers were invited to contact the technical team service desk for advice and help with transitioning.
We liaised with a number of non-targeted publishers following this (and will continue to do so) and will provide a more detailed update about the number of v1 datasets/activities as a whole in the coming weeks.
Thanks loads for the continued updates @JoshStanley - are the ones who have not changed going to be removed from the dashboard? And if so, are we under 1,000 again?
Since last time Josh did a data pull 418 files have been moved from v1, equating to 30,213 activities. You can see the details on the July 2019 - Data tab
@matmaxgeds the plan in the future is to remove v1 data from the Dashboard. However, this wonât change the count of IATI publishers.
