Python 2.x Deprecation Notice

Notifications
#1

As part of the work planned for this quarter, the Tech Team will be upgrading the programming language used across a number of our tools.

As of 1st January 2020, Python 2.x will no longer be supported globally. To ensure that our tools remain functional and up-to-date, we are commencing upgrades now.

Standard tools affected will be:

As a part of a wider reconfiguration of tools, the following internal tools are being considered for deprecation, and therefore arenā€™t a priority for upgrade:

The Datastore is currently undergoing a major rebuild using Python 3.x as default, so no action will be taken on the current repository. The new Datastore is being released this quarter.

For developers, the upgrade means that virtual environments must be configured to use Python 3.x and dependencies upgraded. We urge that any contributions use Python 3 complicit code.

Tool documentation will be updated with instructions for using Python 3 virtual environments.

1 Like
#2

Hooray for python 3! :tada: This is great ā€“ hopefully a good opportunity to make some incremental improvements to some old codebases.

One thing to note: CSV2IATI definitely should not be upgraded to python 3, since it was decommissioned last year. I donā€™t know about the missing activity checker ā€“ I donā€™t know if that is in use.

#3

Iā€™ve converted IATI Registry Refresher from PHP to python3, in case thatā€™s useful.

1 Like
#4

Andy, thatā€™s great, thank you! Weā€™ll look into the registry refresher.

CSV2IATI definitely should not be upgraded to python 3

This has indeed been decommissioned, however there is potential that some publishers are using this tool. So, for now, weā€™ve marked this upgrade as a low priority and are keeping it on our radar whilst we scope who might still be using it and consider steps for rounding off the decommission completely.

1 Like
Python3 Upgrade: Proposed Publishing Statistics Changes (comment by 15th April)
#5

Python Software Foundation wrote about this today.

The IATI python upgrade featured in Aprilā€™s technical update, but not Julyā€™s.

I understand a bunch of work has happened on this since April, which is brilliant. Given that, would it be possible to get an update from @IATI-techteam on current progress & work remaining, with respect to the repositories referenced above?

Thanks

#6

Hello Andy, thanks for re flagging this. We were going to update at work complete but an interim update wonā€™t hurt.

As of now, the tools that have not yet been upgraded to Python 3 are:

  • IATI Dashboard: now separated from Publishing Statistics

  • IATI Publishing Statistics: stand alone; did not exist at the time we wrote the original post

  • IATI Stats: hard dependency on the split work that happened above

  • IATI Datastore: soon to be replaced by the new one

  • IATI Registry Refresher does not really fit this list as itā€™s currently in a different language and does not require an upgrade

Dashboard, Publishing Statistics and Stats will be our next upgrade focus, followed by the Datastore (based on the new Datastoreā€™s progress).

Registry Refresher is at the bottom of our priority list because of it not really being affected by the deprecation deadline, being in PHP so weā€™ll see about this as we move forward and complete more critical work.

4 Likes
#7

Thanks for this update. Sounds like a lot of progress has been made on this, which is great.

^^ Looks like these two havenā€™t been converted to python 3. Was the conclusion of the scoping exercise mentioned above that these wonā€™t be converted?

#8

CSV2IATI has been deprecated and put offline, no further work will happen on it.

Missing Activity Checker has been deprecated as it did not seem to be in use, no further work will happen on it.

Both codebases will remain available. The Tech Team is available in case further contact is needed.

1 Like
#9

Is there also a plan to upgrade the IATI Registry to python 3?

I realise the registry code is managed by an agency. Is upgrading to python 3 on their radar?

It looks like the next minor version of CKAN will have python 3 support, so I guess that is a big part of the upgrade.

#10

Hi Andy, some of the background work has been done to upgrade the custom ckan-iati code that currently exists, and the agency weā€™re working with expects to finish and deliver this work by year end.
It appears that CKAN itself will support python 3 as of version 2.9, but that might not come out until the first quarter of 2020. So while the bespoke IATI code will support python 3, the ckan base will not until said update is rolled out.
If anything else comes up regarding this Iā€™ll make sure to update this thread.

2 Likes
#11

Is there an update on the python3 upgrade? This was a focus for the tech team in 2019, and I see it was covered in the Q1 2020 technical update but not since.

A few things to note from looking at the open source code:

  • Thereā€™s an approved but unmerged PR to upgrade the IATI-Dashboard to python3. The live branch is still using python2.
  • Similarly for the IATI-Publishing-Statistics code, thereā€™s a draft PR that includes the python3 upgrade. The master branch is still using python2.
  • The live branch of the IATI-Stats code is still using python2.

Itā€™s possible python3 is in use on production, which would be great. If so, the public code on github should be updated to reflect that.

Re. the registry: it looks as though CKAN v2.9 was released two weeks ago, with python3 support. So upgrading to that version would make it possible to upgrade the registry to python3.

#12

Derilinx are currently working on the CKAN python three upgrade work, so we hope this will be deployed soon.

Weā€™ve got a few people from the team on holiday currently, so we will give an update about Publisher Stats and the Dashboard in a couple of weeks time. Thank you for your patience.

1 Like
#13

Update on the Registry work:

  • The IATI Registry is currently at CKAN 2.8.5. Changelog with ckan 2.8.5 can be accessed in here.
  • The CKAN Python 3 upgrade will then happen in a two step approach:
    Update to ckan 2.9 (Python 2) - major change: pylons have been removed and now everything is in flask.
    Update on ckan 2.9 to (Python 3)- migrate extensions to python 3 and deploy in staging which should undergo exhaustive testing.
    Details and updates will added on Github and we will keep the community posted on Discuss as well.