PLIP: Port Plone to Python 3

[pbauer]

Proposer: Philip Bauer

Seconder: Alexander Loechel

Abstract

Support Python 3 and 2.7.

Plone 5.2 should run on Python 2.7, 3.6 and 3.7

For Plone 6 Python 3 should be the default but it should still run on 2.7.

Motivation

On 2020-01-01 support for Python 2.7 ends. Migrating to Python 3 is a requirement to ensure the continued success of Plone.

Assumptions

Most of the dependencies of Plone (Zope, ZODB, CMFCore, GenericSetup, DCWorkflow, RestrictedPython...) already run on Python 3 or are actively being ported.

Proposal & Implementation

Keep the functionality the same but allow it to run in Python 3 as well as in Python 2.7.

To achieve compatibility with both versions use six, e.g. for changed imports.

In most cases incompatible code can be replaced by code that works in both versions.

Whenever a change is needed the code should be changed in such a way that Python 3 is the default and 2.7 is the fallback that can be removed by deleting complete lines. This should make it easier to drop support for 2.7 in the future.

Approaches to porting Plone to Python 3

There are five strategies to porting Plone to Python 3. We need to follow all of them to make it in reasonable time:

1. Migrate packages that have no dependency to CMFPlone and test them in Python 2 and 3. For most packages from the plone-ecosystem this is already done but some are not finished yet (e.g. GenericSetup, DCWorkflow, plone.rfc822).
2. Prepare packages for Python 3 without being able to test them in Python 3. This is already done for all packages with the help of sixer or python-modernizer. See Start migrating Plone to Python 3 #2217 for a list of pull-requests that did this.
3. Start Plone on Python 3 and fix whatever breaks. I started down that path at the Alpine City Sprint in January 2018 and by now (March 30th) Plone starts up with Python 3 and you can create a instance and play around with it. Many features still do not work, most noticeable static resources and add- and edit-forms. But a running instance allows parallelizing work (remote and at sprints). We can continue this work in parallel by manually testing features and fixing issues as they appear.
4. Port plone.testing and plone.app.testing to Python 3 and WSGIPublisher. Work on this was started at the Alpine City Sprint in 2018 but is not finished. When this task is done we can start with approach 5:
5. Tests in Python 3 and fix all breaking tests.

All changes must result in pull-requests that are tested with Python 2.7 and eventually in Python 3.

Python versions

Plone will continue to support Python 2.7 until a PLIP to drop support for Python 2.7 is written and implemented. Continued support for Python 2.7 beyond 2020-01-01 will ensure that everyone can upgrade choosing their own speed but are not necessarily stuck on a old version of Plone.

Plone will support Python 3.7 and 3.6. Python 3.7 will be released on 2018-06-15 and will be supported until 2023. 3.6 is supported until 2021-12-23 and 3.5 is supported until 2020-09-13.

Python 3.4 has its end of life (EOL) on 2019-03-16 and it is not worth the effort to support it for the limited time it still has. Python 3.5 EOL is 2020-09-13.

See the Python release- and support-roadmap at https://devguide.python.org/#status-of-python-branches

ZServer, WSGI, FTP and WebDAV

There are plans to reimplement ZServer for Python 3 on top of aiohttp at the Sauna Sprint in July 2018. ZServer is not yet ported to Python 3. Since ZServer holds support for FTP and WebDAV these features will need to also be implemented there or replaced.

Currently plone.recipe.zope2instance already supports waitress and uwsgi. The WSGIPublisher in Zope will probably be the default publisher.

When running in Python 2 the old ZServer will still work but is not available in Python 3.

(see discussion below)

ZODB

The ZODB is compatible with Python 3 but a DB created in Python 2.7 cannot be used in Python 3 without being modified before.
There are currently two approaches to such a migration that need to be evaluated: https://github.com/zopefoundation/zodbupdate#migration-to-python-3 and https://zodbpy3migrate.readthedocs.io/

Archetypes deprecation and removal

Plone running on Python 3 will not support Archetypes because it makes no sense to maintain two frameworks and probably nobody will invest the time to migrate AT to Python 3.

The following packages should be deprecated:

• Products.Archetypes
• Products.ATContentTypes
• plone.app.blob
• plone.app.collection
• plone.app.imaging
• plone.app.folder

As far as I can see the only code from them we need to move is GopipIndex (the index for sorting against getObjPositionInParent) from plone.app.folder.

On Python 2 Archetypes will still work but needs to be added as a dependency (maybe as a extra).

Upgrade-Process

The following is half-educated guesswork:

1. Upgrade a existing instance to Plone 5.2 while still in Python 2.7
2. Remove addons that are not compatible with Python 3.
3. Run upgrade-steps to prepare the switch to Python 3.
4. Migrate the ZODB Database with the help of https://github.com/zopefoundation/zodbupdate#migration-to-python-3 or https://zodbpy3migrate.readthedocs.io/
5. Build and start Plone 5.2 with Python 3
6. Run additional upgarde-steps in Python 3 (optional)

Timeline

The real timeline depends a lot on other PLIPs and the momentum that the community can muster. I nevertheless suggest to aspire to follow a timeline since 2020-01-01 is a hard deadline that is beyond our influence. Having no stable release on Python 3 some time before that will seriously impact Plone.

• November 2018: Final release of Zope 4
• November 2018: First alpha-release of Plone 5.2 (by the time of the conference).
• January 2019: Final release of Plone 5.2

This will allow people to test and migrate some addons before 2020.

I do not think that waiting for Plone 6 with this PLIP is an viable option since Plone 6 will most likely depend on the state of pastanaga/plone-react and having two hard dependencies for a mayor release is a great recipe for delays.

Deliverables

• Provide a buildout that allows to build Plone with Python 3: https://github.com/plone/buildout.coredev/blob/5.2/py3.cfg
• Document best-practices for porting Plone to Python 3 while keeping support for 2.7 (Best practices for making code compatible with Python2 and Python3 #2184)
• Document building and running Plone on Python 3 (Document Python 3 buildout #2041)
• Document deploying Plone with WSGI
• Port the testing-infrastructure (plone.testing and plone.app.testing) to Python 3
• Port packages without dependecy on CMFPlone to Python 3
• Get Plone to run on Python 3
• Deprecate Archetypes and remove it from tests in Python 3 (Deprecate Archetypes #2390)
• Port plone.testing and plone.app.testing
• Setup our test-infrastructure to test all pull-requests against Python 2.7 and 3
• Fix all tests in Python 3
• Document upgrading Plone to 5.2 with Python 3
• Document and test migrations for existing Projects and the ZODB
• Provide and document replacements or alternatives for FTP and WebDAV
• Document best-practices for porting Addons (https://github.com/plone/documentation/blob/5.2/manage/upgrading/version_specific_migration/upgrade_to_python3.rst)
• Test performance in Python 3 (Test performance of Plone 5.2 in Python 3 and 2 #2635)
• Upgrade the Plone Trainings to mention and eventually use Python 3 (Mastering Plone works with Python 3)

Risks

Tough

• If we do not pull this off our world will be coming to an end.
• The upgrade-procedure is not as easy as clicking a button and waiting for upgrade-steps to finish.
• Migrating a existing ZODB from Python 2 to 3 still experimental, can be tricky and could lead to data-loss.
• So far there is no solution for FTP and WebDAV support. Simply dropping support for these might turn away users that rely on that.

Medium

• Early-adopters of Plone on Python 3 will have a much smaller ecosystem of addons that already supports Python 3. Luckily porting addons is really easy unless they use Archetypes.

Easy

• Compatability of addons will be more confusing since they not only need to state which Plone-Versions they support but also which Python-Versions.
• Supporting two Python versions might lead to confusion as to which Python-Version is recommended.
• Continuing support for running Plone on Python 2.7 might lead to the expectation that the Plone community will provide security support for Python 2. We need to be explicit that people who still use Python 2.7 after 2020-01-01 will do so at their own risk.

Participants

Leads: Philip Bauer (@pbauer), ...

We need separate leads for many of the deliverables (e.g. plone.testing, ZODB-Upgrades, FTP/WebDAV etc...). Please consider championing one of the deliverables and add your name here.

Current Status

As of now (March 30th) Plone already starts up with Python 3 and you can create a instance and play around with it. Many features still do not work, most noticeable static resources. But this foundations allows parallelizing work.

How can I help?

• Pick any of the tasks in https://github.com/plone/Products.CMFPlone/projects/4

• Come to one or more of the upcoming sprints that are dedicated to Python 3:

  • Zope 4 Welcome Sprint, May 16.-18. 2018 in Halle, Germany: https://www.meetup.com/de-DE/Zope-Sprint/events/248187180/
  • Buschenschanksprint, June 11.-15. 2018 in Berghausen, Austria: https://community.plone.org/t/buschenschanksprint-2018-11-6-15-6-2018/6154
  • Blue Bear Sprint, September 3.-7. 2018 in Bern, Switzerland
  • ...

[datakurre]

@pbauer Just FYI that we begin reimplementing ZServer for Python 3 on top of aiohttp (unless we find better base) at Sauna Sprint in July 2018. We are serious, because we rely heavily on current asyncore base of ZServer and async patches for wsgi don’t look interesting when asyncio could be an option.

[fulv]

Just pointing out that the deliverable *Document best-practices for porting Addons* in turn depends on several of the other dependencies, because if you want to port an Add-on you need a 5.2 build that can run on Python 3, and you need all the testing infrastructure and plone.testing and plone.app.testing to work on Python 3.

…

On Fri, Mar 30, 2018 at 10:43 AM Asko Soukka ***@***.***> wrote: @pbauer <https://github.com/pbauer> Just FYI that we begin reimplementing ZServer for Python 3 on top of aiohttp (unless we find better base) at Sauna Sprint in July 2018. We are serious, because we rely heavily on current asyncore base of ZServer and async patches for wsgi don’t look interesting when asyncio could be an option. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2368 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAcHxr5QETTBzryM1rQkEje_cU20lfIRks5tjm6-gaJpZM4TB4fz> .

[pbauer]

@datakurre Great to hear, I totally forgot about that. I updated the info in the PLIP. Is there already a ticket to track the progress and the discussion that I can link to?

[pbauer]

Also: I'm looking for a partner in crime for this. See
https://community.plone.org/t/porting-plone-to-python-3/6226

[jaroel]

Awesome, the p2.7 support is a BIG plus IMHO!

Is there a specific reason to support 3.5, or does that just happen to come OOTB with six?
AFAICT 3.6+ should be good enough, maybe 3.7+ even; we probably won't finish this before 3.7 is stable.

[datakurre]

@pbauer Not yet. We know better only in the summer once we have POC’d it.

[pbauer]

@jaroel Yes, there is nothing to be gained from supporting (and testing) 3.5. We should support the last two versions only. 3.6 is what I'm using now and I will switch to 3.7 when it is released.

[gforcada]

IMHO as we are doing the porting now I would stick with 3.6 for now, there's no much gain to keep our CI matrix more complex for the sake of it... As we have a steady working version on 3.6, we can add the +3.6 python versions on the CI matrix.

I can try to work on the CI bits... one thing that I would like to work on would be on catching regressions, with all the six porting for example, if new pull requests are not tested for regressions, we can endlessly play hide and seek.

[pbauer]

@gforcada can you please create a PLIP-job for this in jenkins? The job should at first only follow the steps in #2041 (comment). Running tests is not yet possible. That would help us catch errors from merged or broken branches.
As a bonus it would be nice to have the job also start the instance and catch the output from the terminal but I don't know how to stop it again after the startup.

[gforcada]

@pbauer there you go: https://jenkins.plone.org/view/PLIPs/job/plip-py3 it is already complaining about Products.statusmessages branch not being found 😄

[ale-rt]

It should be fixed by this PR plone/buildout.coredev#452: the Products.statusmessages branch has been merged to master

[pbauer]

@gforcada thanks for the jenkins-job. The buildout works fine now. But the test itself makes no sense: ./bin/instance run test.py

The instance-script cannot work since it pulls in ZServer and other stuff that does not work in Python 3. Same goes for the test-script since AT is a test-dependency for most packages.
Can we have something that starts up Plone like this and stops after 10 seconds:

timeout 10 ./bin/wsgi.py etc/waitress.ini

It should pass if the last line of output is something like Serving on http://0.0.0.0:6543.

For me the output looks like this:

$ timeout 10 ./bin/wsgi.py etc/waitress.ini
Class Products.PythonScripts.PythonScript.PythonScript has a security declaration for nonexistent method 'manage_historyCopy'
Class Products.PythonScripts.PythonScript.PythonScript has a security declaration for nonexistent method 'manage_beforeHistoryCopy'
Class Products.PythonScripts.PythonScript.PythonScript has a security declaration for nonexistent method 'manage_afterHistoryCopy'
Class Products.PythonScripts.PythonScript.PythonScript has a security declaration for nonexistent method 'manage_historyCopy'
Class Products.PythonScripts.PythonScript.PythonScript has a security declaration for nonexistent method 'manage_beforeHistoryCopy'
Class Products.PythonScripts.PythonScript.PythonScript has a security declaration for nonexistent method 'manage_afterHistoryCopy'
Class Products.CMFCore.PortalContent.PortalContent has a security declaration for nonexistent method 'manage_FTPstat'
Class Products.CMFCore.PortalContent.PortalContent has a security declaration for nonexistent method 'manage_FTPlist'
Class Products.CMFCore.PortalContent.PortalContent has a security declaration for nonexistent method 'manage_FTPstat'
Class Products.CMFCore.PortalContent.PortalContent has a security declaration for nonexistent method 'manage_FTPlist'
WARNING:Init:Class Products.PythonScripts.PythonScript.PythonScript has a security declaration for nonexistent method 'manage_historyCopy'
WARNING:Init:Class Products.PythonScripts.PythonScript.PythonScript has a security declaration for nonexistent method 'manage_beforeHistoryCopy'
WARNING:Init:Class Products.PythonScripts.PythonScript.PythonScript has a security declaration for nonexistent method 'manage_afterHistoryCopy'
Serving on http://0.0.0.0:6543

Once the testlayers in plone.app.testing are ported to python3 we can start proper testing for real.

[ebrehault]

@pbauer the @plone/framework-team has approved this PLIP.

[pbauer]

Here comes a quick status-update before the sprint in Halle next week: We've made great progress and most features of Plone work in python 3.

A couple of core-features still need some love (e.g. login in Plone, adding files and images, schema-editor etc.). I'm optimistic that we can make good progress on many of these during the sprint.

We even started testing in jenkins (https://jenkins.plone.org/view/PLIPs/job/plip-py3/). For now we only run the tests of plone.app.contenttypes but for those we already get: Total: 280 tests, 18 failures, 4 errors and 119 skipped in 1 minutes 4.915 seconds. The skipped tests are the migration-tests, they only run when Archetypes is installed.

There is still a lot of work to be done but thanks to everyone involved the progress is encouraging.

[janjaapdriessen]

We have migrated our 120M Python2 objects Relstorage database to Python3 using mdtools.relstorage. We've made the repository public for other ZODB users to use as inspiration:

https://github.com/minddistrict/mdtools.relstorage

[pbauer]

At the sprint in Halle we made very good progress. Plone is running nicely on Python 3 but now we have to fix over 500 failing tests. I wrote a blogpost at https://www.starzel.de/blog/porting-plone-to-python-3 with some more details.

[pbauer]

Status report

Here is report of the current state of Plone on Python 3. It covers what was done in the last couple of weeks since the Buschenschank Sprint (https://community.plone.org/t/this-was-the-buschenschanksprint-2018/6711) and lists some things that still need to be done.

&tldr;~Things are looking good but we still need to do a lot of work to migrate some remaining packages, fix all tests and provide a solid database migration-story. If you can, please come to the sprint in Halle in October (https://www.meetup.com/de-DE/Zope-Sprint/events/252468356/)!

On June 25-27 there was a Mini-Sprint in Munich where Jens Klein, David Glick and Philip Bauer worked mainly on fixing the tests with Python 3.

For that sprint we started out with 7459 Tests with 138 failures and 40 errors and ended with 8147 Tests with 113 failures and 56 errors.

We worked on a bunch of tasks:

Weird tests

There were a lot of tests failing where the issue was not obvious. We fixed all of the the following:

• -s plone.app.z3cform -t test_widget_base_notimplemented
• -s plone.dexterity -t test_container_manage_delObjects
• testUnicodeSplitter.py
• -s Products.CMFPlone -t safe_unicode
• -s Products.ZopeVersionControl
• -s Products.CMFEditions -t test06_retrieveWithNoMoreExistentObject
• -s Products.CMFEditions -t test16_delete_history_on_content_deletion
• -s Products.CMFPlone -t test_site_logo_is_stored_in_registry: "Object is of wrong type" in fieldErrorBox. Check def validate()...
• -s plone.protect -t test_change_password_on_root_does_not_throw_other_csrf_protection
• -s plone.formwidget.namedfile -t widget.rst

Fail on Jenkins only

Some tests still fail when running in Jenkins but pass when running locally. Weird.

• -s z3c.form -t datamanager.txt
• -s five.customerize

plone.rfc822

After the sprint Jens tackled the package plone.rfc822 that deals with serialization (also of schemata) and fixed all remaining issues (I think...). Here are a couple of tests that failed before:

• -s plone.rfc822
• -s plone.namedfile -t marshaler.rst
• -s plone.app.contentrules -t TestMailAction
• -s Products.CMFPlone -t test_request_reset_password

Port plone.app.robotframework

David ported plone.app.robotframework to Python 3. For this we had to re-add XMLRPC support to Zope (it was removed for Zope 4) and ported it to Python 3. We also had to work around a pretty tough bug in Python itself. See plone/plone.app.robotframework#80 for details.

Locally the robot-test are now running and some are passing. We still need to setup Jenkins properly to run them there and then fix the failing robot-tests.

Migrate tests away from PloneTestCase and PloneTestCaseFixture

We migrated a ton of tests from PloneTestCase to plone.app.testing. I also changed PloneTestCase to use Dexterity instead of Archetypes to make porting some tests easier that rely heavily on self.publish (see plone/plone.app.testing#50). A lot of additional tests are now running because of this change but I also found a couple of issues where dexterity is failing (e.g. testClearFindAndRebuildKeepsModificationDate).

Add PY23DoctestChecker

To run doctests in Python 3 and Python 2 we use a pattern PY23DoctestChecker that can modify the expected and received values in doctests for Python 2 (for Python 3 they should pass without any tricks). See https://github.com/plone/plone.app.relationfield/pull/24/files for a simple example.

Strategy to deal with random ordering

In Python 3 (as actually in Py2) keys in a dict do not have a deterministic order. There are many tests which implicitly test the order of dicts. They had to be fixed to work in Py2 and Py3).

We applied a couple of different patterns for different tests. The best is to simply sort the keys.

Where are some of the test that used to fail:

• -s Products.DCWorkflow -t test_exportimport
• -s plone.app.multilingual -t test_formvariables_sequences
• -s plone.app.contentrules -t testExport

Cannot adapt lists

A issue with __slice__ prevented lists of brains from being adaptable to IContentListing. See #2429

Fix issues in GenericSetup

David fixed some more issues in the import/export code of GenericSetup. The main issue there was (as usual) serialization of lists of bytes and text.

Issue with IObjectInitializedEvent

When creating Dexterity-content with invokeFactory there is a issue when a new item is moved before it's completely added. Using createContentInContainer instead fixes this. See plone/plone.app.contentrules#37

Open tasks

Plone runs fine with the current configuration (https://github.com/plone/buildout.coredev/blob/5.2/py3.cfg). But there are a lot of open tasks that need to be done before we can make a release.

Packages that need to be migrated

plone.outputfilters is being worked on by @MrTango @petschki et al (see plone/plone.outputfilters#27 plone/plone.outputfilters#31).

Products.PortalTransforms still has a lot of failing tests and needs someone to take care of those.

plone.rest and plone.restapi. Both packages were merged into Plone 5.2 but they are not yet compatible with Python 3. I started working on this in plone/plone.rest#75 and plone/plone.restapi#542. See #2474.

Fix all tests

We are currently at 9408 tests with 170 failures and 331 errors. The added tests and failures are mostly because I changed PloneTestCase to use Dexterity and also included plone.rest and plone.restapi in the Python 3 build.

Merge all open pull-requests

We currently have 88 open pull-requests that need to be reviewed and merged. They also need change notes and updated trove classifiers to specify support for python3:

(updated by @jensens at 2018-09-27)

DONE:

1. fix relative import for py3 collective/Products.DateRecurringIndex#4
2. Fix setup_localrole_plugin and aq in localroles in Python 3 borg.localrole#9
3. Python3 plone.api#402
4. Python3 plone.app.caching#44
5. Python3 plone.app.content#155
6. fix sorting of portletmanager-menuitems in py3 and fix tests in py3 plone.app.contentmenu#23
7. migrate all tests to use dexterity, fix most tests plone.app.contentrules#37
8. Fix indexers, link-view and most tests in Python 3 plone.app.contenttypes#471
9. Python3 plone.app.dexterity#274
10. Python3 plone.app.discussion#133
11. Python 3 fixes plone.app.event#289
12. Python 3 fixes plone.app.event#289
13. fix GopipIndex for py3 plone.app.folder#17
14. fix tests for py3 plone.app.i18n#7
15. Python3 plone.app.iterate#64
16. Implement viewlet dummy ordering needed for Python 3 sorting plone.app.layout#152
17. Fix tests in Python 3 plone.app.linkintegrity#66
18. fix python 3 support plone.app.portlets#107
19. fix tests in py3 plone.app.relationfield#24
20. Fixes for Python 3 plone.app.robotframework#82
21. Python3 plone.app.users#75
22. Python3 plone.app.versioningbehavior#39
23. fix issue with a test in plone.app.customerize plone.app.viewletmanager#17
24. Python 3 support plone.app.vocabularies#54
25. Python 3 and GenericSetup 2.0 plone.app.workflow#17
26. Python 3 conpatibility plone.app.z3cform#93
27. fix sorting when group.order is None plone.autoform#34
28. Python3 plone.batching#20
29. fix tests for py3 plone.browserlayer#8
30. Python3 plone.cachepurging#13
31. fix caching and tests in python 3 plone.caching#5
32. fix tests in py3 plone.contentrules#8
33. More Python 3 fixes plone.dexterity#84
34. fix ordering of content in folder in py3 plone.folder#9
35. Python3 plone.i18n#23
36. fix tests for py3 plone.indexer#5
37. Python3 plone.locking#17
38. fix static portlet for py3 plone.portlet.static#16
39. Fix for python 3 plone.recipe.alltests#5
40. More Python 3 fixes plone.resource#25
41. Fix tests and functionality in python 3 plone.resourceeditor#21
42. Python3 plone.schemaeditor#60
43. Fix handling session-cookies (and login) in py3 plone.session#13
44. Python 3 plone.stringinterp#12
45. Python 3 plone.subrequest#19
46. Add support for Python3 plone.supermodel#21
47. fix diff and tests in py3 Products.CMFDiffTool#33
48. [WIP] Use ulines for view_methods in python 3 Products.CMFDynamicViewFTI#15
49. Fix dict key sorting in Python 3 Products.CMFFormController#15
50. Migrate tests away from PloneTestCase Products.CMFPlacefulWorkflow#23
51. fix tests for py3 Products.CMFQuickInstallerTool#20
52. Python 3 Compatibility Products.MimetypesRegistry#15
53. More python 3 issues Products.PlonePAS#34
54. fix tests for py3 zopefoundation/five.customerize#6
55. Python3 zopefoundation/plone.z3cform#16
56. [WIP] Add support for Python 3.5 and 3.6. zopefoundation/Products.CMFUid#5
57. [WIP] Python 3 support zopefoundation/Products.GenericSetup#51
58. fix listPluginTypeInfo in py3 zopefoundation/Products.PluginRegistry#9
59. Python3 compatible imports zopefoundation/Products.ZopeVersionControl#1
60. fix code and test for python 3 zopefoundation/z3c.formwidget.query#7
61. re #36: Make fields hashable in Python 3. zopefoundation/zope.schema#38

Migration-Story

We need to start testing the migration-script zodbupdate with Plone and need to add and write the required zodbupdate_decode_dict for the packages in Plone that need it. For details see https://blog.gocept.com/2018/06/07/migrate-a-zope-zodb-data-fs-to-python-3/. This is likely the biggest and most important task for the sprint in Halle.

Document Support for Windows

So far no one tested Plone with Python 3 on Windows.

zope2instance

We need to add the necessary scripts to the WSGI start script:

• run (for running scripts)
• debug
• adduser

Improve Jenkins setup

The Jenkins setup for the PLIP job is basically OK but we soon need to start testing all pull-requests and changes in Python 2 and Python 3. For this we need some serious Tox know how.

Migrate mtest to python 3

@Rotonen did a lot of work on test parallelization with serious speed-improvements (see https://community.plone.org/t/ci-run-speedups-for-buildout-coredev/6225/14). As soon as we test everything with Py2 and Py3 tests take even longer. We are looking forward for the tests speedup be in use.

Adapt Plone's ZMI pages

There is a lot of work being done to port the ZMI to Bootstrap (see zopefoundation/Zope#249). We will have to make a list all ZMI templates that Plone adds and test/fix them the new ZMI theme.

Port development plugins

The most important developer tools are probably plone.reload, Products.PDBDebugMode, plone.app.debugtoolbar and Products.PrintingMailHost. These packages need to be ported.

Port other plugins

We would love to see that some frequently used plugins are already compatible with Python 3 when when the first alpha of 5.2 is released.
Two that come to mind are plone.app.easyform and Mosaic, where mosaic is probably much more work since it comes with a whole bunch of dependencies.

Test against Python 3.7

Python 3.7 is out. We need to update the jenkins-jobs to test against it. One of the changes that might be relevant is that dicts preserve the order of items by insertion.

Deal with Deprecation- and ResourceWarnings

Running Plone on Python 3 yields many deprecation warnings. We should fix these. Also there are a lot of ResourceWarnings for unclosed files, we want to get rid of these by using with open(x) as fd: ...

Final words

Please pitch in to bring Plone to Python 3:

• Come to the sprint in Halle (https://www.meetup.com/de-DE/Zope-Sprint/events/252468356/). It will be fun!
• Pick one of the open tasks mentioned above or in https://github.com/plone/Products.CMFPlone/projects/4 and work on it
• Review some of the pull-requests

[pbauer]

There is now a demo-site for this PLIP: http://demo-latest-py3.plone.de/de

[jensens]

FYI: I changed the above "Merge all open pull-requests" into a check list style. I am atm about to check the packages.

[jensens]

Checked all, please keep up to date

[pbauer]

The Saltlabs-Sprint in Halle (https://www.meetup.com/de-DE/Zope-Sprint/events/252468356) is approaching. More than 30 people are coming to the sprint (!!!) and I hope we will get a lot done. In preparation for the sprint there is some work under way:

Merge branches: @jensens, @thet and I are currently trying to merge as many python3-branches as possible. We're currently down to 44 unmerged braches. Many of the remaining branches work fine in python 3 but depend on changes in other packages (mosty plone.app.testing, plone.testing, Products.CMFPlone or plone.rfc822), so they are not easily merged.

Fix failing tests: We currently have 216 failing tests in Python 3, most of those are in CMFPlone (because I changed PloneTestCase to use Dexterity) and in plone.restapi.

With https://jenkins.plone.org/view/PLIPs/job/py3_on_py2/ I created a jenkins-job that runs the branches of this PLIP in Python 2. That should make it easier to find issues that need to be fixed before merging.

I would have loved to have a alpha-release of 5.2 out before the sprint but fixing all remaining tests and merging all remaining branches seems more and more unlikely (even I have to sleep sometimes and need to earn some money...).

But: Things are looking good:

• http://demo-latest-py3.plone.de is working most of the time (unless someone merges a branch without updating the py3-config)
• There are promising first result from migrating Plone Databases to Python 3
• @gforcada will add some additional jenkins nodes so that jenkins will not be the bottleneck

If you want to help:

• review, test and merge some branches
• fix some failing tests
• come to the sprint

The tasks for the sprint are basically still the same as mentioned above. We'll publish a more detailed list next week.

[gforcada]

@pbauer regarding more servers, I already asked @polyester to get a server (someone at the board should have the hetzner account, I also got an account there, but with no billing permissions 🤑 )

[gforcada]

@pbauer as I saw some commits from you updating trove classifiers, I thought about updating https://jenkins.plone.org/py3/ 162 out of 290 packages ported

[jensens]

I updated the above list of open PRs and we're down to 28.

[jensens]

We are at 21 PR left - 61 done. Also, there were 10 AT BBB related PRs merged (not listed here).

[jensens]

State of tests on Python 3: 57 failures.
On Python 2.7 way more, mainly (but not only) because of exception and string outputs in doctests. Should be easy to fix most of them.

[ale-rt]

I switched the Zope branch to plone-py3 which contains the current Zope master plus the two patches:

• Re-add support for XML-RPC zopefoundation/Zope#290
• Fix XML Page template files in Python 3 zopefoundation/Zope#320

[thet]

Regarding WebDAV and WSGI - there is a WSGI middleware for WebDAV (which also includes a standalone WebDAV server):

https://github.com/mar10/wsgidav
https://wsgidav.readthedocs.io/en/latest/reference_guide_architecture.html

This project looks quite promising:

• It has a good software architecture - everything is implemented against interface specifications.
• It comes with default DAVProvider, PropertyManager and LockManager implementations and allows for providing more specific ones.
• It's tested against a lot of WebDAV clients: Windows File Explorer and drive mapping, MS Office, Ubuntu, Mac OS X, …

This could help us a lot providing WebDAV on WSGI for Plone/Zope.

[davisagli]

There are some test failures in CMFCore due to Plone monkey patches that change the behavior that CMFCore tests for (at least patches/dateIndexPatch and patches/speed). Ideally each patch should come with a function to apply it and a function to unapply it and we could apply them as part of the Plone test fixture and during Zope startup instead of at import time. But that's probably quite some work so the practical thing to do for now is to trust that CMFCore is fully tested prior to release and stop running its tests as part of Plone's test suite.

[pbauer]

A super-short update:

• We have 5 remaining failing robot-tests on the job, that is running the whole python3-branches plus Archetypes in Python 2.
• And the py3 job https://jenkins.plone.org/view/PLIPs/job/plip-py3 has also 5 remaining faiing robot tests.
• Zope 4.0b6 is released. Updating our versions leads to some more issues that need to be fixed (Update to Zope 4.0b7 #2590)
• Once 4.0b6 is used and both jobs use are green we need to merge a lot of branches at the same time.
• The sprint in Halle was a huge success and we made a lot of progress. A report will be published soon (I was down with a nasty cold until yesterday)

[pbauer]

I declare this done. There are follow-up tasks but the PLIP is merged and Plone 5.2b1 released.