Start migrating Plone to Python 3

[pbauer]

Meta-ticket to collect todos for this task.

Initial work was documented in https://docs.google.com/document/d/1rYN817-9SjAQulpes36k1veTt4pABZRbl0T0BbuZrHA/edit#

Follow the best practice doumentation in #2184 (comment) when working on this!

Tasks

• Document setup of different Python 3 versions, tox and related tools to test and maintain Python 3 compatibility.
• Migrate GenericSetup and MailHost to Python 3 [@gotcha and @witsch] (Note Prevent AttributeError 'NoneType' object has no attribute 'decode'. zopefoundation/Products.GenericSetup#52 for a small fix that is needed.)
• Update the jenkins-job for python3 (http://jenkins.plone.org/view/py3)
• Migrate packages without CMFPlone dependencies (e.g. finish migration of plone.rfc822)
• Fix all import-errors everywhere (use 2to3 and six to create backwards-compatible imports)
• Look at plone.testing and plone.app.testing if we can get the lowermost testlayers to start on python 3

Packages that were worked on regarding Python 3

• AccessControl [@rudaporto] - Python3 compat module. zopefoundation/AccessControl#48
• archetypes.schemaextender [@mauritsvanrees] - Add Python 2 / 3 compatibility. archetypes.schemaextender#4
• borg.localrole [@jensens] - Py3 and cleanup borg.localrole#6
• collective.monkeypatcher [@mauritsvanrees] - nothing to do
• collective.xmltestreport [@mauritsvanrees] - nothing to do
• plone.alterego [@pbauer] - nothing to do
• plone.api [@rudaporto] - Python 2 / 3 compatible imports plone.api#394
• plone.app.blob [@rudaporto, @pbauer, @davilima6] - Python 2 / 3 compatible imports. plone.app.blob#43 & More Python 2 / 3 compatibility adjustments plone.app.blob#48
• plone.app.caching [@b4oshany, @jensens, @davilima6] Imports are Python3 compatible. plone.app.caching#40 & More Python 2 / 3 compatibility adjustments plone.app.caching#41
• plone.app.collection [@rudaporto] - nothing to do
• plone.app.content [@ale-rt, @davilima6] - Imports are compatibile with Python3 plone.app.content#143 & Prepare for Python 2 / 3 compatibility plone.app.content#151
• plone.app.contentlisting [@rudaporto] - nothing to do
• plone.app.contentmenu [@pbauer] - nothing to do
• plone.app.contentrules [@pbauer] - Add Python 2 / 3 compatibility plone.app.contentrules#35
• plone.app.contenttypes [@pbauer] - Add Python 2 / 3 compatibility plone.app.contenttypes#447
• plone.app.controlpanel [@vincero] - nothing to do
• plone.app.customerize [@vincero] - nothing to do
• plone.app.dexterity [@vincero] - Python 2 / 3 compatible imports plone.app.dexterity#265
• plone.app.discussion [@pbauer] - Add Python 2 / 3 compatibility plone.app.discussion#129
• plone.app.event [@pbauer] - Add Python 2 / 3 compatibility plone.app.event#274
• plone.app.folder [@pbauer] - Add Python 2 / 3 compatibility plone.app.folder#15
• plone.app.i18n [@pbauer] - nothing to do
• plone.app.imaging [@ale-rt] - Imports are Python3 compatible plone.app.imaging#34
• plone.app.intid [@pbauer] - nothing to do
• plone.app.iterate [@vincero] - nothing to do
• plone.app.layout [@ale-rt, @jensens] - Imports are Python3 compatible plone.app.layout#134
• plone.app.layout part2 [@pbauer] - py3 fixes plone.app.layout#140
• plone.app.lockingbehavior [@pbauer] - nothing to do
• plone.app.ldap [ale-@rt] - Imports are Python3 compatible plone.app.ldap#33
• plone.app.linkintegrity - [ale-rt] Imports are Python3 compatible plone.app.linkintegrity#61
• plone.app.locales [@rudaporto] - nothing to do
• plone.app.mosaic [@b4oshany] - Imports are Python3 compatible plone.app.mosaic#390
• plone.app.multilingual [@ale-rt, @jensens] - WIP Imports are Python3 compatible plone.app.multilingual#296
• plone.app.openid [@b4oshany] - Imports are Python3 compatible. plone.app.openid#12
• plone.app.portlets [@ale-rt] - Imports are Python3 compatible plone.app.portlets#104
• plone.app.portlets part2 [@pbauer] - fix python 3 support plone.app.portlets#107
• plone.app.querystring [@pbauer] - Add Python 2 / 3 compatibility plone.app.querystring#84
• plone.app.redirector [@vincero] - Add Python 2 / 3 compatibility plone.app.redirector#9
• plone.app.registry [@pbauer] - Python 2 / 3 compatible imports plone.app.registry#29
• plone.app.relationfield [@rudaporto] - nothing to do
• plone.app.referenceablebehavior [@pbauer] - nothing to do
• plone.app.robotframework [@ale-rt, @b4oshany, @davilima6] - Imports are Python3 compatible plone.app.robotframework#70
• plone.app.standardtiles [@b4oshany] - Imports are Python3 compatible plone.app.standardtiles#82
• plone.app.testing [@rudaporto] - nothing to do
• plone.app.textfield [@b4oshany] - Imports are Python3 compatible. plone.app.textfield#26
• plone.app.theming [@pbauer] - Prepare for Python 2 / 3 compatibility plone.app.theming#139
• plone.app.upgrade [@rudaporto] - nothing to do
• plone.app.users [@rudaporto] - nothing to do
• plone.app.uuid [@pbauer] - nothing to do
• plone.app.versioningbehavior [@ale-rt @pbauer] - Imports are Python3 compatible plone.app.versioningbehavior#35
• plone.app.vocabularies [vincero] - Add Python 2 / 3 compatibility plone.app.vocabularies#53
• plone.app.widgets [@pbauer] - Add Python 2 / 3 compatibility plone.app.widgets#178
• plone.app.workflow [@rudaporto] - nothing to do
• plone.app.z3cform [@pbauer] - Prepare for Python 2 / 3 compatibility plone.app.z3cform#85
• plone.app.z3cform part 2 [@pbauer] - fix issue for py3 startup plone.app.z3cform#86
• plone.autoform [@pbauer] - Prepare for Python 2 / 3 compatibility plone.autoform#31
• plone.batching [@pbauer - nothing to do
• plone.behavior [@rudaporto] - nothing to do
• plone.browserlayer [@pbauer] - Add Python 2 / 3 compatibility plone.browserlayer#7
• plone.cachepurging [@pbauer] - Add Python 2 / 3 compatibility plone.cachepurging#9
• plone.caching [@vincero] - nothing to do
• plone.contentrules [@pbauer] - Add Python 2 / 3 compatibility plone.contentrules#7
• plone.dexterity [@pbauer] - Prepare for Python 2 / 3 compatibility plone.dexterity#76
• plone.dexterity part 2 [@pbauer] - fix issues during py3 startup plone.dexterity#77
• plone.event [@vincero] - nothing to do
• plone.folder [@vincero] - Add Python 2 / 3 compatibility plone.folder#5
• plone.formwidget.namedfile - [@ale-rt, @pbauer] - Prepare for Python 2 / 3 compatibility plone.formwidget.namedfile#30
• plone.formwidget.recurrence [@pbauer] - Add Python 2 / 3 compatibility plone.formwidget.recurrence#23
• plone.i18n [@pbauer] - Add Python 2 / 3 compatibility plone.i18n#19
• plone.indexer [@vincero] - nothing to do
• plone.intelligenttext - nothing to do
• plone.keyring [@pbauer] - nothing to do
• plone.locking [@pbauer] - Add Python 2 / 3 compatibility plone.locking#15
• plone.memoize [@vincero] - nothing to do
• plone.namedfile [@ale-rt] - Imports are Python3 compatible plone.namedfile#60
• plone.namedfile part 2 [@pbauer] - Prepare for more Python 2 / 3 compatibility plone.namedfile#61
• plone.namedfile part 3 [@pbauer] - fixes for py3 startup plone.namedfile#62
• plone.outputfilters [@pbauer] - Add Python 2 / 3 compatibility plone.outputfilters#26
• plone.portlet.collection [@vincero] - nothing to do
• plone.portlet.static [@vincero] - Add Python 2 / 3 compatibility plone.portlet.static#15
• plone.portlets [@pbauer] - nothing to do
• plone.protect [@vincero] - Add Python 2 / 3 compatibilityJ plone.protect#70
• plone.registry [@gogobd] - nothing to do
• plone.resource [@gogobd] - Add Python 2 / 3 compatibility plone.resource#22
• plone.resourceeditor [@pbauer] - Add Python 2 / 3 compatibility plone.resourceeditor#20
• plone.rfc822 [@ale-rt, @pbauer, @davilima6] - WIP Python3 plone.rfc822#4
• plone.scale [@pbauer] - nothing to do
• plone.schema [@vincero] - nothing to do
• plone.schemaeditor [@pbauer] - Prepare for Python 2 / 3 compatibility plone.schemaeditor#59
• plone.session [@vincero] - Add Python 2 / 3 compatibility plone.session#10
• plone.stringinterp [@pbauer] - nothing to do
• plone.subrequest [@pbauer] - Python3 plone.subrequest#16
• plone.supermodel [@pbauer, @b4oshany, @davilima6] -
  Add support for Python3 plone.supermodel#21
• plone.synchronize [@vincero] - nothing to do
• plone.testing [@rudaporto, @pbauer, @icemac, @loechel] - Add support for Python 3.6. plone.testing#48
• plone.theme [@pbauer] - nothing to do
• plone.transformchain [@vincero] - Add Python 2 / 3 compatibility plone.transformchain#7
• plone.uuid - nothing to do
• plone.z3cform [@pbauer, @davilima6, @ale-rt] - Prepare for Python 2 / 3 compatibility zopefoundation/plone.z3cform#11, Python 3 compatibility: use the adapter and implementer decorators zopefoundation/plone.z3cform#12
• plonetheme.barceloneta - nothing to do
• Products.Archetypes [@pbauer] - (keep your hands off this)
• Products.ATContentTypes - TODO
• Products.CMFCore [@rudaporto, @davilima6] - Python 2 / 3 compatible imports. zopefoundation/Products.CMFCore#29 & More Python 2 / 3 compatibility changes zopefoundation/Products.CMFCore#32
• Products.CMFDynamicViewFTI [@davilima6] - Add Python 2 / 3 compatibility Products.CMFDynamicViewFTI#11
• Products.CMFDynamicViewFTI part 2 [@pbauer] - fix py3 issue Products.CMFDynamicViewFTI#12
• Products.CMFEditions [@davilima6] - Add Python 2 / 3 compatibility Products.CMFEditions#47 Make imports Python 3 compatible Products.CMFEditions#49
• Products.CMFFormController [@rudaporto, @pbauer] - Python 2 / 3 compatible imports. Products.CMFFormController#13
• Products.CMFPlacefulWorkflow [@davilima6] - plone/Products.CMFPlacefulWorkflow@b1746ce
• Products.CMFUid part2 [@pbauer] - Fix startup in python3 zopefoundation/Products.CMFUid#3
• Products.CMFPlone [@pbauer] - make imports work in python3 #2181
• Products.CMFPlone part 2 [@pbauer] - Prepare for Python 2 / 3 compatibility with a little help from sixer #2277
• Products.CMFQuickInstallerTool [@mauritsvanrees] - Add Python 2 / 3 compatibility. Products.CMFQuickInstallerTool#17
• Products.contentmigration [@davilima6] - Add Python 2 / 3 compatibility Products.contentmigration#17 & More compatibility changes for Python 2 / 3 Products.contentmigration#18
• Products.DateRecurringIndex [@vincero] - nothing to do
• Products.DCWorkflow [@davilima6] - Prepare for Python 2 / 3 compatibility (master branch) zopefoundation/Products.DCWorkflow#9
• Products.DiffTool [@davilima6] - Add Python 2 / 3 compatibility Products.CMFDiffTool#28
• Products.ExtendedPathIndex [@vincero] - nothing to do
• Products.ExternalEditor [@davilima6] - Prepare for Python 2 / 3 compatibility (master branch) zopefoundation/Products.ExternalEditor#8 & Prepare for Python 2 / 3 compatibility (1.1.x branch) zopefoundation/Products.ExternalEditor#9
• Products.ExternalMethod [@vincero] - nothing to do
• Products.GenericSetup [@gotcha, @davisagli, @pbauer, @ale-rt] - Updates to davisagli-py-3 branch zopefoundation/Products.GenericSetup#60 Some more python 3 fixes zopefoundation/Products.GenericSetup#69
• Products.i18ntestcase [@b4oshany, @davilima6] - Imports are compatibile with Python3 Products.i18ntestcase#1 & More Python 2 / 3 compatibility changes Products.i18ntestcase#2
• Products.Marshall [@pbauer] - Prepare for Python 2 / 3 compatibility Products.Marshall#5
• Products.PlacelessTranslationService [@ksuess] - WIP
• Products.PlonePAS [@pbauer] - Fix imports for py3 Products.PlonePAS#26
• Products.PlonePAS part2 [@pbauer] - Fix startup issues in python3 Products.PlonePAS#30
• Products.PluggableAuthService [@rudaporto] - Python 2 / 3 compatible imports. zopefoundation/Products.PluggableAuthService#10
• Products.PluginRegistry [@mauritsvanrees] - Basic Python 3 compatibility zopefoundation/Products.PluginRegistry#4
• Products.PortalTransforms [@mauritsvanrees] - Add Python 2 / 3 compatibility. Products.PortalTransforms#26
• Products.PortalTransforms part2 [@pbauer] - Begin fixing startup in py3 Products.PortalTransforms#28
• Products.ResourceRegistries [@mauritsvanrees] - Add Python 2 / 3 compatibility Products.ResourceRegistries#25
• Products.ResourceRegistries part2 [@pbauer] - fix relative imports for py3 Products.ResourceRegistries#26
• Products.Sessions [@pbauer] - Fix startup and syntax in python3 zopefoundation/Products.Sessions#3
• Products.SiteErrorLog [@pbauer, @icemac] - make ZServer a dependency for py2 only zopefoundation/Products.SiteErrorLog#4
• Products.statusmessages [@davilima6] - Add Python 2 / 3 compatibility Products.statusmessages#8
• Products.TemporaryFolder [@davilima6] - nothing to do - Released as 5.0 [@icemac]
• Products.validation [@davilima6] - nothing to do
• Products.ZCatalog [@davilima6] - nothing to do
• Products.ZCTextIndex [@davilima6] - nothing to do
• Products.ZopeVersionControl [@rudaporto] - Python3 compatible imports zopefoundation/Products.ZopeVersionControl#1
• Products.ZSQLMethods [@davilima6] - nothing to do
• z3c.objpath - [@ale-rt] Make imports Python 3 compatible zopefoundation/z3c.objpath#1
• z3c.relationfield - TODO [@ale-rt] [undefined name 'cmp'] Python 3 compatibility: use the implementer decorator zopefoundation/z3c.relationfield#9
• zc.relation [@ale-rt] - Py3 compat zopefoundation/zc.relation#3
• zc.relationship [@ale-rt] - Python3 compatibility: print function zopefoundation/zc.relationship#1

[ale-rt]

Take care that many of that PR use the _compat module approach instead of sixer.
We should not merge them as they are, see:

• make imports work in python3 #2181

[pbauer]

Packages that still need to be looked at for this ticket: https://public.etherpad-mozilla.org/p/Plone_on_Python_3
Many of those are already ok (like WebOb). It is simply a diff between the list above and the packages used in ./bin/test
[Edit: the relevant items of the etherpad list have been copied to the big list here above.]

[pbauer]

Some pull-requests need to replace replace _compat with six:

• Imports are Python3 compatible plone.app.imaging#34
• Imports are Python3 compatible plone.app.linkintegrity#61
• Imports are Python3 compatible plone.app.theming#137
• Prepare for Python 2 / 3 compatibility plone.app.z3cform#85
• Fix imports for Python 3 plone.dexterity#74

Please update this list when you encounter any.

[jensens]

I close this in favour of keeping thing central at #2368

[andreasma]

I worked on the package cioppino.twothumbs and updated it locally with sixer and python-modernize and some fixes of imports. The package works in my local environment. I tried to test my pull request for python 2 to 3 with Travis-CI, but I can't get Travis to work. I need some hints how I could fix this. My current work is available here: collective/cioppino.twothumbs#23 (I used branch python3 inside my own fork).