Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Checkout Zope and use its master versions. #844

Merged
merged 6 commits into from
Mar 10, 2023

Conversation

mauritsvanrees
Copy link
Member

There are several major updates there, which should be fine: they drop Python 2 support.

I do see that several pins have been dropped, so we need to add more ourselves. This is the output when allowing unpinned versions:

[versions]
mock = 5.0.1
zope.sendmail = 5.3

# Required by:
# plone.recipe.zope2instance==6.12.0
ZEO = 5.4.0

# Required by:
# repoze.xmliter==0.6.1
future = 0.18.3

# Required by:
# ZEO==5.4.0
zdaemon = 4.4

# Required by:
# five.customerize==2.1.0
# plone.contentrules==2.1.2
zope.componentvocabulary = 2.3.0

# Required by:
# plone.app.caching==3.0.4.dev0
# plone.memoize==3.0.0
zope.ramcache = 2.4

I added them.

@mauritsvanrees
Copy link
Member Author

plone.dexterity tests fail on Zope master, because Zope fixed a typo 'succeded'. I am fixing that in plone/plone.dexterity#173.
Let's retry the current PR when that one is merged.

@mauritsvanrees mauritsvanrees force-pushed the maurits-checkout-zope-on-60 branch 2 times, most recently from de321fd to 03bf5d9 Compare March 1, 2023 11:45
@mauritsvanrees mauritsvanrees marked this pull request as ready for review March 1, 2023 12:02
@mauritsvanrees
Copy link
Member Author

I rebased, squashed, and force pushed. Should work now.

@jenkins-plone-org please run jobs

@jensens
Copy link
Member

jensens commented Mar 1, 2023

I would say, merge yourself if ready.

@mauritsvanrees
Copy link
Member Author

In all jobs 19 failing tests. Really? :-(
example 3.11 job
At least some have the same root cause, so maybe it is just one or two fixes somewhere.
Possibly we need a few other new packages, I don't know.
I am not likely to continue this today, so if someone else wants to try: go ahead.

@jensens
Copy link
Member

jensens commented Mar 1, 2023

In all jobs 19 failing tests. Really? :-(

see my condition - I would not call it ready ;)

@jensens
Copy link
Member

jensens commented Mar 1, 2023

@ale-rt
Copy link
Member

ale-rt commented Mar 1, 2023

We could change the signature in Zope master to def __init__(self, aFieldStorage, charset="latin-1"): to make it backward compatible.

mauritsvanrees added a commit to zopefoundation/Zope that referenced this pull request Mar 7, 2023
Without this change, some tests (and possibly live code) in Plone fail because they do not pass the new required `charset` parameter.
See plone/buildout.coredev#844
And see discussion that I just started here:
#1094 (comment)

Also fixes an unclosed file in a test.
mauritsvanrees added a commit to zopefoundation/Zope that referenced this pull request Mar 8, 2023
Without this change, some tests (and possibly live code) in Plone fail because they do not pass the new required `charset` parameter.
See plone/buildout.coredev#844
And see discussion that I just started here:
#1094 (comment)

Also fixes an unclosed file in a test.
@mauritsvanrees mauritsvanrees force-pushed the maurits-checkout-zope-on-60 branch from 03bf5d9 to bd33f98 Compare March 8, 2023 10:52
There are several major updates there, which should be fine: they drop Python 2 support.

I do see that several pins have been dropped, so we need to add more ourselves.
This is the output when allowing unpinned versions:
```
mock = 5.0.1
zope.sendmail = 5.3

ZEO = 5.4.0

future = 0.18.3

zdaemon = 4.4

zope.componentvocabulary = 2.3.0

zope.ramcache = 2.4
```

Pinned several others as well that used to be pinned by Zope.
They are not getting pulled in on Python 3.11 on my Mac, so hard to say where these dependencies comes from.
@mauritsvanrees mauritsvanrees force-pushed the maurits-checkout-zope-on-60 branch from bd33f98 to cb0b066 Compare March 8, 2023 11:03
@mauritsvanrees
Copy link
Member Author

@jenkins-plone-org please run jobs

@mauritsvanrees
Copy link
Member Author

Three robot tests fail. They are the same across all four Python versions.

TLDR: I found the problem and it needs a one-line fix in Zope.

I looked at this one and I have the same test failure locally and can reproduce it in an actual site:

  • Use the buildout.coredev branch from this PR, use Python 3.11 to bootstrap and buildout it.
  • Create a Plone Site.
  • Go to the folder contents of the site root.
  • Click the Rearrange button and reverse the sort order: nothing happens, and I see no error.
  • Alternatively, drag and drop an item to change the order. Now you get an error:
Traceback (innermost last):
  Module ZPublisher.WSGIPublisher, line 181, in transaction_pubevents
  Module ZPublisher.WSGIPublisher, line 390, in publish_module
  Module ZPublisher.WSGIPublisher, line 285, in publish
  Module ZPublisher.mapply, line 85, in mapply
  Module ZPublisher.WSGIPublisher, line 68, in call_object
  Module plone.app.content.browser.contents.rearrange, line 43, in __call__
KeyError: 'delta'

That is on this line:

delta = self.request.form["delta"]

I traced this back to Zope: if I go to src/Zope, checkout tag 5.8, and retry, all is well. Looking more specifically, it is caused by the changes from zopefoundation/Zope#1094, which replaces cgi.FieldStorage by multipart. With zopefoundation/Zope@5b324f6 it fails, with one earlier, it works.

@d-maurer, that is your PR. Do you have an idea what causes this?

On a (POST) request where reordering works, I have this:

(Pdb++) self.request
<WSGIRequest, URL=http://localhost:8080/Plone5/fc-itemOrder>
(Pdb++) self.request.form
{'delta': '-1', 'id': 'events', '_authenticator': '19599becb83ff63f067ba031a4622eab0ff90ccb', 'subsetIds': '["news","events","Members"]'}
(Pdb++) pp self.request.items()
...
 ('delta', '-1'),

On a request where reordering fails, I get:

(Pdb++) self.request
<WSGIRequest, URL=http://localhost:8080/Plone5/fc-itemOrder>
(Pdb++) self.request.form
{}
(Pdb++) pp self.request.items()
...
 ('BODY', b'delta=-1&id=events&_authenticator=4a94398e168eb87302d55598839313413329f964&subsetIds=%5B%22news%22%2C%22events%22%2C%22Members%22%5D'),

So when it fails, the request has a BODY and no form.

Ah, found it! It is this line from your PR:

elif content_type == "application/x-www-form-urlencoded": 

In my case I have this:

(Pdb++) content_type
'application/x-www-form-urlencoded; charset=UTF-8'

So the condition does not match. It should be:

elif content_type.startswith("application/x-www-form-urlencoded"):

Then reordering works, and the test passes.

@d-maurer does that make sense?
I can make a small PR, but I am getting sleepy, so that will have to wait a bit.
Maybe also get my other PR in the same area approved and merged first.

@d-maurer
Copy link

d-maurer commented Mar 9, 2023 via email

@mauritsvanrees
Copy link
Member Author

@d-maurer wrote:

application/x-www-form-urlencoded
is defined by "https://www.w3.org/TR/2014/REC-html5-20141028/iana.html":
This means: application/x-www-form-urlencoded is expected to have no parameters - and especially no charset parameter. For that reason, I used "==" instead of startswith.

Interesting. Most others in that document allow a charset parameter, but indeed this one not.

Not sure whether we should make Zope support the use of a parameter not supported by the relevant standard and which it would not interpret. I suggest, you make the tests standard conform, instead.

This is not only in tests. I see this in an actual browser: it makes reordering in folder contents fail.

Okay, the folder contents code in mockup explicitly sets the Content-Type header this way here and here. So we could fix it there.

I see a few more though, at least these with a simple grep, there may be more:

plone/formwidget/recurrence/tests/test_z3cwidget.py:55:                "ajaxContentType": "application/x-www-form-urlencoded; charset=UTF-8",
plone/formwidget/recurrence/z3cform/widget.py:60:            ajaxContentType="application/x-www-form-urlencoded; charset=UTF-8",
zope/publisher/tests/test_browserrequest.py:422:            'CONTENT_TYPE': 'application/x-www-form-urlencoded; charset=UTF-8',
zope/publisher/tests/test_browserrequest.py:446:                'application/x-www-form-urlencoded; charset=ISO-8859-1'),
zope/publisher/tests/test_browserrequest.py:458:                'application/x-www-form-urlencoded; charset=ISO-8859-13'),

plone.formwidget.recurrence is the only one in this list where the problem is also in live code. This should mean that recurring events in Plone may have a problem as well, but I don't see this reflected in the tests. I did not try anything though.

For zope.publisher here is a list of search results.
Some of its tests that use the no allowed charset parameter, were changed in this PR from 2021, which handles similar encoding problems as you did. Maybe the examples in the tests were just badly chosen.

So: I could update mockup and plone.formwidget.recurrence to no longer send the charset parameter. But I do wonder if it makes sense to make the Zope code more lenient, to avoid problems for other code out there that makes the same mistake.

mauritsvanrees added a commit to zopefoundation/Zope that referenced this pull request Mar 9, 2023
…1101)

FileUpload: use default encoding as charset when nothing is passed.

Without this change, some tests (and possibly live code) in Plone fail because they do not pass the new required `charset` parameter.
See plone/buildout.coredev#844
And see discussion that I just started here:
#1094 (comment)

Also fixes an unclosed file in a test.
@d-maurer
Copy link

d-maurer commented Mar 9, 2023 via email

Branch maurits-fileupload-charset-latin-1 was merged.
@d-maurer
Copy link

d-maurer commented Mar 9, 2023 via email

mauritsvanrees added a commit to plone/mockup that referenced this pull request Mar 9, 2023
…ncoded header.

Adding a charset to this Content-Type is not supported by the definition, and fails with Zope master.

See plone/buildout.coredev#844 (comment)
mauritsvanrees added a commit to plone/plone.formwidget.recurrence that referenced this pull request Mar 9, 2023
Adding a charset to this Content-Type is illegal according to the definition, and fails with Zope master.
See plone/buildout.coredev#844 (comment)

In our case, when you click to add a recurrence, you get a dialog box saying 'error' because the call to `@@json_recurrence` failed.
@mauritsvanrees
Copy link
Member Author

I made two PRs to remove the charset:

Those fix a few real problems in the folder contents and for recurring events.

But reordering in the folder contents still does not work. And I have no idea which code is responsible for adding the Content-Type header application/x-www-form-urlencoded; charset=UTF-8. I would expect either mockup or Patternslib, but I don't see anything, other than what I am already fixing in the above PRs.

@thet @petschki Something in the folder contents code results in a Content-Type header application/x-www-form-urlencoded with a charset. This worked until now, although it is against the rules (as I learned today). With Zope master it fails. You can see this when you try to reorder items: the main Reorder button with a form does nothing. Dragging and dropping is clearer because it gives an error. Can you find out which code is responsible for this? I can't find it. I have a mockup PR linked above that solves similar problems in folder contents.

@d-maurer What about accepting the charset in the header when it is the same as the default_encoding? We would still ignore it, but at least we would use the wanted code path.
When the charset does not match, we could log a warning and then ignore it.

mauritsvanrees added a commit to zopefoundation/Zope that referenced this pull request Mar 9, 2023
Warn when this does not fit the default encoding.
This is my proposal from plone/buildout.coredev#844 (comment)
It fixes several problems in Plone.
@mauritsvanrees
Copy link
Member Author

@jenkins-plone-org please run jobs

@d-maurer
Copy link

d-maurer commented Mar 9, 2023 via email

@petschki
Copy link
Member

petschki commented Mar 9, 2023

@thet @petschki Something in the folder contents code results in a Content-Type header application/x-www-form-urlencoded with a charset. This worked until now, although it is against the rules (as I learned today). With Zope master it fails. You can see this when you try to reorder items: the main Reorder button with a form does nothing. Dragging and dropping is clearer because it gives an error. Can you find out which code is responsible for this? I can't find it. I have a mockup PR linked above that solves similar problems in folder contents.

See my comment here plone/mockup#1292 (comment)

@mauritsvanrees
Copy link
Member Author

So all/most jQuery Ajax calls are the problem! Thanks for figuring that out @petschki.

At least with all the various PRs included, the jobs pass, so we are in the right direction.

Use the standard branches from mockup and plone.formwidget.recurrence again.
Although my fixes on the previous branches are fine, they should not be needed anymore.
@mauritsvanrees
Copy link
Member Author

With Dieter's branch from zopefoundation/Zope#1104 both the folder contents and recurring events work again. Thanks!
The only other PR we need is a test fix in plone/plone.formwidget.namedfile#60
That is tested in this coredev PR.

Let's see if Jenkins is happy too:

@jenkins-plone-org please run jobs

@mauritsvanrees mauritsvanrees merged commit fad9794 into 6.0 Mar 10, 2023
@mauritsvanrees mauritsvanrees deleted the maurits-checkout-zope-on-60 branch March 10, 2023 20:31
mister-roboto pushed a commit that referenced this pull request Mar 10, 2023
Branch: refs/heads/master
Date: 2023-03-09T22:00:47+01:00
Author: Maurits van Rees (mauritsvanrees) <[email protected]>
Commit: plone/plone.formwidget.recurrence@25d552c

Do not add charset to application/x-www-form-urlencoded header.

Adding a charset to this Content-Type is illegal according to the definition, and fails with Zope master.
See #844 (comment)

In our case, when you click to add a recurrence, you get a dialog box saying 'error' because the call to `@@json_recurrence` failed.

Files changed:
A news/844.bugfix
M plone/formwidget/recurrence/tests/test_z3cwidget.py
M plone/formwidget/recurrence/z3cform/widget.py
Repository: plone.formwidget.recurrence

Branch: refs/heads/master
Date: 2023-03-10T21:31:21+01:00
Author: Maurits van Rees (mauritsvanrees) <[email protected]>
Commit: plone/plone.formwidget.recurrence@25e1695

Merge pull request #31 from plone/maurits-no-charset-with-x-www-form-urlencoded

Do not add charset to application/x-www-form-urlencoded header.

Files changed:
A news/844.bugfix
M plone/formwidget/recurrence/tests/test_z3cwidget.py
M plone/formwidget/recurrence/z3cform/widget.py
@mauritsvanrees
Copy link
Member Author

All is well now. I merged this manually, plus the related plone.formwidget.namedfile/recurrence PRs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants