Fix `asyncio.Task.cancelling` issues #790

gschaffner · 2024-09-22T09:31:40Z

Changes

This is a small patch that fixes the more obscure case that we skipped yesterday (#774 (comment)).

After a night's rest I woke up this morning with the realization that this case shouldn't involve whack-a-mole as we feared. IIUC it seems to be relatively straightforward to fix.

d342ae1 is the core patch here in its simplest form. The subsequent two commits are some minor follow-up refactoring for performance optimization:

Avoid doing repeated computations of _effectively_cancelled & _parent_cancellation_is_visible_to_us in CancelScope.__exit__ (I missed this during review).
Move the counter back from TaskState to CancelScope. This makes the impl. a little harder to understand IMO (as this makes the impl. diverge a bit further from Trio's model of delaying deciding which scope a level cancellation exception belongs to), but it may(?) save a little memory (as the counter is only needed for host tasks).

Checklist

If this is a user-facing code change, like a bugfix or a new feature, please ensure that
you've fulfilled the following conditions (where applicable):

You've added tests (in tests/) added which would fail without your patch
You've updated the documentation (in docs/, in case of behavior changes or new
features)
You've added a new changelog entry (in docs/versionhistory.rst).

src/anyio/_backends/_asyncio.py

agronholm · 2024-11-30T14:21:56Z

I'm reviewing this now for inclusion in the next release. Would you mind resolving the merge conflicts? I'm not 100% confident about doing that myself in this case.

gschaffner · 2024-12-05T06:59:18Z

Rebased in order to: resolve conflicts with #806 and verify whether this needed changes for #822. Also fixed the changelog entry being too vague.

If you want, we could also add a changelog entry about the improved performance of CancelScope.__exit__ (by deduplicating the _effectively_cancelled and _parent_cancellation_is_visible_to_us calls that were repeated for every _uncancel call). I'm not sure if it's worth noting that in the changelog or not; I haven't benchmarked it.

Also added a fix for #832 to this PR. That could be done in a separate PR, but I figured testing/review is probably easier if both things are done in one PR since they are both about cancelling().

graingert · 2024-12-05T07:05:22Z

src/anyio/_backends/_asyncio.py

-        and isinstance(exc.args[0], str)
-        and exc.args[0].startswith("Cancelled by cancel scope ")
-    )
+    # Sometimes third party frameworks catch a CancelledError and raise a new one, so as


Can you include this in the change log?

I'm a bit conflicted on this, I think 3rd party frameworks should change behavior, so maybe this should log a warning?

That __context__ walk isn't new in this PR—it's from #774. It's just in the diff here due to refactoring it out of _uncancel in c891630

Oh I see oh well

Sometimes third party frameworks catch a CancelledError and raise a new one

Ugh.
+1 on raising a warning when somebody does that.

Warning also seems reasonable to me. Ref #606 --- has asyncpg fixed this in their case by now?

Let's do (or don't) the warning on another PR. I'll investigate asyncpg

looks like it's fixed in asyncpg: https://github.com/MagicStack/asyncpg/blob/e9bb695413b2ee923310f9106277198cc19c47b3/asyncpg/_asyncio_compat.py#L25-L73

src/anyio/_backends/_asyncio.py

Co-authored-by: Thomas Grainger <[email protected]>

src/anyio/_backends/_asyncio.py

Bumps [anyio](https://github.com/agronholm/anyio) from 4.6.2.post1 to 4.7.0. <details> <summary>Release notes</summary> Sourced from <a href="https://github.com/agronholm/anyio/releases">anyio's releases</a>. <blockquote> <h2>4.7.0</h2> <ul> <li>Updated <code>TaskGroup</code> to work with asyncio's eager task factories (<a href="https://redirect.github.com/agronholm/anyio/issues/764">#764</a>)</li> <li>Added the <code>wait_readable()</code> and <code>wait_writable()</code> functions which will accept an object with a <code>.fileno()</code> method or an integer handle, and deprecated their now obsolete versions (<code>wait_socket_readable()</code> and <code>wait_socket_writable()</code>) (PR by <a href="https://github.com/davidbrochart"><code>@davidbrochart</code></a>)</li> <li>Changed <code>EventAdapter</code> (an <code>Event</code> with no bound async backend) to allow <code>set()</code> to work even before an async backend is bound to it (<a href="https://redirect.github.com/agronholm/anyio/issues/819">#819</a>)</li> <li>Added support for <code>wait_readable()</code> and <code>wait_writable()</code> on <code>ProactorEventLoop</code> (used on asyncio + Windows by default)</li> <li>Fixed a misleading <code>ValueError</code> in the context of DNS failures (<a href="https://redirect.github.com/agronholm/anyio/issues/815">#815</a>; PR by <a href="https://github.com/graingert"><code>@graingert</code></a>)</li> <li>Fixed the return type annotations of <code>readinto()</code> and <code>readinto1()</code> methods in the <code>anyio.AsyncFile</code> class (<a href="https://redirect.github.com/agronholm/anyio/issues/825">#825</a>)</li> <li>Fixed <code>TaskInfo.has_pending_cancellation()</code> on asyncio returning false positives in cleanup code on Python >= 3.11 (<a href="https://redirect.github.com/agronholm/anyio/issues/832">#832</a>; PR by <a href="https://github.com/gschaffner"><code>@gschaffner</code></a>)</li> <li>Fixed cancelled cancel scopes on asyncio calling <code>asyncio.Task.uncancel</code> when propagating a <code>CancelledError</code> on exit to a cancelled parent scope (<a href="https://redirect.github.com/agronholm/anyio/pull/790">#790</a>; PR by <a href="https://github.com/gschaffner"><code>@gschaffner</code></a>)</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> Sourced from <a href="https://github.com/agronholm/anyio/blob/master/docs/versionhistory.rst">anyio's changelog</a>. <blockquote> <h1>Version history</h1> This library adheres to <code>Semantic Versioning 2.0 <http://semver.org/></code>_. 4.7.0 <ul> <li>Updated <code>TaskGroup</code> to work with asyncio's eager task factories (<code>[#764](agronholm/anyio#764) <https://github.com/agronholm/anyio/issues/764></code>_)</li> <li>Added the <code>wait_readable()</code> and <code>wait_writable()</code> functions which will accept an object with a <code>.fileno()</code> method or an integer handle, and deprecated their now obsolete versions (<code>wait_socket_readable()</code> and <code>wait_socket_writable()</code>) (PR by <a href="https://github.com/davidbrochart"><code>@davidbrochart</code></a>)</li> <li>Changed <code>EventAdapter</code> (an <code>Event</code> with no bound async backend) to allow <code>set()</code> to work even before an async backend is bound to it (<code>[#819](agronholm/anyio#819) <https://github.com/agronholm/anyio/issues/819></code>_)</li> <li>Added support for <code>wait_readable()</code> and <code>wait_writable()</code> on <code>ProactorEventLoop</code> (used on asyncio + Windows by default)</li> <li>Fixed a misleading <code>ValueError</code> in the context of DNS failures (<code>[#815](agronholm/anyio#815) <https://github.com/agronholm/anyio/issues/815></code>_; PR by <a href="https://github.com/graingert"><code>@graingert</code></a>)</li> <li>Fixed the return type annotations of <code>readinto()</code> and <code>readinto1()</code> methods in the <code>anyio.AsyncFile</code> class (<code>[#825](agronholm/anyio#825) <https://github.com/agronholm/anyio/issues/825></code>_)</li> <li>Fixed <code>TaskInfo.has_pending_cancellation()</code> on asyncio returning false positives in cleanup code on Python >= 3.11 (<code>[#832](agronholm/anyio#832) <https://github.com/agronholm/anyio/issues/832></code>_; PR by <a href="https://github.com/gschaffner"><code>@gschaffner</code></a>)</li> <li>Fixed cancelled cancel scopes on asyncio calling <code>asyncio.Task.uncancel</code> when propagating a <code>CancelledError</code> on exit to a cancelled parent scope (<code>[#790](agronholm/anyio#790) <https://github.com/agronholm/anyio/pull/790></code>_; PR by <a href="https://github.com/gschaffner"><code>@gschaffner</code></a>)</li> </ul> 4.6.2 <ul> <li>Fixed regression caused by (<code>[#807](agronholm/anyio#807) <https://github.com/agronholm/anyio/pull/807></code>_) that prevented the use of parametrized async fixtures</li> </ul> 4.6.1 This release contains all the changes from both v4.5.1 and v4.6.0, plus: <ul> <li>Fixed TaskGroup and CancelScope producing cyclic references in tracebacks when raising exceptions (<code>[#806](agronholm/anyio#806) <https://github.com/agronholm/anyio/pull/806></code>_) (PR by <a href="https://github.com/graingert"><code>@graingert</code></a>)</li> </ul> 4.6.0 This release is the successor to v4.5.0 with Python 3.8 support dropped, and does not contain the changes from v4.5.1. <ul> <li>Dropped support for Python 3.8 (as <code>[#698](agronholm/anyio#698) <https://github.com/agronholm/anyio/issues/698></code>_ cannot be resolved</li> </ul>  </blockquote> ... (truncated) </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/agronholm/anyio/commit/c967f5c644b4e535af4cb230d9bcb8be06e2559f"><code>c967f5c</code></a> Bumped up the version</li> <li><a href="https://github.com/agronholm/anyio/commit/f47ac5eb32b695ba76b94f811c4fa0e2d040c311"><code>f47ac5e</code></a> Rearranged changelog entries</li> <li><a href="https://github.com/agronholm/anyio/commit/f316ce5cefb3fa4ebb87518fa58ea205230ba24e"><code>f316ce5</code></a> Allowed Event to be set before it's bound to an async backend (<a href="https://redirect.github.com/agronholm/anyio/issues/835">#835</a>)</li> <li><a href="https://github.com/agronholm/anyio/commit/93a57466c3b226ef2bc6fb2f0d361ed9ce8f707c"><code>93a5746</code></a> Fixed <code>asyncio.Task.cancelling</code> issues (<a href="https://redirect.github.com/agronholm/anyio/issues/790">#790</a>)</li> <li><a href="https://github.com/agronholm/anyio/commit/39cf394713f06f1d008a18b489ad042252ddb469"><code>39cf394</code></a> Avoid exposing extra variables key and value in anyio.abc (<a href="https://redirect.github.com/agronholm/anyio/issues/833">#833</a>)</li> <li><a href="https://github.com/agronholm/anyio/commit/a2150f525ea96fa6c14f9a1adca5ee36b9b61992"><code>a2150f5</code></a> Fixed design issues in PR template (<a href="https://redirect.github.com/agronholm/anyio/issues/834">#834</a>)</li> <li><a href="https://github.com/agronholm/anyio/commit/0f8061148b3c038c2211a2e37c63f7ce94c0a2a4"><code>0f80611</code></a> Added support for wait_readable() and wait_writable() on ProactorEventLoop (#...</li> <li><a href="https://github.com/agronholm/anyio/commit/97d5fe6f92608807f4e1a2feed8be80d280250a8"><code>97d5fe6</code></a> Made asyncio TaskGroup work with eager task factories (<a href="https://redirect.github.com/agronholm/anyio/issues/822">#822</a>)</li> <li><a href="https://github.com/agronholm/anyio/commit/44405f4a6c33cea9c0e0e4cf169248fd800e6ffe"><code>44405f4</code></a> Updated downstream test workflows and their target Python versions</li> <li><a href="https://github.com/agronholm/anyio/commit/93c0cd61b5c3b296fa4115957eab8f76a9cf23a1"><code>93c0cd6</code></a> Replaced mentions to run_sync_in_process with to_process.run_sync (<a href="https://redirect.github.com/agronholm/anyio/issues/829">#829</a>)</li> <li>Additional commits viewable in <a href="https://github.com/agronholm/anyio/compare/4.6.2.post1...4.7.0">compare view</a></li> </ul> </details> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=anyio&package-manager=pip&previous-version=4.6.2.post1&new-version=4.7.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Shrink TaskState to save a little memory

7c0c02a

agronholm reviewed Sep 22, 2024

View reviewed changes

src/anyio/_backends/_asyncio.py Outdated Show resolved Hide resolved

gschaffner added 2 commits October 14, 2024 00:00

Fix uncancel() being called too early

579f8d1

Refactor to avoid duplicate computation

72c334a

gschaffner added 3 commits December 4, 2024 00:00

Test TaskInfo.has_pending_cancellation in cleanup code

8cf35fd

Fix TaskInfo.has_pending_cancellation in cleanup code on asyncio

e7ecae0

Test that uncancel() isn't called too early

7464dc3

gschaffner force-pushed the fix-uncancel-too-early branch from ec27a9d to fe3ee45 Compare December 5, 2024 06:57

gschaffner changed the title ~~Fix asyncio.Task.uncancel() being called too early~~ Fix asyncio.Task.cancelling issues Dec 5, 2024

graingert reviewed Dec 5, 2024

View reviewed changes

src/anyio/_backends/_asyncio.py Outdated Show resolved Hide resolved

Update src/anyio/_backends/_asyncio.py

9ddcd61

Co-authored-by: Thomas Grainger <[email protected]>

gschaffner force-pushed the fix-uncancel-too-early branch from 14b6cd4 to 9ddcd61 Compare December 5, 2024 08:22

Merge branch 'master' into fix-uncancel-too-early

810f6fe

agronholm reviewed Dec 5, 2024

View reviewed changes

src/anyio/_backends/_asyncio.py Outdated Show resolved Hide resolved

Update src/anyio/_backends/_asyncio.py

3893475

agronholm approved these changes Dec 5, 2024

View reviewed changes

agronholm merged commit 93a5746 into agronholm:master Dec 5, 2024
16 checks passed

gschaffner deleted the fix-uncancel-too-early branch December 5, 2024 23:45

gschaffner mentioned this pull request Dec 5, 2024

TaskInfo.has_pending_cancellation returns false positives in shielded cleanup on asyncio >= 3.11 #832

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix `asyncio.Task.cancelling` issues #790

Fix `asyncio.Task.cancelling` issues #790

gschaffner commented Sep 22, 2024 •

edited

Loading

agronholm commented Nov 30, 2024

gschaffner commented Dec 5, 2024

graingert Dec 5, 2024

graingert Dec 5, 2024

gschaffner Dec 5, 2024

graingert Dec 5, 2024

smurfix Dec 5, 2024

gschaffner Dec 5, 2024

graingert Dec 5, 2024

graingert Dec 5, 2024

Fix asyncio.Task.cancelling issues #790

Fix asyncio.Task.cancelling issues #790

Conversation

gschaffner commented Sep 22, 2024 • edited Loading

Changes

Checklist

agronholm commented Nov 30, 2024

gschaffner commented Dec 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Fix `asyncio.Task.cancelling` issues #790

Fix `asyncio.Task.cancelling` issues #790

gschaffner commented Sep 22, 2024 •

edited

Loading