Make the computation of `coroutine_captures_by_ref_ty` more sophisticated #123660

compiler-errors · 2024-04-09T01:11:26Z

Currently, we treat all the by-(mut/)ref borrows of a coroutine-closure as having a "closure env" borrowed lifetime.

When we have the given code:

let x: &'a i32 = ...;
let c = async || {
    let _x = *x;
};

Then when we call:

c()
// which, because `AsyncFn` takes a `&self`, we insert an autoref:
(&c /* &'env {coroutine-closure} */)()

We will return a future whose captures contain &'env i32 instead of &'a i32, which is way more restrictive than necessary. We should be able to drop c while the future is alive since it's not actually borrowing any data originating from within the closure's captures, but since the capture has that 'env lifetime, this is not possible.

This wouldn't be true, for example, if the closure captured i32 instead of &'a i32, because the 'env lifetime is actually necessary since the data (i32) is owned by the closure.

This PR identifies two criteria where we need to take the borrow with the closure env lifetime:

If the closure borrows data from inside the closure's captures. This is not true if the parent capture is by-ref, OR if the parent capture is by-move and the child capture begins with a deref projection. This is the example described above.
If we're dealing with mutable references, since we cannot reborrow &'env mut &'a mut i32 into &'a mut i32, only &'env mut i32.

See the documentation on should_reborrow_from_env_of_parent_coroutine_closure for more info.

important: As disclaimer states on that function, luckily, if this heuristic is not correct, then the program is not unsound, since we still borrowck and validate the choices made from this function -- the only side-effect is that the user may receive unnecessary borrowck errors.

Fixes #123241

compiler-errors · 2024-04-09T14:52:26Z

compiler/rustc_middle/src/ty/closure.rs

+            }
+
+            field_used_at_least_once = true;
+            break for_each((parent_field_idx, parent_capture), (child_field_idx, child_capture));


The only thing this fails to do is assert that the parents captures are all used at least once. I'm tempted to use a coroutine for this and just use iter::from_coroutine...

bors · 2024-04-10T05:38:15Z

☔ The latest upstream changes (presumably #123708) made this pull request unmergeable. Please resolve the merge conflicts.

rustbot · 2024-04-10T16:14:26Z

Some changes occurred to MIR optimizations

cc @rust-lang/wg-mir-opt

compiler-errors · 2024-04-10T16:14:28Z

r? oli-obk

bors · 2024-04-10T17:21:15Z

☔ The latest upstream changes (presumably #123725) made this pull request unmergeable. Please resolve the merge conflicts.

…closures

oli-obk · 2024-04-11T08:10:43Z

Nice (and great docs, thanks!)

@bors r+ rollup

bors · 2024-04-11T08:10:45Z

📌 Commit 599d456 has been approved by oli-obk

It is now in the queue for this repository.

…iaskrgr Rollup of 4 pull requests Successful merges: - rust-lang#123660 (Make the computation of `coroutine_captures_by_ref_ty` more sophisticated) - rust-lang#123738 (Call lower_const_param instead of duplicating the code) - rust-lang#123774 (Fix typo MaybeUnit -> MaybeUninit) - rust-lang#123790 (correct the handling of `bootstrap-cache-path` option) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of rust-lang#123660 - compiler-errors:coroutine-closure-env, r=oli-obk Make the computation of `coroutine_captures_by_ref_ty` more sophisticated Currently, we treat all the by-(mut/)ref borrows of a coroutine-closure as having a "closure env" borrowed lifetime. When we have the given code: ```rust let x: &'a i32 = ...; let c = async || { let _x = *x; }; ``` Then when we call: ```rust c() // which, because `AsyncFn` takes a `&self`, we insert an autoref: (&c /* &'env {coroutine-closure} */)() ``` We will return a future whose captures contain `&'env i32` instead of `&'a i32`, which is way more restrictive than necessary. We should be able to drop `c` while the future is alive since it's not actually borrowing any data *originating from within* the closure's captures, but since the capture has that `'env` lifetime, this is not possible. This wouldn't be true, for example, if the closure captured `i32` instead of `&'a i32`, because the `'env` lifetime is actually *necessary* since the data (`i32`) is owned by the closure. This PR identifies two criteria where we *need* to take the borrow with the closure env lifetime: 1. If the closure borrows data from inside the closure's captures. This is not true if the parent capture is by-ref, OR if the parent capture is by-move and the child capture begins with a deref projection. This is the example described above. 2. If we're dealing with mutable references, since we cannot reborrow `&'env mut &'a mut i32` into `&'a mut i32`, *only* `&'env mut i32`. See the documentation on `should_reborrow_from_env_of_parent_coroutine_closure` for more info. **important:** As disclaimer states on that function, luckily, if this heuristic is not correct, then the program is not unsound, since we still borrowck and validate the choices made from this function -- the only side-effect is that the user may receive unnecessary borrowck errors. Fixes rust-lang#123241

… r=oli-obk An async closure may implement `FnMut`/`Fn` if it has no self-borrows There's no reason that async closures may not implement `FnMut` or `Fn` if they don't actually borrow anything with the closure's env lifetime. Specifically, rust-lang#123660 made it so that we don't always need to borrow captures from the closure's env. See the doc comment on `should_reborrow_from_env_of_parent_coroutine_closure`: https://github.com/rust-lang/rust/blob/c00957a3e269219413041a4e3565f33b1f9d0779/compiler/rustc_hir_typeck/src/upvar.rs#L1777-L1823 If there are no such borrows, then we are free to implement `FnMut` and `Fn` as permitted by our closure's inferred `ClosureKind`. As far as I can tell, this change makes `async || {}` work in precisely the set of places they used to work before rust-lang#120361. Fixes rust-lang#125247. r? oli-obk

Rollup merge of rust-lang#125259 - compiler-errors:fn-mut-as-a-treat, r=oli-obk An async closure may implement `FnMut`/`Fn` if it has no self-borrows There's no reason that async closures may not implement `FnMut` or `Fn` if they don't actually borrow anything with the closure's env lifetime. Specifically, rust-lang#123660 made it so that we don't always need to borrow captures from the closure's env. See the doc comment on `should_reborrow_from_env_of_parent_coroutine_closure`: https://github.com/rust-lang/rust/blob/c00957a3e269219413041a4e3565f33b1f9d0779/compiler/rustc_hir_typeck/src/upvar.rs#L1777-L1823 If there are no such borrows, then we are free to implement `FnMut` and `Fn` as permitted by our closure's inferred `ClosureKind`. As far as I can tell, this change makes `async || {}` work in precisely the set of places they used to work before rust-lang#120361. Fixes rust-lang#125247. r? oli-obk

…i-obk Stabilize async closures (RFC 3668) # Async Closures Stabilization Report This report proposes the stabilization of `#![feature(async_closure)]` ([RFC 3668](https://rust-lang.github.io/rfcs/3668-async-closures.html)). This is a long-awaited feature that increases the expressiveness of the Rust language and fills a pressing gap in the async ecosystem. ## Stabilization summary * You can write async closures like `async || {}` which return futures that can borrow from their captures and can be higher-ranked in their argument lifetimes. * You can express trait bounds for these async closures using the `AsyncFn` family of traits, analogous to the `Fn` family. ```rust async fn takes_an_async_fn(f: impl AsyncFn(&str)) { futures::join(f("hello"), f("world")).await; } takes_an_async_fn(async |s| { other_fn(s).await }).await; ``` ## Motivation Without this feature, users hit two major obstacles when writing async code that uses closures and `Fn` trait bounds: - The inability to express higher-ranked async function signatures. - That closures cannot return futures that borrow from the closure captures. That is, for the first, we cannot write: ```rust // We cannot express higher-ranked async function signatures. async fn f<Fut>(_: impl for<'a> Fn(&'a u8) -> Fut) where Fut: Future<Output = ()>, { todo!() } async fn main() { async fn g(_: &u8) { todo!() } f(g).await; //~^ ERROR mismatched types //~| ERROR one type is more general than the other } ``` And for the second, we cannot write: ```rust // Closures cannot return futures that borrow closure captures. async fn f<Fut: Future<Output = ()>>(_: impl FnMut() -> Fut) { todo!() } async fn main() { let mut xs = vec![]; f(|| async { async fn g() -> u8 { todo!() } xs.push(g().await); }); //~^ ERROR captured variable cannot escape `FnMut` closure body } ``` Async closures provide a first-class solution to these problems. For further background, please refer to the [motivation section](https://rust-lang.github.io/rfcs/3668-async-closures.html#motivation) of the RFC. ## Major design decisions since RFC The RFC had left open the question of whether we would spell the bounds syntax for async closures... ```rust // ...as this... fn f() -> impl AsyncFn() -> u8 { todo!() } // ...or as this: fn f() -> impl async Fn() -> u8 { todo!() } ``` We've decided to spell this as `AsyncFn{,Mut,Once}`. The `Fn` family of traits is special in many ways. We had originally argued that, due to this specialness, that perhaps the `async Fn` syntax could be adopted without having to decide whether a general `async Trait` mechanism would ever be adopted. However, concerns have been raised that we may not want to use `async Fn` syntax unless we would pursue more general trait modifiers. Since there remain substantial open questions on those -- and we don't want to rush any design work there -- it makes sense to ship this needed feature using the `AsyncFn`-style bounds syntax. Since we would, in no case, be shipping a generalized trait modifier system anytime soon, we'll be continuing to see `AsyncFoo` traits appear across the ecosystem regardless. If we were to ever later ship some general mechanism, we could at that time manage the migration from `AsyncFn` to `async Fn`, just as we'd be enabling and managing the migration of many other traits. Note that, as specified in RFC 3668, the details of the `AsyncFn*` traits are not exposed and they can only be named via the "parentheses sugar". That is, we can write `T: AsyncFn() -> u8` but not `T: AsyncFn<Output = u8>`. Unlike the `Fn` traits, we cannot project to the `Output` associated type of the `AsyncFn` traits. That is, while we can write... ```rust fn f<F: Fn() -> u8>(_: F::Output) {} ``` ...we cannot write: ```rust fn f<F: AsyncFn() -> u8>(_: F::Output) {} //~^ ERROR ``` The choice of `AsyncFn{,Mut,Once}` bounds syntax obviates, for our purposes here, another question decided after that RFC, which was how to order bound modifiers such as `for<'a> async Fn()`. Other than answering the open question in the RFC on syntax, nothing has changed about the design of this feature between RFC 3668 and this stabilization. ## What is stabilized For those interested in the technical details, please see [the dev guide section](https://rustc-dev-guide.rust-lang.org/coroutine-closures.html) I authored. #### Async closures Other than in how they solve the problems described above, async closures act similarly to closures that return async blocks, and can have parts of their signatures specified: ```rust // They can have arguments annotated with types: let _ = async |_: u8| { todo!() }; // They can have their return types annotated: let _ = async || -> u8 { todo!() }; // They can be higher-ranked: let _ = async |_: &str| { todo!() }; // They can capture values by move: let x = String::from("hello, world"); let _ = async move || do_something(&x).await }; ``` When called, they return an anonymous future type corresponding to the (not-yet-executed) body of the closure. These can be awaited like any other future. What distinguishes async closures is that, unlike closures that return async blocks, the futures returned from the async closure can capture state from the async closure. For example: ```rust let vec: Vec<String> = vec![]; let closure = async || { vec.push(ready(String::from("")).await); }; ``` The async closure captures `vec` with some `&'closure mut Vec<String>` which lives until the closure is dropped. Every call to `closure()` returns a future which reborrows that mutable reference `&'call mut Vec<String>` which lives until the future is dropped (e.g. it is `await`ed). As another example: ```rust let string: String = "Hello, world".into(); let closure = async move || { ready(&string).await; }; ``` The closure is marked with `move`, which means it takes ownership of the string by *value*. The future that is returned by calling `closure()` returns a future which borrows a reference `&'call String` which lives until the future is dropped (e.g. it is `await`ed). #### Async fn trait family To support the lending capability of async closures, and to provide a first-class way to express higher-ranked async closures, we introduce the `AsyncFn*` family of traits. See the [corresponding section](https://rust-lang.github.io/rfcs/3668-async-closures.html#asyncfn) of the RFC. We stabilize naming `AsyncFn*` via the "parenthesized sugar" syntax that normal `Fn*` traits can be named. The `AsyncFn*` trait can be used anywhere a `Fn*` trait bound is allowed, such as: ```rust /// In return-position impl trait: fn closure() -> impl AsyncFn() { async || {} } /// In trait bounds: trait Foo<F>: Sized where F: AsyncFn() { fn new(f: F) -> Self; } /// in GATs: trait Gat { type AsyncHasher<T>: AsyncFn(T) -> i32; } ``` Other than using them in trait bounds, the definitions of these traits are not directly observable, but certain aspects of their behavior can be indirectly observed such as the fact that: * `AsyncFn::async_call` and `AsyncFnMut::async_call_mut` return a future which is *lending*, and therefore borrows the `&self` lifetime of the callee. ```rust fn by_ref_call(c: impl AsyncFn()) { let fut = c(); drop(c); // ^ Cannot drop `c` since it is borrowed by `fut`. } ``` * `AsyncFnOnce::async_call_once` returns a future that takes ownership of the callee. ```rust fn by_ref_call(c: impl AsyncFnOnce()) { let fut = c(); let _ = c(); // ^ Cannot call `c` since calling it takes ownership the callee. } ``` * All currently-stable callable types (i.e., closures, function items, function pointers, and `dyn Fn*` trait objects) automatically implement `AsyncFn*() -> T` if they implement `Fn*() -> Fut` for some output type `Fut`, and `Fut` implements `Future<Output = T>`. * This is to make sure that `AsyncFn*()` trait bounds have maximum compatibility with existing callable types which return futures, such as async function items and closures which return boxed futures. * For now, this only works currently for *concrete* callable types -- for example, a argument-position impl trait like `impl Fn() -> impl Future<Output = ()>` does not implement `AsyncFn()`, due to the fact that a `AsyncFn`-if-`Fn` blanket impl does not exist in reality. This may be relaxed in the future. Users can work around this by wrapping their type in an async closure and calling it. I expect this to not matter much in practice, as users are encouraged to write `AsyncFn` bounds directly. ```rust fn is_async_fn(_: impl AsyncFn(&str)) {} async fn async_fn_item(s: &str) { todo!() } is_async_fn(s); // ^^^ This works. fn generic(f: impl Fn() -> impl Future<Output = ()>) { is_async_fn(f); // ^^^ This does not work (yet). } ``` #### The by-move future When async closures are called with `AsyncFn`/`AsyncFnMut`, they return a coroutine that borrows from the closure. However, when they are called via `AsyncFnOnce`, we consume that closure, and cannot return a coroutine that borrows from data that is now dropped. To work around around this limitation, we synthesize a separate future type for calling the async closure via `AsyncFnOnce`. This future executes identically to the by-ref future returned from calling the async closure, except for the fact that it has a different set of captures, since we must *move* the captures from the parent async into the child future. #### Interactions between async closures and the `Fn*` family of traits Async closures always implement `FnOnce`, since they always can be called once. They may also implement `Fn` or `FnMut` if their body is compatible with the calling mode (i.e. if they do not mutate their captures, or they do not capture their captures, respectively) and if the future returned by the async closure is not *lending*. ```rust let id = String::new(); let mapped: Vec</* impl Future */> = [/* elements */] .into_iter() // `Iterator::map` takes an `impl FnMut` .map(async |element| { do_something(&id, element).await; }) .collect(); ``` See [the dev guide](https://rustc-dev-guide.rust-lang.org/coroutine-closures.html#follow-up-when-do-async-closures-implement-the-regular-fn-traits) for a detailed explanation for the situations where this may not be possible due to the lending nature of async closures. #### Other notable features of async closures shared with synchronous closures * Async closures are `Copy` and/or `Clone` if their captures are `Copy`/`Clone`. * Async closures do closure signature inference: If an async closure is passed to a function with a `AsyncFn` or `Fn` trait bound, we can eagerly infer the argument types of the closure. More details are provided in [the dev guide](https://rustc-dev-guide.rust-lang.org/coroutine-closures.html#closure-signature-inference). #### Lints This PR also stabilizes the `CLOSURE_RETURNING_ASYNC_BLOCK` lint as an `allow` lint. This lints on "old-style" async closures: ```rust #![warn(closure_returning_async_block)] let c = |x: &str| async {}; ``` We should encourage users to use `async || {}` where possible. This lint remains `allow` and may be refined in the future because it has a few false positives (namely, see: "Where do we expect rewriting `|| async {}` into `async || {}` to fail?") An alternative that could be made at the time of stabilization is to put this lint behind another gate, so we can decide to stabilize it later. ## What isn't stabilized (aka, potential future work) #### `async Fn*()` bound syntax We decided to stabilize async closures without the `async Fn*()` bound modifier syntax. The general direction of this syntax and how it fits is still being considered by T-lang (e.g. in [RFC 3710](rust-lang/rfcs#3710)). #### Naming the futures returned by async closures This stabilization PR does not provide a way of naming the futures returned by calling `AsyncFn*`. Exposing a stable way to refer to these futures is important for building async-closure-aware combinators, and will be an important future step. #### Return type notation-style bounds for async closures The RFC described an RTN-like syntax for putting bounds on the future returned by an async closure: ```rust async fn foo(x: F) -> Result<()> where F: AsyncFn(&str) -> Result<()>, // The future from calling `F` is `Send` and `'static`. F(..): Send + 'static, {} ``` This stabilization PR does not stabilize that syntax yet, which remains unimplemented (though will be soon). #### `dyn AsyncFn*()` `AsyncFn*` are not dyn-compatible yet. This will likely be implemented in the future along with the dyn-compatibility of async fn in trait, since the same issue (dealing with the future returned by a call) applies there. ## Tests Tests exist for this feature in [`tests/ui/async-await/async-closures`](https://github.com/rust-lang/rust/tree/5b542866400ad4a294f468cfa7e059d95c27a079/tests/ui/async-await/async-closures). <details> <summary>A selected set of tests:</summary> * Lending behavior of async closures * `tests/ui/async-await/async-closures/mutate.rs` * `tests/ui/async-await/async-closures/captures.rs` * `tests/ui/async-await/async-closures/precise-captures.rs` * `tests/ui/async-await/async-closures/no-borrow-from-env.rs` * Async closures may be higher-ranked * `tests/ui/async-await/async-closures/higher-ranked.rs` * `tests/ui/async-await/async-closures/higher-ranked-return.rs` * Async closures may implement `Fn*` traits * `tests/ui/async-await/async-closures/is-fn.rs` * `tests/ui/async-await/async-closures/implements-fnmut.rs` * Async closures may be cloned * `tests/ui/async-await/async-closures/clone-closure.rs` * Ownership of the upvars when `AsyncFnOnce` is called * `tests/ui/async-await/async-closures/drop.rs` * `tests/ui/async-await/async-closures/move-is-async-fn.rs` * `tests/ui/async-await/async-closures/force-move-due-to-inferred-kind.rs` * `tests/ui/async-await/async-closures/force-move-due-to-actually-fnonce.rs` * Closure signature inference * `tests/ui/async-await/async-closures/signature-deduction.rs` * `tests/ui/async-await/async-closures/sig-from-bare-fn.rs` * `tests/ui/async-await/async-closures/signature-inference-from-two-part-bound.rs` </details> ## Remaining bugs and open issues * rust-lang#120694 tracks moving onto more general `LendingFn*` traits. No action needed, since it's not observable. * rust-lang#124020 - Polymorphization ICE. Polymorphization needs to be heavily reworked. No action needed. * rust-lang#127227 - Tracking reworking the way that rustdoc re-sugars bounds. * The part relevant to to `AsyncFn` is fixed by rust-lang#132697. ## Where do we expect rewriting `|| async {}` into `async || {}` to fail? * Fn pointer coercions * Currently, it is not possible to coerce an async closure to an fn pointer like regular closures can be. This functionality may be implemented in the future. ```rust let x: fn() -> _ = async || {}; ``` * Argument capture * Like async functions, async closures always capture their input arguments. This is in contrast to something like `|t: T| async {}`, which doesn't capture `t` unless it is used in the async block. This may affect the `Send`-ness of the future or affect its outlives. ```rust fn needs_send_future(_: impl Fn(NotSendArg) -> Fut) where Fut: Future<Output = ()>, {} needs_send_future(async |_| {}); ``` ## History #### Important feature history - rust-lang#51580 - rust-lang#62292 - rust-lang#120361 - rust-lang#120712 - rust-lang#121857 - rust-lang#123660 - rust-lang#125259 - rust-lang#128506 - rust-lang#127482 ## Acknowledgements Thanks to `@oli-obk` for reviewing the bulk of the work for this feature. Thanks to `@nikomatsakis` for his design blog posts which generated interest for this feature, `@traviscross` for feedback and additions to this stabilization report. All errors are my own. r? `@ghost`

Stabilize async closures (RFC 3668) # Async Closures Stabilization Report This report proposes the stabilization of `#![feature(async_closure)]` ([RFC 3668](https://rust-lang.github.io/rfcs/3668-async-closures.html)). This is a long-awaited feature that increases the expressiveness of the Rust language and fills a pressing gap in the async ecosystem. ## Stabilization summary * You can write async closures like `async || {}` which return futures that can borrow from their captures and can be higher-ranked in their argument lifetimes. * You can express trait bounds for these async closures using the `AsyncFn` family of traits, analogous to the `Fn` family. ```rust async fn takes_an_async_fn(f: impl AsyncFn(&str)) { futures::join(f("hello"), f("world")).await; } takes_an_async_fn(async |s| { other_fn(s).await }).await; ``` ## Motivation Without this feature, users hit two major obstacles when writing async code that uses closures and `Fn` trait bounds: - The inability to express higher-ranked async function signatures. - That closures cannot return futures that borrow from the closure captures. That is, for the first, we cannot write: ```rust // We cannot express higher-ranked async function signatures. async fn f<Fut>(_: impl for<'a> Fn(&'a u8) -> Fut) where Fut: Future<Output = ()>, { todo!() } async fn main() { async fn g(_: &u8) { todo!() } f(g).await; //~^ ERROR mismatched types //~| ERROR one type is more general than the other } ``` And for the second, we cannot write: ```rust // Closures cannot return futures that borrow closure captures. async fn f<Fut: Future<Output = ()>>(_: impl FnMut() -> Fut) { todo!() } async fn main() { let mut xs = vec![]; f(|| async { async fn g() -> u8 { todo!() } xs.push(g().await); }); //~^ ERROR captured variable cannot escape `FnMut` closure body } ``` Async closures provide a first-class solution to these problems. For further background, please refer to the [motivation section](https://rust-lang.github.io/rfcs/3668-async-closures.html#motivation) of the RFC. ## Major design decisions since RFC The RFC had left open the question of whether we would spell the bounds syntax for async closures... ```rust // ...as this... fn f() -> impl AsyncFn() -> u8 { todo!() } // ...or as this: fn f() -> impl async Fn() -> u8 { todo!() } ``` We've decided to spell this as `AsyncFn{,Mut,Once}`. The `Fn` family of traits is special in many ways. We had originally argued that, due to this specialness, that perhaps the `async Fn` syntax could be adopted without having to decide whether a general `async Trait` mechanism would ever be adopted. However, concerns have been raised that we may not want to use `async Fn` syntax unless we would pursue more general trait modifiers. Since there remain substantial open questions on those -- and we don't want to rush any design work there -- it makes sense to ship this needed feature using the `AsyncFn`-style bounds syntax. Since we would, in no case, be shipping a generalized trait modifier system anytime soon, we'll be continuing to see `AsyncFoo` traits appear across the ecosystem regardless. If we were to ever later ship some general mechanism, we could at that time manage the migration from `AsyncFn` to `async Fn`, just as we'd be enabling and managing the migration of many other traits. Note that, as specified in RFC 3668, the details of the `AsyncFn*` traits are not exposed and they can only be named via the "parentheses sugar". That is, we can write `T: AsyncFn() -> u8` but not `T: AsyncFn<Output = u8>`. Unlike the `Fn` traits, we cannot project to the `Output` associated type of the `AsyncFn` traits. That is, while we can write... ```rust fn f<F: Fn() -> u8>(_: F::Output) {} ``` ...we cannot write: ```rust fn f<F: AsyncFn() -> u8>(_: F::Output) {} //~^ ERROR ``` The choice of `AsyncFn{,Mut,Once}` bounds syntax obviates, for our purposes here, another question decided after that RFC, which was how to order bound modifiers such as `for<'a> async Fn()`. Other than answering the open question in the RFC on syntax, nothing has changed about the design of this feature between RFC 3668 and this stabilization. ## What is stabilized For those interested in the technical details, please see [the dev guide section](https://rustc-dev-guide.rust-lang.org/coroutine-closures.html) I authored. #### Async closures Other than in how they solve the problems described above, async closures act similarly to closures that return async blocks, and can have parts of their signatures specified: ```rust // They can have arguments annotated with types: let _ = async |_: u8| { todo!() }; // They can have their return types annotated: let _ = async || -> u8 { todo!() }; // They can be higher-ranked: let _ = async |_: &str| { todo!() }; // They can capture values by move: let x = String::from("hello, world"); let _ = async move || do_something(&x).await }; ``` When called, they return an anonymous future type corresponding to the (not-yet-executed) body of the closure. These can be awaited like any other future. What distinguishes async closures is that, unlike closures that return async blocks, the futures returned from the async closure can capture state from the async closure. For example: ```rust let vec: Vec<String> = vec![]; let closure = async || { vec.push(ready(String::from("")).await); }; ``` The async closure captures `vec` with some `&'closure mut Vec<String>` which lives until the closure is dropped. Every call to `closure()` returns a future which reborrows that mutable reference `&'call mut Vec<String>` which lives until the future is dropped (e.g. it is `await`ed). As another example: ```rust let string: String = "Hello, world".into(); let closure = async move || { ready(&string).await; }; ``` The closure is marked with `move`, which means it takes ownership of the string by *value*. The future that is returned by calling `closure()` returns a future which borrows a reference `&'call String` which lives until the future is dropped (e.g. it is `await`ed). #### Async fn trait family To support the lending capability of async closures, and to provide a first-class way to express higher-ranked async closures, we introduce the `AsyncFn*` family of traits. See the [corresponding section](https://rust-lang.github.io/rfcs/3668-async-closures.html#asyncfn) of the RFC. We stabilize naming `AsyncFn*` via the "parenthesized sugar" syntax that normal `Fn*` traits can be named. The `AsyncFn*` trait can be used anywhere a `Fn*` trait bound is allowed, such as: ```rust /// In return-position impl trait: fn closure() -> impl AsyncFn() { async || {} } /// In trait bounds: trait Foo<F>: Sized where F: AsyncFn() { fn new(f: F) -> Self; } /// in GATs: trait Gat { type AsyncHasher<T>: AsyncFn(T) -> i32; } ``` Other than using them in trait bounds, the definitions of these traits are not directly observable, but certain aspects of their behavior can be indirectly observed such as the fact that: * `AsyncFn::async_call` and `AsyncFnMut::async_call_mut` return a future which is *lending*, and therefore borrows the `&self` lifetime of the callee. ```rust fn by_ref_call(c: impl AsyncFn()) { let fut = c(); drop(c); // ^ Cannot drop `c` since it is borrowed by `fut`. } ``` * `AsyncFnOnce::async_call_once` returns a future that takes ownership of the callee. ```rust fn by_ref_call(c: impl AsyncFnOnce()) { let fut = c(); let _ = c(); // ^ Cannot call `c` since calling it takes ownership the callee. } ``` * All currently-stable callable types (i.e., closures, function items, function pointers, and `dyn Fn*` trait objects) automatically implement `AsyncFn*() -> T` if they implement `Fn*() -> Fut` for some output type `Fut`, and `Fut` implements `Future<Output = T>`. * This is to make sure that `AsyncFn*()` trait bounds have maximum compatibility with existing callable types which return futures, such as async function items and closures which return boxed futures. * For now, this only works currently for *concrete* callable types -- for example, a argument-position impl trait like `impl Fn() -> impl Future<Output = ()>` does not implement `AsyncFn()`, due to the fact that a `AsyncFn`-if-`Fn` blanket impl does not exist in reality. This may be relaxed in the future. Users can work around this by wrapping their type in an async closure and calling it. I expect this to not matter much in practice, as users are encouraged to write `AsyncFn` bounds directly. ```rust fn is_async_fn(_: impl AsyncFn(&str)) {} async fn async_fn_item(s: &str) { todo!() } is_async_fn(s); // ^^^ This works. fn generic(f: impl Fn() -> impl Future<Output = ()>) { is_async_fn(f); // ^^^ This does not work (yet). } ``` #### The by-move future When async closures are called with `AsyncFn`/`AsyncFnMut`, they return a coroutine that borrows from the closure. However, when they are called via `AsyncFnOnce`, we consume that closure, and cannot return a coroutine that borrows from data that is now dropped. To work around around this limitation, we synthesize a separate future type for calling the async closure via `AsyncFnOnce`. This future executes identically to the by-ref future returned from calling the async closure, except for the fact that it has a different set of captures, since we must *move* the captures from the parent async into the child future. #### Interactions between async closures and the `Fn*` family of traits Async closures always implement `FnOnce`, since they always can be called once. They may also implement `Fn` or `FnMut` if their body is compatible with the calling mode (i.e. if they do not mutate their captures, or they do not capture their captures, respectively) and if the future returned by the async closure is not *lending*. ```rust let id = String::new(); let mapped: Vec</* impl Future */> = [/* elements */] .into_iter() // `Iterator::map` takes an `impl FnMut` .map(async |element| { do_something(&id, element).await; }) .collect(); ``` See [the dev guide](https://rustc-dev-guide.rust-lang.org/coroutine-closures.html#follow-up-when-do-async-closures-implement-the-regular-fn-traits) for a detailed explanation for the situations where this may not be possible due to the lending nature of async closures. #### Other notable features of async closures shared with synchronous closures * Async closures are `Copy` and/or `Clone` if their captures are `Copy`/`Clone`. * Async closures do closure signature inference: If an async closure is passed to a function with a `AsyncFn` or `Fn` trait bound, we can eagerly infer the argument types of the closure. More details are provided in [the dev guide](https://rustc-dev-guide.rust-lang.org/coroutine-closures.html#closure-signature-inference). #### Lints This PR also stabilizes the `CLOSURE_RETURNING_ASYNC_BLOCK` lint as an `allow` lint. This lints on "old-style" async closures: ```rust #![warn(closure_returning_async_block)] let c = |x: &str| async {}; ``` We should encourage users to use `async || {}` where possible. This lint remains `allow` and may be refined in the future because it has a few false positives (namely, see: "Where do we expect rewriting `|| async {}` into `async || {}` to fail?") An alternative that could be made at the time of stabilization is to put this lint behind another gate, so we can decide to stabilize it later. ## What isn't stabilized (aka, potential future work) #### `async Fn*()` bound syntax We decided to stabilize async closures without the `async Fn*()` bound modifier syntax. The general direction of this syntax and how it fits is still being considered by T-lang (e.g. in [RFC 3710](rust-lang/rfcs#3710)). #### Naming the futures returned by async closures This stabilization PR does not provide a way of naming the futures returned by calling `AsyncFn*`. Exposing a stable way to refer to these futures is important for building async-closure-aware combinators, and will be an important future step. #### Return type notation-style bounds for async closures The RFC described an RTN-like syntax for putting bounds on the future returned by an async closure: ```rust async fn foo(x: F) -> Result<()> where F: AsyncFn(&str) -> Result<()>, // The future from calling `F` is `Send` and `'static`. F(..): Send + 'static, {} ``` This stabilization PR does not stabilize that syntax yet, which remains unimplemented (though will be soon). #### `dyn AsyncFn*()` `AsyncFn*` are not dyn-compatible yet. This will likely be implemented in the future along with the dyn-compatibility of async fn in trait, since the same issue (dealing with the future returned by a call) applies there. ## Tests Tests exist for this feature in [`tests/ui/async-await/async-closures`](https://github.com/rust-lang/rust/tree/5b542866400ad4a294f468cfa7e059d95c27a079/tests/ui/async-await/async-closures). <details> <summary>A selected set of tests:</summary> * Lending behavior of async closures * `tests/ui/async-await/async-closures/mutate.rs` * `tests/ui/async-await/async-closures/captures.rs` * `tests/ui/async-await/async-closures/precise-captures.rs` * `tests/ui/async-await/async-closures/no-borrow-from-env.rs` * Async closures may be higher-ranked * `tests/ui/async-await/async-closures/higher-ranked.rs` * `tests/ui/async-await/async-closures/higher-ranked-return.rs` * Async closures may implement `Fn*` traits * `tests/ui/async-await/async-closures/is-fn.rs` * `tests/ui/async-await/async-closures/implements-fnmut.rs` * Async closures may be cloned * `tests/ui/async-await/async-closures/clone-closure.rs` * Ownership of the upvars when `AsyncFnOnce` is called * `tests/ui/async-await/async-closures/drop.rs` * `tests/ui/async-await/async-closures/move-is-async-fn.rs` * `tests/ui/async-await/async-closures/force-move-due-to-inferred-kind.rs` * `tests/ui/async-await/async-closures/force-move-due-to-actually-fnonce.rs` * Closure signature inference * `tests/ui/async-await/async-closures/signature-deduction.rs` * `tests/ui/async-await/async-closures/sig-from-bare-fn.rs` * `tests/ui/async-await/async-closures/signature-inference-from-two-part-bound.rs` </details> ## Remaining bugs and open issues * rust-lang/rust#120694 tracks moving onto more general `LendingFn*` traits. No action needed, since it's not observable. * rust-lang/rust#124020 - Polymorphization ICE. Polymorphization needs to be heavily reworked. No action needed. * rust-lang/rust#127227 - Tracking reworking the way that rustdoc re-sugars bounds. * The part relevant to to `AsyncFn` is fixed by rust-lang/rust#132697. ## Where do we expect rewriting `|| async {}` into `async || {}` to fail? * Fn pointer coercions * Currently, it is not possible to coerce an async closure to an fn pointer like regular closures can be. This functionality may be implemented in the future. ```rust let x: fn() -> _ = async || {}; ``` * Argument capture * Like async functions, async closures always capture their input arguments. This is in contrast to something like `|t: T| async {}`, which doesn't capture `t` unless it is used in the async block. This may affect the `Send`-ness of the future or affect its outlives. ```rust fn needs_send_future(_: impl Fn(NotSendArg) -> Fut) where Fut: Future<Output = ()>, {} needs_send_future(async |_| {}); ``` ## History #### Important feature history - rust-lang/rust#51580 - rust-lang/rust#62292 - rust-lang/rust#120361 - rust-lang/rust#120712 - rust-lang/rust#121857 - rust-lang/rust#123660 - rust-lang/rust#125259 - rust-lang/rust#128506 - rust-lang/rust#127482 ## Acknowledgements Thanks to `@oli-obk` for reviewing the bulk of the work for this feature. Thanks to `@nikomatsakis` for his design blog posts which generated interest for this feature, `@traviscross` for feedback and additions to this stabilization report. All errors are my own. r? `@ghost`

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Apr 9, 2024

This comment has been minimized.

Sign in to view

compiler-errors mentioned this pull request Apr 9, 2024

Async closures are not allowed to reference all captured lifetimes if one of them is invariant #123241

Closed

compiler-errors commented Apr 9, 2024

View reviewed changes

compiler-errors force-pushed the coroutine-closure-env branch from 7dbc68b to 76b84f5 Compare April 9, 2024 15:51

This comment has been minimized.

Sign in to view

compiler-errors force-pushed the coroutine-closure-env branch 2 times, most recently from 3784660 to c5efc2d Compare April 10, 2024 15:50

compiler-errors marked this pull request as ready for review April 10, 2024 16:14

rustbot assigned oli-obk Apr 10, 2024

This comment has been minimized.

Sign in to view

compiler-errors force-pushed the coroutine-closure-env branch from c5efc2d to a183b48 Compare April 10, 2024 16:22

compiler-errors added 2 commits April 10, 2024 13:39

Use a helper to zip together parent and child captures for coroutine-…

568703c

…closures

Make the computation of coroutine_captures_by_ref_ty more sophisticated

599d456

compiler-errors force-pushed the coroutine-closure-env branch from a183b48 to 599d456 Compare April 10, 2024 17:40

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 11, 2024

matthiaskrgr mentioned this pull request Apr 11, 2024

Rollup of 4 pull requests #123795

Merged

bors merged commit ee73660 into rust-lang:master Apr 11, 2024
11 checks passed

rustbot added this to the 1.79.0 milestone Apr 11, 2024

matthiaskrgr mentioned this pull request Apr 11, 2024

ICE: we captured CapturedPlace .. but it was not used in the child coroutine? #123821

Closed

compiler-errors mentioned this pull request Apr 21, 2024

add a couple tests for fixed ICEs. #124240

Merged

This was referenced May 18, 2024

async closure does not implement FnMut because it captures state from its environment for async closures, while closure returning async block works fine #125247

Closed

An async closure may implement FnMut/Fn if it has no self-borrows #125259

Merged

compiler-errors mentioned this pull request Nov 6, 2024

Stabilize async closures (RFC 3668) #132706

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make the computation of `coroutine_captures_by_ref_ty` more sophisticated #123660

Make the computation of `coroutine_captures_by_ref_ty` more sophisticated #123660

compiler-errors commented Apr 9, 2024 •

edited

Loading

This comment has been minimized.

compiler-errors Apr 9, 2024

This comment has been minimized.

bors commented Apr 10, 2024

rustbot commented Apr 10, 2024

compiler-errors commented Apr 10, 2024

This comment has been minimized.

bors commented Apr 10, 2024

oli-obk commented Apr 11, 2024

bors commented Apr 11, 2024

Make the computation of coroutine_captures_by_ref_ty more sophisticated #123660

Make the computation of coroutine_captures_by_ref_ty more sophisticated #123660

Conversation

compiler-errors commented Apr 9, 2024 • edited Loading

This comment has been minimized.

compiler-errors Apr 9, 2024

Choose a reason for hiding this comment

This comment has been minimized.

bors commented Apr 10, 2024

rustbot commented Apr 10, 2024

compiler-errors commented Apr 10, 2024

This comment has been minimized.

bors commented Apr 10, 2024

oli-obk commented Apr 11, 2024

bors commented Apr 11, 2024

Make the computation of `coroutine_captures_by_ref_ty` more sophisticated #123660

Make the computation of `coroutine_captures_by_ref_ty` more sophisticated #123660

compiler-errors commented Apr 9, 2024 •

edited

Loading