How to parse keeping spans

[antoyo]

Hi.
I tried using the git version (of both syn and quote) to benefit from proc-macro2 but I'm unable to keep the span information.
Here is the code I use:

#[proc_macro_attribute]
pub fn attribute(attributes: TokenStream, input: TokenStream) -> TokenStream {
    //input // Doing that keeps span info, but the following does not keep it.
    let ast: Item = syn::parse(input).unwrap();
    let result = quote! {
        #ast
    };
    let tokens: proc_macro2::TokenStream = result.into();
    tokens.into()
}

Is it how I'm suppose to use syn and quote to have span info?

Also, is it normal that I need to call into() twice?

I use rust nightly:

rustc 1.20.0-nightly (696412de7 2017-07-06)

Thanks for your help.

[dtolnay]

Is it how I'm suppose to use syn and quote to have span info?

You need to opt in to using the new unstable TokenStream API that preserves span info. Otherwise syn is using a compatibility layer around the old TokenStream which does not have span info.

proc-macro2 = { version = "0.1", features = ["unstable"] }

Although it seems to not work right now... @alexcrichton. Without features = ["unstable"]:

 --> src/main.rs:6:1
  |
6 | #[antoyo]
  | ^^^^^^^^^ expected u8, found reference

With features = ["unstable"]:

 --> <macro expansion>:1:1
  |
1 | fn main() { let _: u8 = ""; }
  | ^^ expected u8, found reference

Without #[antoyo], and what I would expect:

 --> src/main.rs:8:17
  |
8 |     let _: u8 = "";
  |                 ^^ expected u8, found reference

[dtolnay]

Also, is it normal that I need to call into() twice?

Looks like it. serde-rs/serde#980 does effectively the same thing:

proc_macro2::TokenStream::from(expanded).into()

I filed #179 to follow up.

[alexcrichton]

Although it seems to not work right now...

Yeah :(. This is rust-lang/rust#43081

[kdy1]

Is rust-lang/rust#43081 really a cause of this?

• returning proc_macro::TokenStream directly works well.

So.. proc_macro::TokenStream has correct span.

• proc_macro2::TokenStream is just a shim

So it also has correct spans.

I found that syn and quote does not expose or enable it's "unstable" feature.

If it is the cause, what do you think about detecting nightly and automatically enabling "unstable" from proc_macro2 build script?

---

Edit: Clarify meaning of pm2 is a shim.

[kdy1]

And I'm not absolutely sure but current incorrect behavior seems like a that of pm2 without unstable feature (or just proc macros 1.1).

[kdy1]

Enabling "unstable" feature does not work.

For experiment, I tried below and I think it's a parsing bug or quasi quoting bug.

// Span is correct
return proc_macro2::TokenStream::from(input).into();

// All span points `fn`
return proc_macro2::TokenStream::from(parse::<Item>(function).unwrap().into_tokens()).into();

I will investigate more as I need this to make error reporting accurate.

---

Current behavior (futures-await) is

error[E0308]: mismatched types
   --> tests/lifetime.rs:12:12
    |
 12 |     #[async]
    |            ^ expected enum `std::result::Result`, found ()
    |
    = note: expected type `std::result::Result<u64, std::num::ParseIntError>`
               found type `()`

[kdy1]

When I change span of fn_token to default, error points annotation instead of fn. So ToTokens seems innocent.
And if I add pub before fn, error points pub.

So I concluded that parser does not advance span. I'll try to fix it.
Can you help me? cc @dtolnay

[kdy1]

Oh I was wrong. Sorry. I noticed that TokenStream was NtItem and has only one valid span, and other spans are just (0..0). I think currently there's no way to bypass this limitation.
I tried collecting it to vec, but it didn't work.

[dtolnay]

I think this must have been fixed upstream as of rustc 1.24.0-nightly (77e189cd7 2017-12-28). I tried the following macro:

#![feature(proc_macro)]

#[macro_use]
extern crate quote;
extern crate syn;
extern crate proc_macro;
use proc_macro::TokenStream;

#[proc_macro]
pub fn roundtrip_item(tokens: TokenStream) -> TokenStream {
    let item = syn::parse::<syn::Item>(tokens).unwrap();
    quote!(#item).into()
}

#![feature(proc_macro)]

extern crate mac;

mac::roundtrip_item! {
    struct Antoyo(Missing);
}

fn main() {}

Building with cargo build --features proc-macro2/unstable I get all the right spans.

error[E0412]: cannot find type `Missing` in this scope
 --> src/main.rs:6:19
  |
6 |     struct Antoyo(Missing);
  |                   ^^^^^^^ not found in this scope

error: aborting due to previous error

Without proc-macro2/unstable the span points to the macro invocation which seems reasonable.

error[E0412]: cannot find type `Missing` in this scope
 --> src/main.rs:5:1
  |
5 | / mac::roundtrip_item! {
6 | |     struct Antoyo(Missing);
7 | | }
  | |_^ not found in this scope

error: aborting due to previous error