parser: Keep current and previous tokens precisely #69006
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
including their unnormalized forms. Add more documentation for them.
rust-highfive
added
the
S-waiting-on-review
Centril
reviewed
Feb 10, 2020
| prev_token_kind: PrevTokenKind, | ||
| /// Equivalent to `unnormalized_prev_token().span`. | ||
| /// FIXME: Remove in favor of `(unnormalized_)prev_token().span`. |
There was a problem hiding this comment.
This is used so frequently that we might want self.prev_span() (unless those occurrences are meant to be replaced with self.prev_token.span).
|
@bors r+ |
|
📌 Commit cd7a428 has been approved by |
bors
added
S-waiting-on-bors
Dylan-DPC-zz
pushed a commit
to Dylan-DPC-zz/rust
that referenced
this pull request
Feb 10, 2020
parser: Keep current and previous tokens precisely ...including their unnormalized forms. Add more documentation for them. Hopefully, this will help to eliminate footguns like rust-lang#68728 (comment). I'll try to address the FIXMEs in separate PRs during the next week. r? @Centril
bors
added a commit
that referenced
this pull request
Feb 10, 2020
Rollup of 6 pull requests Successful merges: - #68694 (Reduce the number of `RefCell`s in `InferCtxt`.) - #68966 (Improve performance of coherence checks) - #68976 (Make `num::NonZeroX::new` an unstable `const fn`) - #68992 (Correctly parse `mut a @ b`) - #69005 (Small graphviz improvements for the new dataflow framework) - #69006 (parser: Keep current and previous tokens precisely) Failed merges: r? @ghost
Dylan-DPC-zz
pushed a commit
to Dylan-DPC-zz/rust
that referenced
this pull request
Feb 12, 2020
parser: Remove `Parser::prev_token_kind` Follow-up to rust-lang#69006. r? @Centril
Centril
added a commit
to Centril/rust
that referenced
this pull request
Feb 18, 2020
parser: Simplify treatment of macro variables in `Parser::bump` Follow-up to rust-lang#69006. Token normalization for `$ident` and `$lifetime` is merged directly into `bump`. Special "unknown macro variable" diagnostic for unexpected `$`s is removed as preventing legal code from compiling (as a result `bump` also doesn't call itself recursively anymore and can't make `prev_token` inconsistent). r? @Centril
This was referenced Feb 22, 2020
Dylan-DPC-zz
pushed a commit
to Dylan-DPC-zz/rust
that referenced
this pull request
Feb 23, 2020
parser: Cleanup `Parser::bump_with` and its uses Follow-up to rust-lang#69006. r? @Centril
Centril
added a commit
to Centril/rust
that referenced
this pull request
Feb 28, 2020
parser: `token` -> `normalized_token`, `nonnormalized_token` -> `token` So, after rust-lang#69006, its follow-ups and an attempt to remove `Parser::prev_span` I came to the conclusion that the unnormalized token and its span is what you want in most cases, so it should be default. Normalization only makes difference in few cases where we are checking against `token::Ident` or `token::Lifetime` specifically. This PR uses `normalized_token` for those cases. Using normalization explicitly means that people writing code should remember about `NtIdent` and `NtLifetime` in general. (That is alleviated by the fact that `token.ident()` and `fn parse_ident_*` are already written.) Remembering about `NtIdent`, was, however, already the case, kind of, because the implicit normalization was performed only for the current/previous token, but not for things like `look_ahead`. As a result, most of token classification methods in `token.rs` already take `NtIdent` into account (this PR fixes a few pre-existing minor mistakes though). The next step is removing `normalized(_prev)_token` entirely and replacing it with `token.ident()` (mostly) and `token.normalize()` (occasionally). I want to make it a separate PR for that and run it though perf. `normalized_token` filled on every bump has both a potential to avoid repeated normalization, and to do unnecessary work in advance (it probably doesn't matter anyway, the normalization is very cheap). r? @Centril
bors
added a commit
that referenced
this pull request
Mar 8, 2020
Centril
added a commit
to Centril/rust
that referenced
this pull request
Mar 9, 2020
rustc_parse: Remove `Parser::normalized(_prev)_token` Perform the "normalization" (renamed to "uninterpolation") on the fly when necessary. The final part of rust-lang#69579 rust-lang#69384 rust-lang#69376 rust-lang#69211 rust-lang#69034 rust-lang#69006. r? @Centril
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
...including their unnormalized forms.
Add more documentation for them.
Hopefully, this will help to eliminate footguns like #68728 (comment).
I'll try to address the FIXMEs in separate PRs during the next week.
r? @Centril