feat: new formatter (#6300)

# Description

## Problem

Resolves #5281
Resolves #4768
Resolves #4767
Resolves #4766
Resolves #4558
Resolves #4462
Resolves #3945
Resolves #3312

## Summary

I thought about trying to extend the existing formatter to format more
code, but I couldn't understand it very well: it partially implemented
things, and it lacked comments and some explanation of how it works. I
think some chunks might have been copied from the Rust formatter. I also
took a look into it but though it might be too complex.

[I wrote a formatter in the past for
Crystal](https://github.com/crystal-lang/crystal/blob/master/src/compiler/crystal/tools/formatter.cr)
with a technique that I saw was used in the Java formatter written for
Eclipse. The idea is to traverse the AST together with a Lexer, writing
things along the way, bumping into comments and formatting those, etc.
It works pretty well! But that's not enough for the Noir formatter
because here we also want to automatically wrap long lines (Crystal's
formatter doesn't do that). That part (wrapping) is partially based on
[this excellent blog
post](https://yorickpeterse.com/articles/how-to-write-a-code-formatter/).

Benefits:
- All code kinds are formatted. For example previously traits weren't
formatted.
- Comments are never lost.
- Lambdas appearing as the last argument of a call are formatted nicely
(this was maybe the most trickier part to get right).
- I believe the new code ends up being simpler than before, even though
it's (slightly) longer (previously is was 2138 lines, now it's 6139,
though 2608 lines are tests, so it ends up being 3531 lines, but this
new formatter does many things the old one didn't). I tried to comment
and document it well.
- Adding new formatting rules, new configurations, or enabling
formatting of new syntax should be relatively easy to do.
- There are lots and lots of tests for each of the different scenarios.
The existing overall formatter tests were kept, but with unit tests it's
easier to see how edge cases are formatted.

[Here's Aztec-Packages formatted with the new
formatter](https://github.com/AztecProtocol/aztec-packages/pull/9292/files):
- Max line width seems to be respected more (making it more readable)
- `unsafe { ... }` and `comptime { ... }` expressions will be inlined if
possible (shortening the code and making it more readable)

## Additional Context

Some things are subjective. For example Rust will put a function `where`
clause in a separate line, with each trait bound on its own line. The
new formatter does that too. But Rust will sometimes put an `impl` where
clause on the same line. It's not clear to me why. I chose to always put
`where` clauses on a separate line, but this could easily be changed if
we wanted to.

## Documentation

Check one:
- [x] No documentation needed.
- [ ] Documentation included in this PR.
- [ ] **[For Experimental Features]** Documentation to be submitted in a
separate PR.

# PR Checklist

- [x] I have tested the changes locally.
- [x] I have formatted the changes with [Prettier](https://prettier.io/)
and/or `cargo fmt` on default settings.

---------

Co-authored-by: Akosh Farkash <[email protected]>
Co-authored-by: Tom French <[email protected]>

compiler/noirc_frontend/src/ast/mod.rs

@@ -173,6 +173,12 @@ pub enum GenericTypeArg {
     Named(Ident, UnresolvedType),
 }
 
+#[derive(Debug, PartialEq, Eq, Clone, Hash)]
+pub enum GenericTypeArgKind {
+    Ordered,
+    Named,
+}
+
 #[derive(Debug, Default, PartialEq, Eq, Clone, Hash)]
 pub struct GenericTypeArgs {
     /// Each ordered argument, e.g. `<A, B, C>`
@@ -181,6 +187,9 @@ pub struct GenericTypeArgs {
     /// All named arguments, e.g. `<A = B, C = D, E = F>`.
     /// Used for associated types.
     pub named_args: Vec<(Ident, UnresolvedType)>,
+
+    /// The kind of each argument, in order (in case traversing the generics in order is needed)
+    pub kinds: Vec<GenericTypeArgKind>,
 }
 
 impl GenericTypeArgs {
@@ -351,7 +360,11 @@ impl UnresolvedType {
         let last_segment = path.segments.last_mut().unwrap();
         let generics = last_segment.generics.take();
         let generic_type_args = if let Some(generics) = generics {
-            GenericTypeArgs { ordered_args: generics, named_args: Vec::new() }
+            let mut kinds = Vec::with_capacity(generics.len());
+            for _ in 0..generics.len() {
+                kinds.push(GenericTypeArgKind::Ordered);
+            }
+            GenericTypeArgs { ordered_args: generics, named_args: Vec::new(), kinds }
         } else {
             GenericTypeArgs::default()
         };

compiler/noirc_frontend/src/hir/comptime/display.rs

@@ -860,6 +860,7 @@ fn remove_interned_in_generic_type_args(
         named_args: vecmap(args.named_args, |(name, typ)| {
             (name, remove_interned_in_unresolved_type(interner, typ))
         }),
+        kinds: args.kinds,
     }
 }
 

compiler/noirc_frontend/src/hir/comptime/hir_to_display_ast.rs

@@ -303,7 +303,8 @@ impl Type {
             Type::Struct(def, generics) => {
                 let struct_def = def.borrow();
                 let ordered_args = vecmap(generics, |generic| generic.to_display_ast());
-                let generics = GenericTypeArgs { ordered_args, named_args: Vec::new() };
+                let generics =
+                    GenericTypeArgs { ordered_args, named_args: Vec::new(), kinds: Vec::new() };
                 let name = Path::from_ident(struct_def.name.clone());
                 UnresolvedTypeData::Named(name, generics, false)
             }
@@ -312,7 +313,8 @@ impl Type {
                 // alias' definition was changed
                 let type_def = type_def.borrow();
                 let ordered_args = vecmap(generics, |generic| generic.to_display_ast());
-                let generics = GenericTypeArgs { ordered_args, named_args: Vec::new() };
+                let generics =
+                    GenericTypeArgs { ordered_args, named_args: Vec::new(), kinds: Vec::new() };
                 let name = Path::from_ident(type_def.name.clone());
                 UnresolvedTypeData::Named(name, generics, false)
             }
@@ -330,7 +332,7 @@ impl Type {
                 let named_args = vecmap(&generics.named, |named_type| {
                     (named_type.name.clone(), named_type.typ.to_display_ast())
                 });
-                let generics = GenericTypeArgs { ordered_args, named_args };
+                let generics = GenericTypeArgs { ordered_args, named_args, kinds: Vec::new() };
                 let name = Path::from_single(name.as_ref().clone(), Span::default());
                 UnresolvedTypeData::TraitAsType(name, generics)
             }

compiler/noirc_frontend/src/lexer/lexer.rs

@@ -575,7 +575,11 @@ impl<'a> Lexer<'a> {
             return self.lookup_word_token(word, start, end);
         }
 
-        let delimiter = self.next_token()?;
+        let mut delimiter = self.next_token()?;
+        while let Token::Whitespace(_) = delimiter.token() {
+            delimiter = self.next_token()?;
+        }
+
         let (start_delim, end_delim) = match delimiter.token() {
             Token::LeftBrace => (Token::LeftBrace, Token::RightBrace),
             Token::LeftBracket => (Token::LeftBracket, Token::RightBracket),

compiler/noirc_frontend/src/lexer/token.rs

@@ -377,7 +377,7 @@ impl fmt::Display for Token {
             }
             Token::Keyword(k) => write!(f, "{k}"),
             Token::Attribute(ref a) => write!(f, "{a}"),
-            Token::InnerAttribute(ref a) => write!(f, "#![{a}]"),
+            Token::InnerAttribute(ref a) => write!(f, "#![{}]", a.contents()),
             Token::LineComment(ref s, style) => match style {
                 Some(DocStyle::Inner) => write!(f, "//!{s}"),
                 Some(DocStyle::Outer) => write!(f, "///{s}"),
@@ -1010,28 +1010,32 @@ impl SecondaryAttribute {
     pub(crate) fn is_abi(&self) -> bool {
         matches!(self, SecondaryAttribute::Abi(_))
     }
-}
 
-impl fmt::Display for SecondaryAttribute {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+    pub(crate) fn contents(&self) -> String {
         match self {
-            SecondaryAttribute::Deprecated(None) => write!(f, "#[deprecated]"),
+            SecondaryAttribute::Deprecated(None) => "deprecated".to_string(),
             SecondaryAttribute::Deprecated(Some(ref note)) => {
-                write!(f, r#"#[deprecated({note:?})]"#)
+                format!("deprecated({note:?})")
             }
-            SecondaryAttribute::Tag(ref attribute) => write!(f, "#['{}]", attribute.contents),
-            SecondaryAttribute::Meta(ref attribute) => write!(f, "#[{}]", attribute.contents),
-            SecondaryAttribute::ContractLibraryMethod => write!(f, "#[contract_library_method]"),
-            SecondaryAttribute::Export => write!(f, "#[export]"),
-            SecondaryAttribute::Field(ref k) => write!(f, "#[field({k})]"),
-            SecondaryAttribute::Abi(ref k) => write!(f, "#[abi({k})]"),
-            SecondaryAttribute::Varargs => write!(f, "#[varargs]"),
-            SecondaryAttribute::UseCallersScope => write!(f, "#[use_callers_scope]"),
-            SecondaryAttribute::Allow(ref k) => write!(f, "#[allow(#{k})]"),
+            SecondaryAttribute::Tag(ref attribute) => format!("'{}", attribute.contents),
+            SecondaryAttribute::Meta(ref attribute) => attribute.contents.to_string(),
+            SecondaryAttribute::ContractLibraryMethod => "contract_library_method".to_string(),
+            SecondaryAttribute::Export => "export".to_string(),
+            SecondaryAttribute::Field(ref k) => format!("field({k})"),
+            SecondaryAttribute::Abi(ref k) => format!("abi({k})"),
+            SecondaryAttribute::Varargs => "varargs".to_string(),
+            SecondaryAttribute::UseCallersScope => "use_callers_scope".to_string(),
+            SecondaryAttribute::Allow(ref k) => format!("allow({k})"),
         }
     }
 }
 
+impl fmt::Display for SecondaryAttribute {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        write!(f, "#[{}]", self.contents())
+    }
+}
+
 #[derive(PartialEq, Eq, Hash, Debug, Clone, PartialOrd, Ord)]
 pub struct CustomAttribute {
     pub contents: String,

compiler/noirc_frontend/src/parser/parser/generics.rs

@@ -130,9 +130,11 @@ impl<'a> Parser<'a> {
             match generic {
                 GenericTypeArg::Ordered(typ) => {
                     generic_type_args.ordered_args.push(typ);
+                    generic_type_args.kinds.push(crate::ast::GenericTypeArgKind::Ordered);
                 }
                 GenericTypeArg::Named(name, typ) => {
                     generic_type_args.named_args.push((name, typ));
+                    generic_type_args.kinds.push(crate::ast::GenericTypeArgKind::Named);
                 }
             }
         }

cspell.json

@@ -164,6 +164,7 @@
         "plonkc",
         "PLONKish",
         "pprof",
+        "precomputes",
         "preimage",
         "preprocess",
         "prettytable",
@@ -178,6 +179,7 @@
         "reqwest",
         "rfind",
         "rustc",
+        "rustfmt",
         "rustup",
         "sboxed",
         "schnorr",
@@ -229,6 +231,8 @@
         "wasi",
         "wasmer",
         "Weierstraß",
+        "whitespace",
+        "whitespaces",
         "zkhash",
         "zshell"
     ],

noir_stdlib/src/array/check_shuffle.nr

@@ -1,7 +1,10 @@
 use crate::cmp::Eq;
 
-unconstrained fn __get_shuffle_indices<T, let N: u32>(lhs: [T; N], rhs: [T; N]) -> [Field; N] where T: Eq {
-    let mut shuffle_indices: [Field;N ] = [0; N];
+unconstrained fn __get_shuffle_indices<T, let N: u32>(lhs: [T; N], rhs: [T; N]) -> [Field; N]
+where
+    T: Eq,
+{
+    let mut shuffle_indices: [Field; N] = [0; N];
 
     let mut shuffle_mask: [bool; N] = [false; N];
     for i in 0..N {
@@ -35,7 +38,10 @@ unconstrained fn __get_index<let N: u32>(indices: [Field; N], idx: Field) -> Fie
     result
 }
 
-pub(crate) fn check_shuffle<T, let N: u32>(lhs: [T; N], rhs: [T; N]) where T: Eq {
+pub(crate) fn check_shuffle<T, let N: u32>(lhs: [T; N], rhs: [T; N])
+where
+    T: Eq,
+{
     unsafe {
         let shuffle_indices = __get_shuffle_indices(lhs, rhs);
 
@@ -59,7 +65,7 @@ mod test {
     struct CompoundStruct {
         a: bool,
         b: Field,
-        c: u64
+        c: u64,
     }
     impl Eq for CompoundStruct {
         fn eq(self, other: Self) -> bool {
@@ -102,14 +108,14 @@ mod test {
             CompoundStruct { a: false, b: -100, c: 54321 },
             CompoundStruct { a: true, b: 5, c: 0xffffffffffffffff },
             CompoundStruct { a: true, b: 9814, c: 0xeeffee0011001133 },
-            CompoundStruct { a: false, b: 0x155, c: 0 }
+            CompoundStruct { a: false, b: 0x155, c: 0 },
         ];
         let rhs: [CompoundStruct; 5] = [
             CompoundStruct { a: false, b: 0x155, c: 0 },
             CompoundStruct { a: false, b: 0, c: 12345 },
             CompoundStruct { a: false, b: -100, c: 54321 },
             CompoundStruct { a: true, b: 9814, c: 0xeeffee0011001133 },
-            CompoundStruct { a: true, b: 5, c: 0xffffffffffffffff }
+            CompoundStruct { a: true, b: 5, c: 0xffffffffffffffff },
         ];
         check_shuffle(lhs, rhs);
     }