Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bevy_reflect: Improve serialization format #4561

Closed
wants to merge 24 commits into from

Conversation

MrGVSV
Copy link
Member

@MrGVSV MrGVSV commented Apr 22, 2022

Note: See #5723 for an even more improved (but slightly more controversial) format

Objective

The current serialization format used by bevy_reflect is both verbose and error-prone. Taking the following structs1 for example:

// -- src/inventory.rs

#[derive(Reflect)]
struct Inventory {
  id: String,
  max_storage: usize,
  items: Vec<Item>
}

#[derive(Reflect)]
struct Item {
  name: String
}

Given an inventory of a single item, this would serialize to something like:

// -- assets/inventory.ron

{
  "type": "my_game::inventory::Inventory",
  "struct": {
    "id": {
      "type": "alloc::string::String",
      "value": "inv001",
    },
    "max_storage": {
      "type": "usize",
      "value": 10
    },
    "items": {
      "type": "alloc::vec::Vec<alloc::string::String>",
      "list": [
        {
          "type": "my_game::inventory::Item",
          "struct": {
            "name": {
              "type": "alloc::string::String",
              "value": "Pickaxe"
            },
          },
        },
      ],
    },
  },
}

Aside from being really long and difficult to read, it also has a few "gotchas" that users need to be aware of if they want to edit the file manually. A major one is the requirement that you use the proper keys for a given type. For structs, you need "struct". For lists, "list". For tuple structs, "tuple_struct". And so on.

It also requires that the "type" entry come before the actual data. Despite being a map— which in programming is almost always orderless by default— the entries need to be in a particular order. Failure to follow the ordering convention results in a failure to deserialize the data.

This makes it very prone to errors and annoyances.

Solution

Using #4042, we can remove a lot of the boilerplate and metadata needed by this older system. Since we now have static access to type information, we can simplify our serialized data to look like:

// -- assets/inventory.ron

{
  "my_game::inventory::Inventory": {
    "id": "inv001",
    "max_storage": 10,
    "items": [
      {
        "name": "Pickaxe"
      },
    ],
  },
},

This is much more digestible and a lot less error-prone (no more key requirements and no more extra type names).

Custom Serialization

Additionally, this PR adds the opt-in ability to specify a custom serde implementation to be used rather than the one created via reflection. For example1:

// -- src/inventory.rs

#[derive(Reflect, Serialize)]
#[reflect(Serialize)]
struct Item {
  #[serde(alias = "id")]
  name: String
}
// -- assets/inventory.ron

{
  "my_game::inventory::Inventory": {
    "id": "inv001",
    "max_storage": 10,
    "items": [
      (
        id: "Pickaxe"
      )
    ],
  },
},

Because we specified a custom Serialize implementation with #[reflect(Serialize)], we actually get the RON-style syntax for a struct.

Note: If we had done this for Inventory, it would still define the type and continue to serialize the data normally.

See Example
// -- assets/inventory.ron

{
"my_game::inventory::Inventory": (
 id: "inv001",
 max_storage: 10,
 items: [
   (
     name: "Pickaxe"
   ),
 ],
),
}

By allowing users to define their own serialization methods, we do two things:

  1. We give more control over how data is serialized/deserialized to the end user
  2. We avoid having to re-define serde's attributes and forcing users to apply both (e.g. we don't need a #[reflect(alias)] attribute).

Tuples!

Tuples and tuple-likes now use serde's de/serialize_tuple method. This provides better semantics— being fixed-size sequences, we should be able to indicate that size at when de/serializing. It also addresses some of serde's own limitations (see sections below).

Tuples

Obviously, tuples now use this format. This is better because tuples do have a maximum size according to serde. I don't think we actually run into this limitation, but it's a good reminder to avoid large tuples in case we want to be compatible with other formats.

{
  "(f32, f32)": (1.0, 1.0)
}
Old Format
{
  "type": "(f32, f32)",
  "tuple": [
    {
      "type": "f32",
      "value": 1.0
    },
    {
      "type": "f32",
      "value": 1.0
    }
  ]
}
Tuple Structs

This also applies to tuple structs. Though, it should be noted that there is a dedicated de/serialize_tuple_struct method in serde. However, it requires a static name, which is not yet provided by TypeInfo. Even if it was, we have no way of handling Dynamic*** types (see bevyengine/rfcs#56).

{
  "my_crate::Bar": ("Hello World!")
}
Old Format
{
  "type": "my_crate::Bar",
  "tuple_struct": [
    {
      "type": "alloc::string::String",
      "value": "Hello World!"
    }
  ]
}
Arrays

It may be a bit surprising to some, but arrays now also use the tuple format. This is because they essentially are tuples (a sequence of values with a fixed size), but only allow for homogenous types. Additionally, this is how RON handles them and is probably a result of the 32-capacity limit imposed on them (both by serde and by bevy_reflect).

{
  "[i32; 3]": (1, 2, 3)
}
Old Format
{
  "type": "[i32; 3]",
  "array": [
    {
      "type": "i32",
      "value": 1
    },
    {
      "type": "i32",
      "value": 2
    },
    {
      "type": "i32",
      "value": 3
    }
  ]
}

Changelog

  • Re-worked serialization/deserialization for reflected types
  • Added TypedReflectDeserializer for deserializing data with known TypeInfo
  • Renamed ReflectDeserializer to UntypedReflectDeserializer
  • Replaced usages of deserialize_any with deserialize_map for non-self-describing formats Reverted this change since there are still some issues that need to be sorted out (in a separate PR). By reverting this, crates like bincode can throw an error when attempting to deserialize non-self-describing formats (bincode results in DeserializeAnyNotSupported)
  • Tuples, tuple structs, and arrays are now all de/serialized as tuples (similar to RON)

Migration Guide

  • This PR reduces the verbosity of the scene format. Scenes will need to be updated accordingly:
// Old format
{
  "type": "my_game::item::Item",
  "struct": {
    "id": {
      "type": "alloc::string::String",
      "value": "bevycraft:stone",
    },
    "tags": {
      "type": "alloc::vec::Vec<alloc::string::String>",
      "list": [
        {
          "type": "alloc::string::String",
          "value": "material"
        },
      ],
    },
}

// New format
{
  "my_game::item::Item": {
    "id": "bevycraft:stone",
    "tags": ["material"]
  }
}
  • There are also some smaller changes to how certain types are serialized/deserialized. Tuples, tuple structs, and arrays are all de/serialized as standard tuples now (if using a de/serializer like RON):
// Old format
{
  "type": "my_crate::Foo",
  "tuple": {
    "type": "(f32, f32)",
    "tuple": [
        {
          "type": "f32",
          "value": 1.0
        },
        {
          "type": "f32",
          "value": 1.0
        }
      ]
    },
    "tuple_struct": {
      "type": "my_crate::Bar",
      "tuple_struct": [
        {
          "type": "alloc::string::String",
          "value": "Hello World!"
        }
      ]
    },
    "array": {
      "type": "[i32; 3]",
      "array": [
        {
          "type": "i32",
          "value": 1
        },
        {
          "type": "i32",
          "value": 2
        },
        {
          "type": "i32",
          "value": 3
        }
      ]
    }
  }
}

// New format
{
  "my_crate::Foo": {
    "tuple": (1.0, 1.0),
    "tuple_struct": ("Hello World!"),
    "array": (1, 2, 3),
  }
}

Entering fewer/more than the exact amount of values results in an error when deserializing.

  • ReflectDeserializer has been renamed to UntypedReflectDeserializer

Footnotes

  1. Some derives omitted for brevity. 2

@github-actions github-actions bot added the S-Needs-Triage This issue needs to be labelled label Apr 22, 2022
@alice-i-cecile alice-i-cecile added A-Reflection Runtime information about types C-Usability A targeted quality-of-life change that makes Bevy easier to use and removed S-Needs-Triage This issue needs to be labelled labels Apr 22, 2022
@alice-i-cecile
Copy link
Member

Previously discussed in #92 and #4153. To make life easier for reviewers, how does this compare to this proposals?

@MrGVSV
Copy link
Member Author

MrGVSV commented Apr 22, 2022

Previously discussed in #92 and #4153. To make life easier for reviewers, how does this compare to this proposals?

I actually hadn't seen #92, so thanks for pointing that out!

In regards to #4153, though, I think this is a good intermediate step towards a syntax like that. The main "issue" with the syntax proposed in that PR is that it's pretty reliant on the RON format and likely wouldn't translate to JSON or other formats well— especially if they're generated by a non-Rust tool (see @james7132's comment).

So I think this is (hopefully) a less controversial improvement to the format while we decide and finalize on something like #4153.

@alice-i-cecile alice-i-cecile added the X-Controversial There is active debate or serious implications around merging this PR label Apr 25, 2022
@MrGVSV MrGVSV force-pushed the reflect-improved-serialization branch from 81c6d50 to 8537bcd Compare June 1, 2022 07:04
@MrGVSV MrGVSV force-pushed the reflect-improved-serialization branch from 7bbb300 to 5e6b188 Compare June 11, 2022 23:55
@MrGVSV MrGVSV marked this pull request as ready for review June 12, 2022 00:34
@PROMETHIA-27
Copy link
Contributor

Anecdotally, this will be really useful in a surprising way; I'm using reflection to power messages for my editor, so cutting down on the verbosity is a performance win as well as a reliability win for how much types can change or move before messages become invalid between versions.

@alice-i-cecile
Copy link
Member

as well as a reliability win for how much types can change or move before messages become invalid between versions.

This would be appreciated. I've had cases where I trivially reorganized code and completely broke all of my serialized structs in frustrating ways that are hard for beginners to debug.

Copy link
Contributor

@PROMETHIA-27 PROMETHIA-27 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@MrGVSV MrGVSV force-pushed the reflect-improved-serialization branch from 80c8db4 to 3debda4 Compare June 27, 2022 02:23
Copy link
Contributor

@nicopap nicopap left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work, awesome change! Just a tinny nit and a question. I'm a bit wary from the repetition, but I know from experience it's basically impossible to work around it.

crates/bevy_reflect/src/serde/de.rs Outdated Show resolved Hide resolved
crates/bevy_reflect/src/serde/de.rs Outdated Show resolved Hide resolved
crates/bevy_reflect/src/serde/de.rs Outdated Show resolved Hide resolved
@MrGVSV MrGVSV force-pushed the reflect-improved-serialization branch from 053d265 to 9d656b1 Compare August 1, 2022 02:29
@Metadorius
Copy link

Metadorius commented Aug 2, 2022

My suggestion on making the serialization syntax even more human-processable: omit type and value tags and express them as key-value pairs.

Example

Instead of

{
  "type": "my_game::inventory::Inventory",
  "value": {
    "id": "inv001",
    "max_storage": 10,
    "items": [
      (
        id: "Pickaxe"
      )
    ],
  },
},

we would write

"my_game::inventory::Inventory": {
  "id": "inv001",
  "max_storage": 10,
  "items": [
    (
      id: "Pickaxe"
    )
  ],
},

Why?

TL;DR it is just a less error-prone and more intuitive representation without artificial to JSON / RON / YAML rules, instead these rules are expressed naturally through JSON / RON / YAML syntax.

  1. With the current implementation you need to introduce artificial "you have to write type before value" rule. Ordering of key-values is unnatural for such data representations. With my suggestion you simply can't write value before key as there's no such syntax.
  2. Same deal with "you can't add multiple components of the same type" rule. Because ECS (Bevy's in particular) don't allow for multiple components, the data structure that represents an ECS entity's components is a component set (or a type-value dictionary/map which is almost the same) and absolutely not an array. Current format uses arrays, my suggestion uses dictionary/set representation as you simply can't define the same key (type) twice on one object. I don't think using arrays to represent sets is a good idea.
  3. It is in-line with Serde's suggested default serialization style (typetag also mentions this as Serde's default among other options).
  4. As an added bonus it is also more compact and has less indentation levels while not sacrificing readability.

Alternatives considered

#4153 is a good pitch too and aims to solve the same issue, except the solution @AronDerenyi proposed only solves one of the two inconsistencies. For RON it looks nicer, but it still uses arrays to represent entity's components, which allows for multiple components on a syntax level, which is semantically invalid and has to be limited, whereas what I propose doesn't allow for that. Also I don't think you really can express the same syntax with JSON, YAML etc. meaning that it is less portable.

@inodentry
Copy link
Contributor

I really like this last suggestion here, by @Metadorius , for the format. :) Agreed with the reasoning. Seems like this is perhaps the best proposal we have so far.

@MrGVSV
Copy link
Member Author

MrGVSV commented Aug 2, 2022

I updated the code and the PR description to follow @Metadorius' suggestion! It now uses {type_name: data} in place of {type: type_name, value: data}.

Copy link
Contributor

@sarkahn sarkahn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Example scene needs to be updated to the new format, other than that looks great. Really looking forward to this getting merged!

@MrGVSV MrGVSV force-pushed the reflect-improved-serialization branch 3 times, most recently from c70308e to aa1f1aa Compare August 10, 2022 16:07
@MrGVSV MrGVSV force-pushed the reflect-improved-serialization branch from aa1f1aa to c7475e9 Compare August 17, 2022 06:14
@MrGVSV MrGVSV force-pushed the reflect-improved-serialization branch from 04c28e2 to 90433d1 Compare September 1, 2022 05:14
@alice-i-cecile
Copy link
Member

Closed in favor of #5723 per #5723 (comment) 🎉

bors bot pushed a commit that referenced this pull request Sep 20, 2022
> Note: This is rebased off #4561 and can be viewed as a competitor to that PR. See `Comparison with #4561` section for details.

# Objective

The current serialization format used by `bevy_reflect` is both verbose and error-prone. Taking the following structs[^1] for example:

```rust
// -- src/inventory.rs

#[derive(Reflect)]
struct Inventory {
  id: String,
  max_storage: usize,
  items: Vec<Item>
}

#[derive(Reflect)]
struct Item {
  name: String
}
```

Given an inventory of a single item, this would serialize to something like:

```rust
// -- assets/inventory.ron

{
  "type": "my_game::inventory::Inventory",
  "struct": {
    "id": {
      "type": "alloc::string::String",
      "value": "inv001",
    },
    "max_storage": {
      "type": "usize",
      "value": 10
    },
    "items": {
      "type": "alloc::vec::Vec<alloc::string::String>",
      "list": [
        {
          "type": "my_game::inventory::Item",
          "struct": {
            "name": {
              "type": "alloc::string::String",
              "value": "Pickaxe"
            },
          },
        },
      ],
    },
  },
}
```

Aside from being really long and difficult to read, it also has a few "gotchas" that users need to be aware of if they want to edit the file manually. A major one is the requirement that you use the proper keys for a given type. For structs, you need `"struct"`. For lists, `"list"`. For tuple structs, `"tuple_struct"`. And so on.

It also ***requires*** that the `"type"` entry come before the actual data. Despite being a map— which in programming is almost always orderless by default— the entries need to be in a particular order. Failure to follow the ordering convention results in a failure to deserialize the data.

This makes it very prone to errors and annoyances.


## Solution

Using #4042, we can remove a lot of the boilerplate and metadata needed by this older system. Since we now have static access to type information, we can simplify our serialized data to look like:

```rust
// -- assets/inventory.ron

{
  "my_game::inventory::Inventory": (
    id: "inv001",
    max_storage: 10,
    items: [
      (
        name: "Pickaxe"
      ),
    ],
  ),
}
```

This is much more digestible and a lot less error-prone (no more key requirements and no more extra type names).

Additionally, it is a lot more familiar to users as it follows conventional serde mechanics. For example, the struct is represented with `(...)` when serialized to RON.

#### Custom Serialization

Additionally, this PR adds the opt-in ability to specify a custom serde implementation to be used rather than the one created via reflection. For example[^1]:

```rust
// -- src/inventory.rs

#[derive(Reflect, Serialize)]
#[reflect(Serialize)]
struct Item {
  #[serde(alias = "id")]
  name: String
}
```

```rust
// -- assets/inventory.ron

{
  "my_game::inventory::Inventory": (
    id: "inv001",
    max_storage: 10,
    items: [
      (
        id: "Pickaxe"
      ),
    ],
  ),
},
```

By allowing users to define their own serialization methods, we do two things:

1. We give more control over how data is serialized/deserialized to the end user
2. We avoid having to re-define serde's attributes and forcing users to apply both (e.g. we don't need a `#[reflect(alias)]` attribute).

### Improved Formats

One of the improvements this PR provides is the ability to represent data in ways that are more conventional and/or familiar to users. Many users are familiar with RON so here are some of the ways we can now represent data in RON:

###### Structs

```js
{
  "my_crate::Foo": (
    bar: 123
  )
}
// OR
{
  "my_crate::Foo": Foo(
    bar: 123
  )
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "my_crate::Foo",
  "struct": {
    "bar": {
      "type": "usize",
      "value": 123
    }
  }
}
```

</details>

###### Tuples

```js
{
  "(f32, f32)": (1.0, 2.0)
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "(f32, f32)",
  "tuple": [
    {
      "type": "f32",
      "value": 1.0
    },
    {
      "type": "f32",
      "value": 2.0
    }
  ]
}
```

</details>

###### Tuple Structs

```js
{
  "my_crate::Bar": ("Hello World!")
}
// OR
{
  "my_crate::Bar": Bar("Hello World!")
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "my_crate::Bar",
  "tuple_struct": [
    {
      "type": "alloc::string::String",
      "value": "Hello World!"
    }
  ]
}
```

</details>

###### Arrays

It may be a bit surprising to some, but arrays now also use the tuple format. This is because they essentially _are_ tuples (a sequence of values with a fixed size), but only allow for homogenous types. Additionally, this is how RON handles them and is probably a result of the 32-capacity limit imposed on them (both by [serde](https://docs.rs/serde/latest/serde/trait.Serialize.html#impl-Serialize-for-%5BT%3B%2032%5D) and by [bevy_reflect](https://docs.rs/bevy/latest/bevy/reflect/trait.GetTypeRegistration.html#impl-GetTypeRegistration-for-%5BT%3B%2032%5D)).

```js
{
  "[i32; 3]": (1, 2, 3)
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "[i32; 3]",
  "array": [
    {
      "type": "i32",
      "value": 1
    },
    {
      "type": "i32",
      "value": 2
    },
    {
      "type": "i32",
      "value": 3
    }
  ]
}
```

</details>

###### Enums

To make things simple, I'll just put a struct variant here, but the style applies to all variant types:

```js
{
  "my_crate::ItemType": Consumable(
    name: "Healing potion"
  )
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "my_crate::ItemType",
  "enum": {
    "variant": "Consumable",
    "struct": {
      "name": {
        "type": "alloc::string::String",
        "value": "Healing potion"
      }
    }
  }
}
```

</details>

### Comparison with #4561

This PR is a rebased version of #4561. The reason for the split between the two is because this PR creates a _very_ different scene format. You may notice that the PR descriptions for either PR are pretty similar. This was done to better convey the changes depending on which (if any) gets merged first. If #4561 makes it in first, I will update this PR description accordingly.

---

## Changelog

* Re-worked serialization/deserialization for reflected types
* Added `TypedReflectDeserializer` for deserializing data with known `TypeInfo`
* Renamed `ReflectDeserializer` to `UntypedReflectDeserializer` 
* ~~Replaced usages of `deserialize_any` with `deserialize_map` for non-self-describing formats~~ Reverted this change since there are still some issues that need to be sorted out (in a separate PR). By reverting this, crates like `bincode` can throw an error when attempting to deserialize non-self-describing formats (`bincode` results in `DeserializeAnyNotSupported`)
* Structs, tuples, tuple structs, arrays, and enums are now all de/serialized using conventional serde methods

## Migration Guide

* This PR reduces the verbosity of the scene format. Scenes will need to be updated accordingly:

```js
// Old format
{
  "type": "my_game::item::Item",
  "struct": {
    "id": {
      "type": "alloc::string::String",
      "value": "bevycraft:stone",
    },
    "tags": {
      "type": "alloc::vec::Vec<alloc::string::String>",
      "list": [
        {
          "type": "alloc::string::String",
          "value": "material"
        },
      ],
    },
}

// New format
{
  "my_game::item::Item": (
    id: "bevycraft:stone",
    tags: ["material"]
  )
}
```

[^1]: Some derives omitted for brevity.
bors bot pushed a commit that referenced this pull request Sep 20, 2022
> Note: This is rebased off #4561 and can be viewed as a competitor to that PR. See `Comparison with #4561` section for details.

# Objective

The current serialization format used by `bevy_reflect` is both verbose and error-prone. Taking the following structs[^1] for example:

```rust
// -- src/inventory.rs

#[derive(Reflect)]
struct Inventory {
  id: String,
  max_storage: usize,
  items: Vec<Item>
}

#[derive(Reflect)]
struct Item {
  name: String
}
```

Given an inventory of a single item, this would serialize to something like:

```rust
// -- assets/inventory.ron

{
  "type": "my_game::inventory::Inventory",
  "struct": {
    "id": {
      "type": "alloc::string::String",
      "value": "inv001",
    },
    "max_storage": {
      "type": "usize",
      "value": 10
    },
    "items": {
      "type": "alloc::vec::Vec<alloc::string::String>",
      "list": [
        {
          "type": "my_game::inventory::Item",
          "struct": {
            "name": {
              "type": "alloc::string::String",
              "value": "Pickaxe"
            },
          },
        },
      ],
    },
  },
}
```

Aside from being really long and difficult to read, it also has a few "gotchas" that users need to be aware of if they want to edit the file manually. A major one is the requirement that you use the proper keys for a given type. For structs, you need `"struct"`. For lists, `"list"`. For tuple structs, `"tuple_struct"`. And so on.

It also ***requires*** that the `"type"` entry come before the actual data. Despite being a map— which in programming is almost always orderless by default— the entries need to be in a particular order. Failure to follow the ordering convention results in a failure to deserialize the data.

This makes it very prone to errors and annoyances.


## Solution

Using #4042, we can remove a lot of the boilerplate and metadata needed by this older system. Since we now have static access to type information, we can simplify our serialized data to look like:

```rust
// -- assets/inventory.ron

{
  "my_game::inventory::Inventory": (
    id: "inv001",
    max_storage: 10,
    items: [
      (
        name: "Pickaxe"
      ),
    ],
  ),
}
```

This is much more digestible and a lot less error-prone (no more key requirements and no more extra type names).

Additionally, it is a lot more familiar to users as it follows conventional serde mechanics. For example, the struct is represented with `(...)` when serialized to RON.

#### Custom Serialization

Additionally, this PR adds the opt-in ability to specify a custom serde implementation to be used rather than the one created via reflection. For example[^1]:

```rust
// -- src/inventory.rs

#[derive(Reflect, Serialize)]
#[reflect(Serialize)]
struct Item {
  #[serde(alias = "id")]
  name: String
}
```

```rust
// -- assets/inventory.ron

{
  "my_game::inventory::Inventory": (
    id: "inv001",
    max_storage: 10,
    items: [
      (
        id: "Pickaxe"
      ),
    ],
  ),
},
```

By allowing users to define their own serialization methods, we do two things:

1. We give more control over how data is serialized/deserialized to the end user
2. We avoid having to re-define serde's attributes and forcing users to apply both (e.g. we don't need a `#[reflect(alias)]` attribute).

### Improved Formats

One of the improvements this PR provides is the ability to represent data in ways that are more conventional and/or familiar to users. Many users are familiar with RON so here are some of the ways we can now represent data in RON:

###### Structs

```js
{
  "my_crate::Foo": (
    bar: 123
  )
}
// OR
{
  "my_crate::Foo": Foo(
    bar: 123
  )
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "my_crate::Foo",
  "struct": {
    "bar": {
      "type": "usize",
      "value": 123
    }
  }
}
```

</details>

###### Tuples

```js
{
  "(f32, f32)": (1.0, 2.0)
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "(f32, f32)",
  "tuple": [
    {
      "type": "f32",
      "value": 1.0
    },
    {
      "type": "f32",
      "value": 2.0
    }
  ]
}
```

</details>

###### Tuple Structs

```js
{
  "my_crate::Bar": ("Hello World!")
}
// OR
{
  "my_crate::Bar": Bar("Hello World!")
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "my_crate::Bar",
  "tuple_struct": [
    {
      "type": "alloc::string::String",
      "value": "Hello World!"
    }
  ]
}
```

</details>

###### Arrays

It may be a bit surprising to some, but arrays now also use the tuple format. This is because they essentially _are_ tuples (a sequence of values with a fixed size), but only allow for homogenous types. Additionally, this is how RON handles them and is probably a result of the 32-capacity limit imposed on them (both by [serde](https://docs.rs/serde/latest/serde/trait.Serialize.html#impl-Serialize-for-%5BT%3B%2032%5D) and by [bevy_reflect](https://docs.rs/bevy/latest/bevy/reflect/trait.GetTypeRegistration.html#impl-GetTypeRegistration-for-%5BT%3B%2032%5D)).

```js
{
  "[i32; 3]": (1, 2, 3)
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "[i32; 3]",
  "array": [
    {
      "type": "i32",
      "value": 1
    },
    {
      "type": "i32",
      "value": 2
    },
    {
      "type": "i32",
      "value": 3
    }
  ]
}
```

</details>

###### Enums

To make things simple, I'll just put a struct variant here, but the style applies to all variant types:

```js
{
  "my_crate::ItemType": Consumable(
    name: "Healing potion"
  )
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "my_crate::ItemType",
  "enum": {
    "variant": "Consumable",
    "struct": {
      "name": {
        "type": "alloc::string::String",
        "value": "Healing potion"
      }
    }
  }
}
```

</details>

### Comparison with #4561

This PR is a rebased version of #4561. The reason for the split between the two is because this PR creates a _very_ different scene format. You may notice that the PR descriptions for either PR are pretty similar. This was done to better convey the changes depending on which (if any) gets merged first. If #4561 makes it in first, I will update this PR description accordingly.

---

## Changelog

* Re-worked serialization/deserialization for reflected types
* Added `TypedReflectDeserializer` for deserializing data with known `TypeInfo`
* Renamed `ReflectDeserializer` to `UntypedReflectDeserializer` 
* ~~Replaced usages of `deserialize_any` with `deserialize_map` for non-self-describing formats~~ Reverted this change since there are still some issues that need to be sorted out (in a separate PR). By reverting this, crates like `bincode` can throw an error when attempting to deserialize non-self-describing formats (`bincode` results in `DeserializeAnyNotSupported`)
* Structs, tuples, tuple structs, arrays, and enums are now all de/serialized using conventional serde methods

## Migration Guide

* This PR reduces the verbosity of the scene format. Scenes will need to be updated accordingly:

```js
// Old format
{
  "type": "my_game::item::Item",
  "struct": {
    "id": {
      "type": "alloc::string::String",
      "value": "bevycraft:stone",
    },
    "tags": {
      "type": "alloc::vec::Vec<alloc::string::String>",
      "list": [
        {
          "type": "alloc::string::String",
          "value": "material"
        },
      ],
    },
}

// New format
{
  "my_game::item::Item": (
    id: "bevycraft:stone",
    tags: ["material"]
  )
}
```

[^1]: Some derives omitted for brevity.
james7132 pushed a commit to james7132/bevy that referenced this pull request Oct 19, 2022
> Note: This is rebased off bevyengine#4561 and can be viewed as a competitor to that PR. See `Comparison with bevyengine#4561` section for details.

# Objective

The current serialization format used by `bevy_reflect` is both verbose and error-prone. Taking the following structs[^1] for example:

```rust
// -- src/inventory.rs

#[derive(Reflect)]
struct Inventory {
  id: String,
  max_storage: usize,
  items: Vec<Item>
}

#[derive(Reflect)]
struct Item {
  name: String
}
```

Given an inventory of a single item, this would serialize to something like:

```rust
// -- assets/inventory.ron

{
  "type": "my_game::inventory::Inventory",
  "struct": {
    "id": {
      "type": "alloc::string::String",
      "value": "inv001",
    },
    "max_storage": {
      "type": "usize",
      "value": 10
    },
    "items": {
      "type": "alloc::vec::Vec<alloc::string::String>",
      "list": [
        {
          "type": "my_game::inventory::Item",
          "struct": {
            "name": {
              "type": "alloc::string::String",
              "value": "Pickaxe"
            },
          },
        },
      ],
    },
  },
}
```

Aside from being really long and difficult to read, it also has a few "gotchas" that users need to be aware of if they want to edit the file manually. A major one is the requirement that you use the proper keys for a given type. For structs, you need `"struct"`. For lists, `"list"`. For tuple structs, `"tuple_struct"`. And so on.

It also ***requires*** that the `"type"` entry come before the actual data. Despite being a map— which in programming is almost always orderless by default— the entries need to be in a particular order. Failure to follow the ordering convention results in a failure to deserialize the data.

This makes it very prone to errors and annoyances.


## Solution

Using bevyengine#4042, we can remove a lot of the boilerplate and metadata needed by this older system. Since we now have static access to type information, we can simplify our serialized data to look like:

```rust
// -- assets/inventory.ron

{
  "my_game::inventory::Inventory": (
    id: "inv001",
    max_storage: 10,
    items: [
      (
        name: "Pickaxe"
      ),
    ],
  ),
}
```

This is much more digestible and a lot less error-prone (no more key requirements and no more extra type names).

Additionally, it is a lot more familiar to users as it follows conventional serde mechanics. For example, the struct is represented with `(...)` when serialized to RON.

#### Custom Serialization

Additionally, this PR adds the opt-in ability to specify a custom serde implementation to be used rather than the one created via reflection. For example[^1]:

```rust
// -- src/inventory.rs

#[derive(Reflect, Serialize)]
#[reflect(Serialize)]
struct Item {
  #[serde(alias = "id")]
  name: String
}
```

```rust
// -- assets/inventory.ron

{
  "my_game::inventory::Inventory": (
    id: "inv001",
    max_storage: 10,
    items: [
      (
        id: "Pickaxe"
      ),
    ],
  ),
},
```

By allowing users to define their own serialization methods, we do two things:

1. We give more control over how data is serialized/deserialized to the end user
2. We avoid having to re-define serde's attributes and forcing users to apply both (e.g. we don't need a `#[reflect(alias)]` attribute).

### Improved Formats

One of the improvements this PR provides is the ability to represent data in ways that are more conventional and/or familiar to users. Many users are familiar with RON so here are some of the ways we can now represent data in RON:

###### Structs

```js
{
  "my_crate::Foo": (
    bar: 123
  )
}
// OR
{
  "my_crate::Foo": Foo(
    bar: 123
  )
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "my_crate::Foo",
  "struct": {
    "bar": {
      "type": "usize",
      "value": 123
    }
  }
}
```

</details>

###### Tuples

```js
{
  "(f32, f32)": (1.0, 2.0)
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "(f32, f32)",
  "tuple": [
    {
      "type": "f32",
      "value": 1.0
    },
    {
      "type": "f32",
      "value": 2.0
    }
  ]
}
```

</details>

###### Tuple Structs

```js
{
  "my_crate::Bar": ("Hello World!")
}
// OR
{
  "my_crate::Bar": Bar("Hello World!")
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "my_crate::Bar",
  "tuple_struct": [
    {
      "type": "alloc::string::String",
      "value": "Hello World!"
    }
  ]
}
```

</details>

###### Arrays

It may be a bit surprising to some, but arrays now also use the tuple format. This is because they essentially _are_ tuples (a sequence of values with a fixed size), but only allow for homogenous types. Additionally, this is how RON handles them and is probably a result of the 32-capacity limit imposed on them (both by [serde](https://docs.rs/serde/latest/serde/trait.Serialize.html#impl-Serialize-for-%5BT%3B%2032%5D) and by [bevy_reflect](https://docs.rs/bevy/latest/bevy/reflect/trait.GetTypeRegistration.html#impl-GetTypeRegistration-for-%5BT%3B%2032%5D)).

```js
{
  "[i32; 3]": (1, 2, 3)
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "[i32; 3]",
  "array": [
    {
      "type": "i32",
      "value": 1
    },
    {
      "type": "i32",
      "value": 2
    },
    {
      "type": "i32",
      "value": 3
    }
  ]
}
```

</details>

###### Enums

To make things simple, I'll just put a struct variant here, but the style applies to all variant types:

```js
{
  "my_crate::ItemType": Consumable(
    name: "Healing potion"
  )
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "my_crate::ItemType",
  "enum": {
    "variant": "Consumable",
    "struct": {
      "name": {
        "type": "alloc::string::String",
        "value": "Healing potion"
      }
    }
  }
}
```

</details>

### Comparison with bevyengine#4561

This PR is a rebased version of bevyengine#4561. The reason for the split between the two is because this PR creates a _very_ different scene format. You may notice that the PR descriptions for either PR are pretty similar. This was done to better convey the changes depending on which (if any) gets merged first. If bevyengine#4561 makes it in first, I will update this PR description accordingly.

---

## Changelog

* Re-worked serialization/deserialization for reflected types
* Added `TypedReflectDeserializer` for deserializing data with known `TypeInfo`
* Renamed `ReflectDeserializer` to `UntypedReflectDeserializer` 
* ~~Replaced usages of `deserialize_any` with `deserialize_map` for non-self-describing formats~~ Reverted this change since there are still some issues that need to be sorted out (in a separate PR). By reverting this, crates like `bincode` can throw an error when attempting to deserialize non-self-describing formats (`bincode` results in `DeserializeAnyNotSupported`)
* Structs, tuples, tuple structs, arrays, and enums are now all de/serialized using conventional serde methods

## Migration Guide

* This PR reduces the verbosity of the scene format. Scenes will need to be updated accordingly:

```js
// Old format
{
  "type": "my_game::item::Item",
  "struct": {
    "id": {
      "type": "alloc::string::String",
      "value": "bevycraft:stone",
    },
    "tags": {
      "type": "alloc::vec::Vec<alloc::string::String>",
      "list": [
        {
          "type": "alloc::string::String",
          "value": "material"
        },
      ],
    },
}

// New format
{
  "my_game::item::Item": (
    id: "bevycraft:stone",
    tags: ["material"]
  )
}
```

[^1]: Some derives omitted for brevity.
bors bot pushed a commit that referenced this pull request Oct 27, 2022
# Objective

Currently scenes define components using a list:

```rust
[
  (
    entity: 0,
    components: [
      {
        "bevy_transform::components::transform::Transform": (
          translation: (
            x: 0.0,
            y: 0.0,
            z: 0.0
          ),
          rotation: (0.0, 0.0, 0.0, 1.0),
          scale: (
            x: 1.0,
            y: 1.0,
            z: 1.0
          ),
        ),
      },
      {
        "my_crate::Foo": (
          text: "Hello World",
        ),
      },
      {
        "my_crate::Bar": (
          baz: 123,
        ),
      },
    ],
  ),
]
```

However, this representation has some drawbacks (as pointed out by @Metadorius in [this](#4561 (comment)) comment):

1. Increased nesting and more characters (minor effect on overall size)
2. More importantly, by definition, entities cannot have more than one instance of any given component. Therefore, such data is best stored as a map— where all values are meant to have unique keys.


## Solution

Change `components` to store a map of components rather than a list:

```rust
[
  (
    entity: 0,
    components: {
      "bevy_transform::components::transform::Transform": (
        translation: (
          x: 0.0,
          y: 0.0,
          z: 0.0
        ),
        rotation: (0.0, 0.0, 0.0, 1.0),
        scale: (
          x: 1.0,
          y: 1.0,
          z: 1.0
        ),
      ),
      "my_crate::Foo": (
        text: "Hello World",
      ),
      "my_crate::Bar": (
        baz: 123
      ),
    },
  ),
]
```

#### Code Representation

This change only affects the scene format itself. `DynamicEntity` still stores its components as a list. The reason for this is that storing such data as a map is not really needed since:
1. The "key" of each value is easily found by just calling `Reflect::type_name` on it
2. We should be generating such structs using the `World` itself which upholds the one-component-per-entity rule

One could in theory create manually create a `DynamicEntity` with duplicate components, but this isn't something I think we should focus on in this PR. `DynamicEntity` can be broken in other ways (i.e. storing a non-component in the components list), and resolving its issues can be done in a separate PR.

---

## Changelog

* The scene format now uses a map to represent the collection of components rather than a list

## Migration Guide

The scene format now uses a map to represent the collection of components. Scene files will need to update from the old list format.

<details>
<summary>Example Code</summary>

```rust
// OLD
[
  (
    entity: 0,
    components: [
      {
        "bevy_transform::components::transform::Transform": (
          translation: (
            x: 0.0,
            y: 0.0,
            z: 0.0
          ),
          rotation: (0.0, 0.0, 0.0, 1.0),
          scale: (
            x: 1.0,
            y: 1.0,
            z: 1.0
          ),
        ),
      },
      {
        "my_crate::Foo": (
          text: "Hello World",
        ),
      },
      {
        "my_crate::Bar": (
          baz: 123,
        ),
      },
    ],
  ),
]

// NEW
[
  (
    entity: 0,
    components: {
      "bevy_transform::components::transform::Transform": (
        translation: (
          x: 0.0,
          y: 0.0,
          z: 0.0
        ),
        rotation: (0.0, 0.0, 0.0, 1.0),
        scale: (
          x: 1.0,
          y: 1.0,
          z: 1.0
        ),
      ),
      "my_crate::Foo": (
        text: "Hello World",
      ),
      "my_crate::Bar": (
        baz: 123
      ),
    },
  ),
]
```

</details>
james7132 pushed a commit to james7132/bevy that referenced this pull request Oct 28, 2022
> Note: This is rebased off bevyengine#4561 and can be viewed as a competitor to that PR. See `Comparison with bevyengine#4561` section for details.

# Objective

The current serialization format used by `bevy_reflect` is both verbose and error-prone. Taking the following structs[^1] for example:

```rust
// -- src/inventory.rs

#[derive(Reflect)]
struct Inventory {
  id: String,
  max_storage: usize,
  items: Vec<Item>
}

#[derive(Reflect)]
struct Item {
  name: String
}
```

Given an inventory of a single item, this would serialize to something like:

```rust
// -- assets/inventory.ron

{
  "type": "my_game::inventory::Inventory",
  "struct": {
    "id": {
      "type": "alloc::string::String",
      "value": "inv001",
    },
    "max_storage": {
      "type": "usize",
      "value": 10
    },
    "items": {
      "type": "alloc::vec::Vec<alloc::string::String>",
      "list": [
        {
          "type": "my_game::inventory::Item",
          "struct": {
            "name": {
              "type": "alloc::string::String",
              "value": "Pickaxe"
            },
          },
        },
      ],
    },
  },
}
```

Aside from being really long and difficult to read, it also has a few "gotchas" that users need to be aware of if they want to edit the file manually. A major one is the requirement that you use the proper keys for a given type. For structs, you need `"struct"`. For lists, `"list"`. For tuple structs, `"tuple_struct"`. And so on.

It also ***requires*** that the `"type"` entry come before the actual data. Despite being a map— which in programming is almost always orderless by default— the entries need to be in a particular order. Failure to follow the ordering convention results in a failure to deserialize the data.

This makes it very prone to errors and annoyances.


## Solution

Using bevyengine#4042, we can remove a lot of the boilerplate and metadata needed by this older system. Since we now have static access to type information, we can simplify our serialized data to look like:

```rust
// -- assets/inventory.ron

{
  "my_game::inventory::Inventory": (
    id: "inv001",
    max_storage: 10,
    items: [
      (
        name: "Pickaxe"
      ),
    ],
  ),
}
```

This is much more digestible and a lot less error-prone (no more key requirements and no more extra type names).

Additionally, it is a lot more familiar to users as it follows conventional serde mechanics. For example, the struct is represented with `(...)` when serialized to RON.

#### Custom Serialization

Additionally, this PR adds the opt-in ability to specify a custom serde implementation to be used rather than the one created via reflection. For example[^1]:

```rust
// -- src/inventory.rs

#[derive(Reflect, Serialize)]
#[reflect(Serialize)]
struct Item {
  #[serde(alias = "id")]
  name: String
}
```

```rust
// -- assets/inventory.ron

{
  "my_game::inventory::Inventory": (
    id: "inv001",
    max_storage: 10,
    items: [
      (
        id: "Pickaxe"
      ),
    ],
  ),
},
```

By allowing users to define their own serialization methods, we do two things:

1. We give more control over how data is serialized/deserialized to the end user
2. We avoid having to re-define serde's attributes and forcing users to apply both (e.g. we don't need a `#[reflect(alias)]` attribute).

### Improved Formats

One of the improvements this PR provides is the ability to represent data in ways that are more conventional and/or familiar to users. Many users are familiar with RON so here are some of the ways we can now represent data in RON:

###### Structs

```js
{
  "my_crate::Foo": (
    bar: 123
  )
}
// OR
{
  "my_crate::Foo": Foo(
    bar: 123
  )
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "my_crate::Foo",
  "struct": {
    "bar": {
      "type": "usize",
      "value": 123
    }
  }
}
```

</details>

###### Tuples

```js
{
  "(f32, f32)": (1.0, 2.0)
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "(f32, f32)",
  "tuple": [
    {
      "type": "f32",
      "value": 1.0
    },
    {
      "type": "f32",
      "value": 2.0
    }
  ]
}
```

</details>

###### Tuple Structs

```js
{
  "my_crate::Bar": ("Hello World!")
}
// OR
{
  "my_crate::Bar": Bar("Hello World!")
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "my_crate::Bar",
  "tuple_struct": [
    {
      "type": "alloc::string::String",
      "value": "Hello World!"
    }
  ]
}
```

</details>

###### Arrays

It may be a bit surprising to some, but arrays now also use the tuple format. This is because they essentially _are_ tuples (a sequence of values with a fixed size), but only allow for homogenous types. Additionally, this is how RON handles them and is probably a result of the 32-capacity limit imposed on them (both by [serde](https://docs.rs/serde/latest/serde/trait.Serialize.html#impl-Serialize-for-%5BT%3B%2032%5D) and by [bevy_reflect](https://docs.rs/bevy/latest/bevy/reflect/trait.GetTypeRegistration.html#impl-GetTypeRegistration-for-%5BT%3B%2032%5D)).

```js
{
  "[i32; 3]": (1, 2, 3)
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "[i32; 3]",
  "array": [
    {
      "type": "i32",
      "value": 1
    },
    {
      "type": "i32",
      "value": 2
    },
    {
      "type": "i32",
      "value": 3
    }
  ]
}
```

</details>

###### Enums

To make things simple, I'll just put a struct variant here, but the style applies to all variant types:

```js
{
  "my_crate::ItemType": Consumable(
    name: "Healing potion"
  )
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "my_crate::ItemType",
  "enum": {
    "variant": "Consumable",
    "struct": {
      "name": {
        "type": "alloc::string::String",
        "value": "Healing potion"
      }
    }
  }
}
```

</details>

### Comparison with bevyengine#4561

This PR is a rebased version of bevyengine#4561. The reason for the split between the two is because this PR creates a _very_ different scene format. You may notice that the PR descriptions for either PR are pretty similar. This was done to better convey the changes depending on which (if any) gets merged first. If bevyengine#4561 makes it in first, I will update this PR description accordingly.

---

## Changelog

* Re-worked serialization/deserialization for reflected types
* Added `TypedReflectDeserializer` for deserializing data with known `TypeInfo`
* Renamed `ReflectDeserializer` to `UntypedReflectDeserializer` 
* ~~Replaced usages of `deserialize_any` with `deserialize_map` for non-self-describing formats~~ Reverted this change since there are still some issues that need to be sorted out (in a separate PR). By reverting this, crates like `bincode` can throw an error when attempting to deserialize non-self-describing formats (`bincode` results in `DeserializeAnyNotSupported`)
* Structs, tuples, tuple structs, arrays, and enums are now all de/serialized using conventional serde methods

## Migration Guide

* This PR reduces the verbosity of the scene format. Scenes will need to be updated accordingly:

```js
// Old format
{
  "type": "my_game::item::Item",
  "struct": {
    "id": {
      "type": "alloc::string::String",
      "value": "bevycraft:stone",
    },
    "tags": {
      "type": "alloc::vec::Vec<alloc::string::String>",
      "list": [
        {
          "type": "alloc::string::String",
          "value": "material"
        },
      ],
    },
}

// New format
{
  "my_game::item::Item": (
    id: "bevycraft:stone",
    tags: ["material"]
  )
}
```

[^1]: Some derives omitted for brevity.
james7132 pushed a commit to james7132/bevy that referenced this pull request Oct 28, 2022
# Objective

Currently scenes define components using a list:

```rust
[
  (
    entity: 0,
    components: [
      {
        "bevy_transform::components::transform::Transform": (
          translation: (
            x: 0.0,
            y: 0.0,
            z: 0.0
          ),
          rotation: (0.0, 0.0, 0.0, 1.0),
          scale: (
            x: 1.0,
            y: 1.0,
            z: 1.0
          ),
        ),
      },
      {
        "my_crate::Foo": (
          text: "Hello World",
        ),
      },
      {
        "my_crate::Bar": (
          baz: 123,
        ),
      },
    ],
  ),
]
```

However, this representation has some drawbacks (as pointed out by @Metadorius in [this](bevyengine#4561 (comment)) comment):

1. Increased nesting and more characters (minor effect on overall size)
2. More importantly, by definition, entities cannot have more than one instance of any given component. Therefore, such data is best stored as a map— where all values are meant to have unique keys.


## Solution

Change `components` to store a map of components rather than a list:

```rust
[
  (
    entity: 0,
    components: {
      "bevy_transform::components::transform::Transform": (
        translation: (
          x: 0.0,
          y: 0.0,
          z: 0.0
        ),
        rotation: (0.0, 0.0, 0.0, 1.0),
        scale: (
          x: 1.0,
          y: 1.0,
          z: 1.0
        ),
      ),
      "my_crate::Foo": (
        text: "Hello World",
      ),
      "my_crate::Bar": (
        baz: 123
      ),
    },
  ),
]
```

#### Code Representation

This change only affects the scene format itself. `DynamicEntity` still stores its components as a list. The reason for this is that storing such data as a map is not really needed since:
1. The "key" of each value is easily found by just calling `Reflect::type_name` on it
2. We should be generating such structs using the `World` itself which upholds the one-component-per-entity rule

One could in theory create manually create a `DynamicEntity` with duplicate components, but this isn't something I think we should focus on in this PR. `DynamicEntity` can be broken in other ways (i.e. storing a non-component in the components list), and resolving its issues can be done in a separate PR.

---

## Changelog

* The scene format now uses a map to represent the collection of components rather than a list

## Migration Guide

The scene format now uses a map to represent the collection of components. Scene files will need to update from the old list format.

<details>
<summary>Example Code</summary>

```rust
// OLD
[
  (
    entity: 0,
    components: [
      {
        "bevy_transform::components::transform::Transform": (
          translation: (
            x: 0.0,
            y: 0.0,
            z: 0.0
          ),
          rotation: (0.0, 0.0, 0.0, 1.0),
          scale: (
            x: 1.0,
            y: 1.0,
            z: 1.0
          ),
        ),
      },
      {
        "my_crate::Foo": (
          text: "Hello World",
        ),
      },
      {
        "my_crate::Bar": (
          baz: 123,
        ),
      },
    ],
  ),
]

// NEW
[
  (
    entity: 0,
    components: {
      "bevy_transform::components::transform::Transform": (
        translation: (
          x: 0.0,
          y: 0.0,
          z: 0.0
        ),
        rotation: (0.0, 0.0, 0.0, 1.0),
        scale: (
          x: 1.0,
          y: 1.0,
          z: 1.0
        ),
      ),
      "my_crate::Foo": (
        text: "Hello World",
      ),
      "my_crate::Bar": (
        baz: 123
      ),
    },
  ),
]
```

</details>
bors bot pushed a commit that referenced this pull request Jan 4, 2023
# Objective

Fixes #6891

## Solution

Replaces deserializing map keys as `&str` with deserializing them as `String`.

This bug seems to occur when using something like `File` or `BufReader` rather than bytes or a string directly (I only tested `File` and `BufReader` for `rmp-serde` and `serde_json`). This might be an issue with other `Read` impls as well (except `&[u8]` it seems).

We already had passing tests for Message Pack but none that use a `File` or `BufReader`. This PR also adds or modifies tests to check for this in the future.

This change was also based on [feedback](#4561 (comment)) I received in a previous PR.

---

## Changelog

- Fix bug where scene deserialization using certain readers could fail (e.g. `BufReader`, `File`, etc.)
bors bot pushed a commit that referenced this pull request Jan 4, 2023
# Objective

Fixes #6891

## Solution

Replaces deserializing map keys as `&str` with deserializing them as `String`.

This bug seems to occur when using something like `File` or `BufReader` rather than bytes or a string directly (I only tested `File` and `BufReader` for `rmp-serde` and `serde_json`). This might be an issue with other `Read` impls as well (except `&[u8]` it seems).

We already had passing tests for Message Pack but none that use a `File` or `BufReader`. This PR also adds or modifies tests to check for this in the future.

This change was also based on [feedback](#4561 (comment)) I received in a previous PR.

---

## Changelog

- Fix bug where scene deserialization using certain readers could fail (e.g. `BufReader`, `File`, etc.)
alradish pushed a commit to alradish/bevy that referenced this pull request Jan 22, 2023
# Objective

Fixes bevyengine#6891

## Solution

Replaces deserializing map keys as `&str` with deserializing them as `String`.

This bug seems to occur when using something like `File` or `BufReader` rather than bytes or a string directly (I only tested `File` and `BufReader` for `rmp-serde` and `serde_json`). This might be an issue with other `Read` impls as well (except `&[u8]` it seems).

We already had passing tests for Message Pack but none that use a `File` or `BufReader`. This PR also adds or modifies tests to check for this in the future.

This change was also based on [feedback](bevyengine#4561 (comment)) I received in a previous PR.

---

## Changelog

- Fix bug where scene deserialization using certain readers could fail (e.g. `BufReader`, `File`, etc.)
ItsDoot pushed a commit to ItsDoot/bevy that referenced this pull request Feb 1, 2023
> Note: This is rebased off bevyengine#4561 and can be viewed as a competitor to that PR. See `Comparison with bevyengine#4561` section for details.

# Objective

The current serialization format used by `bevy_reflect` is both verbose and error-prone. Taking the following structs[^1] for example:

```rust
// -- src/inventory.rs

#[derive(Reflect)]
struct Inventory {
  id: String,
  max_storage: usize,
  items: Vec<Item>
}

#[derive(Reflect)]
struct Item {
  name: String
}
```

Given an inventory of a single item, this would serialize to something like:

```rust
// -- assets/inventory.ron

{
  "type": "my_game::inventory::Inventory",
  "struct": {
    "id": {
      "type": "alloc::string::String",
      "value": "inv001",
    },
    "max_storage": {
      "type": "usize",
      "value": 10
    },
    "items": {
      "type": "alloc::vec::Vec<alloc::string::String>",
      "list": [
        {
          "type": "my_game::inventory::Item",
          "struct": {
            "name": {
              "type": "alloc::string::String",
              "value": "Pickaxe"
            },
          },
        },
      ],
    },
  },
}
```

Aside from being really long and difficult to read, it also has a few "gotchas" that users need to be aware of if they want to edit the file manually. A major one is the requirement that you use the proper keys for a given type. For structs, you need `"struct"`. For lists, `"list"`. For tuple structs, `"tuple_struct"`. And so on.

It also ***requires*** that the `"type"` entry come before the actual data. Despite being a map— which in programming is almost always orderless by default— the entries need to be in a particular order. Failure to follow the ordering convention results in a failure to deserialize the data.

This makes it very prone to errors and annoyances.


## Solution

Using bevyengine#4042, we can remove a lot of the boilerplate and metadata needed by this older system. Since we now have static access to type information, we can simplify our serialized data to look like:

```rust
// -- assets/inventory.ron

{
  "my_game::inventory::Inventory": (
    id: "inv001",
    max_storage: 10,
    items: [
      (
        name: "Pickaxe"
      ),
    ],
  ),
}
```

This is much more digestible and a lot less error-prone (no more key requirements and no more extra type names).

Additionally, it is a lot more familiar to users as it follows conventional serde mechanics. For example, the struct is represented with `(...)` when serialized to RON.

#### Custom Serialization

Additionally, this PR adds the opt-in ability to specify a custom serde implementation to be used rather than the one created via reflection. For example[^1]:

```rust
// -- src/inventory.rs

#[derive(Reflect, Serialize)]
#[reflect(Serialize)]
struct Item {
  #[serde(alias = "id")]
  name: String
}
```

```rust
// -- assets/inventory.ron

{
  "my_game::inventory::Inventory": (
    id: "inv001",
    max_storage: 10,
    items: [
      (
        id: "Pickaxe"
      ),
    ],
  ),
},
```

By allowing users to define their own serialization methods, we do two things:

1. We give more control over how data is serialized/deserialized to the end user
2. We avoid having to re-define serde's attributes and forcing users to apply both (e.g. we don't need a `#[reflect(alias)]` attribute).

### Improved Formats

One of the improvements this PR provides is the ability to represent data in ways that are more conventional and/or familiar to users. Many users are familiar with RON so here are some of the ways we can now represent data in RON:

###### Structs

```js
{
  "my_crate::Foo": (
    bar: 123
  )
}
// OR
{
  "my_crate::Foo": Foo(
    bar: 123
  )
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "my_crate::Foo",
  "struct": {
    "bar": {
      "type": "usize",
      "value": 123
    }
  }
}
```

</details>

###### Tuples

```js
{
  "(f32, f32)": (1.0, 2.0)
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "(f32, f32)",
  "tuple": [
    {
      "type": "f32",
      "value": 1.0
    },
    {
      "type": "f32",
      "value": 2.0
    }
  ]
}
```

</details>

###### Tuple Structs

```js
{
  "my_crate::Bar": ("Hello World!")
}
// OR
{
  "my_crate::Bar": Bar("Hello World!")
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "my_crate::Bar",
  "tuple_struct": [
    {
      "type": "alloc::string::String",
      "value": "Hello World!"
    }
  ]
}
```

</details>

###### Arrays

It may be a bit surprising to some, but arrays now also use the tuple format. This is because they essentially _are_ tuples (a sequence of values with a fixed size), but only allow for homogenous types. Additionally, this is how RON handles them and is probably a result of the 32-capacity limit imposed on them (both by [serde](https://docs.rs/serde/latest/serde/trait.Serialize.html#impl-Serialize-for-%5BT%3B%2032%5D) and by [bevy_reflect](https://docs.rs/bevy/latest/bevy/reflect/trait.GetTypeRegistration.html#impl-GetTypeRegistration-for-%5BT%3B%2032%5D)).

```js
{
  "[i32; 3]": (1, 2, 3)
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "[i32; 3]",
  "array": [
    {
      "type": "i32",
      "value": 1
    },
    {
      "type": "i32",
      "value": 2
    },
    {
      "type": "i32",
      "value": 3
    }
  ]
}
```

</details>

###### Enums

To make things simple, I'll just put a struct variant here, but the style applies to all variant types:

```js
{
  "my_crate::ItemType": Consumable(
    name: "Healing potion"
  )
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "my_crate::ItemType",
  "enum": {
    "variant": "Consumable",
    "struct": {
      "name": {
        "type": "alloc::string::String",
        "value": "Healing potion"
      }
    }
  }
}
```

</details>

### Comparison with bevyengine#4561

This PR is a rebased version of bevyengine#4561. The reason for the split between the two is because this PR creates a _very_ different scene format. You may notice that the PR descriptions for either PR are pretty similar. This was done to better convey the changes depending on which (if any) gets merged first. If bevyengine#4561 makes it in first, I will update this PR description accordingly.

---

## Changelog

* Re-worked serialization/deserialization for reflected types
* Added `TypedReflectDeserializer` for deserializing data with known `TypeInfo`
* Renamed `ReflectDeserializer` to `UntypedReflectDeserializer` 
* ~~Replaced usages of `deserialize_any` with `deserialize_map` for non-self-describing formats~~ Reverted this change since there are still some issues that need to be sorted out (in a separate PR). By reverting this, crates like `bincode` can throw an error when attempting to deserialize non-self-describing formats (`bincode` results in `DeserializeAnyNotSupported`)
* Structs, tuples, tuple structs, arrays, and enums are now all de/serialized using conventional serde methods

## Migration Guide

* This PR reduces the verbosity of the scene format. Scenes will need to be updated accordingly:

```js
// Old format
{
  "type": "my_game::item::Item",
  "struct": {
    "id": {
      "type": "alloc::string::String",
      "value": "bevycraft:stone",
    },
    "tags": {
      "type": "alloc::vec::Vec<alloc::string::String>",
      "list": [
        {
          "type": "alloc::string::String",
          "value": "material"
        },
      ],
    },
}

// New format
{
  "my_game::item::Item": (
    id: "bevycraft:stone",
    tags: ["material"]
  )
}
```

[^1]: Some derives omitted for brevity.
ItsDoot pushed a commit to ItsDoot/bevy that referenced this pull request Feb 1, 2023
# Objective

Currently scenes define components using a list:

```rust
[
  (
    entity: 0,
    components: [
      {
        "bevy_transform::components::transform::Transform": (
          translation: (
            x: 0.0,
            y: 0.0,
            z: 0.0
          ),
          rotation: (0.0, 0.0, 0.0, 1.0),
          scale: (
            x: 1.0,
            y: 1.0,
            z: 1.0
          ),
        ),
      },
      {
        "my_crate::Foo": (
          text: "Hello World",
        ),
      },
      {
        "my_crate::Bar": (
          baz: 123,
        ),
      },
    ],
  ),
]
```

However, this representation has some drawbacks (as pointed out by @Metadorius in [this](bevyengine#4561 (comment)) comment):

1. Increased nesting and more characters (minor effect on overall size)
2. More importantly, by definition, entities cannot have more than one instance of any given component. Therefore, such data is best stored as a map— where all values are meant to have unique keys.


## Solution

Change `components` to store a map of components rather than a list:

```rust
[
  (
    entity: 0,
    components: {
      "bevy_transform::components::transform::Transform": (
        translation: (
          x: 0.0,
          y: 0.0,
          z: 0.0
        ),
        rotation: (0.0, 0.0, 0.0, 1.0),
        scale: (
          x: 1.0,
          y: 1.0,
          z: 1.0
        ),
      ),
      "my_crate::Foo": (
        text: "Hello World",
      ),
      "my_crate::Bar": (
        baz: 123
      ),
    },
  ),
]
```

#### Code Representation

This change only affects the scene format itself. `DynamicEntity` still stores its components as a list. The reason for this is that storing such data as a map is not really needed since:
1. The "key" of each value is easily found by just calling `Reflect::type_name` on it
2. We should be generating such structs using the `World` itself which upholds the one-component-per-entity rule

One could in theory create manually create a `DynamicEntity` with duplicate components, but this isn't something I think we should focus on in this PR. `DynamicEntity` can be broken in other ways (i.e. storing a non-component in the components list), and resolving its issues can be done in a separate PR.

---

## Changelog

* The scene format now uses a map to represent the collection of components rather than a list

## Migration Guide

The scene format now uses a map to represent the collection of components. Scene files will need to update from the old list format.

<details>
<summary>Example Code</summary>

```rust
// OLD
[
  (
    entity: 0,
    components: [
      {
        "bevy_transform::components::transform::Transform": (
          translation: (
            x: 0.0,
            y: 0.0,
            z: 0.0
          ),
          rotation: (0.0, 0.0, 0.0, 1.0),
          scale: (
            x: 1.0,
            y: 1.0,
            z: 1.0
          ),
        ),
      },
      {
        "my_crate::Foo": (
          text: "Hello World",
        ),
      },
      {
        "my_crate::Bar": (
          baz: 123,
        ),
      },
    ],
  ),
]

// NEW
[
  (
    entity: 0,
    components: {
      "bevy_transform::components::transform::Transform": (
        translation: (
          x: 0.0,
          y: 0.0,
          z: 0.0
        ),
        rotation: (0.0, 0.0, 0.0, 1.0),
        scale: (
          x: 1.0,
          y: 1.0,
          z: 1.0
        ),
      ),
      "my_crate::Foo": (
        text: "Hello World",
      ),
      "my_crate::Bar": (
        baz: 123
      ),
    },
  ),
]
```

</details>
ItsDoot pushed a commit to ItsDoot/bevy that referenced this pull request Feb 1, 2023
# Objective

Fixes bevyengine#6891

## Solution

Replaces deserializing map keys as `&str` with deserializing them as `String`.

This bug seems to occur when using something like `File` or `BufReader` rather than bytes or a string directly (I only tested `File` and `BufReader` for `rmp-serde` and `serde_json`). This might be an issue with other `Read` impls as well (except `&[u8]` it seems).

We already had passing tests for Message Pack but none that use a `File` or `BufReader`. This PR also adds or modifies tests to check for this in the future.

This change was also based on [feedback](bevyengine#4561 (comment)) I received in a previous PR.

---

## Changelog

- Fix bug where scene deserialization using certain readers could fail (e.g. `BufReader`, `File`, etc.)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-Reflection Runtime information about types C-Usability A targeted quality-of-life change that makes Bevy easier to use S-Ready-For-Final-Review This PR has been approved by the community. It's ready for a maintainer to consider merging it X-Controversial There is active debate or serious implications around merging this PR
Projects
Archived in project
Status: Done
Development

Successfully merging this pull request may close these issues.

9 participants