Dynamic API For Use With Scripting

[zicklag]

Hey there, I'm considering building a game engine on top of shipyard, but scripting support is essential. What I think I need is an API to shipyard that doesn't require the use of Rust's fancy type checking. I'm not 100% sure I know what I need to accomplish what I need to accomplish, but I want to start the discussion to start figuring it out. 🙂

Firstly, don't get me wrong, I love Rust's fancy type system and, though I haven't use shipyard extensively yet, I think I like the direction that you have been going with in terms of ergonomics. The problem, though is that I need to be able to insert new components and systems that are not known at compile time. I'm looking to accomplish something similar to Amethyst's Scripting RFC ( you don't necessarily need to read it, I'll explain ).

So the goal is to have an API, potentially accessible over a C FFI, though not necessary for my use-case yet ( and maybe that FFI could be a separate crate? ), that we can use to store components, which will be opaque byte arrays. The component data will likely be stored in the C memory layout, but that is not important to the ECS itself it just needs to store the bytes.

I need to be able to register components at run-time, though and so I need a way to distinguish between all of the different component types, even though they are all byte-arrays. In other words, we can't use the Rust type system to identify components. We might need to identify components by strings or integers or otherwise have some sort of ComponentID.

I also need to be able to query the components in systems that are created at runtime. I think workloads would need to be defined at runtime, too.

---

I don't know anything about the internals of shipyard yet, and I may not have explained that very well, but here's the gist:

• I need to create components at runtime:
  • Each component will be a byte array
  • Each component needs to have a component ID I can use to query the component in systems
• I need to create systems and workloads at runtime:
  • I will know at runtime which component IDs I need to have read or write access to for each system like normal
• An FFI is optional
  • The API to do the actual runtime registration of components and systems can be a base for an FFI if necessary

I might be able to help with this, but no promises. I'll probably be checking out the internals of shipyard a bit to see how it works and how easy it might be to implement.

[leudz]

Hi 🙋 This was discussed a long time ago on gitter but the api changed a lot since then.

At the time, a few things were implemented:

• StorageId is the equivalent of ComponentID
• systems as functions and not trait

It was a lot of work and nobody needed it so the systems fn got commented out and StorageId was never used. With the new systems as functions the old code was removed.

Making shipyard use StorageId isn't a lot work, you can then use it to add components Rust has no information about.
The hard part is the rest.

I don't think I'll personally need it and there are other features that will benefit more people.
So I can help but that's all I can promise.

[zicklag]

Cool, thanks for the reply. I might take a look at it and I'll ask questions here if I have any.

[leudz]

Apparently I choose the wrong day to not look at the forum 😆
Just so everyone can follow, here's a link.
I read the forum's discussion but I'm pretty sure we can make something simpler.

Here's what I had in mind, I haven't tested it so there might be some incorrect or missing things.

First, since FFI types are all going to come in the shape of byte arrays, we can store them in a special SparseSet<u8>.
One u8 won't mean one component and it'll have an Option<fn(*mut c_void)> (or Option<fn(*mut u8)>? I'm not sure) to drop data.
We probably will have to emulate a Vec with alloc for the data.

When users want to create a FFI storage they'll have to call a method that takes:

• u64 as storage id
• *mut u8 for the data
• c_uint for its size
• c_uint for its alignment
• Option<extern fn(*mut c_void)> to drop the data if needed

Then to retrieve the data, returning one type, two types or an enum is a trade-off.

• Two types, like in Rust, would allow finer grained mutation control at compile time.
• A single type would defer issues to runtime for a simpler api.
• An enum would make the potential runtime issues clearer while keeping a simple api.

I think enum is the best bet.

For the views themselves, there are a few ways to go about it. I think the simplest one would be to make the storage carry its lock. This way we can make the view a simple pointer to the storage and have all the information needed. Views would just be something like:

#[repr(transparent)]
struct FFIViewConst(*const FFISparseSet);

#[repr(transparent)]
struct FFIViewMut(*mut FFISparseSet);

#[repr(C)]
enum FFIView {
    Shared(FFIViewConst),
    Exclusive(FFIViewMut),
}

The storage will stay borrowed until a free function is called on the view.

For lots of methods, shipyard uses tuples of various sizes and generics. We can't use this in FFI but I think variadic functions can have the same role.

I probably forgot a ton of things. What are your thoughts?

[zicklag]

Well, it started out as a "Can you do dyn Any<'lifetime>?" question, and then it turned out more complicated than that and I was going to have to add details about the use-case so I renamed it. I didn't want to bug you until I got my current code pushed up so you could see the direction I was going, but I was going to mention you to get you in on it.

I just pushed my code and here is the direction I was going:

https://github.com/leudz/shipyard/pull/99/files#diff-5c4fb3ad249d34ee03fab6185b3aef4aR12

I haven't had time to process your ideas, but I'll leave my thoughts as soon as I can! 😄

[zicklag]

So I've gotten some time to look at this and I have some questions/thoughts. Note that this new idea is a little different than the one in the PR currently.

Separation of FFI and Dynamic Systems Issues

So, while I'm not 100% sure this is the best idea, I would propose that we separate the issues FFI and dynamic systems into separate problems to solve individually, starting with dynamic systems.

What I mean is that while we will definitely want an FFI, currently I actually don't need one ( yet ), and I think it would be nice to start by providing a way from pure Rust to create systems and components that are only known at runtime, without having to use constructs such as unsafe pointers that would be required for FFI. Dynamic systems is something that is useful in plain Rust, even without an FFI. ( For instance, I want to bind the ECS to the RustPython implementation, which has a Rust API so I don't have to use any FFI ).

After we have established a Rusty way to create dynamic systems we could add an FFI.

Component Storage

So, normally we store components in the data vector of the SparseSet<T> where rust will know the type and size at compile time so the Vec can allocate memory for it, but the issue with dynamic components is that you don't know the size at compile time, which means the Vec can't know how much memory to allocate.

I didn't understand what you meant when you said SparseSet<u8> because that would just be a one byte component right?

We probably will have to emulate a Vec with alloc for the data.

Yeah, I agree. I think we might be able to do something like this playground example where we just build a DynamicVec directly on top Vec ( though we could maybe optimize some parts with a little unsafe? ). With something like that DynamicVec we could replace the data Vec in SparceSet with DynamicVec and for all non script-defined components the element_size would just be 1. Then we could store normal and scripted types in the same storage without a generic, hopefully helping us to reduce the amount of difference in the way we handle scripted and non-scripted components.

Views

At this point views may hardly need to change? If we use a DynamicVec to back the data of the SparseSet, then the views will actually get &[T] when indexing into the SparceSet data and that will be of whatever length the component was initialized with.

---

I think this design lets us keep almost every aspect about the way everything is already designed. The workloads and system scheduler should be able to remain unchanged. I haven't finished combing through the code-base yet, though so I could be horribly wrong about all this.

I don't understand it enough yet to know whether or not it will work until I try to put it all down and see if rustc still likes me when I'm done. 😃

[zicklag]

Yeah, I'm not sure the whole DynamicVec strategy to replace the data Vec of SparseSet is a good idea. The issue is that instead of having a component T you have to have every component a &[T], which means certain function return types, which used to be T have to be Vec<T> because the return value needs to be owned and therefore can't be &[T]. That means we have to make at least some allocations for the Vec<T>, which would incur some unnecessary cost if you weren't doing scripting.

I'm really not the most experienced with strategies for this kind of thing. Is there any way to store some sort of special type inside of a Vec where we only know the size at runtime, without causing extra allocations? I can't figure anything out.

Or does the SparseSet just need to change a little bit to avoid the allocations that would occur if tried to use something like the DynamicVec? Any ideas?

[leudz]

Sorry it took me so long to respond, I knew someone on zulip used Python with shipyard so I asked them a few questions. I also looked into pyo3 and a few other things.

My few questions yielded interesting answers.
They told me that as long as all logic happens on the Rust side, you can just use shipyard as is. I don't think that's what you're looking for though.
They're not using Python anymore (and scripting altogether) but switched to Ron files.
They also mentioned Mun as something to look into in the future.

Assuming you really need scripting, I agree with you that you probably don't always need the full FFI experience.

So first level of dynamism: custom storages.
Instead of choosing between SparseSet or unique storages, users could use anything that implements the UnknownStorage trait.
These custom storages would use StorageId::Custom.

Second level: fully dynamic storages.
This kind of storage can be used to store types Rust doesn't even know.
We can't use a Vec because it needs the size and alignment at compile time.
Vec<u8> and size doesn't work either because of alignment.
That's why I talked about alloc, I think it's the only way. Even with alloc it'll probably require a derive on components used in a storage like this (or take them from FFI) to make sure we don't get into trouble with uninitialized memory.

Here's what custom storages would look like:

struct MyStorage(Vec<usize>);

impl UnknownStorage for MyStorage {
    // -- snip --
}

let world = World::new();

world.add_custom(0, MyStorage(Vec::new()));
world.add_custom(1, SparseSet::<[u8; 8]>::new());

let my_storage: CustomView<MyStorage> = world
    .borrow_custom(0)
    .map(|storage| storage.as_any().downcast_ref().unwrap());

CustomView is nothing special, it's just like a UniqueView. It has a reference to the storage and two borrows.

For workloads the syntax isn't final, but users will have to give this kind of info:

fn sys1(world: &World) -> Result<(), error::Run> {
    let my_storage: CustomView<MyStorage> = world
        .borrow_custom(0)
        .map(|storage| storage.as_any().downcast_ref().unwrap());

    Ok(())
}

fn sys1_infos(infos: &mut BorrowInfos) -> (bool, bool) {
    infos.extend_from_slice(&[(0, Mutation::Exclusive), (1, Mutation::Shared)]);

    (true, true)
}

world
    .add_workload("...")
    .with_custom_system(sys1, sys1_infos)
    .build();

What do you think of custom storages and systems? Do you think you'll need the second level?

[zicklag]

Sorry it took me so long to respond, I knew someone on zulip used Python with shipyard so I asked them a few questions. I also looked into pyo3 and a few other things.

No problem. :)

They also mentioned Mun as something to look into in the future.

I've done a little looking into Mun, and it looks great, but I think we can enable great hot-reloading in a language agnostic way with some extra logic around the scripted ECS. That way you could have hot reloading by dynamically reloading systems/components for any programming language that you bind to the ECS.

I think. :)

---

+1 for custom storages, that sounds like the right approach.

Do you think you'll need the second level?

Would your example count as "Second level"? I'm not sure I understood the difference. If we can implement custom storages by implementing UnknownStorage that means that we can use any sort of fully dynamic struct that we want to back it, right?

I think that we will need to build something that uses either alloc ( like you said ) or just provides special interactions over Vec like the DynamicVec playground experiment, but if we can have that custom type implement UnknownStorage then that should be all we need from shipyard.

I would think that a built-in custom component storage ( like a byte-array based one or something ) could be useful, or maybe provided in a different crate/example, but the important part I think is just being able to add CustomStorages.

---

For workloads the syntax isn't final, but users will have to give this kind of info:

That looks pretty good to me.

[leudz]

"Level 2" will be built on top of "level 1". And yes in theory users could make it but it'll involve a lot of unsafe so shipyard will provide it, ready to be used.
Assuming you want a SparseSet-like storage, I think the question is: does Rust has a definition for the type?

If yes then a custom SparseSet<T> should be enough.
If Rust has no idea what the type looks like, then you'd need "level 2".

---

We can't use Vec without copying a lot and it wouldn't work with Drop types either.
The "alloc path" doesn't have all these issues but I basically have to implement a Vec from scratch.

[zicklag]

Ah, OK, then yes, "Level 2" would be required.

The idea is to have something like this schema definition example ( which looks essentially like Rust ) that would describe the data for each component. These would be loaded at runtime and the byte representation of the component would be determined from the schema, potentially by using the repr(C) byte representation of the equivalent Rust structure ( or some custom representation? It would only need to be custom if the C representation was not flexible enough to handle stuff like generics or something, I think. ). This byte representation would be stored by shipyard in the custom component storage and would be mutated by the scripted systems.

Each different scripting language integration would need to have some implementation that takes the byte representation of the component and the associated schema file, and produces a native object or otherwise some native interface to represent the component in the target scripting language.

For example, for Python, you would need something to translate the raw bytes into a Python Object. That Object would be manipulated by the Python script and those manipulations would be translated into modifications of the raw bytes stored in the ECS. This you would do for each scripting language. Because you standardize on the byte representation and schema definition, you provide an interface for any number of scripting languages to operate on the same components in the same ECS world seamlessly!

---

I'm glad to help with this however I can and can find time for ( with my time being pretty inconsistent, just a warning 😄 ). I'm not super acquainted with unsafe and the rules for writing sound code so that might not be the best place for me to contribute, but I'll try whatever especially if I've got some guidance.

[leudz]

It's an interesting project, very ambitious.

I'm going to check what needs to be done for "level 1", if it's not much work, I'll do it. If it requires more time I'll post a comment explaining what needs to be done and how.

Regardless of what I do, you'll get updates here.

[zicklag]

Hey @leudz is there anything I can help with yet? Might get some time over the weekend for this.

Don't worry if you haven't gotten to anything yet. 🙂

[leudz]

Hey! I don't think you can help yet but I've made some progress on what I have to do before taking care of this issue.
Shared components should be bug free or close to it and I made some progress on serde. This issue comes right after it.

[zicklag]

Friendly ping. :) Any progress or help I can give?

[zicklag]

With the release of the Bevy game engine it looks like I'm going to be building on that for my game engine and therefore probably won't need this anymore. I'm really sorry if that makes any time you spent on this a huge waste. 😬

Thanks for the useful discussion, though. 👍 😃

[leudz]

No waste of time, don't worry. Best of luck for your project =)