Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for ARM #53

Closed
ethanfrey opened this issue Mar 11, 2020 · 52 comments
Closed

Add support for ARM #53

ethanfrey opened this issue Mar 11, 2020 · 52 comments

Comments

@ethanfrey
Copy link
Member

This is now supported in wasmer (as of 0.11) and has been requested by Zach. If we can update the build-scripts to support it, that would be a cool addition

https://medium.com/wasmer/running-webassembly-on-arm-7d365ed0e50c

@ethanfrey
Copy link
Member Author

ethanfrey commented Oct 5, 2020

Update: I tried but there were still some issues with wasmer. Also, they are going through a re-write to 1.0, so not worth the time now.

I would ask for help on this for some Pi-lover once we have a solid wasmer release.

@faddat
Copy link
Contributor

faddat commented Oct 5, 2020

Well I am certainly pi lover, so let me know when you think we have a solid wasmer release 👍

@webmaster128
Copy link
Member

Wasmer claims to have good support for the new ARM based MacBook now. So it should be possible to support both aarch64-apple-darwin and aarch64 Linux.

@Kwaskoff
Copy link

Kwaskoff commented Jan 13, 2021

Well I am certainly pi lover, so let me know when you think we have a solid wasmer release +1

We got Wasmer release already!
https://github.com/wasmerio/wasmer/releases/tag/1.0.1

@webmaster128
Copy link
Member

webmaster128 commented Jan 13, 2021

@Kwaskoff was there any relevant change in Wasmer between 1.0.0 (integrated here already) and 1.0.1?

This ticket is unblocked but the Confio team will probably not work on it until we have developers with ARM Macs as dev machines. If this is important to you, feel free to look into it. A good starting point is the compatibility table here: https://github.com/CosmWasm/wasmvm#overview

@faddat
Copy link
Contributor

faddat commented Mar 18, 2021

I... believe that this is solvable.

@faddat
Copy link
Contributor

faddat commented Mar 18, 2021

Graviton2, Ubuntu 20.10:

sudo apt install llvm-11-dev llvm-11
export IS_LINUX=1
export IS_AARCH64=1

What if it was not the singlepass compiler?

May I use cranelift and expect it to work?

@faddat
Copy link
Contributor

faddat commented Mar 18, 2021

#####
#
# The Matrix
#
#####

# The matrix is the product of the following columns:
#
# |------------|--------|----------|--------------|-------|
# | Compiler   ⨯ Engine ⨯ Platform ⨯ Architecture ⨯ libc  |
# |------------|--------|----------|--------------|-------|
# | Cranelift  | JIT    | Linux    | amd64        | glibc |
# | LLVM       | Native | Darwin   | aarch64      | musl  |
# | Singlepass |        | Windows  |              |       |
# |------------|--------|----------|--------------|-------|
#
# Here is what works and what doesn't:
#
# * Cranelift with the JIT engine works everywhere,
#
# * Cranelift with the Native engine works on Linux+Darwin/`amd64`,
#   but it doesn't work on */`aarch64` or Windows/*.
#
# * LLVM with the JIT engine works on Linux+Darwin/`amd64`,
#   but it doesn't work on */`aarch64` or Windows/*.
#
# * LLVM with the Native engine works on
#   Linux+Darwin/`amd64`+`aarch64`, but it doesn't work on Windows/*.
#
# * Singlepass with the JIT engine works on Linux+Darwin/`amd64`, but
#   it doesn't work on */`aarch64` or Windows/*.
#
# * Singlepass with the Native engine doesn't work because it doesn't
#   know how to output object files for the moment.
#
# * Windows isn't tested on `aarch64`, that's why we consider it's not
#   working, but it might possibly be.

@ethanfrey
Copy link
Member Author

Cranelift compiles better. BUT it is not safe for a blockchain context.
It is an optimizing compiler and it is expected that giving it arbitrary malicious wasm to compile could open up attack vectors, minimally O(N^2) compile time (where N is the size of the wasm contract)

We need this to compile with singlepass to be useful for wasmd

@faddat
Copy link
Contributor

faddat commented Mar 18, 2021

"Cranelift with the JIT engine" is OK?

@ethanfrey
Copy link
Member Author

If this is not supported by wasmer, feel free to jump on their github and raise an issue. (It will have more chance of getting addressed)

@ethanfrey
Copy link
Member Author

ethanfrey commented Mar 18, 2021

All design and testing has been with singlepass. It is the only engine we feel comfortable with using in a blockchain setting.

@faddat
Copy link
Contributor

faddat commented Mar 18, 2021

All design and testing has been with singlepass. It is the only engine we feel comfortable with using in a blockchain setting.

dang.

LLVM is a no-go for the same reasons that Cranelift is a no-go?

@ethanfrey
Copy link
Member Author

ethanfrey commented Mar 18, 2021

  • Singlepass with the JIT engine works on Linux+Darwin/amd64, but
    it doesn't work on /aarch64 or Windows/.

Thus the long-open tickets to add arm64 and windows support.

Every time someone says it's working in wasmer now, it turns out it is a different backend.

@ethanfrey
Copy link
Member Author

ethanfrey commented Mar 18, 2021

LLVM is a no-go for the same reasons that Cranelift is a no-go?

LLVM has blazing fast code execution (I clocked wasm doing loops of sha256 pre-compiled with LLVM but inside the wasmer vm to be within a factor of two of the same rust code compiled directly to native binary). However, the compilation times are long (think close to 1 second) and surely have far, far more compiler bombs lurking than cranelift. I benchmarked it in the beginning, but discarded it for use in a blockchain setting shortly afterwards

@faddat
Copy link
Contributor

faddat commented Mar 18, 2021

got it. Ok, so to fix this (I'm pretty OK w/ arm ports....) I would need to make "singlepass with the JIT engine" work on arm64, yes?

https://developers.redhat.com/blog/2020/01/20/mir-a-lightweight-jit-compiler-project/

"use in a blockchain context" == determinism concerns?

https://medium.com/nearprotocol/wasm-for-blockchain-2019-d093bfeb6133

I'm getting this feeling you've read both articles, however.

Or possibly mir == the JIT compiler mentioned in the config.

https://github.com/rust-lang/miri

wasmerio/wasmer#713

@faddat
Copy link
Contributor

faddat commented Mar 18, 2021

NEAR seems to be using LLVM

Too scary?

Perhaps they've optimized?

@faddat
Copy link
Contributor

faddat commented Mar 18, 2021

https://github.com/wasmerio/wasmer/tree/63a2d8129c7ad5ca670d041859de45e01292dd12/lib/singlepass-backend/src

In the past, they had a translator for the x86 asm instructions in the emitter.

It is my understanding that codegen is often used to great effect in such cases. Not sure if that'll work here.

https://censoredusername.github.io/dynasm-rs/language/index.html

@ethanfrey
Copy link
Member Author

NEAR seems to be using LLVM
Too scary?
Perhaps they've optimized?

I'd be happy to see a link of what they are doing.

That said, they have some very skilled developers building a new platform from the ground up and I have a lot of respect for their engineering. If they use LLVM, they probably manage to run the compilation in some "background process" / "offchain worker" and sandbox it somehow to avoid run-away processs.

I have considered doing some hybrid approach, where it is available as singlepass right away and some selected contracts get recompiled in LLVM in an offchain worker. But there is no such infrastructure available for the Cosmos SDK and I didn't feel like inventing yet another subsystem.

@faddat
Copy link
Contributor

faddat commented Mar 18, 2021

@ethanfrey
Copy link
Member Author

"use in a blockchain context" == determinism concerns?

Kind of (which is why we ban some floats). But mainly it is ability to gas meter all code execution or have a predictable performance. Singlepass gives us predictable O(N) compilation time, so we can just charge based on wasm bytecode size.

Optimizing compilers provide neither and have no upper limit on execution time (which blocks all other processing on every full node in the blockchain). This could be an extremely dangerous attack vector.

@ethanfrey
Copy link
Member Author

If you want to work on adding arm64 bytecode emission to wasmer, that could be a fun rust project (IIFC, you are skilled in Rust) and a nice help to the ecosystem.

If your employer would give you time for that...

@faddat
Copy link
Contributor

faddat commented Mar 18, 2021

Yeah, this seems to affect many projects that want ARM64 build targets and CosmWasm

  • Desmos
  • Stargaze
  • Splash
  • Juno

Alternative is a little silly: x86_64 single board computers in place of Raspberry Pi's

I am pretty solidly booked, time-wise, but knew I'd be up insanely late tonight as I'm going on a podcast. Figured that I could bang this out.

Perhaps not though.

The hopeful bit is:

the emitter uses dynasm and dynasm fully supports aarch64.

I'm not even remotely skilled in Rust, yet.

@ethanfrey
Copy link
Member Author

If you really want to dig in, there was a magic streaming/optimizing compiler that produced highly efficient bytecode in a singlepass.

At least this is what Parity claimed: https://www.parity.io/lightbeam-webassembly-compiler/

I'm not sure what the current state is, but you can find the code here: https://github.com/bytecodealliance/wasmtime/tree/main/crates/lightbeam

Maybe someone could get this working with arm64 and integrated with wasmer 🤯

@faddat
Copy link
Contributor

faddat commented Mar 18, 2021

Somehow, I figure that humans aren't writing these 7-8k LOC files like codegen_x86.rs

Could be wrong though, people have done stranger things :P

This won't get finished tonight, that's for sure.

@ethanfrey thanks for helping me understand the issues.

@faddat
Copy link
Contributor

faddat commented Mar 19, 2021

@ethanfrey TBH I don't know how much I truly want to dig in.

Just, I want this on my Pi's.

image

Could you let me know what you think?

https://crates.io/crates/wasmer-vm-near/reverse_dependencies

faddat pushed a commit to faddat/go-cosmwasm that referenced this issue Dec 10, 2021
faddat pushed a commit to faddat/go-cosmwasm that referenced this issue Dec 10, 2021
Prevent storing wasm code in simulation mode
@webmaster128
Copy link
Member

THERE IS NO ARM SUPPORT IN SINGLEPASS, PERIOD.

@antrixy
Copy link

antrixy commented Dec 21, 2021

Will this be resvoled soon. I am raspberry lover. Its been more than two years since this issue was opened.

@0xjame5
Copy link

0xjame5 commented Dec 25, 2021

@webmaster128 could you clarify more, do you mean indefinitely there will be no support for Arm based development using wasmnvm?

@ethanfrey
Copy link
Member Author

@webmaster128 could you clarify more, do you mean indefinitely there will be no support for Arm based development using wasmnvm?

Please read the whole thread. Like this #53 (comment)

There is currently no upstream support for arm64 in wasmer singlepass compiler.

This is our largest request for support from Wasmer and can be tracked here https://github.com/wasmerio/wasmer/labels/project-confio

This is a known issue and hard to solve and we are patiently awaiting upstream.

@faddat
Copy link
Contributor

faddat commented Dec 28, 2021

I'd like to second @ethanfrey here:

I spent a lot of time trying to fix this at one point, that is why this issue's thread is so long. It is very difficult and the fix is likely best done upstream.

@mandrean
Copy link

FYI there's a release candidate of wasmer with singlepass arm64 support now!

https://github.com/wasmerio/wasmer/releases/tag/2.2.0-rc1

@shanev
Copy link
Contributor

shanev commented Feb 18, 2022

There's now an RC2: https://github.com/wasmerio/wasmer/releases/tag/2.2.0-rc2. Almost there!

@webmaster128
Copy link
Member

The integration PR for cosmwasm-vm is here: CosmWasm/cosmwasm#1224.

Please don't get too excited about this yet. We'll have to create a build system, which more or less doubles the number of builds we create. And every time 1 becomes n, you create a new dimension and things get harder in a non-trivial way. It might even be blocked by #277 but I'm not sure about that. We might get away with Rust 1.55.0 even if not officially supported by the Wasmer team.

@mandrean
Copy link

mandrean commented Feb 19, 2022

Please don't get too excited about this yet

IMHO what most people are excited about is running local dev environments on their new M1 Macs. Some quirks and bugs are acceptable. Being able to use your new expensive laptop is the biggest win

I've actually been doing this with a fork of wasmvm and cosmwasm to get terrad to run natively on my laptop since ~1 month back:

mandrean/terra-core@aebba18
mandrean@b9e4201
mandrean/cosmwasm@90cb117

Now, production-worthy arm64 for Linux VMs in the cloud is a different story. Maybe that will take a longer time to sort out. But this is already great stuff for devs with new Macbooks.

@webmaster128
Copy link
Member

This table should capture where we are and where we go with the build system.

OS family Arch Linking Supported Wasmer 2.2+ Note
Linux (glibc) x86_64 shared ✅​libwasmvm.so ✅​libwasmvm.so
Linux (glibc) x86_64 static 🚫​ 🚫​ Would link libwasmvm statically but glibc dynamically as static glibc linking is not recommended. Potentially interesting for Osmosis.
Linux (glibc) aarch64 shared 🚫​ ✅​libwasmvm.aarch64.so
Linux (glibc) aarch64 static 🚫​ 🚫​
Linux (musl) x86_64 shared 🚫​ 🚫​ Possible but not needed
Linux (musl) x86_64 static ✅​libwasmvm_muslc.a ✅​libwasmvm_muslc.a
Linux (musl) aarch64 shared 🚫​ 🚫​ Possible but not needed
Linux (musl) aarch64 static 🚫​ ✅​libwasmvm_muslc.aarch64.a
macOS x86_64 shared ✅​libwasmvm.dylib ✅​libwasmvm.dylib Fat/universal library can contain multiple archs
macOS x86_64 static 🚫​ 🚫​
macOS aarch64 shared 🚫​ ✅​libwasmvm.dylib Fat/universal library can contain multiple archs
macOS aarch64 static 🚫​ 🚫​
Windows (mingw) x86_64 shared 🏗 wasmvm.dll 🏗 wasmvm.dll See #288
Windows (mingw) x86_64 static 🚫​ 🚫​
Windows (mingw) aarch64 shared 🚫​ 🚫​
Windows (mingw) aarch64 static 🚫​ 🚫​

Turns out that it is possible to have two architectures in a .dylib at the same time. So we can start with M1 support relatively easily as this does not add new filenames to distinguish builds.

@webmaster128
Copy link
Member

mandrean/terra-core@aebba18
mandrean@b9e4201
mandrean/cosmwasm@90cb117

Interesting approach to focus on aarch64 Linux and then use that for M1.

@webmaster128
Copy link
Member

webmaster128 commented Feb 23, 2022

#296 shows this universal library for macOS is working well.

@webmaster128
Copy link
Member

The good news is, we have an libwasmvm_muslc.aarch64.a now. The bad news is, it's not working. If anyone wants to try it in their chain or understands the error message, I'd be happy to discuss in #306.

@webmaster128
Copy link
Member

webmaster128 commented Mar 24, 2022

libwasmvm_muslc.aarch64.a is shipped as part of v1.0.0-beta9 and is working when the linker flags are set correctly.

@webmaster128
Copy link
Member

See https://github.com/CosmWasm/wasmvm/wiki/Linking-wasmvm_muslc for how to use the new static libraries.

@migueldingli1997
Copy link

What's the latest status around this please?

@webmaster128
Copy link
Member

What's the latest status around this please?

We have aarch64 support (64 bit ARM) done and released. So I think this ticket can be closed. Is there anything you are missing?

@Kwaskoff
Copy link

Kwaskoff commented Jul 9, 2022

What's the latest status around this please?

We have aarch64 support (64 bit ARM) done and released. So I think this ticket can be closed. Is there anything you are missing?
I can check it out this month

@webmaster128
Copy link
Member

ARM support is done for now. I'd not recommend using it for consensus critical infrastructure yet. There was some issue popping up here: CosmWasm/cosmwasm#1628. We can follow up with more specific tickets.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

10 participants