State of WebAssembly and Rust?

1

Good afternoon Rustilians! Today I’d like to canvas the community and ask about the state of how WebAssembly (or asm.js) works with Rust. For some time now we’ve had targets like asmjs-unknown-emscripten and wasm32-unknown-emscripten, but these to me at least don’t feel like they’re “over the hump” of becoming close to Tier 1 targets.

I would be particularly interested in a few questions about wasm:

  • If you’re using wasm/asmjs and Rust today, how’s it going? Do things generally work well? Are there weird pain points? Is crates.io missing support for key functionality? Is key functionality missing from rustc?
  • If you looked at using wasm/asmjs and Rust historically, but decided not to do so, why not? What are the blockers for enabling your use case with wasm/asmjs and Rust?
  • In general do you have an idea for a use case with wasm/asmjs and Rust on the web? We’re wondering if there’s some sweet uses of wasm/asmjs which really showcase how Rust can shine on the web.

For those of you also curious it may also be good to give a bit of an overview as to the state of these targets in Rust today. We build and distribute the asmjs and wasm32 targets via rustup, which also means that each PR ensures that these two targets are still building. Notably, however, I’m not aware of any “official” documentation about how to leverage these targets. On CI we also have a dedicated builder running the Emscripten test suite (asmjs) on each PR, so we’ve got quite a broad amount of test coverage of libstd and such in the repo. These two targets, however, require the Emscripten toolchain to be present, and it’s rustc’s job to feed Emscripten gobs of IR which Emscripten then processes into a JS file (or wasm).

The standard library has many stubbed out components which (as far as I know) immediately return errors when used. For example using threads will immediately return errors (likely then causing a panic) and I believe similar functionality like TCP/UDP will also tend to return an error pretty quickly. I’m not personally sure what the state of the crates ecosystem is for running on the web, and I’m quite curious as to what you’ve run into and used along the way!

Some examples of Rust-related WebAssembly/asmjs links we’ve collected are:

And of course in general if you’ve just got thoughts about Rust and the web, we’d love to hear about them!

29 Likes
2

I’ve done a little experimenting with wasm and Rust. Two indispensable tools in in the ecosystem are stdweb which wraps DOM apis and cargo-web sort of like xargo for the web.

Most of the pain points I’ve hit with wasm are not issues with Rust, just general immaturity with the ecosystem. Note that I haven’t really done anything with wasm since about May because of immaturity with wasm in general.

  • Having to download the external emscripten sdk is pretty painful. It would be really nice if this could make into LLVM trunk. cargo-web will handle this for you on linux, but even there there’s a lot that could be improved. Perhaps rustup could be enhanced to deal with this for you in the meantime?
  • Sharing any non-trivial data structures with JS is a pain, and often means that I have to resort to serializing to json. I believe wasm wants to specify a higher level ABI, it will be critical for rust to easily bind to that when it happens.
  • Related, lack of native DOM apis makes using Rust outside of a <canvas> usually a net loss for speed. I suspect there won’t be significant of adoption of wasm until this gets sorted out.

Things Rust can improve

  • Perhaps cargo-web and xargo can join forces so I don’t need two cargo cross compilation tools
  • Rust could do a better job of generating test harnesses so that I can run my tests easily with cargo test (assuming I have node or chrome headless). Note that cargo-web helps does allow you do this kind of testing, but it definitely is not as comfortable or well supported as cargo test.
  • RLS doesn’t handle #[cfg()] blocks all that well, having on the fly type checking of my platform specific code would shorten up the dev cycle a lot.
  • Stabilized opaque refs would make refering to JS objects much cleaner
  • Some way to say that an object is owned by an external GC would make some js wrapper code much easier to deal with and is especially important if/when the ABI for wasm is stabilized.
17 Likes
3

I attempted to try it out only briefly when the targets were first added, but I couldn’t even manage to install the emscripten toolchain. I don’t remember the specifics now, but I remember it being a comedy of errors that made me once again curse the evil of the JavaScript ecosystem and give up. I believe work is being done (or maybe it’s done already) to add native wasm compilation to LLVM, which would be wonderful.

As far as use case, I’m interested in using Rust + a “functional reactive” library to replace things that I’d currently use Elm for today. I love the application development model of Elm, but I don’t love the Haskell style of the language, and would much rather just use Rust, but following the same application design patterns. The one library in Rust I’m aware of that’s working on what I’m describing is domafic, by the venerable @cramertj.

2 Likes
4

I happened to try to compile my pet project through emscripten for the first time yesterday. The experience was pretty bad overall. Installing the emscripten toolchain was slow and painful, and in the end the installation took 25 gigabytes. That’s more than all the other software on my laptop combined, including the operating system.

When I finally got it running, the compiler crashed with some assertion error deep in the emscripten internals. My other, smaller projects compiled just fine, so I think my toolchain is sound, I guess it couldn’t handle some of my project’s dependencies. I haven’t gotten around to posting a proper issue anywhere, mostly because I don’t really know where to start, I’m not familiar enough with the toolchain to know which part failed.

It’s possible none of these problems were really Rust’s fault, but wasm and asmjs are definitely far from being Tier 1 targets.

4 Likes
5

Hi! In May of this year, I used Rust to make a simple text adventure called Werewolves and Wanderer which can compile either to a standard command-line app or a web app via emscripten.

I’m not sure if things have changed since May, but overall I found the wasm support great but a bit confusing. In particular, it seemed like the output file(s) generated often had an unpredictable hash in their filename, and it took me some time to figure out the directory structure of the output. The following lines from my build script are an example of the sort of weird contortions I had to do in order to get all the files in the right place:

export WASM32_TARGET=wasm32-unknown-emscripten
export WASM32_ROOT_DIR=target/${WASM32_TARGET}/debug

cp ${WASM32_ROOT_DIR}/werewolves-and-wanderer.js ${OUTPUT_DIR}
cp ${WASM32_ROOT_DIR}/deps/werewolves_and_wanderer*.wasm ${OUTPUT_DIR}

IIRC, the wildcard in that last line is there because of the hash.

Aside from that, I remember finding it a bit frustrating that there seemed to be no way to pass parameters directly to emcc. I forget exactly why I wanted to do this, but I remember feeling like the “bridge” between cargo/rustc and emcc felt very opaque; I even tried digging into the source code for cargo and/or rustc to figure out how it was calling emcc, to no avail.

Perhaps this has all been fixed at this point, in which case that’s awesome! Thanks again for adding wasm support to rust.

3 Likes
6

The big issue is the lack of library support for Rust<->JS interop. Interacting with any web API is painful; even grabbing a reference to a canvas takes quite a bit of effort with little documentation. You’re also fighting a battle on two fronts, having to be familiar with both emscripten and Rust on top of it. Crates like stdweb will be a huge boon, so I’ve been actively trying to contribute here. Once these crates are mature, I’ll expect there’ll be a large uptick in usage of the asmjs + wasm targets.

2 Likes
7

(foreword: I've done a lot with emscripten, and I now work in Rust, but I haven't yet combined the two, so my perspective is driven by using emscripten on a bunch of C and C++ projects)

As long as Rust still requires using Emscripten to build 'to the web', I think there's going to be a fair drag factor on adoption - it's a fantastic tool and does a great job of compiling existing C/C++ projects today, but I have found it rare for anything nontrivial to 'just work' (this contrasts to Cargo, which mostly does 'just work'). For example, there tons of Emscripten settings you can tweak, including some that magically add other libraries to your include and link path(!), others that are only valid for asm.js and so on.

I have a vague belief that we can do better than this, by deciding on some 'minimal working experience' we want to enable and grabbing just the relevant parts of Emscripten - as long as people can still 'drop down' to emscripten if they want to, nothing has been lost and the happy path becomes very happy. For example, you could just take all the default options in wasm mode, create a stripped down emscripten script and just use that for wasm and asmjs, with zero configurability and the emscripten ports (sdl2 etc) available as Cargo packages.

Of course, talk is cheap and the details are hazy (if you don't have emcc you don't have clang, so you can't compile C and there's no point in having ports).

I personally used to use ./emsdk/emsdk install --build=Release sdk-incoming-64bit to reduce the size to about 2GB. This disables assertions and debug info in the emscripten LLVM so isn't generally recommended, but I'd never seen them trigger and my disk space was fairly limited. Your mileage may vary!

While I never saw the emscripten LLVM fail, emscripten itself most definitely did and I've contributed a number of bugfixes back in the past. Feel free to just raise an issue on the Rust repository with whatever detail/error message you have and if we can figure it out and it's appropriate to forward it on, we can do so.

There are a number of magic environment variables emcc will listen to, but I would be interested to hear what you were trying to do if it comes back to you.

2 Likes
8

We are using wasm to hold the business logic of our front end app, so we don’t actually need DOM interaction in the mid term. The biggest pain point we have is the long compilation times because of the need to use emscripten. We did try https://github.com/WebAssembly/binaryen to compile to WASM, which was a lot faster, and did successfully create a webassembly, but we need to exchange byte arrrays, strings etc. with Javascript and this is not directly supported (or we don’t know how). If someone can figure out a rust way to do the JS wrapper of EMScripten minus the posix stuff (the wrapper does a lot more than we need, e.g. file system abstraction, etc.), and use binaryen to create the wasm that would be fantastic.

9

We've been developing a pretty extensive project using the emscripten target(along with win32, android armv7 + x86, osx and a few other odds and ends).

Overall it's in a pretty good shape and very usable, if you're looking for specifics here's what we've hit.

We've discussed this elsewhere but win32 seems to fall through the cracks on this. Rust 1.20 broke the whole emscripten toolchain(thanks for the quick fixes!) but would love to see win32 as one of the targets here.

When building in debug mode I've seen some really wierd LLVM compiler crashes. I've tried to bake it down to a simple proof of concept but I can't seem to isolate it. Thankfully building release seems to skip the crash but it's a little worrying. If I didn't have a private repo I'd be happy to share the full repro.

+1 on needing to build LLVM. It's pretty painful and also doesn't give a really good "fixed" point to get a stable toolchain from.

Interop with JS hasn't been an issue for us. Using external C functions that call to JS on the page seems to work well.

Overall really glad that the toolchain is an option and happy to expand on any of the points above if it helps.

10

Hello all and thanks for making web targets on Rust work as well as they do! I have spent quite a decent amount of time working with asmjs and wasm targets and I have run into some problems. As someone else mentioned, it’s unclear and not easy to configure emcc. I’ve had to use makefiles/shell scripts to configure emcc.

Also, linking with c libraries (compiled with emcc) is difficult, because it wasn’t clear to me exactly how that should be done. Things like “What optimization level should I use when compiling a c library with emcc to link with a rust program?”

I’ve also had some instances where the compiler simply hangs building to wasm (but not asmjs). No real idea what that’s about and I don’t know how to debug it.

Finally, I’ve noticed some library failures on emscripten targets. For instance, the image crate seems to panic on decoding jpg but pngs work ok. (EDIT: It’s been pointed out to me that this is due to lack of threading support, though it would be nice if it either failed on compile, or fell back to a single threaded version for emscripten targets.)

That said, for the most part, I have been able to accomplish what I want to with rust and web targets, but it does take effort.

11

Finally, I’ve noticed some library failures on emscripten targets. For instance, the image crate seems to panic on decoding jpg but pngs work ok.

The jpeg decoding uses threading, which isn't supported by emscripten.

1 Like
12

I tried it a few months ago for Pathfinder, hit an LLVM assertion, and then gave up and moved the Rust code to the server.

13

I’ve tried emscripten in my project http://ivanceras.github.io/spongedown-editor/ which is running very slow, and very high memory consumption. Rendering the whole markdown text into html which may took only a few hundred milliseconds in rust, may took ~6 seconds in the browser. Older version of firefox (56 and below) will freeze the user’s computer. I been trying to find a way to make the processing faster, rust std::threads doesn’t work on empscripten. Since my app re-render the markdown to html every keystroke, I haven’t found a way to cancel the previous call of the render function since it is synchronous. This might be possible by adding checking of status in between loops whether the function has been cancelled or not, but it will make the code messy.

14

For the record, I put together a small Docker image that contains the emscripten SDK and rustup. The image auto-updates itself at each new version of Rust or emscripten (EDIT: it auto-updates itself on dockerhub, you still have to docker pull the updated version). In my opinion docker is by far the most practical way to compile for emscripten, especially on Windows.

Other than that, I’ve recently had to turn a Rust Vec<f32> into a Javascript Float32Array in order to use the WebAudio API. Right now it’s not possible to inject Javascript at compile-time through rustc (you would have to pass a --pre-js <file> flag to emcc), so my solution has been this unfortunate hack.

11 Likes
15

In a JS implementation of the Ethereum Virtual Machine (https://github.com/ethereumjs/ethereumjs-vm) they’re using ZCash/Parity’s Rust implementation of the Alt-bn-128 curve compiled to ASM.js: https://github.com/ethereumjs/rustbn.js

Seems to work so far! We’ve also had success implementing a simple pure-Rust implementation of the secp256k1 curve and using that as a WASM module.

4 Likes
17

I tried the “Rust to WASM” toolchain a few times in the past year or so and wasn’t too happy about it. Thus, I never really used it for a real project. Main pain points:

Documentation is not very good. This actually surprised me a lot: to me it seemed like WASM was hyped a lot and that the huge web-dev community cared a lot about that. But I often got stuck researching basic things about WASM or Emscripten.

Furthermore, I don’t really like Emscripten. It just seems like a big, hacky blackbox to me which is focused on C and C++ codebases. The author (kripken) does an amazing job advancing the whole eco system around asm.js/wasm, but again, I’m not too happy with Emscripten.

Do you have a main()? Do you compile with default settings? Fine, then it might work without problems. Sure, you get multiple MBs of JS and asm file, but it probably just works. But if you want to do anything else than that (including a library, modify any settings, …), that’s really hard to do (also due to lacking documentation).

I also find it quite annoying to have a huge runtime “linked” to my program every time. If the Rust code is compiled with Emscripten, you can’t even simply write your own runtime. Often, most of the Emscripten runtime magic is not necessary. I’m a lot more interested in writing specific programs for the web (meaning: I won’t use the file system or other desktop stuff) instead of compiling a random old codebase to the web and have it just work™ thanks to Emscripten linker magic™.

A lot of this is of course somewhat due to WASMs immaturity. But it looks like the specification will be extended to solve the most common pain-points.

My dreams :rainbow: for the Rust-WASM future:

  • the codegen is done by the (then) mature LLVM WASM backend
  • the WASM specification is extended to make a few important things easier (use threads, access DOM?, linking without the need for a huge runtime)
  • Rust gets rid of Emscripten and has its own minimal runtime and a lot of documentation on how to modify said runtime as well as how to write the runtime yourself

As a result, in my dreams:

  • compiling to WASM won’t have external linker dependencies (other than the built-in LLVM stuff)
  • the resulting WASM file and the runtime are very small
  • Rust becomes the go-to language for WASM development :heart:
10 Likes
18

It's true that a huge part of the emscripten runtime right now consists in emulating features of other platforms (such as sockets, pthreads, or a filesystem). In the future it would be nice if this layer disappeared.

7 Likes
19

Would this work https://github.com/ReactiveX/RxRust - it’s not ‘functional reactive’ though in the Conalian sense

1 Like
20

@bbatha

Awesome thanks for pointing out the stdweb/cargo-web projects, those are pretty nifty! Mind detailing a bit about what cargo-web is doing? I’m curious if there’s functionality that we may wish to bake directly into Cargo to give it wasm/asm a more “first class” feeling, although starting with a custom subcommand sounds great.

I find the point about general immaturity pretty interesting too. It points that many possible use cases simply aren’t enabled just yet due to lack of APIs, but it’s sort of out of our control. That being said I wouldn’t want that to stop us from providing an as-good-as-it-gets experience today, and that way we can help ensure that most future development on wasm happens in Rust :slight_smile:

Could you also elaborate on some of the implementation details of stdweb? Are there various features we need to add to rustc to get things to work? Maybe work nicer and/or less brittle than they may be right no? I’m not even sure how a js! macro would work!

@jimmycuadra

Heh it seems like there’s a very common trend here of “emscripten is a little hard to acquire”! I’ve certainly felt that myself as I can never remember quite what options to flag. AFAIK there’s still a good bit of upstream development needed to get a native compilation target working, but I’d love to see us head towards that as well!

@toolness

FWIW I think https://github.com/rust-lang/cargo/pull/4570 may help quite a bit with these “extraneous files” problems. I think after that we should no longer have weird hashes in filenames and the filename should be deterministic (I think). There may still be a bug or two to shake out though!

I’m perpetually very curious about how Emscripten works with all this, and it certainly soundsl ike you’ve had quite a rodeo with it. I’m personally interested in seeing if we can provide a much more stripped down experience that’s just the bare essentials needed to get Rust working, but that being said I’m not actually sure what such essentials are! It definitely seems like emscripten has a lot of “magic” baked into it, and certainly starting from scratch would lose a lot of that.

@Herschel

To clarify, do you mean interop in the “how do I do it?” kind or the “this just isn’t possible today” kind? In that if we had documentation of how to do it (I’m not so sure how it’s done myself) would that help? Or does wasm/asmjs not support the level of interop needed today?

For example while Rust can interact with C “easily” it tends to be somewhat difficult to do in a robust fashion. Frameworks exist to make this interop much more smooth (and less unsafe), and I’m curious if that’s the sort of tool needed for JS or if we’re waiting on upstream wasm support.

@aidanhs

Oh awesome! I had no idea you were so used to emscripten :). So one question I’d have for you is do you know what an emscripten-less world would look like? How, for example, would the standard library look like? Would you only have libcore? How do things like allocation/printing work? (I’m not even sure how canvases work…)

@newtack

Thanks for the post! Is your project public? I’d love to poke around the slow compile times myself to see what’s going on. If not, do you know if this is rustc’s fault? Or emscripten’s “fault”?

Also I’m curious, if you’re not using emscripten’s runtime, what’s your logic doing? Does it require any resources from the browser, or is it basically just running computations to generate answers?

I’m also personally curious about the bridge between the browser and Rust. Sort of like C and Rust I’d expect it to be a pretty rough experience (aka not beginner-friendly) today, but I’m curious how it’s working for you!

@vvanders

Heh sorry again about the breakage! I agree it’d be best if we got an emscripten builder on Windows running (to ensure that keeps working). I think to me that’d certainly block a “tier 1” status.

Out of curiosity, have you reported any of the debug mode LLVM crashes? I’m sure emscripten and/or us would be quite interested in investigating!

Also when you say it’s painful to build LLVM, do you mean it’d be best to not have to install the emscripten toolchain itself? In that it’d be best (easiest?) to just have rustup manage all toolchain dependencies?

Also, like with many others I’m asking, have you done much Rust/C interop? If so, was doing it in JS “just as hard”, easier, harder?

@gregwtmtno

I’m curious, what sort of C libraries are you using? It sounds like a lot of this thread’s sentiment is to move away from the emscripten runtime, which would sort of move to a “runtime” with no C compiler, so I’m not even sure how pulling in a project written in C would work!

@tomaka

Awesome thanks for pointing out that docker image! I wonder if we could perhaps integrate that with the cross tool to make testing even easier?

Also I’m curious, could you elaborate a bit on the injecting JS? I’m sort of curious just how this all works at the underlying layer. If you’re emitting an asmjs output, does this mean that you basically inject some JS somewhere that’s something you wrote? (in that the final output js file literally contains the textual content of other stuff you wrote). Or does other sort of “linking” need to happen?

If you’re on wasm, what’s being injected? Is that like a shim with which you bridge the JS/wasm boundary? Binding functions in your crate?

(I fear I may not even be knowledgeable enough to know what are the right questoins to ask!)

@rphmeier

Neat! I forget, was that deployed already or still in development? Additionally, do you have any comments (like many others I’m asking) about the JS/Rust bridge? Is it straightforward enough to do, difficult, could be easier, etc?

@LukasKalbertodt

I’m curious, when you say the documentation isn’t that great, was that how to get Rust and wasm/asmjs up and running? Or was that mostly “once I compiled hello world how do I do real things?” For example I’m personally still not sure how I’d create a web page with a wasm module that passes data to Rust, Rust then queries some JS, and then finally returns an answer to the original query. (although I’m sure it’s probably straightforward, just not sure where to look these things up!)

I also wonder, can you quantify the “hugeness” of the emscripten runtime. I tested the other day and a “hello world” rust program compiled to wasm with optimizations was a 93k wasm file. The equivalent C file, however, was compiled to 12k wasm. Is there a threshold for “too large” to bring in?

Also, do know know what a “minimal Rust runtime” would look like? Is that just libstd but with all I/O things returning errors? (but presumably a working allocator?)

2 Likes
21

Let me give a quick summary.

As you know, the asmjs/emscripten target works very similarly to other targets. Rustc/LLVM turn Rust code into Javascript code, then emcc links this Javascript code to the Emscripten runtime code which is also in Javascript. You also have to possibility to link other Javascript files by passing options to emcc.

This is similar to when you compile for Linux for example, where you turn your Rust code into object files, which are then linked together by ld alongside with the C runtime library. And similarly you also have the possibility to link to other libraries by passing options to the linker.

But there's a problem. Emscripten treats FFI calls as calls to Javascript functions. But not to global Javascript functions, only to Javascript functions that are part of the things you passed to emcc. And this is for a good reason, as the function arguments and return values are not of the same nature. That means that you can't, for example, call the alert() function directly. If you want to do so, you have to write a .js file that contains an intermediary function that turns the string parameter into a Javascript String and calls alert, and pass this file to emcc at compile-time.

Right now Rust doesn't provide a stable way to do this. You cannot inject Javascript code in your binary. You can maybe by using linker args or the asm! macro, but I didn't try. Technically it could be possible to add some sort of macro or attribute or something that passes the right parameters to emcc, but that doesn't exist yet and is maybe not desirable.

So in order to bypass this problem you have to use the emscripten_run_script function provided by Emscripten, that takes as parameter a script and calls eval() on it.

If you’re on wasm, what’s being injected? Is that like a shim with which you bridge the JS/wasm boundary? Binding functions in your crate?

On wasm, things are blurry because there's no interface to "external" things like alert yet. There's a mechanism for interfacing between Javascript and webassembly, but it has to be done manually.

1 Like