Need help with emscripten port

@tomaka wasm v1 itself does not provide any facilities for calling web APIs, but emscripten does and will. Here’s the user facing documentation on it. Doing the same thing in Rust will require some digging into how the mechanism works under the hood.

The basic idea is to compile down strings of javascript into LLVM IR, following some pattern that emscripten recognizes. emscripten will then emit these as plain javascript and add them to the asm.js import list. I don’t know exactly how this will translate to wasm, but presumably emscripten will have a similar facility for that format as well.

The focus seems to be on computation at the beginning, so it “doesn’t” a whole lot right now, in the sense that interacting with existing DOM APIs directly doesn’t seem to be planned in the MVP, nor soon after (PostMVP) but probably after that: https://github.com/WebAssembly/design/blob/master/GC.md However, the browser environment and JS — and this already does work right now — can make javascript functions accessible to wasm modules. Modules can import functions and call them, like you’d import a function from another wasm module. For now, it seems the “only way” would be to have the wasm modules import some JS glue to the web APIs you want to interact with, (IIRC another planned thing is being able to export the wasm memory to be accessible to JS as a Uint8Array. I think it’ll be so you can interact with the wasm with more flexiblity than calling fns with just 4 numeric types) until one can interact with the real APIs directly and remove that glue code and overhead.

That's interesting. I wonder how we could integrate such a JS glue file in Rust/Cargo's build system. Everything will probably remain a bit blurry until we have some sort of prototype.

I think this is the same as the approach I linked previously. You treat it like web-specific assembly code and emscripten packages it up for you.

@brson yes probably, maybe some of it can even be generated by emscripten (was it the WebIDL binder maybe?)

@lqd @tomaka @jer It was pointed out to me that the obvious way to generate DOM bindings is to translate from WebIDL. And in Rust we already have a sophisticated bindings generator, from Servo!

So it seems to me the obvious thing to do, and the most maintainable long term, would be to piggyback on Servo’s bindings generator so we get the benefit of their ongoing work. I’m sure that today it’s pretty tightly coupled to Servo internals, so it could be quite a task getting started. One might be able to imagine creating an adapter layer that basically implements Servo’s API on top of emscripten.

@kripken has some opinions about the right way to do this.

OK, just one more idea.

@kripken just reminded me that one of the early ways we can promote Rust on the web is the exact same way we promote Rust for other software stacks - write a high-performance module in Rust and call it from JS. This plays to our strengths for all the usual reasons, doesn’t require solving DOM bindings. And we’ve got quite a few crates these days that are approaching world-class performance for their domains.

So another tactic we might take to bring the thunder for the wasm launch is to prepare several of these projects to be packaged as reusable JavaScript modules, along with a demo that we can show the world and blog about. Anybody have ideas about what modules we can provide that JavaScript programmers will care about?

The technical obstacle to this is that we’ll need a way to usefully represent some set of Rust types to JavaScript clients, and that will require experimentation. It will almost certainly involve typed arrays.

I have several high-priority tasks on my plate still for the immediate future, but I hope I can turn the corner on those and pitch in technically myself.

2 Likes

With just a few changes to Rust and a freshly compiled emscripten-fastcomp and emscripten-fastcomp-clang (from their next-merge branches) I have a stage1 compiler producing more or less working JavaScript now (stage2 running as I write).

I cheated and copy’n’pasted a definition, but unless it fails, it works. I will write down simple instructions now. If anyone can help with eliminating that hack in libpanic_unwind that would be appreciated. Is there a way to disable it completely and rely on libpanic_abort completely?

I will go through @tomaka’s old PR and see which changes from there we need.

3 Likes

There we go, a simple 20-step recipe to build an up-to-date Rust with Emscripten support: https://gist.github.com/badboy/2086757d09b7019e9f4ec8e98ee17054

2 Likes

Someone had already done exactly the same thing a few months ago: http://ashleysommer.com.au/how-to/articles/asm-js-code-using-rust-and-emscripten

Eventually the instructions became outdated. Your Rust fork will unfortunately become outdated as well in a few weeks. Ideally we should avoid repeating this circle again.

D’oh! Didn’t see that at all. I’d like to get as much of the changes into upstream. Once next-merge becomes the incoming/master branch for emscripten, the whole setup will be easier again.

If we have that we might get to a point where we can provide an external LLVM to Rust and it will be able to enable Emscripten on the fly.

Update: AFAIK next-merge has been merged into the incoming branch in emscripten. Getting closer!

It has indeed: https://github.com/kripken/emscripten-fastcomp/commit/7edfb846787427c23eea4357a68d710f2d09bead

That will make my 20-step guide a little bit shorter as now a emsdk install sdk-incoming-64bit && emsdk activate sdk-incoming-64bit should do.

So I’ve never really looked under the hood of Rust before, but I think having an emscripten compile target would be really awesome. Could anyone who knows the process take a minute lay out in broad strokes the next steps that need to be taken in order to eventually see an asmjs-unknown-emscripten target in rustup? I would love to help where I can!

The critical step in getting asmjs support is merging emscripten’s LLVM fork into Rust’s LLVM fork. Here’s how I suggest we do it:

  • Take all of emscripten’s patches against LLVM and squash them into one. We don’t particularly care about the details of them, and I think it’ll be easier to manage if the history just says ‘emscripten patches’.
  • Cherry-pick that patch onto our fork.
  • Run the test suite on x86_64 and see if it still passes. It almost certainly won’t because issues mentioned previously.
  • Make another patch that reverts whatever is breaking the x86/x86_64 backend in emscripten’s patches. Once we’ve got this figured out, we’ll have this patch in our pockets to re-apply every time we upgrade LLVM.
  • Submit the combined LLVM branch to Rust’s LLVM fork.
  • Submit a patch to Rust with the new LLVM branch. Once that merges successfully we’re pretty much home free.

At that point we need to fix any remaining bugs that prevent std from building for asmjs. Once we can build a std for asmjs we can begin publishing nightlies. Then all that’s lift is rooting out test suite failures and we’re off to the races.

1 Like

awesome!

In the name of exploring all the options - would it be possible to package the emscripten version of llvm only in the asmjs toolchain? If possible, this seems like it would be a more flexible option going forward, as it decouples rust asmjs work from all the other targets, and avoids the x86 problems you mention. It would also (theoretically) open the way for Rust to use other forks of llvm for other backends in the future. Then again, maybe this is an even bigger hassle than the patch-and-revert process you describe, or its undesirable for other reasons.

1 Like

That step is easy, maybe 20 lines of code to fix.

You mean the various #[cfg(not(target_os="emscripten")))] things? I have yours rebased. I can send them upstream later today. I also re-added most of your // ignore-emscripten to tests (but still have a lot others failing). The ones I already have can go upstream. I’m not sure about your fix to libtest. It was mentioned this should be done by conditional compiling. That shouldn’t be too hard.

I can already compile libstd. With the fixes above they should be mostly stable as well, just not passing the test suite.

Added the test changes from your old PR here: https://github.com/rust-lang/rust/pull/35574

It is technically possible, but there are some bad tradeoffs. The way things exist right now we would have to define an artificial host triple for every host platform that wants to compile to asmjs, like x86_64-foremscripten-linux-gnu which would give you a compiler that runs on x86_64 linux and only targets emscripten. We would need one of these for every host platform that wants to target emscripten (so in the limit all of them). Quite ugly and I'd only consider it a temporary solution.

One could also imagine converting rustc to load LLVM dynamically, producing LLVM packages, and adding a mechanism to switch between them. Quite a bit of work.

@jer yep. Thanks!