Need help with emscripten port

[jer]

So I took @shepmaster’s changes, applied them to a recent Rust checkout, checked out upstream master for llvm and compiler-rt and was able to compile stage 1 (full build is currently running).

@tomaka: I’ll take a look at that when I have the LLVM thing running.

[brson]

So I took @shepmaster’s changes, applied them to a recent Rust checkout, checked out upstream master for llvm and compiler-rt and was able to compile stage 1 (full build is currently running).

Beautiful! Thanks for tackling this.

[jer]

Current status: I got a full succesfull build, but make check fails in 4 cases (3 times LTO, where it can’t find main, it might be optimized out?). I really only took the existing changes from shepmaster and applied them, code lives here: https://github.com/badboy/rust/tree/llvm-upgrade

I hope to get around to clean it up next.

[brson]

@jer Awesome! Only 4 failures is really encouraging. @kripken is about to start forward porting emscripten’s port to the same merge-base as yours.

[jer]

Down to one actually, because I partly understood what I’m doing now. @alexcrichton said I should submit this as a PR, so it gets seen by a wider audience to help out fixing last few bits. I will get around to do this this week.

[brson]

@jer has submitted the LLVM upgrade PR. @habnabit did some cleanup on my emscripten branch of Rust. It looks though like emscripten hasn’t yet updated the incoming branch of their fastcomp LLVM fork.

Next steps are to merge the LLVM upgrade, get @kripken to finish upgrading fastcomp, upgrade my ‘empscripten’ branch of Rust, port our LLVM patches onto fastcomp and submit that as a Rust PR, submit patches to turn on emscripten support in Rust, set up automation to keep emscripten target building.

[alexcrichton]

The LLVM upgrade has now landed

[brson]

Update: The Rust LLVM upgrade finally landed (thanks @jer and everyone who helped - it was a brutal slog).

Next @kripken is going to upgrade the emscripten incoming branch to the same merge base. But we don’t need to wait for that probably. He’s already prepared a rebase that should be in the ballpark of the same commit: the fastcomp next-merge branch.

So our next step is to take all of their patches, apply them to our tree and merge that into Rust.

Unfortunately, there’s a gotcha - @kripken didn’t realize that our plan was to apply the emscripten LLVM fork to all of our targets and he’s a bit worried about it. The reason is that the fastcomp branch is only tested for the emscripten target, and it’s based on the old pnacl fork, which has a bunch of ancillary patches of unknown value. So we may run into unanticipated problems (and in fact last time we had this working the x86_64 backend miscompiled things).

Furthermore, the fastcomp history is lengthy and messy, with all that pnacl legacy. So we are going to need to scrutinize it carefully and see e.g. if it’s doing anything to the x86 backend. We might, e.g just squash their entire history into one ‘emscripten’ commit, then revert large chunks of it. It could be a big mess. (We also have the option of pursuing a strategy where rustc can optionally use an emscripten-specific LLVM).

It’s also worth noting that the upstream LLVM wasm backend is still in progress and the last I heard the wasm team is trying to have it ready for the wasm launch late this year. So the longer this process goes on the more likely we are to just end up on the upstream wasm backend and skip the asm.js backend completely. Still, I think it’s worth continuing on the path we’re on and seeing how far we can get.

[brson]

OK, another tantalizing avenue of attack here.

The LLVM wasm backend, while it isn’t fully working, and there are still changes to the wasm spec incoming, works to a relatively large degree, and Rust has it in tree now.

It would be very interesting to see what happens if we turn it on, start laying the groundwork for a wasm-unknown-emscripten target.

Emscripten already has experimental support, documented here. Might be some useful leads there.

[brson]

Another idea: even though the road to getting working support for emscripten in-tree is still a bit off on the horizon, we’ve already proven it works, and it would be relatively easy to get it working out of tree again. With a working build it would be incredibly valuable for somebody to sprint ahead and begin implementing a webapi-rs or dom-rs crate that exposes web API bindings to Rust. If we do this work in parallel then by the time we are ready to produce a wasm target we will not only have something that runs on the web, but something ready to challenge JavaScript directly. We’ll be way ahead of everyone.

[tomaka]

How does WebAssembly communicate with the “environment” (like canvas, WebAudio, fullscreen, etc.)? I imagine all the interfaces that the W3C defines (like this or this for example) will be accessible, but I can’t find any document that describes how.

[jer]

That was my plan indeed. There’s some old code around I want to re-use for the dom-rs thingy. I will work on compiling emscripten out-of-tree again in the coming days. I will post instructions once I have a successful build.

[brson]

@tomaka wasm v1 itself does not provide any facilities for calling web APIs, but emscripten does and will. Here’s the user facing documentation on it. Doing the same thing in Rust will require some digging into how the mechanism works under the hood.

The basic idea is to compile down strings of javascript into LLVM IR, following some pattern that emscripten recognizes. emscripten will then emit these as plain javascript and add them to the asm.js import list. I don’t know exactly how this will translate to wasm, but presumably emscripten will have a similar facility for that format as well.

[lqd]

The focus seems to be on computation at the beginning, so it “doesn’t” a whole lot right now, in the sense that interacting with existing DOM APIs directly doesn’t seem to be planned in the MVP, nor soon after (PostMVP) but probably after that: https://github.com/WebAssembly/design/blob/master/GC.md However, the browser environment and JS — and this already does work right now — can make javascript functions accessible to wasm modules. Modules can import functions and call them, like you’d import a function from another wasm module. For now, it seems the “only way” would be to have the wasm modules import some JS glue to the web APIs you want to interact with, (IIRC another planned thing is being able to export the wasm memory to be accessible to JS as a Uint8Array. I think it’ll be so you can interact with the wasm with more flexiblity than calling fns with just 4 numeric types) until one can interact with the real APIs directly and remove that glue code and overhead.

[tomaka]

For now, it seems the “only way” would be to have the wasm modules import some JS glue to the web APIs you want to interact with

That’s interesting. I wonder how we could integrate such a JS glue file in Rust/Cargo’s build system. Everything will probably remain a bit blurry until we have some sort of prototype.

[brson]

That’s interesting. I wonder how we could integrate such a JS glue in Rust/Cargo’s build system.

I think this is the same as the approach I linked previously. You treat it like web-specific assembly code and emscripten packages it up for you.

[lqd]

@brson yes probably, maybe some of it can even be generated by emscripten (was it the WebIDL binder maybe?)

[brson]

@lqd @tomaka @jer It was pointed out to me that the obvious way to generate DOM bindings is to translate from WebIDL. And in Rust we already have a sophisticated bindings generator, from Servo!

So it seems to me the obvious thing to do, and the most maintainable long term, would be to piggyback on Servo’s bindings generator so we get the benefit of their ongoing work. I’m sure that today it’s pretty tightly coupled to Servo internals, so it could be quite a task getting started. One might be able to imagine creating an adapter layer that basically implements Servo’s API on top of emscripten.

@kripken has some opinions about the right way to do this.

[brson]

OK, just one more idea.

@kripken just reminded me that one of the early ways we can promote Rust on the web is the exact same way we promote Rust for other software stacks - write a high-performance module in Rust and call it from JS. This plays to our strengths for all the usual reasons, doesn’t require solving DOM bindings. And we’ve got quite a few crates these days that are approaching world-class performance for their domains.

So another tactic we might take to bring the thunder for the wasm launch is to prepare several of these projects to be packaged as reusable JavaScript modules, along with a demo that we can show the world and blog about. Anybody have ideas about what modules we can provide that JavaScript programmers will care about?

The technical obstacle to this is that we’ll need a way to usefully represent some set of Rust types to JavaScript clients, and that will require experimentation. It will almost certainly involve typed arrays.

I have several high-priority tasks on my plate still for the immediate future, but I hope I can turn the corner on those and pitch in technically myself.

[jer]

With just a few changes to Rust and a freshly compiled emscripten-fastcomp and emscripten-fastcomp-clang (from their next-merge branches) I have a stage1 compiler producing more or less working JavaScript now (stage2 running as I write).

I cheated and copy’n’pasted a definition, but unless it fails, it works. I will write down simple instructions now. If anyone can help with eliminating that hack in libpanic_unwind that would be appreciated. Is there a way to disable it completely and rely on libpanic_abort completely?

I will go through @tomaka’s old PR and see which changes from there we need.