@lqd@tomaka@jer It was pointed out to me that the obvious way to generate DOM bindings is to translate from WebIDL. And in Rust we already have a sophisticated bindings generator, from Servo!
So it seems to me the obvious thing to do, and the most maintainable long term, would be to piggyback on Servo’s bindings generator so we get the benefit of their ongoing work. I’m sure that today it’s pretty tightly coupled to Servo internals, so it could be quite a task getting started. One might be able to imagine creating an adapter layer that basically implements Servo’s API on top of emscripten.
@kripken has some opinions about the right way to do this.
@kripken just reminded me that one of the early ways we can promote Rust on the web is the exact same way we promote Rust for other software stacks - write a high-performance module in Rust and call it from JS. This plays to our strengths for all the usual reasons, doesn’t require solving DOM bindings. And we’ve got quite a few crates these days that are approaching world-class performance for their domains.
So another tactic we might take to bring the thunder for the wasm launch is to prepare several of these projects to be packaged as reusable JavaScript modules, along with a demo that we can show the world and blog about. Anybody have ideas about what modules we can provide that JavaScript programmers will care about?
The technical obstacle to this is that we’ll need a way to usefully represent some set of Rust types to JavaScript clients, and that will require experimentation. It will almost certainly involve typed arrays.
I have several high-priority tasks on my plate still for the immediate future, but I hope I can turn the corner on those and pitch in technically myself.
With just a few changes to Rust and a freshly compiled emscripten-fastcomp and emscripten-fastcomp-clang (from their next-merge branches) I have a stage1 compiler producing more or less working JavaScript now (stage2 running as I write).
I cheated and copy’n’pasted a definition, but unless it fails, it works. I will write down simple instructions now.
If anyone can help with eliminating that hack in libpanic_unwind that would be appreciated. Is there a way to disable it completely and rely on libpanic_abort completely?
I will go through @tomaka’s old PR and see which changes from there we need.
Eventually the instructions became outdated. Your Rust fork will unfortunately become outdated as well in a few weeks.
Ideally we should avoid repeating this circle again.
D’oh! Didn’t see that at all.
I’d like to get as much of the changes into upstream. Once next-merge becomes the incoming/master branch for emscripten, the whole setup will be easier again.
If we have that we might get to a point where we can provide an external LLVM to Rust and it will be able to enable Emscripten on the fly.
So I’ve never really looked under the hood of Rust before, but I think having an emscripten compile target would be really awesome. Could anyone who knows the process take a minute lay out in broad strokes the next steps that need to be taken in order to eventually see an asmjs-unknown-emscripten target in rustup? I would love to help where I can!
The critical step in getting asmjs support is merging emscripten’s LLVM fork into Rust’s LLVM fork. Here’s how I suggest we do it:
Take all of emscripten’s patches against LLVM and squash them into one. We don’t particularly care about the details of them, and I think it’ll be easier to manage if the history just says ‘emscripten patches’.
Cherry-pick that patch onto our fork.
Run the test suite on x86_64 and see if it still passes. It almost certainly won’t because issues mentioned previously.
Make another patch that reverts whatever is breaking the x86/x86_64 backend in emscripten’s patches. Once we’ve got this figured out, we’ll have this patch in our pockets to re-apply every time we upgrade LLVM.
Submit the combined LLVM branch to Rust’s LLVM fork.
Submit a patch to Rust with the new LLVM branch. Once that merges successfully we’re pretty much home free.
At that point we need to fix any remaining bugs that prevent std from building for asmjs. Once we can build a std for asmjs we can begin publishing nightlies. Then all that’s lift is rooting out test suite failures and we’re off to the races.
In the name of exploring all the options - would it be possible to package the emscripten version of llvm only in the asmjs toolchain? If possible, this seems like it would be a more flexible option going forward, as it decouples rust asmjs work from all the other targets, and avoids the x86 problems you mention. It would also (theoretically) open the way for Rust to use other forks of llvm for other backends in the future. Then again, maybe this is an even bigger hassle than the patch-and-revert process you describe, or its undesirable for other reasons.
You mean the various #[cfg(not(target_os="emscripten")))] things? I have yours rebased. I can send them upstream later today.
I also re-added most of your // ignore-emscripten to tests (but still have a lot others failing). The ones I already have can go upstream.
I’m not sure about your fix to libtest. It was mentioned this should be done by conditional compiling. That shouldn’t be too hard.
I can already compile libstd. With the fixes above they should be mostly stable as well, just not passing the test suite.
It is technically possible, but there are some bad tradeoffs. The way things exist right now we would have to define an artificial host triple for every host platform that wants to compile to asmjs, like x86_64-foremscripten-linux-gnu which would give you a compiler that runs on x86_64 linux and only targets emscripten. We would need one of these for every host platform that wants to target emscripten (so in the limit all of them). Quite ugly and I'd only consider it a temporary solution.
One could also imagine converting rustc to load LLVM dynamically, producing LLVM packages, and adding a mechanism to switch between them. Quite a bit of work.
Based on @jer’s instructions I’ve started poking at the emscripten port again, and I’m very encouraged that we can get this across the finish line soon.
If you look at the fastcomp squash you’ll see that almost all the code is in two asmjs-specific subsystems: the pnacl legalizer, and the asmjs backend. There’s only a small amount of common code, minor fixes and optimizations. The whole patch applies cleanly to our branch. So that gives me a lot of hope that we can use a single LLVM build for all existing targets + emscripten, and that it can be maintained, at least until the wasm backend can replace it.
So I think the way is clear to get this landed and get builds out.
Next steps:
ignore-emscripten all the currently-failing tests.
Merge the fastcomp squash into the rust llvm fork. It may be worth pushing this to a temporary branch on the assumption that it’s not going to land easily in Rust, but I actually think this will go pretty smoothly.
Submit the PR to Rust that adds asmjs support.
Add an auto- builder that builds the asmjs target. We probably won’t start testing it right away because testing this target is quite slow.
Add a dist- builder to create the asmjs target packages.
Those steps have to happen sequentially and will result in in-tree support and release builds for asmjs (not wasm). I will today start as a background task finding and ignoring run-pass tests, but doing so for the full test suite will take time. If anybody wants to help please say so and jump in. This needs to happen fast.
You can get a working build by following @jer’s previously-linked instructions but substituting the repo’s I linked above, (sorry I haven’t submitted PR’s to @jer yet), and running tests with e.g.
In parallel, and starting right now, we can do the following:
Begin fixing the failing tests on emscripten
Add the wasm32-unknown-emscripten target to rustc. It will be almost the same as the asmjs target. For the initial implementation we will not use the wasm llvm backend, but instead use the asmjs backend and ask emcc to convert the asmjs IR to wasm. Thusly we will have wasm support.
Once the in-tree support has landed we can create a quest issue for fixing the broken emscripten tests, documenting how to run the tests and itemizing every one into its own bug, stir up community support to get everything working.
If anybody is interested in helping with any of these steps, please speak up; help is needed as always.
When I worked on that a few months ago, I already submitted PRs that fixed all the legitimate test failures.
Unless new bugs have been introduced since then, all the tests that are failing are either because they use threads or because they spawn a new process.
@tomaka I haven’t investigated the causes yet, but there are many some number of test failures now, not sure how many. They need to be dealt with in some fashion in order to land.
Looks to me like most of the failures have to do with unwinding infrastructure, with backtraces like:
/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/extern-pass-u32.stage2-asmjs-unknown-emscripten.js:110
throw ex;
^
abort(-1) at Error
at jsStackTrace (/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/extern-pass-u32.stage2-asmjs-unknown-emscripten.js:1092:13)
at stackTrace (/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/extern-pass-u32.stage2-asmjs-unknown-emscripten.js:1109:12)
at abort (/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/extern-pass-u32.stage2-asmjs-unknown-emscripten.js:26641:44)
at _rust_dbg_extern_identity_u32 (/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/extern-pass-u32.stage2-asmjs-unknown-emscripten.js:1612:73)
at Array.__ZN15extern_pass_u324main17h0ac086c698e22245E (/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/extern-pass-u32.stage2-asmjs-unknown-emscripten.js:5517:8)
at Array.__ZN3std9panicking3try7do_call17h5abeb5e2a7efe99fE (/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/extern-pass-u32.stage2-asmjs-unknown-emscripten.js:14445:43)
at Object.dynCall_vi (/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/extern-pass-u32.stage2-asmjs-unknown-emscripten.js:26206:31)
at invoke_vi (/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/extern-pass-u32.stage2-asmjs-unknown-emscripten.js:5269:25)
at ___rust_maybe_catch_panic (/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/extern-pass-u32.stage2-asmjs-unknown-emscripten.js:14943:2)
at __ZN3std2rt10lang_start17h08beb9a157543f15E (/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/extern-pass-u32.stage2-asmjs-unknown-emscripten.js:14211:15)
------------------------------------------
thread '[run-pass] run-pass/extern-pass-u32.rs' panicked at 'explicit panic', src/tools/compiletest/src/runtest.rs:2356
note: Run with `RUST_BACKTRACE=1` for a backtrace.
Probably prudent to investigate the “Patch panic_unwind to compile, but this is surely broken” patch.
Though, actually I can’t imagine offhand why the rust_dbg_extern_identity_u32 function would be calling abort. Maybe this doesn’t have to do with unwinding and we’re miscompiling that C file.