There we go, a simple 20-step recipe to build an up-to-date Rust with Emscripten support: https://gist.github.com/badboy/2086757d09b7019e9f4ec8e98ee17054
Someone had already done exactly the same thing a few months ago: http://ashleysommer.com.au/how-to/articles/asm-js-code-using-rust-and-emscripten
Eventually the instructions became outdated. Your Rust fork will unfortunately become outdated as well in a few weeks. Ideally we should avoid repeating this circle again.
D’oh! Didn’t see that at all.
I’d like to get as much of the changes into upstream. Once
next-merge becomes the incoming/master branch for emscripten, the whole setup will be easier again.
If we have that we might get to a point where we can provide an external LLVM to Rust and it will be able to enable Emscripten on the fly.
Update: AFAIK next-merge has been merged into the incoming branch in emscripten. Getting closer!
That will make my 20-step guide a little bit shorter as now a
emsdk install sdk-incoming-64bit && emsdk activate sdk-incoming-64bit should do.
So I’ve never really looked under the hood of Rust before, but I think having an emscripten compile target would be really awesome. Could anyone who knows the process take a minute lay out in broad strokes the next steps that need to be taken in order to eventually see an asmjs-unknown-emscripten target in rustup? I would love to help where I can!
The critical step in getting asmjs support is merging emscripten’s LLVM fork into Rust’s LLVM fork. Here’s how I suggest we do it:
- Take all of emscripten’s patches against LLVM and squash them into one. We don’t particularly care about the details of them, and I think it’ll be easier to manage if the history just says ‘emscripten patches’.
- Cherry-pick that patch onto our fork.
- Run the test suite on x86_64 and see if it still passes. It almost certainly won’t because issues mentioned previously.
- Make another patch that reverts whatever is breaking the x86/x86_64 backend in emscripten’s patches. Once we’ve got this figured out, we’ll have this patch in our pockets to re-apply every time we upgrade LLVM.
- Submit the combined LLVM branch to Rust’s LLVM fork.
- Submit a patch to Rust with the new LLVM branch. Once that merges successfully we’re pretty much home free.
At that point we need to fix any remaining bugs that prevent std from building for asmjs. Once we can build a std for asmjs we can begin publishing nightlies. Then all that’s lift is rooting out test suite failures and we’re off to the races.
In the name of exploring all the options - would it be possible to package the emscripten version of llvm only in the asmjs toolchain? If possible, this seems like it would be a more flexible option going forward, as it decouples rust asmjs work from all the other targets, and avoids the x86 problems you mention. It would also (theoretically) open the way for Rust to use other forks of llvm for other backends in the future. Then again, maybe this is an even bigger hassle than the patch-and-revert process you describe, or its undesirable for other reasons.
That step is easy, maybe 20 lines of code to fix.
You mean the various
#[cfg(not(target_os="emscripten")))] things? I have yours rebased. I can send them upstream later today.
I also re-added most of your
// ignore-emscripten to tests (but still have a lot others failing). The ones I already have can go upstream.
I’m not sure about your fix to libtest. It was mentioned this should be done by conditional compiling. That shouldn’t be too hard.
I can already compile libstd. With the fixes above they should be mostly stable as well, just not passing the test suite.
It is technically possible, but there are some bad tradeoffs. The way things exist right now we would have to define an artificial host triple for every host platform that wants to compile to asmjs, like
x86_64-foremscripten-linux-gnu which would give you a compiler that runs on x86_64 linux and only targets emscripten. We would need one of these for every host platform that wants to target emscripten (so in the limit all of them). Quite ugly and I’d only consider it a temporary solution.
One could also imagine converting rustc to load LLVM dynamically, producing LLVM packages, and adding a mechanism to switch between them. Quite a bit of work.
@jer yep. Thanks!
Here’s what I’ve done so far:
- Squashed all the fastcomp LLVM commits and cherry-picked them onto our LLVM revision
- Rebased @jer’s branch onto Rust master and added a commit to make the test suite work under rustbuild.
If you look at the fastcomp squash you’ll see that almost all the code is in two asmjs-specific subsystems: the pnacl legalizer, and the asmjs backend. There’s only a small amount of common code, minor fixes and optimizations. The whole patch applies cleanly to our branch. So that gives me a lot of hope that we can use a single LLVM build for all existing targets + emscripten, and that it can be maintained, at least until the wasm backend can replace it.
So I think the way is clear to get this landed and get builds out.
- ignore-emscripten all the currently-failing tests.
- Merge the fastcomp squash into the rust llvm fork. It may be worth pushing this to a temporary branch on the assumption that it’s not going to land easily in Rust, but I actually think this will go pretty smoothly.
- Submit the PR to Rust that adds asmjs support.
- Add an auto- builder that builds the asmjs target. We probably won’t start testing it right away because testing this target is quite slow.
- Add a dist- builder to create the asmjs target packages.
Those steps have to happen sequentially and will result in in-tree support and release builds for asmjs (not wasm). I will today start as a background task finding and ignoring run-pass tests, but doing so for the full test suite will take time. If anybody wants to help please say so and jump in. This needs to happen fast.
python src/bootstrap/bootstrap.py --step check-rpass --target=asmjs-unknown-emscripten
In parallel, and starting right now, we can do the following:
- Begin fixing the failing tests on emscripten
- Add the
wasm32-unknown-emscriptentarget to rustc. It will be almost the same as the asmjs target. For the initial implementation we will not use the wasm llvm backend, but instead use the asmjs backend and ask emcc to convert the asmjs IR to wasm. Thusly we will have wasm support.
Once the in-tree support has landed we can create a quest issue for fixing the broken emscripten tests, documenting how to run the tests and itemizing every one into its own bug, stir up community support to get everything working.
If anybody is interested in helping with any of these steps, please speak up; help is needed as always.
This is it. It’s on.
When I worked on that a few months ago, I already submitted PRs that fixed all the legitimate test failures. Unless new bugs have been introduced since then, all the tests that are failing are either because they use threads or because they spawn a new process.
@tomaka I haven’t investigated the causes yet, but there are many some number of test failures now, not sure how many. They need to be dealt with in some fashion in order to land.
I just let
check-rpass run with @brson’s branch, these are the test cases failing: https://gist.github.com/badboy/447b30da7ba1fa47882374f2c4394af3
(I add the full failure log later)
Looks to me like most of the failures have to do with unwinding infrastructure, with backtraces like:
/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/extern-pass-u32.stage2-asmjs-unknown-emscripten.js:110 throw ex; ^ abort(-1) at Error at jsStackTrace (/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/extern-pass-u32.stage2-asmjs-unknown-emscripten.js:1092:13) at stackTrace (/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/extern-pass-u32.stage2-asmjs-unknown-emscripten.js:1109:12) at abort (/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/extern-pass-u32.stage2-asmjs-unknown-emscripten.js:26641:44) at _rust_dbg_extern_identity_u32 (/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/extern-pass-u32.stage2-asmjs-unknown-emscripten.js:1612:73) at Array.__ZN15extern_pass_u324main17h0ac086c698e22245E (/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/extern-pass-u32.stage2-asmjs-unknown-emscripten.js:5517:8) at Array.__ZN3std9panicking3try7do_call17h5abeb5e2a7efe99fE (/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/extern-pass-u32.stage2-asmjs-unknown-emscripten.js:14445:43) at Object.dynCall_vi (/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/extern-pass-u32.stage2-asmjs-unknown-emscripten.js:26206:31) at invoke_vi (/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/extern-pass-u32.stage2-asmjs-unknown-emscripten.js:5269:25) at ___rust_maybe_catch_panic (/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/extern-pass-u32.stage2-asmjs-unknown-emscripten.js:14943:2) at __ZN3std2rt10lang_start17h08beb9a157543f15E (/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/extern-pass-u32.stage2-asmjs-unknown-emscripten.js:14211:15) ------------------------------------------ thread '[run-pass] run-pass/extern-pass-u32.rs' panicked at 'explicit panic', src/tools/compiletest/src/runtest.rs:2356 note: Run with `RUST_BACKTRACE=1` for a backtrace.
Probably prudent to investigate the “Patch panic_unwind to compile, but this is surely broken” patch.
Though, actually I can’t imagine offhand why the rust_dbg_extern_identity_u32 function would be calling abort. Maybe this doesn’t have to do with unwinding and we’re miscompiling that C file.
When I look at
Edit: I see the fix for this issue needs to go into gcc-rs. Will update a patch.
Here’s the patch to gcc-rs. Once that lands I rustbuild should pick it up and a lot of tests should start to pass.
Edit: Nope, I had to update the lockfile. I’ve done so and pushed a commit to update gcc-rs.
So now I’m going to focus again on ignore-emscriptening tests.