Need help with emscripten port

1

I’ve been working on an emscripten port since December and have finally landed most of it. The in-tree work, when paired with an LLVM upgrade to the emscripten-fastcomp/incoming LLVM branch and the sdk-incoming-64bit emscripten sdk is enough to get hello.rs and 2000 run-pass tests working. My emscripten branch of Rust, contains the LLVM patches.

Unfortunately, I don’t have a great deal of time to continue working on this right now, so I’m interested in help. Fortunately, the immediate way ahead is clear: run the test suite in my branch against the asmjs-unknown-emscripten target, fix them, and submit patches (either to rust-lang or my fork).

Here’s how to do it, more or less:

$ curl -O https://s3.amazonaws.com/mozilla-games/emscripten/releases/emsdk-portable.tar.gz
$ tar -xzf emsdk-portable.tar.gz
$ source emsdk_portalbl/emsdk_env.h
$ emsdk update
$ emsdk install sdk-incoming-64bit
$ emsdk activate sdk-incoming-64bit
$ git clone git://github.com/brson/rust
$ (cd rust/src/llvm && git remote add kripken https://github.com/kripken/emscripten-fastcomp && git fetch kripken)
$ cd rust
$ git reset origin/emscripten --hard
$ ./configure --target=asmjs-unknown-emscripten
$ make check-stage2-T-asmjs-unknown-emscripten-H-x86_64-unknown-linux-gnu-rpass-exec

That will run just the rpass tests, crossing from 64-bit linux to asmjs. There are other sets of tests to run as well, but rpass are the biggest.

19 Likes
2

It looks like your branch isn’t quite consistent (or something didn’t get pushed) in the LLVM submodule:

$ git submodule update
fatal: reference is not a tree: 6e11a1442a65c6dba432731cabe2394419c0955a
Unable to checkout '6e11a1442a65c6dba432731cabe2394419c0955a' in submodule path 'src/llvm'
3

Ooh sorry. You’re right. The LLVM commit is from kripken/emscripten-fastcomp. I didn’t update the remote. So to get it to work one needs to fetch the proper remote.

Edit: Updated OP w/ directions for fetching fastcomp

4

Here’s how I see the path forward for Rust on emscripten:

Fix the test suite and upstream those patches to rust-lang/rust.

Rebase Rust’s LLVM on top of emscripten’s. I believe emscripten will be willing to forward port their incoming LLVM when we do our nightlies; they don’t want to block us. They carry a lot of patches that will never be upstreamed, so this would be a big commitment, though we won’t have to carry emscripten patches forever. Eventually the upstream LLVM wasm backend will be ready and we can convert from wasm to asmjs, drop the asmjs LLVM backend.

Come up with a strategy for pairing emscripten SDKs with Rust builds. Both projects must be on the exact same version of LLVM to interoperate, so we need to be able to communicate exactly which emscripten to pair it with, and a way to obtain the paired emscripten SDK. This will integrate with rustup.rs to automatically install the SDK.

Get popular crates working and show cool demos.

1 Like
5

I compiled Rust and got a “hello world” working.

However things got messy when I tried to port glutin to it. For the record, I had tried your emscripten branch back in december, and glutin was working just fine. So all the problems here are either new or due to the fact that I built in release mode (most likely the latter). EDIT: I built rustc in release mode, not glutin.

Part of the code of glutin requires calling functions from the emscripten SDK. One of these functions is emscripten_GetProcAddress, defined like this:

extern {
    pub fn emscripten_GetProcAddress(name: *const libc::c_char) -> *const libc::c_void;
}

And I call the function like this:

fn get_proc_address(&self, addr: &str) -> *const () {
    let addr = CString::new(addr.as_bytes()).unwrap();
    let addr = addr.as_ptr();
    unsafe {
        ffi::emscripten_GetProcAddress(addr as *const _) as *const _
    }
}

But when I do so, emscripten reports in the console that it received some garbage data. I initially thought that it was a problem with the ABI, however when I pass a static value (for example emscripten_GetProcAddress(b"hello world\0" as *const _)), it works (emscripten still complains that the name is wrong, but at least it’s no longer garbage). More importantly, it also works if I replace addr.as_ptr() with into_raw (which leaks the string).

Even if I manage to pass the correct strings to emscripten, things continue to be bad afterwards. For example calling glGetString(GL_VERSION) returns the string A fantastic window!, which is a static string located in the app.

I don’t exactly know what happens, but it seems to be a low-levelish problem.

6

If you want to try it, I merged the change in master.

So you can just clone glutin, remove this line (because it will trigger an unimplemented!()), and build the window example.

The problematic code I talked about is here.

7

Thanks for the report @tomaka.

8

I just recompiled Rust with --enable-debug and that fixed the problem, so it’s caused by release mode.

9

[quote=“brson, post:3, topic:3154, full:true”] Ooh sorry. You’re right. The LLVM commit is from kripken/emscripten-fastcomp. I didn’t update the remote. So to get it to work one needs to fetch the proper remote.

Edit: Updated OP w/ directions for fetching fastcomp [/quote]Great, thanks.

After running the tests and having every single one fail, I found that tests are being run with the nodejs command (which I don’t have) whereas the Node binary provided with emscripten is just node; changing it in the test runner fixes the issue. Where did that difference come from?

10

nodejs is Debian-specific alias of node. For details, see Resolution of node/nodejs conflict by Debian Technical Committee. (Since this was resolved by Technical Committee, it is unlikely to change on Debian, if ever.)

11

Emscripten should automatically detect nodejs - https://github.com/kripken/emscripten/blob/42fb486/tools/shared.py#L227

What does your ~/.emscripten file say?

Anyway, I think the build instructions are in the wrong order - source emsdk_env.sh should come after install and activate, per https://github.com/juj/emsdk#installing-emsdk-directly-from-github

12

emscripten is working fine; the Rust test runner is trying to invoke nodejs to run tests whereas I believe it should be using node (or dynamically picking one of those that works). See runtest.rs.

13

I didn't even realize emscripten came with node! (are you sure?) My system had both 'node' and 'nodejs' and I picked one. Probably compiletest should dynamically figure out which one to use.

14 
$ which node
/home/tari/emsdk_portable/node/4.1.1_64bit/bin/node
$ which emcc
/home/tari/emsdk_portable/emscripten/incoming/emcc

(emcc for sanity verification)

I don’t have any version of Node installed system-wide.

15

I fixed the most obvious tests in https://github.com/rust-lang/rust/pull/31532

I can spot three more failure reasons:

  1. pthreads support. Right now emscripten exports pthreads symbol, but they are all dummy functions that print an error message and abort. I’m going to try using -s USE_PTHREADS=2, which enables experimental support for the pthreads library and checks at runtime whether threads are supported by the JS implementation.

  2. asm! doesn’t work. I don’t know what to do with that. Should the tests be disabled, or should asm! be fixed to output javascript code (if that’s possible at all)?

  3. Some failures because jemalloc isn’t compiled. Some tests beginning with allocator_ fail because of something like can't find crate allocator_jemalloc. I don’t really know how all that works, so I’m not qualified to fix that.

That’s for debug mode. Compiling rustc in release mode will probably bring more test failures.

16

-s USE_PTHREADS=2 seems to be a purely browserish thing (not that I expected threads to work with node, but I at least wanted some proper error codes being returned).

When I use this flag all the tests fail because the Javascript code tries to access document, which doesn’t exist in node.

17

I may not be able to help much with the pthreads-related tests, but I can help with the other two!

Feel free to just disable these, I think that asm! is incompatible with emscripten and forever will be (same true for webassembly I believe)

Yeah right now all these tests start with // ignore-platform for a bunch of platforms that don't have jemalloc compiled. The situation isn't that great here, and ideally tests wouldn't try to use jemalloc at all!

18

In my opinion the best course of actions about that pthreads problem is to force single-threading by modifying the functions within libstd/sys and making them either do nothing or return an error, so that pthreads isn’t used at all.

Then ignore all tests that require a thread to be spawned.

Would that be ok? I don’t think any other solution will come any time soon.

19

I’d personally prefer to avoid stubbing out std::thread primitives for single-threaded ones and instead just proxy error codes up through the normal pathways (e.g. pthread_create should return an error, right?).

I’d be fine adding // ignore annotations for now, but we may be adding quite a few…

20

The problem is that it doesn't :-/ And it's not just pthread_create. All mutexes and rwlocks (and maybe also TLS), even when used in a single-threaded environment, also straight up crash when you lock them.

The emscripten "linker" automatically detects which pthreads symbols are used by the program and replaces them with dummy functions that print missing function and call abort(-1). This is not something specific to pthreads, it's how the linker works.

The only way to enable pthread symbols is to use the USE_PTHREADS option, which is documented here. But as I said above it then tries to use the document object, which doesn't exist in node.

Maybe we may want to ping kripken on this problem.

Yes, that's around 120 tests rpass tests I think.