# Running tests on wasm32-unknown-unknown

**URL:** <https://internals.rust-lang.org/t/running-tests-on-wasm32-unknown-unknown/15010>\
**Category:** tools and infrastructure\
**Created:** [July 12, 2021, 2:00pm UTC](https://internals.rust-lang.org/t/running-tests-on-wasm32-unknown-unknown/15010 "2021-07-12T14:00:55Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![matklad](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/matklad/32/12266_2.png) [@matklad](https://internals.rust-lang.org/u/matklad)\
**Post date:** [July 12, 2021, 2:00pm UTC](https://internals.rust-lang.org/t/running-tests-on-wasm32-unknown-unknown/15010/1 "2021-07-12T14:00:55Z")

</div>

Today I've realised a surprising thing: `cargo test` just works for `wasm32-unknown-unknown`:

```rust
$ bat -p src/lib.rs
#[test]
fn arithmetics_works() {
    #[cfg(feature = "bug")]
    assert_eq!(2 + 2, 5);
    #[cfg(not(feature = "bug"))]
    assert_eq!(2 + 2, 4);
}

$ bat -p run.sh 
#!/bin/sh
wasmtime "$@" --invoke main 0 0

$ export CARGO_TARGET_WASM32_UNKNOWN_UNKNOWN_RUNNER=./run.sh

$ cargo t --target wasm32-unknown-unknown
    Finished test [unoptimized + debuginfo] target(s) in 0.00s
     Running unittests (target/wasm32-unknown-unknown/debug/deps/wt-e2cbdd00108cd06b.wasm)
warning: using `--invoke` with a function that takes arguments is experimental and may break in the future
warning: using `--invoke` with a function that returns values is experimental and may break in the future
0

$ cargo t --features bug --target wasm32-unknown-unknown
    Finished test [unoptimized + debuginfo] target(s) in 0.00s
     Running unittests (target/wasm32-unknown-unknown/debug/deps/wt-01d9e8c358b9bc58.wasm)
warning: using `--invoke` with a function that takes arguments is experimental and may break in the future
Error: failed to run main module `/home/matklad/tmp/wt/target/wasm32-unknown-unknown/debug/deps/wt-01d9e8c358b9bc58.wasm`

Caused by:
    0: failed to invoke `main`
    1: wasm trap: unreachable
       wasm backtrace:
         0: 0x3aaaf - <unknown>!__rust_start_panic
         1: 0x3a7bb - <unknown>!rust_panic
         ...
         25: 0xd30 - <unknown>!main
       
error: test failed, to rerun pass '--lib'

Caused by:
  process didn't exit successfully: `/home/matklad/tmp/wt/./run.sh /home/matklad/tmp/wt/target/wasm32-unknown-unknown/debug/deps/wt-01d9e8c358b9bc58.wasm` (exit status: 134)

```

For `wasm32-unknown-unknown`, `cargo t` produces a wasm binary with a `main` function, which you can call, and which will either exit with zero (when the test pass) or else hit `unreachable` if there are test failures. So, it is possible to specify a custom cargo runner which will execute the resulting wasm.

Note that this doesn't impose a lot of constraints on the environment -- wasm can import or export arbitrary functions, as long as the runner actually sets all of the imports. The only non-fully general thing here is that the test-harness exposes the `main` export, which might conflict with some existing export.

There's a big missing piece here -- when test harness prints textual results, it all goes to ["/dev/null"](https://github.com/rust-lang/rust/blob/e97c29bda238cf55ddd7eec772deb823364b1846/library/std/src/sys/unsupported/stdio.rs#L25-L33).

**Question:** would it be a good idea to allow libtest to specify hooks (wasm-imports) to allow the wasm-runner intercept prints and forward them to the terminal?

One way to achieve something similar is to just compile to the `wasm32-wasi` target, but that seems to invasive -- it requires the wasm runner to implement a particular environment. In contrast, today with `wasm32-unknown-unknown` we don't care about environment at all, the code under test decides which functions it imports/exports. Stdio and entry point seem to be the only two exceptiions where libtest iteslf needs to interact with environment. For the entry point, we already export main. We don't currently import stdio-related hooks, but we already have the logic to substitute stdio for tests, so it seems that morally extending it a bit might be a good unlocking capability?

---

<div class="post-metadata">

**Author:** ![josh](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/josh/32/5934_2.png) [@josh](https://internals.rust-lang.org/u/josh)\
**Post date:** [July 12, 2021, 7:44pm UTC](https://internals.rust-lang.org/t/running-tests-on-wasm32-unknown-unknown/15010/2 "2021-07-12T19:44:39Z")

</div>

I/O is a pretty substantial thing to add, and I don't know that we should standardize a non-standard interface for it just for tests. But it _would_ be nice to get test output from wasm32-unknown-unknown tests.

Perhaps it would make sense to export a subset of WASI, _just_ for the test interface to use, not for use in the rest of the code. I think WASI is the right interface for the test harness, even if we're compiling the rest of the code for `wasm32-unknown-unknown`.

---

<div class="post-metadata">

**Author:** ![matklad](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/matklad/32/12266_2.png) [@matklad](https://internals.rust-lang.org/u/matklad)\
**Post date:** [July 12, 2021, 8:17pm UTC](https://internals.rust-lang.org/t/running-tests-on-wasm32-unknown-unknown/15010/3 "2021-07-12T20:17:33Z")

</div>

How small would be the minimal WASI subset that would work here? One risk here is indeed pulling the whole of I/O, with file descriptors and error handling. Conceptually, it seem that for this particular case we need something much simpler, like `fn log(buf: *const u8, len: usize) -> ()`.

---

<div class="post-metadata">

**Author:** ![CAD97](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/cad97/32/3460_2.png) [@CAD97](https://internals.rust-lang.org/u/CAD97)\
**Post date:** [July 12, 2021, 10:24pm UTC](https://internals.rust-lang.org/t/running-tests-on-wasm32-unknown-unknown/15010/4 "2021-07-12T22:24:55Z")

</div>

Using just the WASI spec, it seems like the minimal way to get IO is to use just [`fd_write(fd: fd, iovs: ciovec_array) -> Result<size, errno>`](https://github.com/WebAssembly/WASI/blob/main/phases/snapshot/docs.md#-fd_writefd-fd-iovs-ciovec_array---resultsize-errno), along with a non-WASI way to get the stdout handle. (I don't know the WASI way, or if it exists, scanning the existing docs.)

The simplest solution imo is to say that the test runner (optionally) uses a WASI execution environment, but that the tested code runs full wasm32-unknown-unknown, if such separation is possible. The test runner would only query and use the `fd_write` (or equivalent) permission (along with however it gets the stdout fd).

The next simplest is the minimal approach of just defining a module that provides `cargo_test_harness_print(io: ciovec)` (where `ciovec` is basically just `(*const u8, size)`) and nothing else. Given it's just the one function, it doesn't seem like that much of a burden to stabilize that interface, especially if we delegate to WASI's definition of `ciovec`.

---

<div class="post-metadata">

**Author:** ![newpavlov](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/newpavlov/32/3290_2.png) [@newpavlov](https://internals.rust-lang.org/u/newpavlov)\
**Post date:** [July 12, 2021, 10:41pm UTC](https://internals.rust-lang.org/t/running-tests-on-wasm32-unknown-unknown/15010/5 "2021-07-12T22:41:08Z")

</div>

`wasm32-unknown-unknown` is already a somewhat weird target with its `std`, which arguably should not have been implemented in the first place. I hope that we will not continue to pile weirdness further and instead proper environment specific targets will be added instead (`wasm32-web` and similar).

---

<div class="post-metadata">

**Author:** ![programmerjake](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/programmerjake/32/5893_2.png) [@programmerjake](https://internals.rust-lang.org/u/programmerjake)\
**Post date:** [July 13, 2021, 6:08pm UTC](https://internals.rust-lang.org/t/running-tests-on-wasm32-unknown-unknown/15010/6 "2021-07-13T18:08:05Z")

</div>

for wasi, stdout is just fd `1` and stderr is just fd `2`, just like unix.

---

<div class="post-metadata">

**Author:** ![matklad](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/matklad/32/12266_2.png) [@matklad](https://internals.rust-lang.org/u/matklad)\
**Post date:** [July 14, 2021, 11:57am UTC](https://internals.rust-lang.org/t/running-tests-on-wasm32-unknown-unknown/15010/7 "2021-07-14T11:57:44Z")

</div>

Thinking more about this, it seems there's a much better approach here. The original proposal tries to make the existing test harness work, but that's not right. The harness assumes pretty standard environment with terminal and such, but that's not necessary what is available in any particular `wasm32-unknown-unknown` environment.

A better approach is to let the environment itself provide the test harness. This can be achieved, by, eg, the `#[test]` attribute just marking the functions with `#[export_name]` attribute with some prefix, say, `$rust-test$`. A cargo runner can then import wasm module, list its exports, and run them as tests.

One problem here is that each language's ecosystem may come up with their own conventions for exporting tests functions, while it feels like a wasm-level concern. So, it might not be too prudent to encode a specific general convention into Rust's libtest just now.

Luckily, this can be poly-filled with a custom test macro without much loss of ergonomics:

```rust
$ cat src/lib.rs
#[cfg(test)]
mod tests {
    use webassembly_test::webassembly_test;

    #[webassembly_test]
    fn it_works() {
        assert_eq!(2 + 2, 4);
    }

    #[webassembly_test]
    fn it_does_not_work() {
        assert_eq!(2 + 2, 5);
    }

    #[webassembly_test]
    #[ignore]
    fn it_is_ignored() {
        assert_eq!(2 + 2, 5);
    }
}

$ cargo test --target wasm32-unknown-unknown
     Running `webassembly-test-runner target/wasm32-unknown-unknown/debug/deps/hello_world.wasm`

running 3 tests
test hello_world::tests::it_works ... ok
test hello_world::tests::it_does_not_work ... FAILED
test hello_world::tests::it_is_ignored ... ignored

test result: FAILED. 1 passed; 1 failed; 1 ignored;

```

This is now published as [https://crates.io/crates/webassembly-test](https://crates.io/crates/webassembly-test) (mind a somewhat aggressive MSRV: 1.54.0).

---

<div class="post-metadata">

**Author:** ![josh](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/josh/32/5934_2.png) [@josh](https://internals.rust-lang.org/u/josh)\
**Post date:** [July 14, 2021, 3:03pm UTC](https://internals.rust-lang.org/t/running-tests-on-wasm32-unknown-unknown/15010/8 "2021-07-14T15:03:01Z")

</div>

That _does_ seem like a much better plan. Then the test harness can be outside the sandbox, and call any native functionality it wants for reporting.

---

<div class="post-metadata">

**Author:** ![bjorn3](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/bjorn3/32/2736_2.png) [@bjorn3](https://internals.rust-lang.org/u/bjorn3)\
**Post date:** [July 14, 2021, 4:28pm UTC](https://internals.rust-lang.org/t/running-tests-on-wasm32-unknown-unknown/15010/9 "2021-07-14T16:28:51Z")

</div>

Using custom test runners is already possible on nightly: [Testing | Writing an OS in Rust](https://os.phil-opp.com/testing)

---

<div class="post-metadata">

**Author:** ![dhm](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/dhm/32/4879_2.png) [@dhm](https://internals.rust-lang.org/u/dhm)\
**Post date:** [July 14, 2021, 11:20pm UTC](https://internals.rust-lang.org/t/running-tests-on-wasm32-unknown-unknown/15010/10 "2021-07-14T23:20:28Z")

</div>

> [@matklad](#):
>
> (mind a somewhat aggressive MSRV: 1.54.0).

Is that to be able to feed a macro invocation as the rhs of an attribute? By funneling the macro invocation through an `:expr` metavariable, you should be able to lower the MSRV 🙂

> **.diff**
>
> ```diff
> diff --git a/src/lib.rs b/src/lib.rs
> index 1c41625..a886084 100644
> --- a/src/lib.rs
> +++ b/src/lib.rs
> @@ -15,11 +15,16 @@ pub fn webassembly_test(_attr: TokenStream, item: TokenStream) -> TokenStream {
> ignore = "ignore$"
> }
>  
> - let res = quote! {
> - #[cfg(test)]
> - #[export_name = concat!("$webassembly-test$", #ignore, module_path!(), "::", #name)]
> - #item
> - };
> + let res = quote!(
> + macro_rules! __helper__ {( $export_name:expr $(,)? ) => (
> + #[cfg(test)]
> + #[export_name = $export_name]
> + #item
> + )}
> + __helper__! {
> + ::core::concat!("$webassembly-test$", #ignore, ::core::module_path!(), "::", #name)
> + }
> + );
> // eprintln!("{}", res);
> res.into()
> }
> 
> ```
> 
> - Tested with `1.51.0`
> 
> - (the ` __helper__!` thing could be factored out, obviously)

---

<div class="post-metadata">

**Author:** ![system](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/system/32/14092_2.png) [@system](https://internals.rust-lang.org/u/system)\
**Post date:** [October 12, 2021, 11:20pm UTC](https://internals.rust-lang.org/t/running-tests-on-wasm32-unknown-unknown/15010/11 "2021-10-12T23:20:59Z")

</div>

Closed on October 12, 2021, 11:20pm UTC

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.
