Today I've realised a surprising thing: cargo test just works for wasm32-unknown-unknown:
$ bat -p src/lib.rs
#[test]
fn arithmetics_works() {
#[cfg(feature = "bug")]
assert_eq!(2 + 2, 5);
#[cfg(not(feature = "bug"))]
assert_eq!(2 + 2, 4);
}
$ bat -p run.sh
#!/bin/sh
wasmtime "$@" --invoke main 0 0
$ export CARGO_TARGET_WASM32_UNKNOWN_UNKNOWN_RUNNER=./run.sh
$ cargo t --target wasm32-unknown-unknown
Finished test [unoptimized + debuginfo] target(s) in 0.00s
Running unittests (target/wasm32-unknown-unknown/debug/deps/wt-e2cbdd00108cd06b.wasm)
warning: using `--invoke` with a function that takes arguments is experimental and may break in the future
warning: using `--invoke` with a function that returns values is experimental and may break in the future
0
$ cargo t --features bug --target wasm32-unknown-unknown
Finished test [unoptimized + debuginfo] target(s) in 0.00s
Running unittests (target/wasm32-unknown-unknown/debug/deps/wt-01d9e8c358b9bc58.wasm)
warning: using `--invoke` with a function that takes arguments is experimental and may break in the future
Error: failed to run main module `/home/matklad/tmp/wt/target/wasm32-unknown-unknown/debug/deps/wt-01d9e8c358b9bc58.wasm`
Caused by:
0: failed to invoke `main`
1: wasm trap: unreachable
wasm backtrace:
0: 0x3aaaf - <unknown>!__rust_start_panic
1: 0x3a7bb - <unknown>!rust_panic
...
25: 0xd30 - <unknown>!main
error: test failed, to rerun pass '--lib'
Caused by:
process didn't exit successfully: `/home/matklad/tmp/wt/./run.sh /home/matklad/tmp/wt/target/wasm32-unknown-unknown/debug/deps/wt-01d9e8c358b9bc58.wasm` (exit status: 134)
For wasm32-unknown-unknown, cargo t produces a wasm binary with a main function, which you can call, and which will either exit with zero (when the test pass) or else hit unreachable if there are test failures. So, it is possible to specify a custom cargo runner which will execute the resulting wasm.
Note that this doesn't impose a lot of constraints on the environment -- wasm can import or export arbitrary functions, as long as the runner actually sets all of the imports. The only non-fully general thing here is that the test-harness exposes the main export, which might conflict with some existing export.
There's a big missing piece here -- when test harness prints textual results, it all goes to "/dev/null".
Question: would it be a good idea to allow libtest to specify hooks (wasm-imports) to allow the wasm-runner intercept prints and forward them to the terminal?
One way to achieve something similar is to just compile to the wasm32-wasi target, but that seems to invasive -- it requires the wasm runner to implement a particular environment. In contrast, today with wasm32-unknown-unknown we don't care about environment at all, the code under test decides which functions it imports/exports. Stdio and entry point seem to be the only two exceptiions where libtest iteslf needs to interact with environment. For the entry point, we already export main. We don't currently import stdio-related hooks, but we already have the logic to substitute stdio for tests, so it seems that morally extending it a bit might be a good unlocking capability?
I/O is a pretty substantial thing to add, and I don't know that we should standardize a non-standard interface for it just for tests. But it would be nice to get test output from wasm32-unknown-unknown tests.
Perhaps it would make sense to export a subset of WASI, just for the test interface to use, not for use in the rest of the code. I think WASI is the right interface for the test harness, even if we're compiling the rest of the code for wasm32-unknown-unknown.
How small would be the minimal WASI subset that would work here? One risk here is indeed pulling the whole of I/O, with file descriptors and error handling. Conceptually, it seem that for this particular case we need something much simpler, like fn log(buf: *const u8, len: usize) -> ().
Using just the WASI spec, it seems like the minimal way to get IO is to use just fd_write(fd: fd, iovs: ciovec_array) -> Result<size, errno>, along with a non-WASI way to get the stdout handle. (I don't know the WASI way, or if it exists, scanning the existing docs.)
The simplest solution imo is to say that the test runner (optionally) uses a WASI execution environment, but that the tested code runs full wasm32-unknown-unknown, if such separation is possible. The test runner would only query and use the fd_write (or equivalent) permission (along with however it gets the stdout fd).
The next simplest is the minimal approach of just defining a module that provides cargo_test_harness_print(io: ciovec) (where ciovec is basically just (*const u8, size)) and nothing else. Given it's just the one function, it doesn't seem like that much of a burden to stabilize that interface, especially if we delegate to WASI's definition of ciovec.
wasm32-unknown-unknown is already a somewhat weird target with its std, which arguably should not have been implemented in the first place. I hope that we will not continue to pile weirdness further and instead proper environment specific targets will be added instead (wasm32-web and similar).
Thinking more about this, it seems there's a much better approach here. The original proposal tries to make the existing test harness work, but that's not right. The harness assumes pretty standard environment with terminal and such, but that's not necessary what is available in any particular wasm32-unknown-unknown environment.
A better approach is to let the environment itself provide the test harness. This can be achieved, by, eg, the #[test] attribute just marking the functions with #[export_name] attribute with some prefix, say, $rust-test$. A cargo runner can then import wasm module, list its exports, and run them as tests.
One problem here is that each language's ecosystem may come up with their own conventions for exporting tests functions, while it feels like a wasm-level concern. So, it might not be too prudent to encode a specific general convention into Rust's libtest just now.
Luckily, this can be poly-filled with a custom test macro without much loss of ergonomics:
$ cat src/lib.rs
#[cfg(test)]
mod tests {
use webassembly_test::webassembly_test;
#[webassembly_test]
fn it_works() {
assert_eq!(2 + 2, 4);
}
#[webassembly_test]
fn it_does_not_work() {
assert_eq!(2 + 2, 5);
}
#[webassembly_test]
#[ignore]
fn it_is_ignored() {
assert_eq!(2 + 2, 5);
}
}
$ cargo test --target wasm32-unknown-unknown
Running `webassembly-test-runner target/wasm32-unknown-unknown/debug/deps/hello_world.wasm`
running 3 tests
test hello_world::tests::it_works ... ok
test hello_world::tests::it_does_not_work ... FAILED
test hello_world::tests::it_is_ignored ... ignored
test result: FAILED. 1 passed; 1 failed; 1 ignored;
Is that to be able to feed a macro invocation as the rhs of an attribute? By funneling the macro invocation through an :expr metavariable, you should be able to lower the MSRV
