Proposal: Support setting fixed hash seed for benchmarking purposes

[itamarst]

New here, so forgive me if this is the wrong place to post proposals, or if I'm doing it wrong.

Use case

When benchmarking, one wants to get rid of sources of noise. Randomized hash functions are one such source of noise. The solution is to set a fixed seed—somehow.

Solutions

Custom logic (not good enough)

One alternative is to have each application or libraries have its own custom logic for doing this. However, dependencies may also be creating HashMaps; it would be unworkable to have everyone create their own.

Also, as far as I can tell Rust's default hash doesn't even have the option to set a custom seed?

Environment variable (proposed solution)

If Rust's default hasher, and hopefully third-party hash libraries, checked for a well-known environment variable as their seed, it would be possible to set a single environment variable and have all libraries and dependencies use a fixed seed. Python has PYTHONHASHSEED, for example.

Rust could use RUST_HASHSEED, which can be a string of a u64, e.g. export RUST_HASHSEED=12345. Or if the resulting security impact is a concern, RUST_INSECURE_HASHSEED just to emphasize what you're getting into.

Different hashers have different seed requirements, of course. E.g. ahash takes four u64. But that's fine, the goal isn't to enable a production source of randomness, the goal is just to be able to set a fixed seed for benchmarking only—even if it only covers a subset of the seed space that's not an issue.

Other solutions?

Got to be more than just two, I assume; more ideas are welcome.

[josh]

It isn't safe to make arbitrary Rust code check that environment variable. Setuid/setgid programs, for instance, or other cases where the environment may be partly or fully untrusted.

We could have programs opt into this, via a call at the top of main, or a compile-time option. That would allow for benchmarking without introducing potential insecurity in other code.

[itamarst]

main() probably isn't general enough (my use case involves benchmarking a shared library), but opt-in would be fine too, so long as it's in a way that percolates down to dependencies and transitive dependencies.

[cuviper]

For Linux, recently we made the internal getrandom calls try to use the libc wrapper, rather than always making a raw SYS_getrandom call. This affects the standard library's RandomState, and you could write your own #[no_mangle] fn getrandom to intercept this and behave however you like. If you write that in the main executable, it should take precedence in the dynamic loader, or you could use an LD_PRELOAD library.

linux: try to use libc getrandom to allow interposition

Random Number

[itamarst]

That sounds promising, even if it's just on one platform, especially insofar as it's broader than just hashes. Is this a guaranteed interface once it hits stable, though? Seems more of an implementation detail.

The flip side is that something like RUST_HASHSEED is nice insofar as you can e.g. file an issue against random hash libraries saying "you should really implement this standard", whereas using getrandom may just be an implementation choice they happen to not to have taken.

[cuviper]

True, I would indeed call getrandom an implementation detail. It could very well change to something else in the future, without being considered a breaking change.

[itamarst]

So here's another proposal, inspired by above.

#[cfg(benchmark)]

When building code one could set a benchmark option via the compiler configuration options (I am very hazy on how this works, sorry). Libraries could then have deterministic implementations they would swap in in situation-specific ways, e.g. fixed seed for hashes, but also e.g. fixed seed for other sources of randomness.

This would require recompiling dependencies as well, and is unrelated to "bench" mode, though "bench" mode would likely enable it (see below).

[mbrubeck]

Currently at least, Cargo doesn't rebuild dependencies in "test" or "bench" mode. (Instead, it links tests and benchmarks to the normal debug or release builds of their dependencies.) Unless this changes, compile-time switches like #[cfg] won’t work for changing behavior of dependencies.

[itamarst]

This is probably orthogonal to cargo bench, e.g. I have my own benchmarking setup using Cachegrind, there's multiple benchmarking tools for Rust, etc.. So maybe comparing it to #[cfg(test)] is merely confusing (I'll edit proposal above), and better to think of it as something more akin to a target feature? So you'd typically have two variants of code, #[cfg(benchmark)] and #[cfg(not(benchmark))].

And yes, it would require full recompile (unless compiler was smart enough to notice certain compilation units don't have conditionals on this cfg flag).

[itamarst]

Another problem with an environment variable: I'm doing that now, and I'm getting reentrancy-caused crashes because I'm using LD_PRELOAD and getenv is happening as result of unsetenv, which makes relevant macOS locks very unhappy.