Proposing compiletest for the nursery

[erickt]

I’d like to propose migrating compiletest crate into the nursery, which is a crates.io-ified version of the rustc compiletest framework. I wrote about it recently on my blog, where I wrote about how fundamentally important it is to test for compile failures when using Unsafe Rust. In it, I talk about a use-after-free bug I had in zmq where a pointer was able to escape it’s lifetime. I fixed the bug, but accidentally reverted it later on because I had no test that guaranteed that the region system was catching and rejecting this error. I think it’s critical in almost all libraries that use Unsafe Rust should have compile-fail tests, so it would make sense to have compiletest be a part of the officially supported platform.

[alexcrichton]

I’ve long wanted the ability to test for code not compiling as well, but my main worry in this regard is that the tests are quite brittle. The compiler provides no guarantee about the stability of its error messages and such, so these sorts of tests are bound to break over Rust releases as messages are tweaked, codes are changed, etc.

I unfortunately don’t know of a great way around this without a lot of cooperation from the compiler as well, which may not be super easy to do at this time.

[aturon]

Definitely a legitimate concern, but maybe this is something that could be addressed before transition into full rust-lang (via the requisite RFC)? Or do you think it’ll require a total overhaul?

[erickt]

I know the feeling, but now that we’ve got real production users, I’d rather pay the “brittle test” cost now than risk security holes leaking into our ecosystem. I think we got two things going for us. First, compiletest doesn’t need to be stable, it can and should iterate on it’s design while being in the nursery. Second, it’s a great forcing function on the compiler itself to generate machine-parsable error messages so that libraries like compiletest can be made more robust.

[alexcrichton]

Or do you think it’ll require a total overhaul?

Ah yes this is a good point! I highly doubt that any overhaul will be required to get it working, it’d primarily just be a policy-ish thing to deal with.

First, compiletest doesn’t need to be stable, it can and should iterate on it’s design while being in the nursery. Second, it’s a great forcing function on the compiler itself to generate machine-parsable error messages so that libraries like compiletest can be made more robust.

Yeah these indeed are good points!

---

Oh one other thing I just remembered is that I’m not sure we’ve explicitly made a decision one way or the other about nursery crates requiring nightly Rust vs compiling on stable. For example compiletest looks like it still requires nightly Rust, so would that be a point against it? A neutral point?

I’d personally prefer to require that crates in the nursery work on stable in some capacity as it may indicate that a large overhaul is needed to move it into rust-lang eventually?

[Manishearth]

but my main worry in this regard is that the tests are quite brittle. The compiler provides no guarantee about the stability of its error messages and such,

We don’t need to set backwards compatibility guarantees that compile-fail tests won’t break. In that context, this isn’t an issue. Brittle tests become the problem of the test maintainers.

(Besides, backcompat is an issue for downstream users, not developers of the library affected)

[erickt]

Oh one other thing I just remembered is that I’m not sure we’ve explicitly made a decision one way or the other about nursery crates requiring nightly Rust vs compiling on stable. For example compiletest looks like it still requires nightly Rust, so would that be a point against it? A neutral point?

I’d personally prefer to require that crates in the nursery work on stable in some capacity as it may indicate that a large overhaul is needed to move it into rust-lang eventually?

I don’t think it’d be that hard to work around. Most of the feature flags can be rewritten. The only place I can tell that’s really using a librustc library is a default config method that’s looking up the host platform. Is there a stable way to determine the host platform?

[brson]

I totally agree that testing compile-fail is an important use case, and I’m glad compiletest has legs, but it (at least the in-tree version) is a mess of upstream-rust-specific stuff. I’d like to reconsider the requirements and design from scratch instead of pushing forward with what is some of the oldest and roughly-maintained Rust code in the world!

Edit: also, we can reasonably let this bake in the community for a long time to see if it picks up steam - not everything of strategic importance needs to be in the nursery.

[brson]

It just occurred to me that it may make sense for rust-lang to maintain compiletest just to reduce the duplication with the in-tree compiletest. As long as compiletest is going to be used both in and out-of-tree, and assuming it doesn’t need to diverge functionally from the in-tree version, if we used the out-of-tree version we would share the maintenance load, and possibly benefit from real improvements to compiletest But we’re not going to use the out-of-tree version if it’s not rust-lang.

In other words, as long as compiletest persists in- and out-of-tree, we may as well merge the two and be responsible for it.

But not endorse it. Maybe it can be evolved into something we can recommend.

[Gankro]

A possible back-compat story: error if it compiles; warn if it has the wrong message.

[alexcrichton]

Thanks for the nomination @erickt! To follow up, the libs team discussed this the other day and the conclusion was to not move this into the nursery quite just yet. It’s certainly a critical piece of infrastructure for the compiler, and it may end up living in even rust-lang one day to be shared among the compiler and the ecosystem, but there’s quite a few outstanding decisions to make about the tool so the time doesn’t seem quite ripe to move to the nursery just yet.

[erickt]

@alexcrichton: No problem! Just as long as this kind of functionality is prominent somewhere in our ecosystem, be it rust-lang, or in a well maintained external crate makes me happy.

[erickt]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.