Rustc dependency policy


#1

So, thanks to all the hard work on ./x.py, we have the ability to pull in crates from crates.io and elsewhere. This is wonderful. However, we currently impose some pretty minimal requirements on said crates. As far as I know, we just check that they have suitable licenses.

Recently, rayon encountered some problems due to its use of compiletest-rs, which turns out to be depending (on windows) on the dbghelp library. This is because of miri which uses backtrace. This strikes me as a kind of warning: our set of dependencies is rapidly growing and I don’t know that we are paying any sort of attention. This increases brittleness.

I am wondering if we should consider a whitelist on the transitive crates we employ. I feel like adding a new dependency to rustc should be a “more momentous” (but not necessarily very momentous) decision – perhaps one that we FCP.

Thoughts? Am I over-reacting?

UPDATE: @retep998 opened this issue describing the particular problem rayon hit. To be clear, though, I’m not saying that this problem itself is necessarily a problem, but it does seem to me like the set of dependencies is growing sort of rapidly. Not sure if this is necessarily a problem though.


#2

The “dependency allowed to exist” part of this recent post seems relevant. At least in the sense that there is certainly a use-case for this level of control.


#3

compiletest_rs is just a dev-dependency of rayon, and just intuitively, I don’t think it should be pulled in, in any way.


#4

Thoughts? Am I over-reacting?

I don’t think so. Every dependency might cause work in the future.

Trying to keep dependencies low certainly is a good idea, but perhaps even more important to ensure that they’re maintained.


#5

yes, it’s a dev-dependency. The problem we were encountering was travis failures. We use it to check that things are failing to compile when they should. I’m interested in other solutions to that problem, but that’s kind of a separate question.

Mostly what I’m trying to ask here is whether we should try to at least throw up some roadblock to adding new dependencies, so that we have a chance to look at them as they go in. I think we don’t have to be sticklers about it, but I suspect many of them are kind of “accidental”.

In this case, for example, miri depends on backtrace, but that’s not really needed from within the compiler; I think it’s used for prettier errors. We could have potentially made it an optional feature.

(Should we have? Not sure.)


#6

I would personally be a fan of trying out a whitelist for compiler dependencies (false positives through cargo/rustfmt/rls/etc shouldn’t affect this). The compiler dependencies end up being much trickier in practice I think than rls/cargo/etc because we continue to use them to generate programs, whereas cargo/rls are just standalone programs.

I’m also starting to get worried about our build times for the compiler itself. Whenever we put more dependencies in the compiler those are crates we have to build and it can take longer and longer to do that. I’ve just done some analysis which shows that we’re clearly regressing on how long it takes to bootstrap rustc over time, it’s just not clear to me what’s actually causing the problem. I don’t think pulling in tons of crates from crates.io help much, however.

Could we perhaps try an explicit whitelist for awhile of allowed crates? See how it goes? If it’s too onerous we can reconsider and otherwise I’d agree that having a double check for “are you sure you wanted to add this dependency?” would be a good idea.


#7

I’m in favor of trying it out. Is it hard to do? =) I’d hate to add more onto your plate personally, I’m not really following who hacks around the build system though.


#8

Hm it won’t be too hard but I think it will be nontrivial.

We’ve already got dependency checking which verifies license, so it should be mostly just adding an array next to EXCEPTIONS for a whitelist (and then checking the names of directories and such). The trick is that we only want to verify the dependencies of rustc itself, so we’d have to run cargo metadata to learn about the dependency graph. The build system already does that though so the logic could perhaps just be moved there.