Posting this in advance of today’s Cargo meeting.
I’d like to propose a step towards a solution for the problem of integrating a Cargo build process with native dependencies, and with broader build systems or projects, such as massive mono-repo build systems, or Linux distributions.
Right now, the biggest problem facing such systems involves
build.rs scripts and the arbitrary things those scripts can do. Such systems typically need more information about native dependencies that are embedded in
build.rs, so that they can provide their own versions of those dependencies, or encode appropriate dependencies in another metadata format such as the dependencies of their packaging system or build system. Right now, such systems often have to override the
build.rs script themselves, and do custom per-crate integration work, manually; there’s no way to introspect what
build.rs does, or get a declarative semantic description of the build script.
However, I don’t believe that we should enshrine a single solution for that problem into Cargo itself; at least, not at this time. We need to explore the problem space further, and I believe we should do so via the crate ecosystem.
Thus, I’d like to propose the following:
- Introduce a simple way to declare in
Cargo.toml that the crate’s build script (
build.rs) should come from one of its build dependencies, rather than from the crate itself.
- Build scripts provided by a build-dependency crate would implement a trivial interface: they get called with no parameters, they should get all their information by parsing declarative
Cargo.toml, they print the standard build script outputs that Cargo expects, they potentially generate code, and they return an Error if anything goes wrong.
- Start experimenting in the crates ecosystem with a family of crates that provide various fully declarative build system steps. These steps are easily aggregated without further help from Cargo: we can have a “metabuild” crate that allows listing multiple such build steps, and its build script would invoke each of the individual build steps in a well-defined way, allowing them to feed into each other if needed.
- Work with the ecosystem of broader build systems and packaging systems to create well-defined ways to override those individual declarative build steps, such that those systems can know that if they handle each individual step of a build process, they’ve completely subsumed the necessary functionality of the build process.
Over time, we may also make it possible to pull out and override some of Cargo’s built-in functionality into build scripts, such as dependency location, resolution, and even building of dependencies, enhancing currently hard-coded mechanisms for doing so. In doing so, I’d recommend trying to treat those steps of Cargo as a library, rather than as a layer that has specific points for extensibility; for a parallel, see the discussion of the midlayer mistake in the context of the Linux kernel. However, we need a good declarative hook mechanism first, and that’s what I’m proposing here as a first step.