External dependencies in declarative format

[ignatenkobrain]

Hello from Fedora maintainer of over 500 crates!

We are working on a way to generate our RPM packages more automatically. The only problem (hopefully) which persists from cargo side is external dependencies.

For example, there is glib-sys crate which checks for the glib on the system using pkg-config. Depending on a activated feature, different version of it.

There was a project https://github.com/joshtriplett/metadeps, but it seems to be pretty much dead.

It would be cool if that info could be stored inside Cargo.toml in some standardized way. How can we help with that?

[CAD97]

The way to do it would be to write up and submit a Cargo RFC for the feature. Bonus points if you have a working prototype.

A more quickly actionable solution is to build a Cargo subcommand and petition some of the packages you package to use it. If it’s better than what they’re doing currently, they’ll probably consider using it. And if it gains popularity, that’s some weight behind adding it to Cargo directly.

IIRC (can’t find the reference to it), the [package.metadata] table in Cargo.toml is free to be whatever, and Cargo-integrated tools are encouraged to put package metadata there.

Cargo’s future is a little rough right now. But if you build something good and properly useful (and that works at least on all tier-1 platforms), people will (hopefully) use it.

[kornel]

Cargo-deb has an interesting hack for this problem: it runs ldd on the produced executables to find which dynamic libraries they actually use, and queries dpkg to find which packages provide these libraries. This bypasses Cargo entirely and doesn’t require knowing anything about any build scripts, configurations, features, etc.

[matklad]

The way to do it would be to write up and submit a Cargo RFC for the feature. Bonus points if you have a working prototype.

The metabuild RFC/project is exactly that: https://doc.rust-lang.org/cargo/reference/unstable.html#metabuild

I think the idea behind it is great: instead of many build scripts for various projects, you have a single build script parametrized by information from Cargo.toml.

The way forward is perhaps making the metabuild ready for the use, and sending PRs to a couple of projects, replacing their custom build scripts with metabuild.

If the approach is proven to work, Cargo docs could officially recommend using a metabuild.

[matklad]

One clarification: the metabuild feature in Cargo is absolutely not required for using metabuild project. The feature is a cosmetic thing, which allows you to not write build.rs at all. Without it, need a trivial build.rs file which just calls metabuild::main()

[josh]

metadeps isn’t dead so much as done in its current state, dormant and waiting on metabuild to improve further. With metabuild now available in cargo, we should start writing metabuild-enabled crates that incorporate metadata from Cargo.toml.

Another option: with https://github.com/rust-lang/rfcs/pull/2627 we could have crates that build against shared libraries without needing those libraries installed at build time. (We’d still need to generate the runtime dependencies for the package, though.)

[ignatenkobrain]

metadeps isn’t dead so much as done in its current state, dormant and waiting on metabuild to improve further. With metabuild now available in cargo, we should start writing metabuild-enabled crates that incorporate metadata from Cargo.toml .

Can you merge Pull Requests and fix Issues then? I probably don’t fully understand how this metabuild thing works. Do you have some example how it would be potentially integrated with pkg-config dependencies?

Another option: with https://github.com/rust-lang/rfcs/pull/2627 we could have crates that build against shared libraries without needing those libraries installed at build time. (We’d still need to generate the runtime dependencies for the package, though.)

That would be awful because it would be impossible to do any checking for undefined symbols or so during build.

[josh]

Can you merge Pull Requests and fix Issues then?

Working on that now, thank you. Sorry that I lost track of those.

I probably don’t fully understand how this metabuild thing works. Do you have some example how it would be potentially integrated with pkg-config dependencies?

metabuild would let you use crates like metadeps without writing a build.rs at all. The goal is to standardize almost every common use of build.rs into a crate that reads declarative metadata.

That would be awful because it would be impossible to do any checking for undefined symbols or so during build.

The linker can check for undefined symbols, but it can’t (for instance) check for defined symbols with different interfaces, or any number of other issues.

The idea would be that the test suite needs the library present, but building the crate doesn’t.

[bascule]

There’s a problem semi-related to this I’m interested in: many *-sys crates look for if the system dependency is available in build.rs, and if it’s not, do wacky things like use git2 to clone a repo.

It’d be nice to have a reproducible, declarative alternative to that for optional build artifacts, e.g. “fetch this tarball over HTTPS and ensure its SHA-256 digest is X, or clone this get repo and set HEAD to commit Y”

[kornel]

I usually recommend crates to bundle the source instead of downloading it. The sources usually aren’t that big, and having them in the crate solves all the problems of matching versions, downloading, verifying, cleaning up, etc.

[CAD97]

Then whatever tool it is should probably provide a guide on adding a simple submodule for the bundled source (and/or best practices for just having the source in /vendor or smth).

[bascule]

I usually recommend crates to bundle the source instead of downloading it.

Yeah, that’s a great point. One alternative to having the build script use git2 to fetch the code is using a git submodule in the original project (for easy updates), but when the crate is published, it can include the sources from the submodule directly in the published crate.

[kornel]

Yes. Fortunately git submodules work out of the box and automatically become regular directories in the published package.