Non-Rust artifact dependencies for build scripts

tools and infrastructure cargo
1

As detailed in this larger piece surveying build.rs scripts, a lot of build.rs scripts that generate FFI bindings (and thus need to pull in external sources) or need to pull in external corpuses of information (e.g. NLP-oriented crates) often need to download external artifacts themselves.

By artifacts, I mean:

  • remote HTTP archives or Git repositories
  • on-disk paths to executables the build.rs needs to shell out to and expects to be in $PATH
  • on-disk paths to foreign library information the build.rs needs to generate linker arguments/Rust code with

Usually, these artifacts are too large to be reasonably vendored in Git, either because of their size, or that the intention of the crate is actually to build against whatever the current OS's pkg-build provides, or because they're bindings over Absolute VendorWare that takes up 200 GiB and refuses to live anywhere other than /opt/NameBeforeTheAcquisition/PainAndSuffering/202x.y/Sdk

The ways build scripts accomplish this widely varies and any given build script may do any combination of the following:

  • they make HTTP requests to download the file and maybe check the hash
  • they git-clone the upstream repo and maybe check out a specific commit or tag.
  • they download a specific release tarball from the upstream project website
  • they use pkg-build to locate the OS installation of the headers
  • they search the filesystem for an existing installation of some VendorWare SDK at specific paths

With increased interest in sandboxing build scripts and increasing their security, it's worth investigating mechanisms through which said build scripts can better manage external dependencies. A lot of these build.rs scripts end up acting as ad-hoc, informally-specified, bug-ridden, slow (to compile) implementations of half of Common Lisp Nix.

Bringing them closer to Nix by encouraging explicit manifests of file/path/executable dependencies encourages reproducibility, auditability, caching, build times, and error handling when said artifacts are missing. It also means that said build scripts need less dependencies internally as they no longer have to bundle an HTTP client.

One might imagine a section in the Cargo.toml loosely like this:

[artifact-dependencies]
# git cloned to $OUT_DIR/artifacts/lib-source
lib-source = { git = "https://github.com/tukaani-project/xz.git", rev = "4b73f2ec19a99ef465282fbce633e8deb33691b3" }
# 700 MiB zipfile downloaded to $OUT_DIR/artifacts/build-rs.zip?
# You get a hash though.
"build-rs.zip" = { src = "https://sidequests.reduxrobotics.com/gacha/build-rs.zip", sha256 = "629ab10e62eeafa1efe34c0c45aec9607ec45d83032e976a400cb28d09d4f63b" }
# Only export as a path...Still some unsolved problems? Also platform-dependent behavior?
vendorware = { type = "path", env = "VENDORWARE_SDK", src = "file:/opt/UnpleasantVendorWare/sdk/include" }
os-package.pkg-config = true

[target.'cfg(target_os = "linux")'.artifact-dependencies]
# linux-only
bpf-scripts = { ... }

[patch.artifact-dependencies]
# what if i want to override things?????

[build-commands]
# We should probably also have some way of listing what commands need to exist for a build.rs to run.
# Nix flakes/derivations take it a step farther and let you pick _exactly_ which binaries 
required-commands = ["git", "cbindgen", "rc.exe"]

This isn't a one-size fits-all solution, though. There's still quite a few open questions:

  • If these artifacts are so large they are fetched externally, does it make sense to still cargo-vendor them?
    • For some people it might, for others it might not. The status quo is that vendored build scripts would still download external binaries.
  • For build environments that are already sandboxed or provide their own reproducible artifact primitives, how can we pass them to Cargo without Cargo yelling about how it can't git-clone the remote?
    • Can we do this at runtime without evil Cargo.toml hacking?
  • How much complexity in platform-dependent artifact selection do we want to support via this happy path?
    • For some artifacts, this is non-trivial
  • For executables, you can use it to ask permission from the users to execute, but this manifest wouldn't tell the build script the conditions for the executables to actually be valid

This isn't going to solve every external dependency need a build.rs has, but I think there's a strong case for a nice happy path. I would probably treat this feature like const in the standard library: a strongly-encouraged nice-to-have that slowly becomes more available for a wider number of user cases, but not mandatory to get the job done.

2

Before getting into any further thoughts, I wanted to check in on how this is expected to work when packages using this are published to a registry where a core design principle is to be leftpad proof (all content must be accessible from that registry). For example, publish is blocked when git-only dependencies are present.

3

In a sense, we’re already not “leftpad proof” when packages are downloading outside resources.

One can say “they shouldn’t do so” — and build script sandboxing would give a means to enforce that — but that won’t address the use cases which can’t reasonably be solved by publishing more bytes to the registry.

An upside of explicitly declared non-Rust dependencies would be that people can identify the packages that have them and choose to avoid or use those packages. (And it’s not just a binary choice; for example, someone looking for Rust packages to bind to system libraries might want to search by which system libraries the packages do depend on.)

1 Like
4

While it is possible today, there is a difference in providing a paved path which advertises and endorses it which could lead to more packages doing it rather than vendoring, making the problem worse.

I'm also skeptical of using existing packages not being leftpadst safe as justification. Before the linked post was shared with me, I was wondering if it would be within our compatibility guarantees to block build scripts from accessing the network without an edition changing a setting default because of the assumption that this shouldn't be happening, especially for registry packages. It would help the discussion to have concrete examples to understand the need, both in terms of how critical it is and how widespread it is.

3 Likes
5

I'm one of those that has a build.rs that downloads an external file if it isn't already present -- I do check the hash. In my case the build script downloads a PDF, and does something like OCR (except it knows which characters are rendered where, rather than having to recognize characters by their shapes), and then spits out some XML with some of the info extracted from the PDF.

1 Like
6

What is the use case? Is there a licensing limitation for distributing the xml but its fine to generate it?

7

Looks like https://libre-chip.org/OPF_PowerISA_v3.1C.pdf indeed requires accepting the EULA to be allowed to redistribute it.

1 Like
8

I think there should be a clear separation of (1) "get all the data needed to build this program" and (2) "build it in a reproducible, deterministic way looking only at this data and nothing else". It's unfortunate that build.rs fulfills both roles today.

So, (1) would comprise things like, copying system headers, downloading things from the internet

And (2) should run in some micro VM or container or something that enforces a clear boundary: it can't read outside files, nor can't access the Internet.

The only issue is what to do with system tools and user installed tools that would make the build non deterministic, like calling the C compiler. My idea is that (1) should be tasked with referencing all the tools that (2) might call, then they should be available in the VM/container. This would enable a Cargo subcommand that packages all files, programs and data required to build the program in a fully deterministic way, plus the hashes of all artifacts. If you distribute that, it would be guaranteed that the resulting artifacts are identical, byte for byte, and thus the hashes must match regardless of where you build. If the hashes don't match, we guarantee there is a bug somewhere.

Maybe there should be a flag in Cargo.toml to deprecate the old build.rs and create new files for both (1) and (2) (say, build_setup.rs and build_execute.rs or something)

Then in a new Rust edition, the new build system should be made the default. And in a much later Rust edition (if at all), the old build system could be made unavailable, and the setting removed.

1 Like
9

I assume you are referring to this text:

By downloading or using the POWER® Instruction Set Architecture (“ISA”) Specification, you agree to be bound by the terms and conditions of the OpenPOWER Power ISA End User License Agree- ment (EULA).

Leaving aside the enforceability of a "contract" that applies before you even seen it, this means the crate is having users download the document, binding them to the EULA without their awareness. That sounds a bit problematic.

10

the use case is that the crate uses the pdf to get the official instruction definitions, so I'd rather have it use the original source rather than distribute a generated artifact. also IIRC the XML doesn't include the license since I didn't get around to adding a section for it.

I would have it download from the official website but I figured repeatedly downloading in CI is rather rude and less reliable than mirroring it.

11

It is worth noting that there are considerable number of people who do not have global access to the Internet. For instance, the raw.githubusercontent.com is blocked in my region, which is one of the most common sources for binary artifacts. There are countless frustrating times for me to use a crate when its build.rs hang forever since the artifact cannot be fetched in my network.

I'd appreciate it if there is an "override" approach for downstream users to avoid fetching the hardcoded URL, such as providing an alternative mirror URL or a local file path (it can be unsafe, though).

2 Likes
12

TBH, I think all access from build scripts to things not in the package nor target folders should be blocked by default.

There's no good reason to connect to a SQL database to generate stuff in the build.rs. If you want to do something like that, have a separate tool that connects to it and produces some file that can be checked in (maybe a toml-formatted schema summary), then at build-time if you want to do codegen off that, sure.

But going off-box isn't deterministic, so shouldn't be part of a build.

3 Likes
13

if cargo still verifies the file hash even when overridden to get the file from somewhere else, it's still pretty safe. (e.g. that's why it's fine that many linux distros default to getting their updates through http even though http is insecure, since they verify the cryptographic file hashes and verify the source of the file hashes has been cryptographically signed by a key they trust. that said, http doesn't prevent others from knowing which packages you're installing, so you may prefer https anyway for privacy reasons.)

14

Miri has a crate that does this, but that crate is not meant to ever be put on the registry. So if cargo only applies limitations for crates.io crates, that'd be fine for me.

Specifically, we have a C++ dependency (optional and off-by-default, though planned to become optional and on-by-default eventually), and all ways of including the C++ code into our repo are terrible (submodules, subtrees), so we just fetch the sources from the git repo of that dependency (pinned to a particular commit).

I don't think packages that depend on non-redistributable artifacts should be on crates.io. In Debian terms, that kind of stuff does not belong in main, it belongs somewhere separate where I can avoid accidentally depending on it.

1 Like
15

I like the idea, but two ways to "define arbitrary code that runs on build" wouldn't help if one runs outside of the sandbox/VM/container. In my opinion (1) should be built in a way that a malicious repository cannot execute code outside of the VM (at least when the LSP is sandboxed, too), so changing build.rs should be done in a way that is additionally beneficial to security, for example by having cargo capable of fetching relevant files even if it doesn't come from crates.io or is aware of the Rust ecosystem at all.