Perfecting Rust Packaging - The Plan

This is being worked on in Need ability to add dependencies based on `#[cfg()]` · Issue #1007 · rust-lang/cargo · GitHub / https://github.com/rust-lang/rfcs/pull/1361

This was a reason I suggested the possibility of adding a make dist style cargo command, especially if the other option on the table is each distro manipulating the Cargo.toml which I have no interest in managing.

I use -Wl,-z,relro,-z,now in my build of Rust for Yocto. Its a fork from a previous repo and is a bit of a mess but I’m working to clean it up. The repo is called meta-rust.

I’ve actually landed a patch in Rust master to treat i386, i486, i586, and i686 in mk/platforms.mk recently. It had already done i386 and i686. Yocto also targets i586 and the meta-rust I cloned my repo from had been patching that spot for some time.

Sorry for interfering. Any plans for two specific things:

  1. Binding to custom crate repositories?
  2. Allowing to release pre-built artifacts for crates?

Thanks

@gus Thanks for working through all that and giving us the details. Next week the Rust team is meeting in person and we’ll try to regroup to understand the problems you are dealing with, how Cargo can ease them.

@target_san

  1. It is possible to use other repos than crates.io, though not super tested in the wild. See this unit test, which frobs registry.index in .cargo/config. @alexcrichton says you might also want to look at the implementation of cargo-vendor.

  2. I don’t think there are specific plans for releasing binaries, though it’s known to be a desirable feature.

It'd be helpful to have something that can reproduce some of the issues mentioned -- like Cargo talking to crates.io despite everything being tied to paths. @gus, is the work you've done something you could easily send us in a tarball?

Some updates.

@gus, @alexcrichton, @lucab and I had a quick video call today to discuss Debian’s progress. The notes are available though I guess they will be pretty hard to understand.

Some of the major points:

  • Cargo really needs better support for sourcing registry crates from the local filesystem; path rewriting is too painful. Issue.
  • Debian’s tools for scraping version updates from websites don’t understand crates.io’s pages. I’ve filed an issue, but don’t know enough about it to say what needs to be done yet. Somebody more knowledgeable might fill in the details.
  • The Rust trademark policy’s inclusion of the sentence “This document supplements the official Mozilla trademark policy which governs use of all Mozilla trademarks” is a huge source of confusion. We consider this a bug and are trying to fix it, but working with the lawyers is slow.
  • We think that sourcing packages from crates.io tarballs is the way to go, not to try to trace the source back to git.

While I haven’t been doing a lot of coding yet on the bugs to come out of these discussions, I have started a patch to teach cargo to interpret a RUSTFLAGS environment variable, ala CFLAGS.

5 Likes

A somewhat tangential point, but I've seen two instances of people who needed to be able to locally mirror the index and packages. I've also personally set up a local mirror of the index because constantly hitting the network (and the occasional outage) are a tremendous pain in the ass.

The major problem is that this makes it impossible to work on public binary packages, because changing the index alters the paths in Cargo.lock. It would be really nice if whatever is worked out here helps with the above.

1 Like

I've just realised, for maximum compatibility on 32-bit x86, the stage0 rustc binary could be made into the generic i686 version.

There's been a new stage0 snapshot release (2015-12-18) and the i686 version is unchanged. Have it your way.

Thanks for bringing this up again. If there is a change you are looking for here perhaps you can open an issue to suggest it. I don't recall all the context, and changing the snapshot configuration hasn't been on my radar.

FWIW though, we are planning on removing snapshots completely in favor of using the releases, so any change in snapshots may be short lived.

Ok, thanks for replying. There was never an issue about it so it must have slipped @alexcrichton’s mind.

The idea was to enable the snapshot to be run on pre-P4 hardware too, keeping the default code-generation setting unchanged. Completely transparent but still preserving the option of modifying -C target-cpu= at build-time.

Hey folks.

I’ve gone through and done a quick summary of the status of each of the issues identified in this thread. The TL;DR is that while several of the issues have been fixed, there’s a lot of work left to do. I have not put much focused effort into these myself.

I think the most notable development is that and @alexcrichton and @wycats agree in principal to adding a new “local registry”, where the packages live on the local filesystem instead of crates.io. This was originally called a “non-goal”, and we expected distros to use path-rewriting to redirect dependencies locally. In @gus’s experience that solution was quite unpleasant, and we want to provide official support for sourcing packages locally.

Task: Compiler command line customization in the Rust makefile

Issue: https://github.com/rust-lang/rust/issues/29554

Status: Fixed. Thanks @bltaveras

Task: Bootstrapping from previous releases

Issue: https://github.com/rust-lang/rust/issues/29555

Status: No progress. @alexcrichton’s build system rewrite though removes snapshots, which is part of the solution. This should be solved sooner than a build system rewrite.

Task: Re-bootstrapping from the current release

Issue: https://github.com/rust-lang/rust/issues/29556

Status: No progress.

Task: Bootstrapping unstable code from a stable compiler

Issue: https://github.com/rust-lang/rust/issues/295577

Status: I’ve put some thought into it, solicited opinions about how to do it, and scrapped a partial patch.

Task: Dynamic LLVM

Issue: https://github.com/rust-lang/rust/issues/29558

Status: Fixed. Thanks @DiamondLovesYou

Task: Add an i586-unknown-linux-gnu target spec

Issue: TODO

Status: Still don’t know the right solution for this. Debian’s default i386 target is older than Rust’s. For now it’s a Debian problem.

Task: Teach the makefile to mix in additional ‘extra filename’ information

Issue: https://github.com/rust-lang/rust/issues/29559

Status: PR open

Task: Ensure that SxS installation of crates from multiple compilers works reliably.

Issue: https://github.com/rust-lang/rust/issues/29560

Status: No progress.

Task: Fix --libdir issues

Issue: https://github.com/rust-lang/rust/issues/29561

Status: Fixed. Thanks @wthrowe

Task: Disambiguate system-installed crates during resolution

Issue: https://github.com/rust-lang/rust/issues/16402

Status: No progress.

Task: Update Homebrew packages

Issue: https://github.com/rust-lang/rust/issues/29562

Status: No progress.

Task: Produce packaging guidelines

Issue: https://github.com/rust-lang/rust/issues/29563

Status: No progress.

Task: Publish source tarballs of Cargo releases

Issue: https://github.com/rust-lang/cargo/issues/2107

Status: No progress.

Task: Pair Cargo releases with Rust releases

Issue: https://github.com/rust-lang/cargo/issues/2108

Status: Partially fixed. The Cargo that Rust releases are paired with is tagged with a release number. Still don’t have the automation up to make Cargo release artifacts.

Task: Make binary releases of Cargo

Issue: https://github.com/rust-lang/cargo/issues/2109

Status: No progress.

Task: Validate that Cargo releases build with the corresponding Rust release

Issue: TODO

Status: No progress.

Task: Bootstrap Cargo without Cargo

Issue: https://github.com/rust-lang/cargo/issues/2110

Status: No progress. I’m having second thoughts about doing this at all though. ISTM that once a distro has a binary version of Cargo they can build future

Task: Enable Cargo to work without any network access

Issue: https://github.com/rust-lang/cargo/issues/2111

Status: No progress, though there’s a new issue about making Cargo work with local registries instead of crates.io.

Task: Compiler command line customization

Issue: https://github.com/rust-lang/cargo/issues/2112

Status: PR open.

2 Likes

@brson you've got a typo in the issue link, should be Bootstrapping unstable code from a stable compiler · Issue #29557 · rust-lang/rust · GitHub

Is i586 really still needed for Debian? A thread from September suggests not.

Unfortunately, i686 (Pentium Pro etc) is still too old for Rust’s default configuration, which wants SSE2 instructions (introduced Pentium 4). I think this was discussed upthread.

1 Like

Debian just dropped 586 support

In that case, they can either modify a single line before bootstrapping their i686 rustc and make it x87 for everyone, or, and that's what I'd probably do, make just the compiler runtime generic i686 and ship two sets of i686 crates.
Default to the newer one, instruct the user to pass a separately named --target= in case produced code doesn't run.

Or maybe detect the required version at installation time and download just the right i686 set.

Hey everyone,

I’ve been working on cleaning up and improving the rustc and cargo builds for openSUSE the last few days, and as part of that I ended up working on improving the cargo bootstrap.py script by @dhuseby. I’ve got a patchset in development here, comments / ideas are more than welcome.

Right now I think I have a rough idea of how we could package crates and rust projects for openSUSE - there’s still a lot of work left to do, but I also thought it might be valuable to share my main inspiration, which is the golang packaging macros for rpm developed for packaging golang modules for openSUSE here.

Here’s a question, though: From 1.10, each rustc version willl be bootstrappable from the previous release (so 1.10 with 1.9, 1.11 with 1.10 presumably, and so on). This is good news, but I’m wondering - are you also committing to ensure that rustc 1.10 is bootstrappable with 1.10? If that’s the case, it simplifies things even more…

@krig The self-bootstrap is #29556. I have a WIP patch for this, which I’ll try to post tonight or tomorrow.