# Perfecting Rust Packaging - The Plan

**URL:** <https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767>\
**Category:** Uncategorized\
**Created:** [October 9, 2015, 11:23pm UTC](https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767 "2015-10-09T23:23:14Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![brson](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/brson/32/5197_2.png) [@brson](https://internals.rust-lang.org/u/brson)\
**Post date:** [November 4, 2015, 10:17pm UTC](https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767/21 "2015-11-04T22:17:42Z")

</div>

@wthrowe thanks for the explanation and link to the other thread. I think I basically understand your critiques of how Rust handles linkage.

---

<div class="post-metadata">

**Author:** ![gus](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/gus/32/2924_2.png) [@gus](https://internals.rust-lang.org/u/gus)\
**Post date:** [November 5, 2015, 1:19am UTC](https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767/22 "2015-11-05T01:19:00Z")

</div>

> [@brson](#):
>
> @gus What do you think of @eefriendman's concerns that encoding the i586-ness of the plattform in the target triple is inconsistent with gcc, not the right way to encode CPU features?

I'm afraid I don't really understand the concern. As far as I understand it, triples are just a handy way to encode a bunch of cpu architecture / ABI / platform options into a single string. My understanding (please correct me) was that gcc can indeed be configured with separate i386/i486/i586/i686-linux-gnu triples, and yes they just map to different default values for `-march`, `-msse`, etc. I'm not sure if @eefriedman is arguing that adding more triples is bad, or that adding triples is not sufficient/scalable and we need to add cargo support for passing arbitrary codegen compiler flags. I'd like a triple for my platform, _and_ I'd like the ability to pass arbitrary codegen flags 😉

I'm fine with adding a more specific `i586-debian-linux-gnu`, if we want to make it clear that it means only "the abi/cpu features that Debian assumes". I'm also fine with adding that just to the Debian rustc.deb if there's resistance to carrying it upstream. Really, I just need _something_ I can pass to rustc/LLVM to get it to produce output that fits the Debian "i386" architecture definition (basically gcc's `i586-linux-gnu`), and doesn't assume the existence of pentium2 instructions. I can pass a bunch of compiler flags around, or I can create the triple that represents those same flags - either way I need enough support/hooks in my cargo executable to be able to do that, and right now that means a new triple.

---

<div class="post-metadata">

**Author:** ![eefriedman](https://avatars.discourse-cdn.com/v4/letter/e/ec9cab/32.png) [@eefriedman](https://internals.rust-lang.org/u/eefriedman)\
**Post date:** [November 5, 2015, 2:15am UTC](https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767/23 "2015-11-05T02:15:38Z")

</div>

My primary point is just that a system where “i686-pc-linux-gnu” mean pentium4, and “i586-pc-linx-gnu” mean pentium1 is really confusing. Also, if Debian retires i586 in favor of i686 sometime in the next few years, you’re going to run into trouble because i686 won’t mean what you need it to mean.

On a side note, I’m pretty sure the testsuite will fail on an “i586” target because of differences between SSE and x87 math.

---

<div class="post-metadata">

**Author:** ![glaebhoerl](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/glaebhoerl/32/1978_2.png) [@glaebhoerl](https://internals.rust-lang.org/u/glaebhoerl)\
**Post date:** [November 5, 2015, 8:58am UTC](https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767/24 "2015-11-05T08:58:14Z")

</div>

i686 should actually be Pentium Pro?

---

<div class="post-metadata">

**Author:** ![brson](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/brson/32/5197_2.png) [@brson](https://internals.rust-lang.org/u/brson)\
**Post date:** [November 16, 2015, 11:41pm UTC](https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767/25 "2015-11-16T23:41:14Z")

</div>

I’ve filed an issue against Rust [to apply `-W,-z,relro` by default](https://github.com/rust-lang/rust/issues/29877). This is one of the reasons for wanting to [apply custom command line arguments to rustc](https://github.com/rust-lang/cargo/issues/2112) (which we still plan to do). I don’t know how feasible it is but it seems like the sort of option Rust might like.

---

<div class="post-metadata">

**Author:** ![gus](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/gus/32/2924_2.png) [@gus](https://internals.rust-lang.org/u/gus)\
**Post date:** [November 20, 2015, 10:33am UTC](https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767/26 "2015-11-20T10:33:23Z")

</div>

Just to update everyone here, I tried to put together a quick proof-of-concept of getting cargo to use local packages. It doesn’t work yet, but I’ve learnt a few things along the way.

In no particular order (or difficulty):

- Even something “simple” like cargo requires a lot of crates 😛 I think I’ve built 37 “library” packages in order to build cargo, and that’s with me cheating a bit and collapsing some dependency chains that specified supposedly incompatible versions of the same crate (but I hack the Cargo.toml dependency and reuse the one version for both).
- Lots of over-specified version requirements across [crates.io](http://crates.io)
- Several crates declare blanket dependencies on winapi/kernel32 that aren’t actually required unless you’re building for a windows target (for example, cargo itself)
- Often the only indication of copyright is the keyword in Cargo.toml entry (no explicit LICENSE text nor copyright comments in readme/source)
- Many upstream git repos don’t actually make releases (or declare git tags, etc) other than the snapshot that happens to get uploaded to [crates.io](http://crates.io). This effectively means we need to build distro packages from what’s in [crates.io](http://crates.io) or else we have no hope of matching the semver dependencies used between packages.
- A number of \*-sys crates ship a full upstream C library source, which they only use as a fallback for when the local platform version couldn’t be found. I hadn’t actually noticed that before, and it makes packaging (only slightly) interesting because now we have a bunch of extra files to either strip out, or audit licenses.
- The good news with distro packaging is that we can use the cross-language distro package dependencies to just make sure we always have pkg-config, the required C library, headers, etc installed giving us a much simpler and more predictable result. In just about all cases, we hit that first pkg-config line in [build.rs](http://build.rs) and the rest is skipped entirely.
- Repackaging from [crates.io](http://crates.io) rather than upstream github isn’t great because:
- Can’t share work between upstream repos that contain multiple crates. Probably can’t do this anyway (easily), since the “sub crates” often declare wildly different crate versions so we wouldn’t be able to do a 1:1 crate version and distro package version.
- A number of files are often missing from what gets uploaded to [crates.io](http://crates.io). In particular, documentation, examples, license files - things that aren’t actually cargo buildables.
- Several “sub crates” assume the original source subdirectory layout, and derive their crate name from the directory. I needed to patch several of these to add explicit Cargo.toml `name=...` directives once I started shipping them in my own (differently named) `$name-$version` directories.
- [crates.io](http://crates.io) doesn’t make it easy to verify sources. There _are_ checksums buried away in the metadata (retrieved via a git checkout), but ideally there’d be a simple http-accessible signature alongside the source download (like there is for rustc source itself).

Some implementation details (all open for discussion, this is mostly just a strawman POC):

I hacked up a quick/horrible python script to fetch source from [crates.io](http://crates.io) and it autogenerates most of the bare minimum debian/\* required to package up a library. The end result is (currently) a debian package named “librust-$crate-dev” that contains the crate source, with a patched `Cargo.toml`. The astute here will notice this package naming scheme implies we can only support a single version of the crate at once, without spawning off more explicitly versioned Debian package names (this is probably something that needs to be changed). The patched `Cargo.toml` has all dependencies rewritten from `libc = "0.1"` to `libc = { path = "/usr/share/rustsrc/libc-0.1" }`. I also rudely truncate all more explicit semver dependencies to 2-digit _x.y_ (to prevent having to update all the Cargo.toml paths all the time), and use the Debian package metadata to preserve the original more-specific version requirement.

When you install the package, the crate source gets dumped into (rather arbitrary) `/usr/share/rustsrc/$crate-$x.$y.$z` with a symlink from `$crate-$x.$y` for Cargo.toml `path` convenience. I made the guess that _x.y_ would be good enough to at least verify the approach, and so far I’ve hit other problems first. Note that I _completely ignore Cargo.lock_ - I’m sort of unclear on whether that’s a bad thing, or expected.

So! I eventually have an example non-library Rust thing that I’m trying to build in this environment (in my case “cargo” itself). The `debian/rules` packaging script currently tries to build this with `CARGO_HOME` set to a temporary (empty) local directory to isolate from the local user’s settings, and I run `./configure --prefix=/usr` (works fine), then `make` (and `cargo build --release` fails).

Some issues:

- As far as I can see, cargo insists on updating the [crates.io](http://crates.io) registry, even though there’s no dependency (afaics) which is using [crates.io](http://crates.io). Should I create a dummy registry checkout in order to prevent this?
- Cargo goes ahead and ignores all my Cargo.toml path-rewriting work, and downloads all the crates from [crates.io](http://crates.io) again, including things like `winapi` that shouldn’t even be in the dependency chain after my edits! Any suggestions on where cargo is getting this idea of the dependency graph from?

Note all of this is all entirely unrelated to the cargo package recently actually added to Debian unstable. That package uses a simpler (working!) build approach - I’m trying to explore the more general challenge of building cargo-using apps without vendoring all the dependent crates.

---

<div class="post-metadata">

**Author:** ![steveklabnik](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/steveklabnik/32/4524_2.png) [@steveklabnik](https://internals.rust-lang.org/u/steveklabnik)\
**Post date:** [November 20, 2015, 3:28pm UTC](https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767/27 "2015-11-20T15:28:50Z")

</div>

> [@gus](#):
>
> Lots of over-specified version requirements across [crates.io](http://crates.io)

I'd be interested in hearing more about this.... if anything, we've had an _under_ specified problem, I'd think.

---

<div class="post-metadata">

**Author:** ![aturon](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/aturon/32/3272_2.png) [@aturon](https://internals.rust-lang.org/u/aturon)\
**Post date:** [November 20, 2015, 9:40pm UTC](https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767/28 "2015-11-20T21:40:17Z")

</div>

> [@gus](#):
>
> Just to update everyone here, I tried to put together a quick proof-of-concept of getting cargo to use local packages. It doesn't work yet, but I've learnt a few things along the way.

Just wanted to say thanks for all the work you've put into blazing the trail here!

---

<div class="post-metadata">

**Author:** ![jethrogb](https://avatars.discourse-cdn.com/v4/letter/j/43a26b/32.png) [@jethrogb](https://internals.rust-lang.org/u/jethrogb)\
**Post date:** [November 21, 2015, 12:13am UTC](https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767/29 "2015-11-21T00:13:24Z")

</div>

> [@gus](#):
>
> Several crates declare blanket dependencies on winapi/kernel32 that aren't actually required unless you're building for a windows target (for example, cargo itself)

This is being worked on in [Need ability to add dependencies based on `#[cfg()]` · Issue #1007 · rust-lang/cargo · GitHub](https://github.com/rust-lang/cargo/issues/1007) / [https://github.com/rust-lang/rfcs/pull/1361](https://github.com/rust-lang/rfcs/pull/1361)

---

<div class="post-metadata">

**Author:** ![bryteise](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/bryteise/32/1072_2.png) [@bryteise](https://internals.rust-lang.org/u/bryteise)\
**Post date:** [November 23, 2015, 10:03pm UTC](https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767/30 "2015-11-23T22:03:53Z")

</div>

> [@gus](#):
>
> Many upstream git repos don't actually make releases (or declare git tags, etc) other than the snapshot that happens to get uploaded to [crates.io](http://crates.io). This effectively means we need to build distro packages from what's in [crates.io](http://crates.io) or else we have no hope of matching the semver dependencies used between packages.

This was a reason I suggested the possibility of adding a make dist style cargo command, especially if the other option on the table is each distro manipulating the Cargo.toml which I have no interest in managing.

---

<div class="post-metadata">

**Author:** ![cardoe](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/cardoe/32/4054_2.png) [@cardoe](https://internals.rust-lang.org/u/cardoe)\
**Post date:** [November 25, 2015, 3:38am UTC](https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767/31 "2015-11-25T03:38:56Z")

</div>

I use `-Wl,-z,relro,-z,now` in my build of Rust for Yocto. Its a fork from a previous repo and is a bit of a mess but I’m working to clean it up. The repo is called [meta-rust](https://github.com/starlab-io/meta-rust).

---

<div class="post-metadata">

**Author:** ![cardoe](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/cardoe/32/4054_2.png) [@cardoe](https://internals.rust-lang.org/u/cardoe)\
**Post date:** [November 25, 2015, 3:44am UTC](https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767/32 "2015-11-25T03:44:14Z")

</div>

I’ve actually landed a patch in Rust master to treat i386, i486, i586, and i686 in [mk/platforms.mk](https://github.com/rust-lang/rust/blob/master/mk/platform.mk#L17) recently. It had already done i386 and i686. Yocto also targets i586 and the meta-rust I cloned my repo from had been patching that spot for some time.

---

<div class="post-metadata">

**Author:** ![target\_san](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/target_san/32/4164_2.png) [@target\_san](https://internals.rust-lang.org/u/target_san)\
**Post date:** [November 26, 2015, 12:49pm UTC](https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767/33 "2015-11-26T12:49:00Z")

</div>

Sorry for interfering. Any plans for two specific things:

1. Binding to custom crate repositories?
2. Allowing to release pre-built artifacts for crates?

Thanks

---

<div class="post-metadata">

**Author:** ![brson](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/brson/32/5197_2.png) [@brson](https://internals.rust-lang.org/u/brson)\
**Post date:** [November 30, 2015, 10:25pm UTC](https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767/34 "2015-11-30T22:25:06Z")

</div>

@gus Thanks for working through all that and giving us the details. Next week the Rust team is meeting in person and we’ll try to regroup to understand the problems you are dealing with, how Cargo can ease them.

@target_san

1. It is _possible_ to use other repos than [crates.io](http://crates.io), though not super tested in the wild. See [this unit test](https://github.com/rust-lang/cargo/blob/master/tests/support/registry.rs#L29-L47), which frobs registry.index in .cargo/config. @alexcrichton says you might also want to look at the implementation of [cargo-vendor](https://github.com/alexcrichton/cargo-vendor).

2. I don’t think there are specific plans for releasing binaries, though it’s known to be a desirable feature.

---

<div class="post-metadata">

**Author:** ![aturon](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/aturon/32/3272_2.png) [@aturon](https://internals.rust-lang.org/u/aturon)\
**Post date:** [November 30, 2015, 11:11pm UTC](https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767/35 "2015-11-30T23:11:27Z")

</div>

> [@brson](#):
>
> @gus Thanks for working through all that and giving us the details. Next week the Rust team is meeting in person and we'll try to regroup to understand the problems you are dealing with, how Cargo can ease them.

It'd be helpful to have something that can reproduce some of the issues mentioned -- like Cargo talking to [crates.io](http://crates.io) despite everything being tied to paths. @gus, is the work you've done something you could easily send us in a tarball?

---

<div class="post-metadata">

**Author:** ![brson](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/brson/32/5197_2.png) [@brson](https://internals.rust-lang.org/u/brson)\
**Post date:** [December 16, 2015, 12:25am UTC](https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767/36 "2015-12-16T00:25:21Z")

</div>

Some updates.

@gus, @alexcrichton, @lucab and I had a quick video call today to discuss Debian’s progress. The [notes are available](https://public.etherpad-mozilla.org/p/rust-packaging) though I guess they will be pretty hard to understand.

Some of the major points:

- Cargo really needs better support for sourcing registry crates from the local filesystem; path rewriting is too painful. [Issue](https://github.com/rust-lang/cargo/issues/2212).
- Debian’s tools for scraping version updates from websites don’t understand [crates.io](http://crates.io)’s pages. I’ve [filed an issue](https://github.com/rust-lang/crates.io/issues/238), but don’t know enough about it to say what needs to be done yet. Somebody more knowledgeable might fill in the details.
- The Rust trademark policy’s inclusion of the sentence “This document supplements the official Mozilla trademark policy which governs use of all Mozilla trademarks” is a huge source of confusion. We consider this a bug and are trying to fix it, but working with the lawyers is slow.
- We think that sourcing packages from [crates.io](http://crates.io) tarballs is the way to go, not to try to trace the source back to git.

While I haven’t been doing a lot of coding yet on the bugs to come out of these discussions, I have started a patch to teach cargo to interpret a `RUSTFLAGS` environment variable, ala `CFLAGS`.

---

<div class="post-metadata">

**Author:** ![DanielKeep](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/danielkeep/32/1332_2.png) [@DanielKeep](https://internals.rust-lang.org/u/DanielKeep)\
**Post date:** [December 16, 2015, 4:02am UTC](https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767/37 "2015-12-16T04:02:43Z")

</div>

> [@brson](#):
>
> Cargo really needs better support for sourcing registry crates from the local filesystem; path rewriting is too painful. Issue.

A somewhat tangential point, but I've seen two instances of people who _needed_ to be able to locally mirror the index and packages. I've also personally set up a local mirror of the index because constantly hitting the network (and the occasional outage) are a _tremendous_ pain in the ass.

The major problem is that this makes it impossible to work on public binary packages, because changing the index alters the paths in `Cargo.lock`. It would be _really_ nice if whatever is worked out here helps with the above.

---

<div class="post-metadata">

**Author:** ![PeteVine](https://avatars.discourse-cdn.com/v4/letter/p/ce7236/32.png) [@PeteVine](https://internals.rust-lang.org/u/PeteVine)\
**Post date:** [December 23, 2015, 6:38pm UTC](https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767/38 "2015-12-23T18:38:20Z")

</div>

> I've just realised, for maximum compatibility on 32-bit x86, the stage0 rustc binary could be made into the generic i686 version.

There's been a new stage0 snapshot release (2015-12-18) and the i686 version is unchanged. Have it your way.

---

<div class="post-metadata">

**Author:** ![brson](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/brson/32/5197_2.png) [@brson](https://internals.rust-lang.org/u/brson)\
**Post date:** [January 11, 2016, 9:14pm UTC](https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767/39 "2016-01-11T21:14:50Z")

</div>

> [@PeteVine](#):
>
> There's been a new stage0 snapshot release (2015-12-18) and the i686 version is unchanged. Have it your way.

Thanks for bringing this up again. If there is a change you are looking for here perhaps you can open an issue to suggest it. I don't recall all the context, and changing the snapshot configuration hasn't been on my radar.

FWIW though, we are planning on removing snapshots completely in favor of using the releases, so any change in snapshots may be short lived.

---

<div class="post-metadata">

**Author:** ![PeteVine](https://avatars.discourse-cdn.com/v4/letter/p/ce7236/32.png) [@PeteVine](https://internals.rust-lang.org/u/PeteVine)\
**Post date:** [January 11, 2016, 10:13pm UTC](https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767/40 "2016-01-11T22:13:15Z")

</div>

Ok, thanks for replying. There was never an issue about it so it must have slipped @alexcrichton’s mind.

The idea was to enable the snapshot to be run on `pre-P4` hardware too, keeping the default code-generation setting unchanged. Completely transparent but still preserving the option of modifying `-C target-cpu=` at build-time.

[Previous page](https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767.md?page=1)

[Next page](https://internals.rust-lang.org/t/perfecting-rust-packaging-the-plan/2767.md?page=3)
