Perfecting Rust Packaging - The Plan


#4

Not to argue with this, just a FYI: Tests (at least one in libcoretest) will fail once you get rust to compile on i586 hardware. i686 is the earliest hardware with SSE2. That’s not simply a performance boost, it also affects the correctness of floating point calculations: pre-SSE2, basic arithmetic operations are sometimes rounded incorrectly. There has been some discussion about how to deal with this and similar problems, but so far there is neither consensus nor an implementation.


#5

I’m going to say one last time that this is only hard because of your weird decision to unpack and bundle archives found in /usr/lib.


#6

The way that i586 vs. i686 is getting discussed is really confusing. i586 usually refers to something like gcc’s -march=i586. i686 usually refers to something like gcc’s -march=i686. Rust’s default is something like gcc’s -march=pentium4.


#7

I think this is almost possible with paths in .cargo/config – it just needs a specification to look at all subdirectories under .../crates/ rather than having to enumerate all individual crate paths. Maybe just write this as crates = "/path/to/crates/" and ignore crates.io entirely if that’s present.

This idea is still a bit of a kludge, but perhaps nicer than rewriting Cargo.toml.


#8

Certainly nicer! Then a drop-in .cargo/config with the system-wide crate root should work in the source tree of any package that is shippable on crates.io.

You could have a helper script, or a packaging macro, to install that config file into the CWD; I suggest naming it cargo-embargo :smiley:


#9

I haven’t forgotten about this thread. Just working on other infrastructure stuff the last week. Sorry for the delays.


#10

I am wondering if something like make dist would be useful to add to cargo that basically creates an archive of the crate and all its build deps in source form. This “full” source archive can then be distributed on crates.io and used as an upstream source archive that distros would then be able to cache and use to offline cargo build with whatever compiler flags and install for packaging.

I think this is similar to what go does which seems the easiest way to go until dynamic linking is working properly.


#11

I don’t think a full-source bundle is what we want for distros. That makes it really hard to track what crates are built where, in case there’s ever an urgent need to patch and update one.


#12

I agree that it makes patching libfoo used in multiple crates much harder as your cve checking and the like now needs to understand cargo dependencies and patches must be done to all of those crates until upstream releases an update with the fix.

That said quite often upstream releases are the ones making these fixes anyway as a person spending time looking at cve fixes flow in to my distro so it may not be so bad.

Getting cargo install to know a /usr/share/ type location to store rust crates that cargo can use to build for the package install would be much easier to directly patch but I wanted to give other options too.


#13

I’ve just realised, for maximum compatibility on 32-bit x86, the stage0 rustc binary could be made into the generic i686 version. It’s only needed for stage0 and doesn’t affect anything that happens later so it’s a no-brainer really.

In other words, if the target is left unchanged the build process would yield the standard rustc exactly as before whereas a modified i686 target could be built natively on anything pre-P4.


#14

Yes it will be less convenient.


#15

Can you explain this? I don’t believe I’ve ever heard you say this before and I don’t know what you mean.


#16

Is this something we are doing wrong? Does bundling an i586-unknown-linux-gnu target not make sense in your understanding?


#17

The linked issue describes the use case from @jauhien, and passing rustc -L /usr/lib is exactly what they are doing. A point was also made on that thread that rustc can’t even be compiled when it’s already installed (though that is surprising to me…). This is apparently blocking Gentoo though I don’t fully understand their installation model.

Edit: I’ve modified the text to state this is a rustc problem, not cargo.


#18

There’s two ways you can look at it. One is that our current “i686” target is buggy because it doesn’t actually work on all i686-class machines. The other way of looking at it is that using target triples to specify CPU features is a bad idea because we try to attach too many different meanings to the triple, so the right solution is not to add an “i586” triple, but rather add some other mechanism to distinguish CPU features.

Either way, adding an “i586” target to rustc without any other changes would be extremely confusing.


#19

I’ve filed issues on everything but the i585 triple, which @eefriedman has concerns about, and CI for building Cargo with its own release of Rust (since @alexcrichton is cool on the idea and it’s a lot of work for marginal gain).

@gus What do you think of @eefriendman’s concerns that encoding the i586-ness of the plattform in the target triple is inconsistent with gcc, not the right way to encode CPU features?


#20

My understanding of the problem is that there are three cases for linking LLVM: static bundled copy, static system copy, and dynamic system copy. The first is easy to handle because it’s decided by the configure script so we know when we’re in that case and can do whatever we need to, so the only difficulty is handling the two system library cases.

The current solution involves using #[link(..., kind = "static")] for static libraries and #[link(..., kind = "dylib")] (or usually leaving it off because it’s the default) for dynamic ones, which is hard because pkg-config and similar tools don’t tell you whether the library you’re linking against is static or dynamic. The reason they don’t tell you is because in the C world you basically don’t care: in either case you just pass -lfoo when running the linker (there are edge cases that don’t work, but those don’t work with Rust’s method either). As far as I can tell there’s nothing preventing Rust from doing the same thing.

That behavior is what kind = "dylib" produces. kind = "static" searches the system for the archive file, unpacks it, and then bundles all the objects into the resulting rlib (or whatever you’re producing). This is useful if the archive is some bundled library that you don’t want to install separately (like the bundled LLVM case), but it’s a weird thing to do with a system library. Just using the “dylib” behavior for any system library should just work because it matches the C behavior that linking was built up around.

(As a disclaimer I can only comment from a Linux point of view. As I understand it things are more complicated in Windows. There are some more details and discussion of this whole thing (including some Windows stuff) in a previous internals thread.)


#21

@wthrowe thanks for the explanation and link to the other thread. I think I basically understand your critiques of how Rust handles linkage.


#22

I’m afraid I don’t really understand the concern. As far as I understand it, triples are just a handy way to encode a bunch of cpu architecture / ABI / platform options into a single string. My understanding (please correct me) was that gcc can indeed be configured with separate i386/i486/i586/i686-linux-gnu triples, and yes they just map to different default values for -march, -msse, etc. I’m not sure if @eefriedman is arguing that adding more triples is bad, or that adding triples is not sufficient/scalable and we need to add cargo support for passing arbitrary codegen compiler flags. I’d like a triple for my platform, and I’d like the ability to pass arbitrary codegen flags :wink:

I’m fine with adding a more specific i586-debian-linux-gnu, if we want to make it clear that it means only “the abi/cpu features that Debian assumes”. I’m also fine with adding that just to the Debian rustc.deb if there’s resistance to carrying it upstream. Really, I just need something I can pass to rustc/LLVM to get it to produce output that fits the Debian “i386” architecture definition (basically gcc’s i586-linux-gnu), and doesn’t assume the existence of pentium2 instructions. I can pass a bunch of compiler flags around, or I can create the triple that represents those same flags - either way I need enough support/hooks in my cargo executable to be able to do that, and right now that means a new triple.


#23

My primary point is just that a system where “i686-pc-linux-gnu” mean pentium4, and “i586-pc-linx-gnu” mean pentium1 is really confusing. Also, if Debian retires i586 in favor of i686 sometime in the next few years, you’re going to run into trouble because i686 won’t mean what you need it to mean.

On a side note, I’m pretty sure the testsuite will fail on an “i586” target because of differences between SSE and x87 math.