Updates on Rust's CI uploads

#1

We’ve implemented a few changes recently that if you download from the rust-lang-ci S3 bucket you may wish to be aware of. Users of rustup or other official tools have no need to worry!

The rust-lang-ci bucket was mistakenly created in the us-east-1 region instead of the us-west-1 region where the rest of our infrastructure resides. Additionally it’s been growing at an alarming rate that we’ve wanted to get a handle on. As a result, there are two changes we’ve implemented:

  • A new bucket, rust-lang-ci2, has been created in the us-west-1 region.
  • A policy has been added to the bucket for each of the folders it contains:
    • rustc-builds - the contents of this folder is deleted after 90 days
    • rustc-builds-alt - like above, contents are deleted after 90 days
    • rustc-builds-try - contents are deleted after 30 days

As of a few weeks ago we have started to upload all artifacts to the rust-lang-ci2 bucket instead of the rust-lang-ci bucket. The uploads are mirrored for now, however, to the rust-lang-ci bucket. It is our intention to disable this mirroring and let existing contents of rust-lang-ci expire naturally after 90 days.

If you’ve got scripts or automation which is downloading from rust-lang-ci, now is the time to update them! You’ll want to download from the rust-lang-ci2 bucket instead. Note that these buckets are intended for temporary artifacts, however, so you may wish to move off the buckets altogether if you can!

Public, Stable Rust Services
#2

(This thread should probably be in the “Announcements” category.)

I was not aware of that. It would have been nice to communicate some more on this important aspect.

Here is the Servo timeline, for what it’s worth.

  • 2015-11-07: Servo start downloading Cargo and Servo from static.rust-lang.org (instead of the custom servo-rust bucket), indexed by Nightly date.
  • 2016-11-28: Last update to static.rust-lang.org/cargo-dist.
  • 2016-12-06: Servo starts downloading Cargo from the replacement instead, rust-lang-ci/cargo-builds, indexed by commit hash. (That directory seems to be gone too, btw.)
  • 2017-02-15: rustc-builds-alt is created with builds of rustc that have LLVM assertions disabled and compile 20~50% faster than Nightly.
  • 2017-02-15: Servo switches to downloading rustc from rust-lang-ci/rustc-builds-alt (by default) or rust-lang-ci/rustc-builds, indexed by commit hash.
  • 2017-05-31: Last update to rust-lang-ci/cargo-builds. Cargo builds are now distributed together with rustc builds.
  • 2017-08-31: Servo starts downloading Cargo from rust-lang-ci/rustc-builds based on the same commit hash as rustc. (Previously they were versioned separately.)
  • 2017-09-12: Servo switches back to storing a nightly date in the repo instead of a commit hash (in rust-toolchain to be compatible with rustup overrides) but still downloads from rust-lang-ci/rustc-builds-alt and rust-lang-ci/rustc-builds

So old Servo versions starting from 2016-12-06 (for example when running git bisect) fail to bootstrap or will soon. I’ll work on switching back to static.rlo as much as possible, but alt builds are not on there yet and we want to keep them the default.

#3

Simon, and the rest of the Servo team: my sincere apologies about this! If there’s work we can do to help you recover, please let me know.

The infra team will conduct a post mortem on what happened here and how to prevent it in the future. The most obvious issue here is that public artifacts were deleted without prior notice, which caused us to miss important stakeholders. We’ll enact policies to prevent that from happening in the future.

We should also discuss the overall vision here. My understanding is that nightlies should be kept in perpetuity; the 90 day window is intended only for PRs going through bors. We should probably consider a bigger window, however.

#4

One thing that would help going forward is a compiler team or infra team decision on alt builds (rustup issue, forum thread). My preferred solution, particularly given this deletion policy, would be to add a new nightly-alt channel with builds available by date on static.rlo.

#5

Agreed, and sorry for the long wait on that – it’s been very difficult to reach lasting consensus among the stakeholders. Now that I’m back from PTO, I’ll try again to bottom it out.

#6

I see that Servo also uses https://s3.amazonaws.com/rust-lang-ci/rust-ci-mirror/openssl-${OPENSSL_VERSION}.tar.gz when building for android. How ephemeral are these URLs?

Edit: https://s3.amazonaws.com/rust-lang-ci2/rust-ci-mirror/openssl-1.0.2k.tar.gz responds with HTTP/1.1 301 Moved Permanently but no Location header. I’ll leave that one in the old bucket for now.

Edit 2: https://s3-us-west-1.amazonaws.com/rust-lang-ci2/rust-ci-mirror/openssl-1.0.2k.tar.gz works