Which CI platform should Rust use?

[ghuntley]

Folks may not realise but Martin is being kinda modest here. He’s the founder of the .NET foundation, one of the key folks behind the open-source culture change and a high ranking individual within the Microsoft organisation chart. ie. he can open doors and make shit happen for rust that others cannot. He’s also a standup guy. In case you’re wondering, I do not work for microsoft.

https://www.linkedin.com/in/martinwoodward/

[bascule]

I’ll mention Google Cloud Build, however noting it doesn’t provide native macOS support, which is probably a showstopper (it does provide Windows support).

Namely, Cloud Build is “Docker native”, and there is presently no first-class solution for running macOS inside of Docker (at least that I’m aware of). Solve that problem (here’s a science experiment which attempts to do that) and it would meet the requirements.

Other than that it would seem great given the other requirements:

• I’ve been pretty pleased with GCP support and am in frequent contact with my account executives/reps (via email and telephone). Technical / tier 2 support requires purchase of a support package but is also available via phone or support tickets.
• Highly scalable (up to 32-core machines)
• Configurable timeouts
• Docker native: each step in a build can be a different Docker container, and they share a common filesystem mount throughout a build. This allows builds to be composed in terms of different Docker containers that each do one particular job.
• Google isn’t going to disappear any time soon (but I can’t say the same about any given Google product, they don’t have the greatest track record there)

The main con (other than lack of native macOS support, and Google being known to capriciously kill products) in my experience has been the UI/UX: it feels a bit half-baked. but that’s kind of par for the course with much of GCP, and also Cloud Build is a relatively new product which is about 1½ years old. I’ve brought it up with my account reps and been told it’s a common complaint. I’ve seen UI/UX improvements arrive in other parts of the platform, and can only hope similar improvements can come soon, but until then it feels pretty rough around the edges compared to other CI/build systems.

That said, the core functionality is very solid, and most things can be done through (highly flexible and fully-featured) configuration files and API calls.

[drahnr]

Concourse CI

https://concourse-ci.org

Concourse is the internal deployment strategy of pivotal which is part of the couldfoundry initiative/group and is available as an open source project and is part of their cloudfoundry offering.

It is essentially a thing-doer, a meta system where resources can be defined and used as triggers on which jobs - a collection of serial, parellel, failable, actionable tasks are execute on workers. Each worker can either be windows, linux or mac os. The whole concept is built around containers, each and every task has an container based environment which as of today is mostly docker based but can be extended to whatever is needed. Concourse has a directory level cache support, but it is also possible to integrate external services with it, such as sccache/redis.

I’ve been using concourse 2 throughout 4 for open source as well as work environments and while not being a devops at heart, it was fairly easy to get into it.

A few distinctive key features are:

• being able to beam yourself into the build container and see what went wrong in there, poke around - reduces turn around time for fixing a broken test environment
• command line utility fly to update the pipeline, independently of commits/git
• credential management via vault or injectable json, thus the descriptive yaml can be version with git
• graphical pipeline representation and dashboard (see i.e https://ci.spearow.io )
• monitoring via influxdb/riemann/prometheus possible …

I missed a few things, but I am confident that this is something that could potentially fit the needs.

The components are loosely coupled, i.e. the container runtime can be change as an argument for the worker. The whole thing is written in go as of today.

downside

• I have no experience using a hosted version, so far I always spun up my own network/nodeset

[aidanhs]

I’ve posted a new thread here to talk a little bit about our investigation process after all of these great suggestions.

[matthieum]

News Flash: after acquiring Travis CI about ~30 days ago, Ibera just sent a massive wave of termination letters to Travis CI employees.

There is no mention of shutting down Travis CI, or shutting it down for Open Source projects, however the loss of experienced employees is unlikely to help stabilizing the platform.

[VitalyAnkh]

I have a bold idea: how about building a CI platform in Rust on our own? Maybe the rust infra team will start a company and make money from it (for the strength of Rust, the platform must be very excellent). I don’t have experience in build CI platforms so it’s just an idea.

[mark-i-m]

@VitalyAnkh This is actually less bold than you might expect Discussion (continued): Building our own CI

[VitalyAnkh]

@mark-i-m Cool! Looking forward to more work.

[skade]

I have a bold idea: how about building a CI platform in Rust on our own? Maybe the rust infra team will start a company and make money from it (for the strength of Rust, the platform must be very excellent). I don’t have experience in build CI platforms so it’s just an idea.

I find it quite an assumption that we could quickly form a company and build a consumer-usable platform around our project. Rust has some very specific needs that don’t map what the general audience needs.

Also, our tooling doesn’t need to be excellent, it needs to work. A lot of our tooling is surprisingly unsophisticated at close look. That’s fine in an org that knows the limits, but it would hit the ceiling quick if you’d try to generalise its use.