My post earlier this week generated a lot of feedback, much of it negative. I really appreciate the detailed thoughts that people gave, which drew out many drawbacks and outlined some simpler alternatives. I want to summarize some of that feedback, then revisit what I see as the core goals. I’ll catalog some of the alternative ideas that people have proposed, as well as a couple I’ve been mulling over.
First, to recap the original proposal: the main thrust was to develop a kind of “extended standard library” made up of curated crates from the ecosystem. The platform would be available as a single “Cargo metapackage” snapshotting a collection of compatible versions, enabling integration testing and documentation cross-referencing. The hope was to provide the experience of a large standard library, without the pitfalls that tend to accompany it.
The feedback
Here are some of the concerns people raised:
-
Let’s start with lower-hanging fruit. A common refrain from the Rust community was: we love the combination of Rust’s minimal core, plus incredibly easy access to the ecosystem through Cargo. If discoverability, integration, or quality are issues, let’s address them by making our existing tools better, e.g. by improving search or ranking within crates.io.
-
Picking winners has significant downsides. The platform would introduce a sharp cliff between the crates it includes and the rest of the ecosystem. That introduces several risks. First, in precisely the cases where it’s most valuable – where there is no clear “winner” in an area – it risks ratcheting up the pressure and contention by trying to centrally decide on one. Second, it reinforces first-mover advantage, which could lead to the same kind of stagnation one sees in large standard libraries. Finally, it’s time consuming, and that time might be better spent working to improve the libraries themselves.
-
The similar Haskell platform is not a rousing success. Many from the Haskell community brought up the recent transition toward Stackage and general feeling that the Haskell Platform had a lot of significant problems. However, there are fundamental differences in the way that Rust’s dependency management works that make it hard to know what lessons to draw. As one Haskell user wrote: “the problem that Haskell Platform created was that only a single version of critical packages could exist in the central store at a time.” Cargo doesn’t impose any such restrictions, and indeed a lot of the details of my original post were precisely about the ability to easily juggle versions. Regardless of what direction we take, though, we should learn where we can from Haskell’s experience.
With those concerns in mind, I want to revisit what we’re trying to achieve, and think broadly about how we get there.
Revisiting the goals
Starting at a high level, the goal in my mind is for the Rust community to:
- Provide commonly-desired libraries and tools;
- Make them easy to find, acquire, and use;
- Make them high quality;
- Make them work well together.
These are all fairly obvious things to work toward, but I want to underscore that lack of maturity in these areas was a major theme in the 2016 Survey, and is also something the core team hears constantly when talking to existing and potential Rust users.
Now, work toward this goal is and will continue to happen organically over time. Maturity takes time, and it’s still early days.
But I think it’s worth looking at the big picture to see how we can rally as a community around these goals, both in terms of building global infrastructure, and in terms of lending muscle to particular tools/libraries. The platform proposal was one attempt to do this, but perhaps an over-reaching one.
So I want to try to focus in on three important things that the platform proposal was trying to do, to look at how else we might get there.
Improving discoverability
“Discoverability” is not just about finding any crate that covers a need. You want to be reasonably confident that the crate is robust, maintained, documented, and that it will work well with the rest of the ecosystem. “Blessing” a small set of crates with these attributes is one way to provide such confidence, but by no means the only one.
Many people responding to the original post proposed improving discoverability on crates.io (or some other site) by providing clearer ranking, and better categorization and search. An example site that already does this for Rust is Awesome Rust.
Of course, finding good heuristics for ranking is a hard problem, and risks cementing first movers. The number of github stars on a project doesn’t necessarily tell you whether it plays well with the ecosystem or is well-maintained. But it’s also conceivable to include a lot of information about a package, e.g. in the form of badges for platform compatibility, or even information about compatibility with other libraries. There’s a spectrum between fully automatic ranking and pure curation, and we can iterate until we find the right balance. Ember Observer was raised as a good example of this kind of mix – though it’s worth noting that the project is an enormous effort.
Improving discoverability on crates.io is forward-compatible with having a small set of “blessed crates” should we decide to go in that direction later on. But it’s a more conservative place to start, and one that benefits the whole ecosystem right away. I think we should do it!
Improving quality
Another aspect of the platform idea was improving quality by focusing attention on a small number of core crates, for which we can provide integration testing, integrated documentation, standardization, and a high level of review/scrutiny.
In at least some of these cases, we might take steps in this direction in an ecosystem-wide way:
-
chriskrycho points out that ongoing work to globally host documentation for crates.io has the potential to provide an integrated docs experience, ecosystem-wide. Ideally, when an API in one crate uses a type from another, the documentation makes that type a link to the other crate. More generally, highest leverage comes from focusing on tools that improve quality across the board.
-
For basic quality measures, like having docs, cross-platform compatbility, CI, and so on, we can use badges or some other way of flagging quality within a crate discovery system, much like Ember Observer.
-
There’s a lot we can do to standardize things like API conventions, documentation, or even the use of Cargo feature flags for crate integration. We did a lot of this kind of work when stabilizing the standard library for 1.0, but there’s no reason we can’t continue to push on it in a more global way. And in fact, there have been recent RFCs doing exactly that for crate documentation. We should be on the lookout for emerging idioms that need standardization, and we should revive efforts to gather guidelines in a central place.
The Rust Ecosystem as a product
One goal of the platform idea was to emphasize that Rust is much more than rustc, Cargo and std. The arrival of a new async IO library can have as much impact as shipping a new language feature. As a community, we need to make sure we’re thinking clearly about this bigger picture, and organizing our efforts accordingly.
I’ve got some thoughts on what that means and how to do it without the “platform” concept, but they really belonged as part of the roadmap discussion. So I’ve added a comment over there.
Where to go from here?
This post was mainly meant to recognize the negative reception, and go back and look at the goals/constraints, exploring some other ideas. I’m hoping we can continue iterating in this design space; I continue to think there’s really important work to be done here.