Proposal: The Rust Platform

[tomaka]

Why are community crates holding back from 1.0?

As for glutin, the reason why it’s not 1.0 is that I have actually no idea what I’m doing when working on it. I started the library so that there’s at least something. At the time (something like early 2014 I think) the only existing alternative was glfw-rs, which was very tedious to use. Again, the lack of manpower.

Glium depends on glutin, so it can’t be 1.0 either. All my other libraries are largely unfinished.

[anp]

I can speak for hyper: HTTP is a complex problem, and iterating on the design means breaking changes. Especially as the community tries to figure out non-blocking IO. I’m also usually working alone, with occasional contributions (much appreciated!), and in my free time. This makes iterating take longer. A “platform” wouldn’t make me tag 1.0. It still needs plenty of work.

As for glutin, the reason why it’s not 1.0 is that I have actually no idea what I’m doing when working on it.I started the library so that there’s at least something. At the time (something like early 2014 I think) the only existing alternative was glfw-rs, which was very tedious to use. Again, the lack of manpower.

I think it’s definitely worth collecting input from other community crate authors, but it’s starting to sound to me like a significant reason holding back community 1.0’s is that there’s not enough effort directed at contributions to them, and that they haven’t done enough iteration on their API. Is there something (either alongside or in place of the platform proposal) that the Rust teams could be doing to funnel more contributions and dev hours towards these important community crates?

[steveklabnik]

Is there something (either alongside or in place of the platform proposal) that the Rust teams could be doing to funnel more contributions and dev hours towards these important community crates?

The way that I see it is that we need to grow in general. It’s not like we have a ton of people with spare time that don’t know where to put it; everyone I see in Rust world is working extremely hard at shipping all kinds of awesome stuff. The best way to lighten the load is to get even more Rustaceans on board, which is why the focus on new users and growth is important.

[ruudva]

Can we provide a “batteries included” experience for Rust that doesn’t lead to stagnation, one that instead works directly with and through the ecosystem?

We could start by automatically hosting documentation for crates on crates.io, and if there ever is a set of “endorsed” crates, host their documentation directly on doc.rust-lang.org.

Speaking from Haskell experience, Hoogle is amazing. Part of what makes it so great is that it doesn’t only search through the prelude, but also through core third-party packages. All API reference searches can be served from a single endorsed place, and you can do a cross-package search from that same place. And Hackage serves API documentation for the entire ecosystem. Currently most Rust crates have online API documentation available, but cross-referencing across crates is limited, and there is no uniform search. Having all documentation in one place like Hackage with Hoogle is a game changer.

The biggest open question about what I described above is: how does curation work?

A small feature suggestion for crates.io that I think could help a lot: allow users to “vote for” or “star” a crate, similar to voting in the AUR, starring a repository on GitHub, or starring/+1’ing an issue in a bug tracker. The current metrics to judge packages by, downloads and dependees, are not necessarily good proxies for recommending crates. Downloads in particular is heavily influenced by the number of dependent (popular) crates and by the time the crate has been around. (Though that would be true for votes as well.)

[briansmith]

If you look at crates.io lots (most?) of the crates there haven’t hit 1.0 yet. This is a signal to many developers that Rust simply isn’t ready for prime-time. These programmers don’t want to spend time messing with APIs that are breaking underneath them. They’re too busy trying to solve their actual problem instead of dealing with maintenance issues. Once they see this, Rust is a non-starter.

There is no incentive as a library author to publish a 1.0 crate. When you publish 1.0, people expect you to start mangling your code to maintain backward compatibility. Yet, there’s no benefit to the library author for doing that extra work. Thus, publishing a 1.0 crate is a non-starter for the library authors.

There’s more to it than just the version numbers, though they’re important. Rust also needs a shared vocabulary. We’re seeing issues where there are lots of ways to do something. While this is great in a way because there’s a lot of choice, it’s also is a drag on developer time trying to find the right one.

Could you point to some study, even an informal one, that shows this is true? In my experience, it isn’t a significant timesink.

For example, by having an established set of traits and capabilities we build networking on, we can build even more sophisticated networking functionality on top it because we know it’s stable and supported. Of course, replace “networking” with a variety of common programming tasks.

Conversely, I’m looking forward to people building new replacements (perhaps the same crates refactored, perhaps new crates) for almost all the currently-commonly-used networking libraries. The current generation is useful, but I think there are significant improvements that can be made.

With that shared vocabulary, we can do things like write guides and tutorials that assume these features are available. This lets us make much better starting examples help users quickly solve real problems. This saves them time and gains the community another Rust developer in the process.

We don’t need the Rust teams to spend time on that. The early adopters of the various libraries do this and will do this. This is relatively easy to do and is a good way to start contributing to the Rust community. Conversely, it’s much harder for a newcomer to add alignas to Rust or add 64bit * 64bit -> 128bit multiplication intrinsics to Rust or do other things that the Rust teams are really better suited at doing.

give crates a reason to hit 1.0

To the extent that this gives crates an incentive to publish 1.0, it seems coercive (no doubt unintentionally) and not in the best interests of the library maker. It would be better to find incentives that actually benefit the library makers (e.g. money).

[ruudva]

If you look at crates.io lots (most?) of the crates there haven’t hit 1.0 yet. This is a signal to many developers that Rust simply isn’t ready for prime-time. These programmers don’t want to spend time messing with APIs that are breaking underneath them. They’re too busy trying to solve their actual problem instead of dealing with maintenance issues. Once they see this, Rust is a non-starter.

I think in many cases this is not actually an issue with maturity. In my opinion, many authors are simply too conservative regarding versioning. At the time of writing, none of the most-downloaded crates on crates.io are stable, yet they are the most widely used and battle-tested crates out there. 1.0 doesn’t have to be perfect. There’s no shame in releasing 2.0. Or 52.0 for that matter.

Are there decentralized alternatives to provide incentives to release a 1.0?

Make Cargo colour version numbers < 1.0 red and version numbers >= 1.0 green. Although that might actually scare people away from unstable packages instead of nagging authors. Perhaps only colour the version number for the root crate?

[steveklabnik]

We could start by automatically hosting documentation for crates on crates.io,

This project is already largely underway, see

User interface

[aturon]

Thanks, everybody, for the excellent feedback so far – both here and on various other forums. I’ve been reading and digesting all the comments.

To state the obvious, it’s quite clear that the overall response to the proposal as written is negative. Which is fine! I think people are raising a lot of good points. On the other hand, there is definitely room for improvement in the areas under discussion (e.g. discoverability, maturity, interoperability), and many people have been proposing lighter-weight approaches for doing so.

Given the large number of comments here and elsewhere, and the repeated themes, I’m not going to try to respond individually. Instead, I’m going to put together a short follow-up post, gathering the downsides people have pointed out, looking more closely at the goals, and summarizing some of the alternatives being proposed.

[briansmith]

The way that I see it is that we need to grow in general. It’s not like we have a ton of people with spare time that don’t know where to put it; everyone I see in Rust world is working extremely hard at shipping all kinds of awesome stuff. The best way to lighten the load is to get even more Rustaceans on board, which is why the focus on new users and growth is important.

The way it is described above seems like this would attract organizations that don’t want to contribute significant effort towards maintaining the libraries they use–i.e. freeloaders. But, that’s not the kind of organization that we really need right now, in the world of Rust library development. We need organizations that are willing to get their hands dirty and improve stuff with the people who are building the libraries, or who are willing to at least sponsor the maintenance of the libraries in a meaningful way.

I was just talking with a friend that does a lot of node.js stuff, and he said almost all the negativity in his project comes from people who are demanding fixes be backported and/or new NPM releases of his library be made immediately, and who never contribute positively. I’ve seen how that works myself firsthand. That’s not a position I want to encourage Rust library authors to get into.

[nugend]

This is exactly what I was going to type up but even more eloquently said. I would add that if the Rust language committee or whoever would be in charge of such a list sees a gap in tooling or libraries or what have you, then there should be empowerment to fill it in an appropriate way until something else supersedes it. By simply having a well curated list that gets a bit of official support like documentation and the like, no one who is just upgrading a version gets put out by something no longer working as expected (say behind a corporate firewall). They will have to figure out the normal path to get their libraries and when the community starts preferring a different library, they still get the old library they were pleased as plums with.

If the issue is how to make it easier to get libraries for people behind firewalls (or some otherwise restricted environment), then address that issue explicitly.

[jntrnr]

@briansmith

There is no incentive as a library author to publish a 1.0 crate. When you publish 1.0, people expect you to start mangling your code to maintain backward compatibility.

While we should definitely also talk about ways to encourage crates to stabilize, I’d like to at least make sure we fully understand the end-user experience before switching topics. Both are important and need to be understood.

You also seem to have a very strong take against crates.io as a concept, at least insofar as it becoming a place that people can find stable, supported packages. I think that’s a concern we’re trying to address, and I’d love to hear your thoughts in terms of how best to address it.

Could you point to some study, even an informal one, that shows this is true? In my experience, it isn’t a significant timesink.

While I’m not sure I understand what you’re asking about specifically here, my comment is more about the basic mechanics. It’s building out of composable parts. If the crates don’t work together well, you spend time stitching them together. Having the shared vocabulary also means we spend more time building the new thing rather than recreating the old thing.

The survey results underscore this. The need to cover major programming areas like UIs, networking, async I/O, etc are the specific areas that need strong support.

To the extent that this gives crates an incentive to publish 1.0, it seems coercive (no doubt unintentionally) and not in the best interests of the library maker.

Are you saying here that you’re worried that library authors will rush to hit 1.0 before their ready? That’s definitely a valid concern, though I hope some of the curation ideas mentioned here help ameliorate it.

[briansmith]

There is no incentive as a library author to publish a 1.0 crate. When you publish 1.0, people expect you to start mangling your code to maintain backward compatibility.

While we should definitely also talk about ways to encourage crates to stabilize, I’d like to at least make sure we fully understand the end-user experience before switching topics. Both are important and need to be understood.[/quote]

Just to be clear, I wrote this because you said this proposal was intended to provide incentives for crates to get to 1.0.

You also seem to have a very strong take against crates.io as a concept, at least insofar as it becoming a place that people can find stable, supported packages. I think that’s a concern we’re trying to address, and I’d love to hear your thoughts in terms of how best to address it.

As a security-focused engineer I don’t think things like NPM are good practices from a security standpoint. I believe in vendoring dependencies. However, I don’t have anything against crates.io and my crates will probably show up there soon.

While I’m not sure I understand what you’re asking about specifically here, my comment is more about the basic mechanics. It’s building out of composable parts. If the crates don’t work together well, you spend time stitching them together.

I think that is true in the abstract, but I would be surprised if it were what’s holding people back from using Rust. It would be good to see some concrete examples of where this is causing real problems.

[quote]To the extent that this gives crates an incentive to publish 1.0, it seems coercive (no doubt unintentionally) and not in the best interests of the library maker.

Are you saying here that you’re worried that library authors will rush to hit 1.0 before their ready? That’s definitely a valid concern, though I hope some of the curation ideas mentioned here help ameliorate it.

Publishing a 1.0 creates more work for the library author in terms of support costs and maintaining backward compatibility and whatnot. (Otherwise, the version numbers are meaningless.) Including a library in a curated set on the condition that they publish a 1.0 means they have to choose between being doing that additional work or being an outcast. Both are bad choices for people that just want to keep making the code better. It would be great to find other incentives.

[withoutboats]

Publishing a 1.0 creates more work for the library author in terms of support costs and maintaining backward compatibility and whatnot. (Otherwise, the version numbers are meaningless.) Including a library in a curated set on the condition that they publish a 1.0 means they have to choose between being doing that additional work or being an outcast. Both are bad choices for people that just want to keep making the code better. It would be great to find other incentives.

I don’t understand this reasoning. Not endorsing a library is not some sort of punishment (surely someone who doesn’t manage such a crate is not an ‘outcast’), and the Rust team has the right to define the qualities of a library which are the minimum for endorsement, and stability and backward compatibility are very reasonable qualities to require.

There are existential questions about why anyone would want to maintain an open source libraries, but those seem outside the scope of this conversation.

[steveklabnik]

We need organizations that are willing to get their hands dirty and improve stuff with the people who are building the libraries, or who are willing to at least sponsor the maintenance of the libraries in a meaningful way.

I certainly agree this would be awesome. My experiences with other ecosystems have not made me confident that it happens often, though

[eminence]

This might be a simplistic question, but I’ll ask it anyway:

Say my library is already using a large fraction of the rust-platform, as individual dependencies. Will there be any advantages in “migrating” to the rust platform? The obvious one seems to be that I will no longer have to investigate any compatibility issues between various crates. But are there any others?

[chriskrycho]

A number of good points have been raised above and elsewhere, so I’ll try to avoid repeating those and just offer some brief suggestions, noting first that I’m not hostile to the ideas raised here, but I don’t think they’re particularly necessary, and I think some of the other pain points brought up here (e.g. the UI on crates.io) and on HN and Reddit are as or more important than whether we have a blessed meta-package or not.

1. I strongly endorse the idea of enabling “meta-packages” without expressly endorsing any. Making the build and packaging tooling robust around this will enable the community to coalesce around the things which are valuable, and those can overlap in unique ways. Some of the best stuff going on in the Python ecosystem is around exactly those kinds of community-driven sets-of-libraries/metapackages, but the official tooling has lagged so badly that there are entire alternative distribution tools that have grown up around them (e.g. Conda etc.).

2. Put a high priority on supporting and shipping the in-work tool for supplying docs in a centralized location (presumably, ultimately, as part of crates.io), and work at a cultural level to make that the default location for documentation.

   • This will help enormously with discovery, as well as providing the opportunity for integration with the “docset” tools out there (Dash/Zeal/etc. and anything compatible with that). I know I find that really nice when I want to go look at something for Elixir: you can just pull up hexdocs.pm, and those docs can be integrated into any docset tool.

   • As part of that effort, build tools to support the docs/various-content.md model which is used for the Rust docs themselves (which both @steveklabnik and I have taken abortive stabs at in the past) and make those integrate with the documentation hosting as well.

   It is hard to overstate how significant an impact I think this will have on the community.

3. Look at building out something like Ember Observer. When I’m evaluating a Ember add-on for integration into our application, I evaluate it there first. This is a community-maintained resource, and there are definitely some things that would need to be handled differently for Rust crates than for Ember add-ons—but there are more than a few similarities, too.

Each of those could very well be integrated in with the broader idea you’ve thrown out, but they can also be useful regardless of whether any official Rust Platform shipped, and would help address many of the same underlying issues.

---

Separate from all of that: I think the idea of shipping a small set of standard tools (rustfmt, clippy, etc.) is great, and that should definitely happen. Small support applications feel very different to me (and, I suspect, to most other developers) than do libraries, especially libraries which can basically function as standard libraries.

[aidanhs]

And, even if a library was on 1.0, if a rust platform became available and I used it, I’d expect to be able to continue using it for a reasonable amount of time…but inclusion in a platform doesn’t mean the maintainer has the bandwidth to support an LTS version of their library with only cherry-picked fixes (and there’s at least one popular open source project with plenty of resources that simply doesn’t ‘do’ patch branches - if you’re not on the latest version and are having an issue, tough). So now you need to decide what compatibility guarantees a platform has and figure out if included projects can apply for exceptions either because of important fixes or lack of resources. If I were a maintainer of a popular library (I’m not, so I may be wrong), I might actively not want to be included in a platform due to the possible additional burden.

Taking a thought from upthread, potentially if some team or company wants to make a platform then they could release a -platform crate and just pub extern everything. At this point, I would consider them to have agreed to act as the gatekeepers for any issues that requires them to negotiate with upstream library authors, and possibly branch off themselves. If this (being willing to shoulder work themselves, rather than always defer to/push work upstream) seems like a huge burden and nobody wants to do it, maybe that’s a red flag!

The dependency bloat problem (mentioned upthread) also worries me. I’ve never written a line of haskell, and my single experience with it was downloading it to build a tool, at which point it seemed like half the world was being compiled and I ended up with a 100MB binary. Some crates feel like they’re on this road (which I can live with, the creators of these crates are delivering immense value) but I don’t know if the tradeoff is right for a platform (even with pre-compilation).

I haven’t really mentioned anything other than libs (e.g. compiler, cargo, and particularly cross-compilation targets) because I think there’s important lower hanging fruit/prerequisite work for them all - third parties right now can’t easily/officially swap out/release replacement parts of the toolchain, and a platform feels like it’d add to this list.

[cuviper]

Taking a thought from upthread, potentially if some team or company wants to make a platform then they could release a -platform crate and just pub extern everything. At this point, I would consider them to have agreed to act as the gatekeepers for any issues that requires them to negotiate with upstream library authors, and possibly branch off themselves.

That’s basically what happens anyway when an enterprise/lts distro includes a set of packages. It becomes a “platform” which that distro promises to maintain.

[aidanhs]

Yes, I had linux distros and rails LTS in mind for this bit. But the blog post isn’t keen on the libs team doing it for a couple of reasons. My off-the-cuff suggestion was for ‘lib platforms’ to be always unofficial (just a normal crate, maybe with some bells and whistles) and require whatever team/company doing it to have the resources to support a distro model (rather than the proposed libs team collaboration model). I have a few reasons for not being a fan of the collaboration idea, but I’ll wait for the next blog post.

[critiqjo]

I like this approach (better than the simpler version I proposed in reddit). Also, the reason I prefer this, over the simple meta-crate alternative that many others brought up, is the fine-grained control over which crates are fetched and compiled. I have a few things to add (let’s call this tight group of crates: a clique):

[cliques]
platform = "2.6"
security = "1.1"
piston = "1.9"

[dependencies]
libc = "platform"
regex = "platform"
crypto = "security"
...

A few more advantages apart from the ones I mentioned in reddit:

• Distributed curation of various cliques
• Minimize effects of “endorsement”

But of course this brings an additional layer of version conflicts, and adds to the complexity; such as crate collisions (with conflicting versions) between cliques. One way to adapt to this is to strictly follow a few conventions. One that comes to my mind:

• Every clique version should be compatible with at least one version of the “platform”.
• Every clique version should maintain a list of rustc versions it compiles with.

To make this easy for clique maintenance, the “platform” clique should only contain a very few and extremely popular/useful crates (may be around 10-20) with a high degree of maintenance guarantee.

To make this easy for general users, may be cargo should suggest clique/crate versions when there are conflicts based on compatibility metadata with rustc and platform.

Update: -ed!

Update 2: – OR –

Avoid having library users from dealing with such complexty and let these “cliques” be used exclusively for automatic version matching and suggestion by cargo.

The users should not specify which cliques it depends on. And all crates should have membership metadata on various cliques for use by cargo intelligence.

PS: Sadly, the documentation concern is completely ignored!