Proposal: The Rust Platform


#36

I don’t believe a metapackage is necessary. However I like something like awesome-rust, but up-to-date and maybe package rankings driven by github stars? I’m fairly new to Rust and when I’m deciding on a package to use, if there is none obvious candidate, I look upon the usage examples and documentation of each package and consider stars ammount and the stability of commits since beginning. Also the stability or disclaimers in README that say it is not stable.

I suggest some site, that would contain these criteria and that would be updated automatically from cargo and github. The information is already provided there, but could be upvoted and downvoted by the community, much like reddit posts. This would be represented by some score ?

Also, I disagree with something like C++ boost. It is a giant library, much like jQuery, where jokes are being made of plugins for adding integers.


#37

In general I’m a fan of anything that makes it easier to adopt Rust (whether in free time or at work, but especially the latter). And I’m probably more in favor of this proposal than many others in this thread.

However, perhaps there are some ways to improve some of the issues presented without adding as much new work to the subteam’s plates?

For example, I think that crates.io could have significantly better discoverability by leveraging the metadata that crates provide. Being able to browse by keyword, explore popular keywords, etc., could not only make it easier for the specific high-traffic cases presented in this proposal, but also improve discoverability across the ecosystem. This might be significantly cheaper in dev hours than maintaining integration tests and backporting security patches and bugfixes for a growing set of community crates. (which gets at another issue – who does the work in the Rust Platform to fix, for example, a TLS issue that’s been fixed since the old version of hyper in the platform?)

In terms of selecting versions of libraries with compatible dependency versions, is this something which requires a new heavyweight process to improve? I would imagine that effort spent pinning versions together could also be spent keeping popular libraries’ dependencies in sync for their current versions. Is there a specific reason why the “official” choices need to be held back, as opposed to contributing to keeping the popular libraries up to date? IMO, it could be more in line with the Rust culture to as a community keep the most up-to-date versions of libraries in sync with each other.

I also think several of the parts of this proposal are fairly uncontroversial, and I think it may be beneficial to consider some of these things as separate initiatives. It’d be much easier to discuss alternatives for each individual part of this.

I do really like allowing rustup to distribute rustfmt, clippy, language bridges, etc., with better support for cross compilation and other tooling. I wonder to what extent that may be orthogonal to the idea of a “blessed” set of crates. Do these things really need to be considered together?

Does having cargo support for metapackages mean that there necessarily needs to be an official metapackage? I see value in supporting this feature regardless of the direction this proposal goes.


#38

I fully agree.

Moreover, I would rather the community experiments and polishes meta-packages, and how to use them, before it gets used for anything semi-official.


#39

Before we really dig into the solution, I wanted to at least enumerate the problems.

Our primary motivation is to make Rust as successful as possible. I don’t think we say that enough. I don’t mean in a “Rewrite it in Rust” way. I mean in a “people should always be able to choose Rust for their project”. We should be removing any blocks in the way of someone picking up Rust.

We’ve already been doing great work here with things like rustup, clippy, and all the IDE plugins. We’re hacking away at where we see these blocks, but here I wanted to spell out the problems that motivating things about the platform.

From the survey results, we got feedback on what things need to be addressed:

  • Learning curve
  • Lack of IDE support
  • Immaturity of libraries

Learning curve was a huge point of feedback. Looking at the comments, people mentioned it more than anything else. Why is learning curve important?

While some of us have time to really dig into Rust, most of us simply just don’t. Those of us who have put the time in know that Rust is worth it. You get out what you put in, and sometimes the result can be surprisingly efficient even in your first attempt. This makes Rust fairly unique among languages. The problem is that learning curve sharp. Some users describe it as the “nightmare week” before they understand enough of the language to be effective.

The lack of IDE support is important, and I’d happily talk about that for hours, but I wanted to skip to the immaturity of the libraries and dig into that.

If you look at crates.io lots (most?) of the crates there haven’t hit 1.0 yet. This is a signal to many developers that Rust simply isn’t ready for prime-time. These programmers don’t want to spend time messing with APIs that are breaking underneath them. They’re too busy trying to solve their actual problem instead of dealing with maintenance issues. Once they see this, Rust is a non-starter.

There’s more to it than just the version numbers, though they’re important. Rust also needs a shared vocabulary. We’re seeing issues where there are lots of ways to do something. While this is great in a way because there’s a lot of choice, it’s also is a drag on developer time trying to find the right one. Additionally, it’s simply easier to “build higher from the 17th floor than the ground floor”. For example, by having an established set of traits and capabilities we build networking on, we can build even more sophisticated networking functionality on top it because we know it’s stable and supported. Of course, replace “networking” with a variety of common programming tasks.

That stability opens up the doors to a lot of new users. Going back to the comment about people not having enough time. With that shared vocabulary, we can do things like write guides and tutorials that assume these features are available. This lets us make much better starting examples help users quickly solve real problems. This saves them time and gains the community another Rust developer in the process.

If we can solve the learning curve, give crates a reason to hit 1.0, and build a shared vocabulary, we’ve given Rust a much stronger platform to build on and more quickly get new Rust users going.


#40

@aturon, I think a lot of the reservations in this thread wrt stagnation and blessing could be addressed by providing a preliminary idea of which libraries you would nominate for the platform, to give us some idea of its expected scope. In particular, I sort of get the impression that you don’t intend this platform to be anywhere near the size of the Haskell platform (which is 30+ packages).


#41

@jntrnr Thanks for that perspective. It’s really important. Increasing Rust adoption is our most critical goal, and we rarely drill down into our motivations far enough to tie individual decisions to that ultimate concern. None of our effort matters if Rust doesn’t continue to gain new and happy users. Hopefully the more structured roadmapping process we’re heading toward will help clarify that.


#42

Ah yes! This makes what you’re saying make a lot more sense to me. Periodically, some people try to argue that Mozilla has some sort of nefarious goals with Rust; someone on HN thought that this was some sort of monetization strategy, for example. So I was mostly responding to the Mozilla part, not the rest of it. s/Mozilla/Rust teams/ clarifies a lot, thank you.


#43

I think these are excellent goals.

I’d be curious to see the community response to everything in the proposal except the standardized crates. I don’t see anyone against distributing clippy or rustfmt, for example.

Are these goals still served if the community organically evolves a series of metapackages for different domains, as opposed to a monolithic official metapackage? Is it fundamentally different to write a guide about using the async metapackage than writing a guide about using rust-platform/mio? Is there perhaps benefit to having a web metapackage maintained separately from the embedded metapackage? If metapackages would be available to all Cargo users, what value is added by an official Rust Platform metapackage?

Also, I don’t maintain any popular crates so perhaps someone could do me the benefit of explaining:

  • Why are community crates holding back from 1.0?
  • What part of having an official distribution of community crates would resolve the perceived blocks to release a crate’s 1.0?
  • Are there decentralized alternatives to provide incentives to release a 1.0?

#44

I can speak for hyper: HTTP is a complex problem, and iterating on the design means breaking changes. Especially as the community tries to figure out non-blocking IO. I’m also usually working alone, with occasional contributions (much appreciated!), and in my free time. This makes iterating take longer. A “platform” wouldn’t make me tag 1.0. It still needs plenty of work.


#45

I think it would be extremely beneficial to rehtink how one would find a crate to fulfill a certain need. I don’t think bundling a limited set of crates as proposed is a good idea, though. We just need to make it much easier to find crates and I think this could generally be done by just improving crates.io.

Here are some general thoughts about improving crates.io:

  1. Keyword search on crates.io is pretty terrible. A main cause of this, I think, is that humans are pretty bad at determining what is useful to know about a crate, which results in bad descriptions and keywords. Often, googling just crates.io gives much better results than the crates.io search.
  • The search results don’t let you do any meaningful sorting or filtering.
  • There is not a good measure of ‘popularity’ for a crate. There’s just the number of downloads (which you can’t sort by), which is a poor metric, because being a dependency of a popular crate just pushes up your numbers. I think a graph-based ranking system could work.
  • There is no structure to the crates listed on crates.io. Meaning: no category system, no directory. Perhaps this is a result of a line of thought about everyone using search for everything these days, but that only works if the search is as good as Google. Also, I think developers are more likely to want to browse a structured representation of data than average users.
  • Related to points 1, 3 and 4: give users of crates.io more influence in how data is presented. Because crate owners don’t necessarily know how there crate will be used, users need to be able to add useful information to the database. Perhaps users could vote or you can import data from GitHub (stars and forks). Users could add missing keywords or otherwise improve how a crate is presented. This needs to have low overhead (lower than filing a PR at the crate’s source)

#46

I would like to mention that finding an interesting crate is only half the work.

Writing lots and lots of inefficient and error-prone scaffolding boilerplate so that the dependencies you ended up selecting can talk to each other is the second half (because they use incompatible types or incompatible versions of the same types).

A meta-package prepared for a given domain would allow solving both problems in one fell swoop; that’s rather neat.


#47

@dikaiosune - yeah, I like the idea of metapackage “distros”. People that care about certain areas could help curate and ensure the crates work well together.

@seanmonstar - it makes me sad you’re the only person really working on hyper. Things like “http” should be what we as a community rally around solving. They’re core pieces of what it means to program, well, any language these days really.


#48

I already said it on reddit: everybody’s working on the compiler because it’s somehow prestigious, and nobody is working on the libraries. Piston is one person. Gfx-rs is half a person. Glutin and glium are not maintained because their author doesn’t have time. The situation is disastrous.


#49

As for glutin, the reason why it’s not 1.0 is that I have actually no idea what I’m doing when working on it. I started the library so that there’s at least something. At the time (something like early 2014 I think) the only existing alternative was glfw-rs, which was very tedious to use. Again, the lack of manpower.

Glium depends on glutin, so it can’t be 1.0 either. All my other libraries are largely unfinished.


#50

I think it’s definitely worth collecting input from other community crate authors, but it’s starting to sound to me like a significant reason holding back community 1.0’s is that there’s not enough effort directed at contributions to them, and that they haven’t done enough iteration on their API. Is there something (either alongside or in place of the platform proposal) that the Rust teams could be doing to funnel more contributions and dev hours towards these important community crates?


#51

The way that I see it is that we need to grow in general. It’s not like we have a ton of people with spare time that don’t know where to put it; everyone I see in Rust world is working extremely hard at shipping all kinds of awesome stuff. The best way to lighten the load is to get even more Rustaceans on board, which is why the focus on new users and growth is important.


#52

Can we provide a “batteries included” experience for Rust that doesn’t lead to stagnation, one that instead works directly with and through the ecosystem?

We could start by automatically hosting documentation for crates on crates.io, and if there ever is a set of “endorsed” crates, host their documentation directly on doc.rust-lang.org.

Speaking from Haskell experience, Hoogle is amazing. Part of what makes it so great is that it doesn’t only search through the prelude, but also through core third-party packages. All API reference searches can be served from a single endorsed place, and you can do a cross-package search from that same place. And Hackage serves API documentation for the entire ecosystem. Currently most Rust crates have online API documentation available, but cross-referencing across crates is limited, and there is no uniform search. Having all documentation in one place like Hackage with Hoogle is a game changer.

The biggest open question about what I described above is: how does curation work?

A small feature suggestion for crates.io that I think could help a lot: allow users to “vote for” or “star” a crate, similar to voting in the AUR, starring a repository on GitHub, or starring/+1’ing an issue in a bug tracker. The current metrics to judge packages by, downloads and dependees, are not necessarily good proxies for recommending crates. Downloads in particular is heavily influenced by the number of dependent (popular) crates and by the time the crate has been around. (Though that would be true for votes as well.)


#53

There is no incentive as a library author to publish a 1.0 crate. When you publish 1.0, people expect you to start mangling your code to maintain backward compatibility. Yet, there’s no benefit to the library author for doing that extra work. Thus, publishing a 1.0 crate is a non-starter for the library authors.

There’s more to it than just the version numbers, though they’re important. Rust also needs a shared vocabulary. We’re seeing issues where there are lots of ways to do something. While this is great in a way because there’s a lot of choice, it’s also is a drag on developer time trying to find the right one.

Could you point to some study, even an informal one, that shows this is true? In my experience, it isn’t a significant timesink.

For example, by having an established set of traits and capabilities we build networking on, we can build even more sophisticated networking functionality on top it because we know it’s stable and supported. Of course, replace “networking” with a variety of common programming tasks.

Conversely, I’m looking forward to people building new replacements (perhaps the same crates refactored, perhaps new crates) for almost all the currently-commonly-used networking libraries. The current generation is useful, but I think there are significant improvements that can be made.

With that shared vocabulary, we can do things like write guides and tutorials that assume these features are available. This lets us make much better starting examples help users quickly solve real problems. This saves them time and gains the community another Rust developer in the process.

We don’t need the Rust teams to spend time on that. The early adopters of the various libraries do this and will do this. This is relatively easy to do and is a good way to start contributing to the Rust community. Conversely, it’s much harder for a newcomer to add alignas to Rust or add 64bit * 64bit -> 128bit multiplication intrinsics to Rust or do other things that the Rust teams are really better suited at doing.

give crates a reason to hit 1.0

To the extent that this gives crates an incentive to publish 1.0, it seems coercive (no doubt unintentionally) and not in the best interests of the library maker. It would be better to find incentives that actually benefit the library makers (e.g. money).


#54

I think in many cases this is not actually an issue with maturity. In my opinion, many authors are simply too conservative regarding versioning. At the time of writing, none of the most-downloaded crates on crates.io are stable, yet they are the most widely used and battle-tested crates out there. 1.0 doesn’t have to be perfect. There’s no shame in releasing 2.0. Or 52.0 for that matter.

Make Cargo colour version numbers < 1.0 red and version numbers >= 1.0 green. Although that might actually scare people away from unstable packages instead of nagging authors. Perhaps only colour the version number for the root crate?


#55

This project is already largely underway, see