Proposal: The Rust Platform

1

A programming language is much more than its compiler and standard library. It's a community. Tools. Documentation. An ecosystem. All of these elements affect how a language feels, how productive it is, and how applicable it is.

Rust is a very young language -- barely a year past 1.0 -- and building out and maturing the full complement of ecosystem and tooling is crucial to its success. That building is happening, but sometimes at an explosive rate that makes it hard to track what's going on, to find the right library for a task, or to choose between several options on crates.io. It can be hard to coordinate versions of libraries that all work well together. And we lack tools to push toward maturity in a community-wide way, or to incentivize work toward a common quality standard.

On the other hand, the core parts of Rust get a tremendous amount of focus. But we have tended to be pretty conservative in what is considered "core": today, essentially it's rustc, cargo, libstd/libcore, and a couple of other crates. The standard library also takes a deliberately minimalistic approach, to avoid the well-known pitfalls of large standard libraries that are versioned with the compiler and quickly stagnate, while the real action happens in the broader ecosystem ("std is where code goes to die").

In short, there are batteries out there, but we're failing to include them (or even tell you where to shop for them).

Can we provide a "batteries included" experience for Rust that doesn't lead to stagnation, one that instead works directly with and through the ecosystem?

I think we can, and I want to lay out a plan that's emerged after discussion with many on the core and subteams.

What is "The Rust Platform"?

The core ideas here draw significant inspiration from the Haskell Platform, which is working toward similar goals for Haskell.

The basic idea of the Rust Platform is simple:

  • Distribute a wide range of artifacts in a single "Rust Platform Package", including:

    • The compiler, Cargo, rust-lang crates (e.g. std, libc), docs
    • Best-of-breed libraries drawn from the wider ecosystem (going beyond rust-lang crates)
    • Best-of-breed Tools drawn from the wider ecosystem (e.g. rustfmt, clippy, NDKs, editor plugins, lints)
    • Cross-compilation targets
    • "Language bridges": integration with other languages like JS, Ruby, Python, etc.

  • Periodically curate the ecosystem, determining consensus choices for what artifacts, and at what versions, to distribute.

In general, rustup is intended to be the primary mechanism for distribution; it's expected that it will soon replace the guts of our official installers, becoming the primary way to acquire the Rust Platform and all that comes with it.

As you'd expect, the real meat here is in the details. For example, it's probably unclear what it even means to "distribute" a library, given Cargo's approach to dependency management. Read on!

This is the dedicated discuss post for feedback on the proposal. Fire away!

6 Likes
2

Somehow I feel strongly negative towards this :frowning2:

Looks like the platform is supposed to solve two problems:

  • visibility and documentation (your problem X is solved by a tool/library Y)
  • code distribution

As for the distribution, I think that Cargo and crates.io are just awesome. Why would you need a “global lockfile” of packages? You almost never care about dependencies of the dependencies anyway, so as long as each crate has it’s dependencies right, any combinations of crate should work? Precompilation seems a minor advantage to me (and what about different compler flags?). However forcing everyone to download a ton of unnecessary (for them) stuff does not feel right (and Haskell distribution is just ridiculously huge. IntelliJ IDEA is smaller :slight_smile: ).

As for the first problem, I agree that it does exist. But can it be solved by just some kind of meta documentation page? Something like awesome Rust or stdx list, but just actively maintained, serachable and with included crates documentation.

What I am trying to say is that you’ll need some kind of unified docs for the platform anyway, and the platform would be useless without docs, but the docs will be perfectly useful even without the platform.

Also what bothers me is that inclusion of some crate into the platform would put other crates at disadvantage. And blessed crates are not necessary the best ones in every aspect! See, for example, the docs for env_logger ( https://doc.rust-lang.org/log/env_logger/index.html ) and fern ( https://dabo.guru/rust/fern/fern/ )

EDIT: I use the word “crate” here, but I have the same feelings towards tools like rustfmt.

15 Likes
3

I do like the idea of metapackages, though I’m less sure about one "officia"l rust-platform meta (other than rust-lang crates being a rust-platform). I’d rather see, for instance Are we web yet? is already a set of crates for doing web/networking, where a networking metapackage could be maintained. Awesome Rust links to a number that could be a meta. In other words, make a meta package tool (generating combined docs, allowing easy visibility, providing easier integration testing, etc.) general enough for anyone to create meta packages and let unofficial meta curation grow organically. This would also make it easy enough for metas to act as incubators for any inclusion in rust-lang/rust-platform without dictating official crates; multiple competing visions for platforms could co-exist and let everyone be happy. When some crates end up being used across multiple metas and become essential, that’s an indication of more official centralized involvement, rather than trying to arrive on consensus ahead of time.

In general, I do think Cargo needs some sort of metapackage-style thing for many of the reasons listed in the proposal. Grouping multiple libraries into single distributions has a lot of benefits, but it also has its drawbacks if they’re too official, as we see with std and rust-lang crates.

1 Like
4

As you are probably aware, the Haskell community’s experience with the Haskell Platform has been… mixed, and probably not overall positive. A large part of that is likely attributable to “cabal issues”, and to GHC versions being tied to Platform versions, both of which as far as I can tell this proposal would presumably avoid. But I’ve only been observing from a distance, so I definitely couldn’t say for sure, myself, what other issues may or may not potentially exist. I think it would be wise to consult the Haskellers about this directly, so as to avoid repeating any of their mistakes, as far as possible. (A good person to get in touch with is probably Michael Snoyman - incidentally, the “Global coherence” idea is strongly reminiscent of the Stackage project which he spearheaded.)

5 Likes
5

Distribution maintenance is a hard problem, as is deciding what goes in ‘core’. Thank you for trying to tackle how to keep libraries fresh but also in core. I am sorry that I don’t really agree with this proposal as a solution to the problem of “we lack tools to push toward maturity in a community-wide way, or to incentivize work toward a common quality standard”. I do agree that we need to “tell you where to shop for [batteries]”.

I would rather see an effort put in making finding and using crates easier, than hiding them and picking a distribution that fits ‘everyone’. A rust platform will lead to major confusion when reading official documentation for packages , and creates a further split than just stable/nightly. Users will have to know if a package version/usage/documentation requires nightly/stable or platform. Or they will need to unpack what version is platform. I think this causes more problems than it solves.

Improving external tools is another great weakness you have identified in this RFC. However, if rustfmt is great, efforts should be made to upstream it to the cargo tool rather than come up with yet another way to install stuff for all platforms. If clippy should be easier to install, put work into making clippy installable on stable and not just nightly.

If we go with the platform route, tutorials will have to explain the difference between ‘rustup platform/stable/nightly’ and cargo new will need a cargo new --platform or cargo new --without-platform.

I am not sure what evidence there is for newcomers or experienced rustaceans that we need a batteries-included distribution of the language. It’s especially awesome to see people hop in #rust-beginners and ask ‘can I compile to arduino’ and hear a resounding ‘yes! check this tutorial out!’. In your proposal, serde is recommended. Its an awesome library, but one that not everyone needs, and would be confusing why my hello world has a json parser in it. If rust/cargo’s build tools are super smart about this and don’t compile stuff in, maybe we could take it further and just imply a Cargo.toml based on ‘extern crate’ declarations, but that is getting off-topic.

It is not clear to me what new and experienced rustaceans are struggling with from a tooling/platform maturity perspective, and I would love to see examples from irc and other places so we can understand what is most important.

Anecdotally, as a newcomer to rust (~5 months), platform-related issues i’ve come across include:

  • clippy neeeding nightly, which lead to
  • rustup.rs being the most recommended but not-on-its-own-website way to install rust, conflicting with brew install rust and rustup.sh on the offical website
  • unborrow! needing nightly, leading me to just not use unborrow! so I don’t propagate the nightly issue for my own project

I really hope you continue to evolve this RFC, as I do recognize there are rough edges around working with the rust ecosystem. Tools like https://rust.libhunt.com/ can help, and maybe making stuff like cargo search show ‘recommended’ crates first could help make things more human-friendly.

4 Likes
6

  1. Let’s say that a crate is chosen to be in the “Rust Platform” and the people developing the crate don’t want to be in the “Rust Platform.” What happens then?

  2. Let’s say the best crate for solving some problem has some non-technical issues that make it difficult to include in the “Rust Platform.” Would a worse crate for solving the same problem be added to the “Rust Platform” instead?

I would prefer to see a proposal going in the opposite direction: Less centralization of the management of libraries by Mozilla, and more distributed development of libraries and diversity of library development, including in particular more diversity of the management and leadership of libraries. Mozilla is doing so many great things with Rust, but it seems like resources there are stretched really thin. Maybe it’s time to let go of some aspects of library development, even the “core” libraries, to focus on improving the core areas of Rust faster, in particular improving the language itself and the rustc compiler, and adding features to the toolchain as necessary to make distributed library development work better.

5 Likes
7

I think it would be worthwhile to think about how non-Rustlang managed “metapackages” could also be integrated into this scheme. For example, a “piston” metapackage that imports all of the crates from the piston project. The “Rust Platform” metapackage of course would have distinct versioning and distribution questions, which this post works on resolving; I’m just saying that I think the concept of a metapackage in cargo should include independent metapackages for projects which distribute multiple crates, and a consideration of how a cargo registry would distribute those metapackages.

I’m not as critical sure that providing a “Rust platform metapackage” as some of the other posters so far seem to be, but I wonder if it is really high priority right now though. What is motivating this effort? Is there evidence a lack of curated stdx crates is blocking adoption?

4 Likes
8

The core ideas here draw significant inspiration from the Haskell Platform, which is working toward similar goals for Haskell.

Primary problem Haskell Platform has been solving is incompatibility of dependencies between packages (its even written in the first paragraph of its README!). The secondary one – Windows experience (even if its something that was solved seemingly by chance). Both of these are non-problems in rust-land, and thus, This sentence misinterprets/misrepresents Haskell Platform’s approaches as its goals.

I feel, like looking for inspiration in that project is a bad idea, especially provided that it solved the problems to some extent, it wasn’t a particularly nice solution.

But I've only been observing from a distance, so I definitely couldn't say for sure, myself, what other issues may or may not potentially exist.

I’ve had a great opportunity to observe the rise of stackage firsthand, and even before stackage was announced haskell platform wasn’t a first choice for majority of developers. Rather, I remember it as a “get haskell platform to try things out, setup stuff properly later” shortcut. This niche is already well covered by tools like rustup.

As one of the primary motivators for this seems to be difficulties surrounding crate discovery, then perhaps, we should work on making discovery easier?

6 Likes
9

How is this proposal centralizing things with Mozilla?

10

My initial impression is that this sounds a lot like what boost does for C++. I think it’s a great idea, but there are some long term issues. For example, boost never actually got to a 2.0 and I doubt it will. I’m not suggesting that this effort will have the same issues, but there are definitely things to be learned from.

11

Responding in the same vein as your question: I never said it was.

12

It seems to me that this would add extra complexity for very little gain.

Although having cargo implicitly import numerous crates via one name may seem like a very slight improvement for writers, it also seems like more magic to grok for readers.

Some questions a new user might have:

  • Why is it that this rust-platform dependency (or other metapackages) can auto-add a bunch of crates to my project, but these other dependencies do not?
  • Why does this Cargo.toml have regex listed when the rust docs says it already comes with the rust-platform (perhaps unaware of non-obvious version overriding)
  • Where are these crates being imported from in this src code? I can’t see them in my lib.rs or Cargo.toml? (if we did remove the need for extern crateing them as you mention as a possibility).
  • Which one of these metapackages listed does this extern crate come from? (in the case that metapackages become something that users can create, and multiple are used in one project)

Even in the very few crates that you’ve listed as examples, I personally have only used one of them (serde) in a very small handful of my projects (I must have at least 30+ going now). As it is proposed, I foresee this rust-platform as just something I’ll have to get rid of when setting up new projects using cargo new.

Adding crates with cargo has never crossed my mind as a strenuous process, so I did find this proposal quite surprising, especially at a time when the core team seems inundated with so much other work. This is of course just my own limited perspective, and I may be alone in having this reaction! In general, my vote is in favour of keeping cargo simple and explicit.

9 Likes
13

I’m against this, particularly the part about how it affects dependency resolution, and also bringing in a whole load of unused crates.

I think a “rust platform” as purely a discovery/stability helper would more promising. There would be no changes to cargo or dependencies, and the rust platform would not actually exist as a thing within the tooling: it would simply be a collection of “best in class” crates, with more stringent requirements on testing and stability. These crates would still be installed individually with no changes, but it would provide a way for beginners to more easily find good crates, and would serve to make rust feel like it has batteries included.

It would need to be carefully designed to avoid disadvantaging new crates though.

5 Likes
14

I don’t think any of these are insurmountable educational problems:

  • “Metapackages” should probably have a clearer name (“bundle”? idk) and be listed separately from single-crate dependencies. It doesn’t seem super hard to me to figure out what’s going on here:
[dependencies]
serde = "1.0.0"

[dependency-bundles]
piston = "1.4.0"

extern crate serde;
extern crate piston_2d;
extern crate piston_editor;

  • This just doesn’t seem like something a new user is going to notice and run into often, but the docs should probably be clear that you can get regex from the ‘platform metapackage’ or from crates.io.

  • This behavior would be documented in the explanation of the ‘platform,’ and seems unsurprising to me based on my experience with other languages’ standard libraries.

  • Cargo and crates.io interfaces should be able to answer these kinds of questions.

1 Like
15

I’m happy to see an official effort here, though I think that the blog post failed to articulate that the value of this wouldn’t just be greater discoverability, but also to push more packages into committing to a stable 1.0 release.

My greatest concern at this time is that I don’t want the existence of a Rust platform to be used as an argument against ever again adding anything to libstd. I’m not trying to argue maximalism, but we need standard and shipped-by-default libs for certain things (such as an eventual replacement for rustc-serialize).

Do you foresee any restrictions on licensing? I can imagine that people would be quite cross at copyleft libraries being added to their project without fanfare.

1 Like
16

It seems to me that most people want the ability to answer the questions: “What is the best library for doing X? Is it ready and stable?” The Rust Platform proposes to solve this by curating the best libraries and providing them to users. However, I’m not sure this will solve the core issue, and might be more confusing to users than a directory of curated packages. It also prevents Cargo.toml from signalling which crates a package uses.

It would be nice to review the occasions when this has previously been done, including languages with large standard libraries that get updated slowly. Python has some successes and failures, e.g. functional programming in Python uses a very clear set of builtins and additions from the itertools and functools packages, but some tasks such as “get the current time in UTC as a UNIX timestamp” are still messy combinations of standard library tools. I personally don’t know any project which has pulled this off without fracturing the ecosystem, and my ignorance may be souring me towards this proposal.

In the end, I won’t be upset if this happens, but I will prefer to have an installation without the platform and to write packages without the platform declared as a dependency.

2 Likes
17

Huge +1 to having an official or semi-official directory of best-of-category crates, and doing whatever is necessary to make that “inner ecosystem” more mature and stable. It took me several months to find awesome-rust; if that or something equivalent were made “community official” and linked from the main Rust documentation page, I think that’d be a big improvement.

1 Like
18

Hm, I guess I’m not understanding what your point is. You said that you want “the opposite of this proposal”, which implies that this proposal is doing these things.

19

Interesting initiative, although I don’t like so aggressive injection into existing process. I don’t think adding it into “cargo new” and removing “extern crate” are good ideas. First will ship a lot of bloatware like Samsung phones, second will pollute namespace and create naming collisions.

Part about promoting libraries and tools of the high quality - maybe a good idea. It can motivate authors to write libraries of better quality. From other side, it will demotivate them to write alternatives for already existing crates, included into the Platform (since chances to replace it are quite small).

20

Right, and your question implied that you don't see how the proposal is doing those things. I can say the same thing a different way:

We don't need the Rust teams* to endorse specific libraries over others, and in fact I think such endorsements would definitely lead to stagnation. Right now we have certain libraries that seem like clearly the best choices for certain tasks, but in many cases the best libraries are really the least worst; i.e. we have a long way to go to improve the library ecosystem, and it is likely that which libraries are "best" at certain tasks will change radically, especially if and when important changes to the language and toolchain are made.

Further, if the Rust teams start endorsing particular libraries then it will become necessary for libraries to work towards receiving such endorsements. This would mean, in particular, that library developers would then have to waste time doing politics to get such endorsements if they want to build a successful community around their library, and organizations that are more politically-connected to the Rust teams would have a huge advantage in such politics. Thus I'd rather have the Rust teams avoid such endorsements. This includes, in particular, avoiding adding things to libstd that can reasonably be done outside of libstd, as adding something in libstd is in a sense the ultimate endorsement. Instead, I'd rather have the Rust teams do the technical work necessary to make a small standard library work.

* In my previous message, I used "Mozilla" instead of "Rust teams." Whether this is a important distinction is itself a political issue which I regret having brought up, especially as a former Mozilla employee. Note in particular that I'm definitely not saying that Mozilla is bad in any way or that Mozilla would abuse its political leverage with the Rust teams any more or any less than any other organization would in the same situation.

3 Likes