Expansion of standard library

after working in rust for 3 months i had to switch to go lang for writing back-end micro-service as too many things

during my time in go , i realized that rust indeed needs a basic HTTP I/O (a little more abstracted than TCP/IP , something like basic version of go langs net/http . like routes , http request structures and response writers )library built into the language itself and my points are :

  1. it will be easier for the people who want to keep dependencies at bare minimum , either due to company policies or simple desire of self contained source code
  2. not everyone likes those big frameworks that take ages to compile
  3. async is coming directly into rust , giving framework makers a chance to restructure project and reducing amount of code in project to maintain .by giving them abstraction over HTTP implementation we help them focus on high level abstractions and features
  4. with rust , services can be scalable , fast , efficient
  5. with this we can make rust look more reliable and see even faster adoption of rust on corporate
3 Likes

We've had this discussion a few times. For Rust, crates.io is the standard library.

crates.io plans to have precompiled crates, so the speed won't be any different from shipping the same crates via rustup.

Cloudflare uses Actix-web in production globally. Note that Actix-web had 8 breaking releases, giving it many chances to improve it's public API (and it will be able to change it in the future). The maximum number of breaking releases allowed in the standard library is 0, so such large and complex codebase wouldn't be able to gradually evolve.

35 Likes

I think it should maybe have been reduced, e.g. std::fs, std::net and std::process probably shouldn’t be in the standard library.

The only things that need and should be in the standard library are data structures expected to be shared between various libraries and used by almost any crate (e.g. Result, Vec, HashMap, etc.), since putting them in an external crate would result in multiple versions of them being present in a single project and thus an inability to share data betwen crates.

14 Likes

i would like know more on your point against std :: fs and std :: net i believe they provide important functionality

Rust teaches that anything can fail , why crates.io won’t ? why everyone is against a bigger standard library ?

Python is going through this same issue today. See https://lwn.net/Articles/790677/ for more information.

Elevating something to the standard library picks a “winner” among potentially competing options, forces everyone to have it available, forces every platform to support it as well as possible, and similar. Having something on crates.io allows an ecosystem of options with different tradeoffs and assumptions, available on the platforms that they make sense.

19 Likes

The key point is that Rust comes out-of-the-box with a toolchain that makes adding a 3rd party crate to your project a completely hassle-free non-issue. That creates a complete paradigm shift for how we think about a “standard library”.

In many other language ecosystems, especially C++ (and I think Go to a lesser extent, until recently?), the reality is that the standard library is the only library for many if not most users, thus the bar for inclusion in the standard library is fundamentally not that far from “is this useful to people?” Any problem that’s “been solved already” becomes something we want to “standardize” just for the sake of allowing the rest of the C++ community to use it, because in practice 3rd party libraries are simply not worth the hassle.

In Rust, none of that applies. When all 3rd party code is just as easy to depend on as the standard library, there’s no longer any special availability advantage to being in the standard library. The idea that any “already solved problem” needs to be “standardized” simply evaporates into a non-problem, because once it’s on crates.io, then “everyone has it” in practice already. Moving it into std does not increase the availability any further.


So the “what should go in std?” question is completely different in Rust. Typical compelling arguments for putting things in the Rust standard library include “vocabulary types” (e.g. the whole Rust ecosystem needs to agree on a single str type just so everyone’s code can talk to each other), code that needs to interact with the compiler in some magical way, or tricky unsafe operations (because we trust std verification more than 3rd party crate verification, though even that could potentially change in the future), or platform abstractions that help keep a lot of Rust code portable-by-default (though that strategy didn’t scale as well as we hoped, so that might change too). All of these are pretty fuzzy and case-by-case criteria, not remotely official in any way. In fact, in the specific case of an HTTP server, there is an obvious vocabulary types problem, but the ecosystem ended up agreeing on a 3rd party crate for that and AFAIK no one’s ever asked for it to be moved into std.

27 Likes

I don't against big standard lib. If Rust has a stdlib the size of Go, I'll start all my later side project only with Rust.

Go had a big big standard lib that is finely integrated together, and good integration is one biggest advantage a huge stdlib can provide. It's not about "forces" for whatever that means, it's about integration.

And let's don't forget people can import any number of lib they want to use, and glue them with the existing stdlib when needed, just like what you would when using a third-party lib.

But apparently, Rust embraced another ideology that is "Let people choice", and expecting the ecosystem will automatically organically grow into place. It could happen, after the language itself become agile enough while also setup a standard way of doing things.

3 Likes

In certain areas, rust have achieved strong standards through crates. The std does not contain randomness, or many common maths functions. Yet everyone uses the rand crate for randomness, and most maths crates build on top of num.

The standard library only needs to contain the types and traits, that have syntactic sugar. While not having these apis in the std allows breaking changes without breaking of legacy code.

I would possibly go as far as suggesting reducing what the std contains, allowing for more flexibility in areas such as std::time.

8 Likes

I am still unsure regarding random and num, I still think their functionality will be better in the stdlib in some more years.

For what it’s worth, there’s presently an open issue about integrating the getrandom crate into std/core

This is one key detail, though: I do think the most important job of the standard library is to standardize key vocabulary like Write and Debug and Display and Add. Sometimes that can occur in the ecosystem, but sometimes that does need the standard library (e.g. futures).

I may be wrong on this fact, but I though futures where moved into the standard library do to the async await syntax. Anything that has syntactic sugar has to be in the standard library right?

If so, both Add and Future are bad examples, because the have to be there

I also like smallstd. I even think that some things shouldn't be part of it (e.g. hostname lookup, or even if we are being more extreme the whole std::net).

It's important to note, that this proposal is strictly limited to retrieving system entropy and there are no plans for adding other stuff from rand project to std. We have two reasons for adding getrandom to std instead of keeping it in a separate crate:

  • Randomness is used for seeding hash maps, so considering non-triviality of retrieving system entropy on some targets it's worth to keep a single well-reviewed code base for it.
  • Hypotheticall future addition of #[getrandom] lang item, which will be somewhat similar to #[global_allocator].
7 Likes

I’ve posted this here before, but here’s an excellent article by Titus Winters about what he thinks should go into the (C++) standard library. Tl; dr:

  • Hooks into the compiler/language/runtime that can’t actually be implemented outside of std. Intrinsics, magic macros, magic traits, &c.
  • Vocabulary (string, vec, …) without which it would be difficult for non-std libraries to talk to each other.
  • Portability functions, platform agnostic ways of talking to the fs, to the network, basic concurrency primitives, and the various underlying platform specific bases. Note for the OP: these don’t necessarily have to be “fully featured”.
  • CS primitives for sorting, searching, filtering. Things that are simple enough that I would implement it myself before looking for a crate, but complex enough that that’s a papercut. dbg! was a recent pretty clear cut addition in this category. itertools is a nice collection of things on the border.
  • Batteries. Big unicode libraries, random number generation libraries, matrix libraries. Standard C++ libraries for this are often not the best in this domain and a proper (C++) dependency system would make these unnecessary. Rust has crates.io so pretty much all large domain-specific libraries should just be crates.

Rust has other categories of crates that receive extra special attention/auditing, such as

  • the nursery
  • crates the compiler, and thus everybody, depends upon
  • crates various large projects (e.g. servo) depend on

I’d really like to see a What should go into std document that answers this question in a more or less official way.

6 Likes

I like current small standard library (builtin library). But if Rust gets large standard library (libraries governed by Rust team) in crates.io, I welcome it.

I’m in favor of a small standard library, but I think the people who want a larger one have real legitimate needs, and I’m frustrated that there isn’t much of an attempt to find some other way of addressing these. I don’t have a 100% solid understanding of what those needs are (and would like a better one), but my general impression is that they want to rely on libraries that are proven in production, won’t make breaking changes, won’t be deprecated in two years, won’t be compromised and used as a malware vector, if bugs are reported they’ll be fixed, etc. C++'s package management situation may be terrible, but it’s far from the only way that dependencies can be a liability.

Unfortunately I have no idea what to do about this. I like the idea of curated stacks but that went over like a lead balloon last time it was proposed. Again, though, it’s frustrating that this has been the end of the conversation.

11 Likes

Despite the discussion about bigger or smaller std (I’m all for thin std fwiw) is there maybe still a point that Rust could benefit from a more leightweight crate for doing https?

1 Like

There was a concerted effort to bring many more crates to 1.0 some time ago, which may potentially help.

I do absolutely understand that it can be a challenge to pick out which crate to use in certain areas. I’d love to see some clearer guides on things like error handling, randomness, and a dozen other things that people commonly reach for.

1 Like

I don’t think the problem’s just discoverability, I think it’s trust. I’ve certainly talked to people who are very reluctant to add any dependency because they expect it to bite them down the road.

10 Likes

I am glad you mentioned these issues and I am surprised the security is not getting more attention.

With so many dependencies it is becoming close to impossible to guarantee that the 3rd party code you are blindly bringing into your app is not compromised today and won’t become malicious tomorrow.

How can we avoid issues that are becoming widespread in the NPM world?

8 Likes