What should go into the standard library?

[SimonSapin]

Continuing the discussion from The Life and Death of an API:

The Life and Death of an API

In the long term, what should live in the standard library, in other rust-lang crates, and in crates.io?

The Python standard library is sometimes said to be where modules go to die, as improvements to it can take 1 to 2 years to be released. Rust is better with 6~12 weeks from landing a PR to a stable release, but that’s still nowhere the flexibility of cargo publish. Additionally, moving into the standard library means you have to make strong stability promises: no breaking change until we increment the major version number for the entire language, which hopefully will be very rare.

Any advantage to moving into std needs to outweigh these downsides.

So what are reasons to add some new feature in the standard library? I can think of:

1. It can not reasonably be done outside. For example adding a method to a standard library types that need access to private fields.

• It offers interfaces that should be “standardized” to make it easier for unrelated libraries to interoperate with each other. Things like the Iterator or Error traits.
• Some people don’t like having dependencies.
• Curation: being included is a sign of quality and scrutiny, whereas I have no idea how good is a library I just found on crates.io

In my opinion, respectively:

1. Simpler low-level functionality like Vec::from_raw_parts that enable building complex stuff externally should be preferred over including the complex stuff.

• The bar should be very high for this IMO, and possibly require some experimentation time on crates.io
• Cargo makes dependencies easy enough that this argument is invalid. (If it doesn’t yet, Cargo should be improved.)
• I think this reason is just not good enough: there are other ways to do this. Having crates in the rust-lang github org could be one of them. (But perhaps creates an undesirable “us versus them” division.)

[eminence]

So what are reasons to add some new feature in the standard library? I can think of:

4. Curation: being included is a sign of quality and scrutiny, whereas I have no idea how good is a library I just found on crates.io

There are several crates on crates.io that are published by “The Rust Project Developers”. Personally, I put a high degree of faith in the Rust Project Developers to product high quality code[1]. So code doesn’t necessarily need to be in the stdlib to meet this ‘Curation’ requirement.

If a new feature doesn’t meet the necessary requirements for inclusion into the standard lib, could the feature go into an “Official Rust Crate”? By “Official Rust Crate” I think I mean a crate that been flagged as being maintained by the rust project, and so has similar guarantees as the core rust project (it won’t become stale, has the same level of code review and support that one expects of the rust project)

(I think you might be thinking along the same lines based on your 4th bullet. I am also sensitive to the “us versus them” division, but I think there are ways to frame this idea to avoid that notion)

[1] This does not imply that I don’t trust others. I just don’t have as much data about other crate authors

[alexcrichton]

I think it’s definitely tough to strike the right balance here, and I’m personally somewhat wary that we can create a set of concrete guidelines which aren’t likely to change over time, but it’s certainly worth a shot! One rule of them I normally take is that once something is in the standard library, it must be supported until the end of time, so it should be worth it!

I believe one of the main charters of the standard library is to set the idioms and standards for the rest of the Rust ecosystem as much as possible (you alluded to this as well). For example traits like Iterator, Error, Read, Into, etc all belong in the standard library. These traits define an idiomatic and common interface to allow libraries to interoperate with one another. In terms of idioms, I also think the standard library’s APIs and what it exposes on core types like Vec and String are strong precedents for custom downstream APIs in terms of functionality exposed. Essentially, I definitely agree with (2) above, and I definitely don’t think it should be taken lightly.

One reason I might add to your list above is: if a feature is used ubiquitously throughout the ecosystem, it should be in the standard library. For example there’s really no need for TCP/UDP support to be in the standard library in the sense that the compiler and distribution do not depend on it. It’s such a core interface, however, and so ubiquitously used, that it should definitely be included. A rule of thumb here might be that if a dependency shows up in 90% of Cargo projects, then it’s a strong candidate for being in the standard library.

To comment on some other points you made:

Some people don’t like having dependencies.

My only reservation about this is that it’s often nice to write small one-off scripts, but Cargo does not currently support this very well. For example I can’t gist a playpen snippet which makes an HTTP request or parses JSON. Otherwise, though, I definitely agree that in general “having fewer dependencies” is not a great motivating factor for moving content into the standard library. There are some strong benefits from being used as a dependency:

• Debuggability is increased because Cargo can compile dependencies with debug information and unoptimized
• Iteration on design can happen much more quickly outside the standard library, especially adding new features
• Living outside the standard library means that multiple versions can exist in an application. For example if large breaking changes want to be made to an API, it can be done without breaking other downstream code as you’ll just include both versions.

Curation: being included is a sign of quality and scrutiny, whereas I have no idea how good is a library I just found on crates.io

I agree that the rust-lang organization is a prime location for “curated crates”, but it will take some time to get here. I am not personally proud of the API exposed by the time crate today, for example, and reworking some of the existing crates we have will take time. Overall, though, I expect this to become a good place for high-quality libraries to live.

[SimonSapin]

My only reservation about this is that it’s often nice to write small one-off scripts, but Cargo does not currently support this very well. For example I can’t gist a playpen snippet which makes an HTTP request or parses JSON.

Sure, having an HTTP client an JSON parser in the standard library is one way to fix that. But it’s not necessarily the only way. Could we make playpen accept a list of crates.io dependencies?

[DanielKeep]

My only reservation about this is that it’s often nice to write small one-off scripts, but Cargo does not currently support this very well.

<fantasy>

It would be nice if Cargo supported some kind of “all-in-one” format that was a Rust source file with a Cargo.toml prepended.

Give it a different file extension (say, .crs for “Cargo’d Rust Script”) and use cargo script to run them. Cargo should also learn to aggressively cache packages and compiled crates so that it doesn’t require a network trip every time.

See also rdmd.

</fantasy>

[alexcrichton]

@SimonSapin yeah that’s definitely one way to tackle the problem, and it may not be too too hard for the playpen to do some Cargo business, although it does make is much less sand-boxy as lots of network traffic would be needed.

@DanielKeep I agree that sounds like an awesome idea!

[SimonSapin]

Right, maybe that playpen thing isn’t realistic without something like Daniel’s idea (or at least the aggressive caching part). But that still doesn’t mean “std all the popular things!” is the right way.

[tomaka]

There’s also another advantage in putting things in the standard library: you are sure that every crate that you are compiling depends on the same version of the stdlib.

Let’s say that I write a library A. If I write a function that looks like fn foo() -> String, everything is fine because String is standard.

But now if I add another function that looks like fn bar() -> OtherLibrary::SomeType, then both library A and library A’s users must depend on OtherLibrary (well, not strictly, but you run into other problems if you don’t). And if A and A’s users for some reason depend on different versions of OtherLibrary, then everything crumbles.

I have recently encountered this problem with vectors, matrices and images. I’m currently trying to solve it for vectors by adding impl Into<(f32, f32, f32)> for Vector in math libraries and fn takes_vector<T>(_ :T) where T: Into<(f32, f32, f32)> in my library that accepts vectors. But it causes some other problems like trait implementations being too generic, and it would be nicer if the Vector type was directly in the stdlib.

[killercup]

<fantasy using="attributes">

For playpen and small scripts, something like this might be enough:

#![cargo(dependencies("rust-serialize" = "0.3", "hyper" = "0.5"))]

</fantasy>

[td_]

It offers interfaces that should be “standardized” to make it easier for unrelated libraries to interoperate with each other. Things like the Iterator or Error traits.

This seems to me like the essential and most important role for std to have. Basically everything in Rust aside from the memory model is a library concern; while std can present a sane default for e.g. io, what’s more important are the traits and types defined in std that provide a lingua franca for other libraries to share so that they can be safely and easily composed. I think looser coherence rules will be needed for this to really be effective, though.

It would be nice if Cargo supported some kind of “all-in-one” format that was a Rust source file with a Cargo.toml prepended.

Give it a different file extension (say, .crs for “Cargo’d Rust Script”) and use cargo script to run them. Cargo should also learn to aggressively cache packages and compiled crates so that it doesn’t require a network trip every time.

This would be great! The way that makes sense to me to do it though is to use something like a specially formatted doc comment at the head of the file. Maybe a three-tick code block tagged “Cargo”, e.g.

//! ```Cargo
//! [dependencies]
//! serde = "0.4.0"
//! ```
extern crate serde;
...

rustdoc would then also learn to drop ```Cargo blocks.

[lambda]

I think you could avoid a lot of the network traffic, sandbox issues, and cargo issues by having a curated set of crates that could be used in the playpen, and having them available pre-compiled on the playpen machine.

There are a large number of crates that just wouldn’t make sense due to sandbox restrictions, the platform, the nature of the crate (who’s going to be running GTK apps in the playpen?), and similar issues. You don’t really want to open up a large amount more surface for downloading, compiling, running arbitrary build scripts, and so on from potentially arbitrary crates from crates.io.

So just pick a set of popular and fundamental crates that make sense to use in the playpen, have them available in the search path, and you get to use only the ones that are available there.

[DanielKeep]

@DanielKeep I agree that sounds like an awesome idea!

This would be great!

cough https://github.com/DanielKeep/cargo-script

I might have accidentally that fantasy…

[SimonSapin]

Maybe a three-tick code block tagged “Cargo”, e.g.

//! ```Cargo
//! [dependencies]
//! serde = "0.4.0"
//! ```
extern crate serde;

Strings in attributes can be raw strings:

#[cargo = r#"
[dependencies]
serde = "0.4.0"
"#]
extern crate serde;

[system]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.