The Libs Team met today to discuss a weighty topic: what is its mission as a
team, and are we set up to achieve it?
As team lead, I took the liberty of proposing a mission statement:
To improve the quality of the crate ecosystem, as a product.
Working backwards:
“as a product” means that we need to focus on the end-to-end experience
people have with the ecosystem. It’s not enough to have great libraries if no
one can find them. It can be a problem to have too many libraries. Docs
count for a lot!
“the crate ecosystem” means that the Libs Team needs to look far beyond std
and help look after the library ecosystem as a whole. The Libz Blitz was one
of our first major attempts on this front.
“improve the quality” means that we don’t own or oversee the ecosystem,
but that we work together with library authors to improve the experience. What
quality means, and what aspects to prioritize, is of course also important to
nail down.
That’s a lofty goal! Let’s take a look at how we’ve approached it in the past,
and then talk about the future.
Full post is here – take a look, and leave your thoughts here!
First off, I really enjoyed this post and am excited by many of the suggestions.
I have some thought about issues with -sys crates in the ecosystem which I initially discussed in replies to your tweet of this post. Repeating the main points here for easier discussion.
TLDR One of the challenges I've faced with -sys crates is dealing with redundancies of them in the ecosystem and having high-level APIs tied to one or the other.
There's currently this weird repetition of work on core crates shared by servo and other projects such as piston. Main example is freetype and fontconfig-sys crates. I had to jump through hoops to get this these working correctly for Alacritty.
Piston has nice high-level FreeType bindings, but I had to couple it with servo's freetype-sys for some reason. I'll can go into details some other time. I forget the exact issue. As part of Alacritty, I've developed high-level fontconfig API in Rust, but have been unable to publish it due to this weird redundancy of sys crates.
The point I'm getting at is there's definitely missing sys crates, but there's also redundancy of existing sys crates which makes room for a lot of confusion or difficulties.
FreeType and fontconfig are kind of weird because fontconfig-sys will optionally link to libfreetype to expose additional methods, but this causes issues with Rust's "Only one crate can link to lib" rule.
I’m encouraging existing (and potential!) Libs Team members to add comments about what they find most compelling – i.e., what working groups they might be willing to form and lead.
For my part, discoverability remains top on my list.
Do non-team people post here? Apologies if not, but here goes if so.
For what I’m using rust for, game dev, I’d like to see libs that are no_std whenever possible, and for things that might otherwise just be std to make it all the way to core whenever possible. If this means that the no_std build has only partial features that’s fine too (example: rand can still run an rng using core only even if you now have to seed the rng yourself somehow instead of using the csprng from the OS).
This also means hopefully having some types that accept some memory that I give them, use it while they exist, and then go away without ever allocating/reallocating/de-allocating. I just want a nice interface over what I’ve allocated. For being a “systems language”, rust seems to be somewhat lacking in low level support of that kind (no one on IRC knew of any existing type that did something like that). Obviously there are a lot of catches and such, but those can be ironed out with careful thought I hope.
I’d also like to see crates and libs that have very clear interfaces towards solving whatever problem they solve. Preferably with little 10 or 20 line demo programs that show how you use the crate from start to finish. Rust can get a little Trait happy and doc light, and that can make coming into a library get confusing fast. Hopefully traits could say things like “you should implement this trait on your type if it does X and then it will work with this library” or “values with this trait usually come from the Y or Z methods”. Things in core and std do this quite well, but things outside that usually don’t do this so well.
I have somewhat a fear that the libs team doing too much review of the crates ecosystem (which is of course great) might leave too little time for the standard library which is vital to everyone. Do you share this view?
Also; What are, in your view, the reasons for and against having an extra ‘ecosystem team’ that is dedicated to working on crates, and an std team that works on the standard library (libstd, libcore, liballoc, and other nursery crates)?
I understand where this worry is coming from, but part of the strategy here is to substantially expand the team -- in part by decoupling it from std, so that we can have a smaller, dedicated working group for focusing on std. IOW, if this goes well, I anticipate better maintenance for std, because we'll have a tigher group that's more intentionally focused on it.
That's kind of what I'm proposing, with two differences:
First, it's a working group within the libs team, which ensures some level of coordination with activities around std and work like the API guidelines.
Second, I think that having a single monolithic "ecosystem team" won't work well; it's too vast. In the past, when we've had teams with that large of a scope, things don't function well because any given topic ends up not being that relevant to a good chunk of the people involved. That's why I think it's better to spin up more topic-focused working groups (SIMD, networking, concurrency, ...)
The Lib Blitz was a really exciting time seeing advancements in some crates and some hit 1.0, but the biggest take away I saw from it were the API guidelines that were published. I’m working through them in order to get nix to 1.0, and they’re full of very sage advice. I’d like to see efforts in the Libs Team expanding this knowledge to other areas related to developing great crates like:
Example CI configurations for some of the bigger CI players, especially as this relates to cross-platform compilation and testing (like using trust). I’d especially like if this includes GitLab CI, which generally gets neglected with these kinds of things. The crates deepest in the dependency tree are most important to have cross-platform CI testing as possible. We wouldn’t support as many platforms as we do with nix if libc’s CI testing wasn’t so robust.
This also means getting all supported platforms up to being supported in rustup. That’s the only things stopping us from supporting DragonFlyBSD in nix right now, that we can’t install it easily (same for OpenBSD I believe). It’d be good if there was better documentation and support around how to set up compilation and/or testing for other platforms either using QEMU, docker containers, etc. There’s knowledge hidden across a lot of different crates, but it’d be good to centralize that knowledge so everyone can benefit.
Clippy and rustc’s lints are very powerful for enforcing good code style. gtk-rs and winapi-rs go beyond these lints to enforce additional code style rules that accelerate development. It’d be good to see more effort towards adding more lints to rustc and clippy and to also enable more of them by default.
More validation of crates before uploading to crates.io. Right now a lot of failures when publishing to crates.io happens really late in the process, but it’d be great if the API guidelines above was an automated check run my cargo or a cargo subcommand. The easier the guidelines process is, the more likely people will try to follow it.
Regarding specific libraries I’d like to see some focus on the bottom-of-the-stack: platform libraries. This means winapi-rs, libc, mach/CoreFoundation-sys/IOKit-sys, ioctl-rs, and nix (note: I maintain this crate). These are all very low-level, in that they interact with the system kernel or user space libraries associated with the OS. It’s very hard to build software on an unstable foundation and these libraries are very foundational. The biggest issue is that libc needs a 0.3 release to fix various issues with it, but nobody wants to touch it. That’s the biggest wart I see, but many of these crates need work to build off the progress that’s been made in the ecosystem, like relying on libc because of its good CI testing and validation rather then declaring their own types. It would be helpful to get enough eyes on these foundational OS crates to set them on a path to 1.0 as part of a lib blitz, and maybe help them recruit some more devs to do that work.
(I speak as a hobbyist who watches a lot of game dev talks and tinkers with stuff in his garage but who doesn’t work in the AAA space or anything like that)
The short answer is that you can use std for some game dev, but not all of it.
Could you elaborate on that? Are you talking about games that must run on “weird” platforms (e.g. consoles)? Because apart from that I can’t think of anything where having the standard library around would be a downside.
There was a bigger post there, but it got kinda really bigger because I’m often bad at keeping it terse. jethrogb and I decided that such a discussion would easily derail this thread and so I decided to cut it all down to just the short answer there. Fear not, the full text of the post has been saved, and will be morphed into a larger and longer posting about rust that I plan to do at some point.
How about a rating system for crates.io, where if you’re logged in via github you can either tick a crate - meaning it did what you needed it to, or question-mark it, where you are saying “I couldn’t figure out how to do my thing”, with a comment explaining your problem.
You could then show metrics like “no. of ticks”.
I think if you have a rating system you want it to emphasize positivity - you don’t want people making unconstructive negative comments. That’s why I don’t think the question mark metrics should be publicly available, but they may be useful to the crate developer!
I don’t know how much any of this fall under the responsibilities of the libs team but I think these two could be tied. I’d love to have CI publish a new version on crates.io and push a git tag automatically after checking that that pass. (For example when crates.io doesn’t have yet a version with the same version number as in the commit being tested.)
I think these kinds of things definitely do fall under what the libs team should be doing. It's a part of what I would like the current API guidelines will grow into - a more fully fledged resource for helping people to make high quality crates.
I have looked into setting up published via CI before, but think we'd ideally grow crates.io's authentication support out a bit. In particular, I'd like to be able to create auth tokens that are only allowed to publish a single library to minimize risks if credentials leak.
I’m excited about the granularity and inclusiveness I think working groups will offer! There are a few things I’m personally interested in pursuing with respect to improve the quality of the crate ecosystem right now.
I’d love to see structured logging in log move forward.
It would be good to try rejuvenate some of our long-time nursery residents and revisit their paths to stabilisation that we identified last year. The nursery doesn’t seem like an ideal long-term home for libraries, and the times where we’ve been able to hand over maintainership to the wider community seems to have worked out well in the past. So maybe that’s a way forward for some?
I’d like to see what we can do to help some of our stable or currently stabilising dependencies flesh out their docs. I think serde sets a great example with its docs + book, but not all libraries would need it.
And I’d like to continue growing our resources like @sfackler mentioned, but in a write by doing fashion, where we support authors’ following resources while producing them.
I would like to announce the creation of the ergo crate ecosystem. It is very early stages and I am looking for others who are interested in contributing. All contributions are welcome, from discussion, design goals, maintainers and new crate submissions.
I would love to have the libs-team onboard as owners with a voice and decision making power within the ecosystem. Anyone who is interested please let me know!
One thing I’ve noticed while participating in Impl period is that there is a need for inactivity tracking automation of github pull requests and issues.
Some github issues had comments like “I’d like to take on that”, and that was the last activity on that issue. Sometimes issues like this were retaken and still nothing happened.
There had been another type of inactive issues: someone submitted a pull request, there’s been a discussion, pull rqeuirests has been updated and nothing happened afterwards.
Both of these scenarios could have benefited from pinging automation, like some sort of bot asking “what’s happening?” and changing issue state, like moving it from open - active - review - closed states. Another benefit is that this state may be integrated with impl period/libz blitz list of open issues, so that anyone looking for a way to contribute can checkout open issues only.
