Priorities after 1.0

It should come as no surprise that life lately has been dominated by the upcoming Rust 1.0 release. But now that the beta release is out the door, I’ve started to give some thought to what we ought to do next. Obviously, there is no shortage of good ideas out there for how to grow the language. Just as obviously, we need to be careful about what we implement and when we implement it.

This post represents an initial stab at assigning priorities to the upcoming tasks. It is based on my attempts to survey the many proposals for extending Rust, as well as talking things over with users of Rust (particularly production users). I’d really love to hear more feedback (here on the discuss thread, ideally). I also expect we’ll be constantly adjusting priorities as we go. I plan to draw up a public Google doc that we keep up to date tracking this sort of high-level priorities information; haven’t done it yet.

How to decide what to prioritize

So how should we decide what to work on now versus later? I think the two most important criteria are time-sensitivity and impact:

  1. Time-sensitivity. Some changes are just going to get harder and harder to do over time. Particularly around unsafe code, de facto dependencies and assumptions can crop up and make it hard to introduce changes. Some examples where this might be relevant are dynamic drop, allocators, and tracing support.
  2. Impact. Some features have the potential to affect a large number of problems, or else a large number of users. An example of the former is trait specialization, which can potentially address a number of existing problems and needs. An example of the latter is improved compilation performance, which affects every Rust user.

Top priority

These things seem like they are the most urgent to me. Many of them already have people working on them. Those that don’t will be top priority as people-power becomes available. I’ve put down names of owner(s) for each item: these are the people that are currently planning on driving the design and impl work forward. If you’d like to be more involved in one of these items, they would be good people to contact!

Library APIs. We’ve stabilized a large portion of std, but there are many important things that didn’t make it. Issue 24028 has a more complete list, but I want to highlight a few things in particular as needing attention:

  1. Filesystem APIs. We plan to flesh these out so that we can get the cargo implementation running on stable Rust.
  2. Memory allocation APIs. We know that being able to allocate a bag-of-bytes would be very useful for a number of projects. The main reason we haven’t pursued this is because of concern about possible interactions with GC/tracing support (see below). But we plan to introduce some kind of stable allocator API very soon.

In addition, we would like to put some energy into developing useful libraries that are not (necessarily) part of the standard library but which fill important gaps. For example, I know @aturon is chomping at the bit to build some parallel programming abstractions. Owner: aturon, acrichto.

Better Windows and ARM support. Both Windows and ARM are huge, important markets where Rust support still needs work. Improving our support in these areas will help make Rust attractive for a larger number of people. With respect to Windows, the overall goal is to remove our dependence on the MinGW toolchain but this will take a number of steps to achieve (more discussion here is needed). With respect to ARM, the primary goal is support for the ARM64 architecture. Owner: brson.

Faster compilation. We’ve put a lot of effort into building up the feature set and relatively little into optimizing. There’s a lot of low-hanging fruit here, but also an opportunity to rearchitect parts of the compiler and gain huge wins. I’d like to see progress towards parallel and incremental compilation, for example, both in the backend (LLVM side) and the front end (type checker, etc). I’d also really like to take some time to overhaul the trait matching and type inference scheme; it needs some work to better handle associated types in particular. Owner: nrc, nmatsakis.

Better support for distributing Rust libraries and programs. There are a number of things we can do to make distributing Rust executables and libraries easier. cargo install is an oft-requested feature that will require some careful design. Another piece here is being able to distribute binaries with minimal external dependencies; being able to link to MUSL might be a big help here. Owner: acrichto.

Better tool support. While Rust now has fairly decent debuginfo, it can always be better. And it’d be great to have integration with other tools like fuzzers, line coverage, profilers, etc. Having a healthy ecosystem with quality tools – and in particular, being able to leverage C tools where it makes sense – seems crucial to widespread adoption. Owner: nrc, brson.

Dynamic drop. We need to remove the zeroing flags and implement the dynamic drop semantics we agreed to in RFC 320. It’s important to do this sooner rather than later to avoid de facto dependencies arising from unsafe code (Felix made a mitigating change aimed at encouraging people to write forwards compatible code, but that was only a stopgap measure). Furthermore, this change should improve compilation and execute performance across the board. Owner: pnkfelix.

Allocators/Tracing. We need to offer more control over memory allocation. This touches on a number of areas. For example, it can be useful to use different kinds of allocators in different parts of your program (in the direction of RFC 244). It would also be good to make it easier to control the global allocator. We need a stable routine like malloc/free. And, crucially, we’d like all of that while also having the ability to trace live values on the stack, so that we can integrate with garbage collectors – this is particularly important when linking to dynamic langauges or other GC’d runtimes, but might also be useful in plain Rust, particularly for some kinds of non-blocking data structures. (LLVM’s new and improved support for tracing offers exciting opportunities here.) This is a very time-sensitive topic: if we don’t act soon, the growth of unsafe libraries may severely limit our options here. Owner: pnkfelix.

Specialization. Specialization allows there to be multiple implementations of a given trait for a given set of types. Over the past few months or so, I’ve seen numerous examples of situations where specialization could help to address coherence limitations or to improve performance. I think it’s important to get the ball rolling on a design here. Owner: aturon, nmatsakis.

Virtual structs. For certain use cases, there is an urgent need to generalize the current setup of enum/structs and traits. This led to the so-called “virtual struct” proposal in the past. In the past, we focused on the efficiency questions, but I think there is also a need to make some patterns more convenient to write. In any case, previous experience suggests it’s going to be tricky to get the design right here, so I expect we’ll wind up spending some amount of time iterating in the RFC process. I’ve been doing a lot of thinking on this lately and I hope to put out some blog posts soon working through some aspects of the problem and the design space. Owner: aturon, nrc, nmatsakis.

Possibly high priority

There are some tasks whose priority feels harder to calculate. They are both tasks that will take a fair amount of effort, but it might be worth it!

Borrow checker improvements. I would very much like to start working on a concerete design to solve #6268 and #6393. Prioritizing this feels a bit hard though. On the one hand, both of these problems are (usually) relatively easy to work-around, but they affect a large number of new users, and removing them might go a long way towards improving the initial “Rust experience”. Owner: nmatsakis.

Improved macros and syntax extensions. Rust 1.0 includes the existing macro_rules system, but it has a lot of known problems. Perhaps the biggest is the lack of “privacy” or "unsafety" hygiene (meaning that it’s not possible to define macros that have privileged access over normal code). This prevents macros from being used to define abstraction barriers. There is also no integration with the normal import/export mechanisms of the language. Going further, Rust has always intended to have the ability to link in “syntax extensions”, which are macros whose implementation is written in Rust rather than using a declarative form. There is a prototype now but the API exposes far too many of the compiler internals to be considered stable. We need to better identify the use-cases we’d like to support and what kind of stable API we can offer. This is also somewhat tricky to characterize: any changes here will be backwards compatible, and many of the same effects can be achieved with pre-processing for the time being. On the other hand, several popular libraries (e.g., serde, sql, regex) have demonstrated the power and appeal of syntax extensions. Owner: nrc.

Mid-priority

There are a grab-bag of smaller RFCs and changes I’d like to see forward motion on. These issues seem to be relatively self-contained: it will still take some time to find a design everyone is happy with, but it doesn’t look like the design here will have large interactions with the rest of the language.

  • The try and ? operator for improved error-handling. See rust-lang/rfcs#243.
  • Overloadable += operators. See rust-lang/rfcs#953.
  • Overflow ergonomics. It’d be nice to be able to write overflowing operations more concisely, when you want them. Probably. See this internals thread.
  • Rvalue promotion. The idea here is to allow constants like &[1, 2, 3] to have static lifetime. Currently, they are limited to a lifetime that would permit them to be stored on the stack. A lot of the work on identifying which cases are legal has been done, but it needs to be written up and thought through a bit more. See rust-lang/rfcs#1056.
  • Smart pointer enhancements. In particular, we need support for DerefMove, IndexMove, IndexSet, and NonZero. Unlike the other cases, the design here actually is subtle, but we should be able to base the work on the existing traits in a fairly straightforward way. See rust-lang/rfcs#997.

Longer term

This final group includes changes that I see as being somewhat longer term. These are mostly larger, cross-cutting changes that will deeply affect the language and libraries. I think it’s great to keep thinking and working on these larger items – particularly on their design – but they are less likely to be integrated into Rust proper until other large features have been completed. This is both because it takes a lot of work to finalize the design and because the design will benefit from more time and experience. It seems likely that with an influx of new users, we will find new use cases (or new ways to solve the problem without changing the language).

  • Passing DST by value: It’d be great to be able to pass DST objects by value. At minimum, we need to be able to make Box<FnOnce()> work, but there are other interesting use cases too. See rust-lang/rfcs#990.
  • impl Trait: This case is tricky to prioritize. On the one hand, there is an urgent need to have some way to return closures and iterators without boxing them, which argues for doing it soon. However, this interacts with many parts of the type system, and there could be a lot of subtlety to handle, which argues for holding off until we have time for dust to settle. See rust-lang/rfcs#105.
  • Generic integer parameters: Sometimes it’d be nice to be able to write an impl parameterized over an integer constant, like impl<T,N> Trait for [T;N]. This might also be done through improved support for associated constants: impl<T,N:IsizeConst> Trait for [T; N::Value]. The primary challenge here in terms of the design is to decide how far we will go in trying to evaluate whether two constants are equivalent when we can’t compute a precise value; as far as the implementation, it’d probably be nice to do some refactorings of the associated type implementation first. See rust-lang/rfcs#884.
  • Finalizing the closure traits: Right now you can’t manually implement the closure traits. The reasoning is explained here, bu the key idea is to remove the “rust-call” ABI hack and replace it with…something. This might be variadic generics, or it might just be integrating the ‘untupling’ into the normal Rust ABI. It’s not entirely clear. This also interacts with passing DST by value, described above.
  • Smoother C and C++ integration: How cool would it be to be able to #include header files, or call methods on (simple) C++ objects (and have Rust code be called in return). This could be really useful for integrating Rust into existing projects.
  • async/await/yield: I’d like to see some sort of transform here, and I think integrating it into the compiler and type-system is worthwhile since we can be more accepting. But it bears some thought, we may be able to extend the type system in a generic way and then make yield a pure front-end transform See rust-lang/rfcs#388.
  • HKT: Lots of generic programming traits (e.g. Iterable) seem like they might benefit from higher-kinded types. But higher-kinded types also bring a lot of complexity, and there might be other ways to solve the problem (e.g. the existing higher-ranked trait bounds that we have). See rust-lang/rfcs#324.
  • Negative bounds: This is related to specialization, but different. I am concerned about the use of negative reasoning due to forwards compatibility concerns, though we have a precedent now for how to handle it. Still, I have a feeling that the need for negative bounds will be greatly diminished if we pursue specialization See rust-lang/rfcs#1053.
  • Extended reference types like &own or &uninit: There are various use cases that the existing reference types don’t quite cover. One way to address those is extending the language, though I’m personally very reluctant to do this without great justification. I’d rather wait and see how far we can get with library-based approaches See rust-lang/rfcs#998. This interacts with passing DST by value as well.
16 Likes

I think it's important to link to where and how to give this feedback.

Edited to clarify that this discuss thread is the proper place to, well, discuss.

I’d like to ask that tail call optimization (rust-lang/rfcs#81) be considered for mid-priority. It’s not exactly time sensitive but I think it has the potential to have a large impact on how rust code is written.

7 Likes

Just a random idea, but it would be nice to have precompiled versions of Rust for ARM, Android, and others.

This would not only make it easier to get started, but also allow continuous integration on travis or other services for Rust libraries and programs that target these platforms.

3 Likes

I think I won’t necessarily write any code using HKP, but I’m using libraries that want it. I also think the standard library might benefit from implementing some higher kinded traits so that it’s easier to generalize over common features of the standard library types. Same for generic integers. -> impl Trait will be something I will use in my own code, though.

I think impl Trait should have higher priority. Being able to hide implementation details w/o having to pay the boxing tax is something I’m really looking forward to.

3 Likes

I agree. There are a couple of questions about this from newcomers on reddit every week. Some limited form, restricted to return types at least would be very welcome.

You can already hide implementation details without paying the boxing tax. Look at std::vec::IntoIter, std::collections::hash_map::Keys, std::sync::MutexGuard, etc.

The impl Trait syntax is mostly a convenience that saves you from creating a struct. I don’t think that’s a priority as there are other things that simply can’t be done with Rust today.

I would much rather have compilation performance and dynamic drop (which could improve compilation performance) than any language features (especially impl Trait, implementation specialization, type-level numerics, HKT, etc.). In fact, I’d be quite happy for no new language features to be added for at least several months, because compilation performance is so important.

Laying the groundwork for incremental rebuilds, in particular, would be really awesome.

22 Likes

One of the most requested things in rust (at least that I have seen) is a standardized formatter. Coming from an eco system at my job where go is taking over, a large amount of the success is due to go fmt, go test, and the various go linters.

I think you really should - as a high priority - solve this part of the tooling asap.

1 Like

I basically agree, and this is roughly the plan, but I also want to get started on planning language changes now, as those have a long ramp-up time in the design phase. (And the more time there is for designs to bake, the better.)

4 Likes

+1 - From my (very small) sample, the biggest barrier to contributing compiler improvements is build time. Bringing that down will make it easier to implement every other feature on this list.

Beyond parameterizing with integrals, parameterizing with values in general would be great.

For example, in C++ the absence of parameterizing with strings has been a long-standing issue, and there are crazy meta-template programming work-arounds based on spelling stuff character by character, leading to:

constant_string<'H', 'e', 'l', 'l', 'o'>::cat< constant_string<' ', 'w', 'o', 'r', 'l', 'd'> >::type

The huge push for constexpr in C++11, and the various extensions brought by C++14 and C++17 clearly show the interest of programmers for more compile-time computation; mainly for two reasons:

  • Free Performance: a constexpr Regex::new would have the performance of regex! (without the hassle); build.rs somewhat fulfills part of this gap, but regex! exists because it is much more readable than defining all your Regexes in a file and referencing them throughout the project.
  • Better Types: it has been brought on already that the branching factor of BTree was not configurable until integral parameterization existed; beyond that, though, it would be valuable to parameterize a “pin-handler” by the address/width/mode of its pin (0/1/2/3 is not as readable, so enums?). I believe today it can be somewhat approximated by using a dedicated type providing the necessary constants by implementing a dedicated trait. For single-value arguments, it’s quite heavy-handed.

Beyond C++, in Scala they actually went all the way down with something like Rust procedural macros… hard to detect whether you should rebuild or not when that depends on the content of a database though.


However, I must admit that in terms of implementation, it seems like a really tricky thing to do.

Specifically, the C++ way (constexpr) has led to specific “evaluation code” popping in compilers to allow evaluation of an ever-increasing range of expressions. It seems hardly tenable to be honest, it means implementing features twice (one for constexpr and one for IR) yet guaranteeing that both results match. It also seems quite difficult to “go down”: what of low-level code such as calls into C, arbitrary memory manipulations, syscalls and inline assembly? Some of those are the building bricks of most collections (starting with Vec), and therefore most code remains inaccessible to this.

The other strategy could be quite similar to plugins: just compile and execute. LLVM can JIT code, the lowering to IR has to be implemented anyway, and since the compiler knows the in-memory representation of the result type, it should be able to “convert back” the raw memory of the result to an “AST value”. It solves all calls into C/memory/assembly issues, providing you can execute the code.

Unfortunately, executing the code gets hard in the case of cross-compilation. The result of std::mem::size_of for example depends on the triplet; it means that you would need to JIT for the target’s triplet, and execute it on the target… which is quite at odds with cross-compiling.

Beyond, there is the issue of deciding whether function should be explicitly annotated (better backward compatibility, but annoying when authors forget, leading to re-implementations) or implicitly guessed (easy re-use, but potential backward compatibility trap).

4 Likes

Agreed, compilation speed definitely is one of the biggest problems in any non-hello-world project. Lang/lib problems can have a workaround and even if it is dirty - you’re doing it once and it’s hidden for a while. But slow compilation is always there, every single day.

If prioritizing cargo install would cause people to start uploading Rust applications to crates.io, then it would have the happy coincidence of expanding our automatic regression test suite of community code. Though it’s possible that application authors won’t care about being on the stable channel as much as library authors will.

1 Like

I concur that compiler speed should be very high priority, but I don’t think that necessarily precludes feature development. Allow me to vote for issues that should be considered for high priority:

  1. Borrow checker improvements. First impressions matter, and “the borrow checker is too strict” is a meme with sticking power.
  2. The ? sugar for try. Feedback for this type of sugar seems almost universally positive, and the longer we put it off the more code that will exist using outdated error-handling idioms. Furthermore, adding ? to the grammar has the capability to break macros in the wild.
  3. impl Trait. Of all the features that Rust “lacks” (e.g. placement new), this is the only one that really sticks out as feeling “half-implemented”. Even if we want to hold off on the implementation for a while, we should work towards accepting a preliminary design as soon as possible.
2 Likes

Maybe for the Rust compiler. Compilation times of under a minute don’t hurt as much as bad performance caused by not being able to return an Iterator

Following @bstrie’s example, my humble vote, ordered by importance to me:

  1. Borrow checker improvements. I am constantly burned by borrows in the if-condition lasting the entire if-scope. Although the workaround is simple, I agree that this would help first impressions and reduce frustration.
  2. Generic integer parameters. So I can serialize [T; N]. And to see all those hard-coded trait impls in std go away.
  3. Virtual structs. I go back and forth on this, though, because I have found a few interesting ways to achieve pretty ergonomic single-inheritance using generic structs and traits.
  4. ? sugar for try.
  5. DST by value. Because I think it’s nifty :stuck_out_tongue:

What is expected timing for associated constants and const functions? They are big and mostly implemented changes, but they will not land without serious attention from someone in the core team.