Setting our vision for the 2017 cycle

I would love that. Nobody has yet stepped up to do so, though.

It really, really depends on how you learn. Some people do respond well to videos, but others cannot learn anything from them.

@kennytm - I couldn’t agree more, and it’s been a constant refrain since joining the team. Unfortunately, the solution is not so clear.

You can see what takes the compiler time pass by pass by looking at http://perf.rust-lang.org/table.html?kind=rustc&date=Wed%20Aug%2031%202016%2019%3A56%3A15%20GMT%2B0000%20(UTC)

(I just grabbed a recent build date)

Notice at the bottom across all parts of the compiler build and what pass takes the most time. Poking around, you’ll see the the compiler itself is actually pretty snappy, but a lot of time is spent in translation, linking, and - most certainly - the llvm passes.

As it stands today, the rustc compiler asks quite a bit from llvm, passing it all the specialized functions in order to get the absolute best performance in the output code. But it has a trade-off in terms of how long llvm will have to chew on what it’s given.

Am I making excuses? Absolutely not. I want us to be able to bootstrap at Go and D’s speed. The question is, though, how much can we shave that without cutting into the runtime speed of the output code?

4 Likes

So, my concrete use for it:

I’ve got a large C++ project that is basically WebAudio for desktop. It works by building a directed acyclic dependency graph of node base classes, each base class holding shared data like the dependencies of the node and some pre-allocated buffers. The base handles connection information like channel layouts, holding a mapping of integer keys to property instances, etc. Then, each derived node type comes in, provides a creator function, subclasses, and puts whatever data in it needs to have.

With specialization and fields in traits, this would be translatable to Rust, for the most part; without them, the things you have to do to make it work are decidedly not fun. Even with specialization and fields in traits (assuming fields in traits gets approved), you still can’t just say “here is my base object, I want to override this one method”. Instead, you have to jump through annoying hoops to set up a trait that contains the base field and implement it.

The logic in this project that allows it to integrate with Python’s GC then uses Any. I don’t think Any's differences from dynamic_cast are restricting in this case, though I’d need to read the extremely complicated GC integration thing I did to see if that’s true.

This is a pattern I see a lot with graphs. You want some sort of uniform interface with some sort of default implementation and some bookkeeping that’s done automatically, then you want to override the methods that actually do type-specific things.

I don’t believe in multiple inheritance, but single inheritance like this is an incredibly useful thing to have even in situations that aren’t OOP. Moreso if you can somehow bring in selected trait implementations of the base that you know are going to be correct because you don’t break their invariants.

2 Likes

I’m ops for all of Mozilla Research, and have recently been addressing infrastructure issues on Servo that we established were more urgent than Rust’s. Rust’s goals, especially as clarified by the community roadmap, are one of the things that guide where my time is allocated!

15 Likes

I like to add that the problem with Serde json/xml, to me at least, it is really a question off better reflection / introspection in the core language or in the macro system.

It would be so nice to be able to extract type info for struct’s and functions, in a static way, and of cause also to be able to call function using dynamically collected arguments (it will of cause be unsafe, and properly possible already, but I never came to that :-)). If that is possible we could map data at runtime using info the compiler gives us, and use Rust as our one and only IDL. It is i bit frustrating that this is possible in go and C++, ugly I admit but doable.

If we had real introspection, we could runtime check and map, remote RPC (json/xml/asn1) and keep nice static testing internally and use the really cool type system in Rust, without paying for it at runtime. This would make Rust the natural server language, together with proper parallelism as std.

I am not sure how it would be done in Rust (I have mostly been flirting with it, and I really like most of it), but I know that the compiler knows all kind of stuff, and C++ variadic could be a way to go, or something like that. The important part is that the compiler could “leave” some hints in the code we the could use to make dynamic argument collecting.

In the meantime, I have to stick to C++ (no go for me sorry, I hate its GC and related blackouts) :frowning:

From a game development perspective, if I wanted to write a game engine in Rust I think the main pain points right now would be:

Compile times - already covered here. Compiler plug-in stabilisation - e.g. serde issues already covered here.

Fine grained control over memory allocation and alignment. This is covered by Tomaka’s Vulkano troubles document https://github.com/tomaka/vulkano/blob/master/TROUBLES.md. Specifically I’d call out stabilisation of placement new syntax and fine grained control over alignment (of stucts, members, and heap allocations).

I think game dev needs are similar to embedded but not the same. Gaming platforms are pretty powerful these days and a lot of the time high level constucts are OK, but low level control is still necessary for performance. E.g. it’s common to use special purpose allocators. Dealing with graphics devices requires specific alignment and padding and so on.

This is actually much farther along than expected! There's some patches that need to be sent in for the standard library, but it's largely in a "someone needs to do the work" phase rather than the "super far away" phase.

2 Likes

One thing I didn’t see on the list that I am really hoping for is higher-kinded types, or in general more ways to express to the compiler what is desirable or allowed and what is not. I grant that this actually goes the opposite direction from making Rust more accessible, but in Scala a delicate helping of this is essential for helping me be both safe and productive in larger code bases. It’s possible this is different in Rust (I haven’t worked on a project large enough so that I desperately miss it), but I doubt it. My sense is that if you express this capability in terms of type lambdas with syntax very similar to regular value lambdas / functions, most of the “it’s a function!” concepts transfer over nicely.

2 Likes

‘discoverability’ as a high level goal would be interesting (elsewhere in the thread I think someone mentioned how they choose crates from crates.io - rather than ‘pick the biggest downloads number’, good discoverability would be useful here, as well as for finding crates in general areas of interest), but I’m not aware of a language community with a good solution (maybe aside from very targeted languages like Julia? I’ve never used it). It’s probably made more difficult by the variety of uses you can put rust to. For example, I never realised I needed manual lightweight userland context switches to stacks at custom locations in memory until I saw a post on reddit yesterday. I’d likely never find any use for such a thing in Python…but anything useful in Python probably has value in Rust - the set of discoverable things is bigger.

On the subject of the vision, something else I saw hinted at above (the “compile-time code generation” in @dikaiosune’s most recent summary) is the dependency on nightly in general. I’d love to be on nightly because I enjoy living on the edge, rather than because there are very nice features in nightly or because there are libraries I need that require nightly or because there’s a great library but it’s a compiler plugin rather than a macro so I need to be on nightly. There are always going to be cool things to try out on nightly, that’s one of the points of it, but I cringe at the thought of the papercuts that I’d feel at moving to stable. I can see that this line of thought is mentioned in the 2016 survey post and it arguably falls under “Rust’s core ecosystem should be maturing” in the OP of this thread. In which case, I’d suggest adding “reducing ecosystem reliance on nightly” to “Some potential avenues:” of that heading.

My main points:

  • Fixing bugs in rustc and core libs.
  • Stabilization of APIs. There are essential crates which need rust nightly. Even a performance test needs rust nightly.
  • Stable SIMD support
  • More performance improvements. C is still faster.
1 Like

Great question. Others have mentioned having sections in the documentation for people coming from other languages, which would be a great place to document how to extend those languages using Rust. Helix and Neon (for Ruby and JS) already seem to live on the https://github.com/rustbridge org, but the website doesn’t seem to work. Collecting all the other bindings like rust-cpython, maintaining them, and giving them visibility from the documentation and website would be a good start.

Rust needs a good set of “batteries included” stable libraries and some easy way for people to find them. This might require serde and some form of specialization to work on stable.

Rust’s type system is really missing integer generics, through we don’t know the implications of it. Looks realistic for EOY 2017.

Our tools team needs to have more people than just @alexcrichton (with @petrochenkov and @jseyfried cleaning up our frontend, our compiler is starting to look fairly well - great work! We need something like that but for rustdoc and our metadata loader). Get the FFI story done with!

3 Likes

I don't disagree, but any proposal here would have to make sure to address the failures of the platform proposal, and answer people's objections to it.

I definitely agree here, Rust could absolutely be taught to beginners.

However, I'm not sure I agree wth this, at least at this stage. Beginners aren't going to get Rust more widely adopted in production, which I feel should be the higher priority right now.

3 Likes

I don’t really agree with this; writing Rust extensions for use from Python is pretty much all I do. The problem isn’t really with Rust, it’s with Python. Python’s FFI tooling is (understandably, I suppose) geared towards experienced C and C++ users, and correspondingly assumes that you “know what you’re doing”, forcing you into writing a horrible mix of C-ified Python.

Suffice to say, as someone who came to Rust via Python, entirely skipping C and C++, I do not really “know what I’m doing”, and spend all my time dealing with cryptic or exasperated answers on Stackoverflow, or Usenet, attempting to figure out incredibly basic things like how to instantiate arrays of Structure objects in ctypes.

In my experience, the advent of cffi hasn’t improved this in any way – it also assumes a C background. The Python community (perhaps this is generalisable to other dynamic languages – I don’t know for sure) needs a better “FFI 102” story, which assumes almost no knowledge of C / pointers, and provides intermediate examples (More complex than “here’s how you send an integer to a foreign function”, less daunting than “I want to send a pointer to a pointer to this 35-element struct, there will be callbacks” ).

High-quality, well-documented implementations have helped, here (python-rust-fst is a great example, and Jake Goulding’s Rust FFI omnibus is fantastic), but overall, there’s so much progress still to be made, and I feel that most of it needs to come from the non-Rust language side.

5 Likes

You’re right. I should have said when things come into scope of a more solid release date. Example. MIR got turned on and the whole original AST stuff got removed from nightly. That means bar any problems will likely show up in about 12 weeks (6 in nightly if landed at the beginning of the cycle, 6 in beta, and then stable). An estimated time of delivery when a goal becomes more solid would be nice. The dashboard could also link to reasons behind delays if that happens. It’s one thing to say this will be released by Q4 2017 with no work done vs this will possibly be released in the next 12 weeks and here’s the relevant PRs and discussions surrounding them.

I agree with the OP almost entirely, nice job! I have my own analysis of how we should focus - I think I would categorise the ‘vision statements’ differently, but I think we end up with the same concrete goals and mostly the same priorities.

TL;DR

This is kinda long, sorry.

  • I pretty much agree with Aaron. I think that some of the vision statements could be defined more broadly to perhaps encourage surprising solutions.
  • I think we must do “More systematic outreach/evangelism”, but probably not until the second half of the year.
  • We should be specific about where to hit the learning curve - I feel the whole curve is too broad to meaningfully focus on (my vote is to start at the beginning, second vote for the ownership system, since that is pretty unique to Rust).
  • I’m not sure we have the resources to focus on FFI as broadly as described. I’d be happier to focus on improving our C-interop, and other languages only as necessitated by the focus on a good server experience.
  • “mentoring at all levels” is (IMO) just a part of scaling the project and community, but an important one.

A word on core values

I would put this part (‘a word on core values’) at the top - I think it is really important. In fact, I think it is maybe even undersold. I think a year and a bit on from 1.0, Rust is really finding its own spirit. Beyond ‘better C++’ and the more technical ‘memory safety without GC’.

My interpretation of Rust’s value proposition is that it begins with performance (you only use Rust if you need high performance), but our distinction (once you are in the realm of high performance languages) is that you get vastly superior productivity. In particular, I think safety (whilst important) is really just one aspect of productivity. Safety means you spend less time debugging, in particular at late stages in the engineering process where debugging is very expensive (e.g., once software is released). It makes sense to me, therefore, to really concentrate on how we can make Rust users more productive, throughout their engineering processes.

Goals

When deciding what we are going to do, I think it makes sense to think about what we are trying to achieve. I think Aaron mentioned in the thread (I can’t find it now) that increased adoption is our primary goal. I agree (and I think we should have metrics to quantify that and concrete, numerical goals). I also want to emphasise that ‘quality’ of adoption matters not, just the raw number of Rust users (I’m sure the core team think this, I just want to be clear) - we need users who are committed to using Rust, who are investing back into the ecosystem and community, and have a concrete interest in our success.

I also want to think beyond pure adoption: if we sold out, dumped most of Rust and worked on a new version of Java, even if we got tonnes of users, I think we would still have failed. So, I think a goal has to be the quality of the experience of using Rust, not just the number of people having that experience.

Modeling adoption

My model of adoption:

user.realise_rust_exists();
loop {
    be interested enough to take the next step (and overcome the costs of doing so);
    ++Rust;
    if blocked || frustrated { break; }
    if evaluation.is_good() { continue; } else { wait_or_maybe_break!(); }
}

At each iteration, we need to ensure that users have the perception that the rewards (increased performance and productivity) outweigh the costs (time, risk of failure, friction, opportunity cost). We need to ensure that the experience of using Rust is not stopped by frustration (with the language or ecosystem) or blocked (missing libraries, platforms, inability to integrate with existing systems). Finally, we must ensure that users get the payoff they expect.

My vision for 2017 is motivated by addressing these steps, especially where we are weak.

Note that the first step is all about perception, not reality. But reality has to keep pace with the perception or we fail at the final step.

Focus areas (vision statements)

I think my breakdown is a little broader than Aaron’s (and perhaps less useful as a result), and I have aimed to be even more problem-oriented, rather than solution oriented (for example, “Rust should have 1.0-level libraries for concurrency and parallelism” feels a bit like a specific solution, rather than identifying an area for development, OTOH, my “ergonomics” and “first contact with Rust” areas are something of a solution to the ‘learning curve’ area).

In priority order.

Keep doing what we are doing well

Focus on performance, incremental improvements to libraries, core tools (rustc, Cargo), documentation, addressing bugs and soundness issues, being an awesome, welcoming, and helpful community.

Of course to keep doing what we are doing, we need to scale. That means growing the community, mentoring, building leadership, scaling the leadership teams, etc.

I think this is the most important thing, so important it is pretty much in a different category from everything else. If we can’t continue to get the basics right, it doesn’t matter how well we do the rest.

Maturity and appearance of maturity

We need to reduce the perception that Rust is an immature language or that large area of it are experimental. This will make Rust appear a much less risky proposition. That this is an issue is borne out by survey responses and by comments on non-Rust parts of the internet.

At the risk of getting into too much detail, I think we need to:

  • improve discoverability of libs and tools
  • address the backlog of unstable and unfinished features (DST, macros, SIMD, etc.)
  • improve the maturity of libs (‘the fear of 1.0’ problem)
  • improve reliability and feature-completeness of tools such as debuggers, editors, and IDEs
  • ‘1.0’-level versions of tools such as Rustfmt, Clippy, and IDE plugins

I think this is an important issue since it prevents potentially high-value users from trying Rust. Furthermore in industry, it prevents users getting buy-in from senior people and thus stops them iterating the adoption loop.

Ergonomics

I think, to a large extent, we have got a lot of the big things right in Rust, and at this stage there are diminishing returns on doing more big things. However, there are many, many paper-cuts and small frustrations that can cause users to quit the iteration loop. We should seek these out and address them in the language (NLL is a kind of a big ergonomic improvement here, but there are very many smaller ones too), the compiler (in particular, compile times), libraries, and tools.

I think this area is key, since friction and frustration seem to be reported a lot by Rust users and former Rust users.

Integration with existing systems and tools

It will be very rare for people to use Rust in isolation. Even if making standalone software, the process for doing so in Rust must fit with existing processes. We should focus on integrating with other languages (improved bindgen, language features such as unions) and processes/tools (cross-compilation, Windows support, IDE support, build systems). This is kind of tricky, we don’t want to add so much to the language we turn into C++++, and we could spend an unlimited amount of time and energy working with every possible tool or platform.

I rank this highly because it seems something that users and potential users often report as blocking adoption. Better interactions with other languages and systems also improves ergonomics (see above). However, I’ll qualify this a bit by saying that interaction with other languages is more important right now than supporting a wider range of systems of various kinds.

First contact with Rust

I feel like we need to put effort into converting curiosity into a serious experiment with the language, and in getting more curious users.

The first part of this is that I think we do a pretty good job of encouraging serious use (blog posts, website front page, outreach on HN, etc.), and the experience for people trying to learn the language is great (book, docs, Stack Overflow, irc, etc.). However, we don’t do a good job of getting people to ‘dip a toe in’ to experiment (c.f., React and Angular, for IMO good examples of doing this).

Secondly, whilst Rust is very well known (and has a generally great reputation) among web 2.0/hacker news/startup people, I think it is much less known in the wider industry - it worries me how many random engineers I meet who have never heard of us, and when they do know of us, they don’t know much about us (also true of many in academia). Beyond a point, we are going to have to appeal to the broader industry (Scala, for example, has done this very well).

I think this is the most important area to focus specific docs and communities effort (of course they are involved in all the other areas too). However, I don’t think we should push here too early - this is probably something to work on later in 2017, once we’ve made progress on other improvements.

Spike a specific market

I think we have the bandwidth to focus on a single potential market at a time and really make a push to improve both adoption and quality of the Rust experience. I think we have implicitly done this in the past with the browser and embedded language spaces. I think Aaron is correct that we should focus now on server implementation.

I’ve listed this last because it feels somehow orthogonal to the other areas, we should be doing all those things with this particular application domain in mind.

‘Non-goals’

I don’t think these are things we shouldn’t work on, rather things that aren’t important enough to focus on.

More expressivity in the language (e.g, HKT, int generics)

It would certainly be nice to improve here, but there are so many more important things to work on.

Infrastructure

It would be awesome to have more reliable and useful infrastructure, and I think we will have to invest in the future. However, my feeling at the moment is that while it is not ideal and sometimes frustrating it rarely meaningfully slows us down.

Other potential markets

I believe we can only afford to focus on one at a time.

12 Likes

One more concrete thing that would be useful here is a ready-to-use solution for integrating Rust components into existing CMake based projects. It would include officially supported (analogues of) rust-bindgen (C -> Rust) and rusty-cheddar (Rust -> C) and a set of CMake modules aware of Rust, Cargo and the C <-> Rust converters.

The "ready-to-use" aspect is important because code itself (D) and build system / infrastructure code (I) are often maintained by different people and it's D-people who are normally interested in using Rust, for I-people it's just an extra headache, unless using Rust is a management's initiative. If Rust is is introduced into a codebase as initiative from below, then trivial integration simplifies the process a lot.

I'm glad you've expressed your thoughts with so much precision; it makes it much easier to see where I disagree. :slight_smile:

(emphasis mine)

I don't agree with this at all! I think correctness - which is broader than 'safety' - is one of Rust's key selling points, and honestly should be pushed more. Rust code is performant, productive, and correct; its this whole trifecta that is so compelling. I think we should be messaging more about how Rust helps avoid bugs (not only low level safety issues, but high level bugs as well).

I also think when evaluating features we should be asking how they impact Rust's 'correctness property.' That is, is this feature going to lead users to write code with bugs, even if they aren't memory safety issues, or will it steer users away from common pitfalls? Examples of features that I think are advantageous to Rust's correctness property are - exhaustive matching, Result error handling, and the emphasis on parametric polymorphism over subtyping.


I think more expressivity in and of itself is not a good goal - in fact I think expressivity has certain costs and isn't unambiguously a good thing to pursue given infinite resources - but I think enhancements in expressivity will be a good avenue toward producing the kinds of killer frameworks and libraries that will make people want to use Rust (this is especially true in the server space). I think this should be driven by what feedback from library authors shows is really badly needed, rather than just importing type system features from other languages.

I think the focus here has been pretty good - specialization is easily the most important type system extension in my experience, and its the one that's actually implemented.

6 Likes

The ? operator, the -> impl Trait syntax, specialization, HKTs, and so on are all features that add expressivity to the language and that can be bypassed. In other words, if you truly want to do something without these features, you still can in a more dirty way. They also tend to generate endless debates, so I agree that they should be secondary.

But on the other side, integer generics is the only feature that I can think of that can't be bypassed realistically. There's no way to implement a trait for [T; n] where n is between 0 and 2^32, unless you are ready for some epic compile times. I don't think they are a good example of something that should be delayed just because it adds expressivity.

18 Likes