Setting our vision for the 2017 cycle

I don’t really agree with this; writing Rust extensions for use from Python is pretty much all I do. The problem isn’t really with Rust, it’s with Python. Python’s FFI tooling is (understandably, I suppose) geared towards experienced C and C++ users, and correspondingly assumes that you “know what you’re doing”, forcing you into writing a horrible mix of C-ified Python.

Suffice to say, as someone who came to Rust via Python, entirely skipping C and C++, I do not really “know what I’m doing”, and spend all my time dealing with cryptic or exasperated answers on Stackoverflow, or Usenet, attempting to figure out incredibly basic things like how to instantiate arrays of Structure objects in ctypes.

In my experience, the advent of cffi hasn’t improved this in any way – it also assumes a C background. The Python community (perhaps this is generalisable to other dynamic languages – I don’t know for sure) needs a better “FFI 102” story, which assumes almost no knowledge of C / pointers, and provides intermediate examples (More complex than “here’s how you send an integer to a foreign function”, less daunting than “I want to send a pointer to a pointer to this 35-element struct, there will be callbacks” ).

High-quality, well-documented implementations have helped, here (python-rust-fst is a great example, and Jake Goulding’s Rust FFI omnibus is fantastic), but overall, there’s so much progress still to be made, and I feel that most of it needs to come from the non-Rust language side.

5 Likes

You’re right. I should have said when things come into scope of a more solid release date. Example. MIR got turned on and the whole original AST stuff got removed from nightly. That means bar any problems will likely show up in about 12 weeks (6 in nightly if landed at the beginning of the cycle, 6 in beta, and then stable). An estimated time of delivery when a goal becomes more solid would be nice. The dashboard could also link to reasons behind delays if that happens. It’s one thing to say this will be released by Q4 2017 with no work done vs this will possibly be released in the next 12 weeks and here’s the relevant PRs and discussions surrounding them.

I agree with the OP almost entirely, nice job! I have my own analysis of how we should focus - I think I would categorise the ‘vision statements’ differently, but I think we end up with the same concrete goals and mostly the same priorities.

TL;DR

This is kinda long, sorry.

  • I pretty much agree with Aaron. I think that some of the vision statements could be defined more broadly to perhaps encourage surprising solutions.
  • I think we must do “More systematic outreach/evangelism”, but probably not until the second half of the year.
  • We should be specific about where to hit the learning curve - I feel the whole curve is too broad to meaningfully focus on (my vote is to start at the beginning, second vote for the ownership system, since that is pretty unique to Rust).
  • I’m not sure we have the resources to focus on FFI as broadly as described. I’d be happier to focus on improving our C-interop, and other languages only as necessitated by the focus on a good server experience.
  • “mentoring at all levels” is (IMO) just a part of scaling the project and community, but an important one.

A word on core values

I would put this part (‘a word on core values’) at the top - I think it is really important. In fact, I think it is maybe even undersold. I think a year and a bit on from 1.0, Rust is really finding its own spirit. Beyond ‘better C++’ and the more technical ‘memory safety without GC’.

My interpretation of Rust’s value proposition is that it begins with performance (you only use Rust if you need high performance), but our distinction (once you are in the realm of high performance languages) is that you get vastly superior productivity. In particular, I think safety (whilst important) is really just one aspect of productivity. Safety means you spend less time debugging, in particular at late stages in the engineering process where debugging is very expensive (e.g., once software is released). It makes sense to me, therefore, to really concentrate on how we can make Rust users more productive, throughout their engineering processes.

Goals

When deciding what we are going to do, I think it makes sense to think about what we are trying to achieve. I think Aaron mentioned in the thread (I can’t find it now) that increased adoption is our primary goal. I agree (and I think we should have metrics to quantify that and concrete, numerical goals). I also want to emphasise that ‘quality’ of adoption matters not, just the raw number of Rust users (I’m sure the core team think this, I just want to be clear) - we need users who are committed to using Rust, who are investing back into the ecosystem and community, and have a concrete interest in our success.

I also want to think beyond pure adoption: if we sold out, dumped most of Rust and worked on a new version of Java, even if we got tonnes of users, I think we would still have failed. So, I think a goal has to be the quality of the experience of using Rust, not just the number of people having that experience.

Modeling adoption

My model of adoption:

user.realise_rust_exists();
loop {
    be interested enough to take the next step (and overcome the costs of doing so);
    ++Rust;
    if blocked || frustrated { break; }
    if evaluation.is_good() { continue; } else { wait_or_maybe_break!(); }
}

At each iteration, we need to ensure that users have the perception that the rewards (increased performance and productivity) outweigh the costs (time, risk of failure, friction, opportunity cost). We need to ensure that the experience of using Rust is not stopped by frustration (with the language or ecosystem) or blocked (missing libraries, platforms, inability to integrate with existing systems). Finally, we must ensure that users get the payoff they expect.

My vision for 2017 is motivated by addressing these steps, especially where we are weak.

Note that the first step is all about perception, not reality. But reality has to keep pace with the perception or we fail at the final step.

Focus areas (vision statements)

I think my breakdown is a little broader than Aaron’s (and perhaps less useful as a result), and I have aimed to be even more problem-oriented, rather than solution oriented (for example, “Rust should have 1.0-level libraries for concurrency and parallelism” feels a bit like a specific solution, rather than identifying an area for development, OTOH, my “ergonomics” and “first contact with Rust” areas are something of a solution to the ‘learning curve’ area).

In priority order.

Keep doing what we are doing well

Focus on performance, incremental improvements to libraries, core tools (rustc, Cargo), documentation, addressing bugs and soundness issues, being an awesome, welcoming, and helpful community.

Of course to keep doing what we are doing, we need to scale. That means growing the community, mentoring, building leadership, scaling the leadership teams, etc.

I think this is the most important thing, so important it is pretty much in a different category from everything else. If we can’t continue to get the basics right, it doesn’t matter how well we do the rest.

Maturity and appearance of maturity

We need to reduce the perception that Rust is an immature language or that large area of it are experimental. This will make Rust appear a much less risky proposition. That this is an issue is borne out by survey responses and by comments on non-Rust parts of the internet.

At the risk of getting into too much detail, I think we need to:

  • improve discoverability of libs and tools
  • address the backlog of unstable and unfinished features (DST, macros, SIMD, etc.)
  • improve the maturity of libs (‘the fear of 1.0’ problem)
  • improve reliability and feature-completeness of tools such as debuggers, editors, and IDEs
  • ‘1.0’-level versions of tools such as Rustfmt, Clippy, and IDE plugins

I think this is an important issue since it prevents potentially high-value users from trying Rust. Furthermore in industry, it prevents users getting buy-in from senior people and thus stops them iterating the adoption loop.

Ergonomics

I think, to a large extent, we have got a lot of the big things right in Rust, and at this stage there are diminishing returns on doing more big things. However, there are many, many paper-cuts and small frustrations that can cause users to quit the iteration loop. We should seek these out and address them in the language (NLL is a kind of a big ergonomic improvement here, but there are very many smaller ones too), the compiler (in particular, compile times), libraries, and tools.

I think this area is key, since friction and frustration seem to be reported a lot by Rust users and former Rust users.

Integration with existing systems and tools

It will be very rare for people to use Rust in isolation. Even if making standalone software, the process for doing so in Rust must fit with existing processes. We should focus on integrating with other languages (improved bindgen, language features such as unions) and processes/tools (cross-compilation, Windows support, IDE support, build systems). This is kind of tricky, we don’t want to add so much to the language we turn into C++++, and we could spend an unlimited amount of time and energy working with every possible tool or platform.

I rank this highly because it seems something that users and potential users often report as blocking adoption. Better interactions with other languages and systems also improves ergonomics (see above). However, I’ll qualify this a bit by saying that interaction with other languages is more important right now than supporting a wider range of systems of various kinds.

First contact with Rust

I feel like we need to put effort into converting curiosity into a serious experiment with the language, and in getting more curious users.

The first part of this is that I think we do a pretty good job of encouraging serious use (blog posts, website front page, outreach on HN, etc.), and the experience for people trying to learn the language is great (book, docs, Stack Overflow, irc, etc.). However, we don’t do a good job of getting people to ‘dip a toe in’ to experiment (c.f., React and Angular, for IMO good examples of doing this).

Secondly, whilst Rust is very well known (and has a generally great reputation) among web 2.0/hacker news/startup people, I think it is much less known in the wider industry - it worries me how many random engineers I meet who have never heard of us, and when they do know of us, they don’t know much about us (also true of many in academia). Beyond a point, we are going to have to appeal to the broader industry (Scala, for example, has done this very well).

I think this is the most important area to focus specific docs and communities effort (of course they are involved in all the other areas too). However, I don’t think we should push here too early - this is probably something to work on later in 2017, once we’ve made progress on other improvements.

Spike a specific market

I think we have the bandwidth to focus on a single potential market at a time and really make a push to improve both adoption and quality of the Rust experience. I think we have implicitly done this in the past with the browser and embedded language spaces. I think Aaron is correct that we should focus now on server implementation.

I’ve listed this last because it feels somehow orthogonal to the other areas, we should be doing all those things with this particular application domain in mind.

‘Non-goals’

I don’t think these are things we shouldn’t work on, rather things that aren’t important enough to focus on.

More expressivity in the language (e.g, HKT, int generics)

It would certainly be nice to improve here, but there are so many more important things to work on.

Infrastructure

It would be awesome to have more reliable and useful infrastructure, and I think we will have to invest in the future. However, my feeling at the moment is that while it is not ideal and sometimes frustrating it rarely meaningfully slows us down.

Other potential markets

I believe we can only afford to focus on one at a time.

12 Likes

One more concrete thing that would be useful here is a ready-to-use solution for integrating Rust components into existing CMake based projects. It would include officially supported (analogues of) rust-bindgen (C -> Rust) and rusty-cheddar (Rust -> C) and a set of CMake modules aware of Rust, Cargo and the C <-> Rust converters.

The "ready-to-use" aspect is important because code itself (D) and build system / infrastructure code (I) are often maintained by different people and it's D-people who are normally interested in using Rust, for I-people it's just an extra headache, unless using Rust is a management's initiative. If Rust is is introduced into a codebase as initiative from below, then trivial integration simplifies the process a lot.

I'm glad you've expressed your thoughts with so much precision; it makes it much easier to see where I disagree. :slight_smile:

(emphasis mine)

I don't agree with this at all! I think correctness - which is broader than 'safety' - is one of Rust's key selling points, and honestly should be pushed more. Rust code is performant, productive, and correct; its this whole trifecta that is so compelling. I think we should be messaging more about how Rust helps avoid bugs (not only low level safety issues, but high level bugs as well).

I also think when evaluating features we should be asking how they impact Rust's 'correctness property.' That is, is this feature going to lead users to write code with bugs, even if they aren't memory safety issues, or will it steer users away from common pitfalls? Examples of features that I think are advantageous to Rust's correctness property are - exhaustive matching, Result error handling, and the emphasis on parametric polymorphism over subtyping.


I think more expressivity in and of itself is not a good goal - in fact I think expressivity has certain costs and isn't unambiguously a good thing to pursue given infinite resources - but I think enhancements in expressivity will be a good avenue toward producing the kinds of killer frameworks and libraries that will make people want to use Rust (this is especially true in the server space). I think this should be driven by what feedback from library authors shows is really badly needed, rather than just importing type system features from other languages.

I think the focus here has been pretty good - specialization is easily the most important type system extension in my experience, and its the one that's actually implemented.

6 Likes

The ? operator, the -> impl Trait syntax, specialization, HKTs, and so on are all features that add expressivity to the language and that can be bypassed. In other words, if you truly want to do something without these features, you still can in a more dirty way. They also tend to generate endless debates, so I agree that they should be secondary.

But on the other side, integer generics is the only feature that I can think of that can't be bypassed realistically. There's no way to implement a trait for [T; n] where n is between 0 and 2^32, unless you are ready for some epic compile times. I don't think they are a good example of something that should be delayed just because it adds expressivity.

18 Likes

Great writeup.

I think ‘productivity’ over ‘safety’ is really key - to me, most people (eg: not Google/ Mozilla, who run native code on clients and have huge security budgets) are not going to prioritize security. It’s a “nice to have” but really, most don’t care - and security tends to also bring a long a lot of negative connotations like constraints. Productivity is something that the type system buys you - less time fixing bugs, faster bug turn around, avoiding technical debt without a huge up front cost. That’s going to resonate a lot more than secure code.

And the other important thing you’ve brought up that I think holds the language back is the feeling that rust is immature. In truth, that’s because it is for a lot of areas - the asyncio story has always held me back from recommending the language, and thankfully we’re seeing awesome work there. The need for unstable features is a deal breaker for production, because it increases that feeling of risk, which is already going to be a significant hurdle for anyone undertaking a language. Rust needs to feel like a safe choice.

Those are the two critical areas I see holding production adoption back.

This is really insightful; I'm super into it.

2 Likes

There was a recent discussion about design patterns in the user’s board: Modern Design Patterns in Rust

Some design patterns exist only for the reason that certain programming languages are missing some special features or follow certain paradigms. So I’d expect a guide to be like “A design pattern X solves a certain problem; how can I solve the same problem using Rust?” rather than “How can I translate a design pattern X into Rust?”.

6 Likes

Thanks for writing this up! Here are some personal bits of feedback, wearing my Servo & Firefox hats. Obviously, these are places where we might be willing to make investments from our side to assist in making them happen, as appropriate.

First-class Windows support Things are getting much better on Windows, but the debugging experience, native interop, etc. need one last “push” to close the gap. This is super important for desktop browsers, as >90% of desktop usage is Windows, and today developers really don’t like to try to work on Windows, partially because of toolchain issues.

Make Rust/Cargo first-class citizens in big build systems There are a few more small gaps here (e.g., [replace] usage, host vs. target rustc flags, being able to compile the stdlib as part of the build, etc.) where giving an inch will prevent big build systems from doing stuff like their own custom rustc builds or reverse-engineering crates.io hash procedures / building a custom cargo binary to avoid policies. There are also some simple tools around cloning a dependency to open a PR & setting up all the local environment bits that would aid in the frustrations big build folks (and Servo developers, tbh) feel with hundreds of dependencies.

Finish off a Rust semantics I get a bit worried because this has lingered with a lot of partial efforts for a while and there seem to be lots of pushes for integrating pieces of language features that are known to interact badly (e.g., when traits, modules, and/or HKT co-exist). Haskell gets around this a bit with their language features, which are just assumed not to “play nicely” together and to not be on any standardization/stability path, but so far as I have seen people don’t have the same assumptions about new Rust language features hidden behind unstable. Without getting some sort of tooling or framework in place for checking whether an extension is semantic sugar or deeper, I’m worried that as the language continues to grow in complexity, more safety holes will sneak in.

Some small elaboration on other high-level bits of the original post:

FFI Does this include support for inheritance-like patterns in Rust, as needed in the DOM? IIRC, this gap is still responsible for the vast majority of Servo’s unsafe code. Is there something we can do to help that move along this year? I think it’s now entering Year 4 of work on that, and we should either finish it or decide it’s never happening :slight_smile:

Concurrency and parallelism I’d love to see some more ideas around stuff that could reduce the number of threads. I’m a bit worried here because, especially on 32-bit platforms, we’re finding that with hundreds of threads we end up going OOM and failing thread creates with Rust due to the combination of address space fragmentation and base reserved stack. Further, hundreds of threads just doesn’t work all that great on some of our platforms. These problems will probably lead to ad-hoc solutions that kill our nice architectural separations provided by the use of thread boundaries and channels.

Rust should have a lower learning curve Have you considered reaching out to a user research group? There are several places (academic and commercial) that do formal user studies, recording first-time and experienced users working with a programming language, and then distilling that into some takeaways (possibly publishing a paper, as appropriate ). I’ve found that very useful in the past.

5 Likes

Newbie question: what does FFI stand for?

Foreign Function Interface

Beginners (college new hires) are a core component of any good production team. Not being able to point them at a reference will mean that either:

  • teams that use rust won't hire newbies
  • teams won't use rust because they can't hire newbies

This is particularly sad because rust strikes me as a great language for letting newbies touch the code because of its type system and safety guarantee.

2 Likes

Hmm, I’m not so sure that a college graduate should be considered a “beginner” in the sense that you were describing—i.e., needing an intro to basic computing concepts. This may be subjective, but I think that any college graduate shouldn’t have any problem whatsoever reading The Book. I certainly had no problem reading it as an undergrad.

Not all college grads are trained in software. I am an SET and we hire people trained in quality. Physicists, EEs (with minimal software training) and other disciplines are also frequently hired as level 1 and internship positions. Furthermore, python, perl, ruby, and self taught programmers could really benefit from a computer 101 chapter, which could benefit those groups immensely.

True beginners are the lifeblood of any community. Python is useful because you can pick it up and apply it to almost ANY discipline. Rust could be the same.

Not much to add to the general topic of the thread; it looks very good to me. From where I stand, the particular needs are IDE-level tooling and not-needing-nightly—so, very much in line with much of what has already been noted.

One thing does bear emphasizing, though:

Exactly this. I have an undergrad degree in physics and learned Fortran 90 as part of my senior thesis work (and took one intro-to-CS class taught in Java). I picked up C on the job at the first place that hired me, and there are enough beginner-friendlyapproachable C materials out there that that was possible. I was a "beginner" in the sense that I was very new to a lot of the jargon and terms, and I am quite sure that even in its current (excellent) form, The Rust Programming Language would have been over my head at that point.

To that end, and with my interest in teaching, I'd be very happy to chip in on such an effort... in late 2017. I have a Masters' degree to finish first. :wink:

I’ve spent the last six months reading the specifications and manuals for a wide variety of programming languages – Perl6, Haskell, Clojure, Go, Rust, and many more I’ve forgotten. I’ve read the Rust book, the manual, standard library and always read This Week in Rust, but I settled on Go for my language de-jour. Why?

  1. Rust’s generics are phenomenally hard to grasp for anybody new to generics. The syntax makes progressing in a tutorial actually difficult. Staring at nested angle brackets is like staring at a foreign language expecting it to translate itself. Haskell’s generics are really easy to read and therefore easier to grok

  2. Too much emphasis is put on meta-programming rather than productivity. I opted for Go rather than Rust because it seemed like I’d spend more time designing Traits & generics and not enough time actually making forward progress. Rust comes across very much as a Big-Engineering language. There should be more tutorials for practical projects such as writing a lexer/parser.

That said, I follow Rust closely and will be keeping a constant eye on its progress. Unlike Go, I truly think Rust is the replacement for C and will be one of the most critical programming languages for the next 25 years.

BTW; for IDE support, have you considered publishing a certified EBNF file for the language,and using scripts to automatically generate plugins for the various editors? This way editor plugins can be be kept up to date with the language spec with no effort.

1 Like

I'm working on the generics chapter of the new book right now. Would love your feedback, if you have the time:

I think this is interesting, because in my mind, metaprogramming is one way that you gain productivity. Copying and pasting things over and over isn't productive; in the long run, it's draining. But then again, I come from languages that use metaprogramming extremely extensively, so it might just be that...

This is one of those things that we'd like to do, but nobody has the time to do. It'd be fantastic. Grammar is the closest we've got.

1 Like

I’m not sure. While you can certainly write a safe program in every Turing-complete language (e.g. by writing your program in a proof checker), normally-developed programs end up with the safety problems of their language - C programs with buffer overflows and PHP programs with SQL injections.

Many companies do not give a **** about safety. However, with Rust’s current design I don’t really see it going in a less-safe direction.

In terms of FFI, long-term I do think we want better integration for objects with “foreign lifetimes”: Python objects managed via Python reference counting, JavaScript and DOM objects in a browser, objects managed by a foreign garbage collector, EFI firmware objects, and other “external” objects. I’d love to have a better solution for those than just “write a wrapper using a raw pointer”, when a large family of otherwise unrelated objects all use the same “foreign lifetime” mechanism.

However, I don’t think that should go in the 2017 roadmap. For 2017, I’d like to see exceptional support for C FFI: better library integration, better build system integration, improved cdylib support, sonames, and anything else needed to transparently put Rust anywhere C can go.