Production user research summary

1

Hey, team.

During the week of July 13 members of the Rust core and community teams sat down over four meetings with a number of individuals using Rust in corporate and startup environments. The intent was to establish relationships, gather requirements, and otherwise solicit feedback to inform the project roadmap.

These were amazing calls, and we were very thankful (and kind of astonished) that so many people made the so much time for us.

I've tried to frontload this with the most pertinent information, but the takeaway I think is: we're doing well, our early adopters are happy with the product and happy to be a part of what we're doing, they need some high priority improvements but none of them are surprises.

This is a summary of the minutes taken in these original etherpads:

The organizations represented here include MaidSafe and Ironworks, both writing security-oriented code; tessel.io, who are creating a SBC kit with first-class Rust support; 12Sided, who are doing high performance distributed computing; Chef, who are experimenting with Rust for new tooling; Tilde, who produce a Rails monitor called Skylight that uses Rust for performance, and sandstorm.io and terminal.com, both making cloud platforms and not using Rust significantly (yet!), but employ prominent Rustaceans.

The rest of this document is divided between a prose summary and bulleted notes.

Summary

One of the areas we talked about extensively is the onboarding experience. A summary might be: things are not bad, but there's a lot to learn, and the docs still don't go deep enough in places, especially for experienced C++ programmers. One person mentioned that there's a lot to slog through before you can do anything interesting.

There were no tales of developers hating Rust, and it sounds like many are receptive. Some spoke of coworkers who work in C++ that are skeptical of Rust's benefits, and MaidSafe mentioned that one of their C++ contractors elected not to make the transition and left the project (MaidSafe was fine with this outcome).

Peter from MaidSafe did tell a heartening tale about how switching to Rust had opened development of their core system to the larger audience of web programmers in the project who otherwise wouldn't have touched the C++ codebase.

Documentation-wise, it's clear there's still much to do. We need more docs for C++ users, more examples for high-level things (e.g. requesting data from a URL), more instruction on Rust idioms. The proliferation of outdated docs continues to hamper Googlability.

When it comes to barriers to adoption, this quote from Seth Falcon at Chef summarizes pretty well the phase we're in:

Talking with others and we've got another project where the prototyping is happening in Rust but they expressed reasonable concerns about pushing forward into production (e.g. it may be rewritten in something else). Are we ready to stand up and build a team with enough rust expertise? We now have a handful of engineers who can read/write basic rust, but it's a big gap from having confidence knowing how to structure a larger project and not get lost. Want to end up with a system where you don't hate yourself 5 months in. Not clear what the good patterns and good practice are, unsure whether they exist. Organizationally those are some of our blockers which make using rust for a larger project intimidating.

There's a lot of trepidation about the newness of Rust. We'll overcome this with time and success stories.

Although everybody has their favorite warts they want fixed, more language features are notably not a major concern, with a few exceptions: catch_panic needs to be stable for embedding Rust to be viable, and is the only thing preventing Skylight from being on stable; we've also talked to a customer that is developing a process-sandboxed server app that should just crash on panic, and we don't have a stable way to do that yet (though there is a panic handlers RFC in discussion).

Other minor features were mentioned. The subtle interaction between trait objects and OIBITS continues to frustrate (this is the problem that leads to situations where e.g. some custom error type fails to implement Send and ruins the downstream fun because nobody's encapsulated errors can fulfill the Error trait (or something like that)). It's a big pain for the types of I/O dominant programs like Iron that reem works on.

On libraries, everybody wants one async thing or another and a DB abstraction layer. The unclear serialization story between rustc-serialize and serde needs to be resolved sooner or later.

While tessel supports running Rust, that support is not so robust yet and they need bindings to standard parts of the Linux driver stack: USB, SD flash, ALSA/PortAudio, L4V2, Bluetooth. Contributors wanted I'm sure.

Nobody complained about debuggers. Debugging may not be as vital to Rust as it is to C++, yet.

It's no surprise that lots of people lament compiler speed (though not everybody). We've heard some anecdotes that there is no mode to build Rust that is both reasonably fast to build (ala --opt-level=0) and also fast enough to run. Several people mentioned the long time needed to compile the compiler itself. I found this a bit surprising since our users shouldn't ever need to compile the compiler (of course some of these users also contribute to Rust). Hopefully as we get better platform support this will not matter.

We asked everybody what IDE's they preferred and there were no obvious winners, and a lot of people not needing a full-blown IDE but preferring SumblimeText or Atom. Early adopters are a hearty crew and don't require such niceties.

There are a lot of other minor tooling issues: real-time code reformatting was requested; Racer is still too slow to be practical; the lack of backtraces in errors makes debugging large code bases difficult. With the spotty Windows support, it would be very beneficial to know which crates work on Windows.

Cross-compiling needs to be more friendly - people shouldn't have to build the compiler themselves to cross-compile. We heard a lot of enthusiasm for being able to compile std via Cargo, which would go a long way to fixing the problem (note this isn't currently in the near term plans). Deploying to Android and iOS is entirely a manual figure-it-out-for-yourself affair right now.

Tessel's device is running MIPS, and Rust is miraculously working fine there. They also have a tantalizing ARM co-processor that they don't compile Rust for. There was a suggestion that Rust is still not suitable for microcontrollers because static allocation and initialization is so limited. Cross-compiling embedded code can't be done on stable until #[no_std] lands.

While everybody was happy to sit in two hour meetings with us this one time there was a lot less enthusiasm for making this a regular thing, nor did anybody make any strong requests for additional support channels. So right now I expect us not to, e.g., add new forums or mailing lists, but to follow up perhaps once a quarter with similar touch-base meetings, perhaps with slightly different format and new participants. Manish also suggested that the community team is a good contact point for production users, which I agree with.

Customer priorities and next steps

With what we've heard from customers, it feels like we're in a pretty good place, but there are some pressing issues. These are things that are potential 'deal-breakers' for some of our favorite users if things don't improve quickly.

  • catch_panic - Tilde needs it for embedding. Others do too.
  • panic -> abort / custom panic. We've heard from a customer (not on these calls) that must have the process abort instead of unwinding before they can deploy.
  • #[no_std]. This is keeping embedded developers off stable.
  • compiler performance

Fortunately, catch_panic and #[no_std] are all but done. We need to not let the panic/abort issue slip by though.

Here are other ideas for non-critical areas we may push on to improve the overall Rust experience based on this feedback.

  • docs. There's not a single big thing here, but it seems like there's still a lot of low-hanging fruit that can elevate the user experience. Several pain points were pointed out. (Personally, I think a community-led effort to bolster non-std API docs would have a huge impact.)
  • better cross-compile experience. Several people mentioned that things could be a lot better.
  • backtrace-capturing errors. Is there anything we can do here?
  • teach crates.io to report on the channel/platform compatibity of crates.

I'll set myself a reminder to follow up with another round of outreach in a few months.

Notes

The rest of this is just an outline of notes along several topics: onboarding / docs, platform support / cross-compile, tooling, libraries, language

Onboarding and docs

  • Meetups seem to be valuble (skade)
  • Too much to learn initially before being able to do real work
  • Mutability story with Cell is clarified pretty late in the learning process.
  • High-level examples are hard to find (e.g. get data from URL)
  • Searchable documentation on crates.io
  • Testing documentation is hard to find
  • Non-systems audiences
  • Concerns about hiring problems
  • Hard to know what ideomatic Rust is
  • Not enough docs for C++ users (per reem)
  • Rustdoc side bar is mostly useless
  • Finding all methods on a type is hard
  • Outdated docs on the web continue to suck. One suggestion to develop a culture of tagging Rust versions on blog posts.

Cross-compiling and platform support

  • Many people want easier cross-compilation
    • Building the cross compiler and libs is too unfriendly
  • tessel.io is using MIPS (mipsel-unknown-linux-gnu)
  • Unavailability of Cargo is a problem on smaller platforms (FreeBSD).
    • Bootstrapping Cargo dependencies on new platforms is hard
  • Lots of enthusiasm for being able to cross-compile std via Cargo
  • Plan to distribute std bins for more platforms via online installer seems well-recieved
  • Microcontrollers
    • Tessel's device has a coprocessor Rust could concievably target (ARM Cortex M0)
    • Need static allocation, for which Rust's libraries are not designed. (I think we're pretty happy with the story here - when needed this can be solved in libs ontop of libcore).
    • Need Cargo to support #[no_std].
    • Might need various custom linking behaviors (targets specs may be sufficient)
  • Cargo should give guidance when cross toolchains are not configured correctly

Tooling

  • Compile time could be better of course, but no major complaints
  • Several people complained about the compile times of the compiler itself though
    • (Not sure how real a concern this is for the greater audience)
  • Real-time reformatting
  • Racer still too slow. Is there work we can do here to improve things in the short term?
  • Tracking down error locations continues to be difficult because no backtraces, etc.
  • 12sided is using makefiles, not Cargo
  • More votes to pass arbitrary flags to rustc
  • Cargo should tell you whether crates work on windows
  • Library deployment on android and iOS
  • Need nicer JNI integration on Android, Cocoa bindings
  • Need better deployment to Android via Cargo (matt -Ironworks)
  • Error suggestions are often incorrect
  • No clear IDE winners, IDEs personal choice at all orgs talked to
  • Lots using emacs/vim/sublime/atom, not IDEs
  • Acknowledge that IDE's are valuable

Libraries

  • tessel kits run Linux and a good experience requires access to peripherals:
    • USB
    • SD flash
    • audio (alsa/portaudio)
    • video (l4v2)
    • Bluetooth LE (what stack?)
  • DB abstraction (lots of demand for this recently)
  • More curation - yes
    • rust-lang crates maintained by libs team
  • Lots of people want async I/O, async/await, green threading, async things
    • ethusiasm for building on mio
  • Seth expressed desire for more cross-platform conveniences, e.g. recursive copy (ala Ruby/Python).
  • Unclear serialization story between rustc-serialize/serde
  • Problems with protobuf lib not working (needs a proto compiler? not sure)

Language

  • Reem is dissatisfied with the interaction of trait objects with oibits.
  • Reem wants more introspection (why?)
  • Fixed-size arrays cumbersome because lack of type-level ints
  • Non-lexical lifetimes

Future production user support efforts

  • Enthusiasm for regular meetings is minimal - too much time commitment
  • Manish suggests using the community list as a 'switchboard' for production support
  • Seth is interested in providing feedback on upcoming features
  • Several people expressed needed a more reliable way to stay on top of the roadmap. Between TWIR and subteam reports we're still not surfacing the right info well enough. Maybe a feature dashboard.

FFI

Other topics

10 Likes
2

Thanks for posting this!

Was there any feedback on build systems or larger build integration, especially around Cargo? When I’ve talked with the Gecko team and various hush-hush companies with large autotools or CMake builds, the issues of managing Rust code - especially multiple different pieces of it - and some of Cargo’s restrictions like not allowing concurrent builds come up immediately.

3

@larsberg I haven’t talked to anybody else with the same kinds of large-scale integration concerns that Gecko has, no, and I don’t recall anybody making any pointed complaints about integration of cargo/rustc into other build systems. Unfortunately, I think Servo/Gecko are still trailblazing here. @rovar from 12sided mentioned that they are not using cargo at all and just calling rustc directly from makefiles, but their work is small-scale, using Rust to speed up hot spots. The only company I’ve talked to with scale comparable to Servo are doing a greenfield project completely w/in the Cargo ecosystem.

Perhaps @alexcrichton has information I don’t though.

4

I also don’t have too too many specifics, but I also got the impression that a number of production users were avoiding Cargo for the time being. From what I could tell, much of this stems from a “let’s build Rust exactly like C/C++” mindset. The model for building Rust code and then integrating into another application is different enough that it doesn’t seem to play well with existing build systems all the time.

That being said I’m also unaware of any precise concerns about integrating Cargo other than “that may not be trivial”

5

Perhaps we could automate this by making crater work on windows and then testing all the crates on crates.io and putting badges on each crate indicating which platforms and which versions of rust it builds with.

cough

6

I’m interested in any news on a database abstraction API. I maintain some sqlite3 bindings and I’ve studied sfackler’s postgres bindings. I’m subscribed to https://github.com/rust-lang/rfcs/issues/798 and I’d be interested in a real-time brainstorming session.

(I guess I could add a link in the other direction…)

I’d appreciate links to tracked issues corresponding to other issues mentioned in this article so we can monitor progress.

7

Cross-posting my comment from reddit:

Thanks @brson for organizing this whole thing, as well as painstakingly re-reading and summarizing the minutes. I really enjoyed the calls!

From the point of view of the core team, it’s vital to build connections to people using Rust in diverse contexts and with diverse backgrounds. These perspectives help us keep in touch with pain points when using Rust differently than we do, and to stay grounded in real-world use-cases. That in turn helps focus our efforts to improve the language, tooling, documentation, and ecosystem.

We collect this feedback in a wide variety of ways, from keeping up with this subreddit and our discourse forums, to attending meetups and conferences, to visiting companies interested in Rust. In this particular case, we wanted to solicit feedback from a range of people trying out Rust in a more “production” environment, loosely defined. We wanted to hear about the challenges of using Rust in existing organizations, integrating with large existing products, and in on-ramping new engineers, in addition to the general pain points mentioned above. It was also a chance for users to get to know one another, since each video call had several different organizations represented.

For me personally, it was very beneficial to have several hours of "immersion," hearing directly about people’s experience putting Rust to work, to get a real sense of how things are feeling. It’s hard to get that data any other way.

By sharing this data publicly (the full transcripts are available in addition to the summary), we hope both to inform the broader community of the challenges for Rust in production today, and also to provide a kind of “experience report” for people considering adopting Rust in their own organization.

Finally, I want to emphasize that this is one set of data points among many that the core team pays attention to. We’ve also been talking about holding an open “town hall” kind of call, though the logistics there are much harder to manage. If you have further ideas about how to gather feedback in a high-bandwidth way, I’d love to hear them!

8

This is super interesting. Regarding the lack of error handling docs, I’d been meaning to translate @burntsushi’s epic error handling set into a series of rustbyexample examples but I haven’t gotten to it yet (aside from @steveklabnik planning to import them into the regular docs sometime).

Regarding crate library docs, it’s tricky. Even crates which seem to have good examples such as regex with extensive docs on the front page lack structure aside from the what the library provides with sub-headers. For example, every wikipedia page in existence has a table of contents but regex does not. For something complicated, it’s nice to get an overview but that’s tricky with the docs.

One thing that could be done is have library authors create dummy modules strictly for dividing up complex topics into simpler topics (I’ve never seen anyone do it though)? For regex it would be useful to be able to click and go directly to say “repetition” as opposed to say “capturing” sections.

Aside from that, browsing an unknown lib is tricky. The most relevant post I’ve seen regarding that is this reddit post (maybe it’d be useful to people). It also has listed an example of how someone might browse an unknown library (it basically is almost like clicking things at random until you get a feel for it. Hardly great I guess).

I’d be super interested in seeing how the lib docs might be streamlined for the future. Really useful stuff.

[EDIT] Not suggesting dummy docs should be used, though with better support (maybe from the doc generator), more structured docs might be easier.

1 Like
9

As a potential future Rust indie game developper I have encountered lots of problem with Rust that nobody raised here. Off the top of my head:

  • When I profile my game the SipHasher is always in the top 3 of the functions that eat the most CPU. Saying that it consumes a lot of CPU would be an understatement. It consumes a ton of CPU.
  • Lack of write-only slices (my frames per second would increase by around 50% if I was ok with passing around slices that cause segfaults if you read from them).
  • Lack of clean and easy JSON handling. Some tasks require you to be format-specific and can’t be done with a generic deserialization system.
  • Poor handling of unsized types (I have had to write this kind of code for example).
  • Really a lot of missing libraries. Not just two or three bindings to C libraries, but the whole ecosystem. In fact I have spent most of my time on this problem for the moment (glutin, glium, hlua, cpal, spine-rs, android-rs-glue, and some others).

Since I’m alone, do this on my spare time, and don’t raise any money from it, I don’t have enough weight to be in these meetings (it’s not a complaint, I don’t have any problem with this).

However in my opinion it’s a vicious circle. When evaluating Rust, gamedevers are probably going to notice a lot of these problems and stay away from it. And consequently you won’t find any gamedev production user that will tell you what’s missing from Rust.

11 Likes
10

teach crates.io to report on the channel/platform compatibity of crates.

I started an issue on the cargo project on something similar Cargo.toml should include supported platforms info · Issue #1911 · rust-lang/cargo · GitHub. I will be happy to research and implement this.

11

I’m currently working to make Rust production ready at work on top of cargo but there are some definite pain points. The main one I’m running up against is having build boxes without internet access. Even if they did we wouldn’t want to depend on crates.io for production builds+deployments.

It’s easy enough to mirror the crates.io-index repo and also have a mirror for the package files but the architecture adds some complication. The fact that the dl url has to be specified as part of the index’s config.json means that I have to keep an internal fork of the crates.io-index just to modify the config.json and worry about future merge conflicts if/when something in that file changes.

I would love to see the ability to override the index’s dl url from the .cargo/config as this would simplify basic mirroring. I haven’t had the need to override api yet so I can’t say if there’d be value in allowing an option to override the index for that as well but it might be useful to others.

Edit: Opened issue on the cargo repo here: https://github.com/rust-lang/cargo/issues/1923

4 Likes
12

Thanks for doing this @brson, always good to get feedback.

Racer is still too slow to be practical

This surprises me (racer author). I would have thought the complaint would be 'racer is not complete enough' rather than performance. Performance on a completion on my linux laptop is ~100ms - i.e. the same sort of speed as intellij.

Maybe there's some environment related performance issue I'm missing - e.g. perhaps racer is really slow on windows or when used with a particular editor or something. Did you get a feel on what their environment was like?

Thanks!

13

I think this is a misconception. "production users" in this case was meant in the sense of "people that build projects with the intention of releasing things using Rust". It's very loosely defined. I wasn't involved in the planning, but to my impression, the group of people present was more defined by people the core team knew were using Rust in such a fashion/intention. Making money is also no criterion. It's not meant as an exclusionary attribute. It's just the meeting where advanced issues that might pop up in actual project use later are discussed, as opposed to meetings were problems with beginner teaching are discussed.

This won't be the last of such meetings and your feedback is very valuable and welcome. Would you mind if I get you in touch with the core team?

Florian (Rust community team)

3 Likes
14

Sure!

1 Like
15

We’re working on multiple projects using Rust at https://appsignal.com, ping me at thijs at appsignal.com if you would like us to join next time.

16

It's worth mentioning that @alexcrichton did some experiments and found that -O1 with codegen-units had almost the same compile time as -O0 but almost the same performance as -O3. We should make this an easier option to choose, I think.

17

The image library definitely runs much faster with -O3 than with -O1.

With the code I'm working on, loading the assets at runtime takes ~10.5 seconds with -O1 and ~1 second with -O3

18

@tomaka interesting. I didn’t mean to imply we should stop using -O3, however, just that for many users it may be a useful choice.

19

I can’t claim to have the same depth of experience as @tomaka, but I’m in the same spot of wanting to build and release games using Rust. If there’s anything I can do to contribute let me know.

20

Did you mean the same performance as -O1?