Hi everyone,
After almost seven years of working on Rust I will leave my position in the compiler team at the end of this month and in general step away from the everyday operation of the Rust project. I don't expect to regularly frequent the usual communication channels or read my Github notifications. You might see the odd contribution from me here and there but I intend to mostly stay away from computers for the next few months :)
For those of you who are interested, I'd like to highlight some of the things I've been working on and share some thoughts of what that's been like:
-
Implementing the first proper support for debuginfo was what got me involved with the Rust project in 2013. I was pretty happy with the outcome at the time and there's still a surprising amount of the original code in the current version of the compiler. I'm glad that @brson reached out to me, offering me a chance to keep working on debuginfo after that initial internship. Otherwise I might have spent just those three months with Rust instead of several years
:) -
Implementing incremental compilation in the Rust compiler is probably the most interesting and challenging thing that I've worked on in my career so far, touching in one way or another almost every part of
rustc. I am proud that we were able to go from the somewhat naive beginnings that @nikomatsakis and I came up with to something quite solid and decidedly less messy over the course of a couple of years. It's not uncommon now to see two to seven times faster compiles after small code changes and running, for example, a stage 1codegentest after modifying thelibrustccrate goes from 4min 40s to 2min 7s on my machine when I turn on incremental compilation for the compiler itself. Although there undoubtedly continues to be much room for improvement, I think this is an important feature for increasing Rust's acceptance among developers and I'm glad it's been turned on by default fordebugandcheckbuilds for the past years. -
One of the areas I inadvertently came to know quite intimately was parallel codegen & ThinLTO - both things that already existed before they appeared on my radar but which also had to be adapted considerably in order to make them compatible with incremental compilation. I'm glad I could always rely on @alexcrichton to review the often hairy and complex pull requests I made in this area. That part of the codebase could really use a thorough cleanup. On the other hand, parallel codegen might be the single most important feature for compiler performance and incremental compilation would be almost useless without it. And incremental ThinLTO is quite something, producing fully optimized builds in a fraction of the time.
-
After incremental compilation had been stabilized, the next big feature I worked on was Cross-Language LTO (aka "xLTO"). xLTO enables LLVM's link time optimization to be performed across a mixed C/C++/Rust codebase, making it possible for important optimizations, such as function inlining, to be performed across individual compilation units even if one of the compilation units is written in Rust while the other is written in C++. xLTO is a feature that few of you might be using directly but for big projects like Firefox that gradually want to integrate Rust into their C/C++ codebase it is quite valuable.
-
In late 2018, @eddyb and I developed a new symbol mangling scheme for Rust. It started as a "quick fix" for some immediate issues that arose around the existing mangling scheme but of course once we had touched it we wanted to get it right and, as was to be expected, it took way longer than expected. In the end it turned out really nice and has since helped me a lot debugging various symbol name related issues. @eddyb's implementation in the compiler is still behind the unstable
-Z symbol-mangling-versionflag but I hope it will soon become the default after external tools have picked up the appropriate demangling capabilities. -
One of the things that I worked on most recently is the support for profile-guided optimization in the Rust compiler. Once I figured out how everything fit together, I was able to mostly rely on LLVM to do the heavy lifting. But I think PGO is a really good feature to have in a mature compilation toolchain.
-
Another late addition to the things I've been working on is the Rust compiler's "self-profiling" feature, which consists of a tracing framework on the compiler side and a set of tools that do various things with the profiling data. I honestly was surprised by how very useful this feature turned out to be, mainly facilitated by the great tooling @wesleywiser and others wrote, and @Mark_Simulacrum's perf.rlo integration. It is so very helpful to get a detailed and/or visual representation of where the compiler spends its time. I'm glad I got to work on this feature.
There were also things that I found challenging while working on Rust. It has always been a project with a staggering amount of communication going on - on Github, on Discourse, on the mailing list, on reddit, on Zulip, on Discord, during conferences, at team meetings... you get the picture. I always found it hard to not be overwhelmed while trying to process all this information coming in every day (and night), trying to figure out what to pay attention to and what to discard, trying to moderate and negotiate the manifold claims being made on my time and energy from all kinds of directions. I'll admit that it is with some relief that I'll be stepping away from this communication fire hose for the time being.
I'd also like to mention that working on compiler performance can be a somewhat thankless job. Improvements often go unnoticed and are quickly expected as the new baseline, while at the same time there's a constant stream of complaints about how slow everything is. I get it, Rust compile times can be a real annoyance -- but making a highly optimizing and fast ahead-of-time compiler for a language that was not designed with compilation speed in mind, while evolving an existing codebase that also was not built with compilation speed in mind, is a hard task. I hope you give the folks working on it relentlessly the recognition and appreciation they deserve.
In closing I'd like you all to take a moment and reflect on how truly remarkable the Rust project is and how it managed, with very limited resources, to establish a foothold in the contested space of serious, production-ready programming languages. I'm proud to have made my small contribution to it and I'm thankful that in doing so I was able to meet and work with a set of really exceptional people.
I'm going to wrap up a few things until the end of the month and then I'll take an extended break, focusing on my new role as a full-time dad for a while.
Be nice to each other!
-mw
Thanks for everything!
