All Hands Recap: Tools Team

#1

All-hands recap: Tools

A couple of weeks ago the Rust team had its second All-Hands in Berlin (read about the first one), with people present from all the teams.

The tools teams had a bunch of discussions and a lot of progress was made!

Tools team reorganization

Recently, Nick stepped down from leadership of the tools team, so we took this opportunity to reevaluate how the tools team worked, and if the tools team was necessary to have at all (as opposed to just having per-tool teams).

We concluded that the tools team does have purpose: it’s valuable to have a team that has an overview of what’s going on with the tools, and help coordinate between tools. Furthermore, it’s important to look for “tool vacuums” and try to improve the situation when it comes to missing tools.

The nature of this team means that it doesn’t need to be comprised only of subteam leads; instead its composition can be people who are most useful to provide this overall perspective, which may include subteam leads. Furthermore, we wouldn’t need to meet as often.

To some extent figuring out leadership means figuring out accountability, you need to ensure things get done and this is trickier when there’s a team of people “in charge”. Furthermore, it is harder for the rest of the project to interface with a group of people as opposed to just one or two. So we’re going to have two overall “tools team leads” who are responsible for conducting meetings and following up on action items, as well as interfacing with the rest of the project. Two people leads to some redundancy and allows for more flexibility, but isn’t so large that things may get lost in the confusion.

We’re working on selecting our leadership and will have a separate announcement as that is finalized.

You can find the meeting notes here

Rustdoc

A few things have been decided for rustdoc:

Generate others outputs than HTML (we’d like to start with JSON) The goal behind this would be to allow better integration into other doc hosts, and let people have alternate ways of inspecting their documentation.

Improve documentation and the testing of rustdoc itself We had a lot of regressions this year due to the lack of testing. A plan has been started to add a crater-like server to test the UI directly.

Experiment with an archive server to reduce number of files This is one of the biggest pain points of rustdoc: it generates a lot of files which is a huge issue for Windows users . This is due to us wanting documentation to work without servers. We thought about how to reduce this number while remaining serverless, and the solution we found is a middle ground between the two: allowing the user to start an archive server (so there is only one file generated).

Intra doc linking This feature is one people have wanted for a while. It removes almost all pain from having to write paths in the documentation: rustdoc resolves the name and then generates the path itself. However, this is still incomplete, and we worked with the compiler team to figure out what remained to be done and how. We expect it to move forward very soon!

To sum this week up: the main word was maturity. Finishing features and improving users’ documentation experience will take over adding new features this year.

More information can be found in misdreavus’ post here.

IDEs

We’re excited to say that we spent a considerate amount of time talking about the future of the Rust IDE support and the fruit of that is formation of an experimental RLS 2.0 compiler working group:

Initially built on top of rust-analyzer, the primary goal is to discover an ideal infrastructure on top of which we can build an IDE-aware compiler, while also exploring how we can split rustc into more manageable chunks with better-defined boundaries.

With end-to-end queries (on-demand compiler model) defined as a goal at Rust All-Hands 2018 we discussed remaining work to be done and talked a great deal about possible incrementalization and “lazifying” of the compiler frontend. Currently, parsing, macro expansion, name resolution and partially HIR lowering are not yet translated to the new model and so most of the work and discussion was related to how we can tackle that and design the appropriate interfaces.

Some of the items high on the design todo-list include:

  • A way to concretely represent the program - typical IDE tools/transformations prefer a pure AST approach, whereas macro expansion phase prefers a token-based solution
  • Find a principled way to decouple name resolution and macro expansion - one can directly influence the other, thus limiting the possible incrementalization and laziness

Clippy

Last year was a great year for Clippy — we finally became usable with stable rust and distributed with rustup!

Most of clippy’s growth is just adding new lints, and we plan to continue doing that — we have a healthy contribution pipeline and it probably doesn’t need explicit intervention.

We discussed a bunch of implementation issues from the rustc driver perspective. Clippy is currently implemented as a rustc driver; which is basically a drop-in replacement for rustc. This works pretty okay, but often has bugs (usually windows). Furthermore, other custom drivers (e.g. rustfix) don’t interact well on the cargo side. Ideally, a holistic solution for driver integration in cargo would be nice. This would also help improve use cases like cargo-fuzz.

We also discussed alternative lint-writing approaches. Currently lints are written in plain Rust code, matching on the rustc AST and type information. It’s not too complicated, but ideally this could be easier to do. Furthermore, decoupling things from the AST helps stability overall. Something like Coccinelle for lint-writing would be nice, where lints are just written as Rust code with placeholders and conditions. rerast is one foray into such a space. There’s currently a grad student looking into this stuff.

We also specifically discussed short-term solutions for the driver issues causing cargo fix and clippy to not work nicely together.

You can find the meeting notes here


Rustfix and Cargo updates may come later

18 Likes

Tools team changes
#2

I know we have the Discourse “heart” feature to express gratitude for something or someone, but I just wanted to mention that I highly value these kinds of posts, including all the links to the relevant meeting notes, and the videos posted on the Rust YouTube channel.

It gives great insight into the way the teams work, and the goals set out for each team.

It also gives me a lot of comfort in knowing adults are “in charge”, and decisions are made based on community input, past experiences, and the overall goal of Rust itself.

This is especially welcome, to counter some of my “anxiety” of the relatively small group of core contributors, and the inevitable change in personal lives and job situations that happen to those in charge.

Thanks to everyone involved for all the hard work. I’m very much looking forward to other retrospectives of the 2019 All Hands.

:two_hearts:

10 Likes

#3

As promised, here is the recap from the rustfix meeting.

Rustfix

The release of the 2018 edition also included a very cool new cargo subcommand: cargo fix. It enables you to quickly migrate over to the new edition by automatically applying the suggestions that are part of the compiler’s error/warning messages. But that is by far not the only use case this can have: The underlying mechanism (implemented in the rustfix library) is in theory able to apply any kind of suggestion the compiler can generate – as long as it can tell that the suggestion is unambiguous and leads to code that compiles.

The meeting we had in Berlin however was not about the general progress and future of rustfix or cargo-fix, though, but on one specific issue: Differentiating suggestions with multiple possible fixes from fixes that need to change multiple parts of a file. One issue we had in the past was that the JSON-based diagnostics output that rustfix gets from the compiler flattens the (otherwise recursive) structure of suggestions. This is not a problem for the unambiguous suggestions we get from the edition lints, but there are some lints in the compiler (or in clippy!) that require rustfix to make multiple edits (e.g. this). As a precaution, right now we treat these the same as suggestions with multiple possible fixes (e.g., this). At the meeting we discussed ways to lift that restriction and enable fixes for the first case.

You can find the meeting notes here.

We also plan to establish an official rustfix working group. Expect an announcement soon.

6 Likes

#4

Transform them into one big fix that replaces the old version of the crate with a new version :stuck_out_tongue_closed_eyes:

0 Likes