We had our second steering meeting today. We drilled into some of our plans for the future, and specifically into some ideas around batch compilation. You can find the meeting minutes over on the new compiler-team repository, which will hopefully become the home for “all things compiler team” over the fullness of time. =)
Some action items that I think I would like to get help with:
take another stab at collecting and organizing ideas, incorporating ideas around UX in particular, but also breaking out RLS from batch compilation (nikomatsakis)
gather statistics about where we spend our time from perf and perhaps elsewhere (?)
draw up a proposal for polymorphization analysis / strategy (maybe nikomatsakis?)
collecting some numbers on how parallel queries performs these days (mw)
draw the graph of refactor dependencies (eddyb/nikomatsakis, perhaps?)
I just want to say, even though I unfortunately don’t have time to participate in compiler stuff much, I think the way you’re working in the open on internals.r-l.o and sharing what’s going on in the team is really great. I hope more teams and/or working groups will adopt your style over time.
Given that there is the Mozilla All Hands this week, which has @michaelwoerister and I tied up, I propose that we postpone the steering meeting this week by 1 week (until Dec 14). I further propose that we also postpone the next steering meeting by 1 week until Jan 4 (because otherwise it would be on Dec 28, which is a holiday for many folks).
If there are no objections, I’ll adjust the calendar events. =)
Question: Do we want to have a meeting tomorrow? Will people be around? I’ve not really done much prep for it – I think we should probably however start talking about the upcoming Rust All Hands in a bit more detail, so maybe it makes sense to start that conversation tomorrow (i.e., try to figure out what would be the best meetings to have, etc)
In general, we decided that each of our slots should begin with some
exposition of “how things work
and then segway into “here are problems to resolve”. We will try to
take good notes on these expositions as well.
Some stream of consciousness thought about these issues I raised above follows. I apologize for the length. I’m going to try to post a follow-up comment here that digests these things, but I figured I might as well post this too.
Some of the problems we were talking about:
We are pretty good at tackling regressions and P-high things, but what about the backlog of random ICEs or other P-medium issues?
Can we and/or should we try to systematically attack those?
Can we save some time at the triage meetings somehow to make a bit of time for that?
Honestly, it’s still pretty hard to figure out just how much time we’ll have for dedicated meetings etc. But I think that if we could come away from this meeting with a sense of which are the most important conversations to have, that would be very helpful.
To my mind, the main goal of the All Hands should be us to establish a strategy for “rustc in 2021” – actually the problem may be bigger than rustc itself, since I think we should really also be considering IDE support, and that may imply some interaction with cargo etc. But basically what overall compilation architecture we are shooting for in the next edition.
Looking over the HackMd document, my sense is that the single most important discussion to have is this one:
Consensus on IDE Integration
what is our “end goal” – how do we envision the interaction of rustc, IDE, cargo to be in “2021”?
what are the steps to get there?
how can we get more interaction between rustc team and RLS maintenance?
In order to even have that conversation, though, I think it would be pretty useful to get reviews of how things are working today. For example, I would love to get a brief overview from @Xanewok on how the RLS is setup, and similarly I would like to hear from @matklad about rust-analyzer – the experiments that @jntrnr, @wycats and I have been doing on Lark are relevant, but I’m not sure if it makes sense to present about them specifically.
The above conversation (IDE integration) also touches on a number of other issues I think are relevant and potentially worth talking out:
end-to-end query architecture
I think what I mean by this is: given the goal we established in prior conversation, what are the steps we take to get there from here (how compiler works today)
integrating submodules and libraries
I would like to see rustc start to take a page from Servo and try to break itself up into more independent libraries. Things like chalk and polonius, yes, but perhaps for the type system, MIR transformations, etc.
Managing these as separate repositories has advantages: less bors time, etc. But it has costs: harder to coordinate changes, issue triage is hard to manage. What should we do?
Looking beyond the IDE a bit, and perhaps with a slightly more narrow scope, I think our biggest priorities are going to be:
Parallel compilation: More cores, more speed. What’s not to like.
Improved incremental support: Now that we have the incremental infrastructure, we need to dig into more scenarios and see why some of them aren’t paying off. Also, what can we do to optimize the implementation itself, such as faster serialization, deserialization, etc?
MIR optimizations: We need to set some specific goals, but there are improvements to generated code (fewer memcopies) that can be important, and I think things like MIR-based inlining may also be helpful for reducing time spent in LLVM.
Chalk integration: hopefully improves performance, unlocks GATs and some other more advanced features, hopefully fixes a bunch of bugs.
Of these, I think that MIR optimizations could most benefit from some focused group discussion. I can see various interactions. For example, @eddyb had proposed some changes to how MIR is setup to make it easier to perform optimizations, and it’d be great to go over the optimizations we think we might want to do and try to sketch out interactions with stacked borrows (cc @RalfJung) etc.
Chalk integration too has a fair amount of open questions, and it would be interesting to prepare and go over a lot of the details and try to map out some of the challenges we will hit.
Looking beyond the technical, I think the main thing I would want to talk about is reforming our structure a bit and specifically creating an apprenticeship pathway. For example, I would like to have the steering meetings start to transition to regular check-ins (perhaps in a round-robin fashion) with the various active working groups, getting an update on how things are going etc. Questions like triage and ice-breaking are relevant here too.
So my questions for the group might be:
Does that list seem like a reasonable set of priorities?
Should something on it be removed or replaced?
There is lots of work missing from that list – e.g., finishing NLL transition. Are there discussions to be had about that?
What about technical topics and reviews? I am now thinking that maybe it’s more realistic to do those as part of the lecture series, but I guess it depends on how much time we have.
Find answer to the question, “What does rustc + IDE integration look like in 2021?” and chart a course to get there, including specific goals for 2019. Relate this design to the key problems we are trying to tackle and (bonus points) discuss some benchmarks we can use to track our improvements.
Key conversations to have
I think my list of key conversations to have is something like this. This is only semi-ordered. I’d be curious to know if people think there are conversations that I am missing, and also what relative order you might place on them. In some cases, some of these conversations could perhaps be “conjoined” into one session.
I don’t know how much time we’ll have to get to these conversations. We may not have enough slots. This is why the relative priority is important – going into the scheduling, it’d be great to have a feeling for which conversations we think are most important, and perhaps back-up plans for those we don’t get to.
Consensus on RLS strategy going forward
How will we support/maintain RLS?
Should we adopt rust-analyzer approach? Can we share code with it?
One thing that @pnkfelix, @michaelwoerister and I were thinking is that it might be nice to have some “how things work today” reviews first, covering the RLS + rust-analyzer, and then have this conversation on the next day, once we’ve had some time to think those things over.
I know that @QuietMisdreavus wanted to discuss the relationship of rustdoc to the compiler. This feels strongly related to, but potentially distinct from, the RLS.
There are also some technical topics like how to generate docs for “all configurations at once” (maybe others?)
Major goals for 2019
What will be the immediate steps? Should we make a working group for each one?
Compiler team governance and apprenticeship
How should we organize ourselves and track progress?
Do we want to have a plan re: design documents and the like?
Can we do more to help apprentice-level folks move towards expertise?
Maybe touch on other mentorship topics, like lecture series, rustc-guide, etc
MIR optimizations and MIR 2.0
interacts with borrow checker, stacked borrows
Type-checking pathway: Chalk + NLL + fate of the HIR type checker
discuss the strategy and some of the obstacles we anticipate
maybe discuss things like how to rewrite type checker and inference?
One possibility is having “breakout
where a smaller set of people go off to focus on some questions and
then bring the results back to the group. It’s not clear how or if
this will fit in the schedule.
When it comes to the RLS, we decided to take a “tech first”
– basically focus on the technical design we want, and then talk
about the best way to get it (e.g., improve RLS in place, start over a
la rust-analyzer, etc). matklad, xanework, and nikomatsakis held a
separate discussion to settle on an initial agenda
I feel like we made some good progress on the concept of how to approach IDE discusson.:
We will start by elaborating technical design and let that guide us the rest of the way.
We will have some review materials available before the all hands
We will have a short meeting (like, nowish) to try and create agenda
it seems like we consider organizational matters the next most
important thing to discuss as a group. I feel a bit less clear on
what the “agenda” will be, but maybe we’ll feel our way, or else
maybe we should have some more pre-discussion about it. I would at
least be up for trying to prepare a kind of straw-person proposal.
We didn’t really get very far into what that agenda should be. I have definitely been thinking a lot about the RFC / stabilization process, but mostly I guess from the perspective of the lang team. I am very interested in trying to do more “holistic” planning, so that we are not working on language design questions without some idea of who will implement them, and without a rough stabilization plan (I’d like us to work more serially, rather than the current scheme where we just have a series of pipeline stages with unbounded queues that people keep tossing things into willy nilly).
So, definitely a crucial bit for me is the interaction between the lang/compiler team. On the one hand, I think that the “what it ought to do” and “how to do it” are definitely distinct concerns, but of course there is also feedback both ways between them. I think what probably makes sense is that when we are in the implementation stage we kind of have two “tightly coupled” working groups with overlapping members working in concert.
All that said, I wonder if this particular part discussion ought to be had with both lang + compiler team in concert. Interesting question.
Something I’d like to have soon is some kind of written-down policy on how handle out-of-tree crates hosted in github/rust-lang. That’s probably a topic for a steering meeting (but doesn’t have to be this one)?
Good suggestion. In our “meta WG” meeting yesterday, @davidtwco volunteered to go over our notes on that topic from the All Hands and create a summary, so maybe we should wait until that is done and we have a good starting point.