[MIR] compiler-plugins for custom mir-passes

[oli-obk]

What is the motivation for this? It would be nice if changes like this could get some discussion - an internals thread, or even an issue being opened beforehand. I feel we have been burnt by llvm plugin passes and don’t want to repeat that experience. At the very least this facility should be documented.

@nrc : You are very much correct, it’s not good that we now have an interface to add new passes, which is solely designed for my very own use-case. My assumption was that MIR is so very much unstable right now, that it doesn’t matter (because the design of the interface would evolve along with the internal passes).

There are two things I want to be able to do with compiler-plugins and MIR:

1. Be able to implement lints that runs on MIR. MIR is so much easier to analyze than some ast/hir structures, especially once you get to control flow or dataflow. This needs an additional different structure than what we have now and does not require mutable access to the MIR structures.
2. be able to implement and test MIR-passes and operations without having to recompile the compiler.
   • there are different kinds of passes (see https://github.com/rust-lang/rust/pull/31448 )
     1. a MirMapPass that is able to access other Mir datastructures. You need that when you want to inline function calls.
     2. a MirPass that has no access to the MirMap and is thus guaranteed to only operate on one function’s MIR
     3. a MirBlockPass that has no access to the MIR or MirMap, and thus can only reason about very simple things like 5 + 6 should be 11

[nagisa]

@nrc, you said

I feel we have been burnt by llvm plugin passes and don’t want to repeat that experience.

Can you elaborate on this?

[nrc]

@ker thanks for starting this thread and for the explanations

My worry here is that we are adding yet another ad hoc way to plug in to the compiler and at a very early stage in the development of MIR. The current plugin mechanism is grotty and I am pretty sure we will want to change it in the long run, adding more uses of it make that harder.

I would be happy with having the MIR plugin stuff on an unstable and somewhat temporary basis, with the understanding that when we do reform plugins there will be a lot of breakage. Furthermore, once we move syntax extensions away from the registry model, we might want to remove the registry, breaking LLVM and MIR plugins with no immediate replacement (I don’t think we would do this because I don’t see a strong motivation at this stage, but I want it to be a possible course of action if necessary).

I do get very worried about any plugin mechanism to the compiler becoming de facto stable due to use by tools. This could adversely affect development of the MIR and a long term plugin solution.

I’m a little uncomfortable with lints in general - they make use of unstable compiler internals and I don’t see a way forward to making them work stably. I think they are an important tool, and I would love to have more of a sense of a good solution for them. Adding another kind of lint without having a proper vision for lints makes me very nervous.

@nagisa

llvm plugin passes were added basically because someone wanted one, they’re annoying because they are used in two places which is just enough to prevent getting rid of the code, but not enough to justify the code really being there. Furthermore, they are not documented at all, so anyone wanting to actually implement a plugin llvm pass is pretty stuck. I really don’t want to repeat that experience.

[oli-obk]

I’m a little uncomfortable with lints in general - they make use of unstable compiler internals and I don’t see a way forward to making them work stably. I think they are an important tool, and I would love to have more of a sense of a good solution for them. Adding another kind of lint without having a proper vision for lints makes me very nervous.

That applies to HIR-lints. AST-Lints could become stable once we have extendable enums. We’d have to stop passing them a Session object, but I’m sure we can figure out a minimal “session info” that contains things we can guarantee to be available after expansion.

HIR-lints are a different thing, because they currently have already been typechecked. If typeck moves to MIR, then a lot of HIR-lints are gonna break.

I do get very worried about any plugin mechanism to the compiler becoming de facto stable due to use by tools. This could adversely affect development of the MIR and a long term plugin solution.

While I personally don’t think anyone in their right mind would assume that code compiled with a nightly compiler will still compile the next day, I see your point.

[nikomatsakis]

AST-Lints could become stable once we have extendable enums.

This assumes we’d be willing to stabilize the current AST, which is not clear. But I do think your broader point stands: there will eventually be some sort of AST interface we would be willing to stabilize.

[nikomatsakis]

If typeck moves to MIR, then a lot of HIR-lints are gonna break.

Some amount of type-checking will always be done on HIR, since you need the types to build the MIR, but (I think) we won’t have region data available except for in MIR land.

[nikomatsakis]

I’m a little uncomfortable with lints in general - they make use of unstable compiler internals and I don’t see a way forward to making them work stably. I think they are an important tool, and I would love to have more of a sense of a good solution for them. Adding another kind of lint without having a proper vision for lints makes me very nervous.

I agree with this entirely. It’d be good to have some sense of what data lints commonly make use of and try to categorize them. I would hope that the majority would be satisfied with whatever interface we end up providing for IDEs and other similar introspection.

[nikomatsakis]

I’ve been thinking more about this. I’m feeling rather torn. I think it comes down to whether you consider the story with lints to be a success or a failure – and I think the ending to that story has not yet been written. It’s definitely true that clippy is a thing, which means that to some extent our hands are tied by backwards compatibility constraints.

But then the existence of clippy has not prevented us from doing anything yet, and I think it’s plausible that we’ll be able to stabilize a lint interface at some point that clippy can use. And clippy gives us lots of data about what kinds of features lints might want.

Similarly, I think that it’s way too early for MIR plugins. I mean MIR is still rapidly changing. But maybe the existence of them might help us to figure things out.

I am more upset about the process here. It does feel like changes to these sorts of protocols go through without much discussion. MIR plugins are one example, but https://github.com/rust-lang/rust/pull/31562 is another (that’s a change to the lints protocol). I don’t want to add a ton of process, but it feels like there ought to be some sort of middle-ground.

I’d certainly have expected a proposal for MIR plugins to go through the RFC process (also a plugin manager). But then again, I’d have been opposed to any such RFC on the grounds that it’s just too early to think about that stuff yet. So it really comes back to whether it’s good to experiment in tree or whether the risk of lockin is too great.

[oli-obk]

Similarly, I think that it’s way too early for MIR plugins. I mean MIR is still rapidly changing. But maybe the existence of them might help us to figure things out.

While MIR is changing, and anyone using it should be aware of that, the fact that there’s a plugin-way to access them is imo not different from the “using rustc as a library” usage, except that it’s so much more convenient.

The same reasoning applies to lints. We could just as well use rustc as a library and create a static analysis tool that runs the lints. I’ve experimented with that and the interface is not very convenient (even if it’s more powerful than the plugin interface) and I run into problems with loading/finding the standard libs every time. Of course we could make the interface better or create crates that help with finding the libs, but we already have all that with the plugin interface.

I am more upset about the process here. It does feel like changes to these sorts of protocols go through without much discussion.

I think that’s mostly because it’s hard to notice that there’s a PR that changes something of interest. I didn’t know about that PR and I’m interested in compiler-plugins. Maybe there could be some open github groups that anyone can join, and then the related groups can be mentioned in PRs that change something in their interest.

clippy gives us lots of data about what kinds of features lints might want.

All the more reason imo to accept minor changes without much discussion and have the discussion when someone dislikes something or wants to break something.

[gnzlbg]

While I personally don’t think anyone in their right mind would assume that code compiled with a nightly compiler will still compile the next day, I see your point.

I don’t see it. Somebody needed a MIR plugin, now we have them and they are unstable. Worrying about them becoming “de-facto stabilized” just because people use it defeats the point of unstable and goes against “progress without stagnation”.

Those using unstable features are martyrs willing to have their crates broken for the sake of a better Rust. The compiler devs should have peace of mind in breaking unstable code whenever they see fit. Those using MIR plugins are this kind of martyr, thanks to them we will have a solid plugin story in the future.

If worrying about breaking unstable code is truly a thing it seems to me that the real issue is that unstable is not unstable enough and what we really need is really_unstable_dont_use_this_or_puppys_will_die.

[nikomatsakis]

We recently discussed this again in a recent compiler team meeting. Our conclusion was to leave MIR plugins for the time being, but they are to be considered “insta-deprecated”. That is, we plan to remove them once MIR stabilizes more, but we figure that they could be useful for running experiments on how to improve MIR for the time being.

(It may seem surprising that we would remove plugins once MIR is getting more stable, but the reason is that it is precisely then when people can start to ossify real dependencies on MIR plugins. Indeed, precisely this same story is playing out with aster and other places that we prematurely offered plugin support.)

[system]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.