Aster crate

[nrc]

Aster is a crate developed by @erickt which provides tools for building ASTs (similar to, but better than, libsyntax/ext/build.rs). It is used by serde, and thus by Servo among others. It requires to be kept in sync with libsyntax.

There is a PR to add this as an unstable library to the Rust distro. The motivation is to keep it in sync with libsyntax. Given that this is a fairly large (8 kloc) chunk of code and we have tried to reduce the number of libs in the main Rust repo, it would be nice if we could find a better solution. If there is some way for crates like Aster to keep ahead of libsyntax breakage I believe other crates and tools would benefit (Rustfmt certainly would).

So, ideas? Opinions on landing 28006?

[eefriedman]

A perfectly good solution for managing dependencies already exists: it’s called cargo. The only reason why this doesn’t work for Aster is that it uses rustc_private, as opposed to just using the syntex crate from crates.io.

[Gankro]

aster breaking is pretty much the only source of nightly breakage I’ve ever experienced, and it’s incredibly brutal. I literally just hand-editted a Cargo.lock to use the correct aster for my nightly.

I think it’d be great to have some “unstable like std” (where unstable features are more likely to sit still than change) interface for compiler plugins, rather than the current “unstable like rustc” model, which has everyone and their grandmother doing tons of massive refactoring. I’d imagine it would be a minimal burden for those submitting patches to the compiler to fix aster as well. erickt has previously indicated to me that most aster fixes are only a few lines – the hard part is figuring out what the heck changed (which the person submitting a rustc patch knows perfectly).

[Manishearth]

A perfectly good solution for managing dependencies already exists: it’s called cargo. The only reason why this doesn’t work for Aster is that it uses rustc_private, as opposed to just using the syntex crate from crates.io.

Syntex is not equivalent to using libsyntax, and the user experience isn’t as great either (syntex pre-expands code, so you don’t get nice backtraces)

IME libsyntax breaks code far more often than aster’s public interface; switching is a good step in the direction of partial syntax extension stability. If it doesn’t work out, we can always remove it.

[nrc]

I have said before that it would be nice to have libs which are not part of the repo but are part of the testing infra, so PRs cannot land if they break them. It sounds like you would like Aster to be just such a lib. When we’ve discussed this before, the technicalities have stopped us (iirc) - should the libs be submodules? Should we use Cargo for this, etc. If we could solve this, then I think it would be the best solution. I assume it would help the libs team with ‘important, but not std’ libs too?

[alexcrichton]

I would personally prefer we not take on the task of maintaining aster in-tree. I agree that there’s certainly a need here for mitigating breakage on nightly, but I don’t think the best solution is to just push everything in-tree so we’re the ones always mitigating breakage. This unfortunately doesn’t scale to other nightly crates which are also used widely in parts of the ecosystem.

I have said before that it would be nice to have libs which are not part of the repo but are part of the testing infra, so PRs cannot land if they break them.

I’m personally pretty hesitant about this. I’m not so worried about the size of our own repo (e.g. a submodule would be fine), moreso about the maintenance burden (e.g. atomically moving the world forward is much harder).

I think it’d be totally ok to have this as an advisory sort of dashboard, e.g. you could see precisely which PR you started breaking on, but actually gating on many other nightly repos I would not be a huge fan of.

[eefriedman]

Syntex is not equivalent to using libsyntax, and the user experience isn’t as great either (syntex pre-expands code, so you don’t get nice backtraces)

The lack of backtraces aren’t a fundamental limitation of text->text transformation, just a missing feature… for example, C preprocessors annotate expanded includes using “#” notation.

More generally, I’m not sure we want to encourage people to use compiler plugins to implement syntax extensions. Text->text seems easier to support in the long term: it exposes much less API surface than using libsyntax’s AST, even if we wrap semi-stable APIs around it.

Feel free to ignore me if I’m dragging this conversation too far off track.

[nikomatsakis]

I’m personally pretty hesitant about this. I’m not so worried about the size of our own repo (e.g. a submodule would be fine), moreso about the maintenance burden (e.g. atomically moving the world forward is much harder).

I agree with this sentiment and am also concerned. I wonder if there is another way we can mitigate this kind of breakage besides taking responsibility for fixing it. For example, perhaps there is a call for some kind of weekly “alpha” release channel that stands between Nightly and Beta. Then people can track that instead of tracking Nightly, perhaps? (And it gives us a week to fix breakage…?)

Basically, I am torn between two feelings. On the one hand, compiler extensions are unstable, and this is precisely why: they are poking at internals that we are actively refactoring, and I think nobody wants us to stop this process. This sort of refactoring is hard enough to begin with, adding the additional burden that we must maintain legacy code and packages is a worrisome precedent. On the other hand, I know that many people (including servo) rely on compiler plugins, and I want to support those projects. Put another way, I am happy if we can find a way to keep the high-level functionality afloat even as the impl details shift, but moving such packages in tree (or having them block the build) doesn’t feel like the optimal (or only) way to do that.

[steveklabnik]

On the one hand, compiler extensions are unstable, and this is precisely why:… On the other hand, I know that many people rely on compiler plugins, and I want to support those projects.

Well stated.

[nrc]

If fixing these bugs when they happen is easy if you know what is breaking them, then perhaps we can solve this with tooling:

• there is a set of crates which are blessed in this way
• bors builds them all as part of the pre-landing process for each PR
• if a build fails, then bors files an issue referencing the issue number of the PR which broke the crate’s compilation and the error messages

This seems like not much work and should make it easier to keep crates up to date.

[nikomatsakis]

I was going to suggest something similar. @brson and I were kicking around a similar idea linked to crater, not bors – basically a status board listing crates and their current status with stable/nightly builds (plus some notion of priority so that crucial infrastructure crates are kept up to date and hence have better resolution). This wouldn’t give you the precise commit though.

[erickt]

The lack of backtraces aren’t a fundamental limitation of text->text transformation, just a missing feature… for example, C preprocessors annotate expanded includes using “#” notation.

More generally, I’m not sure we want to encourage people to use compiler plugins to implement syntax extensions. Text->text seems easier to support in the long term: it exposes much less API surface than using libsyntax’s AST, even if we wrap semi-stable APIs around it.

Feel free to ignore me if I’m dragging this conversation too far off track.

You’re not off base at all here, in order for syntex to really have a good user experience, we’d have to grow a CPP-ish #line macro so that any errors reference the source files, not the expanded file. I think good error messages are one of the big roadblocks for text to text translation, and would be a big reason why Servo wouldn’t be interested in adopting it.

I’ve been meaning on doing this for a long time, but I haven’t had time to do it yet

[erickt]

I agree with this sentiment and am also concerned. I wonder if there is another way we can mitigate this kind of breakage besides taking responsibility for fixing it. For example, perhaps there is a call for some kind of weekly “alpha” release channel that stands between Nightly and Beta. Then people can track that instead of tracking Nightly, perhaps? (And it gives us a week to fix breakage…?)

An alpha is an interesting idea. I’ve heard from @SimonSapin that Servo has to do some complicated pinning of the nightly version in all their crates to avoid breakages in packages like aster. Perhaps an alpha release channel would be a little easier for them to track without having to explicitly keep repinning nightlies all the time.

[Gankro]

I don’t really see the point in an alpha – does anyone even use beta yet?

Also it’s not sufficient to just “fix” aster out of tree. The problem arises from the fact that random nightlies work with random versions of aster, and there’s no way to tell other than seeing if you build. If you absent-mindedly upgrade your nightly or update your lockfile, your are likely to experience a break.

Barring some way for a crate to specify nightly version reqs, the only way I see to resolve this is to have aster be distributed with nightlies.

[Gankro]

Of course, if the HIR lands and stabilizes libsyntax (if not Truly Stable, at least Basically Stable like std), that’s also grand!

[SimonSapin]

I’ve heard from @SimonSapin that Servo has to do some complicated pinning of the nightly version in all their crates to avoid breakages in packages like aster.

Apparently some people have done this in some crates, but I think it’s completely the wrong solution. When Aster is not up-to-date with nightly the solution is not to use an older nightly, it’s to shave that yak and update Aster.

I don’t see how alpha releases would be useful at all. When something breaks in a crate like Aster with lots of dependents we want to be notified of it (to be able to update) as soon as possible, not a week later.

[nikomatsakis]

The problem arises from the fact that random nightlies work with random versions of aster, and there’s no way to tell other than seeing if you build.

So, the idea of alpha (perhaps better called “Weekly”) was to say that you don’t “randomly upgrade your nightly” but rather you stick to the “Weekly” channel. For it to work, it would require that Aster (and your other dependencies) fix any problems before the next “Weekly” release.

That said, I would prefer a good statusboard with more detailed information – one can even imagine being able to narrow the selection by pointing it at a specific crates.io package and only seeing information about the dependencies of that package.

[erickt]

Another angle to consider would be if Aster (or something similar) would help the compiler itself. I wrote Aster with the intention of it someday replacing syntax::ext::build, which I never found that pleasant to use. Are there any other areas of the compiler that could be made simpler with syntax::ast builders, or is all the heavy lifting taken care of by syntax::visit and syntax::fold? Would a similar builder for rustc::middle::ty help with anything? This style would also be helpful for MIR optimizations too.

[Gankro]

So, the idea of alpha (perhaps better called “Weekly”) was to say that you don’t “randomly upgrade your nightly” but rather you stick to the “Weekly” channel. For it to work, it would require that Aster (and your other dependencies) fix any problems before the next “Weekly” release.

This unfortunately doesn’t fix the issue when you want to update deps or otherwise mess with your lockfile. I’m currently pegged to a random nightly from a couple weeks ago for fear of aster breaking, but I still got hit when I did some project restructuring that caused deps to be re-evaluated and a new version of aster to be downloaded (which of course only works with very recent nightlies).

[nikomatsakis]

It just makes it easier to fix Aster.