Aster is a crate developed by @erickt which provides tools for building ASTs (similar to, but better than, libsyntax/ext/build.rs). It is used by serde, and thus by Servo among others. It requires to be kept in sync with libsyntax.
There is a PR to add this as an unstable library to the Rust distro. The motivation is to keep it in sync with libsyntax. Given that this is a fairly large (8 kloc) chunk of code and we have tried to reduce the number of libs in the main Rust repo, it would be nice if we could find a better solution. If there is some way for crates like Aster to keep ahead of libsyntax breakage I believe other crates and tools would benefit (Rustfmt certainly would).
A perfectly good solution for managing dependencies already exists: itās called cargo. The only reason why this doesnāt work for Aster is that it uses rustc_private, as opposed to just using the syntex crate from crates.io.
aster breaking is pretty much the only source of nightly breakage Iāve ever experienced, and itās incredibly brutal. I literally just hand-editted a Cargo.lock to use the correct aster for my nightly.
I think itād be great to have some āunstable like stdā (where unstable features are more likely to sit still than change) interface for compiler plugins, rather than the current āunstable like rustcā model, which has everyone and their grandmother doing tons of massive refactoring. Iād imagine it would be a minimal burden for those submitting patches to the compiler to fix aster as well. erickt has previously indicated to me that most aster fixes are only a few lines ā the hard part is figuring out what the heck changed (which the person submitting a rustc patch knows perfectly).
Syntex is not equivalent to using libsyntax, and the user experience isn't as great either (syntex pre-expands code, so you don't get nice backtraces)
IME libsyntax breaks code far more often than aster's public interface; switching is a good step in the direction of partial syntax extension stability. If it doesn't work out, we can always remove it.
I have said before that it would be nice to have libs which are not part of the repo but are part of the testing infra, so PRs cannot land if they break them. It sounds like you would like Aster to be just such a lib. When weāve discussed this before, the technicalities have stopped us (iirc) - should the libs be submodules? Should we use Cargo for this, etc. If we could solve this, then I think it would be the best solution. I assume it would help the libs team with āimportant, but not stdā libs too?
I would personally prefer we not take on the task of maintaining aster in-tree. I agree that there's certainly a need here for mitigating breakage on nightly, but I don't think the best solution is to just push everything in-tree so we're the ones always mitigating breakage. This unfortunately doesn't scale to other nightly crates which are also used widely in parts of the ecosystem.
I'm personally pretty hesitant about this. I'm not so worried about the size of our own repo (e.g. a submodule would be fine), moreso about the maintenance burden (e.g. atomically moving the world forward is much harder).
I think it'd be totally ok to have this as an advisory sort of dashboard, e.g. you could see precisely which PR you started breaking on, but actually gating on many other nightly repos I would not be a huge fan of.
The lack of backtraces aren't a fundamental limitation of text->text transformation, just a missing feature... for example, C preprocessors annotate expanded includes using "#" notation.
More generally, I'm not sure we want to encourage people to use compiler plugins to implement syntax extensions. Text->text seems easier to support in the long term: it exposes much less API surface than using libsyntax's AST, even if we wrap semi-stable APIs around it.
Feel free to ignore me if I'm dragging this conversation too far off track.
I agree with this sentiment and am also concerned. I wonder if there is another way we can mitigate this kind of breakage besides taking responsibility for fixing it. For example, perhaps there is a call for some kind of weekly "alpha" release channel that stands between Nightly and Beta. Then people can track that instead of tracking Nightly, perhaps? (And it gives us a week to fix breakage...?)
Basically, I am torn between two feelings. On the one hand, compiler extensions are unstable, and this is precisely why: they are poking at internals that we are actively refactoring, and I think nobody wants us to stop this process. This sort of refactoring is hard enough to begin with, adding the additional burden that we must maintain legacy code and packages is a worrisome precedent. On the other hand, I know that many people (including servo) rely on compiler plugins, and I want to support those projects. Put another way, I am happy if we can find a way to keep the high-level functionality afloat even as the impl details shift, but moving such packages in tree (or having them block the build) doesn't feel like the optimal (or only) way to do that.
I was going to suggest something similar. @brson and I were kicking
around a similar idea linked to crater, not bors ā basically a status
board listing crates and their current status with stable/nightly builds
(plus some notion of priority so that crucial infrastructure crates are
kept up to date and hence have better resolution). This wouldnāt give
you the precise commit though.
You're not off base at all here, in order for syntex to really have a good user experience, we'd have to grow a CPP-ish #line macro so that any errors reference the source files, not the expanded file. I think good error messages are one of the big roadblocks for text to text translation, and would be a big reason why Servo wouldn't be interested in adopting it.
I've been meaning on doing this for a long time, but I haven't had time to do it yet
An alpha is an interesting idea. I've heard from @SimonSapin that Servo has to do some complicated pinning of the nightly version in all their crates to avoid breakages in packages like aster. Perhaps an alpha release channel would be a little easier for them to track without having to explicitly keep repinning nightlies all the time.
I donāt really see the point in an alpha ā does anyone even use beta yet?
Also itās not sufficient to just āfixā aster out of tree. The problem arises from the fact that random nightlies work with random versions of aster, and thereās no way to tell other than seeing if you build. If you absent-mindedly upgrade your nightly or update your lockfile, your are likely to experience a break.
Barring some way for a crate to specify nightly version reqs, the only way I see to resolve this is to have aster be distributed with nightlies.
Apparently some people have done this in some crates, but I think itās completely the wrong solution. When Aster is not up-to-date with nightly the solution is not to use an older nightly, itās to shave that yak and update Aster.
I donāt see how alpha releases would be useful at all. When something breaks in a crate like Aster with lots of dependents we want to be notified of it (to be able to update) as soon as possible, not a week later.
So, the idea of alpha (perhaps better called "Weekly") was to say that you don't "randomly upgrade your nightly" but rather you stick to the "Weekly" channel. For it to work, it would require that Aster (and your other dependencies) fix any problems before the next "Weekly" release.
That said, I would prefer a good statusboard with more detailed information -- one can even imagine being able to narrow the selection by pointing it at a specific crates.io package and only seeing information about the dependencies of that package.
Another angle to consider would be if Aster (or something similar) would help the compiler itself. I wrote Aster with the intention of it someday replacing syntax::ext::build, which I never found that pleasant to use. Are there any other areas of the compiler that could be made simpler with syntax::ast builders, or is all the heavy lifting taken care of by syntax::visit and syntax::fold? Would a similar builder for rustc::middle::ty help with anything? This style would also be helpful for MIR optimizations too.
This unfortunately doesn't fix the issue when you want to update deps or otherwise mess with your lockfile. I'm currently pegged to a random nightly from a couple weeks ago for fear of aster breaking, but I still got hit when I did some project restructuring that caused deps to be re-evaluated and a new version of aster to be downloaded (which of course only works with very recent nightlies).
