I’m now imagining a future where instead of complaining about how long syn takes to compile, you hear complaining about how long polyfill-new-language-feature-blahblahblah2 (or another equally big crate for a different feature) takes to compile in every project that uses it.
More seriously, what about language features whose usage crosses crate boundaries? Some of the most anticipated language features (for me, at least) are those which, if you use them, will be visible in your library crate’s API, and will create new possibilities for how APIs are designed. These include:
const trait impls
TAIT/ATPIT/RTN — things that let you give names to anonymous types
additional const generics features
dyn async traits
I don’t see how you can write a polyfill for these, unless the compiler becomes so hookable that you are patching almost the entire pipeline, from the trait solver to the .rmeta writer. Which seems unlikely to be achievable.
Because proc-macros don't have nearly as much information available as MIR passes do.
You may reply by saying that we should change that. That would require a completely new kind of proc-macro and lots of API bikeshedding and implementation work, literal years of effort. It also would have to go so deep into the compiler that it would incur a permanent tax on all rustc evolution by having to deal with this immutable API in the heart of rustc. That all piles up as a huge amount of cost you're asking us to carry (leaving aside that it's entirely unclear who'd be motivated to design & implement all that). Unless you can convincingly argue that this change will 10x the userbase of Rust, I can't imagine this being worth it.
The cost is gigantic, and the benefit is that some people who don't think rationally may adopt Rust? I am not convinced, not at all. I think rational arguments work on a large enough subset of people and situations to be the preferable strategy. We should not bend over backwards to cater for poor decision processes, not if the costs are so high.
You wrote (emphasis mine):
now we're getting up into the scale of changes where the benefits to code authors actually pay for the costs imposed on everyone else.
Because it makes life easier for all your users at only a modest cost to you?
That does not leave room for any possibility that only a subset of users considers themselves to be negatively affected by the status quo. Your "everyone" and "all" excludes people like Josh and me for whom life would not become the slightest bit easier if crates suddenly decided to support a lot more Rust versions.
The size of the two respective sets (of people that are or are not negatively affected by the status quo) may be up for debate, but the existence of a non-empty set of people that are fine with this part of the status quo is hopefully something we can agree on, so please reflect that in your choice of words. That is all Josh said here.
Yeah that works for some things. It doesn't resolve the dogfeeding aspect and there are many features it doesn't work for (or only works at a non-trivial performance cost).
If you call this experimenting, so this would just be a temporary measure to let people from C get familiar with Rust and once that's done the training wheels can come off?
Then go complain to compliance departments that scream at the top of their lungs about CVEs about some unreachable regex potentially taking too much time.
Without those you can just freeze the compiler and crates in time. Neither rustc nor crates apply pressure to you to update.
I have gotten and offered "no, you were right" in conversations, after some back-and-forth or after trying and realizing I was wrong. It happens.
I suppose "taking time to think/try" already implies some absence-of-pressure. But isn't that normal in engineering fields, that you get at least some time to learn or refresh your knowledge?
That only works for std API surface, not language. And I would resist using macros for language polyfills because they tend to interact poorly with rust-analyzer.
And it's not just about writing the lines of code. Testing it, ensuring it keeps working on all the compiler version, developing new features under those conditions, finding new polyfills etc. all still takes effort too.
So I would have to see some benefit in doing that work. And that benefit is not obvious to me.
If the sole motivation is to provide training wheels to those who want to pick up rust then... yes, we want to be beginner-friendly, but that's still a high cost to the whole ecosystem. And a cost that is difficult to emphasize with because for many who already use Rust don't feel it.
Perhaps this can be achieved more cheaply, e.g. by advertising that some existing projects manage "fine" sticking with an older rust version for a while? (debian, firefox, linux kernel?).
Maybe a retro-rust crates mirror?
More incentives to figure out cross-project caching of intermediate build products, maybe? But I do take your point.
Those are also some of the language features I want most myself. And -- acknowledging that I might be saying this because I want them -- they are also, in my book, the features for which MSRV bumps are most justifiable.
But they don't come along that often. If we could get the general crate ecosystem to "most crates build with any compiler that supports the most recent edition, except when some big new capability for defining APIs comes out mid-edition-cycle" that would already be a lot better than the status quo, in terms of decoupling upgrades from each other.
I kinda do want that, but doing it to Rust doesn't really seem feasible, yeah (it's the sort of thing that would have needed to be designed in from day one).
I will strive to be more precise in future, but IMNSHO what Josh said is most naturally interpreted as "the size of the set of people who are not fine with this is negligible" and that is what I was objecting to.
No, it's meant to be useful to anyone with a code base that currently works just fine with dependency versions A.123, B.234, and C.345 and now they would like to try out the new features of A.latest but there's a risk that updating it breaks either B or C. (See e.g. the time example up above.)
That's just not true. In this very thread we have both people saying that they feel pressure to update and it causes them extra work on an ongoing basis (e.g. here) and people saying that there should be pressure to update (e.g. this comment).
I refer you to what I said about the six-week compiler release cadence being a dealbreaker in and of itself for some people, and the subsequent notes about the hobbyist who was largely satisfied with version X of the language, in this comment. Do not underestimate the psychological cost of not getting to work on the thing you actually wanted to work on because instead you have to do chores.
I don't use rust-analyzer at all myself (it costs me too much laptop on-battery runtime) but I understand where you're coming from here. Probably when I come back with that pre-RFC I was talking about before, it will be restricted to std API surface at least to begin with, as a "let's try out the concept for the easy case first" thing.
I wasn't saying "negligible". I was trying to emphasize that your desires are not universal, and that many changes making things better for some of your desires make things worse for other people's desires. In particular, I would love to see an answer to Ralf's argument about the second-order effects that doesn't just boil down to "I'm okay slowing everyone down".
If you want quantitative numbers, take a look at some of the survey numbers for how many people think Rust is moving at the right speed, vs too fast, vs too slow.
That would be a rustc plugin (for which we removed support several years ago) not a proc macro. MIR is an unstable format. Also async/await required changes to the type system to handle defining the generator type behind it (which in turn required adding support for it to the layout calculation code that previously didn't need to directly get MIR for functions), not just adding a MIR pass. And I believe there was also subtleties around avoiding query cycle errors to avoid MIR generation and layout computation for the generator depending on each other. So before the current query system introduced for incr comp was a thing, there wouldn't really have been a way to implement this even if you managed to change the type system and layout computation. (the first alpha version of incr comp appeared in 2016, which is only 2 years before async/await, and I don't recall if that version was expressive enough to avoid this query cycle and the first alpha of MIR that you need to hook appeared another half a year before that) You have to realize that a lot of newer features, especially around the type system, have only appeared now because of years of refactoring the internals of rustc to make them implementable. Even by overriding entire queries in a custom driver you wouldn't be able to implement them in much older rustc versions without effectively replacing the entire compiler through hooks. The end result of that would be significantly less stable and significantly more likely to cause crashes and miscompilations that just updating your ancient rustc to the latest version.
If I recall correctly the time regression was ambiguity due to a new trait impl. We can't make trait impls conditional on an edition or feature gate. Trait impls are a global thing. The trait solver must give a consistent answer to every query independent of the crate it is invoked in for soundness (otherwise you can transmute types by defining a function in one crate and getting it codegend in another).
I'm willing to believe that the standard library might need to be treated differently from the compiler in this respect. In particular, I don't think rustc needs to depend on much of the stage 0 standard library at all: given that rustc comes with its own copy of the standard library, it could depend on that copy instead (in which case rustc would be able to use all the newest library features, as long as an older compiler was capable of compiling the new standard library). Likewise, the new standard library could depend on itself rather than on the stage 0 standard library for everything other than the most basic functionality.
I can imagine a model in which instead of the current core/alloc/std split in the standard library, there were a four-way split, with core split into two pieces: one piece that contains the extreme basics (integers, raw pointers, slice/dyn, intrinsics, and type system / codegen special cases), and one piece that contains everything that can be implemented in terms of that (everything else, which is almost everything that core currently contains, including things like references and str which could conceptually be implemented in terms of simpler primitives). In this situation, programs would be able to provide basically all the standard library on their own if they wanted to, and rustc is the sort of program which would naturally want to do that as it needs to provide a standard library anyway.
This would effectively mean that the version of the standard library would become mostly untied from that of the compiler, so polyfilling would largely become unnecessary (or to think about it a different way, the entire standard library would be implemented as a polyfill over a minimal subset).
Unfortunately, there is a signfiicant problem with this approach: some library features can't be written without extra compiler features. The stable version of pin! comes to mind: if you are trying to write it purely in a library without compiler help you would make a macro that declares a new variable in order to get the lifetime extension right, but the stable version expands to an expression with weird lifetime extension rules and thus is impossible to polyfill (or implement in a newer version of the standard library) without compiler help. This sort of thing has caused problems in practice in the past, e.g. when part of the standard library turned out to be impossible to implement in 2024 edition, and in general the sort of code that can't be polyfilled is difficult to understand and reason about, so perhaps in retrospect stronger incentives to write standard library code without compiler help would have been a good thing.
What you are describing is the old staging approach that we used until some time last year or so. Under that approach, the in-tree standard library has to build with both bootstrap rustc and in-tree rustc. That turned out to be so painful that we switched things around; now the in-tree standard library only builds with the in-tree rustc, and in exchange in-tree rustc has to build with bootstrap std and in-tree std.
I can't imagine we want to go back to the old model, we switched for a reason.
It's my experience that most of these breakages are those insta-stable things that can't be feature-gated. The one that I linked was a type inference issue (which new rust versions are allowed to cause) from a new trait implementation (which is insta-stable). We also proactively updated because we were about to be hit by the breaking changes for never type fallback in 1.100 (also not gated behind a feature flag). New features that get feature flags are usually different enough from the existing functionality that nothing breaks when they get stabilized.
Or the Rust Foundation may offer LTS as a commercial-only product, so that it gets the revenue while not slowing down the ecosystem, which will not be paying for LTS.
IMO the Rust project should not be making commercial-only software, it's an open-source project for a reason and I contribute because it's open-source and I want my software to be free as in freedom. So I think if the only choices are commercial only or not at all, I pick not at all -- that said I think we can have any LTS version also be open-source and publicly available without a paywall.
My motivation is this: if LTS would be open, it would attract users.
Alternative would be to add a lot of hurdles to using LTS, so that regular users without much motivation won't - such as requiring to compile it themselves, e.g. not distribute at all.
Because it ultimately depends on who the LTS attracts.
The worst case is that the LTS attracts people who were already using stable Rust, and keeping up to date happily, but who are now going to put pressure on everyone in the Rust ecosystem to stick with the LTS. This is directly in opposition to Rust's "stability without stagnation" goal - you've taken people who were happy keeping up, and turned them into people pushing for stagnation.
The middling case, which might still be a problem case, happens if an LTS attracts people who would never have used Rust without the LTS, but those people then put pressure on everyone in the Rust ecosystem to stick with the LTS. This is a balancing act - does the benefit of more people using Rust outweigh the cost of those people pushing back on the very core of "stability without stagnation"?
And the best cases happen if the LTS attracts people who would never have used Rust without it, but who try it, like it, and then decide to either quietly stick to the parts of the ecosystem that agree with them on using the LTS, or decide that actually, now they like Rust, they can try moving on from the LTS to the six weekly stable updates.
But there is a very real risk that we hit one of the problem cases, where the effect of the LTS is to pressure everyone in the Rust ecosystem - not just toolchain developers, but also people working on libraries - into stagnation because it's "obvious" that the LTS is what matters.
This is not a pure hypothetical concern, either - I already have problems when I'm working in C dealing with people who are opposed to me bringing in a library whose header files require C11 to compile, because "we use ANSI C89 here, and you can't use C11 features - C11 is too new to be trustworthy". If an LTS attracts those sorts of users, then Rust is at risk of stagnation.