A/B testing language features

[Soni]

I don’t wanna spend too long writing this but I’d really like if we could A/B test language features.

For example, PITs vs &uninit would be quite interesting to A/B test (but also rather complex).

Another interesting thing to A/B(/C) test would be Into<Result<T, E>> vs TryInto<T, E> vs TryInto<T> with associated E.

We currently cannot do these, and I feel like this gives an unfair advantage to whatever gets implemented first. We should work to remove that bias, so that ppl can choose what they like, rather than being forced to follow the status quo.

[Centril]

For example, PITs vs &uninit would be quite interesting to A/B test (but also rather complex).

Before testing anything you need to implement it; before implementing it, we (the language team + other interested parties) need to see a proposal with motivation, usage guide, technical specification, and so on.

I would encourage you to spend more time focusing on one idea at a time and try to develop it into a fully formed proposal instead of jumping between threads and leaving them abandoned. I think that’s a more productive use of yours and everyone’s time.

[skysch]

Implementing a language feature is usually a lot of work. Implementing two of them is even more work. That’s why there’s a process in place to decide what gets implemented before any implementing begins. If every proposed feature could be implemented, then there’d be no need for an involved RFC process, because everything would get implemented and you could simply decide which one you wanted to use based on how well they work in practice.

I feel like this gives an unfair advantage to whatever gets implemented first

That ‘advantage’ is based on the results of prior discussion. It’s not unfair; it’s been earned through the efforts that’ve been put into motivating the implementation.

[Centril]

That’s why there’s a process in place to decide what gets implemented before any implementing begins.

We do sometimes do experimental RFCs where there’s a known problem and a clear motivation to solve it but where consensus on a specific solution may be lacking or where the details need to be worked out. An example of this is RFC 2497. Even so, these experimental RFCs need to be specified in some detail (as you can see, the aforementioned eRFC is quite long).

[skysch]

Yes, technically that is true and I was over-simplifying. But that also serves to highlight how much work is needed to get a feature into a state where it could even be meaningfully compared to another implementation. (And that’s assuming you don’t have a nigh endless experiment in one of the branches.)

[Centril]

But that also serves to highlight how much work is needed to get a feature into a state where it could even be meaningfully compared to another implementation.

Yes I think this is the crux of the matter; regardless of the experimentality or lack thereof of an RFC the specification work needs to happen so that the compiler team at least has a reasonable chance of implementing it, and so that the language team can assess if the motivation is something we agree is a problem in need of solving, and so that the proposed solution is at least in the general vicinity of being good for the language. In other words: if you want something implemented, you need to show that it is important enough to you that you are willing to put in some effort.

(PS: I should note that I’m a bit of an oddball in that I like writing very long RFCs with lots of details so this is one of the longer RFCs ever written…)

[Soni]

I don’t believe discussion is as good as actually using the thing.

[skysch]

Of course it is not, it is only easier. And if things aren’t truly working well, you don’t waste your life implementing features that don’t work.

[Soni]

If we had an A/B testing framework, and the compiler allowed crates to select between them, then…

I’d be willing to implement things like ! as into, and Into<Result<T, E>, and so on. and comparing those to TryInto/etc

[Nemo157]

Would this not be entirely possible with features? There’s currently the two new use path variants which are both behind different features and effectively available for A/B testing.

The main issue I see with doing A/B testing in the compiler itself is just cluttering the codebase with half implemented, probably never to be stabilised features.

[skysch]

That option is already available, though it is a part of git, not the compiler.

[wesleywiser]

Git branches are definitely an elegant solution here. The problem is not that we have no way of A/B testing, it’s that for all but the most basic features, developing, testing, documenting, reviewing, and stabilizing a feature are expensive. An A/B testing framework doesn’t address any of those issues.

[Soni]

maybe this would be relevant then: Additional useful compiler macros

a crate could come along and provide a wrapper over it that lets you specify “in every expression context, exp ! becomes into” in simple terms

[system]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.