What do you mean by this comment?
fn main() {
let s = "\u{110000}";
}
error: invalid unicode character escape
--> src/main.rs:2:14
|
2 | let s = "\u{110000}";
| ^^^^^^^^^^ invalid escape
|
= help: unicode escape must be at most 10FFFF
Not everyone (including myself) considers that to be good advice: Robustness principle - Wikipedia
Anyway, personally I wouldn't mind Rust increasing the limit on Unicode escape seqeuences and instead just issuing a warning if the user writes a sequence that exceeds Unicode's current limits. That seems the best of both worlds: no hard limit, but still likely to catch mistakes.
I mean that the Rust lexer, which is the party responsible for the 6 character limit, correctly identifies that as a Unicode scalar escape, and passes it onto the next stage, which then tells you that the number is too large to be a Unicode scalar value.
If you enter a 7 digit value, then instead of being told that you've given it too large a number, you get told that the escape is over-long - in other words, the lexer has said that it cannot tokenise this string correctly:
error: overlong unicode escape
--> src/main.rs:2:14
|
2 | let s = "\u{1234567}";
| ^^^^^^^^^^^ must have at most 6 hex digits
If we allowed you to enter more hex digits, you'd still be limited by the fact that a Unicode scalar value is currently defined as a value in the range 0 to 0x10ffff. You might be allowed to write \u{1234567} in a string and have it recognised as a Unicode scalar escape, but you'd then get the error that 0x1234567 is too large for a Unicode scalar.
Changing this requires changing more than just the Unicode scalar escape lexing rules - you also have to change the definition of a char, and of str, to name two places that currently only support Unicode, not an extended superset of Unicode.
I agree, no advice should ever be universally accepted. But listening is always a good idea, LoL.
This would also require expanding the definition of safe strs to include not-valid-in-today's UTF-8 byte sequences. I don't think that's acceptable, as it'd immediately result in everyone trying to accept only Unicode-valid UTF-8 needing to run extra checks.
Not to mention that it's a breaking change to make char bigger anyway, since you can exhaustively match on char: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=1d747466baf240a19236e4cdf8ba2168
pub fn foo(c: char) {
let ('\0'..='\u{10FFFF}') = c;
}
So this isn't happening in Rust in the foreseeable future.
If Unicode gets to the point where they're talking about potentially needing something bigger, then we'll add new types for whatever that is.
I don't really see why this would be a meaningful improvement, even if Unicode were to be expanded.
As I see it, what this will allow is compiling NEW rust code (post Unicode expansion, when new code points have an assigned meaning), to be compiled by an OLD rust compiler. Why not use the new rust compiler for that?
Furthermore, wouldn't std need to know something about the new values, for capitalisation and such to work properly? I'm not knowledgeable in this area, but if anything, the std Unicode handling would be the main issue with forward compatibility: old code running on new data.
It won't even allow that, because after the lexing stage (which is where the 6 hex digit limit is imposed), a later stage will validate that the given scalar value is in the accepted range for the version of Unicode that the compiler knows about.
Increasing the number of allowed hex digits in a Unicode scalar escape simply allows you to add leading zeroes. With 6 hex digits, \u{000041} is a valid way to write 'A'. With 8 hex digits, I could write it as \u{00000041} instead (rejected today, since that's too many hex digits) - but the compiler would still reject \u{12345678} because the maximum allowed value for a Unicode scalar value is \u{10ffff}.
Given that Unicode scalar values are in the range 0 to 0x10ffff, 6 hex digits seems like a reasonable position for today - technically, we could make it an optional 0 or 1 followed by 5 hex digits, without causing issues, but that makes the lexer's job harder.
Expanding the size of char etc so that we support not just Unicode scalar escapes with more than 6 hex digits, but also Unicode scalar values above 0x10ffff is a much bigger job - for starters, Rust currently knows that a char is restricted to the range 0..=0x10ffff, and uses this for exhaustiveness checking and for niche optimizations.
Yes.
If we were going forward with this change, all such checks would be unnecessary, arguably even incorrect.
Ick. That is a pretty severe obstacle, although I'd question whether any real code actually does it.
Only for things that only deal in Rust.
Per https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf section 3.9 Unicode Encoding Forms D92,
As a consequence of the well-formedness conditions specified in Table 3-7, the following byte values are disallowed in UTF-8: C0–C1, F5–FF
And in D93,
A conformant encoding form conversion will treat any ill-formed code unit sequence as an error condition. (See conformance clause C10.) This guarantees that it will neither interpret nor emit an ill-formed code unit sequence.
So if you're interacting with anything that is UTF-8 conformant -- such as implementing a protocol that deals in conformant UTF-8 -- you'd need to apply extra checks.
I'd much rather that Rust's str was actual conformant UTF-8. If someone wants non-conformant handling, they can use a different type.
As already pointed out by someone else upthread, there's no point doing this in Rust without first getting consensus to make the same change everywhere.
And perhaps we should let this thread end with that.
For all I can tell, it is already the case that the upper limit is also enforced during tokenization (e. g. you cannot pass such a literal as a token to a macro that ignores it), so the only actual change you are discussing here would be to tweak what exactly the error message is saying. Arguably the error message that the number is too large is better anyways, in my opinion, it would even make sense to add a check to the >6 digits case whether any of the leading digits is non-zero in order to make the error message the same error about a too large value in such cases, instead of the current error about too many digits. If the adjusted check becomes more accurate in this manner, then thr remaining case with too many leading zeroes could get a useful error that includes a suggestion to remove leading zeroes.
FWIW, lang has recently been saying that tokenization should be fairly flexible in what it accepts, so long as it fits naturally in the regular-language structure of the appropriate lexer definition.
For example, 9999999999999999999999999999999999999999999999999999999999999 is a valid literal token even though it's not a valid literal value for any type that exists today in Rust: https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=06150acd91bbb39b63d0cc6e2aaadc32
Thus it might be reasonable to allow character tokens to allow more things, but reject them once they're used when typed as a char. Though it's still need someone to make a case that it's worth bothering, which is less obvious here than it is for numeric literals -- after all, Rust could add u256 one day, like it added u128, and then all those nines would fit as that number is below 2203, so this is nice forward compatibility for that. But I don't see the same thing for wider chars.
This proposed change has zero practical uses today. There are no codepoints mapped above the supported range, so there's no reason to ever use out-of-range values. Even if it was actually implemented on the literal parsing level, it still wouldn't even work, because the escape code needs to be converted to UTF-8, and that conversion will fail for out-of-range values.
Rust uses UTF-8, and UTF-8 forbids overlong sequences. Rust would have to drop its UTF-8 validity guarantee to allow overlong sequences, which would be a major change to a fundamental stable feature of Rust. And it could cause vulnerabilities in applications that relied on the validity guarantee.
The motivation for this change is purely speculative, and about a thing that has been promised not to change, and has no reason to change in the foreseeable future. It's not even Rust's responsibility for plan for future extension of it.
This is a massively bad idea on every level.
Preparation for Unicode running out of codepoints can be postponed until Rust 1.5000.0 with 100 years to spare.
However, 0000000000000000000000000000000000000000000000000000000000001 is a valid token and value for any integer type. If we accept that useless prefix of zeros, then why not do the same for Unicode escapes?
And for what little it's worth, under the time frame discussed, it's at least as likely that Rust will no longer be sufficient for programming modern hardware as that Unicode will have expanded beyond U+10FFFF. At some point it becomes unreasonable to consider compatibility with speculative futures; if we get to the point where Rust has been around for more than half the time computers have existed at all, I say it's probably done a decent enough job at being future compatible.
Rust won't live forever; it's delusional to believe so. It's a very safe bet that at some point in the next 500 years someone will make a language better suited to development of contemporary programs, because the scale of what we even do with computers is likely to be near incomprehensible to us today. The extent of how helpful a compiler can be will hopefully eclipse what is possible with the Rust language as we know it today.
For the benefit of future developers, I hope Rust doesn't last (remain a go-to choice) for 500 years.
The limitation to the number of hex-digits in escape feels as if we would restrict that we can only have at most 1 million tokens in a single function body. As aforementioned, 00000........1 is a valid integer token in the current Rust, we didn't restrict the number of digits in the integer token on lexical level just because the max width type we support is u128.
If we use the literal
let x = 9999999999999999999999999999999999999999999999999999999999999;
The compiler will just emit an error
integer literal is too large
As we do for 0x110000 in Unicode escape. Analogyly, why do we need to limit the number of hex digits in a Unicode escape on a lexical level just because the current maximum code point value is 0x10FFFF?
With regard to the argument of allowing 9999…9999, I will note that there is a plausible future where Rust has bigints. On the other hand, there is not a plausible future, at least for well in excess of a century, where the existing Unicode range is exhausted.
(I bring up bigints as an example and nothing more; please keep the thread on-topic)
I see where you're coming from, and I don't disagree, but the advantage of the existing behaviour is that, as a human reader of the code, I don't have to strip off too many leading zeroes to work out what the character might be.
It boils down to a tradeoff - even if we allow you to use 16 hex digits to write Unicode escapes, you can still only write valid codepoints, which in today's compiler is numbers from 0 to 0x10_FFFF. The only benefit from allowing more characters is that you can write those codepoints with extra leading zeros - so you can write the same character as \u{1f980} (no leading zeroes, allowed today), \u{01f980} (one leading zero, allowed today since this character is in plane 1), or in future as \u{000000000001f980} (11 leading zeroes, or 16 hex digits).
As leading zeroes have no information content (by definition), I'd prefer to go in the opposite direction - you should have to remove them from Unicode escapes.
And I see this as different to numbers; with integer or floating point literals, you can insert _ separators usefully to group digits in a programmer-friendly fashion. Thus, you might write a 132 bit hex constant as 0x0_0000_0000_0000_0000_0000_0000_0000_0001_u32 to make it clear that your origin number is 132 bit, even though it fits in a u32 (since you've got a lot of leading zeroes). With Unicode escapes, those _ separators are problematic, because they're not standardized, and thus software you look up character codes in doesn't include them (or parse them - Unicode Utilities: Character Properties fails to handle the separator, but is fine without it).
So, for this specific case, where there's an external standard we're following (Unicode), I want us to use the representations the external standard uses. That means that we should be using 4 hex digits for the position in a plane, plus a plane number prefix for characters outside the BMP (like U+19F80, 
). I also want no leading zeroes, or a way to insert separators into big numbers.
These wants conflict - Unicode wants me to write 0041 for a BMP codepoint, not 41 - so I have to compromise to some extent. And I think that saying "you can use 6 hex digits - so you can write 000041, even though that's not what Unicode wants you to do" is a better compromise overall than "you can use _ separators, but when going back to Unicode, you have to remember to remove them". But that's a matter of taste - Unicode's tooling is happy for me to write 0000000000000041 instead, and copes with the leading zeroes, I just think that that's tasteless.
On case I was unclear, I only suggested changing the contents of the error messages, not to allow any syntax that wasn't allowed already.
It's an aside here, but the reason IPv6 transition is not going anywhere is that nobody wants it. It is more complicated and since masquerades are a feature now, few get any benefit from it.