# Is it necessary to limit Unicode-escape to only have at most six hex-digits? **URL:** \ **Category:** language design\ **Created:** [February 8, 2023, 10:00am UTC](https://internals.rust-lang.org/t/is-it-necessary-to-limit-unicode-escape-to-only-have-at-most-six-hex-digits/18304 "2023-02-08T10:00:29Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![steffahn](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/steffahn/32/13288_2.png) [@steffahn](https://internals.rust-lang.org/u/steffahn)\ **Post date:** [February 8, 2023, 4:30pm UTC](https://internals.rust-lang.org/t/is-it-necessary-to-limit-unicode-escape-to-only-have-at-most-six-hex-digits/18304/21 "2023-02-08T16:30:18Z")
> [@farnz](#): > > Rust is liberal in what it accepts - you can enter a Unicode escape with a scalar value of 0x110000, even though the upper limit on USVs is 0x10ffff. What do you mean by this comment? ```rust fn main() { let s = "\u{110000}"; } ``` ```plaintext error: invalid unicode character escape --> src/main.rs:2:14 | 2 | let s = "\u{110000}"; | ^^^^^^^^^^ invalid escape | = help: unicode escape must be at most 10FFFF ``` ---
**Author:** ![mjbshaw](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/mjbshaw/32/5103_2.png) [@mjbshaw](https://internals.rust-lang.org/u/mjbshaw)\ **Post date:** [February 8, 2023, 4:30pm UTC](https://internals.rust-lang.org/t/is-it-necessary-to-limit-unicode-escape-to-only-have-at-most-six-hex-digits/18304/22 "2023-02-08T16:30:34Z")
> [@winksaville](#): > > "Be conservative in what you send, be liberal in what you accept." Not everyone (including myself) considers that to be good advice: [Robustness principle - Wikipedia](https://en.wikipedia.org/wiki/Robustness_principle#Criticism) Anyway, personally I wouldn't mind Rust increasing the limit on Unicode escape seqeuences and instead just issuing a warning if the user writes a sequence that exceeds Unicode's current limits. That seems the best of both worlds: no hard limit, but still likely to catch mistakes. ---
**Author:** ![farnz](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/farnz/32/6587_2.png) [@farnz](https://internals.rust-lang.org/u/farnz)\ **Post date:** [February 8, 2023, 4:43pm UTC](https://internals.rust-lang.org/t/is-it-necessary-to-limit-unicode-escape-to-only-have-at-most-six-hex-digits/18304/23 "2023-02-08T16:43:18Z")
I mean that the Rust lexer, which is the party responsible for the 6 character limit, correctly identifies that as a Unicode scalar escape, and passes it onto the next stage, which then tells you that the number is too large to be a Unicode scalar value. If you enter a 7 digit value, then instead of being told that you've given it too large a number, you get told that the escape is over-long - in other words, the lexer has said that it cannot tokenise this string correctly: ```rust error: overlong unicode escape --> src/main.rs:2:14 | 2 | let s = "\u{1234567}"; | ^^^^^^^^^^^ must have at most 6 hex digits ``` If we allowed you to enter more hex digits, you'd _still_ be limited by the fact that a Unicode scalar value is currently defined as a value in the range 0 to 0x10ffff. You might be allowed to write `\u{1234567}` in a string and have it recognised as a Unicode scalar escape, but you'd then get the error that 0x1234567 is too large for a Unicode scalar. Changing this requires changing more than just the Unicode scalar escape lexing rules - you also have to change the definition of a `char`, and of `str`, to name two places that currently only support Unicode, not an extended superset of Unicode. ---
**Author:** ![winksaville](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/winksaville/32/4321_2.png) [@winksaville](https://internals.rust-lang.org/u/winksaville)\ **Post date:** [February 8, 2023, 4:51pm UTC](https://internals.rust-lang.org/t/is-it-necessary-to-limit-unicode-escape-to-only-have-at-most-six-hex-digits/18304/24 "2023-02-08T16:51:23Z")
I agree, no advice should ever be universally accepted. But listening is always a good idea, LoL. ---
**Author:** ![scottmcm](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/scottmcm/32/2355_2.png) [@scottmcm](https://internals.rust-lang.org/u/scottmcm)\ **Post date:** [February 8, 2023, 5:31pm UTC](https://internals.rust-lang.org/t/is-it-necessary-to-limit-unicode-escape-to-only-have-at-most-six-hex-digits/18304/25 "2023-02-08T17:31:22Z")
> [@zackw](#): > > What I'm saying is that I think Rust (and the software industry in general) should revert to RFC 2044 / original ISO 10646, with the maximum codepoint value set at U+7FFF\_FFFF, as soon as practically feasible (in particular, _not_ waiting for the 17-plane space to be anywhere near full). That work would naturally involve, among other things, changing \u escapes to allow up to eight hex digits. This would also require expanding the definition of safe `str`s to include not-valid-in-today's UTF-8 byte sequences. I don't think that's acceptable, as it'd immediately result in everyone trying to accept only Unicode-valid UTF-8 needing to run extra checks. Not to mention that it's a breaking change to make char bigger anyway, since you can exhaustively match on `char`: [https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=1d747466baf240a19236e4cdf8ba2168](https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=1d747466baf240a19236e4cdf8ba2168) ```rust pub fn foo(c: char) { let ('\0'..='\u{10FFFF}') = c; } ``` So this isn't happening in Rust in the foreseeable future. If Unicode gets to the point where they're talking about potentially needing something bigger, then we'll add new types for whatever that is. ---
**Author:** ![NoamB](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/noamb/32/6998_2.png) [@NoamB](https://internals.rust-lang.org/u/NoamB)\ **Post date:** [February 8, 2023, 5:45pm UTC](https://internals.rust-lang.org/t/is-it-necessary-to-limit-unicode-escape-to-only-have-at-most-six-hex-digits/18304/26 "2023-02-08T17:45:07Z")
I don't really see why this would be a meaningful improvement, even if Unicode were to be expanded. 1. As I see it, what this will allow is compiling NEW rust code (post Unicode expansion, when new code points have an assigned meaning), to be compiled by an OLD rust compiler. Why not use the new rust compiler for that? 2. Furthermore, wouldn't std need to know something about the new values, for capitalisation and such to work properly? I'm not knowledgeable in this area, but if anything, the std Unicode handling would be the main issue with forward compatibility: old code running on new data. ---
**Author:** ![farnz](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/farnz/32/6587_2.png) [@farnz](https://internals.rust-lang.org/u/farnz)\ **Post date:** [February 8, 2023, 6:22pm UTC](https://internals.rust-lang.org/t/is-it-necessary-to-limit-unicode-escape-to-only-have-at-most-six-hex-digits/18304/27 "2023-02-08T18:22:39Z")
> [@NoamB](#): > > As I see it, what this will allow is compiling NEW rust code (post Unicode expansion, when new code points have an assigned meaning), to be compiled by an OLD rust compiler. Why not use the new rust compiler for that? It won't even allow that, because after the lexing stage (which is where the 6 hex digit limit is imposed), a later stage will validate that the given scalar value is in the accepted range for the version of Unicode that the compiler knows about. Increasing the number of allowed hex digits in a Unicode scalar escape simply allows you to add leading zeroes. With 6 hex digits, `\u{000041}` is a valid way to write 'A'. With 8 hex digits, I could write it as `\u{00000041}` instead (rejected today, since that's too many hex digits) - but the compiler would still reject `\u{12345678}` because the maximum allowed value for a Unicode scalar value is `\u{10ffff}`. Given that Unicode scalar values are in the range 0 to 0x10ffff, 6 hex digits seems like a reasonable position for today - technically, we could make it an optional 0 or 1 followed by 5 hex digits, without causing issues, but that makes the lexer's job harder. Expanding the size of `char` etc so that we support not just Unicode scalar escapes with more than 6 hex digits, but also Unicode scalar values above `0x10ffff` is a much bigger job - for starters, Rust currently knows that a `char` is restricted to the range `0..=0x10ffff`, and uses this for exhaustiveness checking and for niche optimizations. ---
**Author:** ![zackw](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/zackw/32/2071_2.png) [@zackw](https://internals.rust-lang.org/u/zackw)\ **Post date:** [February 8, 2023, 7:25pm UTC](https://internals.rust-lang.org/t/is-it-necessary-to-limit-unicode-escape-to-only-have-at-most-six-hex-digits/18304/28 "2023-02-08T19:25:50Z")
> [@scottmcm](#): > > This would also require expanding the definition of safe `str`s to include not-valid-in-today's UTF-8 byte sequences. Yes. > [@scottmcm](#): > > I don't think that's acceptable, as it'd immediately result in everyone trying to accept only Unicode-valid UTF-8 needing to run extra checks. If we were going forward with this change, all such checks would be unnecessary, arguably even _incorrect_. > [@scottmcm](#): > > it's a breaking change to make char bigger anyway, since you can exhaustively match on `char` Ick. That _is_ a pretty severe obstacle, although I'd question whether any real code actually does it. ---
**Author:** ![scottmcm](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/scottmcm/32/2355_2.png) [@scottmcm](https://internals.rust-lang.org/u/scottmcm)\ **Post date:** [February 8, 2023, 7:45pm UTC](https://internals.rust-lang.org/t/is-it-necessary-to-limit-unicode-escape-to-only-have-at-most-six-hex-digits/18304/29 "2023-02-08T19:45:03Z")
> [@zackw](#): > > If we were going forward with this change, all such checks would be unnecessary, arguably even _incorrect_. Only for things that _only_ deal in Rust. Per [https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf](https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf) section 3.9 _Unicode Encoding Forms_ D92, > As a consequence of the well-formedness conditions specified in Table 3-7, the following byte values are disallowed in UTF-8: C0–C1, F5–FF And in D93, > A conformant encoding form conversion will treat any ill-formed code unit sequence as an error condition. (See conformance clause C10.) This guarantees that it will neither interpret nor emit an ill-formed code unit sequence. So if you're interacting with _anything_ that is UTF-8 conformant -- such as implementing a protocol that deals in conformant UTF-8 -- you'd need to apply extra checks. I'd much rather that Rust's `str` was actual conformant UTF-8. If someone wants non-conformant handling, they can use a different type. ---
**Author:** ![zackw](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/zackw/32/2071_2.png) [@zackw](https://internals.rust-lang.org/u/zackw)\ **Post date:** [February 8, 2023, 7:58pm UTC](https://internals.rust-lang.org/t/is-it-necessary-to-limit-unicode-escape-to-only-have-at-most-six-hex-digits/18304/31 "2023-02-08T19:58:17Z")
As already pointed out by someone else upthread, there's no point doing this in Rust without first getting consensus to make the same change _everywhere_. And perhaps we should let this thread end with that. ---
**Author:** ![steffahn](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/steffahn/32/13288_2.png) [@steffahn](https://internals.rust-lang.org/u/steffahn)\ **Post date:** [February 8, 2023, 11:30pm UTC](https://internals.rust-lang.org/t/is-it-necessary-to-limit-unicode-escape-to-only-have-at-most-six-hex-digits/18304/32 "2023-02-08T23:30:51Z")
> [@farnz](#): > > technically, we could make it an optional 0 or 1 followed by 5 hex digits, without causing issues, but that makes the lexer's job harder For all I can tell, it is already the case that the upper limit is also enforced during tokenization (e. g. you cannot pass such a literal as a token to a macro that ignores it), so the only _actual_ change you are discussing here would be to tweak what exactly the error message is saying. Arguably the error message that the number is too large is _better_ anyways, in my opinion, it would even make sense to add a check to the \>6 digits case whether any of the leading digits is non-zero in order to make the error message the same error about a too large value in such cases, instead of the current error about too many digits. If the adjusted check becomes more accurate in this manner, then thr remaining case with too many leading zeroes could get a useful error that includes a suggestion to remove leading zeroes. ---
**Author:** ![scottmcm](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/scottmcm/32/2355_2.png) [@scottmcm](https://internals.rust-lang.org/u/scottmcm)\ **Post date:** [February 9, 2023, 12:20am UTC](https://internals.rust-lang.org/t/is-it-necessary-to-limit-unicode-escape-to-only-have-at-most-six-hex-digits/18304/33 "2023-02-09T00:20:00Z")
> [@steffahn](#): > > it is already the case that the upper limit is also enforced during tokenization FWIW, lang has recently been saying that _tokenization_ should be fairly flexible in what it accepts, so long as it fits naturally in the regular-language structure of the appropriate lexer definition. For example, `9999999999999999999999999999999999999999999999999999999999999` is a valid _literal token_ even though it's not a valid literal _value_ for any type that exists today in Rust: [https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=06150acd91bbb39b63d0cc6e2aaadc32](https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=06150acd91bbb39b63d0cc6e2aaadc32) Thus it _might_ be reasonable to allow character _tokens_ to allow more things, but reject them once they're used when _typed_ as a `char`. Though it's still need someone to make a case that it's worth _bothering_, which is less obvious here than it is for numeric literals -- after all, Rust _could_ add `u256` one day, like it added `u128`, and then all those nines would fit as that number is below 2203, so this is nice forward compatibility for that. But I don't see the same thing for wider `char`s. ---
**Author:** ![kornel](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kornel/32/2711_2.png) [@kornel](https://internals.rust-lang.org/u/kornel)\ **Post date:** [February 9, 2023, 12:32am UTC](https://internals.rust-lang.org/t/is-it-necessary-to-limit-unicode-escape-to-only-have-at-most-six-hex-digits/18304/34 "2023-02-09T00:32:01Z")
This proposed change has zero practical uses today. There are no codepoints mapped above the supported range, so there's no reason to ever use out-of-range values. Even if it was actually implemented on the literal parsing level, it still wouldn't even work, because the escape code needs to be converted to UTF-8, and that conversion will fail for out-of-range values. Rust uses UTF-8, and UTF-8 forbids overlong sequences. Rust would have to drop its UTF-8 validity guarantee to allow overlong sequences, which would be a major change to a fundamental stable feature of Rust. And it could cause vulnerabilities in applications that relied on the validity guarantee. The motivation for this change is purely speculative, and about a thing that has been promised not to change, and has no reason to change in the foreseeable future. It's not even Rust's responsibility for plan for future extension of it. This is a massively bad idea on every level. Preparation for Unicode running out of codepoints can be postponed until Rust 1.5000.0 with 100 years to spare. ---
**Author:** ![cuviper](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/cuviper/32/1897_2.png) [@cuviper](https://internals.rust-lang.org/u/cuviper)\ **Post date:** [February 9, 2023, 12:52am UTC](https://internals.rust-lang.org/t/is-it-necessary-to-limit-unicode-escape-to-only-have-at-most-six-hex-digits/18304/35 "2023-02-09T00:52:19Z")
> [@scottmcm](#): > > For example, `9999999999999999999999999999999999999999999999999999999999999` is a valid _literal token_ even though it's not a valid literal _value_ for any type that exists today in Rust: However, `0000000000000000000000000000000000000000000000000000000000001` is a valid token _and_ value for any integer type. If we accept that useless prefix of zeros, then why not do the same for Unicode escapes? ---
**Author:** ![CAD97](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/cad97/32/3460_2.png) [@CAD97](https://internals.rust-lang.org/u/CAD97)\ **Post date:** [February 9, 2023, 12:56am UTC](https://internals.rust-lang.org/t/is-it-necessary-to-limit-unicode-escape-to-only-have-at-most-six-hex-digits/18304/36 "2023-02-09T00:56:23Z")
And for what little it's worth, under the time frame discussed, it's at least as likely that Rust will no longer be sufficient for programming modern hardware as that Unicode will have expanded beyond U+10FFFF. At some point it becomes unreasonable to consider compatibility with speculative futures; if we get to the point where Rust has been around for more than half the time computers have existed at all, I say it's probably done a decent enough job at being future compatible. Rust won't live forever; it's delusional to believe so. It's a very safe bet that at some point in the next 500 years someone will make a language better suited to development of contemporary programs, because the scale of what we even do with computers is likely to be near incomprehensible to us today. The extent of how helpful a compiler can be will hopefully eclipse what is possible with the Rust language as we know it today. For the benefit of future developers, I hope Rust doesn't last (remain a go-to choice) for 500 years. ---
**Author:** ![xmh0511](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/xmh0511/32/9874_2.png) [@xmh0511](https://internals.rust-lang.org/u/xmh0511)\ **Post date:** [February 9, 2023, 5:08am UTC](https://internals.rust-lang.org/t/is-it-necessary-to-limit-unicode-escape-to-only-have-at-most-six-hex-digits/18304/37 "2023-02-09T05:08:57Z")
The limitation to the number of hex-digits in escape feels as if we would restrict that we can only have at most 1 million tokens in a single function body. As aforementioned, `00000........1` is a valid integer token in the current Rust, we didn't restrict the number of digits in the integer token on lexical level just because the max width type we support is `u128`. If we use the literal ```rust let x = 9999999999999999999999999999999999999999999999999999999999999; ``` The compiler will just emit an error > integer literal is too large As we do for `0x110000` in Unicode escape. Analogyly, why do we need to limit the number of hex digits in a Unicode escape on a lexical level just because the current maximum code point value is `0x10FFFF`? ---
**Author:** ![jhpratt](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/jhpratt/32/11640_2.png) [@jhpratt](https://internals.rust-lang.org/u/jhpratt)\ **Post date:** [February 9, 2023, 5:35am UTC](https://internals.rust-lang.org/t/is-it-necessary-to-limit-unicode-escape-to-only-have-at-most-six-hex-digits/18304/38 "2023-02-09T05:35:21Z")
With regard to the argument of allowing 9999…9999, I will note that there is a plausible future where Rust has bigints. On the other hand, there is _not_ a plausible future, at least for well in excess of a century, where the existing Unicode range is exhausted. (I bring up bigints as an example and nothing more; please keep the thread on-topic) ---
**Author:** ![farnz](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/farnz/32/6587_2.png) [@farnz](https://internals.rust-lang.org/u/farnz)\ **Post date:** [February 9, 2023, 2:41pm UTC](https://internals.rust-lang.org/t/is-it-necessary-to-limit-unicode-escape-to-only-have-at-most-six-hex-digits/18304/39 "2023-02-09T14:41:44Z")
I see where you're coming from, and I don't disagree, but the advantage of the existing behaviour is that, as a human reader of the code, I don't have to strip off too many leading zeroes to work out what the character might be. It boils down to a tradeoff - even if we allow you to use 16 hex digits to write Unicode escapes, you can still only write valid codepoints, which in today's compiler is numbers from 0 to 0x10\_FFFF. The only benefit from allowing more characters is that you can write those codepoints with extra leading zeros - so you can write the same character as `\u{1f980}` (no leading zeroes, allowed today), `\u{01f980}` (one leading zero, allowed today since this character is in plane 1), or in future as `\u{000000000001f980}` (11 leading zeroes, or 16 hex digits). As leading zeroes have no information content (by definition), I'd prefer to go in the opposite direction - you should have to remove them from Unicode escapes. And I see this as different to numbers; with integer or floating point literals, you can insert `_` separators usefully to group digits in a programmer-friendly fashion. Thus, you might write a 132 bit hex constant as `0x0_0000_0000_0000_0000_0000_0000_0000_0001_u32` to make it clear that your origin number is 132 bit, even though it fits in a `u32` (since you've got a lot of leading zeroes). With Unicode escapes, those `_` separators are problematic, because they're not standardized, and thus software you look up character codes in doesn't include them (or parse them - [Unicode Utilities: Character Properties](https://util.unicode.org/UnicodeJsps/character.jsp?a=1_F980&B1=Show) fails to handle the separator, but is fine without it). So, for this specific case, where there's an external standard we're following (Unicode), I want us to use the representations the external standard uses. That means that we should be using 4 hex digits for the position in a plane, plus a plane number prefix for characters outside the BMP (like U+19F80, 🦀). I also want no leading zeroes, or a way to insert separators into big numbers. These wants conflict - Unicode wants me to write 0041 for a BMP codepoint, not 41 - so I have to compromise to some extent. And I think that saying "you can use 6 hex digits - so you can write 000041, even though that's not what Unicode wants you to do" is a better compromise overall than "you can use \_ separators, but when going back to Unicode, you have to remember to remove them". But that's a matter of taste - Unicode's tooling is happy for me to write 0000000000000041 instead, and copes with the leading zeroes, I just think that that's tasteless. ---
**Author:** ![steffahn](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/steffahn/32/13288_2.png) [@steffahn](https://internals.rust-lang.org/u/steffahn)\ **Post date:** [February 9, 2023, 3:05pm UTC](https://internals.rust-lang.org/t/is-it-necessary-to-limit-unicode-escape-to-only-have-at-most-six-hex-digits/18304/40 "2023-02-09T15:05:53Z")
On case I was unclear, I only suggested changing the contents of the error messages, not to allow any syntax that wasn't allowed already. ---
**Author:** ![jan\_hudec](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/jan_hudec/32/4100_2.png) [@jan\_hudec](https://internals.rust-lang.org/u/jan_hudec)\ **Post date:** [February 14, 2023, 9:31am UTC](https://internals.rust-lang.org/t/is-it-necessary-to-limit-unicode-escape-to-only-have-at-most-six-hex-digits/18304/41 "2023-02-14T09:31:36Z")
> [@zackw](#): > > Witness how putting it off as long as possible has made the IPv6 transition take substantially longer, cost more money, and do more collateral damage to things like the end-to-end principle than it would have if we'd collectively ripped the band-aid off in the early 2000s. It's an aside here, but the reason IPv6 transition is not going anywhere is that _nobody wants it_. It is more complicated and since _masquerades are a feature now_, few get any benefit from it. [Previous page](https://internals.rust-lang.org/t/is-it-necessary-to-limit-unicode-escape-to-only-have-at-most-six-hex-digits/18304.md?page=1) [Next page](https://internals.rust-lang.org/t/is-it-necessary-to-limit-unicode-escape-to-only-have-at-most-six-hex-digits/18304.md?page=3)