[Pre-RFC] Add a new offset_of macro to core::mem

[ckaran]

I think you’re collapsing two concepts into one.

You’re right, I am. I’m thinking about it both from the end-user’s perspective, and the perspective of someone who decides to implement a new rust compiler from scratch. In both of those cases, there exist ‘macros’ that aren’t; they have special access that ordinary macros don’t have.

From an end-user’s point of view, they’ll ‘know’ that something can be done with a macro simply because they see what appears to be a macro doing it already. And they may want to do something similar, which won’t work because they don’t have the access that the magic macros have.

From the point of view of a compiler writer, they’ll either have to read through quite a bit of documentation online to learn which are the magic macros, or they’ll learn as they try to implement macros, and realize that they have to hack something into the compiler.

In my opinion, keeping offsetof as a keyword is more ergonomic.

[wesleywiser]

From the point of view of a compiler writer, they’ll either have to read through quite a bit of documentation online to learn which are the magic macros, or they’ll learn as they try to implement macros, and realize that they have to hack something into the compiler.

It’s pretty easy to recognize these macros, because their documentation says

Built-in macros to the compiler itself. These macros do not have any corresponding definition with a macro_rules! macro, but are documented here. Their implementations can be found hardcoded into libsyntax itself. For more information, see documentation for std's macros.

and their source is empty:

    macro_rules! compile_error {
        ($msg:expr) => ({ /* compiler built-in */ });
        ($msg:expr,) => ({ /* compiler built-in */ });
    }

[drXor]

I’ll point out that until Rust has an actual specification (which it has no hope of having without the fruits of the unsafe working group’s mission), we’re stuck in a state of “the spec is whatever stable Rust does”, which, if you’re a compiler vendor, makes you way more sad than whether a certain magic macro does some trivial thing.

[ckaran]

It’s pretty easy to recognize these macros, because their documentation says…<SNIP>

That’s actually part of my point, you have dig around in the source to find that information, which brings up:

’ll point out that until Rust has an actual specification (which it has no hope of having without the fruits of the unsafe working group’s mission), we’re stuck in a state of “the spec is whatever stable Rust does”, which, if you’re a compiler vendor, makes you way more sad than whether a certain magic macro does some trivial thing.

100% true, couldn’t agree with you more! That said, I still don’t want to make things weird with magic macros, etc. As far as it is possible and practical, I’d like rust to remain a fairly clean language, where there are few, if any, surprises. I want to avoid creating a language which (when fully and formally specified) ends up making the C++ draft standard look small (the latest draft is over 1700 pages long). If you’re thinking that the move from offsetof to offset_of! won’t make a difference, you’re probably right. It just doesn’t feel very ergonomic to me though.

That said, if everyone else wants to go with a magic macro, I’ll go with the flow.

[drXor]

I want to avoid creating a language which (when fully and formally specified) ends up making the C++ draft standard look small (the latest draft is over 1700 pages long)

Unfortunately, I think that if you want to target real hardware, and have a powerful template and compile-time-evaluation system, you are going to wind up with something of that order of magnitude. (Our syntax isn’t turing complete, so we have that going for us…)

I think this is a silly point to make, but I think that it’s not worth being afraid, because we have already achieved C+±levels of complexity. =)

[ckaran]

Unfortunately, I think that if you want to target real hardware, and have a powerful template and compile-time-evaluation system, you are going to wind up with something of that order of magnitude.

I’m not sure if that level of complexity is truly necessary. I had a half-baked idea (OK, in all honesty, it was a mostly raw idea) that would simplify templating/runtime evaluation significantly from the end-user perspective. Using it, you could reduce the language size by turning all macros (including the magic ones) into procedural macros, and those would all be ordinary rust (no macro rules). The language itself would shrink, and more stuff would move into the rustc_interface crate. Since that is just a crate, we can do all the usual things of submodules, etc., which would mean that we could have an official core that all compilers support, then each compiler could have its own namespace (rustc_interface::core for everyone, rustc_interface::nightly for experiments, rustc_interface::gnu for gcc, etc.) 99.99% of all code would probably just stick with rustc_interface::prelude::*, which would only depend on rustc_interface::core, but if there is something that you absolutely need out of your compiler, and you don’t mind that you can’t use a different compiler, or need to do it a different way, then you can import the interfaces for the other compiler types.

My concern with C++ levels of complexity are the corner cases; unfortunately, it’s been a number of years since I did C++ often enough to give you good examples, but I do remember skimming through the spec, and realizing that there were some really hairy cases that would require some hard work on the part of the compiler writer to get right. I want to avoid as many weird corner cases as possible.

(Our syntax isn’t turing complete, so we have that going for us…)

I think I know what you’re saying, but… procedural macros are turing complete, right?

[felix.s]

From an end-user’s point of view, they’ll ‘know’ that something can be done with a macro simply because they see what appears to be a macro doing it already. And they may want to do something similar, which won’t work because they don’t have the access that the magic macros have.

From the point of view of a compiler writer, they’ll either have to read through quite a bit of documentation online to learn which are the magic macros, or they’ll learn as they try to implement macros, and realize that they have to hack something into the compiler.

Why isn’t there an analogous problem with intrinsics (i.e. built-in functions and types)? Do you think users convince themselves they can write their own PhantomData or mem::forget, with blackjack and hookers, simply because they are used with regular function-call or type syntax? Why would anyone infer anything about the implementation of something merely because of its usage syntax? I’m pretty sure nobody thinks include_bytes! can be defined as a declarative macro.

And personally, if I were to design a programming language, I’d very much prefer having a simple, orthogonal syntax with few keywords (preferably none) and consistent name-resolution rules, and re-use them as much as possible – even for built-ins – instead of inventing a new syntactic construct every time I need to include an operation that was previously inexpressible. If the segfault keyword proposal is anything to come by, I believe quite a few people share the sentiment, though it may vary to what extent.

My concern with C++ levels of complexity are the corner cases; unfortunately, it’s been a number of years since I did C++ often enough to give you good examples, but I do remember skimming through the spec, and realizing that there were some really hairy cases that would require some hard work on the part of the compiler writer to get right. I want to avoid as many weird corner cases as possible.

In that case, you too should be advocating against adding keywords, because keywords add special cases instead of removing them: they are arbitrarily reserved words that cannot be used as identifiers for user-defined items (despite otherwise conforming to the lexical syntax of identifiers) and aren’t subject to ordinary name resolution (including namespacing). Adding syntactic constructs to the language creates more cases to handle for the parser. Meanwhile, offset_of! as proposed here with macro-like syntax can be already parsed into a syntax tree; implementing this feature is just a matter of assigning them some semantics.

[ckaran]

Why isn’t there an analogous problem with intrinsics (i.e. built-in functions and types)? Do you think users convince themselves they can write their own PhantomData or mem::forget…

Good point, logically there are issues with them.

And personally, if I were to design a programming language, I’d very much prefer having a simple, orthogonal syntax with few keywords (preferably none) and consistent name-resolution rules, and re-use them as much as possible – even for built-ins – instead of inventing a new syntactic construct every time I need to include an operation that was previously inexpressible.

I agree. However, from an ergonomics point of view, something like let c = a + b is much easier for an end user to parse than something like set(new('c'), add(get_value('a'), get_value('b')))[1]. While it is possible to create a complete language using function notation, or some other wholly clean and consistent syntax, at some point we need to start thinking about ergonomics. Programmers are people, not computers; we have opinions, and often have shared opinions on what is ‘easy’ and what is ‘hard’. I will not voluntarily choose to program in Malbolge. I have programmed quite a bit in C, C++, and Python (and some others). From an ergonomics perspective, Rust feels easier to me. However, magic macros feel hard to me; keywords are the magic of the language, so they can do anything, including things that I can’t do on my own via writing my own functions or macros. Functions and macros feel like things that I should be able to do myself. That is why I prefer an offsetof keyword over a magic offset_of!() macro.

Finally, we need to consider crates that have already defined of offset_of!() (simply because they thought they needed it, and rust didn’t yet offer it). How will they deal with a new magic macro? Is the new macro going to be namespaced? Or is it going to be a built-in that stomps on (or gets stomped on by) the crate’s prior definition? Since we already had some keywords that were reserved, everyone knew not to try to use them as identifiers, so there shouldn’t be any crates in the wild that attempted to use offsetof. This makes the change somewhat less painful, and is part of my thinking for why RFC 2421 should be retracted.

[1] I just made up that syntax, it isn’t meant to be rust or any other language

[RalfJung]

Finally, we need to consider crates that have already defined of offset_of!() (simply because they thought they needed it, and rust didn’t yet offer it). How will they deal with a new magic macro? Is the new macro going to be namespaced? Or is it going to be a built-in that stomps on (or gets stomped on by) the crate’s prior definition? Since we already had some keywords that were reserved, everyone knew not to try to use them as identifiers, so there shouldn’t be any crates in the wild that attempted to use offsetof . This makes the change somewhat less painful, and is part of my thinking for why RFC 2421 should be retracted.

It’s going to behave like asm! and include! and any other magic macro: it is going to be properly namespaced and cause none of these issues.

[ckaran]

It’s going to behave like asm! and include! and any other magic macro: it is going to be properly namespaced and cause none of these issues.

If that’s so, then there won’t be a problem. Like I said, I personally dislike magic macros from an ergonomics point of view, but if the consensus is to use them, and they won’t introduce further problems, I’ll accept the consensus and go with it.

[rkruppe]

Summarily responding to the last ten or so comments: I do not think a strong case can be made that either a magic macro/“macro-like syntactic entity” or a keyword for offset_of is significantly worse than the other option in implementation complexity, specification complexity, or user-facing complexity. The main reason I wish for a keyword rather than a magic macro is because of a design preference to have the syntactic form of macros (PATH ! ( TT* )) be reserved for things that expand to token trees and not also as a grab bag of special language constructs that we couldn’t bother designing real syntax for. That is all.

But given that some people are highly allergic to a keyword and “macros = token based expansion” seems to be a minority position anyway, I really won’t lose much sleep over conceding this fight – it’s much more important to get this capability into stable Rust in some way.

However, I do want to note that we have plenty of contextual keywords already, so offsetof being unreserved is no obstacle wrt backwards compatibility.

[scottmcm]

However, I do want to note that we have plenty of contextual keywords already, so offsetof being unreserved is no obstacle wrt backwards compatibility.

offsetof needs to work in expressions, though, which makes it harder. The existing contextual keywords work in items where it’s easier. (And types, I guess, with dyn, but even there it worked poorly enough that it became a real keyword in the edition even though that wasn’t the original plan.)

[ckaran]

The main reason I wish for a keyword rather than a magic macro is because of a design preference to have the syntactic form of macros ( PATH ! ( TT* ) ) be reserved for things that expand to token trees and not also as a grab bag of special language constructs that we couldn’t bother designing real syntax for.

Well said. To me, this is ergonomics; making life easier for the end user.

offsetof needs to work in expressions, though, which makes it harder.

And this may be the real advantage of reserving keywords; since they are reserved, you can do anything with them.

It’s too bad that we don’t have a some kind of known naming convention that was reserved for keywords. E.g., any word that starts with the letter ‘k’ is reserved as a possible keyword in the future. I’ll be the first to admit that there would a lot of problems with such a plan, but at least you’d know what the keywords were, and you could expand the set of keywords as you see fit!

[drXor]

E.g., any word that starts with the letter ‘k’ is reserved as a possible keyword in the future.

This is in a sense Not Wrong; in C, all identifiers staring with _A, for A any capital letter, are reserved. C then asks you import headers that define the user-facing keyword with the gross “real” keyword. I.e, stdatomic.h has the line

#define atomic _Atomic

Fortunately, Rust’s macros are not quite this insane, but it means we can’t play this particular game. There has been a suggestion of introducing k#ident syntax to indicate “I want ident as a keyword, not an identifier”, originally for being able to use 2018 keywords in 2015; I could imagine you could make an argument like “add k#offsetof foo.bar now and turn it into a real identifier in the next edition”. I’m not sure I like this argument.

[ckaran]

There has been a suggestion of introducing k#ident syntax to indicate “I want ident as a keyword, not an identifier”, originally for being able to use 2018 keywords in 2015; I could imagine you could make an argument like “add k#offsetof foo.bar now and turn it into a real identifier in the next edition”. I’m not sure I like this argument.

I actually don’t like that argument; the problem with it is that it sill means that at some point k#ident -> ident, which means that if anyone was using ident for something else, they’d still need to continue using k#ident indefinitely. If we’re going to have to deal with that, we may as well stick to whatever is the accepted naming convention.

So, are we at the point where we’re going to spitball naming conventions for keywords? And if we are, what are the criteria?

[Yato]

I actually don’t like that argument; the problem with it is that it sill means that at some point k#ident -> ident , which means that if anyone was using ident for something else, they’d still need to continue using k#ident indefinitely. If we’re going to have to deal with that, we may as well stick to whatever is the accepted naming convention

They could swap over to using the r#ident syntax in future editions, therefore keeping their names, and using the new stuff. This would be useful if there are some things which couldn’t be backported.

[ckaran]

They could swap over to using the r#ident syntax in future editions, therefore keeping their names, and using the new stuff. This would be useful if there are some things which couldn’t be backported.

Yeah, but that still means someone needs to work their way through the code to find all the places where they need to change to r#ident, and fix them up. Hrmm… I’m starting to sense that rust needs something like 2to3.

Actually, now that I think about it, something like 2to3 might be a really, really good idea. Rust keeps improving, but taking advantage of those improvements requires going back over old code and fixing it. Something like cargo upgrade --old-edition=2015 --new-edition=2021 could be quite helpful.

That said, I think we’ve now officially wandered off the topic by quite a bit. If anyone wants to continue this discussion, say so, and I’ll start a new topic.

[scottmcm]

Something like cargo upgrade --old-edition=2015 --new-edition=2021 could be quite helpful.

It’s quite useful, and called cargo fix --edition

[ckaran]

Someday, I just may learn to read the docs before I type!

But that actually solves the problem, even the original question of an offsetof keyword or an offset_of!() macro. Use whatever works best for now, and if there need to be changes made in the future, let cargo fix handle it.

[catenary]

[Just] let cargo fix handle it.

I think it is the idea to do this very very very sparingly. Switching out a keyword for a macro (if it becomes writable as a macro) seems like an incredibly frivolous use case. Remember especially that example code and memory doesn’t get changed by cargo fix.

I’m in favor of the magic macro form (very mildly) only because it seems probable it will one day be a normal macro, but I agree it’s way more important to get this functionality in in some form.