Include_raw! directive for inclusion of generated code

If you generate code from build.rs, the include! directive isn't quite ideal for including the resulting code, for several reasons:

  • include! doesn't support top-level doc comments.
  • include! doesn't support inner directives like #![no_std].
  • Any module directives in the included file get resolved relative to the included file, not relative to the file including them. So, if you generate lib-generated.rs containing mod error;, and include it from lib.rs, rustc looks for error.rs or error/mod.rs relative to lib-generated.rs.

There's a long history of bug reports related to each of these issues.

I'd like to propose a new directive, include_raw!, which includes the specified file as if its text appeared directly in the module invoking include_raw!. Any doc comments, inner directives, or module directives will get resolved as if they appeared directly in the file invoking include_raw!. Also, unlike include!, include_raw! will only work in top-level item context, not in expression mode. Note that include_raw! can still require that the text properly parse as Rust (e.g. no unmatched delimiters), and can still parse the file and include it as tokens; it should just treat those tokens as if they appeared in the including file.

EDIT:

  • @dtolnay pointed out that we could fix include! to partially address the first two issues (though the documentation and directives wouldn't get propagated to the top level).
  • I've updated this proposal to make it clear that the new directive can still parse the generated code and include it as tokens, rather than raw text.
1 Like

Note that we have #[path = "path/to/mod] mod included; today. For comparison's sake, how exactly would the behavior of include_raw! differ?

The first two bullets sound like limitations we could fix directly in include!. Is the third bullet by itself sufficient to warrant the new macro, since it's usually possible to work around with #[path = "..."] where it comes up?

Speaking of the #[path = "..."], please check out https://github.com/rust-lang/rust/issues/59345 . I'm not sure whether the current behavior is intended...

I hit exactly this problem when trying to update gl_generator recently.

Definitely a relatively easy fix that can go a long way.

  • You can't write #[path = concat!(env!("OUT_DIR"), "/generated.rs")], or otherwise generate the path.
  • The generated code will get treated as a submodule, not as the contents of the current module, which means:
    • You can't add additional code around the generated code
    • You can't use the generated code's top-level documentation as your crate's top-level documentation.
    • You can't use directives that can only appear at the top of the crate.
    • Any directives or mod statements in the generated code will get resolved relative to the generated file, not relative to the including module.

Without breaking backwards compatibility?

1 Like

Yes, since they both refer to things that don't compile today. Making them compile doesn't need to break code that currently compiles.

error: expected outer doc comment
 --> src/../josh.rs:1:1
  |
1 | //! test
  | ^^^^^^^^
  |
  = note: inner doc comments like this (starting with `//!` or `/*!`) can only appear before items

That would certainly help, then. It'd be a pain to work around the third issue (that mod foo; gets resolved relative to the generated file), but I'd still find that substantially better than the current situation.

include! is flawed. To me the worst thing is that it includes only a single expression.

But totally raw text inclusion seems like overreaction to this. It's going to have its own bunch of unsolvable problems (see: C/C++).

At very least it has to be AST-based inclusion, so you can't do questionable things with including { in one file and } in another. Rust so far has managed to enforce that all files are parseable on their own, in one pass, and that everything has balanced parenthesis.

3 Likes

Requiring self-contained parsing is fine, sure. And the single expression limitation doesn't matter if you're using it top-level to include items.

"Textual" may be a bit far. I primarily mean that I would like the contents interpreted as though they were part of the including file, which includes allowing top-level directives, inner doc comments, further includes or mod statements that resolve relative to the including file, and so on.

This should be fixable easily enough, include should work in any contexts where a macro can be invoked, not only in a limited subset hard-coded in the compiler.

Many built-in macros have similar issues and the general solution is to migrate the built-in macro infrastructure to token-based outputs.
For include specifically the issue can be fixed locally without infrastructural changes by making it return ParserAnyMacro like macro_rules macros do.


"Raw inclusion" is incompatible with our existing macro expansion model and will have to use some other mechanism.

3 Likes

Would it help if we had some construct that grouped multiple items into a single item, but without introducing a different namespace? As a strawman, for example, what if the following code was valid at the top level of a module, and was equivalent to the same code without the surrounding braces?

{
    fn foo() {}
    fn bar() {}
}
1 Like

Not quite. That would suppress errors from inner doc comments but not let them become the doc comments of the including module. Likewise inner directives.

I would suggest changing the topic's title then. "raw textual inclusion" could give a heart attack to an unsuspecting IDE writer :slight_smile:

7 Likes

Inner attributes (including doc comments) could likewise be applied at the module level. Although this might get confusing if used outside of generated code...

So, instead of raw text inclusion, this is more related to the included code having the span of the including file, right?

Thus, to also enable doc comments, what about:

a #[include(...)] attribute macro rather than a include_raw! function-like macro ?

This way your intended use-case would be a

mod foo {
    #![include(concat!("path/", "to/included/file.rs"))]
}
1 Like

Interesting. If that would work more easily, sure, that seems reasonable to me.