New include_raw! macro

libs
1

here has a talk about include_raw!: Include_raw! directive for inclusion of generated code

But I want to mention this problem from another side.

serde_json has a json! macro to create a json value, inside this macro is a valid json data. Obviously, if we put the json data in a separate file, it is not only easy to maintain, but also has syntax highlighting.

But rust's existing macro include! can't do the job. Because include! requires the file content to be a valid rust expression.

We can of course put the json! macro in the json file. But is it really a good idea? Rust now has a lot of libraries that can convert some data into valid rust code through macros.(such as askama) Use the include_raw! macro for better maintainability.

2

Note that even if include_raw! exists, json!(include_raw!(path)) still won't do what you expect. It will see "include_raw!(path)" as invalid json and error.

3 Likes
3

This should work though, right?

let json = {
   let raw = include_raw!(path);
   json!(raw)
};
4

then json! will see one token made of letters r, a, and w.

Since it's raw there's no known structure or a data type for the raw variable, it can't have a value, unless we're talking about String, which already has include_str!.

For json! macro to work with something else than text directly between delimiters, there needs to be a new kind of construct in Rust's syntax that flips order of macro expansion — a macro-before-macros kind.

Using include_str! and serde_json::from_str is probably fine. If you already use the parser elsewhere, it should give you smaller executable than the code generated by the json! macro.

3 Likes
5

You could write a json_file macro as a proc-macro, implementing the "include" part yourself.

Alternatively, once the proc_macro_expand feature lands people will make proc macros that enable you to do this without writing a proc macro yourself. For instance, there could be a general purpose macro-composition macro which takes the output of one macro and uses it as the input for another. Let's name this macro compose. You could then implement a json_file macro as something like:

macro_rules! json_file (
    ($path:lit) => (
        compose!(
            raw:expr = include_str!($path);
            json!(#raw)
        )
    );
);
2 Likes
6

Note the Intellij Rust can now do language injection inside macro calls, so that you can get syntax highlighting & editing assistance for JSON syntax in json! macro calls. Unfortunately, currently you need to do it for each json! call separately.

I have doubts that include_raw! would help you with your JSON example. Besides the issues other people noted, typically you want your JSON to depend on existing variables, rather than just be a static text. It is unlikely that you could get proper code assistance for JSON file with external variables, they would be treated like syntax errors, and it would also be super-confusing to edit variables which are declared in entirely different files (and what if you want to use different variables at different call sites for the same JSON?).

7

If my guess is correct, rust should first expand include! and then json! . Then json!(include_raw!()) should work.

10

sorry, I use nvim, so I can't see any syntax highlighting. But it's not the most important part.

I didn't mean to store the json into a variable. I just want to copy it into the source code. In addition to json, it may be any other file that can be converted into valid rust code at compile time. Since it is a compile-time job, it is naturally impossible to use variables to save.

11

Your guess is incorrect. Rust macros take syntax as their argument, not expressions. Look:

macro_rules! reverse_inner (
    ([], [$($tokens:tt)*]) => (
        $($tokens)*
    );
    ([$first:tt $($more:tt)*], [$($tokens:tt)*]) => (
        reverse_inner!([$($more)*], [$first $($tokens)*])
    );
);

macro_rules! reverse (
    ($($tokens:tt)*) => (
        reverse_inner!([$($tokens)*], [])
    );
);

fn main() {
    // prints hello world
    reverse!(("hello world")!println);
}
4 Likes
12

Feel sorry. I presume to comment on this without a close knowledge of macros. I see that rust expands nested macros externally(Eager Expansion). That being the case, include_raw! does not solve my problem. So is there any other way? For example, I now implement a json_wrap! It reads in the file, and then stitches it into json!:

json_wrap! {
   let content = read_file_content("file_name");
   let res = quote! {
     json!(#content);
  }
  res.into()
}
13

That's the basic idea, yes. You'll have to write it as a proc macro though, in a separate proc-macro crate which your program will use as a dependency.

14

I see your code:

macro_rules! json_file (
    ($path:literal) => (
        compose!(
            raw:expr = include_str!($path);
            json!(#raw)
        )
    );
);

but where do I find the compose! macro?

I write a proc macro:

#[proc_macro]
pub fn jsonized_from_file(input: TokenStream) -> TokenStream {
    let mut f = File::open(input.to_string()).unwrap();
    let mut json_data = String::new();
    f.read_to_string(&mut json_data).unwrap();
    let res = quote! {
        println!(#json_data);
    };

    res.into()
}

but it cannot work. it seems I can read file in a macro. because File::open always fail.

15

There are two eventual solutions, with different tradeoffs. The first is extended const capabilities, let json = const { serde_json::from_str(include_str!("foo.json")).unwrap() };, afaik that is “only” blocked on traits-in-const. The main downside is it wouldn’t be possible to emit syntax errors pointing into the source file. The second is using RFC: proc macro `include!` by CAD97 · Pull Request #3200 · rust-lang/rfcs · GitHub to implement a proc-macro which could emit errors that are understood by IDEs and link directly to the actual error location, but requires additional code and increased build time through less reuse (serde_json’s build artifacts can be shared between const and runtime, but not between a proc-macro and runtime).

2 Likes
16

It's probably best to continue this conversation on the users forum rather than here. But to answer your questions:

but where do I find the compose! macro?

compose! doesn't exist. It's an example of something that could exist but requires proc_macro_expand to exist first.

I write a proc macro ... File::open always fail.

You should post the errors messages here so we can help you more easily.

The first problem I see though is that you're calling input.to_string(). That won't give you the literal string passed to the macro (if the user even passed a string), it'll give you the raw tokens formatted as a string. ie.

  • jsonized_from_file!(1 + 2) means input.to_string() will be "1 + 2"
  • jsonized_from_file!("foo.json") means input.to_string() will be "\"foo.json\"" (not "foo.json")

You need to check that the macro input was a string and get that string. The easiest way to do this would be to use the syn crate and parse the input as a LitStr.

use syn::{parse_macro_input, LitStr};

let file_name = parse_macro_input!(input as LitStr);
let file_name = file_name.value();

You might also need to correct the file path (eg. add "src/" to the front of it).

Once you have the contents of the file as a String you'll need to parse it as a TokenStream using TokenStream::from_str so that you have tokens you can give to the json! macro.

You'll also want to use this trick to let the compiler know that the json file is an input to your program. If you don't do this then the compiler won't notice when "foo.json" is modified and won't recompile files that call jsonized_from_file!("foo.json").

I think that's all you need to know, but I haven't tried any of this and checked. If you need more help create a new thread, link it here and ping me.

17

Come to think of it, we could make eager expansion available through macro_rules! by adding new *_eager versions of some fragment specifiers. eg. expr_eager would match an expression just like expr, but if the entire rule matches it expands the matched expression into a token stream before substituting it into the rule body.

You could then write the json_file! macro as:

macro_rules! json_file_inner (
    ($file_contents:expr_eager) => (
        json!($file_contents)
    );
);

macro_rules! json_file (
    ($file_path:lit) => (
        json_file_inner!(include!($file_path))
    );
);
3 Likes
18

Thanks for your help, I solved this problem by your suggestion.

Now I have two solutions:

the first one is using proc_macro:

#[proc_macro]
pub fn include_json(input: TokenStream) -> TokenStream {
    let file_name = parse_macro_input!(input as LitStr);
    let json_data = std::fs::read_to_string(file_name.value()).unwrap();
    let mut data = String::new();
    fmt::write(&mut data, format_args!("json!({})", json_data)).unwrap();
    TokenStream::from_str(&data).unwrap()
}

the second I find it here (it use a crate):

#[cps::cps]
macro_rules! json_include {
    ($source:literal) =>
    let $($json_source:tt)* = cps::include!($source) in
    {
        json!($($json_source)*)
    }
}

There are some restrictions on using this macro. The path to the file must be relative to the configuration of the cargo top-level file.

thanks again :slight_smile:

19

Excellent! Happy I could help :slight_smile:

I didn't know about the cps crate. That looks very useful.

Note that you can change these two lines:

let mut data = String::new();
fmt::write(&mut data, format_args!("json!({})", json_data)).unwrap();

To just:

let data = format!("json!({})", json_data);

Also, if you choose to use your own proc macro instead of the cps crate, then I still recommend doing this. This would change your macro to look like:

let data = format!("{{
    const _: &'static str = include_str!({:?});
    json!({})
}}", file_name.value(), json_data);
20

thanks for your help. I use build.rs to trace file change now. But I give a try on your codes. Unfortunately, it cannot pass compile.

The reason is read_to_string need a path relative to the cargo.toml, but include_str! need a path relative to main.rs. Therefore I need to remove the redundant part of the path:

    let data = format!(
        "{{
            const _: &'static str = include_str!({:?});
            json!({})
        }}",
        file_name.value().replace("some_prefix", ""),
        json_data
    );
21

You can pass the absolute path to include_str!(). Also read_to_string accepts a path relative to the current working directory of the program, which in the case of build scripts is the workspace root by default, not the package root (directory which contains the Cargo.toml for the current crate). When not using cargo workspaces, both are equal.