CharStream macros

Manually forked from

The biggest cited downside to this feature, in the original thread, was the effect it would have on tools like rustfmt, since if the procedural macro itself decides when it ends, it becomes impossible to produce a token tree (token trees are the representation that rustfmt operates on) without compiling the macros.

I personally think that this use case can be best supported by reference non-Rust external files.

#[foreign_lang(javascript, path="do_something.js")]
mod do_something;

Currently proc-macros can read external files, the only true remaining issue here is that it cannot create Spans within these external files(tracked in, so it's bad for diagnostics, but not really a deal-breaker.


What is the use case and the problem being solved here? Why does the combination of macros and raw strings not address that use case?

(I'm leaving aside the possibility that we could make a simpler syntax for a macro taking a raw string as an argument, and I'm just asking for what use case a macro taking a raw string does not suffice.)


I think raw string suffers from the same problem, we can't create spans for a sub-string of it and use that in diagnostics.

1 Like

This is similar to the python! macro explained in this article.

Since in Python syntax whitespace is important, the macro uses spans to reconstruct the original whitespace. However, some Python syntax can't be used in the macro, because it is tokenized as Rust code.

1 Like

Hmm ...

It could make sense with following design:

pub fn javascript(item: CharStream) -> impl Iterator {
    // ...

and then to use it like this:

#[foreign_lang(javascript, path="do_something.js")]
mod do_something;

I don't see why we couldn't, though. Rust already has diagnostics that point into the middle of a string; there's no fundamental reason a proc macro built around raw strings couldn't parse the contents of the string and emit spans for subsets of it.