CharStream macros

[notriddle]

Manually forked from

Macro Keyword

Yeah, should we try to split this into two topics? Bring back item-like macros (while sticking with TokenStream representation) CharStream macros These seem like separate proposals.

---

@redradist

I was so exciting about Rust procedure macros https://blog.rust-lang.org/2018/12/21/Procedural-Macros-in-Rust-2018.html ...

#[proc_macro_attribute]
pub fn hello(attr: TokenStream, item: TokenStream) -> TokenStream {
    let input = syn::parse_macro_input!(item as syn::ItemFn);
    let name = &input.ident;
    let abi = &input.abi;
    // ...
}

Then I thought ... what if to extend it with procedure macro keyword feature ?!

Consider the following example:

[notriddle: removed examples that didn't require the CharStream feature]

javascript! {
   function do_something() {

   }
}

where javascript! is procedure keyword macro like this:

#[proc_macro_keyword]
pub fn javascript(attr: TokenStream, item: CharStream) -> impl Iterator {
    // ...
}

In such way it is possible to embed code or snippets from other languages and simplify library usage across languages !!

@redradist

When Rust will detect macro keyword then it will try to find the proc_macro and then compile involved crate, then crate with this macro will compile tokens and then Rust procede handling remaining part of the file

The biggest cited downside to this feature, in the original thread, was the effect it would have on tools like rustfmt, since if the procedural macro itself decides when it ends, it becomes impossible to produce a token tree (token trees are the representation that rustfmt operates on) without compiling the macros.

[crlf0710]

I personally think that this use case can be best supported by reference non-Rust external files.

#[foreign_lang(javascript, path="do_something.js")]
mod do_something;

Currently proc-macros can read external files, the only true remaining issue here is that it cannot create Spans within these external files(tracked in https://github.com/rust-lang/rfcs/issues/2869), so it's bad for diagnostics, but not really a deal-breaker.

[josh]

What is the use case and the problem being solved here? Why does the combination of macros and raw strings not address that use case?

(I'm leaving aside the possibility that we could make a simpler syntax for a macro taking a raw string as an argument, and I'm just asking for what use case a macro taking a raw string does not suffice.)

[crlf0710]

I think raw string suffers from the same problem, we can't create spans for a sub-string of it and use that in diagnostics.

[Aloso]

This is similar to the python! macro explained in this article.

Since in Python syntax whitespace is important, the macro uses spans to reconstruct the original whitespace. However, some Python syntax can't be used in the macro, because it is tokenized as Rust code.

[redradist]

@redradist

#[proc_macro_keyword] pub fn javascript(attr: TokenStream, item: CharStream) -> impl Iterator { // ... }

Hmm ...

It could make sense with following design:

#[proc_macro_parser]
pub fn javascript(item: CharStream) -> impl Iterator {
    // ...
}

and then to use it like this:

#[foreign_lang(javascript, path="do_something.js")]
mod do_something;

[josh]

@crlf0710

I think raw string suffers from the same problem, we can't create spans for a sub-string of it and use that in diagnostics.

I don't see why we couldn't, though. Rust already has diagnostics that point into the middle of a string; there's no fundamental reason a proc macro built around raw strings couldn't parse the contents of the string and emit spans for subsets of it.