# CharStream macros

**URL:** <https://internals.rust-lang.org/t/charstream-macros/12258>\
**Category:** language design\
**Created:** [April 30, 2020, 4:42pm UTC](https://internals.rust-lang.org/t/charstream-macros/12258 "2020-04-30T16:42:38Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![notriddle](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/notriddle/32/14082_2.png) [@notriddle](https://internals.rust-lang.org/u/notriddle)\
**Post date:** [April 30, 2020, 4:42pm UTC](https://internals.rust-lang.org/t/charstream-macros/12258/1 "2020-04-30T16:42:39Z")

</div>

Manually forked from

> [@Macro Keyword](https://internals.rust-lang.org/t/pre-rfc-macro-keyword/12254/29):
>
> Yeah, should we try to split this into two topics? [Bring back item-like macros](https://internals.rust-lang.org/t/bring-back-item-like-macros/12256) (while sticking with TokenStream representation) [CharStream macros](https://internals.rust-lang.org/t/charstream-macros/12258) These seem like separate proposals.

* * *

> [@Macro Keyword](https://internals.rust-lang.org/t/macro-keyword/12254/1):
>
> I was so exciting about Rust procedure macros [https://blog.rust-lang.org/2018/12/21/Procedural-Macros-in-Rust-2018.html](https://blog.rust-lang.org/2018/12/21/Procedural-Macros-in-Rust-2018.html) ...
> 
> ```rust
> #[proc_macro_attribute]
> pub fn hello(attr: TokenStream, item: TokenStream) -> TokenStream {
> let input = syn::parse_macro_input!(item as syn::ItemFn);
> let name = &input.ident;
> let abi = &input.abi;
> // ...
> }
> 
> ```
> 
> Then I thought ... what if to extend it with _procedure macro keyword_ feature ?!
> 
> Consider the following example:
> 
> _[notriddle: removed examples that didn't require the CharStream feature]_
> 
> ```rust
> javascript! {
> function do_something() {
> 
> }
> }
> 
> ```
> 
> where _javascript!_ is procedure keyword macro like this:
> 
> ```rust
> #[proc_macro_keyword]
> pub fn javascript(attr: TokenStream, item: CharStream) -> impl Iterator {
> // ...
> }
> 
> ```
> 
> In such way it is possible to embed code or snippets from other languages and simplify library usage across languages !!

> [@Macro Keyword](https://internals.rust-lang.org/t/macro-keyword/12254/13):
>
> When Rust will detect macro keyword then it will try to find the proc\_macro and then compile involved crate, then crate with this macro will compile tokens and then Rust procede handling remaining part of the file

The biggest cited downside to this feature, in the original thread, was the effect it would have on tools like `rustfmt`, since if the procedural macro itself decides when it ends, it becomes impossible to produce a token tree (token trees are the representation that rustfmt operates on) without compiling the macros.

---

<div class="post-metadata">

**Author:** ![crlf0710](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/crlf0710/32/1987_2.png) [@crlf0710](https://internals.rust-lang.org/u/crlf0710)\
**Post date:** [April 30, 2020, 5:38pm UTC](https://internals.rust-lang.org/t/charstream-macros/12258/2 "2020-04-30T17:38:44Z")

</div>

I personally think that this use case can be best supported by reference non-Rust external files.

```rust
#[foreign_lang(javascript, path="do_something.js")]
mod do_something;

```

Currently proc-macros can read external files, the only true remaining issue here is that it cannot create `Span`s within these external files(tracked in [https://github.com/rust-lang/rfcs/issues/2869](https://github.com/rust-lang/rfcs/issues/2869)), so it's bad for diagnostics, but not really a deal-breaker.

---

<div class="post-metadata">

**Author:** ![josh](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/josh/32/5934_2.png) [@josh](https://internals.rust-lang.org/u/josh)\
**Post date:** [April 30, 2020, 10:51pm UTC](https://internals.rust-lang.org/t/charstream-macros/12258/3 "2020-04-30T22:51:02Z")

</div>

What is the use case and the problem being solved here? Why does the combination of macros and raw strings not address that use case?

(I'm leaving aside the possibility that we could make a simpler syntax for a macro taking a raw string as an argument, and I'm just asking for what use case a macro taking a raw string does not suffice.)

---

<div class="post-metadata">

**Author:** ![crlf0710](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/crlf0710/32/1987_2.png) [@crlf0710](https://internals.rust-lang.org/u/crlf0710)\
**Post date:** [May 1, 2020, 2:50am UTC](https://internals.rust-lang.org/t/charstream-macros/12258/4 "2020-05-01T02:50:53Z")

</div>

I think raw string suffers from the same problem, we can't create spans for a sub-string of it and use that in diagnostics.

---

<div class="post-metadata">

**Author:** ![Aloso](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/aloso/32/5039_2.png) [@Aloso](https://internals.rust-lang.org/u/Aloso)\
**Post date:** [May 1, 2020, 3:22am UTC](https://internals.rust-lang.org/t/charstream-macros/12258/5 "2020-05-01T03:22:16Z")

</div>

This is similar to the `python!` macro explained in [this article](https://blog.m-ou.se/writing-python-inside-rust-1/).

Since in Python syntax whitespace is important, the macro uses spans to reconstruct the original whitespace. However, some Python syntax can't be used in the macro, because it is tokenized as Rust code.

---

<div class="post-metadata">

**Author:** ![redradist](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/redradist/32/6475_2.png) [@redradist](https://internals.rust-lang.org/u/redradist)\
**Post date:** [May 1, 2020, 7:56am UTC](https://internals.rust-lang.org/t/charstream-macros/12258/6 "2020-05-01T07:56:24Z")

</div>

> [@Macro Keyword](https://internals.rust-lang.org/t/macro-keyword/12254/1):
>
> #[proc\_macro\_keyword] pub fn javascript(attr: TokenStream, item: CharStream) -\> impl Iterator { // ... }

Hmm ...

It could make sense with following design:

```rust
#[proc_macro_parser]
pub fn javascript(item: CharStream) -> impl Iterator {
    // ...
}

```

and then to use it like this:

```rust
#[foreign_lang(javascript, path="do_something.js")]
mod do_something;

```

---

<div class="post-metadata">

**Author:** ![josh](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/josh/32/5934_2.png) [@josh](https://internals.rust-lang.org/u/josh)\
**Post date:** [May 1, 2020, 6:43pm UTC](https://internals.rust-lang.org/t/charstream-macros/12258/7 "2020-05-01T18:43:47Z")

</div>

> [@crlf0710](#):
>
> I think raw string suffers from the same problem, we can't create spans for a sub-string of it and use that in diagnostics.

I don't see why we couldn't, though. Rust already has diagnostics that point into the middle of a string; there's no fundamental reason a proc macro built around raw strings couldn't parse the contents of the string and emit spans for subsets of it.

---

<div class="post-metadata">

**Author:** ![notriddle](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/notriddle/32/14082_2.png) [@notriddle](https://internals.rust-lang.org/u/notriddle)\
**Post date:** [July 30, 2020, 6:57pm UTC](https://internals.rust-lang.org/t/charstream-macros/12258/8 "2020-07-30T18:57:47Z")

</div>

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.