Opinion: `proc_macro::TokenStream` should implement Extend<proc_macro::Group>, etc

JohnScience · June 18, 2023, 10:17pm

At the moment of writing, proc_macro::TokenStream implements core::iter::Extend of proc_macro::TokenStream and proc_macro::TokenTree, i.e. Extend<TokenStream> and Extend<TokenTree>.

I believe that it would be reasonable to provide implementations for all of the types that can be stored in variants of proc_macro::TokenTree, namely

Is there any reason not to provide these implementations?

jhpratt · June 18, 2023, 11:38pm

I'd file an ACP for this. It's more than reasonable and trivial to implement.

JohnScience · June 19, 2023, 12:47am

Done:

github.com/rust-lang/libs-team

proc_macro: Implementation of Extend (parameterized over Group, Ident, Punct, and Literal) for TokenStream

opened 12:46AM - 19 Jun 23 UTC

JohnScience

T-libs-api api-change-proposal

# Proposal ## Problem statement At the moment of writing, [`proc_macro::To…kenStream`](https://doc.rust-lang.org/beta/proc_macro/struct.TokenStream.html) implements parametrizations of [`core::iter::Extend`](https://doc.rust-lang.org/beta/core/iter/trait.Extend.html) over `proc_macro::TokenStream` and [`proc_macro::TokenTree`](https://doc.rust-lang.org/proc_macro/enum.TokenTree.html), i.e. `Extend<TokenStream>` and `Extend<TokenTree>`. However, it is not implemented for parametrizations of `Extend` over 1. [`proc_macro::Group`](https://doc.rust-lang.org/proc_macro/struct.Group.html) 2. [`proc_macro::Ident`](https://doc.rust-lang.org/proc_macro/struct.Ident.html) 3. [`proc_macro::Punct`](https://doc.rust-lang.org/proc_macro/struct.Punct.html) 4. [`proc_macro::Literal`](https://doc.rust-lang.org/proc_macro/struct.Literal.html) which all can can be stored in respective variants of `TokenTree`. I believe that this is a problem because it either 1. causes unnecessary verbosity/boilerplate when one of `Group`, `Ident`, `Punct`, or `Literal`, is added to a `TokenStream`, as will be demonstrated further; or 2. forces the developers to implement a local twin of `Extend` that they would implement for `TokenStream`, either manually or using a non-standard library. Such inconveniences could push developers use [`quote`](https://crates.io/crates/quote), which results in higher compile times, even in situations where `proc_macro` with the proposed changes would suffice. Also, it *probably* results in higher compile times in projects which extensively use procedural macros because these macros have to generate more code. ## Motivating examples or use cases Before: ```rust use proc_macro::{TokenStream, TokenTree, Ident, Span}; let mut ts = TokenStream::new(); ts.extend(iter::once(TokenTree::Ident(Ident::new("new_ident", Span::call_site())))); ``` After: ```rust use proc_macro::{TokenStream, TokenTree, Ident, Span}; let mut ts = TokenStream::new(); ts.extend(iter::once(Ident::new("new_ident", Span::call_site()))); ``` Once (or when) `Extend::extend_one` stabilizes, the difference will be even more noticeable. Before: ```rust use proc_macro::{TokenStream, TokenTree, Ident, Span}; let mut ts = TokenStream::new(); ts.extend_one(TokenTree::Ident(Ident::new("new_ident", Span::call_site()))); ``` After: ```rust use proc_macro::{TokenStream, TokenTree, Ident, Span}; let mut ts = TokenStream::new(); ts.extend_one(Ident::new("new_ident", Span::call_site())); ``` ## Solution sketch ```rust impl Extend<Group> for TokenStream { fn extend(&mut self, item: Group) { self.extend(TokenTree::Group(item)) } } // And so on for the 3 remaining items, or we can use a macro_rules! for that. ``` ## Alternatives Leave it as it is and delegate the implementation to a non-standard library via an extension trait that would be a twin of `Extend`. I dislike this approach because all these types belong to standard libraries and it would result in unnecessary complexity. ## Links and related work * Rust internals discussion ["Opinion: `proc_macro::TokenStream` should implement Extend<proc_macro::Group>, etc "](https://internals.rust-lang.org/t/opinion-proc-macro-tokenstream-should-implement-extend-proc-macro-group-etc/19022).

jhpratt · June 19, 2023, 2:09am

Thanks!

system · September 17, 2023, 2:09am

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
CharStream macros language design	7	1035	July 30, 2020
Proc Macro over external files?	1	649	March 25, 2019
Macro Keyword language design	40	3631	August 5, 2020
[Pre-RFC] Extend "macros 1.1" to support `foo!` style macros language design	13	2286	March 25, 2019
[Idea or pre-RFC] Arbitrary Token Stream Region language design	11	741	August 20, 2019

Opinion: `proc_macro::TokenStream` should implement Extend<proc_macro::Group>, etc

Related topics