Opinion: `proc_macro::TokenStream` should implement Extend<proc_macro::Group>, etc

JohnScience · June 18, 2023, 10:17pm

At the moment of writing, proc_macro::TokenStream implements core::iter::Extend of proc_macro::TokenStream and proc_macro::TokenTree, i.e. Extend<TokenStream> and Extend<TokenTree>.

I believe that it would be reasonable to provide implementations for all of the types that can be stored in variants of proc_macro::TokenTree, namely

Is there any reason not to provide these implementations?

jhpratt · June 18, 2023, 11:38pm

I'd file an ACP for this. It's more than reasonable and trivial to implement.

JohnScience · June 19, 2023, 12:47am

Done:

github.com/rust-lang/libs-team

proc_macro: Implementation of Extend (parameterized over Group, Ident, Punct, and Literal) for TokenStream

opened 12:46AM - 19 Jun 23 UTC

JohnScience

T-libs-api api-change-proposal

# Proposal ## Problem statement At the moment of writing, [`proc_macro::To…kenStream`](https://doc.rust-lang.org/beta/proc_macro/struct.TokenStream.html) implements parametrizations of [`core::iter::Extend`](https://doc.rust-lang.org/beta/core/iter/trait.Extend.html) over `proc_macro::TokenStream` and [`proc_macro::TokenTree`](https://doc.rust-lang.org/proc_macro/enum.TokenTree.html), i.e. `Extend<TokenStream>` and `Extend<TokenTree>`. However, it is not implemented for parametrizations of `Extend` over 1. [`proc_macro::Group`](https://doc.rust-lang.org/proc_macro/struct.Group.html) 2. [`proc_macro::Ident`](https://doc.rust-lang.org/proc_macro/struct.Ident.html) 3. [`proc_macro::Punct`](https://doc.rust-lang.org/proc_macro/struct.Punct.html) 4. [`proc_macro::Literal`](https://doc.rust-lang.org/proc_macro/struct.Literal.html) which all can can be stored in respective variants of `TokenTree`. I believe that this is a problem because it either 1. causes unnecessary verbosity/boilerplate when one of `Group`, `Ident`, `Punct`, or `Literal`, is added to a `TokenStream`, as will be demonstrated further; or 2. forces the developers to implement a local twin of `Extend` that they would implement for `TokenStream`, either manually or using a non-standard library. Such inconveniences could push developers use [`quote`](https://crates.io/crates/quote), which results in higher compile times, even in situations where `proc_macro` with the proposed changes would suffice. Also, it *probably* results in higher compile times in projects which extensively use procedural macros because these macros have to generate more code. ## Motivating examples or use cases Before: ```rust use proc_macro::{TokenStream, TokenTree, Ident, Span}; let mut ts = TokenStream::new(); ts.extend(iter::once(TokenTree::Ident(Ident::new("new_ident", Span::call_site())))); ``` After: ```rust use proc_macro::{TokenStream, TokenTree, Ident, Span}; let mut ts = TokenStream::new(); ts.extend(iter::once(Ident::new("new_ident", Span::call_site()))); ``` Once (or when) `Extend::extend_one` stabilizes, the difference will be even more noticeable. Before: ```rust use proc_macro::{TokenStream, TokenTree, Ident, Span}; let mut ts = TokenStream::new(); ts.extend_one(TokenTree::Ident(Ident::new("new_ident", Span::call_site()))); ``` After: ```rust use proc_macro::{TokenStream, TokenTree, Ident, Span}; let mut ts = TokenStream::new(); ts.extend_one(Ident::new("new_ident", Span::call_site())); ``` ## Solution sketch ```rust impl Extend<Group> for TokenStream { fn extend(&mut self, item: Group) { self.extend(TokenTree::Group(item)) } } // And so on for the 3 remaining items, or we can use a macro_rules! for that. ``` ## Alternatives Leave it as it is and delegate the implementation to a non-standard library via an extension trait that would be a twin of `Extend`. I dislike this approach because all these types belong to standard libraries and it would result in unnecessary complexity. ## Links and related work * Rust internals discussion ["Opinion: `proc_macro::TokenStream` should implement Extend<proc_macro::Group>, etc "](https://internals.rust-lang.org/t/opinion-proc-macro-tokenstream-should-implement-extend-proc-macro-group-etc/19022).

jhpratt · June 19, 2023, 2:09am

Thanks!

system · September 17, 2023, 2:09am

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
The future of syntax extensions and macros	62	13637	March 25, 2019
[Pre-RFC] Generation of item idents in macros bikeshed (deprecated)	9	3905	March 25, 2019
[Pre-RFC] Extend "macros 1.1" to support `foo!` style macros language design	13	2314	March 25, 2019
Why does quote_expr!() requires ExtCtxt and expand to large token tree?	2	1513	March 25, 2019
Pre-RFC: allow retreiving foreign items from proc macros language design	2	609	March 25, 2019

Opinion: `proc_macro::TokenStream` should implement Extend<proc_macro::Group>, etc

Related topics