[Pre-RFC]: cargo new templates (v2)

tools and infrastructure cargo
1

[Pre-RFC]: cargo new templates (v2)

UPDATE: this Pre-RFC has been started as a formal RFC, which can be viewed here:

Summary

Add the ability to reference a template when running cargo new or cargo init

Motivation

We're doing this to make it "easy for frameworks to ship starter templates to users to give users the dependencies, stubs etc that they need" - @withoutboats - [Pre-RFC]: cargo new templates - #10 by withoutboats.

The expected outcome is to:

  • support templates that live on the crates.io registry inside a directory called templates at the root of a crate

Guide-level explanation

Let's create a new project using Cargo and a template from the registry. For example, we'll try this using the hello-world template in the rocket crate. To use this run the following:

cargo new my-rocket-app --template rocket/hello-world

Note: this option is also supported with the init command:

cargo init my-rocket-app --template rocket/hello-world

This will create a new cargo project called my-rocket-app using the hello-world template.

To view other rocket templates, see the templates directory in the root of the rocket repository on GitHub.

Reference-level explanation

cargo new and cargo init will work the same way as usual. The added feature will be the ability to use a template that exists within a package on the crates.io registry.

Templates can be created by any package as long as they follow these conventions:

  • the package is on the crates.io registry
  • the template lives inside the templates directory, which lives at the root of the crate

Example:

my-crate
- templates
    - hello-world
    - basic-app
    - advanced-app

If this were a real crate, the following would be valid ways to initialize a project with the my-crate templates:

  • cargo new my-new-app --template my-crate/hello-world
  • cargo new my-new-app --template my-crate/basic-app
  • cargo new my-new-app --template my-crate/advanced-app

Note: the same examples could be used with cargo init as well.

To summarize, the following is the syntax for using a template

  • cargo new <project-name> --template <crate-name>/<template-name>
  • cargo init <project-name> --template <crate-name>/<template-name>

Drawbacks

Unmaintained templates -> negative effect on the community

With cargo supporting templates, this may lead to a large influx of community-created templates. Over time, this will grow and many will go unmaintained. When community members try to use these templates, it may lead to issues that could have a hurt on the community.

Rationale and alternatives

This is the best design because it keeps the MVP relatively small in scope. It does not worry about:

  • templates that exist outside the crates.io registry
  • templates that exist locally

The purpose behind this is keeping it small and focusing on creating first-level template support.

Prior art

Does this exist in other programming languages?

Looking at other programming communities, similar implementations exist.

JavaScript

In the JS community, two similar concepts have been implemented.

npx create-<initializer>

The npm CLI also added a convention that allows you to initialize a project for any package following the convention: create-<initializer> which can then be used to generate a new project with npx (similar to npm init ). This has been implemented by popular projects such as React and Next.

gatsby new

Another JavaScript framework called Gatsby.js has a command that feels most similar to cargo new , which is gatsby new . Here is how it works:

npx degit sveltejs/template my-svelte-project

The Svelte framework follows a similar pattern to Gatsby. It uses a project scaffolding tool called degit. It follows the pattern npx degit <user/repo> <name-of-project> . You can read more about it in the README .

Yeoman

Yeoman is a "project template generator that has been in development for years, and has grown to have over 5000 project templates." - @kornel

We don't need to have all the same features as Yeoman, but there may be lessons to learn form it.

To scaffold a new project:

yo webapp

To find out the options for the project, pass the --help flag:

yo webapp --help

Python

paster

It appears there is a pip package created by the community called pastescript , which, “[creates] file layouts for packages.” You can use it by running:

paster create --template=basic_package MyPackage

ReasonML

bsb -init my-new-project -theme basic-reason

Using BuckleScript, you can initialize a basic Reason project. This includes the following: README.md bsconfig.json node_modules package.json src and is similar to cargo new .

Has the community suggested this before? Are there crates that solve this problem already?

A similar RFC for cargo templates was written back in April 2017 and shared on the internals forum. It seemed like there were a lot of discussions, but no consensus reached.

The community has also created two crates that solve similar problems:

  • cargo-generate: a developer tool to help you get up and running quickly with a new Rust project by leveraging a pre-existing git repository as a template
  • kickstart: created by @Keats described as "A scaffolding tool to get new projects up and running quickly"
  • cookiecutter : not a Rust project, but "A command-line utility that creates projects from cookiecutters (project templates)"

In addition, there was a lot of discussion both from the Cargo team and the community on this issue. It has been decided that there still remains disagreement among both the Cargo team and the community on how to solve this, hence why this RFC seems to be the logical next step.

What lessons can we learn from what other communities have done here?

There are pros and cons to having templates or some type of template ecosystem. I think the biggest question is **who will maintain them?**It may not be directly related, but it's an important point to consider should templates be added to cargo new .

If we return to our examples from the JavaScript industry, there are two that stick out:

Community maintained templates

In the Gatsby.js community, there are +300 starters. Only a select few are maintained by the Gatsby.js core team. The rest are added by community members.

Pros

  • Community members can contribute
  • There is a wilder selection of options

Cons

  • Members can abandon their starters which can negatively impact the community

Core team maintained templates

Revisiting the create-react-app and the create-next-app templates, those are maintained by core team members (i.e. a select group of individuals). There are some variations to the templates (i.e. regular vs. TypeScript).

Pros

  • Higher-quality
  • Creates a “standard”
  • Reliable

Cons

  • Requires dedicated maintainers
  • Less community involvement (beyond direct contributions)

Unresolved questions

Some questions that may require further discussion depending on how this RFC goes.

Questions that fall in the scope of this RFC:

What is a template ?

A template is a directory that lives inside crate/templates/. It must include the following

The Cargo.toml of a template must include the following text (or some other placeholder) literally at some point, which will be replaced wholesale by the template engine.

[package]
name = ""
authors = []

@CAD97

Will it add the project name, edition, and authors to the Cargo.toml?

"I would expect it would. Ideally, the template would have the power to add/update arbitrary parts of the crate (or workspace) that is being constructed, which includes the Cargo.toml file." - @ckaran

Should it be able to compile immediately?

Nice to have, but not necessary. I can see several use cases where having it generate compile_error!() within the code (with a nice error message) could be useful. @ckaran

Should it specify version control?

There are many good version control systems out there, tying your template engine to just one means that you've locked all the others out. @ckaran

Therefore, we should not have version control system(vcs) be a feature/requirement for templates. They should be vcs agnostic.

Eventually, --templates could support the --vcs flag with --templates. That way, the user could specific which version control system they want to use.

What command line options can be used with templates?

The only two command line options that should be absolutely required are --version and --help . The engine should follow the semver spec, which will guide what people have to look out for when the engine changes (e.g., x.y.z -> x.y.(z + n) changes don't require user analysis, x.y -> x.(y + n) might, etc.). If you see that the version of the template engine has changed, then --help can help you figure out what the changes were. Just keep a good change log around so that users that have used the engine for a while can quickly come up to date on new/modified features without having to dig through piles of stuff they already know about (conventional commits + tools like jilu can help with this). @ckaran

To summarize, it should support the --version and the --help options. Possibly the --vcs, but that can be a lower priority depending on the scope.

How will templates stay up to date with the crates that they're in? (i.e. new versions, which affect dependencies)

Make templates specify the versions of the crates that they work with, but only allow them to use comparison requirements (e.g. foo = {version = ">= 1.2, < 1.5"} ). When the crate is updated, the template authors either need to keep up, or end users will know to avoid those templates. Ideally, this would be paired with better search facilities on crates.io that would allow you to filter templates based on the ranges that they support. @ckaran

To summarize,

  • make templates specify the versions of the crates that they work with
  • allow them to use comparison requirements
  • users can use this information to determine whether or not they should use/avoid those templates

Lower priority and possibly out of the scope of this MVP:

  • "Ideally, this would be paired with better search facilities on crates.io that would allow you to filter templates based on the ranges that they support"

Where do the docs need to be updated in the Rust Lang book or official docs?

I think we'll update The Cargo Book for now. Open to other suggestions too.

What security measures need to be accounted for when Cargo clones from third-party templates? (i.e. what if a template contains a malicious file?)

If a template only permits textual substitution, then there is very little that a malicious template author can do (disregarding billion laughs); before you compile a template, you should look at the code that was generated. @ckaran

My suggestion is that you develop a strong security model, one that encompasses what you're trying to protect as completely as possible (e.g., prevent the template engine from hogging the CPU, overwriting all files, arbitrary communications with the network, whatever). Once you really know what you want, you'll have a better answer to what a template is. - @ckaran

How do people in the community discover templates? (through the crates themselves?)

Once again, these are two sides of the same coin. If there is a really good way to discover all and only the templates that are for a given project on crates.io, then you can keep the docs with the templates themselves. Otherwise, the library authors might mention a few good template crates, and let people search for others on their own. @ckaran

Questions that could be answered after the implementation:

  • What guidelines should the Cargo team provide for creating templates?
  • Will there be any “official” templates?

Related issues that are considered out of scope for this RFC:

  • using local templates
  • using templates that don't live in crates on the crates.io registry
  • extending this to a "full" templating engine

Future possibilities

Nothing at the moment beyond the questions I noted for after the implementation.

Thank you!

This is the second version of a Pre-RFC written for this topic. View v1 here: [Pre-RFC]: cargo new templates - #10 by withoutboats

Special thanks to the following people for reviewing v1:

And special thanks to the following additional people for reviewing v2 (it won't let me mention more than 10 people in a post, so I can't list the name but you know who you are :slight_smile:)

Changes

To be explicit about any edits made, here is a list along with a date:

  • 4/6/20 - modify call syntax to use --templates flag to avoid conflicts + improve UX
  • 4/7/20 - update "Questions that fall in the scope of this RFC" based on feedback from @steffahn and @ckaran
  • 4/11/20 - update proposed change to include cargo init
  • 4/14/20 - update prior art section with feedback from @kornel
  • 4/14/20 - update questions that fall in the scope of this RFC with feedback from @ckaran
  • 4/17/20 - update questions that fall in the scope of this RFC with feedback from @ckaran
  • 4/24/20 - update with feedback from @CAD97
  • 4/24/20 - add thank you for reviewers
  • 5/5/20 - add link to official RFC
7 Likes
2

The proposed syntax of the call needs to change.

You’re proposing to change the current call syntax

cargo new [OPTIONS] PATH (copied from cargo book or --help, actually options can be anywhere)

into cargo new [OPTIONS] [CRATE-NAME/TEMPLATE-NAME] PATH.

Something I immediately don’t really like since the crate-name + slash + template-name part looks like a path, hence the template kind-of steals the position of the destination-path. But here’s the real problem:

The problem

Currently we have also have cargo init:

cargo init [OPTIONS] [PATH] according to the cargo book

more precisely cargo init [OPTIONS] [--] [PATH] according to --help self documentation

Haha, tested it, cargo new accepts the extra -- as well.

And currently cargo init is doing (as far as I can tell) exactly the same, with the same options, as cargo new except for offering two more bits of flexibility: the path argument is optional defaulting to the current directory and the path may be a directory that already exists.

(Today I learned that cargo init seems to offer a strict superset of use cases, i.e. you can replace all your news with inits and it should all still work).

Your new syntax creates trouble now: cargo init crate_x/template_y is currently already a valid command creating a new binary package (without templates of course) in the directory with relative path crate_x/template_y. However, adopting your cargo new [OPTIONS] [CRATE-NAME/TEMPLATE-NAME] PATH syntax into cargo init [OPTIONS] [CRATE-NAME/TEMPLATE-NAME] [PATH] to keep the compatibility between the new and init commands would change (break) that behavior.

1 Like
3

I'm in favor of --template crate/template personally.

I highly suspect the way that people will use --template will be from starting at their "framework library" readme which can have that command directly, so the concrete syntax doesn't matter too much. So sticking with a flag rather than a potentially problematic positional argument seems best.

9 Likes
4

Like i said previously supporting templates will be so cozy for peoples who does have such daily work routine (generating README, LICENSE, etc) if we implement it the right way, with my small refinements to this RFC.

Without changing the call syntax or adding a burden work for doing that, its very simple with a new --template flag and replacing backslash with @ character like <crate-name>@<template-name> hence

cargo new my-rocket-app --template rocket@hello-world

Templates lives inside the .crates directory this might be conventional.

1 Like
5

Just wanted to chime in that I'm also strongly in favor of getting something like this into cargo.

The two points I'd make are that I think it should be using an option, e.g. --template ... since it is an optional modification of the usual new functionality, it seems to semantically fit with what I understand flags/options to be for. The other is that if nailing down the details for how a template is specified are tricky, it's possible to delay making any decisions by only accepting git: or http(s): git repos, where a shallow clone is done and the file/folder contents are used unmodified, since it would be nice to support those in any case.

2 Likes
6

I've updated the proposed call syntax to the following:

cargo new my-project --template rocket/my-app

Thank you all for calling that out!

It seems like most people are in favor of the / rather than the @. I am personally more of a fan of the / because it feels more intuitive IMO. Could be something to discuss further though.

Interesting idea. Not sure how intuitive it would be, but again, something that could be discussed further.

Great point! I agree. I think the simplest MVP route is to follow this:

Thoughts?

1 Like
7

I think there is more to discuss about what a template is. It has to me more involved than a simple copy of a directory or clone of a repository. Even to replicate the current default cargo new --lib. The cargo.toml includes project name, edition, authors so it's generated, not copied. On the other hand, a template should (probably) immediately be able to compile into some kind of hello world example, hence a template from some kind of framework or library needs to import that library, which goes into the cargo.toml as well.

Furthermore I guess there are options quite orthogonal to the choice of template, options like in particular the choice of version control, if any, which can usually be controlled by an extra option to cargo new. I think we need to discuss if and how such options should work with templates.

Last but not least I would appreciate a mention of cargo init in the RFC draft.

5 Likes
8

looks good to me too.

2 Likes
9

I like the idea of having the templates live within the crate. This would make it easier to add them to the crate's CI, which helps make sure they at least compile. It also makes it easier to discover what templates are available without having to build a template search function, instead you can open the templates directory and take a look around. It should also help with problems of different versions of the crate needing different versions of the template.

This RFC is a great idea, thank you!

2 Likes
10

THIS! A project's templates are going to be the starting point for many projects, and as the project evolves, the templates need to as well.

1 Like
11

I think this is fairly simple to do: just set the package.name, package.authors, package.edition, etc. keys in the Cargo.toml using e.g. toml-edit or similar. Or, nuclear option, require all templates to have an identical standard first N lines of the package manifest so that it can be trivially string replaced, and potentially forward-compatible with more templating customization options in the future.

1 Like
12

I've updated the RFC with the questions that we still need to discuss! Feel free to chime in any thoughts and we can update the RFC. The big one seems to be discussing "what is a template?"

I like this idea suggested by @CAD97. I'd be worried of this being blocked if it has to rely on toml-edit or something similar. I saw a lot of discussion here related to toml parsers used in cargo.

For my own learning, I'm going to read up on cargo init and then I will update the RFC. Thanks for calling that out!

13

@steffahn I updated the RFC to mention cargo init.

Looks like last steps then for this Pre-RFC are to answer the following questions:

  • What is a template?
    • Will it add the project name, edition, and authors to the Cargo.toml?
    • Should it be able to compile immediately?
    • Should it specify version control?
    • What command line options can be used with templates?
    • How will templates stay up to date with the crates that they're in? (i.e. new versions, which affect dependencies)

Open to ideas or comments!

14

Everything that is expected to be used widely and persist potentially for years should have version control.

1 Like
15

If you haven't already, I suggest studying Yeoman JS tool. It's a project template generator that has been in development for years, and has grown to have over 5000 project templates.

I'm not suggesting Cargo needs to have all of Yeoman's features, especially not in v1, but I'm sure there have been lessons learned and critical functionality discovered already.

3 Likes
16

I believe it should look into extensibility of adding template with options such as having interactive options when choosing a template, this is useful when bootstrapping a template with multiple options, like how vue and angular CLI does it.

At least, if people would go with plain template, it makes it harder to include more features. As such, there may be template naming something like frontend-yew-backend-actix-router-serde-capnproto to list down all the options and that would increase the template count exponentially. Rather, with plugins available there could just be web and people can choose what they want.

The downside is that this executes some code on the user computer, which may be a security risk.

1 Like
17

Ideas about interactive options did come up in the first thread about cargo new templates already (although in a different context). The conclusion was to seperate any interactiveness of cargo new from the proposal of templates and possibly make that a seperate RFC.

The question of parameterizability of templates is still not quite resolved, especially regarding parameters like version control system that apply to today’s cargo new, too. However I would believe the best way forward would be to start with a really simple template mechanism, and only consider extra features as soon as that’s established.

1 Like
18

I would expect it would. Ideally, the template would have the power to add/update arbitrary parts of the crate (or workspace) that is being constructed, which includes the Cargo.toml file.

Nice to have, but not necessary. I can see several use cases where having it generate compile_error!() within the code (with a nice error message) could be useful.

What do you mean by this? Do you mean that the template is able to specify the version of the crate that it works with, or do you mean something like a git commit hash? The former is useful, especially if you follow the cargo guidelines. The latter is probably harmful as it ties the template to a particular tool. There are many good version control systems out there, tying your template engine to just one means that you've locked all the others out.

The only two command line options that should be absolutely required are --version and --help. The engine should follow the semver spec, which will guide what people have to look out for when the engine changes (e.g., x.y.z -> x.y.(z + n) changes don't require user analysis, x.y -> x.(y + n) might, etc.). If you see that the version of the template engine has changed, then --help can help you figure out what the changes were. Just keep a good change log around so that users that have used the engine for a while can quickly come up to date on new/modified features without having to dig through piles of stuff they already know about (conventional commits + tools like jilu can help with this).

How about going in a different direction? Make templates specify the versions of the crates that they work with, but only allow them to use comparison requirements (e.g. foo = {version = ">= 1.2, < 1.5"}). When the crate is updated, the template authors either need to keep up, or end users will know to avoid those templates. Ideally, this would be paired with better search facilities on crates.io that would allow you to filter templates based on the ranges that they support.

1 Like
19

Thanks so much for the detailed answers! I've updated the RFC.

Remaining questions:

  • What is a template? (I think we should put together an example in a git repository to include)
  • Where do the docs need to be updated?
  • What security measures need to be accounted for when Cargo clones from third-party templates? (i.e. what if a template contains a malicious file?)
  • How do people in the community discover templates? (through the crates themselves?)

@ckaran and @steffahn - you two have been the most active on this Pre-RFC (thank you!). Any thoughts? Otherwise, I can take my best stab and then ping you for feedback.

Others who see this - feel free to answer as well!

20

I think that these are actually two sides of the same coin. If a template only permits textual substitution, then there is very little that a malicious template author can do (disregarding billion laughs); before you compile a template, you should look at the code that was generated.

My suggestion is that you develop a strong security model, one that encompasses what you're trying to protect as completely as possible (e.g., prevent the template engine from hogging the CPU, overwriting all files, arbitrary communications with the network, whatever). Once you really know what you want, you'll have a better answer to what a template is.

Once again, these are two sides of the same coin. If there is a really good way to discover all and only the templates that are for a given project on crates.io, then you can keep the docs with the templates themselves. Otherwise, the library authors might mention a few good template crates, and let people search for others on their own.

That said, a method for discovering objects in the wild that I developed that works surprisingly well is really, really simple; embed a UUID in a file that various search engines are likely to find. The one-liner I use (in bash) is

uuidgen | sed "s/-//g" | sed "y/abcdef/ABCDEF/" | xclip -sel clip

which produces strings like AA0FC9F1D9FA4332B434ECC18A6787FF. Give it a week or two, and Google should pick up that string, and point to this particular post. If projects that have templates could use a similar method, putting their own unique strings into their README.md files (or wherever they discuss about templates), then template authors could reference the string in their own docs. Google will then pick up the original project, as well as any templates that are made for the project.

1 Like