[Pre-RFC] Cargo Templates

  • Feature Name: cargo-template
  • Start Date: 28-03-2017
  • RFC PR: (leave this empty)
  • Rust Issue: (leave this empty)

Summary

This RFC proposes Cargo Templates to generate new projects from existing prototypes.

A simple version was already merged (but not released in Stable) without an RFC, but some useful feedback has underlined the need to have more feedback in the design phase.

Motivation

Writing boiler plate is no fun. Starting a new project often involves writing a lot of repeated code, copying a LICENSE file into place, setting crate dependencies, filling out a hello world example for the crate. In the case of a daemon, setting up the main event loop and signal handling is also something that requires filling out.

We propose Cargo Templates to generate new projects from existing prototypes. This lets people start projects more quickly by not having to type out boiler plate each time.

Also, because the templates can be managed as a repository, they can work as way to spread up to date ‘best practices’ through curated templates.

Detailed design

The design of templates consists of two sides. The interface for users, and the interface for template writers.

User Interface

Users creating a new project will use templates as follows:

$ cargo new foo --template https://gitrepo.com/user/rust-project-template

Template repositories may also have multiple templates within them. Therefore you can specify a subdirectory: may also have sever

$ cargo new foo --template https://gitrepo.com/user/rust-project-templates --template-subdir cmdline

Templates can also be taken from file:// or from the local filesystem (using a path). When using file:// or a path, it does not make sense to use --template-subdir but it

Template Writer Interface

Templates will consist of two parts: a Template.toml file and the files themselves. The Template.toml will consist of two parts: actions and renames.

Actions

Templates actions consist of templating a file or copying a file into place. Copying a file into place is used for binary blobs or files that do not need to be templated (e.g. they may be templates themselves).

Renames

Renames are for files that conflict with files in the template. For example, README.md and Template.toml (if you were to make a template of a template).

A rename entry is a list of <input-name> = output-name>

Example Template.toml

[actions]
template = ["src/main.rs", "src/cmd/new.rs"]
copy = [".travis.yml", "some-template.handlebars"]

[renames]
"README.md.in" = "README.md"
"Template.toml.in" = "Template.toml"

Template helpers

Some strings need to be escaped in different ways. For example a string in a toml file needs \" when escaping quotes and nothing for <. But html uses &lt; for <. This is done by the template system with toml-escape and html-escape:

[package]
name = "{{name}}"
version = "0.1.0"
authors = [{{toml-escape author}}]

Template strings

There are initially three template strings:

  • name : name of the project.
  • author: author of the project as retrieved from cargo settings or git configuration.
  • date: used for injecting the date for e.g. copyright information into the template.

How We Teach This

This should be documented in the crates.io documentation for the Cargo.toml manifest format.

Drawbacks

This adds some complication into Cargo which could possibly be better suited to an external tool, or one that is installed (e.g. clippy, etc).

Alternatives

Templating systems exist. They tend to be focused on their own language communities. e.g. Cookiecutter appears to focus on Python, while Boilr appears to focus on Go.

Creating a template tool that works as a cargo subcommand which could be installed. This is not in line one of the goals of discoverability as new users who want to generate a hello world project would have no simple way to see that the template subcommand exists. However, clippy surely has a dedicated following.

We can use other template libraries. For example, moustache or Askama.

Unresolved questions

4 Likes

Using templates to reduce work is a superb idea!

Now, maybe I don’t understand the rationale for all the details in this pre-rfc but it seems to me like an overly complex solution. A template could just be a repo that you clone, i.e.:

git clone --depth=1 --branch=master https://github.com/framework/template.git foo
rm -rf !$/.git

I don't really agree with the reason not to go with this alternative. There are many very useful third-party cargo subcommands already existing. If the discoverability of these is currently not good enough I think we need to improve that, rather than start shoving random subcommands into the cargo codebase.

Especially in this case, if this is useful enough a lot of large projects (e.g. iron, piston) along with a lot of the "intro to rust" series should be adopting it for their tutorials, giving it publicity on it's own merits instead of just because it's a part of the default cargo install (if it's widely used enough I could even see some argument towards including it as part of the default rust install as well, that still doesn't mean it needs to live in the cargo codebase).

I do agree that it would make sense to have this as a community designed project to ensure it is useful for all use cases, but that could be done by pulling cargo-template into the nursery. I just don't see any need for it to be tightly integrated into cargo itself.

3 Likes

Instead of passing a git url to --template, a crates.io fork/clone/flavor for template could be created. This would shorten

cargo new foo --template https://gitrepo.com/user/rust-project-template

to

cargo new foo --template user/rust-project-template

or even

cargo new foo --template rust-project-template

Additionally, this would allow versioning on templates (user/web-with-angular@^1.0.0 vs user/web-with-angular@^2.0.0) and would make searching for a template for a certain purpose much easier.

1 Like

Then you need to go through and fill out the name, date, and author information. I mean, if using git like you suggest works for you then that’s fine but I don’t think it’s a very nice solution.

I’m very sympathetic to turning this into an external tool. while waiting on the original cargo PR to merge, I actually extricated it into a project I still have on my laptop (called mould) so I could release that instead if it ended up taking forever.

Template Writer Interface:

I agree that there needs to be some support for template strings, but I don’t understand the motivation for most of the other complication in this feature. In particular, I’m not sure why any of the keys in the Template.toml file exist. Why would you want to store files in the template at anything other than their ultimate name (that is, why have the renames section)? Why have a distinction between files to be copied and templated, instead of just supporting an escape syntax in the template?

If templating grows significant enough that complex knobs like this are needed, I’d rather see some sort of programmatic hook provided so that the template could contain Rust code which will be executed, instead of a declarative config file. I suspect that if we open the door like this, what we’ll end up with is a junkdraw of configuration features for templates which are not fully orthogonal to one another.

But in any event, anything more complex than copying all of the files over and inserting the template strings into them seems unnecessary as a first pass at this feature.


User Interace:

What I do think is needed is a more complex system for distributing and using templates than this RFC suggests. The point of templates is to make starting new projects following a certain template ergonomic and easy. Figuring a git repo to clone from is decidedly not that. Its important to support local paths and git repos (just as we need to for dependencies), but I think the normal path should be to just install templates from crates.io and be able to access them at the name you used to install them.

I’m not exactly sure what the mechanism for implementing that is, but I do think it should be analogous to the experience of creating or installing a cargo subcommand - I upload my-template to crates.io, users can install my-template, and then cargo new --template my-template. Possibly they need to exist in their own namespace & use their own subcommands, I’m not certain.

This is why I think an external system won’t work - it wouldn’t be able to host templates on crates.io.

(I also think the --template-subdir flag is overkill for a first pass; I don’t see why we should support multiple templates in a single git repo, it does not seem like a good UX to me.)

4 Likes

I would vote for this being outside of Cargo. From the proposal it seems that it only copies files but I don’t really see the point since you would normally use a git url to get it. Why not use git clone instead?

To be useful like cookiecutter, it needs to ask lots of random potential variables (see example for django here: https://github.com/pydanny/cookiecutter-django#usage) to actually set up a useful project.

1 Like

I’m not a big fan of the proposed “copies and renames” design because requiring the user to manually maintain a list of every file in a template that gets copied into place seems like an anti-feature.

It makes extra work for the template maintainer and it’s just a mistake waiting to happen as people add, remove, and rename files and I think an “excludes and renames” design which makes “copy verbatim” the default action would make more sense.

The problem with your proposed idea for a shorter syntax is that it ties the tool a little too heavily to GitHub.

For example, I’d be using templates off my hard drive which may just happen to be pushed to GitHub if I think they’re ready and useful to others.

(cargo new --template rust-cli-boilerplate being ~/src/rust-cli-boilerplate while $PWD is ~/src)

…and, while I currently use GitHub, I put a lot of effort into maintaining an Exit Strategy document. Your proposed solution would massively inconvenience me if I moved to BitBucket or GitLab.

2 Likes

That's the approach the earlier one was taking. While I was giving the initial feedback which prompted the earlier implementation to be pulled, it brought up the case of "this template needs to contain a binary blob" (could be something that must be incorporated into an embedded build, could be a Windows resource like an icon or cursor, etc.)

The ludicrously hyperbolic but illustrative example I gave was the idea of a Django project requiring you to hex-edit {% verbatim %} and {% endverbatim %} onto the beginning and end of the EXE files you're serving to avoid the risk of 0x7b 0x7b ({{) sequences prompting pathological behaviour.

So that the template repository and the projects it generates don't have to share the same LICENSE, README, .travis.yml, etc. files.

(eg. I have a template which is MIT/Apache licensed but generates GPLv3 projects by default. GitHub's license detector isn't smart enough to read the "This is actually MIT/Apache... it just needs LICENSE to contain the GPL for functional reasons" message in the readme.)

2 Likes

I did in no way propose using “github-user/repo-name” or even using github at all. What I said is that a central index would be a big for the template proposal as it allow shorter names, a global index and proper versioning.

No doubt most template creators will choose github as a platform, be it for technical reasons or just for convenience. But you’re free to host your code anywhere else, whether it is bitbucket or even a self written git server running on free hardware. Which requirements and restrictions exactly will apply on the code hosting will be determined by creators and maintainers of the (still hypothetical) platform.

I agree that you need to able to install template from other source too, but this should be a separate subcommand/flag as this is an edge case mainly required for testing.

You're over-generalizing.

At least one person (me) always makes templates locally and then, maybe, if they've matured enough and have enough general appeal, they'll be pushed to GitHub... but I still insist on initializing new projects from the local copy.

(Among other reasons, because it's one less pointless point of failure. I've already got the authoritative copy on my hard drive. Why should I bounce it over a fallible network connection to a service which has had downtime in the past, and hope that the git->github->cargo-new chain will enforce as much tamper detection as simply never letting the data leave my system in the first place?)

In fact, getting a "too simple to have even considered non-GitHub sources" impression from their READMEs is an actual reason I've dismissed several project templating tools without reading further.

...and given what I already do with shell script and sed (and could do properly with a bit of simple Python glue), I'd be more likely to stick with my home-hacked project-templating solution than to bother building new muscle memory for something where I'd still need to wrap it up in a wrapper like I wrote for ripgrep, which fakes a different set of default command-line flags.

1 Like

As I have said, installing from a directory is a mandatory feature. Having to use a flag for that shouldn’t hurt to much, though:

cargo template --local ~/some-template-of-mine

But please consider that the main point of having templating support isn’t that experienced developers can have their custom workflows simplified, but that guides can offer a one line quickstart with batteries included. Custom workflows are of course a valid demand, but I highly doubt that it is possible to reach any consensus on that. As you’ve already seen for yourself, most experience developers will be best of with some shell, regex and python tailored to their individual preferences.

2 Likes

If examples allowed an extra Cargo.toml, it would be nice to have something like cargo new --from crate_on_crates_io/basic_example.

Update: Another advantage of this approach is that it is in-tree, automatically tested and therefore always works with the relevant library version. Please add this (dealing with different versions) to Unresolved questions.

1 Like

Thanks for explaining the motivation for these features. If we want to support these, I think the easier solutions are:

  1. Instead of copying the template from the root, copy it from a designated template directory (seems easy to see how this could be extended to solve the ‘multiple templates in one repo’ problem also). This solves the renaming issue in a very natural way.
  2. Support a list of files not to run the interpolation on (though that doesn’t seem like an extremely high priority feature). I also think Cargo.toml could support adding a template key for this to fall under instead of creating a new config file.
2 Likes

If I understand the feedback so far:

  1. Just make a cargo-template tool; no need for this to be in cargo proper.
  2. Don't list all the files that need to be copied. Just implicitly copy them. But do list the files that need the template run on them. And do list renames (so template level README can be replaced with project README).
  3. Definitely have multiple templates per repo.
  4. Simplify the CLI.

My prototype for a tool (currently called mold because that's like a 3d template/stencil and also old and decaying like rust, ha, puns!) which isn't in cargo has the following:

  1. Mold.toml file with repos mapping repo type (dir, or git), repo source (path or url) and repo name.
  2. Given ~/.local/conf/mold.d/Mold.toml with:
nursery = {type = "git", url = "https://mygitserver.com/foo.git"}

we can run:

mold new nursery/tokio-tcp mycoolproject

Further extensions could be to add a 'cargo` type which pulls from some cargo-alike for templates.

I think there's not that many files you usually need to template. Cargo.toml, LICENSE, and .<continuous-integration-system>.yml. The rest will usually be verbatim.

I'm fine with an external tool for this, as long as we can agree on one.

mold new nursery/tokio-tcp mycoolproject

I assume tokio-tcp is the name of the template in a repository registered as nursey.

It seems this repositories is just for templates, how to make sure that these templates actually compile? And how to deal with multiple versions, say somebody wrote a tutorial for a library foo 2.0 and a reader wants to try it out but 3.0 was already released?

For the latter one tags would probably be sufficient, but how to test templates if they aren't in tree?

Good question. By default we can use HEAD but I don't see why we couldn't also add a tag, branch or revision ID either at the command line or in the Mold.toml declaration.

Indeed. I see it passes the 'so obvious you can tell what it's doing without having to dig through docs' test. :slight_smile:

I was thinking that I would expect packages -- e.g., tokio -- to have associated templates, so that one could do cargo new --template tokio/hello-world or something like that. This feels awfully close to the existing "examples", in fact.

2 Likes