Pre-RFC: project-based examples for cargo projects


#1

Summary

This RFC enables cargo to run, test and bench examples which are arranged and stored as a project-based cargo project in examples folder, in addition to existing single examples/*.rs and examples/**/main.rs file-based examples. A project-based example imply a complete cargo project with Cargo.toml, src folder, main.rs and maybe other necessary files in it.

Motivation

Recently many projects, especially huge library projects, are used in a bunch of conditions, and to prove the universal use of the project, a wide variety of examples is needed. But differences are there between examples and the project itself because an example:

  1. Might need more external crates than the project itself;
  2. Could be compiled into various cargo targets.

For example, when developing a backend for code editors like Xray, we might need to implement its frontend in a diversity of forms like terminal for vim-like experience, Qt for graphic UI, or even wasm for online judge websites. These forms need to import different crates and are built to different targets. If we only use existing single file examples, we were not even able to import external crates without editing the Cargo.toml for the project itself, which might lead to compiling unnecessary libraries to build this project.

Likewise, other projects, especially developed for embedded platforms, also needs separate dependency for writing examples. When writing examples for them, developers now have to rely on the dev-dependencies and write dependencies together for all examples into the root Cargo.toml, like what projects like f3 did, and we have to compile all of them even if trying to build only one example. If project-based examples could be formed for these projects, developers would be able to run the examples in a more graceful way as well as save compile time.

In addition, many library projects like yew, stdweb and wasm-bindgen, now already somehow made an attempt to place folders into examples folder to form an array of example ‘projects’.

Guide-level explanation

Generalized abstract

Before we start, we should keep in mind that firstly this RFC enhances the search range for parameter --example in subcommand run, test and bench, and secondly this RFC adds --example for subcommand new.

This means, for example, you can create a example project abc using cargo new --example abc, and you may run it with cargo run --example abc as it now searches for all examples/abc.rs file, examples/abc/main.rs and examples/abc/ project folder.

An example project could include its own tests and benches, but to make things more clear, it’s not suggested to have nested example projects inside. Building a nested example project is possible, but as the only way to run, test or bench the nested example is by adding [workspace] members and run with cargo run -p, it’s not convenient by now to deal with the nested example project directly using cargo command from the root project.

When to use project-based examples

When you are developing a library project, trying to build examples in single examples/*.rs files or examples/**/main.rs that could be complex enough to:

  1. import external crates that the library project itself won’t use
  2. be compiled into entirely different framework than the library project itself

you might need to change an approach because you are now able to build project-based examples.

Create a project-based example

Assume that you have a library project my_project and you want to build examples for it. So here we go.

Firstly, use cd command to direct your terminal to project root. Now the terminal might shows like this before your cursor:

XXX: my_project username$ 

Secondly, Create a example project using:

cargo new --example abc

where abc is the name of your example. This command creates a folder for your project-based example abc, which is a complete templated cargo project, in examples folder. So your my_project have a structure like:

.
├── Cargo.lock
├── Cargo.toml
├── src
│   └── lib.rs 
└── examples 
    └── abc # created by the command
        ├── Cargo.lock 
        ├── Cargo.toml # import your crates here
        └── src
            └── main.rs # write your example code here

By creating an example project, you are now ready to write your code in main.rs and Cargo.toml. Feel free to code here with your editor.

Note that a .gitignore file is added to examples/abc/.gitignore as gitignore is supported in nested folders. A simple line !Cargo.lock will be written in it so that the gitignore rule for Cargo.lock is not to be overridden by the .gitignore file in the root. As the .gitignore in subdirectories effects the directory it is in, the .gitignore in root project is not to be effected.

As for Cargo.toml, for your convenience, cargo automatically generate a dependency on your root project like:

[dependencies]
my_project = { path = "../.." }

Thirdly, after your code work, you need to run your example. Instead of using cd examples/abc and cargo run, which would also works but somehow complicated, you could run your example as simple as:

cargo run --example abc

Then you can happily witness you example code being run in console, just as what you did before to *.rs example files.

Additonally, you may include tests and benches into your example project. You can create tests or benches in examples/abc/ folder, and run it by cargo test --example abc etc. Note that it’s not suggested to create examples for examples.

When conflicting with *.rs ot **/main.rs files

Your examples folder may include more than one of *.rs files, **/main.rs files and/or project-based examples as folders, thus might have conflicting names when for example, you have at least two of examples/abc.rs, examples/abc/main.rs and project examples/abc/.

Yet cargo already have supported *.rs and **/main.rs. If there are conflicts between them, or another word you have both of them in your examples folder, you will get this message when trying to run this example:

error: failed to parse manifest at `<Project Root>/Cargo.toml`

Caused by:
  found duplicate example name a, but all example targets must have a unique name

After the third approach implemented as examples/abc/ project, if there are still conflicts, the above message is given as well. And what you need to do to resolve this conflict is to rename the folders or the files to make the names unique.

Notes for path-related macros

As project-based examples might affect path-related macros, there are two macros we should pay attention to: module_path!() and file!(). As project-based examples should be treated like a unique project, the outputs of these two macros should refer to the path of the example instead of root project path. For example, if we have a abc example project for root project my_project and the main function in examples/abc/src/main.rs contains println!("{} {}", module_path!(), file!()), it should print abc src/main.rs rather than abc examples/abc/src/main.rs.

There are plenty of macros like line!() and column!() whose value after being parsed are related to the line number and column number. Fortunately, the parsing logic of these macros above are not effected if being written into project-based example codes.

Conclusion

By including cargo new --example and enhancing cargo run --example etc., this RFC provides a more convenient way for you to write examples for your project.

Reference-level explanation

Enhanced --example for run, test and bench

To implement this feature, firstly we enlarges --example search scope. We’ve already got examples/*.rs and examples/**/main.rs searched by cargo every time it tries to run, test or bench an example. But to implement examples/**/ project we should detect if this folder contains a valid cargo project. This could be done by detecting Cargo.toml the same way we detect when trying to run cargo run in an invalid folder for cargo projects.

When operating example project, we compile the whole example project as what we do on the root project. We could share the target folder with it then incremental compiling can be enabled to save time. The target of example projects are stored just like what we did for file-based examples.

Running, testing and benching project-based examples should be treated the same as what Rust already do for single-file examples. Same toolchain should be applied to them by default.

Operating example projects with --example <NAME> is just like a syntax sugar in programming, which help us save time when executing examples.

For path-related macros, refer to the paragraph ‘Notes for path-related macros’.

cargo new --example <NAME>

Another feature cargo new --example <NAME> requires creating a new argument. If argument --example <NAME> is found, we search for if this folder exists, if so we fire an error like error: example NAME already exists; and if not, a folder is created in examples and a cargo project for runnables where there is main.rs is built inside.

Note that cargo new --example <NAME> can only be executed inside a cargo project. Execution with no cargo project detected should be denied with an error message like error: no cargo project found to create an example or similiar messages.

Additionally, what cargo new --example <NAME> differs from cargo new is only that the former command, as is mentioned above, generates a dependency on the root project using ../.. references. Despite this the procedure should be the same, including what we should write into the .gitignore file for it, so it should be possible to cd into it and run cargo run directly.

Drawbacks

If we implement project-based examples into cargo, it might be backward incompatible if we want to use --example <NAME> in other ways in the future.

Rationale and alternatives

There could be another way to implement project-based examples by introducing cargo example subcommand. However by doing this we must change our way to run examples now by cargo run --example <NAME> which is already widely accepted by rust community.

On .gitignore, it could be a good idea to rewrite the .gitignore file in the root changing the Cargo.lock to /Cargo.lock to avoid it search for every cargo locks nestedly thus a !Cargo.lock is not needed in the example project path. However it would totally change the way how we write .gitignore for Rust, thus this alternative is remained for the Rust authors to judge.

Prior art

None by now.

Unresolved questions

  • Is there any more graceful way to replace the my_project = { path = "../.." } in Cargo.toml file for all example projects?
  • Will it be useful if we provided an approach to test or bench all examples at one time?
  • Should path-related macro file!() refer the path related to the root project path?

#2

This already exists, here’s an example library that has an example in the folder examples/websocket. Adding an extra file examples/websocket.rs fails with:

> cargo run --example websocket
Caused by:
  found duplicate example name websocket, but all example targets must have a unique name

I agree with the premise of allowing foldered examples to provide their own Cargo.toml instead of needing to shove all the examples dependencies into the parent dev-dependencies. One thing to keep in mind is whether workspaces can be leveraged for this somehow, with a Cargo.toml per example you’re basically at the point where examples are sub-binary-crates in the workspace of the main project with some special support for cargo run --example <foo> to be able to target them.


#3

Thanks! This type of writing an example is not easy to discover by me :slight_smile: I’ve modified the doc to make things more proper.

Thank you for reminding me of this. I believe the cargo run command with --example is just like a syntax sugar in programming, which can help us save time from typing cd .. etc.


#4

Shall example’s Cargo.lock be in Git for libraries?

Shall .gitignore setup done by cargo new/cargo init specially consider examples’ Cargo.locks into accout?


#5

Fwiw, .gitignore is indeed supported in nested folders.


#6

As I know now, .gitignore files can be put into subdirectories. When I put a .gitignore into a folder abc and write something in it, it effects and does filter the files it was with. I think every project-based example as folders in examples are unique projects and can be run separately if cd into it and do cargo run. Then the .gitignore files should be treated the same as unique projects. Thus, there should be .gitignores created at examples/abc/.gitignore which effects within examples/abc/. The way it works are separate from the root project, but the same way as it.

Thanks for noticing me.


#7

Examples’s .gitignore should contain !Cargo.lock then to override Cargo.lock ignore in parent .gitignore.


#8

Thanks! I have ignored this. I’ll test it out.


Updates: after my test, .gitignore will search for nested Cargo.lock files when written a Cargo.lock in it, but will only search for that in root project path if we write /Cargo.lock. And if I write !Cargo.lock into nested .gitignore, it works perfectly what we’ve looked forward to. I’ve write it into my article.


#9

An example where the folder structure is already used is stdweb; the reason it’s relevant there is that there is a per-project static/ folder that is most easily used that way. (And per-example dependencies, of course).

I think it could be helpful in the embedded-hal area as well. There, examples often rely on dev-dependencies rather than having their own dependencies neatly handled per-example (eg. madgwick dependency in f3).


#10

Have you considered using Cargo workspaces? If you are inside your project, it should be something like:

  1. cargo new --bin myexample
  2. Edit Cargo.toml to add:
    [workspace]
    members = ["myexample"]
    
  3. cargo run -p myexample

You can also use -p with test and bench as well.


#11

That’s what stdweb is using so far (just pooled in the examples folder). To me, cargo run -p examples/myexample feels much less supported than cargo run --example myexample.

The proposal should evaluate what can already be done with workspaces and show how (or that) integration into --examples provides benefits over it as it progresses.


#12

If filters (like --example and --bin) worked across the entire workspace, would that help? It would change my example above to be cargo run --bin myexample. Or if you really want to use --example, then you can place the example code into examples/myexample/examples which is a little redundant. There is a recently open issue for this (https://github.com/rust-lang/cargo/issues/5819), which offhand seems reasonable.


#13

Thanks for mentioning! I’ll add it.


#14

Thank you for reminding me!

Thanks for reminding me. This easier way of building examples should concern on embedded developing as well.


#15

Thank you everyone for commenting! I’ve sent a RFC pull request for this: https://github.com/rust-lang/rfcs/pull/2517


#16

I haven’t had a chance to read the RFC yet, but thanks so much for working on this! There are a lot of features like this (especially in cargo) that are clearly good ideas, but that are blocked on actually sketching out all their details. I’ve wanted to tackle this one for a year and a half, but never had time. Thanks!