# Private nested Cargo packages **URL:** \ **Category:** cargo\ **Created:** [June 21, 2023, 10:42pm UTC](https://internals.rust-lang.org/t/private-nested-cargo-packages/19042 "2023-06-21T22:42:17Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![kpreid](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kpreid/32/8484_2.png) [@kpreid](https://internals.rust-lang.org/u/kpreid)\ **Post date:** [June 21, 2023, 10:42pm UTC](https://internals.rust-lang.org/t/private-nested-cargo-packages/19042/1 "2023-06-21T22:42:17Z")
One of the current pain points of using Cargo is that sometimes you have to split your code into multiple packages merely for implementation reasons — for proc-macros, build-dependencies, or just smaller compilation units. These multiple packages then have to be published separately, if their dependent is being published at all. ~~This is part (but definitely not all) of the motivation for proposals like [Packages as optional namespaces](https://github.com/rust-lang/rfcs/pull/3243).~~ I have thought of what might be a simpler idea to solve the common cases like proc-macros: **what if published packages could contain `path` dependencies on packages inside them?** The visible changes to Cargo would be: - A dependency that has `path =` without `version =` would be allowed when publishing, provided the path is a relative path without any upward traversal (`../`) that leads outside of the published package. - Such packages would be included in the `.crate` archive when publishing, rather than automatically excluded as all sub-packages currently are. - Probably, `cargo install` should be taught to allow installing binaries from any/all sub-packages packages. If this is done, then such sub-packages should be allowed to depend on the parent package (so that primarily-library packages can include CLI tools). These sub-packages would be ~~“anonymous”~~ private as far as the rest of the dependency graph is concerned — given the no-upward-traversal rule, no published package outside of the parent package will ever be able to depend on them. (If I understand correctly, this is how `path` already works — each path is a unique package identity.) Characteristics of this feature: - A trait-defining package can simply contain its associated proc-macro derive crate. - Dependency resolution between nested packages will always use the matching version of the code — there is no need to think about what version numbering strategy to use. (For an example of the problems with the status quo, using identical version numbers can result in glitches where `foo 1.2.0` is used with `foo-derive 1.3.0` and fails because `foo-derive` assumes wrong things about `foo`'s trait declarations.) In fact, sub-packages ideally would have no version numbers, since the numbers are not used, but that would be a separate proposal. - Does not provide extensible namespacing — you can't publish an additional child package separately, nor can anyone depend on it, so it is not for “plugin” packages, only for providing a single “frontend” package with many internal parts. Now that I've thought of it, this seems like a well-defined solution to several problems, that doesn't require a lot of new engineering. Are there complications I haven't thought of? Has anything like this been proposed before? ---
**Author:** ![jhpratt](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/jhpratt/32/11640_2.png) [@jhpratt](https://internals.rust-lang.org/u/jhpratt)\ **Post date:** [June 21, 2023, 10:50pm UTC](https://internals.rust-lang.org/t/private-nested-cargo-packages/19042/2 "2023-06-21T22:50:14Z")
This is even simpler and nearly as powerful as what I have envisioned in the past. 100% support. ---
**Author:** ![kpreid](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kpreid/32/8484_2.png) [@kpreid](https://internals.rust-lang.org/u/kpreid)\ **Post date:** [June 21, 2023, 10:56pm UTC](https://internals.rust-lang.org/t/private-nested-cargo-packages/19042/3 "2023-06-21T22:56:53Z")
While constructing this proposal, I also thought: what about allowing a package to have multiple `lib` targets? The problems with that are that it would need new manifest syntax and new CLI, and it would not automatically allow specifying dependencies among the crates (or different sets of dependencies on external packages). ---
**Author:** ![Nemo157](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/nemo157/32/11585_2.png) [@Nemo157](https://internals.rust-lang.org/u/Nemo157)\ **Post date:** [June 21, 2023, 10:59pm UTC](https://internals.rust-lang.org/t/private-nested-cargo-packages/19042/4 "2023-06-21T22:59:15Z")
Having multiple `lib` crates within a single package is how I've always assumed something like this would be done. That might also provide impetus to improve handling of crate-specific dependencies, making simultaneous `lib`/`bin` crates more viable. ---
**Author:** ![kpreid](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kpreid/32/8484_2.png) [@kpreid](https://internals.rust-lang.org/u/kpreid)\ **Post date:** [June 21, 2023, 11:05pm UTC](https://internals.rust-lang.org/t/private-nested-cargo-packages/19042/5 "2023-06-21T23:05:02Z")
I agree that multiple `lib` with independent dependencies is a more semantically sensible solution (it makes the notion of “package” more flexible in logical ways), but it would require a lot more additions to make work. I hope that nested packages will be _easy to implement_, yet not create future complexity that needs to be worked around, since it is using existing manifests and build infrastructure, and is invisible to package users. It is compatible with having multiple `lib` support in the future, and users could migrate once that feature is available (but of course there is still “which should I choose” semantic complexity). ---
**Author:** ![epage](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/epage/32/3171_2.png) [@epage](https://internals.rust-lang.org/u/epage)\ **Post date:** [June 22, 2023, 12:09am UTC](https://internals.rust-lang.org/t/private-nested-cargo-packages/19042/6 "2023-06-22T00:09:25Z")
> [@kpreid](#): > > This is part (but definitely not all) of the motivation for proposals like [Packages as optional namespaces](https://github.com/rust-lang/rfcs/pull/3243). imo these use cases are not on my mind at all when working on that proposal > [@kpreid](#): > > Probably, `cargo install` should be taught to allow installing binaries from any/all sub-packages packages. If this is done, then such sub-packages should be allowed to depend on the parent package (so that primarily-library packages can include CLI tools). imo if these are anonymous, then they are truly anonymous and their content can only be accessed by the root package re-exporting them. In my mind this means bins would not be accessible. ---
**Author:** ![epage](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/epage/32/3171_2.png) [@epage](https://internals.rust-lang.org/u/epage)\ **Post date:** [June 22, 2023, 12:10am UTC](https://internals.rust-lang.org/t/private-nested-cargo-packages/19042/7 "2023-06-22T00:10:48Z")
A dangerous question: should we slacken any rules for these anonymous packages, be it orphan rules or item visibility? I think there was another variant of this idea in the past for saying a group of crates could break some of these kind of rules and so I figure this would be asked and that we should have an answer early. ---
**Author:** ![epage](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/epage/32/3171_2.png) [@epage](https://internals.rust-lang.org/u/epage)\ **Post date:** [June 22, 2023, 12:12am UTC](https://internals.rust-lang.org/t/private-nested-cargo-packages/19042/8 "2023-06-22T00:12:53Z")
For the local dev case, are these "anonymous" packages normal packages? I figure they would need to be in some way for being able to have tests on them. Preventing tests in these scenarios seems like it'd be pretty restrictive. ---
**Author:** ![kpreid](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kpreid/32/8484_2.png) [@kpreid](https://internals.rust-lang.org/u/kpreid)\ **Post date:** [June 22, 2023, 12:27am UTC](https://internals.rust-lang.org/t/private-nested-cargo-packages/19042/9 "2023-06-22T00:27:40Z")
> [@epage](#): > > should we slacken any rules for these anonymous packages, be it orphan rules or item visibility? That seems like something that, if desired, could be added later, as a separate design and implementation. That of course has the risk “we would like to do this, but because we didn't think about it early enough, we can't”, but there is an argument for _not_ choosing to provide such incompatible features: I think **it should be possible to turn what was a sub-package into a published package of its own** (e.g. "this utility library has baked for a while and is now something others might find useful"). Given that criterion, sub-packages should not be made _intrinsically_ rather than optionally different, and therefore a package designed for "status quo but with anonymous sub-packages" should be compatible with a later future "and you can do more things with sub-packages if you want". > [@epage](#): > > For the local dev case, are these "anonymous" packages normal packages? I figure they would need to be in some way for being able to have tests on them. Preventing tests in these scenarios seems like it'd be pretty restrictive. Local development doesn't change, because they're _just_ packages referred to with `path` dependencies like you can write today. The only difference is that you are now allowed to publish a package that previously would have been prohibited by the rule that published packages can't have `path`-only dependencies. So, yes, those sub-packages can have `#[test]`s, `tests/`, and even test subpackages of their own. I call them “anonymous” only because other `crates.io` packages can't name them, which is a key difference between this and namespaced package proposals. ---
**Author:** ![kpreid](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kpreid/32/8484_2.png) [@kpreid](https://internals.rust-lang.org/u/kpreid)\ **Post date:** [June 22, 2023, 12:38am UTC](https://internals.rust-lang.org/t/private-nested-cargo-packages/19042/10 "2023-06-22T00:38:14Z")
> [@epage](#): > > imo if these are anonymous, then they are truly anonymous and their content can only be accessed by the root package re-exporting them. In my mind this means bins would not be accessible. There's certainly a reasonable encapsulation logic to that, but being able to publish a CLI tool _with different dependencies_ as part of your package is an important use case within the overall “I need to publish many packages but I would really rather publish one” picture. Also, if you wanted to not publish a binary you could just not publish a binary ——— wait, that will not be true once [artifact dependencies](https://github.com/rust-lang/rfcs/pull/3028) are implemented and a package can depend on _executing_ a binary from another package, and therefore a binary might be an implementation detail. Okay, that's definitely a question with an unclear answer. ---
**Author:** ![BD103](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/bd103/32/12245_2.png) [@BD103](https://internals.rust-lang.org/u/BD103)\ **Post date:** [June 22, 2023, 2:35am UTC](https://internals.rust-lang.org/t/private-nested-cargo-packages/19042/11 "2023-06-22T02:35:24Z")
This sounds like an awesome proposal. I would like to raise two points that may be tricky: - Cargo features, specifically using the `package-name/feature-name` syntax. The anonymous nested packages would still need names, even if inaccessible from crates. - [Dependency overrides](https://doc.rust-lang.org/cargo/reference/overriding-dependencies.html) also have this requirement of a set name. Either way, this sounds awesome! 🙂 ---
**Author:** ![kpreid](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kpreid/32/8484_2.png) [@kpreid](https://internals.rust-lang.org/u/kpreid)\ **Post date:** [June 22, 2023, 3:01am UTC](https://internals.rust-lang.org/t/private-nested-cargo-packages/19042/12 "2023-06-22T03:01:03Z")
> [@BD103](#): > > The anonymous nested packages would still need names, even if inaccessible from crates. They would still have package names in the same way local unpublished `path`-dependency packages do — they just aren't _global_ names in the [crates.io](http://crates.io) registry. (I'm thinking that I really should not have used the word “anonymous” in the title, given this is a recurring confusion.) Even if we decide that the packages in fact have no `package.name` in the `Cargo.toml`, you'd still be able to refer to their features by the same name you entered in the main package's `[dependencies]` table. ---
**Author:** ![kpreid](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kpreid/32/8484_2.png) [@kpreid](https://internals.rust-lang.org/u/kpreid)\ **Post date:** [June 22, 2023, 3:03am UTC](https://internals.rust-lang.org/t/private-nested-cargo-packages/19042/13 "2023-06-22T03:03:16Z")
To reduce confusion, I have replaced the word “anonymous” with “private” in the thread title and original description. ---
**Author:** ![epage](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/epage/32/3171_2.png) [@epage](https://internals.rust-lang.org/u/epage)\ **Post date:** [June 22, 2023, 2:01pm UTC](https://internals.rust-lang.org/t/private-nested-cargo-packages/19042/14 "2023-06-22T14:01:49Z")
> [@kpreid](#): > > There's certainly a reasonable encapsulation logic to that, but being able to publish a CLI tool _with different dependencies_ as part of your package is an important use case within the overall “I need to publish many packages but I would really rather publish one” picture. Could you help me understand what you are referring to. I believe the point in question is whether `cargo install` should be special cased to recurse into the "private" packages to discover `[[bin]]`s to install. How does a "CLI tool with different dependencies" fit it (e.g. "different" than what?). ---
**Author:** ![kpreid](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kpreid/32/8484_2.png) [@kpreid](https://internals.rust-lang.org/u/kpreid)\ **Post date:** [June 22, 2023, 2:13pm UTC](https://internals.rust-lang.org/t/private-nested-cargo-packages/19042/15 "2023-06-22T14:13:02Z")
> [@epage](#): > > How does a "CLI tool with different dependencies" fit it (e.g. "different" than what?). Different than the parent library. Projects often have a library and an associated tool binary (e.g. `wasm-bindgen`, `criterion`, `rerun`, probably many others I'm not remembering at this moment). The tool often needs additional dependencies that the library does not (argument parsing, file formats, GUI or TUI). Currently, projects in this position have the choice between 1. publishing a separate Cargo package (`wasm-bindgen-cli`), 2. using a `feature` that must be activated for the binary target, or 3. having excess dependencies for the library and they usually choose the first option, at least eventually. Allowing installed binaries to be discovered from sub-packages would allow them to publish their binary as part of their main package, which could make for a better user experience: `cargo install wasm-bindgen` instead of `cargo install wasm-bindgen-cli`, so there's no extra package to learn about from documentation, and the command name installed now matches the package they installed. ---
**Author:** ![Neutron3529](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/neutron3529/32/6976_2.png) [@Neutron3529](https://internals.rust-lang.org/u/Neutron3529)\ **Post date:** [June 22, 2023, 2:15pm UTC](https://internals.rust-lang.org/t/private-nested-cargo-packages/19042/16 "2023-06-22T14:15:14Z")
> [@kpreid](#): > > Nested packages are _always_ versioned in lockstep with their siblings and parent — there is no need to think about what versioning strategy to use. (For example, using identical version numbers can result in glitches where `foo 1.2.0` is used with `foo-derive 1.3.0` and fails because `foo-derive` assumes wrong things about `foo`'s trait declarations.) Is this OK? Suppose `Foo` have two sub-crate, `Foo-impl` and `Foo-init`. When published, both of their version locks on `1.0.0` Then you found a bug in `Foo-impl`, with fixing the bug, version of `Foo-impl` becomes `1.0.1` now. And then the question appears, you make ABSOLUTELY NO changes for `Foo-init`, but you have to bump the version to `1.0.1`, too. Is it desired? If tens of sub-crate is used, Should we bump all of the version even we just fix a small bug? ---
**Author:** ![kpreid](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kpreid/32/8484_2.png) [@kpreid](https://internals.rust-lang.org/u/kpreid)\ **Post date:** [June 22, 2023, 2:41pm UTC](https://internals.rust-lang.org/t/private-nested-cargo-packages/19042/17 "2023-06-22T14:41:05Z")
Sorry, by “versioned in lockstep” I meant the actual outcome of dependency resolution, not any requirement on the packages' own version numbers. Because the set of packages is published as one parent package, one `.crate` archive file, you are _physically_ required to publish another such package that contains all of the sub-packages in order to update any one of them. The version numbers written in the sub-packages are irrelevant as long as they are valid, because they are never used in dependency resolution, just like current local `path` dependencies don't use version numbers. (Perhaps it should be permissible to omit `version`s from such packages. If so, then local workspace `path` dependencies should also not require `version`s.) ---
**Author:** ![epage](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/epage/32/3171_2.png) [@epage](https://internals.rust-lang.org/u/epage)\ **Post date:** [June 22, 2023, 2:43pm UTC](https://internals.rust-lang.org/t/private-nested-cargo-packages/19042/18 "2023-06-22T14:43:02Z")
> [@kpreid](#): > > Projects often have a library and an associated tool binary (e.g. `wasm-bindgen`, `criterion`, `rerun`, probably many others I'm not remembering at this moment). The tool often needs additional dependencies that the library does not (argument parsing, file formats, GUI or TUI). Currently, projects in this position have the choice between FYI [Cargo target features by ehuss · Pull Request #3374 · rust-lang/rfcs · GitHub](https://github.com/rust-lang/rfcs/pull/3374) is another solution for improving this situation (though I personally disagree with it). imo keeping this focused on the core problem which is elegantly solved with your solution, rather than adding the CLI case and dealing with the intricacies of what binaries can be included or not, is more likely to make thus proposal for further, faster. ---
**Author:** ![kpreid](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kpreid/32/8484_2.png) [@kpreid](https://internals.rust-lang.org/u/kpreid)\ **Post date:** [June 22, 2023, 2:47pm UTC](https://internals.rust-lang.org/t/private-nested-cargo-packages/19042/19 "2023-06-22T14:47:10Z")
We do need to _define the behavior_ but not doing anything with child packages and saying the parent is the only “public API”, for now, does seem like the simpler and default choice. ---
**Author:** ![epage](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/epage/32/3171_2.png) [@epage](https://internals.rust-lang.org/u/epage)\ **Post date:** [June 22, 2023, 2:47pm UTC](https://internals.rust-lang.org/t/private-nested-cargo-packages/19042/20 "2023-06-22T14:47:17Z")
> [@kpreid](#): > > (Perhaps it should be permissible to omit `version`s from such packages. If so, then local workspace `path` dependencies should also not require `version`s.) The way I would write up a Pre-RFC for this would be - default the version field to `0.0.0` - Only do this for path dependencies without a version requirement that are nested under the current package - A version requirement in this context is meaningless - I believe a normal/build dependency has to have a version requirement to be published today, so there would be no ambiguity - What to do with dev-dependencies is unclear The original proposal mentioned keying off of `package=` not being present but I'm not sure why that would be meaningful and would likely cause a change in behavior (most dependencies likely don't set `package=`) [Next page](https://internals.rust-lang.org/t/private-nested-cargo-packages/19042.md?page=2)