One of the current pain points of using Cargo is that sometimes you have to split your code into multiple packages merely for implementation reasons — for proc-macros, build-dependencies, or just smaller compilation units. These multiple packages then have to be published separately, if their dependent is being published at all. This is part (but definitely not all) of the motivation for proposals like Packages as optional namespaces.
I have thought of what might be a simpler idea to solve the common cases like proc-macros: what if published packages could contain path dependencies on packages inside them?
The visible changes to Cargo would be:
A dependency that has path = without version = would be allowed when publishing, provided the path is a relative path without any upward traversal (../) that leads outside of the published package.
Such packages would be included in the .crate archive when publishing, rather than automatically excluded as all sub-packages currently are.
Probably, cargo install should be taught to allow installing binaries from any/all sub-packages packages. If this is done, then such sub-packages should be allowed to depend on the parent package (so that primarily-library packages can include CLI tools).
These sub-packages would be “anonymous” private as far as the rest of the dependency graph is concerned — given the no-upward-traversal rule, no published package outside of the parent package will ever be able to depend on them. (If I understand correctly, this is how path already works — each path is a unique package identity.)
Characteristics of this feature:
A trait-defining package can simply contain its associated proc-macro derive crate.
Dependency resolution between nested packages will always use the matching version of the code — there is no need to think about what version numbering strategy to use. (For an example of the problems with the status quo, using identical version numbers can result in glitches where foo 1.2.0 is used with foo-derive 1.3.0 and fails because foo-derive assumes wrong things about foo's trait declarations.) In fact, sub-packages ideally would have no version numbers, since the numbers are not used, but that would be a separate proposal.
Does not provide extensible namespacing — you can't publish an additional child package separately, nor can anyone depend on it, so it is not for “plugin” packages, only for providing a single “frontend” package with many internal parts.
Now that I've thought of it, this seems like a well-defined solution to several problems, that doesn't require a lot of new engineering. Are there complications I haven't thought of? Has anything like this been proposed before?
While constructing this proposal, I also thought: what about allowing a package to have multiple lib targets? The problems with that are that it would need new manifest syntax and new CLI, and it would not automatically allow specifying dependencies among the crates (or different sets of dependencies on external packages).
Having multiple lib crates within a single package is how I've always assumed something like this would be done. That might also provide impetus to improve handling of crate-specific dependencies, making simultaneous lib/bin crates more viable.
I agree that multiple lib with independent dependencies is a more semantically sensible solution (it makes the notion of “package” more flexible in logical ways), but it would require a lot more additions to make work. I hope that nested packages will be easy to implement, yet not create future complexity that needs to be worked around, since it is using existing manifests and build infrastructure, and is invisible to package users. It is compatible with having multiple lib support in the future, and users could migrate once that feature is available (but of course there is still “which should I choose” semantic complexity).
imo these use cases are not on my mind at all when working on that proposal
imo if these are anonymous, then they are truly anonymous and their content can only be accessed by the root package re-exporting them. In my mind this means bins would not be accessible.
A dangerous question: should we slacken any rules for these anonymous packages, be it orphan rules or item visibility?
I think there was another variant of this idea in the past for saying a group of crates could break some of these kind of rules and so I figure this would be asked and that we should have an answer early.
For the local dev case, are these "anonymous" packages normal packages? I figure they would need to be in some way for being able to have tests on them. Preventing tests in these scenarios seems like it'd be pretty restrictive.
That seems like something that, if desired, could be added later, as a separate design and implementation.
That of course has the risk “we would like to do this, but because we didn't think about it early enough, we can't”, but there is an argument for not choosing to provide such incompatible features: I think it should be possible to turn what was a sub-package into a published package of its own (e.g. "this utility library has baked for a while and is now something others might find useful").
Given that criterion, sub-packages should not be made intrinsically rather than optionally different, and therefore a package designed for "status quo but with anonymous sub-packages" should be compatible with a later future "and you can do more things with sub-packages if you want".
Local development doesn't change, because they're just packages referred to with path dependencies like you can write today. The only difference is that you are now allowed to publish a package that previously would have been prohibited by the rule that published packages can't have path-only dependencies. So, yes, those sub-packages can have #[test]s, tests/, and even test subpackages of their own.
I call them “anonymous” only because other crates.io packages can't name them, which is a key difference between this and namespaced package proposals.
There's certainly a reasonable encapsulation logic to that, but being able to publish a CLI tool with different dependencies as part of your package is an important use case within the overall “I need to publish many packages but I would really rather publish one” picture.
Also, if you wanted to not publish a binary you could just not publish a binary ——— wait, that will not be true once artifact dependencies are implemented and a package can depend on executing a binary from another package, and therefore a binary might be an implementation detail.
Okay, that's definitely a question with an unclear answer.
This sounds like an awesome proposal. I would like to raise two points that may be tricky:
Cargo features, specifically using the package-name/feature-name syntax. The anonymous nested packages would still need names, even if inaccessible from crates.
They would still have package names in the same way local unpublished path-dependency packages do — they just aren't global names in the crates.io registry. (I'm thinking that I really should not have used the word “anonymous” in the title, given this is a recurring confusion.)
Even if we decide that the packages in fact have no package.name in the Cargo.toml, you'd still be able to refer to their features by the same name you entered in the main package's [dependencies] table.
Could you help me understand what you are referring to. I believe the point in question is whether cargo install should be special cased to recurse into the "private" packages to discover [[bin]]s to install. How does a "CLI tool with different dependencies" fit it (e.g. "different" than what?).
Projects often have a library and an associated tool binary (e.g. wasm-bindgen, criterion, rerun, probably many others I'm not remembering at this moment). The tool often needs additional dependencies that the library does not (argument parsing, file formats, GUI or TUI). Currently, projects in this position have the choice between
publishing a separate Cargo package (wasm-bindgen-cli),
using a feature that must be activated for the binary target, or
having excess dependencies for the library
and they usually choose the first option, at least eventually.
Allowing installed binaries to be discovered from sub-packages would allow them to publish their binary as part of their main package, which could make for a better user experience: cargo install wasm-bindgen instead of cargo install wasm-bindgen-cli, so there's no extra package to learn about from documentation, and the command name installed now matches the package they installed.
Sorry, by “versioned in lockstep” I meant the actual outcome of dependency resolution, not any requirement on the packages' own version numbers.
Because the set of packages is published as one parent package, one .crate archive file, you are physically required to publish another such package that contains all of the sub-packages in order to update any one of them. The version numbers written in the sub-packages are irrelevant as long as they are valid, because they are never used in dependency resolution, just like current local path dependencies don't use version numbers.
(Perhaps it should be permissible to omit versions from such packages. If so, then local workspace path dependencies should also not require versions.)
imo keeping this focused on the core problem which is elegantly solved with your solution, rather than adding the CLI case and dealing with the intricacies of what binaries can be included or not, is more likely to make thus proposal for further, faster.
We do need to define the behavior but not doing anything with child packages and saying the parent is the only “public API”, for now, does seem like the simpler and default choice.
The way I would write up a Pre-RFC for this would be
default the version field to 0.0.0
Only do this for path dependencies without a version requirement that are nested under the current package
A version requirement in this context is meaningless
I believe a normal/build dependency has to have a version requirement to be published today, so there would be no ambiguity
What to do with dev-dependencies is unclear
The original proposal mentioned keying off of package= not being present but I'm not sure why that would be meaningful and would likely cause a change in behavior (most dependencies likely don't set package=)
