Multiple libraries in a cargo project

[glandium]

A related issue that I have is that it’s not possible to choose the crate type through configuration. That is, I have a crate that I’d like to be able to build either as a cdylib or a staticlib. The best I can do at the moment is to generate a Cargo.toml in some way… not really appealing. Or create two sub-crates, one for the cdylib and one for the staticlib, but then I’ll end up with both being built when I only want one.

[CAD97]

Whatever solution is used, it would be nice to support multiple libraries coming from a workspace as well. In existing projects using workspaces or even future projects that need more individual control over the sublibraries, it would be nice to support bundling workspace path dependencies as sublibraries.

[withoutboats]

I don’t understand the situation you’re describing. If you have multiple packages in a workspace, you can just depend on them: there’s no need for one to be a “sublibrary,” and it wouldn’t make sense for it to be.

[matklad]

Previous RFC in this space: https://github.com/rust-lang/rfcs/pull/2224

[matklad]

I definitely agree that we should solve private libraries problem! It is an unfortunate restriction that, as soon as you split your code into crates, the split itself becomes a public API. As for the approach to solve this, I kind of like the “private path dependencies” of the postponed RFC more. In a nutshell, the approach is roughly “if you omit version field from the [package] section of workspace member, it becomes a “private dependency” and is packaged into .crate file with your main package”

In the rest of the post, I’d like to just discuss package/crate separation in detail.

Cargo has a concept of a “package”, and I’ve always had a love-hate relationship with it. In reality, all Rust code is a set of compilation units/crates, which form a DAG. The “package” concept is a Cargo-specific addition, which, in theory, could have not existed.

In practice a package as a way to add “supporting” crates to the main library crate is really, really useful. It feels very appropriate that examples, tests and build scripts are implemented as special-cased built-ins.

One of the more problematic aspects of the package concept is dependency management. Because package has at most single library crate, depending on package, and not on the individual crate, works perfect. However, the fact that you specify dependencies per-package is less than ideal: a common requests is to add binary specific dependencies.

In broad stokes, it seems that the way forward is either to stick with the “package as a unit of dependency” model and make package creation more light weight (postponed RFC), or to expose the underling crate concept more (current proposal).

Without examining the tradeoffs to closely, I’d expect the “lightweight package” route more promising, for two reasons:

• It reuses an existing mechanism for specifying dependencies, which makes Cargo’s interface simpler.
• It does not abandon the existing benefits of the packages (you can have dedicated tests and build-scripts for the private libraries)
• It makes transition from “private library” to “stand-alone package” more straightforward.

[kornel]

The problem is very real, but I think you may be looking at the solution from a wrong perspective. There already is a way to have multiple libraries per project, and have excellent internal boundaries: workspaces.

So the problem is not that one crate can’t have multiple libraries, but that crates.io doesn’t support workspaces.

So I think crates.io needs to be extended to allow publishing multiple crates per package.

---

Cargo has also moved away from from recommending lib and binaries sharing a crate in favor of separate crates, because it doesn’t support separate dependencies, features, etc. for individual targets within a crate. The same problems would affect multiple libraries.

Out of 1600 most popular (>300 downloads per month) crates, 240 (~15%) have a repository with more than one crate published to crates.io. 7% have 3 or more separately published crates per repo.

[djc]

I agree with @kornel that the conceptually more simple approach lies the other way: rather than introducing another layer of complexity, we could instead strengthen the abstractions we already have and fix their leaks. From the list of reasons for wanting to have multiple libraries per package, the latter three are directly fixed by having things in different crates.

While the initial post seems pretty extensive in terms of its discussion of a potential solution, I found it a bit light in terms of talking about the problems or use cases that need to be addressed. The second post discusses a concrete use case, but seems to mainly revolve around the proc-macro issue.

As the author of a somewhat popular custom derive crate, I definitely feel the pain about not being able to distribute procedural macros as part of a larger library. I also agree with @bascule that publishing a number of libraries from a workspace is painful today. However, I don’t think sublibraries are the best way to address these problems.

[withoutboats]

@matklad gets pretty good at the heart of the issue: the relationship between packages and crates:

• crate: a crate is a unit of compilation for a Rust compiler
• package: a package is a set of crates with a single entry in a cargo registry’s index

The relationship between these is often muddled. @matklad is right to say that that we have a love/hate relationship with entire “package” idea, but its built into cargo at this point and I can’t imagine moving away from it. I think that the current relationship between packages and library crates is a contributor to the muddlement: because a package can only contain 1 library, which is a crate, and because the solution is to create multiple packages and compile them in a workspace, the idea that a package is a crate and if you want to have multiple crates you use a workspace is sort of the most obvious interpretation.

Multiple workspaces are about sharing a version resolution among multiple packages, not multiple crates. And I think workspaces are a heavyweight solution for the problem we’re coming at here. The downsides of using a workspace for an internal library is:

1. That library and its public API become part of your public API that you are responsible for semantically versioning and publishing to crates.io. This is a big maintenance burden.
2. Setting it up initially takes more work: you have to create a proper manifest file for your new project at least, possibly deal with the [workspace] section.
3. Now you have to maintain dependency lists between each manifest, which will diverge with time probably, leading to annoyances like “I have to copy my dependency on foobar_baz from Cargo.toml to src/quuxlib/Cargo.toml to get this to compile.”

All of these are burdensome for an internal division, and in my experience what most often happens is that users just don’t break up their crate, even when they want to, because its not worth the cost. That’s very unfortunate: we’ve made it too expensive to do something good.

In contrast, this proposal is designed around making it extremely cheap to create a new crate, essentially as cheap as creating a new module:

1. Create a file at src/lib/foobar/lib.rs. You have a new crate!
2. The rest of your project can depend on it immediately, no additions to your Cargo.toml.
3. That crate has access to all of the dependencies of the package, you don’t need to worry about “exposing” them to it in its manifest.
4. If you need to, eventually you can make it more complicated by creating its own crate-level manifest for specifying its other dependencies.

Diving into more specifics:

• It does not abandon the existing benefits of the packages (you can have dedicated tests and build-scripts for the private libraries)

The problem of build scripts for separate crates within a package I think should be solved regardless of this change. I’m not really clear on the benefits of dedicated tests, is it just an organizational thing?

• It reuses an existing mechanism for specifying dependencies, which makes Cargo’s interface simpler.
• It makes transition from “private library” to “stand-alone package” more straightforward.

I think this is actually not such a hard distinction in practice. I’m thinking possibly that a sublibrary’s manifest would be a Cargo.toml with no [package] section. In that case, all you’d need to do to transition from a sublibrary to a separate package is add [package] to your toml, this is only slightly more heavyweight than adding a version number, at the benefit of making creating the sublibrary initially more lightweight.

While the initial post seems pretty extensive in terms of its discussion of a potential solution, I found it a bit light in terms of talking about the problems or use cases that need to be addressed.

I can go into some concrete examples:

1. I’ve been working on implementing signed registries in my spare time, which involves expanding the sha256 utility into a whole crypto submodule with sha512 and ed25519 primitives as well. This means adding several dependencies and creating a toplevel crypto module, which is actually totally isolated from the rest of cargo. I’d like to make this its own crate to enforce that division, but for the reasons I enumerated above I did not.
2. The original seed of this idea actually came from a conversation with Niko. He was complaining that he wanted some “cratelike” boundary inside of his crate, so that he could easily control visibility by saying pub(crate) but actually meaning this particular module, and so on. I argued that what he actually wanted was not a new language feature, but just for subcrates to be as lightweight to make as modules are.
3. Looking over my own projects, I’ve never actually wanted workspaces at all (and I’ve always found creating them a little confusing… I’m never certain if I’ve actually made it work at first). One of my largest personal projects, cargonauts, was divided into seven crates, none of which I actually wanted to be separate packages, it was just an internal organizational aspect of the project.

[matklad]

I’m not really clear on the benefits of dedicated tests, is it just an organizational thing?

Yeah. I think that if you have the reason to split a part of the library into a separate crate, you might want to split tests as well? For projects which I am not publishing to crates.io, I use workspace heavily (libsyntax2 is seven crates), and each crate typically has dedicated tests. I also often make use of cd crates/some_crate && hack && cargo test && hack && cargo test workflow.

[withoutboats]

We could teach cargo about tests for specific crates as well, e.g. having it treat src/lib/$name/tests specially or whatever.

[matklad]

Yep! The same way, I think we can teach Cargo that libs/mylib/lib.rs creates some kind of virtual default Cargo.toml for a workspace member. Like, I feel we have some kind “either sublibraies or private workspace members” model of thinking, while the end result might actually look almost the same for both approaches?

[kornel]

Cargo’s path dependencies foo = { path = "../foo" } are IMHO very easy to use and work pretty well for most things except: version bumps and publishing, which are chores that still have to be done per crate.

@matklad’s proposal to remove version from them and bundle them in the package sounds great to me, as it solves two main gripes in one go.

[withoutboats]

We discussed this in the cargo meeting today, mainly to think about the backward compatibility hazard as it relates to the edition. We determined that this is backward compatible so long as you always somehow specify something in your Cargo.toml section for each subcrate.

[jjpe]

These comments enumerate some of the design constraints and drawbacks of allowing a single package to contain multiple libraries, but they also contain some motivations for the limitation that I think might be reasonable to challenge at this point.

That is, I think there are good reasons to want to have multiple libraries in a package that are uploaded to crates.io as a single unit:

• Proc macro crates: Proc macros have to be contained in a special crate, leading to splits like serde vs serde_derive . With macro re-exporting, though, its possible to expose proc macros through the non-macro crate. In theory, the “derive” crate could be completely eliminated by having it subsumed as a “sub-library” of the main crate.
• Internal privacy boundaries: Often, a project will have a subsection which exposes a simple interface for very complex internals. Many items inside that submodule will want to be exposed throughout it but not outside it, leading to a lot of pub(in module) declarations. These could be abbreviated to pub(crate) (or even further if we stabilize crate visibility or similar) by making that submodule its own crate.
• Improved internal dependency organization: Crates are required to form a DAG. By breaking off subcrates, you guarantee a certain relationship of dependencies between them, helping you maintain a certain order within your project.
• Improved compile times: Crates are compiled separately and in parallel; it can be worth making a module its own crate for that reason alone.

Maybe it’s just me, but points 3 and 4 seem more like they argue in favor of keeping the status quo, rather than arguing for allowing multiple libraries in a cargo project.

Specifically, point 3 essentially means no cycles on the crate level. As the only reason for cyclic dependencies I’ve found to date is if you have 1 coherent whole but the source file grows too large (try editing a 17 KSLOC file as I’ve recently had to, it’s not a nice experience in any editor due to asymptotic scaling of editing algorithms), allowing cyclic imports on the crate level would seem to me a misfeature.

Point 4 says that separate crates are compiled separately and in parallel. This is true, but not the entire truth: the dependency relation between the importing and imported crates means that they will never be compiled in parallel, or at least I don’t think they should (as an error in a dependency crate must mean a failure to compile of the dependant crate).

FTR, I do agree with allowing multiple libs in a cargo project for e.g. proc_macros so I’m not exactly opposed to this feature, but it seems like it’s easily abused for the wrong reasons.

[rocallahan]

the dependency relation between the importing and imported crates means that they will never be compiled in parallel

This is true but it has to be fixed at some point if Rust compilation speed is to be competitive with C++. I think In principle compilation of an importing crate can be begin as soon as an imported crate has had its interfaces parsed and types resolved.

Returning to the topic, it seems to me that if you break a crate into two crates A and B where A depends on B, you actually slow down non-incremental compilation.

[jjpe]

The thing is, it is unclear how much that could deliver much in the way of speedups: the last time I heard anything about rustc performance, roughly half of the total time was being spent inside of LLVM to do code generation. Code generation cannot start before type checking finishes successfully, and should not be run at all otherwise.
Given that, it means that all dependencies of a crate C should at least have type checked successfully before type checking C itself can finish, and all dependencies must have code generated before C itself can have code generated. So I think a kind of dependency-based, pipelined approach* might be feasible, but it’s not clear that that would actually improve compile times all that much, given the amount of time spent in LLVM (which cannot be parallellized from what I understand).

*By this I mean that the different stages of compiling a Rust program (i.e. parsing, borrow checking, type checking and name resolution, desugaring, code generation etc) could take into account the crate-level dependency graph and schedule operations such that the appropriate dependencies exist between parsing a crate as well as parsing all of its dependencies, as well as a dependency between each 2 consecutive stages e.g. between parsing a crate C and borrow/type checking C. Compilation as a whole would then walk this mega-dependency graph (which would look like a number of interlinked variants of the crate level dependency graph, 1 variant for each compilation stage) and could find the critical path using this (which is important as it’s pointless trying to parallellize that on this level). It could then also parallellize all the non-LLVM things accordingly, and maybe invoke LLVM as a whole in parallel multiple times on different parts of the code-gen subgraph in a manner reminiscent of SIMD-the-idea (contrast to e.g. any specific CPU implementation).

[withoutboats]

@jjpe

Specifically, point 3 essentially means no cycles on the crate level. As the only reason for cyclic dependencies I’ve found to date is if you have 1 coherent whole but the source file grows too large (try editing a 17 KSLOC file as I’ve recently had to, it’s not a nice experience in any editor due to asymptotic scaling of editing algorithms), allowing cyclic imports on the crate level would seem to me a misfeature.

I don’t really understand this comment, but my best guess is that its misunderstood my post: this feature is not about allowing cycles between crates (which is absolutely impossible), but about the desire to separate something into a crate in order to get that guarantee that it does not have an interwoven dependency on other sections of your project. In other words, its about the desire to leverage properties of the crate dependency relationship to improve your code organization, not changing how crates work.

---

Your comments about compiler performance also seem to be off the point to me: a crate is the unit compilation. If you can break a section of your code into a sublibrary, you guarantee that you can compile it separately, and avoid recompiling it, from the other parts of your project that depend on it. Similarly, two sublibraries with no dependency relationship can be compiled in parallel. This is the same principle by which you avoid recompiling all of your dependencies today, and by which many of your dependencies can be compiled in parallel already.

Overall, there’s nothing in this proposal that would involve changing how crates work from how they work today, just allowing you to have multiple private library crates inside of a cargo package.

[wycats]

To me, the big benefit of this proposal is allowing small parts of a larger package to get crate-like isolation without also making them separate packages on crates.io.

Separating them into crates has some very useful benefits:

• the ability to use crate:: (in Rust 2018) to refer to the top level of this newly isolated unit
• the ability to use pub(crate) (and soon crate) to restrict visibility to this newly isolated unit

Keeping them together on crates.io has some benefits too:

• avoids creating public crates that you might not want to support separately
• allows the use of shorter names that wouldn’t be available on crates.io
• since the crates are all part of a single published package, opens up the possibility of expanding the definition of coherence to include the entire group of sub-libraries.

This combination of benefits, and a number of other related benefits makes me think that this is a big improvement. I also feel that the form of this proposal, as currently described, manages to achieve these goals without gratuitous differences from existing mechanisms, making it fit in nicely with what we already have and avoiding the creation of significant new non-compositional concepts.

I like it

[ahmedcharles]

What if this sort of feature was enabled by having a private_members field in [workspace]. I.e.

[workspace]

members = ["path/to/public/member"]

private_members = ["path/to/private/member"]

Disallowing this for virtual manifests would probably be best, since the underlying implementation would add the content of the crate into the package of the root manifest.

Allowing only all private or all public workspaces (ignoring the root) would be analogous to the primary use case of the original proposal, though it would be possible to allow specifying workspaces with both and have various options for how to resolve which packages contained which private crates. I’d suggest just starting with the simple cases.

This seems to have the same implementation complexity (in terms of bundling multiple crates into a single package) but avoids having multiple ways of specifying the same sort of construct. cargo new and cargo init already handle workspace configuration (well, the docs claim they do), so adding a --private flag would allow people to get the right configuration without significantly changing their workflow.

I like the idea, mostly due to the benefits of crates for layering (dependency management), I just don’t want to have two completely distinct ways of specifying the same idea when adding 1 key to a config can accomplish the same thing.

Note, an alternative format could be:

[workspace]

[[members]]
path = "path/to/public/member"

[[members]]
path = "path/to/private/member"
private = true

This would result in members always being an array, either of strings or tables, but unfortunately, this would mostly just be more verbose and likely result in people confused by errors when they try to mix tables and strings or declare members twice, etc.

[withoutboats]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.