Pre-expanding proc macros

tools and infrastructure
21

You're right, it should be.

To do this in a backwards-compatible way (without needing a whole new publish-dependencies table), what if we start out by providing all build-dependencies for both publish.rs and build.rs, and then provide an (ignorable by older cargo and other tools) means of explicitly marking build-dependencies as publish-only so that the user of the crate doesn't need them? (And, at the same time we might as well also mark build-only dependencies that publish doesn't need, though that's less critical.)

22

Hm, no, it should be dev-deps? The difference between build and dep is not whether they are compiler for host or target (proc macros always compile for host, no matter if they are a dep, dev-dep, or buld-dep). Rather , the difference is that build-deps are viral, while dev-deps are not. Downstream projects have upstream build-deps in their Cargo.lock, and this is exactly what we are trying to avoid here.

23

publish.rs would want to use non-proc-macro crates, though, and would still want them compiled for the host rather than the target. build-deps always compile for the host. dev-deps that aren't proc macros compile for the target.

I agree that users of the crate wouldn't need them, but we could solve that by having newer cargo filter out publish-only dependencies from the generated Cargo.toml in the published crate.

1 Like
24

Definitely, things like protobuf code generators should be part of this effort too and they are not proc macros.

25

It should also be possible to have a dependency for both publish.rs and at runtime with different features (recent example I touched, building a syntect pre-compiled SyntaxSet in publish.rs, to embed as a binary blob in the published crate, that is then loaded at runtime by syntect again).

26

Hm, I am 0.8 sure that this works differently: dependencies compile to whatever target(s) is required by the downstream crate. sections of dependencies table don't affect which targets the deps are compiled for, only which deps are available for each particular crate. Eg, a same package can be (transitively) a dev, build, and prod dependency. Cargo will compile it for all the required targets. If the leaf thing Cargo is building is build.rs, than target would be host, if the leaf thing is user's code, target would be the --target argument.

In other words, when Cargo would be building publish.rs, it would build it for host, and that would just cause all dependencies available for publish.rs to be build for host.

27

If I understand your reasoning correctly then this should build:

src/main.rs:

fn main() {}

build.rs:

#[allow(unused)]
use serde::Deserialize;

fn main() {}

Cargo.toml

[dev-dependencies]
serde = "1.0.0"

Command: cargo test (builds dev-deps)

However it fails with undeclared crate or module error.

28

I think what I've said was ambiguous (because in cargo, the word target has two very different meanings: "target triple" (arch/os/libc; host or target) and "target crate" in a package (lib, expamle, test)).

To rephrase:

Sections of dependencies table don't affect which target tripple the deps are compiled for, only which deps are available for each particular target crate.

In your example, dev-dependencies make serde available for the following target crates:

  • lib in unit-testing mode
  • integration tests
  • examples

Notably, serde is not made available for:

  • lib
  • build

From cargo toml alone, it's not possible to say for which target triples the serde will be compiled. If it is included in transitive closure of stuff we build for host, it'll build for host. If it is included in transitive closure of stuff we build for target, it will be build for target. It might be build for both. An illustrative example would be

[dependencies]
proc-macro-that-uses-serde-inside = "1.0.0"

Here, serde will be build for host, despite being (indirectly), in the dependencies table.

The important implication in the current setup is:

dev-deps are not made available for lib or build.rs target crates. As downstream crates only build lib and build, and not any other kind of target crate, they can be ignorant of dev dependencies.

For publish.rs, we only want to run that during development, we don't want to run it during the build of downstream crates. So we want to stick publish.rs deps into the dev-dependencie table.

One drawback here would be that dev-deps would include the union of stuff we need for tests, and the stuff we need for publish, and those might have big symmetric difference. That is the same problem that we have today with the absence of binary-only dependencies: stuff you need in src/main.rs "infects" src/lib.rs also.

1 Like
29

(note that "triple" has a single 'p')

1 Like
30

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.