I still can’t guess what you are doing, but that’s not it either. It works exactly the same if you move those tests directly into the main crate. You can see this PR passing in CI against stable, beta, and nightly: https://github.com/dtolnay/ref-cast/pull/6
I prefer to keep them separate only because it is not possible to have optional dev-dependencies at the moment (this is tracked in rust-lang/cargo#1596). If I need to run my crate’s tests against an older version of rustc than the one required by compiletest_rs, this way makes it easier.
The perfect solution would be a #[should_not_compile] attribute with a optional rustc error code.
It would probably require cargo test to generate for each of these test a rs file with all the other tests removed (so the failing test can be identified) and try to compile them. It would probably slow down test a lot, even if code generation can be skipped.