Some thoughts on showing CI badges on crates.io

[ishitatsuyuki]

Recently crates.io implemented showing various CI badges, however I’m quite against that. I didn’t found the place of discussion, so I’d like to open a thread here.

• Badges are for master, not for the crate. We don’t care if the badge is green or red, the uploaded source should be usable. It means nothing. It doesn’t match the current state.
• CI is designed for noticing mistakes early, not for advertising. We have a badge, 1KB of redundant request, and then what?
• Endless CI providers. You know that there’s tons of providers on market. What direction are we going? Infinite code bloat?

Some projects use homu, others don’t. It’s good to encourage people to use continuous integration, but nothing more than that. I’m proposing here to remove the badges from crates.io (except the version one). They take up page space, bloat code.

[djc]

I don’t agree with most of your points.

An important part of crates.io is helping users select the best crate for a particular job they might have. This means crates.io has to try and give the best possible information about crate qualities to users. Whether the crate is tested using CI can be an important quality indicator – even more so when you’re on Windows, and you can see that a crate is tested on Appveyor, not just Travis.

As for your separate points:

• Mismatch between badge version and crate version: this is actually an interesting point. For something like Travis, it’s actually fairly straightforward to get the badge for the correct version, provided that the developers have tagged their version. It would be useful to show the tag-specific CI result. I guess it depends on whether most developers faithfully tag their versions on GitHub.
• 1 KB is actually a very small amount. It would probably make more sense to worry about the extra round-trip in loading the page, or about the visual information density in the crates.io UI, which is more scarce. In either case, though, I would say this is worth it.
• Typically a particular crate would only have 1 or maybe 2 CI providers, so I don’t see much harm in crates.io supporting many providers. Theoretically the amount of concurrently displayed badges could be limited, but it seems unlikely we’ll have many problems with it.

I would actually be interested in more badges being added, particularly for code coverage (so likely coveralls.io, or codecov.io), as I feel code coverage data provides an even better signal for the author caring about code quality in a meaningful way.

[ishitatsuyuki]

It would be useful to show the tag-specific CI result.

Even a noob developer will try to publish a working release. They must not be broken.

Typically a particular crate would only have 1 or maybe 2 CI providers, so I don’t see much harm in crates.io supporting many providers. Theoretically the amount of concurrently displayed badges could be limited, but it seems unlikely we’ll have many problems with it.

Recently they added support for GitLab CI. Who’s next? I’m not talking about the amount of badges per crate, but the overall code bloat to support all the providers.

as I feel code coverage data provides an even better signal for the author caring about code quality in a meaningful way.

Coverage percentage isn’t the thing that user should blindly trust. I’m a heavy critic of badges, and they are all abused for advertisement while most continuous integration tools are designed for maintaining and make the code quality better.

There are already some cricism about blindly using coverage numbers - the number depends on the purpose of the library, and reaching a number near 100 isn’t always a good thing.

[skade]

Badges are for master, not for the crate.We don’t care if the badge is green or red, the uploaded source should be usable. It means nothing. It doesn’t match the current state.

I think this is more related to the inability to have any kind of interpolation in the manifest, which makes people reach for “master” then anything else:

Variable interpolation in Cargo.toml

Well, that's the danger of using a format marked as version 0.4.0 as the first big user. In that case, such a feature should be discussed with the relevant team before, but that shouldn't factor into the discussion whether that's a desired feature or not for Cargo.

[skade]

Even a noob developer will try to publish a working release. They must not be broken.

Tested on each and every target that they target? I don’t think so.

If they do, they probably prove that through CI, which makes the CI status the most direct piece of info to propagate.

I hate “even noob” arguments. Even advanced developers may be sloppy here.

Recently they added support for GitLab CI. Who’s next? I’m not talking about the amount of badges per crate, but the overall code bloat to support all the providers.

My only surprise here is why they didn’t allow arbitrary shields.

Coverage percentage isn’t the thing that user should blindly trust. I’m a heavy critic of badges, and they are all abused for advertisement while most continuous integration tools are designed for maintaining and make the code quality better.

They haven’t been misused for advertisement, they are for advertisement.

[bbatha]

Badges are for master, not for the crate.

For what its worth I find this information gives me a lot of upfront information about crate quality. Simply knowing that an author spent the time to setup CI for a project tells me a lot about the quality of a project.

[ishitatsuyuki]

Tested on each and every target that they target? I don’t think so.

If they do, they probably prove that through CI, which makes the CI status the most direct piece of info to propagate.

Please note that most project using Travis is testing on x86_64 only. (They can even have no tests)

They haven’t been misused for advertisement, they are for advertisement.

Look at here.

[carols10cents]

If you don’t think CI badges are a useful indicator when deciding which crate to use, then don’t use that information.

As a maintainer of crates.io, the code bloat in crates.io’s code due to badges is not a problem. It’s a very tiny percentage of the codebase.

[carols10cents]

My only surprise here is why they didn’t allow arbitrary shields.

We’re trying to make it easy to set up the badges correctly, so that everyone doesn’t have to go look up the various URLs for the various shields.

We’d like to start rendering READMEs on crates.io, which would be a place for arbitrary shields.

[est31]

Also dont forget that badges like any resources from third parties can be used for tracking and to intrude the privacy of the visitor.

[carols10cents]

Also dont forget that badges like any resources from third parties can be used for tracking and to intrude the privacy of the visitor.

Which you can block if you so choose.

[skade]

Also dont forget that badges like any resources from third parties can be used for tracking and to intrude the privacy of the visitor.

This can - if wanted - be solved just like Github does that with all external resources: transparently serve them from crates.io instead. Even easier in the current implementation approach, as each an every shield is known.

[Storyyeller]

For what its worth I find this information gives me a lot of upfront information about crate quality. Simply knowing that an author spent the time to setup CI for a project tells me a lot about the quality of a project.

What matters is the test quality, and that’s something that you can only judge by looking at the tests in question.

I’m concerned that the badges will lead to a cargo cult checklist mentality where everyone copy pastes a CI file into each new repo without doing any meaningful testing. Speaking of which, it’s also a waste of resources - does your integer addition trait really need to be tested on both Travis and Appveyor? If you aren’t doing anything platform specific (which most crates don’t), then there’s no point.

[CleanCut]

I’m concerned that the badges will lead to a cargo cult checklist mentality where everyone copy pastes a CI file into each new repo without doing any meaningful testing. Speaking of which, it’s also a waste of resources - does your integer addition trait really need to be tested on both Travis and Appveyor?

My personal observation has been that badges lead to the exact opposite of what you fear, both in my projects I have created and projects that I have used.

I see tests as one of the most efficient uses of resources. As a creator of cross-platform projects I have avoided shipping broken code in multiple releases specifically due to Travis and Appveyor – sometimes indicated by the failure of very simple tests. Having any tests matters to some degree, and making the tests more meaningful (higher quality) increases how much they matter.

The fact that the badges aren’t an in-depth analysis of test quality doesn’t really speak to whether or not badges useful for what they are.

As @carols10cents mentioned already, if you don’t personally find badges useful in deciding which crate to use, then don’t use that information. If the quality of the tests is what matters to you, by all means go read the project’s tests!

A lot of the rest of us find the badges useful both as project creators and as users. I am in favor of the feature.

[droundy]

Regarding whether appveyor is needed for portable code, I’ll look point out that anything that uses the filesystem and wants to be portable should be testing on Windows as well as Linux. The semantics are so different that it is very easy to cross platform code that is buggy on Windows.

[michael-zapata]

Which you can block if you so choose.

This is not helping the discussion. Offering a tracker-free experience from the get-go seems like a noble goal in itself.

I’m not against badges, though it creates duplication when you have that information in the crate’s README. Perhaps only having them when searching for a crate but not on the crate page itself?

The other issue is that badges do not reflect the actual state of the crate, but rather the master branch. A way to link a build url at publish time so that we can indicate the true state of the crate could be a solution.

Badges were introduced recently in crates.io, there’s still room for improvement.

[susurrus]

To sum up the discussion we have:

Pros:

• Potentially another indicator of quality
• Indicates status of the project

Cons:

• Badges only reflect status of main branch, not latest release
• Generally redundant with README
• Potential privacy concerns
• Not a good indicator of quality
• May not scale well

Given this, I kind of see @michael-zapata’s points, but I think given the relative difficulty it is in gauging quality when browsing crates.io that these should stand for now as another (debatedaly) useful metric. I believe there are plans to render README’s on crates.io and if so, maybe that would be the time to revisit rendering them separately as well.

I think an easy way to look at this is if we get rid of the badges, how would that affect things. I think right now, it’d just make it harder to determine which crates are worth investigating with almost no upside.

[retep998]

Badges only reflect status of main branch, not latest release

If you update what branch the badge points to when you publish, then it is perfectly capable of reflecting the status of that exact version.

[susurrus]

If you have release branches I guess that’d work, but I use tags for releases instead. I’d be surprised if that isn’t the more common way to handle releases also.

[retep998]

If you have release branches I guess that’d work, but I use tags for releases instead. I’d be surprised if that isn’t the more common way to handle releases also.

You can point the badges to tags as well just fine.