Recently crates.io implemented showing various CI badges, however I’m quite against that. I didn’t found the place of discussion, so I’d like to open a thread here.
Badges are for master, not for the crate.
We don’t care if the badge is green or red, the uploaded source should be usable. It means nothing. It doesn’t match the current state.
CI is designed for noticing mistakes early, not for advertising.
We have a badge, 1KB of redundant request, and then what?
Endless CI providers.
You know that there’s tons of providers on market. What direction are we going? Infinite code bloat?
Some projects use homu, others don’t. It’s good to encourage people to use continuous integration, but nothing more than that.
I’m proposing here to remove the badges from crates.io (except the version one). They take up page space, bloat code.
An important part of crates.io is helping users select the best crate for a particular job they might have. This means crates.io has to try and give the best possible information about crate qualities to users. Whether the crate is tested using CI can be an important quality indicator – even more so when you’re on Windows, and you can see that a crate is tested on Appveyor, not just Travis.
As for your separate points:
Mismatch between badge version and crate version: this is actually an interesting point. For something like Travis, it’s actually fairly straightforward to get the badge for the correct version, provided that the developers have tagged their version. It would be useful to show the tag-specific CI result. I guess it depends on whether most developers faithfully tag their versions on GitHub.
1 KB is actually a very small amount. It would probably make more sense to worry about the extra round-trip in loading the page, or about the visual information density in the crates.io UI, which is more scarce. In either case, though, I would say this is worth it.
Typically a particular crate would only have 1 or maybe 2 CI providers, so I don’t see much harm in crates.io supporting many providers. Theoretically the amount of concurrently displayed badges could be limited, but it seems unlikely we’ll have many problems with it.
I would actually be interested in more badges being added, particularly for code coverage (so likely coveralls.io, or codecov.io), as I feel code coverage data provides an even better signal for the author caring about code quality in a meaningful way.
Even a noob developer will try to publish a working release. They must not be broken.
Recently they added support for GitLab CI. Who's next? I'm not talking about the amount of badges per crate, but the overall code bloat to support all the providers.
Coverage percentage isn't the thing that user should blindly trust. I'm a heavy critic of badges, and they are all abused for advertisement while most continuous integration tools are designed for maintaining and make the code quality better.
There are already some cricism about blindly using coverage numbers - the number depends on the purpose of the library, and reaching a number near 100 isn't always a good thing.
I think this is more related to the inability to have any kind of interpolation in the manifest, which makes people reach for "master" then anything else:
For what its worth I find this information gives me a lot of upfront information about crate quality. Simply knowing that an author spent the time to setup CI for a project tells me a lot about the quality of a project.
This can - if wanted - be solved just like Github does that with all external resources: transparently serve them from crates.io instead. Even easier in the current implementation approach, as each an every shield is known.
What matters is the test quality, and that's something that you can only judge by looking at the tests in question.
I'm concerned that the badges will lead to a cargo cult checklist mentality where everyone copy pastes a CI file into each new repo without doing any meaningful testing. Speaking of which, it's also a waste of resources - does your integer addition trait really need to be tested on both Travis and Appveyor? If you aren't doing anything platform specific (which most crates don't), then there's no point.
My personal observation has been that badges lead to the exact opposite of what you fear, both in my projects I have created and projects that I have used.
I see tests as one of the most efficient uses of resources. As a creator of cross-platform projects I have avoided shipping broken code in multiple releases specifically due to Travis and Appveyor -- sometimes indicated by the failure of very simple tests. Having any tests matters to some degree, and making the tests more meaningful (higher quality) increases how much they matter.
The fact that the badges aren't an in-depth analysis of test quality doesn't really speak to whether or not badges useful for what they are.
As @carols10cents mentioned already, if you don't personally find badges useful in deciding which crate to use, then don't use that information. If the quality of the tests is what matters to you, by all means go read the project's tests!
A lot of the rest of us find the badges useful both as project creators and as users. I am in favor of the feature.
Regarding whether appveyor is needed for portable code, I’ll look point out that anything that uses the filesystem and wants to be portable should be testing on Windows as well as Linux. The semantics are so different that it is very easy to cross platform code that is buggy on Windows.
This is not helping the discussion.
Offering a tracker-free experience from the get-go seems like a noble goal in itself.
I'm not against badges, though it creates duplication when you have that information in the crate's README. Perhaps only having them when searching for a crate but not on the crate page itself?
The other issue is that badges do not reflect the actual state of the crate, but rather the master branch. A way to link a build url at publish time so that we can indicate the true state of the crate could be a solution.
Badges were introduced recently in crates.io, there's still room for improvement.
Badges only reflect status of main branch, not latest release
Generally redundant with README
Potential privacy concerns
Not a good indicator of quality
May not scale well
Given this, I kind of see @michael-zapata’s points, but I think given the relative difficulty it is in gauging quality when browsing crates.io that these should stand for now as another (debatedaly) useful metric. I believe there are plans to render README’s on crates.io and if so, maybe that would be the time to revisit rendering them separately as well.
I think an easy way to look at this is if we get rid of the badges, how would that affect things. I think right now, it’d just make it harder to determine which crates are worth investigating with almost no upside.
If you have release branches I guess that’d work, but I use tags for releases instead. I’d be surprised if that isn’t the more common way to handle releases also.