Requiring 2FA to Publish to Crates.io

Good point. We would have to assume the attacker has these capabilities, given that they can read a local token. Thanks.

Agreed. It's also worth 2FA'ing token creation, which protects against the compromised password (but not local malware) scenario.

Ah, yes. crates.io does need to take on the responsibility of holding onto seeds - I just wanted to make sure I wasn't missing something.

It is already crates.io's responsibility to keep crates safe, I don't think this is asking too much. But this is something to consider vs other alternative approaches (signing).

Oh, this is a great idea.

I had considered this, and I'd like to discuss it more. My main concern was that I felt that key signing has more failure cases - users just storing the keys locally, for example, in plaintext.

Thanks for this in depth reply. I'm going to think a bit further about what the signign experience would entail.

Off the bat I think signing is:

better because it means crates.io holds no secret information better because keys can be moved to other, safer workflows

worse because users have an easier time doing The Wrong Thing

1 Like

A Rust crate is only as trustworthy as its least trustworthy (transitive) dependency. One contributor with poor password hygiene on their Github account is currently enough to compromise dozens or hundreds of downstream crates. The NPM issue is a wake-up call for other developer communities, including this one.

At a high level, trusting a crate requires three things:

  • Trusting the author(s) not to introduce vulnerable or intentionally malicious code
  • Trusting the distribution system in between the authorsā€™ machines and the consumersā€™ machines (i.e., the publish process to crates.io, storage on crates.io, and download from crates.io)
  • Recursively applying the above process on each crate that this crate depends on

This proposal focuses on the second (applied universally to all dependencies).


Regardless of which approach this community ends up taking, itā€™s critical that integrity be required of all crates on crates.io. Opt-in/out-out security mechanisms mean that a crate thatā€™s trustworthy today can become untrustworthy tomorrow, and at that point, what happens to the crates that depend on it? Conscientious authors of dependent crates face a difficult choice of rewriting their crates to avoid the dependency (which may require breaking changes in their public interfaces) or create a polluted ecosystem of securely distributed but poorly maintained forks. Even if these forks are securely distributed, they probably shouldnā€™t be trusted since theyā€™re unlikely to receive security patches (or other important updates) in a timely manner.

Itā€™s in the best interest of the entire Rust community to ensure that all crates be trustworthy, and not to introduce a tiered system where different crates have varying degrees of trustworthiness. Many of us were attracted to Rust precisely because it enforces so many coding practices that help avoid problems. It seems entirely consistent that Rust would take a similar stance to avoid problems in its crate distribution mechanism, even if it introduces a minor inconvenience to the process or precludes certain workflows. Crate distribution is just too important to allow it to continue depending on 1FA with no additional protections.


Now that thatā€™s out of the way, letā€™s talk about the how:

There are basically two approaches to ensuring trust in the crate distribution pipeline:

  1. End-to-end: having authors sign the crates, and verifying the signatures upon downloading the crate from crates.io.
  2. Stepwise: ensuring crates are published to crates.io only by the true author(s), ensuring the crates are not modified while being stored on crates.io, and ensuring the crates are not modified while being downloaded from crates.io.

I think either approach is potentially workable, but we need to adopt something soon and not endlessly debate the merits of the two approaches.

End-to-end approaches tend to give better security guarantees, but they require a fairly complicated PKI, including a key revocation mechanism. Crate signing is only as secure as the private keys used for signing. If authors store their private keys insecurely, weā€™re not gaining much from this approach. Unfortunately, I donā€™t see a way for the Rust community to enforce proper key management practices among its authors (e.g., using a Yubikey).

Stepwise approaches provide slightly weaker guarantees, but theyā€™re potentially easier to manage. The trickiest step to secure, and the subject of this proposal, is the author publishing step. The most likely threat to this step is not a key logger, but rather poor password hygiene (weak passwords, password reuse, etc.). 2FA is an effective countermeasure to this threat. While GitHub supports 2FA, I donā€™t believe its OAuth2 mechanism allows relying parties to check whether 2FA was used. Hence, any approach here will likely require crates.io to store additional authentication information about each author.

In terms of which 2FA factors to support, we should avoid SMS since itā€™s widely viewed as insecure. Itā€™s reasonable to support TOTP, but some authors may unfortunately store their TOTP seeds on their local machines instead of on a separate device. As @bascule pointed out, entering TOTP codes is also vulnerable to keylogging. If weā€™re concerned about keylogging, a simple mitigation is for crates.io only to accept one TOTP code for the author during any 30-second period (preventing an online attacker from using the same code to re-authenticate). U2F and WebAuthn are probably the best options here, though adoption is far from universal. Theyā€™re not vulnerable to keylogging, and authors would have to go out of their way to store the secrets locally. A reasonable compromise is probably to support TOTP and WebAuthn in the short term, and eventually to deprecate TOTP as WebAuthn gains more widespread adoption.


A potential middle ground between end-to-end signing and 2FA is for authors to sign their crates at upload time, and for crates.io to validate the signature against the public key(s) it trusts for the author. This would simplify the PKI, since the downstream consumers of the crate wouldnā€™t need to worry about the signature.

The main downside here is that many (most?) authors will probably store the private keys locally, potentially without any password protection. This approach has most of the downsides of TOTP but with slightly better usability for automated publishing workflows.


One final point: even if crates.io eventually adopts a signing mechanism, updating the list of trusted public keys for an author on crates.io would still be a security-critical operation that should require 2FA. Otherwise, an attacker with the authorā€™s password can just add their own public key to the list of trusted keys, and sign a new package release. So, it makes sense to start implementing 2FA for crates.io even if it isnā€™t ultimately used in the publishing step.

5 Likes

The plan advocated by @withoutboats is to use GitHub as an OpenPGP "certificate" authority for signing keys. This makes sense for several reasons: because GitHub already implements a 2FA-able method for managing the public keys associated with a user (and can do things like verify commits with these keys in-band as part of the UI, as well as requiring signatures on repositories), and crates.io already uses GitHub as an IdP anyway. It does make GitHub, and any individual GitHub user's account, a single point of compromise.

Several years ago I suggested using The Update Framework (TUF) to provide end-to-end integrity for crates:

TUF isn't designed for a delegated authority model like this, however I think TUF could be used in combination with GitHub as a key authority, using something like what you were describing earlier:

TUF supports a notion of delegated targets, where a target in this case would be a crate, and the delegation to some signature key(s). A "TUF target service" for crates.io, when building metadata about crates in the index, can take a point-in-time snapshot of keys it sources from GitHub, and include these in e.g. the crates.io-index repository, which it would then sign (using e.g. @withoutboats proposed commit signing model).

If you're interested in that sort of approach for end-to-end integrity, I think https://github.com/rust-lang/crates.io/issues/75 is probably the right place to discuss it further.

2 Likes

Please correct me if I'm wrong, but I don't believe GitHub provides an API for querying whether users have 2FA enabled. They only recently added the feature that enables organizations to force their members to have 2FA enabled. Without a mechanism for crates.io to enforce that its authors have 2FA enabled, I don't believe we can rely on GitHub for this functionality. As I argued above, an opt-in security mechanisms is not likely to end well.

3 Likes

Exactly.

4 Likes

I think statements like this ignore practical realities of modifying living systems, often for dubious gains.

I think we can assume that more than one person here is charged with keeping systems safe :slight_smile:

I am uninterested in debating credentials.

Am I okay with the fact no progress has really been made in this space? Yes I am

Thatā€™s fine. Iā€™m actually not OK with it. I think package manager security is in a terrible state, and we should be exploring ideas that we can implement quickly for real gains. If you have not made headway in years, I am going to advocate we explore those other ways. Itā€™s been 4 years, and Iā€™m happy to see that it looks like progress is being made.

Itā€™s a reasonable alternative solution to consider.

annoying, it must be for big security gains, or we are only practicing security theater and making peopleā€™s lives more miserable for no security value.

Letā€™s take a step back here.

Who is miserable? 2FA is extremely common. Any company that takes security seriously already enforces 2FA for things like this - authing to production services, authing SSH, authing email, etc. Letā€™s not pretend that 2FA is some ā€˜extraā€™ security mechanism - everywhere else, itā€™s the bar, and package managers just arenā€™t meeting the bar.

If I told you a company did not 2FA access to their production environment, would you say ā€œrolling out mandatory 2FA for that is naiveā€ ? I would hope not - again, iā€™ts the bar.

In what way is this security theater? To me, that implies that we are not addressing the issue - I think it has been explained already that we are addressing a significant issue.

Yes, different threat models, and I'm open to hearing about why TUF is going to deliver what 2FA can not, in some better way. Please, do feel free to elaborate on this.

There is no extant crates.io feature for 2FA. It hasnā€™t even been designed, and 2FA has practical costs .

This discussion is for fleshing the design out. @ramosbugs has already done an excellent job of spelling out some concerns/ benefits of various 2FA approaches.

This is not an RFC. This is not a Pre-RFC.

go write code, ship a production-worthy crates.io feature that people can agree should be merged, and then letā€™s talk about making it mandatory.

No. I have seen people write code for this sort of thing before only to have it rejected. I am not interested in such a thing.

This topic is for soliciting feedback, fleshing out the idea, raising concerns, and gaining consensus. Not ā€œI believe this is right, I will build it in a vacuum, you can reject it after I put weeks of work inā€.

You havenā€™t given a real reason why mandatory is bad, other than ā€œwrite it first and it will be apparentā€.

You havenā€™t actually demonstrated that 2FA does not adequately impact the threat model put forth.

edit: OK, attempted to recover the post.

4 Likes

As someone who has been working on the problem of securing the software delivery lifecycle in both a professional capacity and for open source repositories like crates.io, and as someone who has rolled out 2FA retroactively inside of a company with thousands of employees, I think statements like this ignore practical realities of modifying living systems, often for dubious gains.

For starters, I would strongly suggest studying TUF. One of TUF's foundational principles is that it can be retrofitted into living systems which have not started from a position of having package signing, and can gradually improve security on an opt-in basis from its system's users.

I would like to think I'm an expert in the space of package signing. I opened an issue on crates.io about this in 2014 when no one else was paying attention. I find statements like this particularly disturbing:

The NPM issue is a wake-up call for other developer communities, including this one.

It is not a "wake-up call" for me. I have been focused on this particular problem intently for many years.

Am I okay with the fact no progress has really been made in this space? Yes I am, because I am grounded in reality, and am well-studied in the implications of making changes to living systems. But beyond that, this is a very difficult problem, particularly in terms of finding a reasonable threat model and the difficulty involved in implementing practical solutions which result in actual positive security outcomes instead of a grueling user experience beleaguered with pointless security theater whose net result to the crates.io ecosystem is developers churning because it is too onerous to use.

I also fight for the users, and my opinion is go big or go home: if we're going to make the crate publishing system annoying, it must be for big security gains, or we are only practicing security theater and making people's lives more miserable for no security value.

Given that backstory and expertise, and experience in participating in knee jerk reactions to package system compromises in the past (RubyGems, YAML RCE, circa 2012), and being on the ball about this particular topic years before everyone else because I already experienced a critical compromise of a different package ecosystemand personally worked on implementing a package signing solution when all other knee-jerk efforts, let me please just say:

Stop talking about mandatory 2FA

You are talking about vaporware. There is no extant crates.io feature for 2FA. It hasn't even been designed, and 2FA has practical costs. Which mechanism should be used? What is the recovery mechanism if someone loses their 2FA credential? But please don't directly respond to these questions, go write code, ship a production-worthy crates.io feature that people can agree should be merged, and then let's talk about making it mandatory.

Having worked in retroactively deploying 2FA at multiple companies with millions of users, this is a hard problem, and talking about making it mandatory before the code is even written is in stark ignorance of the practical reality. I've recently just watched a student implement this mechanism as a Google Summer of Code project for RubyGems. I suggest the following:

  1. Implement the feature
  2. Make it publicly available to existing system users
  3. Begin discussions of making it mandatory

I think it's very easy from a naive threat modeling perspective to look to a particular approach as some sort of silver bullet while completely ignoring all practical realities, which is what seems to be going on here. In my experience trying to solve problems this way often makes systems unusable and reduces system security as users actively thwart poorly designed "security" mechanisms.

The reality of retrofitting security mechanisms is it is a long, grueling, and painful process which involves a lot of work, unhappy people, and championing the cause of security despite the criticisms. In doing that, you better damn well be sure you are affecting the intended outcomes. We are absolutely not there yet, and we need to get there before we talk about making anything mandatory.

10 Likes

God damn. I wrote a reply and I donā€™t get this website so I guess I deleted it. Just likeā€¦ read the pre-edited version I guess.

edit: Hilariously, I have now deleted another post in an attempt to revive the first one.

1 Like

@bascule, we're attempting to have a technical discussion here grounded in sound technical arguments about the tradeoffs of different potential options here.

The following are not productive to this discussion:

In the interest of making progress on this important issue, let's please keep the conversation focused on the technical merits, and not who is making them, and why that person believes their opinions are more credible than others'.

Thanks in advance.

4 Likes

I support the basic idea of working towards required 2FA and/or required code signing. I do not think that either of them are too cumbersome, if implemented correctly; however, there are a lot of tricky details in implementing them correctly.

So while the basic outline of the proposed plan in the original comment looks reasonable, I think like @bascule says, it will require first an opt-in implementation, getting sufficient testing, and the plan will probably have to change over time.

In the meantime, letā€™s think about particular threat models. The NPM issue came about because someone who had access to upload a popular package had a password which was shared with other, compromised accounts. This in turn let the attacker upload that popular package, and then when other developers installed the compromised package, it could in turn steal credentials from them.

crates.io doesnā€™t use its own passwords; it delegates to GitHub. But the equivalent situation in the crates.io case is that someone has a GitHub password that gets compromised, allowing for adding a new access token, and then uploading packages with that.

Thereā€™s been some discussion of whether it would be possible to check GitHub if 2FA is enabled, and many people have pointed out that organizations are able to enforce that. So one quick and easy way to enforce the 2FA requirement would be to just require uploaders to be members of a ā€œCrates.io Usersā€ organization, with membership automated by a bot, and with 2FA enabled.

Then we get to what we do about the fact that once one person is compromised by a bad package, their tokens can be compromised to compromise their crates. The original post suggests also requiring 2FA for publish, but there are very few 2FA or signing methods that can actually guarantee that the person is using a different device rather than the same device, unless they require hardware tokens of some sort. So any kind of 2FA is at least going to have to trust that users choose a reasonably independent second factor.

Of course, concerns about one package being able to cause a compromise of others in large part comes down to the terrible security model of desktop workstations, in which every process has access to all data owned by the same user running it, and generally has enough access to install keyloggers or the like to intercept even one-time passwords.

Ideally, building a Rust library shouldnā€™t give the build script access to a users secrets, nor even should running the resulting binary give access to them, unless the user specifically gave it access.

Solving this problem is probably out of the scope of Cargo and Crates.io, as it requires moving to a model a lot more akin to mobile devices or capability based systems like Fuchsia, and I donā€™t know of any systems for controlling access like this but being flexible enough for a developer workstation, let alone being common on all Tier-1 platforms.

Without solving that problem, if you do add 2FA to the publish step as well, and people use it correctly, by using a separate device, that does largely cut down on the risk of malware that is just hoovering up credentials to be used later; it doesnā€™t protect against a full persistent threat that will try to MITM or keylog your every action, but it does protect against fairly simple ways of compromising keys, like having them throw out on hard drives, accidentally check their home directory into public version control, or exploits which are able to read data from your system but not write or control anything. So it is still an improvement over a token permanently on disk.

Signing has additional benefits; it authenticates the crate author to the user directly (at the very least with TOFU based identify verification), without having to even trust the crate repository, while 2FA puts all the trust for verifying authenticity in the crate repository. However, I donā€™t know of many ways to require a second device be involved, at least thatā€™s widespread and reasonably standard and doesnā€™t require specialized hardware tokens. Is it possible to easily secret-share between you laptop and phone, and so have to push an ā€œOKā€ button on your phone to perform a signing operation?

I think that 2FA and signing are probably the most important mitigations for these types of attacks, but itā€™s also worth thinking about similar attacks, and potential solutions to them which might also help with this type of attack.

Some attacks to think about:

  • Actual developer of popular package goes rogue, publishes malicious version of their package.
  • Honest developer gets bad code merged somehow, publishes version with it
  • Crates.io itself is compromised
  • Crates.io gets MITMed, by a nation-state who can issue their own certs, or by someone who exploits a weak link in the WebPKI chain

Some potential other strategies for dealing with them:

  • Be able to display and audit diffs between package versions more easily (actual crates.io package versions, not Git, since whatā€™s on crates.io might differ from Git)
  • Be able to display and audit diffs between Git and the actual published crate (Iā€™m sure this would reveal cases where people had accidentally published code that hasnā€™t been committed to Git or was a different commit than the actual tagged version, much more often than it would find vulnerabilities, but thatā€™s also useful)
  • 2 person signoff rule (could only apply to packages with more than one maintainer, and probably only really usefully to packages with at least 3, to avoid having each person be a single point of failure for actually publishing new versions of the crate)
  • Sandboxing builds and/or capability based permissions for executables (probably requires better platform support)
  • Delay releases for a cool-off time during which they can be audited; possibly even a simple version of this could be implemented on the client side, where cargo would not resolve newer compatible point releases until they were a certain age, but you could still explicitly choose them by setting that version in your Cargo.toml
  • Have some hand picked trusted versions of crates as a ā€œdistributionā€ by some auditors doing at least a cursory look for funny stuff, and who are also generally looking for good development and security processes from crate authors (the ā€œLinux Distributionā€ model)
  • TUF-style signing
9 Likes

I understand you don't like the tone of my argument. I too enjoy technical arguments. But while I have made many technical arguments, your post made none.

My main concern with what you said was:

As I argued above, an opt-in security mechanisms is not likely to end well.

I think opt-in mechanisms are a necessary part of retrofitting new security mechanisms into a living system. In fact the only systems I've seen succeed in retrofitting security mechanisms took a gradual path of this nature.

1 Like

To be clear, I would not be against starting with opt-in, as an approach to getting to mandatory 2FA.

I am against opt-in being the end goal.

Also, your post made no technical arguments whereas ramosbugsā€™ post contained a discussion of potential technical implementations and their tradeoffs.

But perhaps letā€™s stop attempting to discredit or puff up one position, and try discussing it in earnest.

3 Likes

As a user of crates.io, if crates.io adds mandatory 2FA, the 2FA token gets stored in my password manager alongside the password, just like it currently happens with GitHub where organizations can require 2FA, because the only thing mandatory 2FA adds here for me is an additional point of failure. Will I lose my Yubikey within 3 months? Yes. Will I break my smartphone to the point of unusability within 3 months? Yes. Repeat for any possible second factor device. Small physical electronic devices are simply not reliable enough to serve as a second factor.

The only way to deal with decrease of availability added by mandatory 2FA that I find is to reduce it back to a single factor.

9 Likes

Thatā€™s certainly something to consider. It significantly reduces the benefits of 2FA when users do what youā€™re describing. edit: Although, for an attacker who has only compromised the password (such as through reuse on a compromised site) 2FA would still provide benefits.

It sounds like the problem is, specifically, that youā€™re concerned with account recovery in the event of a lost 2FA token?

Iā€™m in full agreement that having a recovery workflow is critical. Doing this safely often means reducing the security to, say, a compromised token + email account, or token + wherever recovery codes are stored.

But I think these feel like reasonable approaches. Is there a recovery workflow youā€™ve encountered that was particularly burdensome/ well put together?

@insanitybit To my understanding, if I lose the GitHub recovery tokensā€“and obviously I will lose them, duh, do you think I will remember to keep a piece of paper safe when I canā€™t keep track of a Yubikey, not to mention that I can lose access to themā€“for example, I often travel to a different country for many monthsā€“there is no way to recover my GitHub account short of asking the GitHub staff.

The GitHub staff will probably require state-provided ID to recover an account without a 2FA token and 2FA recovery codes, and no web service will ever get my state-provided ID because they have no business looking at it, not to mention that because of reasons I am not going to disclose it wouldnā€™t even be very useful to GitHub staff.

Therefore, losing the 2FA token and 2FA recovery codes mean that my account is irrevocably lost, which is unacceptable. That means that both of these credentials are getting backed up as widely as possible and with as few requirements to restore as possible.

At some point, if you are unwilling to keep your account secure, thatā€™s sort of justā€¦ that. Yes, this will mean you can not contribute to crates.io. Similarly, it is required that you password protect your account. This is a burden - you now must store or remember that password. If there were somehow an option to opt out of passwords, I would hope that anyone who does so loses the ability to push.

At some point I think there has to be a bar, and if you are unwilling to meet the bar, then your code does not get hosted. I donā€™t want to sound harsh, this is just how any security control would have to be enforced.

FWIW, Github does not require state ID. I have recovered my account using an SSH key. Most other 2FA implementations will fall back to SMS for recovery, or email. At least, the ones that I have seem.

3 Likes

If you fall back to an SSH key, then it's exactly as secure as the actual code stored in a VCS. In that case, you could make cargo publish authenticate by an SSH key, which has the same benefits as 2FA with recovery by SSH key but none of the pain to end users. Which, I admit, does make some sense, but would greatly inconvenience the unknown (to me) amount of people who authenticate to VCS by password.

If you fall back to SMS, then it's exactly as secure and available as SMS, which is a terrifying proposition to anyone who actually understands the security of SMS--there isn't any, anyone in the world with access to an SS7 control channel can read and reroute SMS messages of anyone else in the world--and the availability issues are even worse. First, when I lose my phone I also lose the SIM card inside, which I can't necessarily easily recover (I can be in a different country) or recover at all (depending on the plan and how much the operator likes my face that day). Second, if you can't get your 2FA token because you can't get an SMS--for a multitude of reasons, starting with "my phone isn't charged" and ending with "SMS to this SIM card don't get delivered in this country"--then you also can't recover the account.

If you fall back to email, then it's exactly as secure as your email, which is obviously fine from my point of view (because I consider my email already secure enough), so I don't have any objections personally, but consider that 2FA is usually supposed to solve the "GitHub password is the same as email password" problem.

Wrong: I am unwilling to opt in into the mandatory security theater you seem to consider necessary. Also wrong: you aren't required to password-protect your crates.io account, because crates.io does not use passwords.

2 Likes

I am only stating what some existing recovery options are. I am not advocating them. SMS recovery has already been called out earlier in this topic as a non starter.

Yes, crates.io uses github for auth. I'm aware. My point stands. It is not the case that crates.io is an unauthenticated service, and there is already a bar for packaging crates. If you do not meet that bar, you do not get to package them. Picking apart minutia to avoid the obvious point is hardly productive.

Sure, that seems worth exploring. It at least deals with one component of the threats described - password reuse.

Is it?

If the assumption is that you remember your password but lost your 2FA, then crates.io password + email password is now required. This seems acceptable to me.

If you lose your password + 2FA at the same time, yeah, not sure I see a good recovery option and that's just a risk. At that point, I'd say create a new account, create a new crate, and deprecate the old one.

Let's not argue? I'm trying to understand your issues with 2FA and see if there's a reasonable way to address them.

1 Like

What I see is mostly you implying that "I'm unwilling to keep my account secure" because I don't follow the steps you consider necessary, not you trying to understand my issues with 2FA. That's not productive (and insulting), and frankly, if crates.io security will be run by someone with this attitude, I will just host my crates on my own server.

crates.io security is not run by me. At all. I am not involved in the projectā€¦ at all.

I am saying that if the bar is 2FA, and you are unwilling to comply with the bar, then you donā€™t get to publish code.

This bar already exists, I am simply advocating for raising it.

Iā€™m sorry if itā€™s coming off as me saying that you are doing something wrong. You are not. These controls donā€™t even exist soā€¦ how could you?

I am saying that I think these controls should exist and that, as with existing controls, compliance is mandatory.

To make that an acceptable option, I want to understand what problems you are facing. You seem to feel that recovery will be a serious option - I have tried to discuss that further with you.

You are not obligated to discuss any of this with me, and Iā€™m not interested in further discussion if youā€™re going to be insulting.

edit: Iā€™m also entirely willing to entertain other approaches, such as TUF, but thatā€™s already work someone else is taking on. I have already asked for elaboration on that projectā€™s state and why itā€™s a better solution.

My interest is in raising the bar against attackers, given specific attacker capabilities. I want to ensure that when doing so, the transition is smooth, that users are still enabled to push code, and that we can recover accounts/ handle the complexity this involves.

3 Likes