Crates.io incident 2018-10-15

[Nemo157]

@Benjamin-L see

rust-lang/rfcs/blob/master/text/2141-alternative-registries.md

There is ongoing work to support this.

[skade]

I don’t think that the correct response the official team not taking namespacing and frontend issues seriously is necessarily to make a new system, but I think the option should be there for a lot of reasons.

The issue is taken seriously, though, just the conclusions don’t match with a very vocal subset of the community.

And getting hazardous about the whole thing like the person causing this incident won’t help moving this issue.

(for the record, I vehemently oppose namespacing)

[ehiggs]

I’m seriously baffled by the amount of what I would call “victim blaming” in this thread. I someone does something which is not acceptable, saying “yeah but you had it coming” is not a great way to react.

There was a denial of service attack on the community. A solution for damage limitation was proposed years before the attack and not accepted and now the attack happened. They are related and interlocked.

Thankfully we haven’t yet had malicious typo-squatting yet with tokyo-core adding a dev-dependency on sys32-encoding (or other extremely dry name that makes you assume it’s super important) which in turn opens a socket in the build.rs and sends data from your build machine to a 3rd party.

This is also a related issue to spamming and namespacing and important enough to impact the design of the crate system and infrastructure.

It’s ok that Cargo’s design made a mistake here. We can fix it. Just default all packages to imply org::rust_lang::legacy. Make a rustfmt rule to fix up packages as we go. Eventually namespaces get introduced, etc.

(Also: I’m all for federation!)

[gwihlidal]

Agree that victim blaming isn’t helpful, and I also agree that the policy around spam reserving crates needs to be addressed, instead of being ignored.

The ability for mass reservation of crate names, even going so far as implying you need to buy the rights from the owner similar to domain name squatting, is one of my biggest disappointments with rust and the ecosystem so far (the language itself is great!).

People like https://crates.io/users/swmon are part of the problem.

As a plea from someone coming from AAA game development and the black pit of C++, please deal with this. One of the biggest strengths of the rust ecosystem is a proper package management and module system, so it is important to spend the effort in protecting and advancing it.

[steveklabnik]

Here’s the blog post https://blog.rust-lang.org/2018/10/19/Update-on-crates.io-incident.html

[Centril]

Here’s the blog post https://blog.rust-lang.org/2018/10/19/Update-on-crates.io-incident.html

From the blog post, my impression is that this was very professionally handled. Nice work!

I’m sure you have already considered this, but in case you have not, I would suggest providing a FAQ like the rust-lang one: https://www.rust-lang.org/en-US/faq.html, with some of these policy questions. This might reduce some of the tension.

It sounds like the community has wanted namespacing from day one of this policy.

I don’t think this thread or other recent ones constitute consensus on what the community wants.

[steveklabnik]

So, I think that you wanted something different from this post. The point of the post wasn’t to try to discuss the overall problem; it was to talk specifically about the incident. For example:

[no mention of namespacing as a possible solution is quoted, and no mention is made of the linked discussion being strongly in favor of namespacing]

We can’t actually know that the person who did this did so because of the namespacing discussion. As such, getting into it is not really relevant.

---

But, getting into the other bits:

Excellent, I’ve been wondering how namespacing could possibly have been overlooked. I wonder what the “rationale behind the policy” was.

The rationale is in the post. Was that not clear, or is this a typo (“wondered”, maybe?)

Are we wasting our effort discussing it?

As the post says;

If you feel that a policy is problematic, the correct place to propose a change is by creating an RFC or messaging the team at help@crates.io.

Discussing it is fine, but discussions here are never going to change policy. In that sense, it is a “waste”, but that’s because discussions here are usually precursors for RFCs, so that’s also not a “waste” in the other sense.

This is starting to feel really weird and adversarial.

I agree that it’s felt adversarial for a long time, but on a personal note, please consider how your framing here promotes that adversity:

finally, an official response to our concerns.

I am not reassured that the crates team has any intention of working with the community on this.

It sounds like the community has wanted namespacing from day one of this policy.

Some of the community has. Much of the community has not. There were official responses, early in the discussions, but then it got fairly nasty, and most people on the “anti” side got tired of going in circles. The “pro” side has constantly promoted this “community vs the team” mentality, when the community is significantly more fragmented than they suggest. And, since this is only internals, and not an RFC, if I only have a limited time and energy, I’m going to save it for the venue where it’s possibly going to affect real policy. I also don’t feel that any new arguments have been made since 2014.

I do not know how to fix this rift. I don’t feel the pro side is ever going to accept any of the arguments.

[skade]

It sounds like the community has wanted namespacing from day one of this policy.

No. There’s large groups in “the community” that don’t see namespacing as a solution. Trying to use the voice of the mass doesn’t help here.

[steveklabnik]

I would suggest providing a FAQ like the rust-lang one: https://www.rust-lang.org/en-US/faq.html,

I actually thought that https://crates.io/policies contained the same text as my 2014 post until yesterday. I certainly agree that it should be more prominent.

[steveklabnik]

it seems disingenuous to imply there is equal support on both sides.

I am not saying that there is equal support. I am saying that we do not know what percent is on what side.

much of the “hush now and wait for the real response” was targeted at discussion of namespaces… You used it to stifle discussion of namespaces earlier

Sorry, maybe I’m making a mistake, but I don’t know what you’re referring to. Could you point me to this? It’s been hard to keep track of all of the responses, but I can’t remember what you’re talking about.

[KillTheMule]

There is a clear majority that wants namespaces.

What makes you sure here, any data on that? I personally am opposed, but I’ve never said anything in any discussion.

[steveklabnik]

The only recent comment I can recall by anyone in the team was @sgrif, who said

Definitely. Nobody is trying to shut the discussion down. I’m just asking that folks try to keep the discussion above “I disagree on the conclusion that has been drawn in the past”, and try to bring new information to the table. There’s been a lot of discussion around this lately, and the team does try to keep on top of it.

Emphasis mine. There have been 34 posts since that comment as well, which doesn’t seem like shutting something down to me.

[steveklabnik]

It’s okay, it happens.

[phaylon]

Some of the community has. Much of the community has not. There were official responses, early in the discussions, but then it got fairly nasty, and most people on the “anti” side got tired of going in circles.

I’d like to point out that this also contributes to this being adverserial. “Some” versus “Much”. I’d also like to add that I’m on the pro side, but am also tired of going in circles. It’s also a reason I try not to participate much in language/ecosystem discussions anymore.

I don’t feel the pro side is ever going to accept any of the arguments.

Can you clarify this? This seems to be putting people squarely into two camps. But I think there are actually many sides. As in, people have different reasons for wanting namespaces, and different amounts of disagreements with the presented arguments.

Besides, this discussion isn’t going to end. A community is made up of people. If people are dissatisfied with the status quo, they’re going to talk about it. Having repeated discussions is just something thats going to happen. Personally, I feel like wasting time discussing things is less of a problem than trying to control what the community talks about. Using phrasing like “relitigation” for example, which I have seen a couple times recently, is problematic in my view. Things will often also have to be discussed multiple times before some good solution is found.

I do not know how to fix this rift.

I’m not sure they are fixable in general. We should maybe try to find ways to minimize animosity, and thus deal with the rifts instead of trying to get rid of them.

[gbutler]

I’ve come to the conclusion, as a participant in these discussions, that changing the way crates.io works is not on the table realistically. It is my conclusion to work on building an alternative that works as seemlessly as possible that incorporates the ideas that those who want name-spacing and squatting prevention want. I don’t mean that in a negative way like, “forget about crates.io, let’s just take our cookies and go home and build our own”, I mean it positively, like, “OK, if we think this would be better, let’s build it and see how it pans out. If it’s truly a better idea, it will grow, if it isn’t it won’t.”

I hope this isn’t taken as any sort of negative reaction or anything negative against the crates team or any other’s in the community. It isn’t intended as that.

[jtgeibel]

I found this paragraph very interesting:

The rate at which this user uploaded packages eventually resulted in our servers being throttled by GitHub, causing a slowdown in all package uploads or yanks. Endpoints which did not involve updating the index were unaffected.

I was not aware that our servers had been throttled by GitHub, but it makes sense given that a publish can result in multiple git interactions with GitHub and given the rate of a publish roughly every 2 seconds.

[steveklabnik]

I’d like to point out that this also contributes to this being adverserial

Yes, I certainly don’t want to suggest that I’m perfect. It’s also why I haven’t been posting as much, because I am not 100% sure how to do it right.

Can you clarify this?

What I have felt in these discussions is, well, what you say:

If people are dissatisfied with the status quo, they’re going to talk about it. Having repeated discussions is just something thats going to happen.

If people are going to endlessly discuss things until they get their way, even though there’s nothing new to say, then well, what is there to do?

Personally, I feel like wasting time discussing things is less of a problem than trying to control what the community talks about.

Different spaces are for different things. The internals forum is supposed to be a place about getting work on the Rust project done. Let’s take this discussion to a different team: what if there was several internals threads, over years, where people said “hey, let’s add GC to Rust.” And the language team says “we’re not interested in adding GC, and we’re also not interested in continuing to discuss this unless something new is brought to the table.” Continuing to discuss the same stuff over and over and over in the team’s space makes the space worse for the team to actually get their jobs done.

This is also not really “controlling” what’s being talked about. If something new is brought to the table, let’s discuss the new information! Furthermore, I’d like to point out that nobody has stopped any discussion either. What has been said is basically “don’t expect the team to respond unless new information is brought up.” There’s only so many hours in the day. I don’t personally think that this is unreasonable.

We should maybe try to find ways to minimize animosity, and thus deal with the rifts instead of trying to get rid of them.

That’s what I mean by “fix.”

[steveklabnik]

Everyone, a user in this thread asked me to delete their account, and I’ve done so. To those of you joining the discussion now, sorry for it being a bit dis-jointed, but respecting their wishes is important.

[gbutler]

but respecting their wishes is important.

Not to belabor it, but, I don’t think someone should be able to say, “I want everything I’ve said publicly purged from the discussion because I no longer want to participate”. That being said, if there are “legal” requirements (maybe GPDR?), then I guess it is what it is.

[sgrif]

I was not aware that our servers had been throttled by GitHub, but it makes sense given that a publish can result in multiple git interactions with GitHub and given the rate of a publish roughly every 2 seconds.

So basically the behavior was that any requests which touched the index started to shoot up to 10s response times. We don’t know for sure this was due to rate limiting on GitHub, since there’s not much documentation on rate limits for normal git push, but it’s the most likely explanation.