Crates.io incident 2018-10-15

[KillTheMule]

There is a clear majority that wants namespaces.

What makes you sure here, any data on that? I personally am opposed, but I’ve never said anything in any discussion.

[steveklabnik]

The only recent comment I can recall by anyone in the team was @sgrif, who said

Definitely. Nobody is trying to shut the discussion down. I’m just asking that folks try to keep the discussion above “I disagree on the conclusion that has been drawn in the past”, and try to bring new information to the table. There’s been a lot of discussion around this lately, and the team does try to keep on top of it.

Emphasis mine. There have been 34 posts since that comment as well, which doesn’t seem like shutting something down to me.

[steveklabnik]

It’s okay, it happens.

[phaylon]

Some of the community has. Much of the community has not. There were official responses, early in the discussions, but then it got fairly nasty, and most people on the “anti” side got tired of going in circles.

I’d like to point out that this also contributes to this being adverserial. “Some” versus “Much”. I’d also like to add that I’m on the pro side, but am also tired of going in circles. It’s also a reason I try not to participate much in language/ecosystem discussions anymore.

I don’t feel the pro side is ever going to accept any of the arguments.

Can you clarify this? This seems to be putting people squarely into two camps. But I think there are actually many sides. As in, people have different reasons for wanting namespaces, and different amounts of disagreements with the presented arguments.

Besides, this discussion isn’t going to end. A community is made up of people. If people are dissatisfied with the status quo, they’re going to talk about it. Having repeated discussions is just something thats going to happen. Personally, I feel like wasting time discussing things is less of a problem than trying to control what the community talks about. Using phrasing like “relitigation” for example, which I have seen a couple times recently, is problematic in my view. Things will often also have to be discussed multiple times before some good solution is found.

I do not know how to fix this rift.

I’m not sure they are fixable in general. We should maybe try to find ways to minimize animosity, and thus deal with the rifts instead of trying to get rid of them.

[gbutler]

I’ve come to the conclusion, as a participant in these discussions, that changing the way crates.io works is not on the table realistically. It is my conclusion to work on building an alternative that works as seemlessly as possible that incorporates the ideas that those who want name-spacing and squatting prevention want. I don’t mean that in a negative way like, “forget about crates.io, let’s just take our cookies and go home and build our own”, I mean it positively, like, “OK, if we think this would be better, let’s build it and see how it pans out. If it’s truly a better idea, it will grow, if it isn’t it won’t.”

I hope this isn’t taken as any sort of negative reaction or anything negative against the crates team or any other’s in the community. It isn’t intended as that.

[jtgeibel]

I found this paragraph very interesting:

The rate at which this user uploaded packages eventually resulted in our servers being throttled by GitHub, causing a slowdown in all package uploads or yanks. Endpoints which did not involve updating the index were unaffected.

I was not aware that our servers had been throttled by GitHub, but it makes sense given that a publish can result in multiple git interactions with GitHub and given the rate of a publish roughly every 2 seconds.

[steveklabnik]

I’d like to point out that this also contributes to this being adverserial

Yes, I certainly don’t want to suggest that I’m perfect. It’s also why I haven’t been posting as much, because I am not 100% sure how to do it right.

Can you clarify this?

What I have felt in these discussions is, well, what you say:

If people are dissatisfied with the status quo, they’re going to talk about it. Having repeated discussions is just something thats going to happen.

If people are going to endlessly discuss things until they get their way, even though there’s nothing new to say, then well, what is there to do?

Personally, I feel like wasting time discussing things is less of a problem than trying to control what the community talks about.

Different spaces are for different things. The internals forum is supposed to be a place about getting work on the Rust project done. Let’s take this discussion to a different team: what if there was several internals threads, over years, where people said “hey, let’s add GC to Rust.” And the language team says “we’re not interested in adding GC, and we’re also not interested in continuing to discuss this unless something new is brought to the table.” Continuing to discuss the same stuff over and over and over in the team’s space makes the space worse for the team to actually get their jobs done.

This is also not really “controlling” what’s being talked about. If something new is brought to the table, let’s discuss the new information! Furthermore, I’d like to point out that nobody has stopped any discussion either. What has been said is basically “don’t expect the team to respond unless new information is brought up.” There’s only so many hours in the day. I don’t personally think that this is unreasonable.

We should maybe try to find ways to minimize animosity, and thus deal with the rifts instead of trying to get rid of them.

That’s what I mean by “fix.”

[steveklabnik]

Everyone, a user in this thread asked me to delete their account, and I’ve done so. To those of you joining the discussion now, sorry for it being a bit dis-jointed, but respecting their wishes is important.

[gbutler]

but respecting their wishes is important.

Not to belabor it, but, I don’t think someone should be able to say, “I want everything I’ve said publicly purged from the discussion because I no longer want to participate”. That being said, if there are “legal” requirements (maybe GPDR?), then I guess it is what it is.

[sgrif]

I was not aware that our servers had been throttled by GitHub, but it makes sense given that a publish can result in multiple git interactions with GitHub and given the rate of a publish roughly every 2 seconds.

So basically the behavior was that any requests which touched the index started to shoot up to 10s response times. We don’t know for sure this was due to rate limiting on GitHub, since there’s not much documentation on rate limits for normal git push, but it’s the most likely explanation.

[phaylon]

Yes, I certainly don’t want to suggest that I’m perfect. It’s also why I haven’t been posting as much, because I am not 100% sure how to do it right.

I think it’s more that I feel we should all stay clear of trying to quantifying need or support or anything like that. I know I also have to put in more effort towards that as well. One reason is that implying one side has less support urges people to jump in who don’t really want to. It’s why I jumped in, even though I knew it probably wouldn’t be enjoyable.

If people are going to endlessly discuss things until they get their way, even though there’s nothing new to say, then well, what is there to do?

Nothing. I think that’s just a fact of life. And people on the “other side” do feel this as well. Remember discussions about how rustfmt shouldn’t be configurable? Or that Ok wrapping pops up again and again. I’ve done that for a couple of topics in the past as well.

The fact that we’re even talking about it being about someone “getting their way” seems problematic. How do you separate people trying to get their way from those venting their frustrations to other community members? In a recent criticism I posted to the subreddit, the first comment said I was making a fuss. How should I even respond to that? We’re moving away from assuming good faith on the other side. I’m sure the core teams have noticed an increase in bad faith assumptions towards them from parts of the community as well.

Different spaces are for different things. The internals forum is supposed to be a place about getting work on the Rust project done.

Namespacing/squatting threads on the subreddit have been locked with the same reasons as well, unless I’m misremembering. And in general I don’t feel like there is such a hard separation in the community in these spaces. Regular disencouragement of something on internals will lead to disencouragement elsewhere.

Let’s take this discussion to a different team: what if there was several internals threads, over years, where people said “hey, let’s add GC to Rust.” And the language team says “we’re not interested in adding GC, and we’re also not interested in continuing to discuss this unless something new is brought to the table.” Continuing to discuss the same stuff over and over and over in the team’s space makes the space worse for the team to actually get their jobs done.

This is also not really “controlling” what’s being talked about. If something new is brought to the table, let’s discuss the new information! Furthermore, I’d like to point out that nobody has stopped any discussion either. What has been said is basically “don’t expect the team to respond unless new information is brought up.” There’s only so many hours in the day. I don’t personally think that this is unreasonable.

Maybe a first step would be a change in the approach. The tone seems to me usually like “we’re not interested until there are arguments that convince us”. But the community needs to be able to talk among themselves to find these new arguments, or new value in old ones. How about instead being more encouraging towards moving discussions to the users forum, or the subreddit? Encourage the search for better solutions instead of sending a discouraging message. I’m sure core would love it if people came up with a good solution that makes most people happy, and leaves others at least not too frustrated. I think it would be helpful to signal that more. Communicate that even when the discussions (in the approproate places) don’t work out, core is still happy the community is involved and trying.

To people on the other side, the responses as they currently are can seem dismissive. I think a more positive approach here would also help to funnel things to where they can be discussed, and have a more positive community wide attitude towards those discussions.

So, instead of seeing it as people trying to get their way, we should maybe all try to see it as people caring about issues even after they’ve been shut down a couple times. For example, when Ok wrapping or implicit Ok(()) insertion is brought up once again, I think the proponents would like the opponents (including me) to assume that they care about ergonomics instead of thinking about it as “getting their way”.

That’s what I mean by “fix.”

One solution I would see, but I don’t think would be liked by the core teams, is to switch from a “one way that fits most” to a “try to find a design that gives solutions to the maximum number of use cases”. I think I brought this up in the past. It just feels like in recent discussions, it’s already clear at the start that one side has to win, and that there can’t be a middle ground.

That is, have less of a streamlined, opinionated approach and more that of a neutral platform. Both with language and ecosystem design. That wouldn’t preclude best practices, idiomatic encouragement, ergonomic defaults and so on.

In the squatting context, that would be trying to find a way for serde to be an easily discoverable top-level project, but winapi not having to reserve every possible Windows library name in advance to maintain consistency. In this specific case, I don’t think anyone even has a good picture of the whole situation and people’s needs yet. It might be fruitful to put together a survey to figure out why people want/don’t want namespaces, what problems they want solved, both from crate publisher and crate consumer standpoint, and then use the survey results as basis for further discussion.

Sorry that this got a bit longer.

[mark-i-m]

And the language team says “we’re not interested in adding GC, and we’re also not interested in continuing to discuss this unless something new is brought to the table.

Maybe I’m wrong, but my impression is that the topic of squatting came up again because it started to be perceived by a non-trivial subset of the community as a growing problem that is becoming a real pain point.

Perhaps a discussion that needs to happen is “Is the rust community willing to re-discuss decisions later? Or are we taking the stance that once we have made a decision, we will own it and see how it plays out?”.

Personally, I think both sides have merits. Re-discussion can help address concerns once we have more information about problematic outcomes. At the same time, if the discussion doesn’t go anywhere, it can just be demoralizing and exhausting for everyone.

[steveklabnik]

I’m about to go to lunch, so I’ll be making more substantial replies later, but I absolutely think that this lies on a spectrum. I chose an extreme example on purpose, I’m not saying that this specific topic is that extreme. Even the 2014 post acknowledged that we’ll feel these out over time.

[steveklabnik]

Sorry that this got a bit longer.

It’s cool! No worries

So, I thought I had a lot to say to respond to you, but now that I re-read it carefully, I’m not sure I do I don’t necessarily disagree with much of what you’ve said.

I do have one small thing, I guess. I don’t even think that some of the discussions need to be moved from internals. Nobody is advocating closing the namespacing threads, for example. But I do think there’s mis-matched expectations about who is supposed to be participating in them. Sometimes, people do participate in internals, but it’s really a more free-flowing space. With limited time, many team members don’t read internals. Since the RFC process is the way to actually affect changes, they stick to that, which is their job. But it seems like some people in these various threads read this as somehow “ignoring” the discussion. Or that the team “isn’t listening.” But that’s because these things aren’t being asked in the right place! Or rather, this is the right place to discuss a bunch of stuff, in order to prepare the work of an RFC. But if you really want a response from the team, posting an RFC or emailing them is a better way to get them to notice than assuming they’ll reply on internals. This is mostly just due to the nature of Rust’s growth; not everyone has the time to keep up with every single possible conversation that might be relevant to their team.

[ahmedcharles]

When I originally read about the incident, I thought there was ‘hacking’ involved, rather than simply creating an account with a disingenuous name. Perhaps in future announcements, this distinction could be given more clarity?

[SimonSapin]

You can split hairs about what qualifies as “hacking”, but pushing a system to the point that it becomes unresponsive for everyone is definitely unacceptable abuse in my opinion. From the blog post:

The rate at which this user uploaded packages eventually resulted in our servers being throttled by GitHub, causing a slowdown in all package uploads or yanks.

• 20:17 UTC: All requests updating the index begin to take 10+ seconds

[kornel]

OTOH a rate of 1 request per 2 seconds is not that much. It’s good that cratesio is getting rate limiting.

[ag_dubs]

i’m replying here for the purposes of clarifying/correcting some of the vibes that folks appear to have gotten.

i am on the crates.io team. this team is very very new. it formed when @sgrif and i were dealing with an incident in April around the project and realized that we clearly needed more focus and support for the service.

i have seen some sentiment as follows:

• the team will only act when it is convinced
• folks should just fork crates.io and make their own service
• we aren’t willing to “rehash” decisions that are already made

i’d like to correct this.

1. we make decisions exactly the same way all the other teams do: RFC -> consensus. there are a lot of emotions here and i think there’s been some confusing comments from everyone.
2. please join the team’s meetings (every thursday 4pm ET) as an observer. you can ping me personally or ping crates-io-team on discord. we also have a channel, ops/#crates-io, we also have an email, help@crates.io. joining as an observer is the first step to joining the team. you can fully participate in our conversations.
3. auditing our current policies and their effectiveness is on our list of goals for the team. we will indeed be discussing them. that may lead to change, or it may not, but i want to be clear that we think talking about it is Actually Good. though, much of the way it has been discussed on this an other forums has made that very hard. (if we spend all our time reacting we cannot take proactive action) - that being said, i do not think a universe where we consistently rehash policy is a good (or possible) one (time is not infinite), i think we can schedule audits, probably on a yearly basis to see if things need to change.

we are a small team currently. we’d like to grow. we cover so many different aspects of the crates.io service. there’s lots to do. getting aggro here is not useful, and i direct that to everyone involved. rust moves slowly and deliberately and we have literally never made policy based on a collection of heated threads from forums and that is not gonna change for this specific request. please take this energy and channel it into constructive participation in the team process.

[ahmedcharles]

My point was about accurately communicating the situation, not about whether it was abusive.

[steveklabnik]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.