Rust's responsibility to engage in good neighborly behavior

1

I'm following up on Cargo issue #1734, Use OS-native config, cache home directories, to discuss the pathological nature of the issue and the ensuing discussion and see if there are opportunities for Rust to take away lessons from it.

This issue is distinct from most feature requests and bug reports in that it disproportionately affects bystanders, which is to say, people who don't use Rust themselves but who are indirectly affected via being a user of some software written in Rust. Inversely, people who use Rust every day are more likely to be tolerant of this problem. As a consequence, the people most affected by the issue are least equipped to participate in a solution (they lack knowledge and skills with Rust programming, experience with the Rust community's norms and practices and processes, they don't know how to write RFCs, and so on), and those least affected are most equipped to solve the problem.

Issue #1734 shows us the outcome of these incentives, which is that ten years later the issue remains unsolved and the "discussion" is a graveyard of passerby who are affected by the bug and upset with Cargo's lack of initiative to fix the problem having their remarks marked as off-topic, spam, or disruptive, with a predictable escalation of tensions and rhetoric.

In my view, this is an example of violating the principle of good neighborly behavior. Concerns internal to the Rust community are the Rust community's problem, and it is effective to move Rust forward to encourage participation from those affected by a problem. But, Rust's place in the public square is the Rust community's responsibility, not the public's. Issue 1734 is about how Cargo treats $HOME, a kind of public square, and the Rust community, not the people external to the Rust community who are harmed by a bug, should be responsible for being good neighbors in the public square.

To insist that non-Rust-users who are affected by Cargo's poor neighborly behavior should be responsible for learning Rust, and its processes and procedures, to commit their labor to fixing Rust just to get one bug fixed and move on -- it smells like victim blaming.

It is my view that Rust as a community should learn as a value that its behavior towards others is a matter of internal priority, and that issues like this should be resolved promptly and, crucially, by those responsible.

18 Likes
2

Would thoughtlessly shipping a breaking change to cargo's behavior be good neighborly behavior, then, or are you content to wait until a migration plan is figured out and enacted?

3 Likes
3

I don't think that sarcastic jabs are appropriate.

Nevertheless, however long it takes is however long it takes, but it ought to be prioritized and I find it hard to believe that this has been a priority for ten years.

3 Likes
4

I believe you are imagining a continuity in organization over those 10 years that did not actually occur, and rather that there were sharp discontinuities in membership where large portions of the contributor base... and thus people who had their eye on various things... ceased being the same people. This has a variety of causes.

Obviously some of the same faces are still around, but they have their own priorities and in most cases have had many, many, many things to keep track of and prioritize. Something like this slipping through those cracks is kind of inevitable. Often, people working on issues like the one you are talking about have to do archaeology to even figure out what happened, because often the person who was initially responsible for the area that had the problem was themselves overworked and has often stopped answering messages.

I also am not sure our priorities would be in the right place, necessarily, if this came first. For example, our compilation for various platforms has been pretty busted. And by busted, I mean in ways that make those targets require much more than 2 lines of shell script in workarounds, or in some cases completely unusable. I've spent quite a bit of energy on improving that, and will spend more. Should I have dropped the task of making software work on computers it didn't work on in favor of fixing this odd inconvenience that is functional but irritating?

4 Likes
5

I also am not sure our priorities would be in the right place, necessarily, if this came first. For example, our compilation for various platforms has been pretty busted. And by busted, I mean in ways that make those targets require much more than 2 lines of shell script in workarounds, or in some cases completely unusable. I've spent quite a bit of energy on improving that, and will spend more. Should I have dropped the task of making software work on computers it didn't work on in favor of fixing this odd inconvenience that is functional but irritating?

Probably! Because the work you're doing for fixing compilers on some platforms is work to improve the experiences of Rust users, and the work needed to fix the cache directory is work to stop harming others. Again, it comes down to the principle of good neighborly behavior.

1 Like
7

Are you sure? Are you sure you want to say that if we ship a compiler that vomits out garbage artifacts, that's not harmful, but dropping a few things in the $HOME directory and asking you to set an environment variable if you want it elsewhere is?

7 Likes
8

Yes! That's my point exactly. A compiler which vomits out garbage artifacts (for presumably niche user configurations, to be certain) affects Rust users. Putting things in $HOME affects everyone else. You have a responsibility not to be a nuisance to everyone else.

9

As someone using Rust very regularly, but on mostly stable and well-supported platforms, I find my biggest gripes with cargo to indeed be with how it handles files. Sure, I have my shell set up to throw things into ~/.local/share as is proper, but cargo's 'good enough' approach to files and directories in my experience extends to other things too. Like the target directory being a mostly innavigable mess (but build script output being such that you often do have to try and go digging for build outputs), many missed or overdue opportunities for deduplication and GC, and odd conventions like project-specific .cargo/config.toml files that are crucial to proper compilation on some targets I use (e.g., embedded), but are relegated to a hidden ("dot") directory.

As someone who isn't a contributor outside of occassional bug reports I don't think it's really my place to say "I think Rust should be focusing on this and that", but I agree with Drew in that cargo/rust could be doing many things better with respect to integrating into my system and organisation.

PS: as an addend, I know this sort of thing also comes up a lot when people are dealing with containerisation (docker) or in CI contexts.

9 Likes
10

Ah! I guess nobody uses Rust in distributed software, making my concern irrelevant, I suppose.

Regardless, these things don't have to be set up to be rivalrous to each other. There's room to fix small problems and also larger ones. I asked the question essentially rhetorically. The real answer is that people work on what they feel motivated to work on, that's kinda how it goes with FOSS. As a result, I just don't think it's very surprising if people don't feel very motivated to work on this topic and work comes at a snail's pace.

If you think "those responsible" should fix this, well, I'm not sure who those are, according to you. If you think "those responsible" are the people who started jamming stuff in $HOME, then you have to go find the people who initially implemented these tools and badger them into fixing it. And as I've mentioned: they often aren't returning calls. That's exactly what I meant when I observed there have been effective discontinuities in membership.

2 Likes
11

It is worth clarifying here that "everyone else" here means software developers. I suspect you implied as much, so forgive my nit. Ordinary users are lucky if they even know dotfiles exist.

I acquired a small dose of Stockholm syndrome long before I knew Rust, when I learned I had caches spread across .opam, .cabal, .npm, etc. So it is a little strange to characterize cargo's issue as disproportionately affecting bystanders, trust me that even the cargo developers are annoyed and pained by this too.

13 Likes
12

It is worth clarifying here that "everyone else" here means software developers. I suspect you implied as much, so forgive my nit. Ordinary users are lucky if they even know dotfiles exist.

Not really. I think that's demeaning towards a lot of Linux/Unix users. Not all of them are software developers but plenty of them understand what dotfiles are.

If you think "those responsible" should fix this, well, I'm not sure who those are, according to you. If you think "those responsible" are the people who started jamming stuff in $HOME, then you have to go find the people who initially implemented these tools and badger them into fixing it. And as I've mentioned: they often aren't returning calls. That's exactly what I meant when I observed there have been effective discontinuities in membership.

Sure, or perhaps the present-day maintainers and leadership of cargo? Certainly not the victims of Rust's poor behavior.

13

Also -- I know how it works in FOSS. I am saying that the project leadership should recognize this pathology of the usual approach and take responsibility for addressing the problem.

1 Like
14

If anything there is a category of folks you'd call "normal users" who in my eyes are software developers. If they can interact with a CLI and write simple shell scripts, they have graduated beyond an ordinary user.

15

A few thoughts here (speaking individually only, and not stating a position of any team):

Note for context on this message: I am, in fact, a big fan of the XDG directory layout. I have, in the past, attempted to further this work myself, and discovered some of these details empirically. (Others have done much more and gotten far further.) So, take that into account when considering the explanations below.

I'm going to venture a guess, here, that many folks are not aware of the amount of time that has gone into this issue over the years. It's easy to see a ten year old issue, even with various discussion on it, and treat it roughly as if it idled for a decade, or hasn't been considered important, rather than that:

  • It has been a regular topic of discussion.
  • People have regularly tried to address it.
  • It's harder to fix than you might think.
  • It is not higher priority than backwards compatibility.
  • Backwards compatibility is more complex than you might think.
    • Backwards compatibility includes people who switch back and forth between different versions of cargo.
    • Backwards compatibility potentially includes not just people who have cargo configuration, but also people who have a large cache of downloaded files and repositories, and whose bandwidth is not such that they can trivially re-download everything.
  • Compatibility needs to take into account that both rustup and cargo have come to share some common directories, which makes it difficult for a version of either to unilaterally move those directories.
  • This issue is extremely prone to scope creep.
    • For example, as long as someone is working on XDG directory layout support, naturally their design should be expected to fully handle the "correct" directory layouts on Windows and macOS and any other OS Rust and Cargo support, right?
    • Not everyone even agrees on what the "correct" directory layout is on every single OS that Rust and Cargo support.
  • Multiple people who have attempted to address it have put substantial effort in before bouncing off.
  • Nobody involved thinks it is a hair-on-fire priority to address, even those who are enthusiastic about the XDG directory structure or the equivalents on other OSes.
  • A huge fraction of the work required here is design and testing, not just showing up with a patch.
  • This is not anyone's top priority, nor is it on anyone's roadmap.
  • People showing up who come across as "go faster, you're being a nuisance, you're not a good neighbor" have an effect ranging from discouraging to actively taking time away from this and other work.
    • It's particularly frustrating when attempts to actually engage with such responses and provide context get treated in a manner that makes people regret making the attempt.
    • It's even more frustrating when such responses express indignance at being subject to moderation for not contributing in any way to the situation, despite the fact that they do not contribute in any way to the situation.
    • No such messages will at any point have a positive effect, let alone an effect aligned with the goals of the people sending such messages.
  • Many, many issues in Cargo have a substantial impact on the ecosystem, both for interoperability with third-party software and for impact on Rust users. It is not, in fact, the case that this one issue and this one issue alone is in the top few issues by importance even if you treat "impact on other people / other people's software" as the top priority and discount everything else.
  • Despite all of the above, there is in fact a current effort working on this issue, which has a good shot of succeeding.
54 Likes
16

For what its worth, I have been working on the weekends to try to improve this. (see #15010, #14125, and #5931 for example).

Also to add to @josh's points, often seemly simple features need many other tasks to be done before we can even start implementing them. I have learned that is especially true for Cargo+the filesystem recently while trying to make progress on improving the target dir.

15 Likes
17

Thank you for going into this detail.

Careful consideration of the impact of changes is a valuable approach, but I don't think that people are pushing for mindless action.

People also don't push for a solution today, next week or even this year. Patience runs thin, but it's been going for a decade, so give us a little slack.

Regarding the complexity of the change: I don't claim to understand all the gotchas, but the thing people desire most is to get ~/.cargo out of the way, and maybe to put reproducable stuff below caches. That can't be so complex. But even if it was, there are a lot of ways to mitigate the impact.

Firstly, the defaults can remain the same in the absence of environment variables. XDG only needs to be considered if a user explicitly sets the XDG_... variables. That allows users to opt in to the preference of XDG without having to know anything about cargo or rust. These variables are rarely set by distros by default, at least none of the distros I use do that. Current CARGO_... variables can have precedence over XDG defaults. Just this simple strategy would make it so that the vast majority of users and non-users alike would not see a difference.

This would not entirely prevent problems resulting from incomplete or buggy implementations, but it would reduce the impact.

You could go fancy and migrate data between the two concepts, merge it if both are present or get the categorization of data stored in $HOME perfectly right at the first attempt. I can see that this would be hard. Perfectionism is always hard. But those people who complain about .cargo do not seek perfection. Those who don't care or prefer .cargo - well they don't care.

As for third party software: The longer it takes for a solution to appear, the more third party will use the current implementation and add to the problem until finally it becomes practically impossible to change. Nobody in their right mind would relocate ~/.ssh today.

I think there is a balance between careful consideration and the ability to change something. If backward compatibility is an absolute goal, then there is no need for deliberation over this issue.

Maybe I'm oversimplifying things here, but I'm working in this field for more than three decades and this really doesn't look like a problem that is so incredibly hard to solve.

The biggest problem is probably third party software, but there is little you can do about it, other than not doing anything. This is a valid choice, but if it's the best choice, you should at some point in time just make it. Bash and SSH will never comply with XDG. I would prefer if they did, but I can understand and accept that they don't.

It's difficult for outsiders to understand the intricacies of a piece of software. But what's new about this? I can't count the occasions when I was confronted with bug reports or "management" asking me why it's so hard to get this or that thing done. They are more often than not impatient and upset. I never took that personally, because it's a consequence of different perspectives. Why would you be frustrated and "regret to have made the attempt"?

If contact to impatient users is frustrating you, then I guess it's more productive if you would not engage with them, because impatient is what users are by their very nature. You have a sufficient number of users such that you will be confronted with each type of user. How is this surprising? Why should this be any more frustrating than to cope with bad weather?

When I read the various reactions to complaints about this issue, then the condensed summary is this: "be polite", "be patient", "we know what we're doing, better than you". Okay.

But while this communication is ongoing, the behavior of your software is perceived as an annoyance to some. These guys can't fix the problem themselves. They don't see how patience is helping them. They feel helpless and that feeling creates tension that has to go somewhere. The most natural and least misguided outlet for such tension is to write grumpy comments.

That's not helping anybody, but neither is "be patient". How is all of this not a perfectly normal situation arising from an unresolved conflict of interest? Whenever you ignore people who depend on you (for whatever reason), they're going to react unfavorably. Again, what's new?

3 Likes
18

I don't see this as "Rust users aren't affected and aren't incentivized while non-Rust users are affected". There are many Rust users who are affected. It's a question of workflows and care abouts which is independent of what tool a person is using. I'm on the other side of the fence and find the use of XDG to be net-neutral on my workflows but I understand its not that way for others and have put in work investigating this but it takes focus-time to work through the details which I'm short on due to other commitments I've previously made.

As for the lack of timeliness, in addition to the compatibility intricacies some important context

  • The Rust project is not currently set up for top-down decisions to prioritize work like this.
  • The Cargo team was understaffed for a while and was in a soft feature freeze for years as it worked to rebuild and reduce technical debt that slowed them down
  • wrt XDG, Cargo is tightly coupled to another tool, Rustup, which is distributed independently and is managed by another team that has been understaffed for a while
  • Since the loss of Mozilla and COVID, this May will be our first in-person, project-wide meeting to help with some of these cross-team work (I've already put XDG on my task list to discuss). We've also had some changes in governance (and a large governance backlog) that has meant there has been little attention to improving cross-team interactions outside of in-person.
  • Even when I say "understaffed", this is for maintenance and design sign off; these teams are the decision makers and it is not sustainable to expect them to exclusively be the ones to implement everything.
  • There are many competing demands on these tools, internal and external. For example, a security improvement has also been stalled out for years, see Cargo and Rustup safe file discovery. by ehuss · Pull Request #3279 · rust-lang/rfcs · GitHub
24 Likes
19

Would people be upset if you would close the ticket with the explanation that the project simply doesn't have the man power to implement it?

You don't need users complaining about XDG violations in order to implement compliance at your leisure.

I am certain that a large part of the frustration results from this situation resembling "waiting for Doors of Stone".

Everybody understands a lack of time and resources. But if you argue that this problem is so complex that 10 years is not enough, that's an entirely different message.

A lot of the drama we are all experiencing here, if from different sides, is due to us communicating badly. The whole complex of meta discussions vs. technical issues is another example. If people like me cannot understand what the technical hold-up is, then we assume that there is some problem on the meta level.

The problems the two of us had is special in that I am other than most not a Rust user. For me, there is no counter value. I don't want to learn about how to configure cargo because I'm not using it. You can expect more patience and tolerance from your users, because they get an immediate counter value.

But even if I'm wrong and my complaints are invalid, both in style and substance, marking posts as off-topic and hiding them is creating more noise and disruption than to answer to them or just ignore them.

I have the impression that your (the project's, not your personal) communication style is making your life unnecessarily hard for you and extremely frustrating for users and affected parties.

This might be just as true for me and my style, but I can just walk away frustrated and nothing changes. I will still have to keep cleaning up my home directory, even if some day cargo will behave. This is actually the outcome no matter what I do.

3 Likes
20

Are you seriously claiming that the only alternative to the current status quo is "thoughtlessly shipping a breaking change"?

Here is a mention of an unresolved security issue. Would a security issue justify a thoughless change more than "good neighborly behavior"?

You committed three fallacies in a single sentence here.

You implied that the only way to fix this issue in a timely manner would also be irresponsible.

You implied that good neighborly behavior is not important.

You tied the two together trying to make the guy presenting his argument look stupid.

Now how is that nice and respectful? This guy listened to complaints on the issue and moved the discussion to a separate channel. He spend time writing up his opinion. He did a good job. The only mistake he made is to have an opinion that doesn't match yours.

Wouldn't that justify an apology?

21

IMO it would be correct for project leadership to prioritize things that its products' audiences are asking for, so I don't think this solves your perceived imbalance in "responsibility taking".

Those people still have the option of not using any software written in Rust if they experience so much harm from a few files placed in places they don't appreciate -- or if that is too much of a cost to bear, maybe the current state isn't so bad after all.

I'm sympathetic to the notion that it sucks that this issue hasn't been fixed for 10 years, and I'm certain that that wouldn't have happened if a few people were paid full-time to work on Cargo/Rust tooling. As it is, there are unfortunately limited resources available to improve Rust tooling (especially relative to the size of their audience) so progress is slow. At the same time, I don't think it's accurate to say that no progress is being made on these issues, although unfortunately that progress hasn't been captured very well in the original issue.

3 Likes