Crossbeam: request for help

[aturon]

It’s probably not news to anyone that crossbeam is effectively unmaintained right now. I simply do not have the time to keep up with it.

At the same time, i think these kinds of concurrent data structures are very important, and would love to see continuing work on the library.

This is a plea for help. If you are interested in getting involved in the maintenance of the library, please let me know on this thread. Ideally, we can get a handful of people together to share the load.

[steveklabnik]

One thing that’s related; I would really like to see a “scoped threads” lib get into the nursery. It’s a shame that this pattern is so widely used, yet we’re not working on stabilizing it.

That doesn’t mean it has to be crossbeam or that it should use crossbeam’s scoped threads exactly, I know there’s a few competing crates. But making this more broadly known would be a big deal.

[aturon]

Agreed. I think this part of crossbeam should be moved out into an external library (or maybe just deprecated in favor of one of the existing ones).

I think the libs team would be quite open to putting some such library in the nursery. Just need someone to propose it.

[arthurprs]

I’d like to help @aturon. I really think crossbeam could turn into a defacto building block in the rust ecosystem.

[aturon]

I’m delighted to hear that! After we wait to hear back from others to join is as well, I’ll get the ball rolling on handing over the reigns.

[kbknapp]

Not strictly crossbeam related, but is there a single place where we could list crates who are effectively unmaintained? Perhaps a section in TWiR? I’ve got a few crates too that I’d like to offer up.

[tupshin]

I’d also like to help. I’ve spent a lot of time in the distributed systems space, of late, and would love to see crossbeam be one the foundations of an ecosystem of parallel and distributed tooling.

[Vtec234]

I am really happy to hear this I have several PRs on crossbeam which are effectively stalled right now, but with more maintainers we could get the library going again.

One person who would be definitely interested in this is @ticki who just forked crossbeam yesterday. So, @ticki, would you be interested in becoming a maintainer of crossbeam instead and merging baguette back into it? Fragmentation is rarely beneficial.

[steveklabnik]

I think the libs team would be quite open to putting some such library in the nursery. Just need someone to propose it.

Great! i’ll put it on my (totally not already full) list of things to do

Not strictly crossbeam related, but is there a single place where we could list crates who are effectively unmaintained? Perhaps a section in TWiR? I’ve got a few crates too that I’d like to offer up.

There has been several discussions about this, but nothing has really taken off.

[jonhoo]

I’d also be happy to help with what limited spare time I have

I’ve already been working on a bunch of related projects like bus (a lock-free broadcast spmc queue), evmap (a highly concurrent HashMap), guardian (convenience type for working with Arc<Mutex>), and arccstr (lower overhead reference counted strings). Am also playing around with ideas for a scalable, bounded, lock-free mpsc queue for use with futures-rs.

[QuietMisdreavus]

I probably wouldn’t mind getting in on this. My head’s been in synchronization territory lately anyway with synchronoise, and I’d love to work on more stuff like it.

[ticki]

Oh, this is nice.

I’m currently doing major refactoring of crossbeam over at baguette (code name).

I’d like to merge upstream, but it is a huge piece of code change.

[ticki]

Maybe it’s better to keep it separated. I’m not sure.

[aturon]

Wow, thanks everyone for the amazing response! I should have done this a long time ago.

I think the best way forward is to move the repo to a dedicated crossbeam org, where I can add the volunteers as admins. I think it’d be good to use this thread to also get on the same page to any deeper changes people want to make (like @ticki and @stjepang, at least).

Any objections to that? Otherwise I’ll move it over shortly.

[tupshin]

Given the interest in growing the maintainer activity of crossbeam, I’d love to get us started off on a unified front.

Any chance of documenting the rationale and goals of your refactoring, and allowing @aturon (and other interested parties) to review and discuss, and if consensus is achieved, base a next major crossbeam release around it?

[ticki]

Okay, so I got increasingly fustrated with the API, and I wanted to make a fork at least temporary.

My intention is (and was the whole time) to eventually merge upstream. As there are some breaking changes, it should be seen as a major-release candidate.

Documentation (80% complete)

• Document every part in details, including private items: People familiar with my style know that I take documentation very seriously, and my crates generally follow a principle of complete documentation including private items. This is because I believe that no code is self-explaining.

• Include comments for everything non-trivial. By my philosphy, any complex or semi-complex piece of code should be written in a semi-literate style, where a comment explains the idea behind a statement.

API changes (90% completed)

• The mem module should be eliminated in favor of having the root providing the epoch module directly. This has the purpose to make the import lists shorter.

• cas and cas_ prefixed methods are renamed according to the issue I opened.

New features (completed)

• Pinned<T> for having an owned guard.
• Actual compare_and_swap.
• map() for Shared<T> allowing internal changes in the reference without some inconviniet third-party dependency like owning_ref

Removals (not started yet)

• Remove epoch::Owned<T> in favor of Box<T>.

Refactoring (complete)

• Use lazy_init instead of manually reimplementing it.

Performance

• I’ve eliminated certain atomic instructions, which can potentially lead to a (small) speed-up.

Bug fixes

• I have found some quite severe bugs in the handling of participant threads, but the bugs are incredibly rare to be triggered (in fact, some of them are only theoretic so far).

Other insignificant changes

• Add Default to all the types.

[Vtec234]

Since your changes are breaking already, you might as well just remove the cas family of functions instead of deprecating them. If we want to merge these improvements to the API, I think we should do it as soon as possible, since all following additions will have to be based on the new names. Anything non-breaking that you want to do (documentation, trait impls) should probably be delayed in favour of breaking stuff for this reason. This should not be a problem since it looks like most of it is done already.

@aturon: how about a general low-level concurrency org? crossbeam is a good name for it, but maybe it could include more than just that crate.

[bstrie]

+1 to fast-tracking a scoped threads lib into the nursery. The original API was only moved out of std because of the leakpocalypse, not because it wasn’t useful (the old broken API is still a cornerstone of the original “Fearless Concurrency” post, to much chagrin…).

After seeing how the Rust team was able to rally support behind Tokio by reaching out to the authors of alternate libs (tailhook, dwrensha) and coming up with a unified solution, I’m hopeful that we can do the same thing here.

[bstrie]

I’d suspect that a general concurrency org would be overkill, since we’d hope that the current situation is just a stepping stone to rust-lang-nursery. Other libs (e.g. Rayon) might not need such a stepping stone. Though I suppose it’s worth asking Niko if he also needs help…

[Amanieu]

I think scoped threads should be deprecated from crossbeam since they don’t really fit in well with the rest of the library. Instead crossbeam should focus more on concurrent data structures and memory reclamation, including data structures such as RCU-lists and RCU-hashtables.