after glancing at them and scoped_threadpool today, I’d agree; the full implementation of scoped_threadpool is smaller than the scoped threads impl in crossbeam alone.
Of course, that’s not always correct, but given that scoped threads are more something you build with crossbeam, they feel like the odd thing out.
Ok, all, I’ve created a crossbeam-rs org (unfortunately crossbeam was taken), and moved the repo there: https://github.com/crossbeam-rs/crossbeam
I’ve also added everyone who volunteered as an owner; you should have received an invite. Let me know if you didn’t, or are interested in being added (did you want to @Amanieu? I know you’d be a great help.)
Obviously we’ll need some level of coordination on the overall direction and management. FWIW, I think that @ticki’s changes look just fine, and I’m not too worried about breakage via new major versions for the core epoch APIs; existing code can continue to pin to the old version and work just fine. There’s no interop concern.
I really have essentially zero time to spend myself running the project; it’s hard enough to keep up with my Rust and Tokio work. But I can try to help at least set up the basic project management structure, if people would like that.
I think that @anon19897381 raises some very interesting questions. I certainly agree with the overall priorities, and at least some of the proposed changes. The issues around object destruction are indeed tricky, and have long been an open question for crossbeam; I’d love to get this resolved.
Regarding exposing pinning – I think that it’s great to see how far we can get while keeping it completely hidden. My hope was that we could offer something like the refcounting API you mention, with a separate API surface for getting your hands on the pin.
This latter question is more to do with the data structure APIs, rather than the core epoch APIs, so it should be quite easy to experiment with either on top of or within crossbeam, without too much risk.
I think, more broadly, it’s fine to treat crossbeam as a highly experimental crate – that’s very much what it is. As you say, it’s going to take a lot of time and effort to really build out this story for Rust. But I think we can get there faster by collaborating on a common foundation.
As a brief follow-up to this, I think implementing some better channel primitives for highly concurrent situations could also fall within the scope of crossbeam. Both mpsc, spmc, and spsc can be implemented with very efficient implementations. My personal target is bounded channels, which turns out to actually be quite tricky (no more CAS linked-lists), especially if you want to combine it with things like futures-rs, but non-bounded implementations would also be useful.
I can help out with Crossbeam. I have some experience implementing a lock-free queue https://sstewartgallus.com/git?p=uevents.git;a=blob;f=README;h=cb0c3a32d568a381d7ea0fe3c04a4ef1f883e5ac;hb=HEAD
I would be interested in contributing. As a start with a non-blocking concurrent queue that I wrote a while ago. It’s at https://github.com/johshoff/concurrent_queue
I could honestly use some help getting it to a finished state as memory reclamation is not yet implemented. I was actually pointed to crossbeam for solving it, but it seems like hazard pointers could be a lot faster. If anyone wants to help out I’d be delighted. I also don’t know if I’m suited to be a maintainer until I grok hazard pointers 
I have to disagree with you here, there are very good use cases for delayed object destruction. One of the main use cases I envision for EBR is something like RCU in the Linux kernel: data structures (linked list, hashtable) highly optimized for read-mostly workloads. This requires that objects remain valid and unmodified while there are still threads reading them. Since write performance is less of an issue, concurrent writers are guarded using a mutex to keep the read path fast.
I haven't had much time to work on my Rust projects lately, but I would be happy to help out if I find the time!
As a side note, if we're considering refactoring the API, could we maybe rename Atomic to something more sensible like ManagedAtomicPtr? The name Atomic somewhat conflicts with my atomic crate which provides a generic Atomic type.
@aturon I’d be glad to start helping again - I previously wasn’t able to contribute a whole lot due to work hours but I have much more time now. I like the idea of crossbeam being an umbrella organization which collects good concurrent data structures/memory management. It already feels a bit stuffed without everybody’s pet datastructure in there.
@jonhoo, I share a similar vision about having high quality bounded channels and having some go-to broadcast queue in the ecosystem. Both you and I have published them, bus and multiqueue, and there’s at least three less-maintained others in the rust ecosystem. A proper multicast queue would be really cool as well.
My other big things I would like to see in crossbeam are
@schets and @Amanieu, you’ve both got invites.
Using the crossbeam-rs org to house repos with data structures built on top sounds great to me! I’d be fine seeing the crossbeam crate itself focus purely on memory management, with particular data structures moved out. It’d be great to explore additional strategies like flat combining and hazard pointers as well.
I like this approach too. It’s not clear that having a single crate with all the data-structures in it is the right move. A more modular approach with crossbeam providing cross-cutting primitives seems like a good path to explore. Would likely encourage both better documentation, better modularity of design, and a better ecosystem over all as well.
I’m happy to pitch in.
Hey there @aturon, I’ve been following the rust ecosystem for a while now but I still feel I lack the skills to directly contribute to the codebase. That said, I’d love to pitch in and help with the documentation of the project at the very least.
I'm in the process of writing (and am almost complete) an atomic hash map.
If I was forced to compare it to a design described in a published paper, it would be Feldman's, but even that would be inaccurate.
Mine is faster than Feldman's, but slightly more memory hungry.
At some point, I will make a blog post describing it in detail, but for now, refer to the code (ticki/tfs).
I hope I can complete a 1.0.0 candidate today.
A lot going on here, but IMHO: @ticki proposals look good and +1 for removing scoped threads from crossbeam.
Unless I am heavily misunderstanding something here, this is simply not true! The necessity of having 'static bounds does not stem from structures referencing each other. Even in a single-thread, one-structure context, non-'static data can reference data local to the current function call's stack. When the function finishes, and another one called on the same thread, on the same structure, tries to run the destructors, it will reference an invalid part of the stack.
There is no necessity of forcing the use of deferred destructors of people. We can still leave the old "just free me" API there. I also disagree with the statement that it "pervades the whole framework". If you look at the implementation here: #57, it's literally just a few lines in the garbage recollection code. In fact, from a quick glance at your own crate (epoch), it seems like it could be added there in a similar way without breaking anything. Please correct me if I'm wrong and it breaks some invariants in your EBR implementation, but it seems that it would only affect recollection times, not correctness.
Looks like I didn’t have the org set up to allow write permissions by default. Should be fixed now, @arthurprs if you want to go ahead and merge PRs.
We’ll also need to add owners to crates.io for publication, and ultimately work toward some leadership/oversight structure here. If people have thoughts on that, let me know!
Could we please wait with merging PRs until my release candidate lands? I’m afraid that the merge conflicts will be a mess.
The review system allows us to queue things to-be-merged and then later merge them.
To add, I except the candidate to be ready this afternoon.
I’m really tired. I’ve been working on this for much longer than I expected. Around 20 hours now.
@ticki: Honestly, if you are tired you should probably have some rest given how notoriously hard concurrency is and the fact that you are more prone to make mistakes when tired. Your changes in baguette are already so extensive that time is needed to review them anyway even if you were to PR them today. I think we already agreed on holding off all other PRs to move the merge conflict burden onto them.
Given the intrinsic difficulty of concurrency, I am of the opinion that nobody should merge their own PRs, and each non-cosmetic change should get at least two reviews from people with good knowledge of the topic at hand. This might seem like much, but we really don't want to introduce ordering bugs that only happen on weakly-ordered hardware and a specific IS, which could be spotted with a formal analysis of correctness.