You need to also brand the allocator with the lifetime. Moreover, you can't let the user choose the lifetime of the allocator because this allows choosing the same lifetime for different instances. Instead you need either a closure based API that requires the closure to be valid for any lifetime (this is how ghostcell does it for example) or some weird macro based API (I remember seeing it somewhere, but I don't remember how it worked). Here you can see an example. Unfortunately this has a bunch of problems, like not being ergonomic and abusing lifetimes for something they weren't designed for.
I think that effect isn't a good name for this feature, because the proposed feature models just one kind of effect: implicit parameters, which is called the Reader monad in Haskell. It's true that languages with first class effects, like Koka, can use effects to implement implicit parameter passing, but this proposal doesn't address effects in general.
Also, Rust already has many kinds of effects. Some are reified as types, some accessible using only language constructs. Off the top of my head, there is: async/await (reified as Future), try/? (reified as Result), unsafe, panics, and probably a few others. I would expect that any effects feature to unify at least a bunch of those.
5 Likes
Also, there is a concern about shadowing\overloading:
struct St;
impl St{
fn new()->Self {...}
}
impl St with alloc: Allocator {
fn new() -> Self {...}
}
How should disambiguation work here?
disallow this? that's a blatant attack on coherence!
... suppose you
- defined an empty trait
UnsafeEffect - did have a syntax to say "this method does not take any other
withparameters except for.."
...because you do need that syntax onpubmethods of a crate
then you could feasibly demarcate which methods transitively contain unsafe blocks.
... suppose you did have a mechanism to unwind stack "through" a parameter received via with - you then could implement non-local returns, aka exceptions
... suppose you defined GreenYield effect and implemented green multi-threading on top of with parameters
It is true that Rust already has all of these defined in a different way. But it is also true that you could make the bold step to augment the language with real effects though this feature.
One way to do this is:
-
withclauses always reborrow:
We make awith-expr. ctx. to reborrow from references it is provided with for some local lifetime, in case of not capturingwith-ctx., or for lifetime of reference which was provided to the clause in case of capturing the ctx. - works like this:
let $bnd = &(mut) *$bnd //for non-captive impls\callsor
let $bnd = ref (mut) $bnd //for capturing ones - the content of a
with-expr scope faces patched borrows which prevents inference from extending\shortening lifetimes => - the example from here actually works as intended.
Appears to be correct
@SkiFire13's example is interesting but also shows some limitations. If this is what with desugars into it would mean that
-
we wouldn't be able to deallocate objects created in one
withstatement from anotherwitheven if the allocator was exactly the same.. -
it's a 2 step process with distinct "create" and "use" phases. Was that really the ambition?
I'd say it might be best to view this approach as orthogonal to with parameters. with parameters should be just parameters at the end of the day. What you can do with them should be similar to what you can do with normal parameters.
I've started thinking that there could be a macros to name and generate the shim struct explicitly. Explicit generation of shims would alleviate the knock on effects and make limitations explicit.
On the off chance itâs helpful, linking an old proposal I had made to achieve a similar result. Sadly in 2018, contexts were too bold an idea and I didnât have the energy to push back against all of the friction. I wish you a better experience.
1 Like
@tmandry I want to apply "Context" into my project. Currently I must pass parameters around and clone() it for usages.
How could I achieve this in the following scenario?
let (bcs, bcc) = broadcast::channel(32);
let wallet_lock = Arc::new(RwLock::new(Wallet::new()));
let bc_lock = Arc::new(RwLock::new(Bc::new(wallet_lock.clone())));
let mp_lock = Arc::new(RwLock::new(Mp::new(wallet_lock.clone())));
then when mp run fn is called, i need to pass these initialization as argument clones:
tokio::select! {
res = crate::mp::run(
mp_lock.clone(),
bc_lock.clone(),
bcs.clone(),
bcc,
) => {
if let Err(err) = res {
eprintln!("err {:?}", err)
}
},
...
I guess that you create your context struct with all the resources you want to share:
struct Ctx {
bc: Bc<WalletLock>,
mp: Mp<WalletLock>,
}
and then pass it around via with crate::your_cap_here = &Ctx bounds.
No need to use Arc & RwLock wrap around Bc & Mp also?
depends on how you want to thread your application: if you really need an Arc (and RwLock inside) for thread-safe shared mutability the contexts are not what you are looking for; however if your code really uses Bc<WL> and Mp<WL> as contexts, e.g. does not pass these to other threads then yes, contexts and with may help you.
1 Like
I love that you're looking at this, and I think Rust will really benefit from the functionality you're trying to deliver here.
This reminds me of a post I made about two years ago.
It even proposes the same with { } syntax to create "dependency containers" that you've proposed.
There are two actors here that I need to refer to: "providers" and "consumers". Under your proposed syntax, the providers are the places in code where a with {} block appears, and the consumers are functions that have a with clause in their signature.
-
The biggest issue I have with this design is that both providers and consumers need to change from status-quo code. If I am an application writer, it means that I can only pass implicit args to libraries that are specifically coded to be a context consumer. Is it necessary for both sides to change? Can't just providers change? That way libraries can be written in a normal, context-agnostic way such that they can work in apps that use context and apps that don't use context. Otherwise, libraries written to use context don't work with apps that don't pass context and vice-versa.
-
Under this syntax, as I understand it, implicit parameters cannot be passed as explicit args using normal function calls. There's no way to override a context.
-
I think it would be helpful to develop your example to show what happens when it's time to add a new context parameter. For instance, let's say in an existing code base I have 1,000
deserialize()calls, and now I want to add an implicit dep on the arena parameter.Each of the 1,000 direct call sites might have 0-50 stack frames between it and the one place in the code where it makes sense to provide an arena. Do I have touch something in 0-50 intermediate stack frames for each of my 1,000 direct
deserialize()call sites to add that implicit dep?
2 Likes
I believe under the design proposed the answer generally is "no". But for those intermediate stack frames which belong to pub functions the answer is "yes".
1 Like
I'd need to think about it more, but I just wanted to say that this looks really compelling for the Neon project ( https://neon-bindings.com ) -- it's an FFI binding to Node.js where all APIs have to thread a context parameter, representing the JavaScript runtime, through all code.
I suspect this proposal would help make a lot of code more ergonomic. I'm especially interested in how this could lighten the burden for our users when writing helper code, where they wouldn't have to worry about all the heavyweight type signatures that come from passing contexts explicitly through their helper code.
So your point is about enabling migration of existing APIs. I think that's a worthwhile discussion to have, I just haven't gone there yet because I think there are a few ways we could go about it and it doesn't seem like the riskiest part of the proposal currently. I hope to have an initiative repo up soon where questions like this can be explored in parallel.
There is. You can introduce another with scope and override an existing context value within that. Whether this should override "captured" contexts your code doesn't know about is still an (important) open question.
Explaining this in a concise and memorable way is likely to be a challenge. The short answer is that anytime you directly use a fn or trait impl that has a with clause you must provide a way to satisfy that clause, either using a with expression or by having your own with clause. This applies transitively. "Directly use" in this case means that by looking at your code I can determine exactly which fn or impl is being used.
Look at deserialize_and_print in the blog post: it can call our deserialize implementation that uses a with clause without knowing about the with clause. It's the caller that names the type and therefore has to provide the context to prove T: Deserialize.
If you're worried about proliferation of with clauses, we can solve that in other ways. One is the fact that you can write a with clause on impl and then use it in all functions defined in that impl.
Another is module-level generics: mod foo with arena::basic_arena;. That is separate from the core proposal, but the two fit together nicely. "Vanilla" module-level generics provides static type and const parameters to an entire module, which is desirable on its own. Adding with clauses allows providing runtime values to an entire module.
1 Like
I have concerns there as well. My other concerns are about how this affects things which take closures. It really sounds, to me, like these constraints apply to fn types which means there is now Fn() -> () with Context which means we need some kind of ?with-like syntax to say "passes through context" and !with for "doesn't support any context". I have no idea how this will end up applying to the myriad impl Iterator types.
Given something like:
fn takes_ctx_a() with A {
âŚ
}
fn takes_ctx_b() with B {
âŚ
}
fn implicit_ctx() { // implicit `with A, B`
takes_ctx_a();
takes_ctx_b();
}
There are questions like:
- How are
AandBpassed? One struct? A tuple? Independently? Which order? What is the ABI here? - If I want to override just
A, how do we model "taking"Bfrom the existing context to "shadow" theA? - What if
Bis a linear type (see other in-flight pre-RFC discussions) and we cannot restore the context afterwards?
I also have other, more general, questions like:
- How do I specify a
mut Acontext that I can mutate? Must I use&mut Aor is ownership allowed? - Can I move out of a passed-in context? Can it be destructured? Replaced? Marked up with lifetimes? How do I specify any of this? What kind of unification rules are needed when mixing various requirements together?
- What if two contexts have the same name, but different types? What is the namespace landscape for these contexts like?
- How do I say "context
fooand contextbarare both satisfied by myCtxinstance, just by different names" (and I can't duplicate because it is owned or is&mut)?
Huh? If the context takes a lifetime, what is the lifetime of this passed context? Is only &'static allowed? If it is ownedâŚwhere does it live? What stack? Is it forced to the heap? With which allocator? I would like to hear the answers to at least some of these questions. What does an extern "C" function inside of this module do? Just not get the context? An invisible extra parameter? Set of parameters (with multiple contexts)?
How can I access the context to pass it to an extern "C" function? Or do I have to rip all of my with usage out from the root and lay down explicit arguments down the entire call chain again just to be able to pass it to another function explicitly? Is this another "color" of functions where APIs taking foo-as-with cannot communicate with foo-as-parameter APIs, just with user-specified colors for each context name/type pair?
I really think the type theory side of this (and how it applies to Rust specifically) needs more attention (or at least description if it has been done) than just "context is passed around". This is a large change that has far reaching consequences from the type system to the ABI and I have yet to see the results of any investigation into all the things that it will touch. The syntax of how it is spelled is among the least of my worries as that can be fixed across an edition, but ABI and type system stuff is (AFAIK) not possible to fix across editions. I suppose the Rust ABI can be fixed across toolchain versions, but I'm pretty sure that the C ABI effects (if any) need to be Done Right⢠from the start.
6 Likes
Wasn't it the intention that in non-pub fn-s you don't need to spell it out? E.g. unless you provide the context yourself the context parameter gets added to your fn without you doing anything?
IDE can show it but you don't have to type that into the source code? Unless the fn is pub?
Contexts are always passed down the call-stack. Ergo references are sufficient. No ownership. No linear types.
Indeed you can't. If duplication is needed non-mut references are the only way.
Excellent question. My personal view is this is a type-level map. The set of contexts is a map from type to reference to a value. Ergo you can't have two contexts of the same exact type. Names are local like for function arguments and don't matter. I'd love more discussion on this.
You write a Rust wrapper which takes with parameters and inside that wrapper pass them to extern C as normal. Is there any difficulty here?
Unspecified like the rest of Rust ABI to leave space for experiments?
My own thinking: each fn is at least conceptually monomorphised to the set of context parameters it takes. If actual monomorphisation is done they could become explicit parameters at LLVM level. Alternatively there could be a pointer to some structure in one of parent stack frames. In either case only addresses (references) are passed - either as explicit LLVM-level parameters or within that structure in one of parent stack frames.
Yet another option is to put a pointer to that struct/tuple on a parent stack frame into a single global thread-local. E.g. implement a virtual register. That way in Rust->C->Rust case those parameters could still be passed. One wild though was that as FS was becoming available for general use on Linux kernels 5.11+ the pointer could sit there 
The hope is to avoid a knock-on effect everywhere. E.g. if you are implementing trait T but your code is with ctx : Ctx somewhere you will get impl T with Ctx type. If your fn invokes it without providing Ctx explicitly its own type is altered to ... with Ctx type. This happens implicitly (I thought) on non-pub fn-s. This will have to be typed out explicitly on pub types.
Most definitely. And this is why I'm loving this thread It's so interesting!
My feeling is that some kind of genericity around the set of with parameters (effects) is needed. E.g. I'm writing a generic fn that is taking some F : Fn.. as a generic parameter. F can have an arbitrary set of with parameters (effects). And I need to declare that my fn will add those to its own set of with parameters.
How does this mesh with:
If the way to pass implicit parameters as explicit args is with another with clause, the C ABI cares a lot about this.
So just because crate A names it ctx: &mut A and crate B says context: &mut A, I'm stuffed (because AFAICS, you're proposing referring to context sastisfaction by name, not type)?
Then the with part that provides the context cannot use the name and must be keyed on the type. This means that with <expr> is allowed, not just with <ident>. Which probably makes more sense anyways, but then you have a formatting conundrum with:
with {
// make context
} { // <-- this looks weird
// } in { maybe? but then there exists `for x in with {} in {}`
// use context
}
The Rust ABI is "unspecified" in that "there are no guarantees what happens today will happen in the future", but we need to do something to make this actually work. The "unspecified" just means "we don't have to get it 100% right the first time", but it does need to be specified enough to be implemented in rustc prior to acceptance.
Sure, if there is only one context being passed. For multiple contexts, you're pointer chasing now. Also, FS probably isn't good because I suspect that would need to be saved on the stack somewhere on any call (since any context-free callee can update the context again internally, so you need to restore it). There are a lot of other platforms to deal with here and modeling it as "as-if" some Rust parameter were added to the end (like #[track_caller]?) makes more sense than "use $register".
I'd like to see more than "hope" in a pre-RFC. But these types need names because I suspect generic code will need to name them to pass them around in some capacity. And if it's public, that means the with-ness needs specified, no? I still don't think we can get away with "just bubble the context around" because:
fn uses_default<T: Default>() { todo!() }
fn calls_uses_default<T>() { uses_default() }
still requires T: Default to be explicit on calls_uses_default. Is there history to that decision rather than deducing requirements based on the body? I suspect it helps a lot with compiler performance in that the body can just be ignored (prior to codegen) in callers.
So does this need to be explicit in the source code?
1 Like
I am a bit surprised with discussion around this.. is there any difficulty?
extern "C" fn external(ctx: Ctx) -> i32;
fn internal() -> i32 with ctx : Ctx {
unsafe{ external(ctx) }
}
Actually I - possibly unlike others, not sure - am proposing to refer to context by type
But &mut indeed can cause major difficulties. If I create a closure from crate A's code "capturing" the context &mut parameter and then pass it to crate B's code that wants the same context parameter.. I can't can I? That'd be two &mut references.
I've started wondering if limiting contexts to & non-mut is the only way. Any mutability in them would need to be internal say via RefCell, e.g. runtime-checked.. Not sure..
Doh! You're right 
I'd love to see a nice way forward.
Referring by name feels wrong conceptually.
But right when writing sample code..
Certainly. I did mention 3 alternatives: thread-local pointer to a hidden tuple/struct on a parent stack frame, single pointer parameter at LLVM level pointing to the same, monomorphise so that each context parameter becomes its own LLVM-level argument. FS was only half-serious
...that'd be a pointer to a hidden tuple or struct in a parent stack frame; because each fn is monomorhpised, it "knows" at least conceptually the ordering of values in that tuple.
This is my perception of the proposal on the table. Just bubble up on non-pub items. Because otherwise it'd be not very ergonomic. If this proposal is passable or even technically feasible, as in won't kill compiler performance is a different question. You can be right and it is not feasible. Or maybe there is some way to do it.. It may also be other ppl interested in the proposal are viewing it differently and want this to be explicit even on non-pub fn-s/items.
This hadn't been brought up in the thread before, but this is what I've been feeling from the start of the thread. In my view this is what has been missing. And yes it'd need to be explicit. Because I can be taking a generic parameter which implements Fn.. "with" some context parameters, but I may be only storing it, not invoking right away. Then I don't need to have that "effect" in this place. But I would be invoking it later. So that effect would need to become part of some of my types and would be "used" later.
This is separate from "conforming" a "with" impl of a trait to a non-with impl of the same trait by packaging a reference to context somewhere in. Which I think needs to be an explicit operation, possibly wrapped in a macros.
1 Like
The context declaration under this proposal is an item - means that as long as ctx in A and context in B refer to the same item things will be fineâ˘.
1 Like