Libsystem, or, the great libstd refactor

[lambda]

I don’t think his comment was dismissive of the idea in general. The current form of the patch isn’t really possible to review; a single giant patch with no discussion whatsoever.

This is a big enough change, that affects enough of the internals of the std library, how easy it is to port, and would cause any other WIP branches to require manual merging so would need some timing and coordination to land, that an RFC does seem reasonable. Basically, something discussing why this move is a good idea, what porting use cases it would help out, if there are any alternatives, etc.

In addition to that, once the RFC process is done, I think this would be better done split up into a series of patches; one huge patch makes it hard to break down and verify that each move is sound and doesn’t change anything important; particular concerns might be privacy, error reporting, and the like.

I would not take his response as dismissive; just saying that once seeing the change, it’s clear that it will require a little more discussion before it’s reviewable and mergeable.

[arcnmx]

I don’t think his comment was dismissive of the idea in general. The current form of the patch isn’t really possible to review; a single giant patch with no discussion whatsoever. I would not take his response as dismissive; just saying that once seeing the change, it’s clear that it will require a little more discussion before it’s reviewable and mergeable.

Sorry, by that I just meant dismissive of the actual design and layout being proposed here. I simply haven’t received any feedback on that whatsoever on any front, which is kind of what I’m looking for… The lack of any discussion or feedback on the changeset itself simply doesn’t instill much hope that an RFC would generate much more discussion. I get that it’s large but that doesn’t preclude being able to see what kind of changes have been made and whether they’re a good idea or not.

I think this would be better done split up into a series of patches; one huge patch makes it hard to break down and verify that each move is sound and doesn’t change anything important; particular concerns might be privacy, error reporting, and the like.

Agreed.

This is a big enough change, that affects enough of the internals of the std library, how easy it is to port, and would cause any other WIP branches to require manual merging so would need some timing and coordination to land, that an RFC does seem reasonable. Basically, something discussing why this move is a good idea, what porting use cases it would help out, if there are any alternatives, etc.

I guess my view is that it’s currently impossibly difficult to port, and libstd changes are far and few enough between right now that it’d be a good time to land. Though mostly I’m just impatient because after the few months it takes an RFC to get through I won’t have the time or energy to actually make the changes anymore - I’m currently on a rare break with some actual time to get things done.

[briansmith]

Well his more recent comment was pretty dismissive so it seems unlikely that’s a good approach to take… I’m not so much looking for approval as much as indication from someone that I’m not doing something very wrong here, but no one seems to care enough to look.

Nobody has time to read a 166 file +3342/-3030 patch. If you want somebody to look at a patch, you need to make one that is very much smaller. If you won’t break up the patch into smaller parts until the Rust community agrees that they want to split all the platform-dependent bits of libstd into separate modules, then the way to get that kind of agreement is through an RFC.

[quote] As for an RFC… “cut-paste anything in libstd within an adhoc conditional #[config(target_os = …)] to the sys module” doesn’t seem like a very productive one to me so meh. [/quote]

Probably more people would be supportive of that wording than the original way it was proposed.

But, your proposal is more than that, right? I’ve made an attempt to help you make the RFC here: https://gist.github.com/briansmith/d4540b86c113eba8ee93. I intentionally left out the technical details in the “Detailed Design” section as you know them better than us. Please fill them in and make other changes for any other parts of your proposal that I’ve misunderstood.

[arcnmx]

Nobody has time to read a 166 file +3342/-3030 patch. If you want somebody to look at a patch, you need to make one that is very much smaller.

Eh one can glance it over and look at the file structure and an example module or two ;_;

I’ve made an attempt to help you make the RFC here

You are good at this communication thing. The “formal verification” part is a bit hard to define though. We’re not really defining the standard library so much as the layer underneath it that’s used to interact with the underlying OS. And even that definition isn’t really that formalized.

[briansmith]

Eh one can glance it over and look at the file structure and an example module or two ;_;

That’s generally not how people work, for better or worse.

The “formal verification” part is a bit hard to define though. We’re not really defining the standard library so much as the layer underneath it that’s used to interact with the underlying OS. And even that definition isn’t really that formalized.

I agree I wasn’t so clear, but I’m not sure how to improve it. The point I was trying to make is that having a clear interface to the platform-specific bits should help the development of a formal definition of the Rust standard library by facilitating the removal of assumptions that depend on the semantics of libc…eventually.

[oli-obk]

How about doing this incrementally?

Add a libsystem, but only port a single thing. Then port the next thing in another PR. At some point libstd should be free of system dependent code, and libsystem should only offer the minimal public API necessary for libstd to work.

[Zoxc]

As someone who wants to add support for another OS to libstd I like the idea. Keeping the OS specific code separated also makes it easier to maintain a fork which supports an additional OS.

[lilred]

+1, I’d love for this to happen.

It’d be really neat if libstd could be implemented inside Rust code the same way libcore is - by just providing x,y,z language items.

[briansmith]

See this Hacker News thread: https://news.ycombinator.com/item?id=11111059

Rust programs vulnerable to buffer overflow because of glibc.

@arcnmx, Are you going to be able to fill out the technical details of the RFC? It doesn’t have to be perfect. Just an outline of the various refactorings you did, before you completely forget what they are.

[arcnmx]

Yeah, sorry, I’ve been busy… I will try to get to that asap so at least we have some talking points and can get to it all sooner rather than later.

[steveklabnik]

Rust programs vulnerable to buffer overflow because of glibc.

Unless you’re using a MUSL-related rustc. Which is admittedly a minority.

[arcnmx]

libstd::sys, the great libstd refactor

[tomjakubowski]

Or, almost definitely, if you’re running Rust on OS X or Windows.

[notriddle]

I thought the MinGW toolchain used glibc…

[retep998]

I thought the MinGW toolchain used glibc…

No, MinGW uses MSVCRT still, it just uses the really really old one in msvcrt.dll that you’re not supposed to use, and because that msvcrt.dll is missing a significant amount of C stuff, MinGW implements the missing functionality on top of it. To use glibc on Windows would require a massive overhaul of it to use windows API instead of linux syscalls.

Furthermore Rust itself uses very little of the C standard library, just strlen and various math functions along with memcpy and friends. Everything else Rust does directly through Windows API.

[eddyb]

I am surprised nobody mentioned allocators!

Most of the libstd code that doesn’t depend on POSIX-compatible I/O and multithreading is not in libcore due to it necessitating dynamic memory allocation.

We already have libcollections (missing HashMap pending defaulting in inference), and with custom allocator support, they wouldn’t even depend on liballoc.

libstd would then contain re-exports, plugging its default allocator as the default for all the collections/smart pointers, and anyone could do the same, skipping all the extra OS-dependent features libstd has.

AFAIK that was the plan, although @nikomatsakis seems to suggest that the necessary “defaulting to type parameter defaults in inference” mechanism might never be stabilized.

[Amanieu]

Another possibility for allowing code that requires memory allocation to be used in kernel/embedded environments would be to add a none system backend to libstd in addition to the existing windows and unix. This backend would simply return an error for all OS-specific functionality to indicate that it is not implemented. This is similar to what the newlib C library does to support running on bare metal systems.

[briansmith]

Eddy, Amanieu, I think that it is also important to figure out how to get more of the standard library to work in environments where allocation needs to be different than the typical malloc/free style heap. But, I think that’s an orthogonal or otherwise separate enough to what’s being proposed here to have it’s own thread.

[arcnmx]

AFAIK that was the plan, although @nikomatsakis seems to suggest that the necessary “defaulting to type parameter defaults in inference” mechanism might never be stabilized.

Wait why is that never being stabilized it’s a very very very useful feature ;_;

(though in the collections/allocator case, could libstd not instead expose it like pub type HashMap<T> = collections::HashMap<T, StdAllocator>;? not quite ideal, though…)

Another possibility for allowing code that requires memory allocation to be used in kernel/embedded environments would be to add a none system backend to libstd in addition to the existing windows and unix. This backend would simply return an error for all OS-specific functionality to indicate that it is not implemented.

This is an experiment I partially wrote the mock bind backend for. You’d get compiler errors if you tried using any unimplemented features, but they’d otherwise be removed by the link/optimize step so you would get a partial libstd that worked as long as you didn’t call into IO or whatever else. It can of course be implemented in pure rust instead, or have methods modified to unconditionally return Err()s or panic at runtime, etc.

I’d like to at some point see libstd functionality become more modular, where it may be possible to build on a platform without threads, or without networking, but still take advantage of the majority of the greater crates.io ecosystem. These proposed changes would be a good start to making it possible and start experimenting with the techniques described above, though it certainly doesn’t provide a robust final solution.

[system]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.