The red zone is one of a number of features that can cause much hair-pulling when enabled on platforms that don’t support it. Any amateur OS implementers using Rust who don’t stumble on this particular blog post (which explains how and why to disable it) are in for a world of pain.
I’m thinking that the red zone should be off by default, and enabled in target files (such as Linux) that support it. The only downside of the red zone being off is decreased performance, which I think is a much better default than occasional stack corruption.
Addendum: another feature which would should be off by default is landing pads, for the same reasons.
I don’t think disabling the red zone by default is the right approach - as you say it is not a discoverable thing and we should not be slower by default for the sake of a small number of programmers. However, you are completely correct that the current situation is horrible, and I think that kernel-level programming is something we should be supporting much better. Perhaps a way forward is to make support for x86_64-unknown-none-gnu more official? We could then disable the red zone (and landing pads?) when using that target. That seems a bit more discoverable than the individual flags, and it sounds like it would have other benefits too (setting it up yourself sounds error prone, for example).
+1 for having a separate target for the kernel ABI. The kernel ABI is distinct enough that it should be separate - e.g. you also need to disable floating point/SSE ops.
However, I feel like landing pads are somewhat orthogonal to this whole ABI business - exceptions are not supported on the x86_64-unknown-kernel-gnu target basically because nobody has bothered to implement them. Maybe we should make compiling on a kernel target with unwinding enabled emit a warning or something.
I also really think that -Z no-landing-pads should have a more friendly name - say -C no-exceptions (as the C++ moral equivalent, -fno-exceptions , is used quite often).
If we disable the red zone by default and enable it in built-in targets, isn’t everyone happy? No change for the >90% of programmers building against built-in targets, and developers building against non-standard or homemade targets stop having stack corruption issues.
What is the advantage of doing it this way around? Do you believe that the majority of programmers with custom targets to do not want the red zone? I have no idea of the numbers, hopefully someone who does can weigh in. My impression is that after subtracting people who would be using x86_64-unknown-kernel-gnu/x86_64-unknown-none-gnu or whatever, then most people using custom targets would want the red zone, but I am pretty much just guessing. Or do you think such targets are a bad idea for other reasons?
I think it’s more of a problem of the whole non-native-target build issues (which is even harder for cross-compilation). If rust had better support for custom targets, those features would be slightly more discoverable (FWIW, I had some hair pulling about red zone too).
I don’t think it’s a good idea to use autodetection magic for things like the red zone. As someone who’s dabbled in Rust OS dev for the last few months, I know the pain of figuring out build flags and ABI stuff when it isn’t yet documented. However, I think that the community will quickly find a canonical set of kernel build parameters and this will no longer be an issue. This is what happened with C. A systems language should leave these decisions to the programmer rather than making an educated guess.
Anecdotally, speaking as a BSD user, this sounds like something that could cause a headache for us. Autoconfigurations like this often don’t work properly on our systems.
I might be mistaken, but I think you’re the first to discuss autodetection/magic/etc. Rust targets are defined explicitly using target files, I don’t think autodetection will ever be part of the story.
In Rust lingo, a “target” is a dictionary of compile options indexed by a target triple (e.g. x86-ubuntu-linux-gnu). Targets are specified for each platform, and new targets are user-definable. A target’s explicit purpose is to determine compile options such as whether to enable the “red zone”.