Indeed, if other threads are created before you set the variable to true, then you could get a data race between writing it from the 'main' thread and reading it from another thread, which is undefined behavior.
Generally speaking, programs are allowed to assume that undefined behavior did not already occur. So you could potentially justify making the variable non-atomic on the grounds that it only converts undefined behavior (spawning threads in the child, assuming the parent was multithreaded) into more undefined behavior (data race). But it sounds like the whole point of your changes is to operate 'as correctly as possible' even if other code does execute undefined behavior. If so, it's probably best to just make it atomic. [edit: and this would only have to be a relaxed atomic load, which has no runtime overhead compared to a non-atomic load.]
The global allocator is not supposed to call handle_alloc_error itself. It should just return a null pointer, and usually the caller will respond to it by calling handle_alloc_error. The exception is methods like try_reserve, which try to cleanly handle allocation failure and thus never call handle_alloc_error.
If you did call handle_alloc_error directly from the allocator, it would unwind, and unwinding out of a global allocator is not allowed.
In any case, I don't believe the default allocation error handler is async-signal-safe, and any custom one set by the user is not likely to be async-signal-safe either. So you may want to consider just calling abort to avoid piling on more undefined behavior.