Should `OpenOptionsExt::custom_flags` be unsafe?

std::os::windows::fs::OpenOptionsExt provides extension methods for creating or opening a file on Windows.

The custom_flags method allows setting arbitrary flags and passing them to the underlying Windows API call. These flags can (potentially) modify the returned file handle in ways that may make otherwise safe File methods unsound. This has already been a problem with FILE_FLAG_OVERLAPPED and if any other such flags are added in the future, the custom_flags method will allows these to be set "safely".

Would it be best to deprecate this method and replace it with an unsafe equivalent?

2 Likes

I can't speak as to unsafety, but if deprecation is the preferred option, let's go for #[deprecated_safe].

Ah, yeah. That looks to be tailor made for this use case.

The dangerous flags could be also masked and later introduced in a different method (unsafe fn custom_flags_unmasked). This doesn't break any sound code and it avoids forcing users to use unsafe even though their usage is sound.

If we go that route, then for forwards compatibility, we should safelist the known-safe flags, and not assume that future OS flags we don't yet know about will be safe.

3 Likes

Yeah, currently the risk is mostly hypothetical at this point. The danger of FILE_FLAG_OVERLAPPED has been defused in the standard library (by aborting) and I'm not aware of any other current flags that would cause similar soundness issues. Which is why I phrased the OP as a question rather than a statement.

People do use FILE_FLAG_OVERLAPPED safely (i.e. they avoid synchronous reads/writes) so if we masked it, we would be breaking sound code. I don't think there's any way to avoid that though. It would just be a question of how we handled a transition.