but this may require reallocating multiple times. size_hint could at least reduce this
let mut buf = String::with_capacity(reader.size_hint().0);
let size_hint = reader.size_hint();
let size = size_hint.1.unwrap_or(size_hint.0);
let mut buf = String::with_capacity(size);
though this option may allocate too much space, which could be undesirable
<std::io::Bytes as Iterator>::size_hint currently operates by "magic", using an internal SizeHint trait that only implements a specific size for &[u8].
Having a size_hint method on Read would allow Bytes::size_hint to have a more obvious implementation, and allow use on more types. BufReader could return the minimum as the amount currently in its buffer, for example.
I like that idea. It leaves a bit of a question as to what the upper limit of Bytes::size should return, but if it doesn't matter as you say (and I would agree) then I'd suggest having it return None (or keep the magic impl, but that seems strange to me)
I didn't end up submitting it as I didn't have the energy/confidence to push for a libs-api addition. Still, here is what I came up with:
Returns the bounds on the number of bytes remaining to be read.
Specifically, remaining_hint() returns a tuple where the first element
is the lower bound, and the second element is the upper bound.
The second half of the tuple that is returned is an Option<usize>.
A None here means that either there is no known upper bound, or the
upper bound is larger than u64.
It is not enforced that a reader implementation yields the declared
number of bytes. A reader may yield less than the lower bound or more
than the upper bound of bytes. A file could be truncated or appended
to after calculating its size, for instance.
remaining_hint() is primarily intended to be used for optimizations
such as reserving space for the elements of the stream, but must not be
trusted to e.g., omit bounds checks in unsafe code. An incorrect
implementation of remaining_hint() should not lead to memory safety
That said, the implementation should provide a correct estimation,
because otherwise it would be a violation of the trait's protocol.
The default implementation returns (0, None) which is
correct for any reader.
Eh. With alignments of allocations from the typical allocators, these bytes only actually add up in usable memory when you're already on a allocation boundary (say 8 or 16 bytes), so effective usage is probably on the order of 6-12% of that "extra" kilobyte in practice (yes, alignments overall probably are some kind of Benford's Law in how their % 8 sizes land, but I'd expect File sizes to be way more even than data structures).
Note that this is bad for File when reading contents out of /sys because the "sizes" returned by stat are complete fabrications. Most everything is either 0 or 4096 (one page). Even "symlinks" have 0 size despite their size being "more known". Things like sys/dev/block/*/stat have a stat size of 4096 but cat stat | wc -c is 153 (in one instance I have here).
I know /sys is not a land of "files", but making them dangerous to use with the standard library doesn't sound good to me (there have been DoS vulns where read loops expected st_size to be trustworthy and it just stalled out despite the EOF state having been reached), but this is just not the case in such "special" places.
Maybe it could be better here, but it's also unclear to me how a Read would normally know a maximum other than infinity or that's the same as the min. Like a TcpStream or a DecompressStream can't really give a meaningful max.
Can you elaborate on where you'd use the extra size here on a 32-bit platform?
What can you do with that fact? Is it even possible to tell whether it was wrong? Is there any way you could write safe code which would misbehave if it wasn't the "minimum at time of calling", since it could change immediately after calling anyway?
Basically, my meta-point here is that if this is for
better optimized pre-allocation for memory
then perhaps it should just be reserve_hint(&self) -> usize, with no requirements other than "should return something reasonably useful". Especially since, unlike for iterators, TOCTOU is an issue for many common Reads and thus it can rarely be relied upon precisely.
(And, for example, I think it would be perfectly fine for it to return a size rounded up to a sector multiple -- at least for multi-sector files -- rather than potentially needing to use a more precise API.)
Either one makes sense to me. The standard library uses both.
Read returns usize when it's reading into a buffer since a buffer can obviously never hold more than usize::MAX bytes.
Other methods like seek, take, and Metadata::len use u64.
I previously thought u64 made the most sense, but given that the intended use case for this hint is to preallocate memory I see that usize would be more convenient. Callers would have to deal with overflow if it were u64.
On the other hand, this hint will always be misleading for files larger than what usize can represent.
I agree that this is fine since we're dealing with allocation, but we have to make sure to communicate that this is the sole purpose of the function, and that it shouldn't be relied on any other thing. (i.e. it shouldn't be used to calculate progress percentage).
Yes. I searched for it, but wasn't able to find it right now. I'll poke around later again.
Edit: OK, just came across it again. It was this subthread where grousing about userspace doing silly things was discussed and it was brought up that anything trusting the size is 1) a TOCTOU watiing to happen and 2) not considering the shared mutable-ness of filesystems. There wasn't actually a DoS in the wild, just silly userspace behaviors.