### Author:  [@scottmcm](https://internals.rust-lang.org/u/scottmcm)
#### Post date: [March 1, 2022, 12:39am UTC](https://internals.rust-lang.org/t/size-hint-for-std-read/16222/8 "2022-03-01T00:39:00Z")
> [@chrefr](#):
>
> The problem with slight overestimations is that they add up
But if it can't overestimate at all -- like `size_hint().0` -- then something like a decompressor will end up having to give a hint that's never big enough to avoid a reallocation.
---
### Author:  [@Cyborus04](https://internals.rust-lang.org/u/Cyborus04)
#### Post date: [March 1, 2022, 12:40am UTC](https://internals.rust-lang.org/t/size-hint-for-std-read/16222/9 "2022-03-01T00:40:31Z")
> [@jkugelman](#):
>
> ```rust
> pub trait Read {
> #[unstable(feature = "remaining_hint", issue = "none")]
> fn remaining_hint(&self) -> (u64, Option) {
> (0, None)
> }
> }
>
> ```
I really like the name! And yeah, `u64` is better here. But I would agree with @scottmcm that simply returning the lower bound might be better
> [@jkugelman](#):
>
> It is not enforced that a reader implementation yields the declared number of bytes. A reader may yield less than the lower bound or more than the upper bound of bytes. A file could be truncated or appended to after calculating its size, for instance.
What about requiring that it be a correct minimum _at the time of calling_, but can change later?
Edit: Reading it again, that might have been what you meant. Was it?
---
### Author:  [@Cyborus04](https://internals.rust-lang.org/u/Cyborus04)
#### Post date: [March 1, 2022, 12:43am UTC](https://internals.rust-lang.org/t/size-hint-for-std-read/16222/10 "2022-03-01T00:43:38Z")
Oh, that's an interesting issue. What about a tuple of `(min, likely)`, instead of just `min` or `(min, max)`? Not sure what would exactly constitute "likely", though.
---
### Author:  [@mathstuf](https://internals.rust-lang.org/u/mathstuf)
#### Post date: [March 1, 2022, 2:48am UTC](https://internals.rust-lang.org/t/size-hint-for-std-read/16222/11 "2022-03-01T02:48:45Z")
> [@chrefr](#):
>
> The problem with slight overestimations is that they add up: if my reader only returns up to one redundant byte, then a collection of 1,000 readers will allocate a spare kilobyte.
Eh. With alignments of allocations from the typical allocators, these bytes only actually add up in usable memory when you're already on a allocation boundary (say 8 or 16 bytes), so _effective_ usage is probably on the order of 6-12% of that "extra" kilobyte in practice (yes, alignments overall probably are some kind of Benford's Law in how their `% 8` sizes land, but I'd expect `File` sizes to be way more even than data structures).
> [@jkugelman](#):
>
> For much of the same reasons you've outlined I figured it would be a more elegant way to optimize not just a couple of specific readers (namely, `File` and `BufReader` ), but any readers that wish to provide hints.
Note that this is _bad_ for `File` when reading contents out of `/sys` because the "sizes" returned by `stat` are complete fabrications. Most everything is either 0 or 4096 (one page). Even "symlinks" have 0 size despite their size being "more known". Things like `sys/dev/block/*/stat` have a `stat` size of 4096 but `cat stat | wc -c` is 153 (in one instance I have here).
I know `/sys` is not a land of "files", but making them dangerous to use with the standard library doesn't sound good to me (there have been DoS vulns where `read` loops expected `st_size` to be trustworthy and it just stalled out despite the EOF state having been reached), but this is just not the case in such "special" places.
---
### Author:  [@scottmcm](https://internals.rust-lang.org/u/scottmcm)
#### Post date: [March 1, 2022, 2:51am UTC](https://internals.rust-lang.org/t/size-hint-for-std-read/16222/12 "2022-03-01T02:51:15Z")
> [@Cyborus04](#):
>
> though this option may allocate too much space
I tried essentially `min(hint.0.saturating_mul(2), hint.1)` for Iterators back in [Try to guess a smarter initial capacity in Vec::from\_iter by scottmcm · Pull Request #53086 · rust-lang/rust · GitHub](https://github.com/rust-lang/rust/pull/53086), but it wasn't an obvious win.
Maybe it could be better here, but it's also unclear to me how a `Read` would normally know a maximum other than infinity or that's the same as the min. Like a TcpStream or a DecompressStream can't really give a meaningful _max_.
> [@Cyborus04](#):
>
> And yeah, `u64` is better here
Can you elaborate on where you'd use the extra size here on a 32-bit platform?
> [@Cyborus04](#):
>
> What about requiring that it be a correct minimum _at the time of calling_ , but can change later?
What can you do with that fact? Is it even possible to tell whether it was wrong? Is there any way you could write safe code which would misbehave if it wasn't the "minimum at time of calling", since it could change immediately after calling anyway?
* * *
Basically, my meta-point here is that if this is for
> better optimized pre-allocation for memory
then perhaps it should just be `reserve_hint(&self) -> usize`, with no requirements other than "should return something reasonably useful". Especially since, unlike for iterators, TOCTOU is an issue for many common `Read`s and thus it can rarely be relied upon precisely.
(And, for example, I think it would be perfectly fine for it to return a size rounded up to a sector multiple -- at least for multi-sector files -- rather than potentially needing to use a more precise API.)
---
### Author:  [@jkugelman](https://internals.rust-lang.org/u/jkugelman)
#### Post date: [March 1, 2022, 3:28am UTC](https://internals.rust-lang.org/t/size-hint-for-std-read/16222/13 "2022-03-01T03:28:08Z")
> [@mathstuf](#):
>
> Note that this is _bad_ for `File` when reading contents out of `/sys` because the "sizes" returned by `stat` are complete fabrications.
This behavior's [already in the standard library](https://doc.rust-lang.org/src/std/fs.rs.html#636-639).
> [@mathstuf](#):
>
> I know `/sys` is not a land of "files", but making them dangerous to use with the standard library doesn't sound good to me (there have been DoS vulns where `read` loops expected `st_size` to be trustworthy and it just stalled out despite the EOF state having been reached), but this is just not the case in such "special" places.
Can you elaborate on what the danger is? What are these DoS vulnerabilities you mention? Is it because somebody relied on the size being accurate instead of treating it as a "hint"?
---
### Author:  [@Cyborus04](https://internals.rust-lang.org/u/Cyborus04)
#### Post date: [March 1, 2022, 3:31am UTC](https://internals.rust-lang.org/t/size-hint-for-std-read/16222/14 "2022-03-01T03:31:25Z")
> [@scottmcm](#):
>
> Maybe it could be better here, but it's also unclear to me how a `Read` would normally know a maximum other than infinity or that's the same as the min.
I can't think of an example either.
> [@scottmcm](#):
>
> Can you elaborate on where you'd use the extra size here on a 32-bit platform?
I was thinking about the fact that `Seek` deals in `u64`s, but seeing as `Read` deals in `usize`, that may indeed make more sense. I really gotta think more before I speak...
> [@scottmcm](#):
>
> What can you do with that fact? Is it even possible to tell whether it was wrong? Is there any way you could write safe code which would misbehave if it wasn't the "minimum at time of calling", since it could change immediately after calling anyway?
I mean that if a file returns the amount of bytes left until the end as the hint, and the file was subsequently shortened, that wouldn't be a violation of the trait
> [@scottmcm](#):
>
> Basically, my meta-point here is that if this is for
>
> > better optimized pre-allocation for memory
>
> then perhaps it should just be `reserve_hint(&self) -> usize` , with no requirements other than "should return something reasonably useful". Especially since, unlike for iterators, TOCTOU is an issue for many common `Read` s and thus it can rarely be relied upon precisely.
That makes sense
---
### Author:  [@jkugelman](https://internals.rust-lang.org/u/jkugelman)
#### Post date: [March 1, 2022, 3:45am UTC](https://internals.rust-lang.org/t/size-hint-for-std-read/16222/15 "2022-03-01T03:45:16Z")
> [@Cyborus04](#):
>
> I was thinking about the fact that `Seek` deals in `u64` s, but seeing as `Read` deals in `usize` , that may indeed make more sense. I really gotta think more before I speak...
Either one makes sense to me. The standard library uses both.
- `Read` returns `usize` when it's reading into a buffer since a buffer can obviously never hold more than `usize::MAX` bytes.
- Other methods like `seek`, `take`, and `Metadata::len` use `u64`.
I previously thought `u64` made the most sense, but given that the intended use case for this hint is to preallocate memory I see that `usize` would be more convenient. Callers would have to deal with overflow if it were `u64`.
> [@Cyborus04](#):
>
> What about requiring that it be a correct minimum _at the time of calling_ , but can change later?
>
> Edit: Reading it again, that might have been what you meant. Was it?
Yep, it is.
---
### Author:  [@Gilnaa](https://internals.rust-lang.org/u/Gilnaa)
#### Post date: [March 1, 2022, 5:39am UTC](https://internals.rust-lang.org/t/size-hint-for-std-read/16222/16 "2022-03-01T05:39:07Z")
> [@jkugelman](#):
>
> Callers would have to deal with overflow if it were `u64` .
On the other hand, this hint will always be misleading for files larger than what usize can represent. I agree that this is fine since we're dealing with _allocation_, but we have to make sure to communicate that this is the sole purpose of the function, and that it shouldn't be relied on any other thing. (i.e. it shouldn't be used to calculate progress percentage).
---
### Author:  [@mathstuf](https://internals.rust-lang.org/u/mathstuf)
#### Post date: [March 1, 2022, 12:10pm UTC](https://internals.rust-lang.org/t/size-hint-for-std-read/16222/17 "2022-03-01T12:10:05Z")
> [@jkugelman](#):
>
> Is it because somebody relied on the size being accurate instead of treating it as a "hint"?
Yes. I searched for it, but wasn't able to find it right now. I'll poke around later again.
Edit: OK, just came across it again. It was [this subthread](https://lwn.net/Articles/814069/) where grousing about userspace doing silly things was discussed and it was brought up that anything _trusting_ the size is 1) a TOCTOU watiing to happen and 2) not considering the shared mutable-ness of filesystems. There wasn't actually a DoS in the wild, just silly userspace behaviors.
---
### Author:  [@system](https://internals.rust-lang.org/u/system)
#### Post date: [May 30, 2022, 12:10pm UTC](https://internals.rust-lang.org/t/size-hint-for-std-read/16222/18 "2022-05-30T12:10:27Z")
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.