BufWriter that doesn't flush on seek

zip-next's ZipArchive needs to seek backward when it finishes adding a file to the archive, to update the file's header with the size and checksum. However, it doesn't have any commonly-used methods that need to seek back past the current file's header.

For situations such as this, it would be useful if BufWriter could be created with no limit on the buffer size and with the option not to flush when seek() was called, but only when dropped, closed or explicitly flushed. Then, using ZipArchive's set_flush_on_finish_file(true), a ZipArchive could be written to a BufWriter for streaming when the underlying Write was e.g. a TCP connection that didn't implement Seek or remember previously-written bytes. One file at a time in the archive would be output.

This seems like a very specialized use case to me that does not warrant addition to the standard library.

You can use BufWriter::get_mut to acess the underlying writer without flushing. You can use BufWriter::with_capacity to create a buffer of an arbitrary size. You can write everything to a Vec first and write the file in one go when you are done.

1 Like

Would this not be better handled by a method on ZipWriter that gets given the entire file data in one go?

Right now, ZipWriter makes no promises about how far back it seeks - it's permitted, as a non-breaking change, to rewind to earlier in the file than the most recently created file if it wants. Your proposed BufWriter change would work with the current implementation of ZipWriter, but would be dependent on ZipWriter not changing the implementation.

With a more complex API, ZipWriter could guarantee to not seek if you use the file data in one go methods, and only allow you to use start_entry followed by write followed by finish_entry if your underlying W is W: Write + Seek instead of just W: Write.

This sounds a little like BufReader::seek_relative. Is that what you were thinking of?

It also sounds a little like a case for Java's InputStream::mark/reset, but even that isn't present on its OutputStream.

It's sneakier than that. The current implementation of ZipWriter in the zip-next crate can be configured so that it never seeks to before the current position after a call to flush; the question is whether we can abuse BufWriter to convert impl Write to impl Write + Seek with the limitation that after a flush, you can't seek backwards to before the current file position at the time you called flush.

The use case is writing ZIP files to a stream, like a TcpStream; this only implements Write, not Seek, but ZipWriter needs to be able to seek so that it can accept the data for a file in chunks, and then seek backwards to fill in the file header with the file size once it's told that it's reached the end of a file.

I think that this is a dangerous direction, and that the requirement is better met by changing ZipWriter so that it can write to a impl Write with restrictions on what you can do, but you get extra abilities if it has an impl Write + Seek. The challenge is that this would then require you to specialise ZipWriter's Drop implementation, depending on whether it was wrapping a impl Write or a impl Write + Seek.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.