Skip to content

More fast IO and improved memory efficiency

Emmanuel Branlard requested to merge f/fast_io into master

This includes:

  • Reading of ASCII out files with variable number of header lines, and a potential line of comment at the end. This is a duplicate of !61 (closed) NOTE: I've added a unit test case and a test file, under wetb\fast\tests\ but since the file is only in my branch, it fails here. The test run fine in my local branch though

  • Fast and memory efficient reading of binary files. This uses about 5 times less memory. I stumbled into this issue while reading a large (100Mb) fast binary file with a 32-bit python. This implementation allocates the data array only once and in the proper type. The reading of the file is done using buffers, while also speeds up the reading (struct.unpack was not as efficient with a huge amount of data). This implementation is backward compatible and uses a flag use_buffer. If this flag is False, the old implementation is used. I've added a unit test that reads a binary file with both the old and new method and compare them.

Merge request reports