Currently we read all of the files into memory in one, including the .dat file which can be several GB. We need a way that:
- Stream the
.dat file when it's needed (the other files are usually a trivial size in comparison so I don't think we need to worry about that).
- Only read the data through once. This sounds simple but the
.cff combined file format complicates this - it's possible to embed the .dat data within the .cff such that there's another file section once the .dat ends (e.g. the .hdr or .inf). For ASCII data format, we have to only read the number of lines we expected from the sample rate specification, i.e. we need to read total_num_samples lines. For binary formats, we have to calculate the total number of bytes expected from the .dat section of the .cff file by using total_num_samples * num_bytes_per_scan where num_bytes_per_scan is 4 + 4 + [size of analog data value] * [number of analog channels] + ceil([number of status channels] / 16) * 2) where [size of analog data value is 2 for binary16 data type (a.k.a. "binary" in the COMTRADE specification - I find this confusing and ambiguous, I presume it's called that for backwards compatibility reasons) and 4 for binary32 and float32 data types. Making this data streaming for the CFF files work with Rust's ownership model is going to be a challenge. It might be that we stream the separate files but read the whole .cff file in, as a compromise.
Currently we read all of the files into memory in one, including the
.datfile which can be several GB. We need a way that:.datfile when it's needed (the other files are usually a trivial size in comparison so I don't think we need to worry about that)..cffcombined file format complicates this - it's possible to embed the.datdata within the.cffsuch that there's another file section once the.datends (e.g. the.hdror.inf). For ASCII data format, we have to only read the number of lines we expected from the sample rate specification, i.e. we need to readtotal_num_sampleslines. For binary formats, we have to calculate the total number of bytes expected from the.datsection of the.cfffile by usingtotal_num_samples * num_bytes_per_scanwherenum_bytes_per_scanis4 + 4 + [size of analog data value] * [number of analog channels] + ceil([number of status channels] / 16) * 2)where[size of analog data valueis 2 for binary16 data type (a.k.a. "binary" in the COMTRADE specification - I find this confusing and ambiguous, I presume it's called that for backwards compatibility reasons) and 4 for binary32 and float32 data types. Making this data streaming for the CFF files work with Rust's ownership model is going to be a challenge. It might be that we stream the separate files but read the whole.cfffile in, as a compromise.