Unable to load historic *.pkl files

[danieludick]

I am trying to convert the attached *.pkl file that was generated with the Stellenbosch University TART, but I am unable to do so using the latest tart package (version 0.15.5).

I am running the following Python code and passing it the attached *.pkl file:

# ==================================================================================================
# Danie Ludick (2020) 
# dludick@sun.ac.za
#
# Description:
#    Converts TART raw data (in *.pkl format to MATLAB *.mat data object)
#    Note, this assumes that you have installed the tart package: "sudo pip3 install tart"

import scipy.io
import pickle
from tart.operation import observation
import argparse

# =================================================================================================
# Main driver
# =================================================================================================
if __name__ == '__main__':

  PARSER = argparse.ArgumentParser(description='Convert raw data from the TART telescope to MATLAB compatible format')
  PARSER.add_argument('--file', required=True, help="The raw data *.pkl file ")
  
  ARGS = PARSER.parse_args()

  print("Convert tart raw data v1.0 from 2020-03-29")
  print("")
  
  # Load the Observation file (i.e. the PKL file)
  obs = observation.Observation_Load(ARGS.file)

  # Extract information from the *.pkl file
  print("  Sampling time-stamp   : %s " % str(obs.timestamp))
  print("  Number of Samples     : %d " % len(obs.data[ANTENNA_INDEX]))
  print("  Number of Antenna     : %d " % obs.config.get_num_antenna())
  fs =  obs.get_sampling_rate()
  print("  Sampling rate [Hz]    : %d " % fs)

  scipy.io.savemat('tart_matlab_data', mdict={'version':1, \
                                              'timestamp': str(obs.timestamp), \
                                              'sampling_rate': fs, \
                                              'number_of_antennas': obs.config.get_num_antenna(), \
                                              'antenna_channels_raw_data':obs.data})

I get the following error:

djludick@forecasts:~/scratch-dl/pkl_reader$ python3 convert_pkl_to_mat.py --file tartza_data.pkl
Convert tart raw data v1.0 from 2020-03-29

not gzipped
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tart/operation/observation.py", line 83, in Observation_Load
d = pickle.load(load_data)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 1: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "convert_pkl_to_mat.py", line 28, in
obs = observation.Observation_Load(ARGS.file)
File "/usr/local/lib/python3.6/dist-packages/tart/operation/observation.py", line 87, in Observation_Load
d = pickle.load(load_data)
_pickle.UnpicklingError: invalid load key, '\x1f'.
djludick@forecasts:~/scratch-dl/pkl_reader$

[danieludick]

I think this issue is merely a result of the data being stored in *.pkl format using Python2 and then I try to extract is using Python3.

When extracting the *.pkl data directly using Python2.7 with the following script, then it seems to work:

load_data = gzip.open(filename, 'rb')
d = pickle.load(load_data)
print(d)

[tmolteno]

I think the longer term solution is to create a conversion utility that moves the old .pkl files to .hdf. This would have to be written in python2.7 so might only work while python2 can be used (it has been deprecated).