Crash on weights for LOFAR data

[tikk3r]

I'm reducing some LOFAR datasets and I'm running into issues regarding the weights where DDFacet will crash with

 - 11:37:10 ClassMS                   [1.9/1.9 4.1/4.1 25.4Gb]   flagged fractions per antenna: 0.15 0.13 0.07 0.08 0.56 0.08 0.12 0.12 0.07 0.10 0.08 0.08 0.08 0.08 0.08 0.09 0.13 0.07 0.09 0.08 0.10 0.08 0.12 0.13 0.08 0.07 0.06 0.45 0.11 0.13 0.07 0.10 0.12 0.11 0.06 0.53 0.39 0.07 0.24 0.27 0.09 0.15 0.08 0.07 0.09 0.77 0.06 0.14 0.07 0.10 0.10 0.09 0.07 0.05 0.08 0.09 0.07 0.08 0.12 0.09 0.07 0.05 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
 - 11:37:10 ClassMS                   [1.9/1.9 4.1/4.1 25.4Gb] Fraction of flagged data: 10.75 %
 - 11:37:10 ClassMS                   [1.9/1.9 4.1/4.1 25.4Gb] Flags updated
 - 11:37:10 ClassMS                   [1.8/1.9 4.1/4.1 25.4Gb] caching flags to ./L846940_123MHz_uv.dp3concat.N0.F0.D    Init CFs..................  70/96 [====================================              ]  72%  - 0'34" 
 - 11:37:10 CacheManager              [1.8/1.9 4.1/4.1 25.4Gb] writing cache hash ./L846940_123MHz_uv.dp3concat.N0.F0.D0.ddfcache/ch0/Flags.npy.hash
    Init CFs..................  73/96 [======================================            ]  76%  - 0'35" 
 - 11:37:11 CacheManager              [1.8/1.9 4.0/4.1 26.9Gb] writing cache hash ./L846940_123MHz_uv.dp3concat.N0.F0    Init CFs..................  78/96 [========================================          ]  81%  - 0'36" 
 - 11:37:12 ClassVisServer            [1.7/1.9 3.9/4.1 27.3Gb]   channel Mapping Gridding  : [0, 0, 0, 0, 0, 0, 0, 0,  ...., 1, 1, 1, 1, 1, 1, 1, 1], (len = 20)
 - 11:37:12 ClassVisServer            [1.7/1.9 3.9/4.1 27.3Gb]   channel Mapping DeGridding: [0, 0, 0, 0, 0, 0, 0, 0,  ...., 0, 0, 0, 0, 0, 0, 0, 0], (len = 20)
 - 11:37:12 AsyncProcessPool          [1.7/1.9 3.9/4.1 27.3Gb] process io00: exception raised processing job DATA:0:0: Traceback (most recent call last):
  File "/opt/lofar/pyenv-py3/lib64/python3.12/site-packages/DDFacet/Other/AsyncProcessPool.py", line 997, in _dispatch_job
    result = call(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lofar/pyenv-py3/lib64/python3.12/site-packages/DDFacet/Data/ClassVisServer.py", line 672, in _handler_LoadVisChunk
    DATA["Weights"] = DATA["Weights"][DATA["sort_index"]]
                      ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
TypeError: 'int' object is not subscriptable

I noticed in other recent DDFacet versions that the check for non-array like weights was commented out, and always just modified this before installing, but it seems that somehow recent LOFAR data can hit this scenario in general after all, so I thought it's perhaps better to open a ticket about it. Perhaps it's the new scalar weights compression or something. After checking with @twshimwell what was in his working container I noticed that it had a version where this check was not yet commented out. I'm guessing this chunk of code might need to handle the case of it not being an array after all?

DDFacet/DDFacet/Data/ClassVisServer.py

Lines 669 to 672 in 2fe05ab

	if DATA.get("sort_index",None) is not None and weights is not None:
	# and DATA["Weights"] is not 1: # OMS 2023/12 they're not "1" ever and this seems a bug
	# a value of '1' is used as a flag by GetVisWeights, but we can't compare with 1 because the weights could be and generally will be an array...
	DATA["Weights"] = DATA["Weights"][DATA["sort_index"]]

[cyriltasse]

Hey @tikk3r can you checkout HackathonRennes_June26 on the dev repo and tell me if the bug is still there? I think it's been fixed already

[tikk3r]

Thanks Cyril. Our cluster where I have the data is down at the moment for security reasons, but I will try that out once it gets back up (but this may pass over the weekend depending on how much updating the cluster needs)