Skip to content

Fix beam pointing antenna-health fallback - #79

Open
peijin94 wants to merge 1 commit into
ovro-lwa:mainfrom
peijin94:agent/fix-beam-pointing
Open

Fix beam pointing antenna-health fallback#79
peijin94 wants to merge 1 commit into
ovro-lwa:mainfrom
peijin94:agent/fix-beam-pointing

Conversation

@peijin94

@peijin94 peijin94 commented Aug 17, 2026

Copy link
Copy Markdown
Member

Summary

  • Validate antenna-health payloads read from DSA Store and raise clear errors for missing, incomplete, or inconsistent
    data.
  • Fall back from the caltable bad-antenna list to selfcorr when caltable state is unavailable.
  • Normalize both LWA-002A and 002A antenna-name formats before converting them to correlator numbers.
  • Add regression tests for the fallback and both antenna-name formats.

Why

Beam 5 scheduled recordings were starting, but each scheduled pointing update aborted while reading /mon/anthealth/ caltable. The missing value was dereferenced as a dictionary, producing:

'NoneType' object is not subscriptable

Consequently, no fresh beam coefficients were sent.

Using selfcorr directly exposed a second issue: its antenna names already include the LWA- prefix, while
control_bf always added another prefix. Names such as LWA-002A therefore became LWA-LWA-002 and failed
correlator mapping.

This patch lets pointing continue with the available selfcorr health list and handles either supported antenna-name
representation.

Validation

  • 9 focused unit tests pass under the deployed deployment Python environment.
  • python -m py_compile mnc/control.py mnc/anthealth.py tests/test_control.py passes.
  • The live beam 5 scheduler resumed periodic Sun tracking.
  • Fresh beam coefficients were issued on all 32 pipelines.
  • The beam 5 UDP power stream remained finite and nonzero after pointing recovery.

Modified files:

  • mnc/control.py
  • mnc/anthealth.py
  • tests/test_control.py

Deployment safety and rollback

Before applying the live recovery patch, copies of the original deployed files were preserved on lwacalim10:

  • /opt/devel/pipeline/envs/deployment/lib/python3.6/site-packages/mnc/control.py.pre-beamfix-20260817
  • /opt/devel/pipeline/envs/deployment/lib/python3.6/site-packages/mnc/anthealth.py.pre-beamfix-20260817

These backups provide a direct rollback path if the deployed changes need to be reverted. The scheduler was not
restarted during the active observation.

@caseyjlaw

Copy link
Copy Markdown
Contributor

I think a patch to improve reliability is good. However, it sounds like the /mon/anthealth/caltable etcd key is not being filled. There is a function caltable_flags that is used to fill that, but I think it is run manually when new tables are available.
Could you test that function with your change on your tables? I would like to be sure that the antenna naming syntax is consistent with your change.

@caseyjlaw caseyjlaw self-assigned this Aug 18, 2026
@caseyjlaw

Copy link
Copy Markdown
Contributor

@peijin94 I'm sorry, but you'll need to rebase after my latest commit to main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants