Skip to content

Report syncfiles failures instead of discarding them - #262

Merged
jjohnson42 merged 2 commits into
xcat2:masterfrom
Obihoernchen:syncfiles
Aug 3, 2026
Merged

Report syncfiles failures instead of discarding them#262
jjohnson42 merged 2 commits into
xcat2:masterfrom
Obihoernchen:syncfiles

Conversation

@Obihoernchen

Copy link
Copy Markdown
Member

Problem

A failed syncfiles run was reported as a success. get_syncresult() caught the sync task's exception, logged a repr server side, and returned 200 OK with a null body. The node then called .get('options') on that null resulted in:

c1: 'NoneType' object has no attribute 'get'

syncfileclient still exited 0, so the deployment carried on as if syncing had succeeded. A dangling symlink in the sync tree is an easy way to hit this.

Fix

Return the error to the requestor as a 500 with an error payload. On the node, unwrap the body that grab_url_with_status raises for a non-success status, print it once, and exit non-zero:

c1: Error performing syncfiles: Syncing failed due to unreadable files: /etc/dangling.conf
c1: 'syncfileclient' exited with code 1

Only a failure the server deliberately reported for this sync is terminal. Anything else, such as a dropped connection, is re-raised so the existing retry loop handles it as before. The polled result is also no longer assumed to be a dict.

Scope

Deployments are not aborted. Nothing cares about the exit status, but the failure is visible now and nodeapply -F exits non-zero instead of reporting success.

@jjohnson42

Copy link
Copy Markdown
Member

This was the impetus I needed to fix the duplicate nature of syncfileclient, unfortunately that means to rework this to just the three copies of syncfileclient left:
-common - the version used by almost everything
-el7 diskful - because python2 and not touching that
-el7 diskless - because no way (yet) to share between two os without going full common

get_syncresult() caught the sync task's exception, logged a repr server
side and returned 200 OK with a null body.  The node then called
.get('options') on that null resulted in:

  c1: 'NoneType' object has no attribute 'get'

and syncfileclient still exited 0 as if syncing had succeeded.

Return the error to the requestor as a 500 with an error payload.  On
the node, unwrap the body that grab_url_with_status raises for a
non-success status, print it once and exit non-zero.  Only a failure the
server deliberately reported for this sync is terminal. Anything else,
such as a dropped connection, is re-raised so the existing retry loop
handles it as before.  The same case now reports

  c1: Error performing syncfiles: Syncing failed due to unreadable files: /etc/dangling.conf
  c1: 'syncfileclient' exited with code 1
@Obihoernchen

Obihoernchen commented Aug 3, 2026

Copy link
Copy Markdown
Member Author

@jjohnson42 ahh yeah way better now 👍
Rebased.
Also deleted two missed files in: 165d229

@jjohnson42
jjohnson42 merged commit a806466 into xcat2:master Aug 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants