Skip to content

[ZTP] Re-assert front panel port config after ZTP profile removal (202311.X) - #7

Merged
gord1306 merged 1 commit into
202311.Xfrom
sync/reassert-port-config-202311.X
Aug 21, 2026
Merged

[ZTP] Re-assert front panel port config after ZTP profile removal (202311.X)#7
gord1306 merged 1 commit into
202311.Xfrom
sync/reassert-port-config-202311.X

Conversation

@gord1306

Copy link
Copy Markdown

Cherry-pick of #6 onto 202311.X, so that the mainline branch carries the same fix as 202311.X_4630_10g_prod. The resulting ztp-profile.sh is byte-identical to the one on the product branch.

Why

ztp-profile.sh remove deletes ZTP|mode from CONFIG_DB and then restarts interfaces-config, which regenerates /etc/network/interfaces from interfaces.j2. The ZTP in-band block of that template is gated on ZTP['mode'], so once the table is gone the regenerated file contains no Ethernet stanza at all.

interfaces-config.sh then runs systemctl restart networking, which performs ifdown -a followed by ifup -a. ifupdown2 tears down every front panel port it had adopted for ZTP in-band DHCP, and ifup -a does not bring them back because they are no longer listed. The netdev is left administratively down with its MTU reset to the kernel default, while CONFIG_DB, APPL_DB and the ASIC all still report the port as up.

Because show interfaces status reads APPL_DB, the CLI shows up/up and the problem is invisible from there, while anything running on the netdev - LLDP and DHCP in particular - silently stops working. Recovery requires a manual config interface shutdown/startup on each affected port.

Nothing corrects this afterwards: portmgrd only acts on CONFIG_DB events and keeps no desired state. It erases each event once applied, and its periodic tick returns immediately when the queue is empty. Since CONFIG_DB never changed, portmgrd receives no event and the drift persists.

Which ports are affected depends only on which ones happened to be oper up when interfaces-config sampled APPL_DB, so the failure looks intermittent. If several ports come up close together they are all adopted and all end up down.

What this does

After the interfaces-config restart in the remove path, re-assert the front panel port configuration. Writing any field of CONFIG_DB PORT|<port> back with its current value produces a keyspace event, and SubscriberStateTable delivers the entire hash to the consumer, so portmgrd re-applies both mtu and admin_status even though only admin_status is written.

Possible concerns

  • Link flap. None. The write does not change CONFIG_DB content, and portmgrd applying admin_status up to a port that is already up is a no-op at the netdev level. Verified on hardware: ip -o link show is byte-identical before and after, and the HSET returns 0 (no field added).
  • Ordering. interfaces-config.service and networking.service are both Type=oneshot, and neither restart above uses --no-block, so ifdown/ifup have completed by the time the loop runs.
  • Ports intentionally left down. The current value is written back, so a port configured admin_status down stays down.
  • Runtime. Two sonic-db-cli calls per port. Measured at 1.8 s for 54 ports on an AS4630-54PE, inside a teardown path that already restarts two services.

Verification

Validated on an AS4630-54PE running in-band ZTP with an official build. After ZTP removes its profile the teardown is confirmed to have happened - /etc/network/interfaces holds no Ethernet stanza and ZTP|mode is cleared - and all 48 cabled front panel ports stay up with MTU 9100. lldpd keeps transmitting and the peer captures LLDP frames at the expected 30 second interval.

Without the fix, the same environment leaves all 48 ports netdev DOWN with MTU 1500 while show interfaces status reports every one of them as up/up, and no LLDP frames are transmitted at all.

When ZTP completes, ztp-profile.sh remove deletes ZTP|mode from CONFIG_DB and
then restarts interfaces-config. interfaces-config regenerates
/etc/network/interfaces from interfaces.j2, whose ZTP in-band block is gated on
ZTP['mode'] being defined. With that table gone the block is no longer rendered
and the regenerated file contains no Ethernet stanza at all.

interfaces-config.sh then runs "systemctl restart networking", which performs
"ifdown -a" followed by "ifup -a". ifupdown2 tears down every front panel port
it had adopted for ZTP in-band DHCP, and ifup -a does not bring them back
because they are no longer listed. The netdev is left administratively down with
its MTU reset to the kernel default of 1500, while CONFIG_DB, APPL_DB and the
ASIC all continue to report the port as up.

portmgrd only acts on CONFIG_DB events and keeps no desired state, so it never
notices nor corrects the drift. The port stays down until something writes
CONFIG_DB, which is why a manual config interface shutdown/startup recovers it.

Anything running on the Linux netdev, LLDP and DHCP in particular, stops working
on the affected port while "show interfaces status" keeps reporting up/up, since
that command reads APPL_DB. Which ports are affected depends on which ones
happened to be oper up when interfaces-config sampled APPL_DB with redis-dump,
so the failure can look intermittent.

Re-assert the front panel port configuration after the interfaces-config
restart. Writing any field of CONFIG_DB PORT|<port> back with its current value
produces a keyspace event, and SubscriberStateTable delivers the entire hash to
the consumer, so portmgrd re-applies both mtu and admin_status. The write is
idempotent and does not flap a port that is already up.

Measured on a 54 port platform: the loop issues 108 portmgrd re-applications,
mtu and admin_status for each port, and completes in 1.8 seconds.

interfaces-config.service and networking.service are both Type=oneshot and
neither restart uses --no-block, so ifdown/ifup have completed by the time the
loop runs.

Signed-off-by: gord_chen <gord_chen@edge-core.com>
(cherry picked from commit dd82ca1)
@gord1306
gord1306 merged commit 274909a into 202311.X Aug 21, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant