[ZTP] Re-assert front panel port config after ZTP profile removal (202311.X) - #7
Merged
Merged
Conversation
When ZTP completes, ztp-profile.sh remove deletes ZTP|mode from CONFIG_DB and then restarts interfaces-config. interfaces-config regenerates /etc/network/interfaces from interfaces.j2, whose ZTP in-band block is gated on ZTP['mode'] being defined. With that table gone the block is no longer rendered and the regenerated file contains no Ethernet stanza at all. interfaces-config.sh then runs "systemctl restart networking", which performs "ifdown -a" followed by "ifup -a". ifupdown2 tears down every front panel port it had adopted for ZTP in-band DHCP, and ifup -a does not bring them back because they are no longer listed. The netdev is left administratively down with its MTU reset to the kernel default of 1500, while CONFIG_DB, APPL_DB and the ASIC all continue to report the port as up. portmgrd only acts on CONFIG_DB events and keeps no desired state, so it never notices nor corrects the drift. The port stays down until something writes CONFIG_DB, which is why a manual config interface shutdown/startup recovers it. Anything running on the Linux netdev, LLDP and DHCP in particular, stops working on the affected port while "show interfaces status" keeps reporting up/up, since that command reads APPL_DB. Which ports are affected depends on which ones happened to be oper up when interfaces-config sampled APPL_DB with redis-dump, so the failure can look intermittent. Re-assert the front panel port configuration after the interfaces-config restart. Writing any field of CONFIG_DB PORT|<port> back with its current value produces a keyspace event, and SubscriberStateTable delivers the entire hash to the consumer, so portmgrd re-applies both mtu and admin_status. The write is idempotent and does not flap a port that is already up. Measured on a 54 port platform: the loop issues 108 portmgrd re-applications, mtu and admin_status for each port, and completes in 1.8 seconds. interfaces-config.service and networking.service are both Type=oneshot and neither restart uses --no-block, so ifdown/ifup have completed by the time the loop runs. Signed-off-by: gord_chen <gord_chen@edge-core.com> (cherry picked from commit dd82ca1)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of #6 onto
202311.X, so that the mainline branch carries the same fix as202311.X_4630_10g_prod. The resultingztp-profile.shis byte-identical to the one on the product branch.Why
ztp-profile.sh removedeletesZTP|modefrom CONFIG_DB and then restartsinterfaces-config, which regenerates/etc/network/interfacesfrominterfaces.j2. The ZTP in-band block of that template is gated onZTP['mode'], so once the table is gone the regenerated file contains no Ethernet stanza at all.interfaces-config.shthen runssystemctl restart networking, which performsifdown -afollowed byifup -a. ifupdown2 tears down every front panel port it had adopted for ZTP in-band DHCP, andifup -adoes not bring them back because they are no longer listed. The netdev is left administratively down with its MTU reset to the kernel default, while CONFIG_DB, APPL_DB and the ASIC all still report the port as up.Because
show interfaces statusreads APPL_DB, the CLI shows up/up and the problem is invisible from there, while anything running on the netdev - LLDP and DHCP in particular - silently stops working. Recovery requires a manualconfig interface shutdown/startupon each affected port.Nothing corrects this afterwards: portmgrd only acts on CONFIG_DB events and keeps no desired state. It erases each event once applied, and its periodic tick returns immediately when the queue is empty. Since CONFIG_DB never changed, portmgrd receives no event and the drift persists.
Which ports are affected depends only on which ones happened to be oper up when
interfaces-configsampled APPL_DB, so the failure looks intermittent. If several ports come up close together they are all adopted and all end up down.What this does
After the
interfaces-configrestart in the remove path, re-assert the front panel port configuration. Writing any field ofCONFIG_DB PORT|<port>back with its current value produces a keyspace event, andSubscriberStateTabledelivers the entire hash to the consumer, so portmgrd re-applies bothmtuandadmin_statuseven though onlyadmin_statusis written.Possible concerns
admin_status upto a port that is already up is a no-op at the netdev level. Verified on hardware:ip -o link showis byte-identical before and after, and theHSETreturns0(no field added).interfaces-config.serviceandnetworking.serviceare bothType=oneshot, and neither restart above uses--no-block, soifdown/ifuphave completed by the time the loop runs.admin_status downstays down.sonic-db-clicalls per port. Measured at 1.8 s for 54 ports on an AS4630-54PE, inside a teardown path that already restarts two services.Verification
Validated on an AS4630-54PE running in-band ZTP with an official build. After ZTP removes its profile the teardown is confirmed to have happened -
/etc/network/interfacesholds no Ethernet stanza andZTP|modeis cleared - and all 48 cabled front panel ports stay up with MTU 9100. lldpd keeps transmitting and the peer captures LLDP frames at the expected 30 second interval.Without the fix, the same environment leaves all 48 ports netdev DOWN with MTU 1500 while
show interfaces statusreports every one of them as up/up, and no LLDP frames are transmitted at all.