[mock_tests] Fix Orch object file descriptor leak in test TearDown

[gord1306]

[mock_tests] Fix Orch object file descriptor leak in test TearDown

Description

Fix file descriptor leaks in mock test fixtures that cause "Too many open files" errors when running the full test suite sequentially.

Root Cause

Each mock test SetUp() creates multiple Orch objects via new and stores them in gDirectory using gDirectory.set(). However, in TearDown():

gDirectory.m_values.clear();  // Only clears map entries, does NOT delete objects

Directory::m_values is std::unordered_map<std::string, Orch*>. Calling clear() removes map entries but never invokes destructors on the pointed-to objects.

Since mock_dbconnector.cpp creates real OS sockets (socket(AF_UNIX, SOCK_DGRAM, 0)) for each DBConnector, and each Orch object internally holds multiple DBConnector/RedisPipeline/Table instances, the leaked sockets accumulate across test runs until the process hits the ulimit -n limit.

Fix

Before gDirectory.m_values.clear(), retrieve and delete all Orch objects stored in gDirectory that are not otherwise cleaned up via global pointer deletion:

// Example from routeorch_ut.cpp
void TearDown() override
{
    auto* mux_orch = gDirectory.get<MuxOrch*>();
    delete mux_orch;
    delete m_tunnel_decap_orch;
    m_tunnel_decap_orch = nullptr;
    delete gDirectory.get<FlexCounterOrch*>();
    delete gDirectory.get<FlowCounterRouteOrch*>();
    gDirectory.m_values.clear();
    // ... existing global pointer deletes ...
}

For TunnelDecapOrch (not stored in gDirectory, passed as local variable to MuxOrch constructor), promote to a class member (m_tunnel_decap_orch) to enable proper cleanup.

For mux_rollback_ut.cpp (uses ut_orch_list for reverse-order deletion), add the missing gBufferOrch to ut_orch_list.

Changed Files (8 files, all under tests/mock_tests/)

File	Leaked Objects Fixed
portsorch_ut.cpp	FlexCounterOrch
routeorch_ut.cpp	MuxOrch, TunnelDecapOrch (member), FlexCounterOrch, FlowCounterRouteOrch
qosorch_ut.cpp	FlexCounterOrch
bufferorch_ut.cpp	FlexCounterOrch
fdborch/flush_syncd_notif_ut.cpp	VxlanTunnelOrch
intfsorch_ut.cpp	MuxOrch, TunnelDecapOrch (member), FlexCounterOrch, VNetOrch, VNetCfgRouteOrch, VNetRouteOrch
flowcounterrouteorch_ut.cpp	MuxOrch, TunnelDecapOrch (member), FlexCounterOrch, VNetOrch, VNetCfgRouteOrch, VNetRouteOrch
mux_rollback_ut.cpp	BufferOrch (added to ut_orch_list)

Test Results

• Build: Successful, zero compile errors
• Mock tests: 119/119 PASSED, 0 FAILED
• Test suites: 23 test suites, all green

Risk Assessment

• Risk Level: Low
• Scope: Test code only (tests/mock_tests/), no production code changes
• Backwards Compatibility: No impact on production behavior

[Copilot]

Pull request overview

This PR fixes file descriptor leaks in mock test fixtures that were causing "Too many open files" errors when running the full test suite sequentially. The root cause is that gDirectory.m_values.clear() only removes map entries without deleting the pointed-to Orch objects, which hold file descriptors through DBConnector instances.

Changes:

• Added explicit deletion of Orch objects from gDirectory before calling gDirectory.m_values.clear() in test TearDown methods
• Promoted TunnelDecapOrch local variables to test fixture member variables in three test files to enable proper cleanup
• Added missing gBufferOrch to ut_orch_list in mux_rollback_ut.cpp for reverse-order deletion

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/mock_tests/routeorch_ut.cpp	Added m_tunnel_decap_orch member and deletion of MuxOrch, TunnelDecapOrch, FlexCounterOrch, FlowCounterRouteOrch before gDirectory.clear()
tests/mock_tests/qosorch_ut.cpp	Added deletion of FlexCounterOrch before gDirectory.clear()
tests/mock_tests/portsorch_ut.cpp	Added deletion of FlexCounterOrch before gDirectory.clear()
tests/mock_tests/mux_rollback_ut.cpp	Added gBufferOrch to ut_orch_list for proper cleanup
tests/mock_tests/intfsorch_ut.cpp	Added m_tunnel_decap_orch member and deletion of MuxOrch, TunnelDecapOrch, FlexCounterOrch, VNetOrch, VNetCfgRouteOrch, VNetRouteOrch before gDirectory.clear()
tests/mock_tests/flowcounterrouteorch_ut.cpp	Added m_tunnel_decap_orch member and deletion of MuxOrch, TunnelDecapOrch, FlexCounterOrch, VNetOrch, VNetCfgRouteOrch, VNetRouteOrch before gDirectory.clear()
tests/mock_tests/fdborch/flush_syncd_notif_ut.cpp	Added deletion of VxlanTunnelOrch before gDirectory.clear()
tests/mock_tests/bufferorch_ut.cpp	Added deletion of FlexCounterOrch before gDirectory.clear()

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[gord1306]

File Descriptor Leak Fix - Test Report

Date: 2026-02-26
Branch: 202311.X
Commit: d417f5e [mock_tests] Fix Orch object fd leak in test TearDown
Build Container: frosty_rhodes
Host: 172.16.3.112

---

Executive Summary

This report provides empirical evidence that commit d417f5e6 effectively resolves
the file descriptor (fd) leak in mock test fixtures. Testing was performed with
4 test scenarios: before/after fix crossed with high/low ulimit -n, demonstrating
both the fd reduction and the elimination of "Too many open files" failures.

---

Test Methodology

1. A monitoring script (fd_monitor.sh) polls /proc/<pid>/fd every ~50ms during
   test execution, recording the fd count timeline and peak value.
2. Four test scenarios were executed:
   • Before Fix + High ulimit (ulimit -n = 1048576)
   • After Fix + High ulimit (ulimit -n = 1048576)
   • Before Fix + Low ulimit (ulimit -n = 800)
   • After Fix + Low ulimit (ulimit -n = 800)
3. The test binary (./tests) runs all 119 mock tests from 23 test suites.
4. Between "before" and "after" runs, the 8 patched files were reverted/restored
   using git checkout, and make -j4 was used to rebuild.

---

Results Summary

Scenario	ulimit -n	Peak FD	Steady-State FD	Tests PASSED	Tests FAILED	"Too many open files" Errors
Before Fix (high ulimit)	1048576	1155	1057	119	0	0
After Fix (high ulimit)	1048576	627	462	119	0	0
Before Fix (low ulimit)	800	800	765	90	29	29
After Fix (low ulimit)	800	628	462	119	0	0

Key Metrics

• Peak FD Reduction: 1155 -> 627 (-45.8%)
• Steady-State FD Reduction: 1057 -> 462 (-56.3%)
• Test Failures Eliminated: 29 -> 0 (with ulimit -n = 800)
• All 119 tests pass in both ulimit scenarios after the fix

---

FD Usage Timeline Comparison

Before Fix (high ulimit) - FDs grow monotonically, never released

FD Count
  1155 |                              *
  1058 |                          *       * * * * * * * * * * * * * * * * *
   938 |                        *
   824 |                      *
   721 |                    *
   558 |                  *
   362 |            * * *
   250 |        * *
    90 |      *
     3 |    *
       +----+----+----+----+----+----+----+----+----+----+
       0   250  500  750 1000 1250 1500 1750 2000 2250 2500  (ms)

After Fix (high ulimit) - FDs rise then stabilize at lower level

FD Count
   627 |                                *
   507 |                      *
   437 |                    *
   327 |                *
   282 |            * * *
   250 |        * *
   184 |          *
    90 |      *                           * * * * * * * * * * * * * * * *
     3 |    *                ( note: 462 steady state, not 1057 )
       +----+----+----+----+----+----+----+----+----+----+
       0   250  500  750 1000 1250 1500 1750 2000 2250 2500  (ms)

---

Detailed Failure Analysis (Before Fix, ulimit -n = 800)

All 29 failures are caused by fd exhaustion during test SetUp():

Test Suite (# Failed)	Error Message
BufferOrchTest (1)	readTextFile: failed to read file: 'producer_state_table_apply_view.lua': Too many open files
FdbOrchTest (7)	SelectableTimer: failed to create timerfd, errno: Too many open files
CoppOrchTest (5)	SelectableTimer: failed to create timerfd, errno: Too many open files
FlowcounterRouteOrchTest (2)	readTextFile: failed to read file: 'producer_state_table_apply_view.lua': Too many open files
IntfsOrchTest (1)	readTextFile: failed to read file: 'producer_state_table_apply_view.lua': Too many open files
MuxRollbackTest (13)	SelectableTimer: failed to create timerfd, errno: Too many open files

Pattern: Tests that run after the fd count exceeds the 800 limit fail. Earlier test suites
(PortsOrchTest, RouteOrchTest, QosOrchTest) complete before the threshold is reached.

---

Root Cause Verification

The fd leak is caused by Orch objects created with new in test SetUp() and stored in
gDirectory, but never deleted in TearDown(). Each Orch creates real OS sockets via
mock_dbconnector.cpp:

// mock_dbconnector.cpp
DBConnector::DBConnector(...) {
    m_conn = (redisContext*)calloc(1, sizeof(redisContext));
    m_conn->fd = socket(AF_UNIX, SOCK_DGRAM, 0);  // Real fd!
}

Before fix: gDirectory.m_values.clear() only removes map entries, fds are never closed.
After fix: delete gDirectory.get<Type*>() properly destroys objects and closes sockets.

Steady-State FD Comparison

Metric	Before Fix	After Fix	Delta
Steady-state FD count	1057	462	-595 (-56.3%)
FD growth per test cycle	~5-8 fd/test	~0 fd/test	Eliminated

The 595 fd difference represents the accumulated leaked sockets from ~100 test SetUp/TearDown
cycles that were never released.

---

Files Modified (8 files)

File	Change Description
portsorch_ut.cpp	delete FlexCounterOrch before gDirectory.m_values.clear()
routeorch_ut.cpp	Add m_tunnel_decap_orch member; delete MuxOrch, TunnelDecapOrch, FlexCounterOrch, FlowCounterRouteOrch
qosorch_ut.cpp	delete FlexCounterOrch before gDirectory.m_values.clear()
bufferorch_ut.cpp	delete FlexCounterOrch before gDirectory.m_values.clear()
fdborch/flush_syncd_notif_ut.cpp	delete VxlanTunnelOrch before gDirectory.m_values.clear()
intfsorch_ut.cpp	Add m_tunnel_decap_orch member; delete MuxOrch, TunnelDecapOrch, FlexCounterOrch, VNetOrch, VNetCfgRouteOrch, VNetRouteOrch
flowcounterrouteorch_ut.cpp	Add m_tunnel_decap_orch member; delete MuxOrch, TunnelDecapOrch, FlexCounterOrch, VNetOrch, VNetCfgRouteOrch, VNetRouteOrch
mux_rollback_ut.cpp	Add gBufferOrch to ut_orch_list for proper reverse-order deletion

---

Conclusion

The fix demonstrably:

1. Reduces peak fd usage by 45.8% (1155 -> 627)
2. Reduces steady-state fd usage by 56.3% (1057 -> 462)
3. Eliminates all 29 "Too many open files" test failures under constrained ulimit
4. Maintains 100% test pass rate (119/119) in both ulimit scenarios
5. Zero production code impact - changes are limited to test fixtures only

fd_monitor.sh