Skip to content

utils.disk: Added comprehensive disk cleanup utilities#6312

Open
maramsmurthy wants to merge 1 commit into
avocado-framework:masterfrom
maramsmurthy:feature-disk-cleanup-utilities-migration
Open

utils.disk: Added comprehensive disk cleanup utilities#6312
maramsmurthy wants to merge 1 commit into
avocado-framework:masterfrom
maramsmurthy:feature-disk-cleanup-utilities-migration

Conversation

@maramsmurthy

Copy link
Copy Markdown
Contributor

This enables consistent disk cleanup across all storage validation tests.

Functions added:

  • cleanup_disks(): Main API for disk cleanup operations
  • normalize_multipath_devices(): Map devices to multipath
  • build_device_dependencies(): Build cleanup dependency graph
  • cleanup_raid_arrays(): Remove software RAID arrays
  • cleanup_lvm_volumes(): Remove LVM logical volumes
  • cleanup_lvm_groups(): Remove LVM volume groups
  • cleanup_lvm_physical_volumes(): Remove LVM physical volumes
  • unmount_filesystems(): Unmount all filesystems on devices
  • wipe_disk_metadata(): Clear filesystem and RAID metadata
  • zero_disk_start(): Zero first 100MB of disks
  • And 8 additional helper functions

@mr-avocado mr-avocado Bot moved this to Review Requested in Default project Jun 18, 2026
@maramsmurthy

Copy link
Copy Markdown
Contributor Author

Verified these changes by adding it to tests in a CR flow

TestSuite TestRun Summary

host_io_disk_cleanup_bonnie_bonnie Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T07.46-502cf83/job.log
| PASS 12 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_rawread_rawread Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T07.50-85c07fb/job.log
| PASS 1 || CANCEL 1 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_tiobench_tiobench Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T07.51-43a5bde/job.log
| PASS 32 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_disktest_disktest Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T08.02-8790c29/job.log
| PASS 6 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_fiotest_fio Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T08.09-5ac6b60/job.log
| PASS 12 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 4 || INTERRUPT 0 |

host_io_disk_cleanup_softwareraid_softwareraid Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T08.19-76b23b2/job.log
| PASS 23 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_ltp_fsstress_ltp_fsstress Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T08.46-663797c/job.log
| PASS 20 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_ltp_fs_ltp_fs Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T09.53-d7c0b36/job.log
| PASS 80 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_parallel_dd_parallel_dd Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T10.35-301a08d/job.log
| PASS 3 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_fs_mark_fs_mark Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T10.36-0ab5718/job.log
| PASS 12 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_ioping_ioping Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T10.38-8bb43a9/job.log
| PASS 2 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_iozone_iozone Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T10.39-8d6cac0/job.log
| PASS 0 || CANCEL 0 || ERRORS 32 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |

host_io_disk_cleanup_lvsetup_lvsetup Run Successfully executed
/home/disk_cleanup_cr/results/job-2026-06-18T10.53-2d9e3d1/job.log
| PASS 15 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |
11:00:11 INFO : Removing temporary mux dir

No issues observed during cleanup and no breakage to the CR flow.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces comprehensive disk management and cleanup utilities in avocado/utils/disk.py, including multipath normalization, dependency resolution, LVM/RAID teardown, and metadata wiping. The review feedback highlights several critical issues, including incorrect kernel name mapping for RAID slaves, dangerous substring matching during device unmounting, ordering bugs in LVM and RAID cleanup sequences, broken retry logic in metadata wiping due to stderr redirection, and potential multi-line parsing issues when retrieving volume groups.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread avocado/utils/disk.py Outdated
Comment thread avocado/utils/disk.py
Comment thread avocado/utils/disk.py
Comment thread avocado/utils/disk.py
Comment thread avocado/utils/disk.py
Comment thread avocado/utils/disk.py
@codecov

codecov Bot commented Jun 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 8.35214% with 406 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.52%. Comparing base (33fb4ab) to head (6f8341a).

Files with missing lines Patch % Lines
avocado/utils/disk.py 8.35% 406 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6312      +/-   ##
==========================================
- Coverage   71.69%   70.52%   -1.18%     
==========================================
  Files         206      206              
  Lines       23480    23922     +442     
==========================================
+ Hits        16835    16870      +35     
- Misses       6645     7052     +407     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This enables consistent disk cleanup across all storage validation tests.

Functions added:
- cleanup_disks(): Main API for disk cleanup operations
- normalize_multipath_devices(): Map devices to multipath
- build_device_dependencies(): Build cleanup dependency graph
- cleanup_raid_arrays(): Remove software RAID arrays
- cleanup_lvm_volumes(): Remove LVM logical volumes
- cleanup_lvm_groups(): Remove LVM volume groups
- cleanup_lvm_physical_volumes(): Remove LVM physical volumes
- unmount_filesystems(): Unmount all filesystems on devices
- wipe_disk_metadata(): Clear filesystem and RAID metadata
- zero_disk_start(): Zero first 100MB of disks
- And 8 additional helper functions

Code Quality Improvements (addressing review feedback):

Exception Handling:
- Replaced bare 'except Exception:' with specific exception types
  (OSError, ValueError, KeyError, TimeoutError) in 4 locations
- Added debug logging for caught exceptions to improve troubleshooting

Magic Numbers:
- Extracted 20 hardcoded values to module-level constants
  (timeouts, retry counts, buffer sizes) for better maintainability
- Updated 15+ locations to use named constants instead of magic numbers

Robustness Improvements:
- Fixed variable shadowing in normalize_multipath_devices()
- Eliminated TOCTOU race conditions in 3 locations
- Added proper error handling for filesystem operations
- Improved handling of permission errors and missing paths

All improvements maintain zero functional impact for normal operation
while significantly improving code robustness and maintainability.

Testing:
- Validated with 13 test suites in avocado-misc-tests
- 203 tests PASS (bonnie, rawread, tiobench, disktest, fiotest,
  softwareraid, ltp_fsstress, ltp_fs, parallel_dd, fs_mark,
  ioping, lvsetup)
- Zero regressions introduced
- Code quality improvements verified with no behavior changes

Signed-off-by: Maram Srimannarayana Murthy <msmurthy@linux.vnet.ibm.com>
@maramsmurthy maramsmurthy force-pushed the feature-disk-cleanup-utilities-migration branch from dcffe4e to 6f8341a Compare June 18, 2026 12:37
@pevogam

pevogam commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Hi @maramsmurthy is it a good idea to perhaps contribute this directly to https://github.com/avocado-framework/aautils/tree/main/autils? This adds a lot of code that is meant to be migrated to begin with. What do you think?

@maramsmurthy

Copy link
Copy Markdown
Contributor Author

Hi @maramsmurthy is it a good idea to perhaps contribute this directly to https://github.com/avocado-framework/aautils/tree/main/autils? This adds a lot of code that is meant to be migrated to begin with. What do you think?

Hi @pevogam, given that this change is currently part of our CR flow, we have a dependency on the existing implementation at this point. That said, I’m open to migrating it to aautils in the future if needed. Please let me know as part of the migration plan if you expect any changes in this code, so we can align early and avoid rework later.

@pevogam

pevogam commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Hi @maramsmurthy is it a good idea to perhaps contribute this directly to https://github.com/avocado-framework/aautils/tree/main/autils? This adds a lot of code that is meant to be migrated to begin with. What do you think?

Hi @pevogam, given that this change is currently part of our CR flow, we have a dependency on the existing implementation at this point.

I assume by your projects depending on this implementation you mean that you use a patch or modified package in some way? Because the pull request is not merged in the repo and as such is not deployed with the default packages built here. If this then integrated as a patch or anything modular then perhaps it maybe we adjusted later on to simply patch on top of aautils?

That said, I’m open to migrating it to aautils in the future if needed. Please let me know as part of the migration plan if you expect any changes in this code, so we can align early and avoid rework later.

Glad to hear this, I would say at this point the aautils is still fairly unexplored in terms of standardization so I would say bolder changes like this will also be merged less conservatively there (meaning that I don't expect any further changes to be needed unless you could of course provide some unit tests which would be much appreciated).

@maramsmurthy

maramsmurthy commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

Hi @maramsmurthy is it a good idea to perhaps contribute this directly to https://github.com/avocado-framework/aautils/tree/main/autils? This adds a lot of code that is meant to be migrated to begin with. What do you think?

Hi @pevogam, given that this change is currently part of our CR flow, we have a dependency on the existing implementation at this point.

I assume by your projects depending on this implementation you mean that you use a patch or modified package in some way? Because the pull request is not merged in the repo and as such is not deployed with the default packages built here. If this then integrated as a patch or anything modular then perhaps it maybe we adjusted later on to simply patch on top of aautils?

That said, I’m open to migrating it to aautils in the future if needed. Please let me know as part of the migration plan if you expect any changes in this code, so we can align early and avoid rework later.

Glad to hear this, I would say at this point the aautils is still fairly unexplored in terms of standardization so I would say bolder changes like this will also be merged less conservatively there (meaning that I don't expect any further changes to be needed unless you could of course provide some unit tests which would be much appreciated).

Hi @pevogam,

Thank you for the detailed feedback and for being open to future migration possibilities.

I'd like to proceed with avocado.utils for this PR, as it maintains consistency with the current codebase and avoids introducing dependencies on aautils while it's still in the exploration phase. I completely understand your perspective on standardization.

Before we finalize the approach, I'd also like to get @PraveenPenguin opinion on this, especially regarding the long-term direction and whether there are any specific migration considerations we should keep in mind.

In the meantime, if you'd like me to make any other improvements to strengthen this implementation, please let me know and I'll be happy to incorporate them.

Looking forward to hearing from both of you!

@pevogam

pevogam commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Alright, let's also hear the opinion of @PraveenPenguin on this. Maybe also same for @harvey0100 but if they are too busy and turn around within enough given time we may find another way forward together on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Review Requested

Development

Successfully merging this pull request may close these issues.

2 participants