Feature Request: Resource Estimation and Parallel Execution for PyCurv Jobs

[hemanthkapa]

Feature Request: Automatic Resource-Aware Parallel Execution for PyCurv Jobs

Summary

Enhance the PyCurv integration to automatically estimate available system resources and run as many PyCurv jobs in parallel as possible, without requiring the user to manually select the number of jobs or files to process at once.

Motivation

Currently, users must manually select which files to run PyCurv on (via checkboxes), or use a "Run All" option. The number of jobs running in parallel is not automatically managed, which can lead to inefficient use of system resources or system overloads.

Automatically estimating available resources and dynamically managing parallel execution will:

• Maximize throughput without overloading the system
• Remove the need for users to manually manage parallelism
• Provide a more robust and user-friendly experience

Requirements

1. Resource Estimation

• Before starting PyCurv jobs, estimate available system resources (RAM, CPU cores, disk space).
• Use the cores parameter from the config as an upper bound for parallel jobs, but do not exceed what is actually available on the system.
• If the user attempts to run more jobs than resources allow (e.g., via "Run All"), automatically limit the number of concurrent jobs.

2. Automatic Parallel Execution

• When the user selects multiple files (or "Run All"), automatically queue and run as many jobs in parallel as resources allow.
• As jobs finish and resources free up, start additional jobs from the queue.
• The user should not need to manually manage how many jobs run at once.

3. UI/UX Improvements

• Clearly display how many jobs will run in parallel, based on current resource estimates.
• Show progress for all queued/running jobs.
• Allow users to cancel individual jobs or the entire batch.
• Provide clear warnings if system resources are low or exhausted.

Acceptance Criteria

• Resource estimation is performed before starting PyCurv jobs, using both system info and the cores config parameter.
• The number of parallel jobs is automatically managed; users do not need to set this manually.
• The GUI displays the number of parallel jobs and progress for all jobs.
• The system remains stable and responsive, even with large batch jobs.
• Users can still select specific files or use "Run All", but parallelism is always resource-aware.

Additional Context

• Related files: src/jobs/pycurv_tab.py, src/jobs/mesh_tab.py, src/main.py