Skip to content

train --max-time-secs overruns 4x — ACE successor phase doesn't check the deadline #930

Description

@avionicharshit-byte

zli train runs far past --max-time-secs. The overrun happens in the "Training ACE graph N/M" phase, which appears not to check the deadline; earlier phases respect it.

Repro (built from a9de25e, macOS arm64, 8 cores):

# sample dir: 3 CSV files, ~8 MB each (any similar samples reproduce it)
zli train --profile csv samples/ -o out.zlc --max-time-secs 120 -f

Observed, all with a 120 s budget, all exit 0:

  • --profile csv, dataset A (NYC 311 export): 495 s and 504 s wall (two runs)
  • --profile csv, dataset B (Chicago crime export): 543 s wall
  • --profile serial: 146 s wall
  • With --no-ace-successors, the budget holds (60 s budget → 49 s wall)

The progress bar makes it visible: the deadline passes while it cycles "Training ACE graph N/15".

Expected: either the ACE phase checks the deadline (stop early, return current best), or the help text states the budget is advisory and excludes the ACE phase.

Can share the exact sample files if useful.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions