Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
52909bb
docs(backlog): open CATCH-UP-PILOT-SCENARIOS lot
davidp57 Jul 12, 2026
e12960c
fix(tests): F-047 pointed at dead menu paths, missing branch enable
davidp57 Jul 12, 2026
eeeb005
fix(troop): refreshMenuSection flight-state override + interactive ia…
davidp57 Jul 13, 2026
3b54dc4
fix(tests): run_ia_scenario.py handles RUNNING re-injection; TFC stal…
davidp57 Jul 13, 2026
227a2ec
fix(tests): retag 4 mistagged ia scenarios to auto-check, RUNNING sup…
davidp57 Jul 13, 2026
8f3270b
fix(tests): TFC step-7 reinjection race + run_ia_scenario.py progress…
davidp57 Jul 13, 2026
f388e6f
fix(tests): ticket-04 tier audit (4/5 -> auto-check) + farp_repack pr…
davidp57 Jul 13, 2026
a5019a8
fix(tests): ticket-04 drone/farp_repack fixes + run_ia_scenario.py ro…
davidp57 Jul 13, 2026
1010591
fix(tests): close ticket 04 (warehouse deferred), audit+retag ticket …
davidp57 Jul 13, 2026
920718c
fix(tests): purge dead ctld_test framework refs from ticket-05 scenarios
davidp57 Jul 13, 2026
e823477
test(dcs): full ia-tier audit -- retag 15 scenarios auto-check, fix m…
davidp57 Jul 13, 2026
bf9e1e1
test(dcs): add auto-slow tier for AI-heli-flight scenarios (exclude f…
davidp57 Jul 13, 2026
236fc1e
test(dcs): move JTAC drone to auto-slow, bump default poll-timeout 60…
davidp57 Jul 13, 2026
842fac1
test(dcs): soft-reset of CTLD player/menu state between scenarios (--…
davidp57 Jul 13, 2026
63126ec
fix(runner): match compound-ID _SCN vars (_SCN_FI_ATK_RESULT etc.)
davidp57 Jul 13, 2026
dcd5b45
fix(tests): make FARP/FI-ATK/FOB-SCN self-pacing + self-contained (sw…
davidp57 Jul 13, 2026
1edfed8
fix(tests): reset clears JTAC state + TFC tears down on failure
davidp57 Jul 13, 2026
871269d
docs(backlog): close CATCH-UP-PILOT-SCENARIOS -- 66/66 headless sweep…
davidp57 Jul 13, 2026
35abf44
test(dcs): remove redundant mt16, purge last dead ctld_test refs
davidp57 Jul 13, 2026
c25a0ff
docs(testing): per-tier run guide -- command + what's expected of the…
davidp57 Jul 13, 2026
f7bdb33
test(dcs): finalize catch-up — EN translation, tier taxonomy, mt08/mt…
davidp57 Jul 13, 2026
da83c2a
docs(testing): rename recette-procedure -> integration-testing, drop …
davidp57 Jul 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 111 additions & 0 deletions .backlog/CATCH-UP-PILOT-SCENARIOS/PRD.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# Lot CATCH-UP-PILOT-SCENARIOS — run the never-executed `ia`-tier scenarios

Status: ✅ done — 66/66 headless sweep green; only optional/manual remainders (see below)
Branch: test/catch-up-pilot-scenarios → PR → develop
Program: re-tooling CTLD on the VMCT model (see `.backlog/README.md`). Sibling to
`FIX-LIVE-DCS-FAILURES` (closed) and `CLEANUP-LEGACY-DCS-TESTS` (planned).

## Problem Statement

Since the migration to VEAF-dcs-bridge (`DCS-BRIDGE-MCP`, `INTEGRATION-TEST-TAGS`), every live-DCS
validation actually run has stayed in the `auto`/`auto-check` tier (headless, no player needed).
The 34 `ia`-tier scenarios (`pilotPassive`/`pilotActive`, needing a live player-controlled unit —
dcs-bridge has no flight-control API) and the 4 `L6` manual sequences
(`tests/manual_test_sequences.md`) have **never been executed** — only one ad-hoc, unscripted
check (Feature Q whole-vehicle spawn) has ever put a human pilot in the loop, and that wasn't even
one of these scenario files.

This lot runs the full backlog once, groups it into pilot-sized sessions, and fixes whatever it
finds (stale assertions vs current code, or real regressions).

## Scope

- 34 `ia`-tier scenario files: 29 `pilotPassive/`, 3 `pilotActive/` (`scenario_warehouse_cycle.lua`
physically lives in `pilotActive/` despite being `pilotPassive`-shaped — not touched here, out of
scope), 2 `noPlayer/` outliers (`F-046`, `F-047`, which ask for a one-off visual F10 check).
- 4 `L6` manual sequences: MT-01, MT-02, MT-03, MT-06 (`tests/manual_test_sequences.md`).
- Each ticket is one pilot session: David runs `tools/integration-runner/run_ia_scenario.py
--scenario <name>` from his own terminal (built during ticket 02 — injects, mirrors in-game
instructions, polls to a verdict, no AI needed for the loop) and flies/does F10 actions
himself, or follows the checklist for L6. Falls back to the `integration-testing` skill's
manual `exec_lua` loop only for genuine visual-judgment scenarios or scenario debugging.

## Tickets — recommended order (fast/foundational first, heaviest batteries last)

| # | Group | Scenarios | Est. pilot time |
|---|-------|-----------|------------------|
| 01 | `noPlayer` `ia` outliers — quick F10 visual checks | F-046, F-047 | ~5 min |
| 02 | L5 F10 menu visual — foundational, other groups assume these menus are correct | `scenario_crate_menu_sol_vol_visual`, `scenario_troop_menu_sol_vol_visual` | ~15 min |
| 03 | Troop/JTAC core cycle — **all 4 turned out mistagged**, retagged `auto-check` (see ticket 03) | `scenarioTroopsFullCycle_v2`, `scenario_extract_menu`, `scenario_jtac_crate_pack`, `scenario_feature_k_jtac_vehicle` | ~20 min |
| 04 | Multi-group / weight / warehouse | `scenario_multigroup_transport`, `scenario_weight_aggregation`, `scenario_unpack_jtac_drone`, `scenario_warehouse_cycle`, `scenario_farp_repack` | ~25 min |
| 05 | FOB / parachute / FARP scenes / RECON | `scenario_fob_scene`, `scenario_p2_fob_parachute`, `scenario_p3_csfarp_parachute`, `scenario_p4_metal_farp`, `scenario_feature_f_recon_farp` | ~25 min |
| 06 | AI transport / AI zones | `scenario_ai_attack_enemy`, `scenario_ai_goto_wpz`, `scenario_ai_transport_visual`, `scenario_ai_troops`, `scenario_feature_i_attack_enemy`, `scenario_feature_i_goto_wpz` | ~25 min |
| 07 | MT-07 to MT-16 full AI battery (heaviest, 10 scripted scenarios) | `scenario_mt07_ai_troops` … `scenario_mt16_countryside_farp` | ~45 min |
| 08 | L6 manual sequences (checklist, no script) | MT-01, MT-02, MT-03, MT-06 | ~60 min (15 min/MT) |

Total estimate: ~3.5h of live-pilot time, splittable across sessions (one ticket = one sitting,
no need to do them back to back).

## Outcome (2026-07-13)

The initial premise — "34 `ia` scenarios each needing a human pilot, ~3.5h" — was wrong. A full
audit + live validation showed almost all of them run headless:

- **66/66 green** in one headless sweep: `python tools/integration-runner/run_scenarios.py
--no-ai --reset-before-each` (all `auto`/`auto-check`, noPlayer + pilotPassive), player just
parked in a slot.
- **Real pilot burden ≈ 2 short flights** — the two menu-visual `ia (fly)` scenarios
(`crate_menu`/`troop_menu_sol_vol_visual`), both already PASS.
- The catch-up surfaced + fixed a pile of **test-harness defects** (not CTLD product bugs):
wholesale `ia`→`auto-check`/`auto-slow` retag, dead `ctld_test` refs, missing terminal
verdicts (infinite loops), premature re-injection (retry guards), a mission-slot hard-code,
compound-ID `_SCN_` var matching, and — the big one — cross-scenario contamination of CTLD's
shared singletons, fixed with a per-scenario soft reset (`tests/dcs/_reset_state.lua`,
`--reset-before-each`) covering player/menu + JTAC state. Also a real product fix:
`CTLDTroopManager:refreshMenuSection` flight-state override.

**Optional / deferred remainders (not blocking):**
- `auto-slow` AI-heli battery (`scenario_ai_troops` + `mt07..mt14`) — minutes of real AI flight
each; logic already covered fast by `noPlayer` F-176..182. Run on demand via `--tier auto-slow`.
- `scenario_mt16_countryside_farp` (`ia (menu)`) — manual, redundant with the auto
`scenario_farp_countryside_spawn.lua`.
- `scenario_warehouse_cycle` (`ia (fly)`) — blocked on the missing `Farp_FG_Petit_Helipad` mod.
- Ticket 08 — the 4 `L6` manual checklist sequences (MT-01/02/03/06): genuine manual pilot work,
left for a dedicated session (not scripted, no runner involvement).

## Non-goals

- `scenario_ai_transport.lua` (`noPlayer`, `auto` tier) — already covered, not `ia`.
- Fixing `CLEANUP-LEGACY-DCS-TESTS` relics — separate lot.
- Rewriting a scenario's tier or folder placement unless a run reveals it's wrong.

## Tier-audit finding (tickets 03–07) — COMPLETE

The `pilotActive/`/`pilotPassive/` → always `ia` default (from `INTEGRATION-TEST-TAGS`) was a
folder-blanket rule, not a per-file semantic check — and it was **wrong for all but 3 of the 34**.
A full static audit (tickets 03–05 read inline; tickets 06–07's 16 scenarios via a subagent that
read each in full) found that the vast majority never gate on the player's `inAir` or wait on an
F10 click — they drive AI helicopters + timers, or self-verify, needing only a BLUE slot for
position/country. **Final tally of the original 34 `ia`:**

- **`auto-check`** (no human, fast headless via `run_scenarios.py --no-ai` — player just occupies
a slot, doesn't fly).
- **`auto-slow`** (no human either, but minutes to resolve — excluded from the fast `--no-ai`
sweep, run with `--tier auto-slow --poll-timeout 900`): the AI-heli-flight battery
(`scenario_ai_troops` + `mt07..mt14`) **and** the JTAC drone (`scenario_unpack_jtac_drone`,
~13 min of internal timers). Core logic of the AI battery already covered fast/headless by the
`noPlayer` `aiTransport_featureT/U` tests (F-176..182), so that end-to-end tier is optional.

Current tier split across `pilotPassive/`+`pilotActive/`: **18 `auto-check`, 10 `auto-slow`,
4 `ia`**.
- **3 genuine `ia`:**
- `scenario_crate_menu_sol_vol_visual.lua` — `ia (fly)` (sol/vol/sol menu, PASS).
- `scenario_troop_menu_sol_vol_visual.lua` — `ia (fly)` (sol/vol/sol menu, PASS).
- `scenario_warehouse_cycle.lua` — `ia (fly)`, **deferred** (needs the `Farp_FG_Petit_Helipad`
mod, absent).
- `scenario_mt16_countryside_farp.lua` — `ia (menu)`, and **redundant** with the auto-tier
`scenario_farp_countryside_spawn.lua`; never emits a terminal PASS. Optional manual UI check.

So after the catch-up, **the real pilot burden is 2 short menu-visual flights** (both already PASS),
not 34 scenarios. The rest is headless. See `tools/integration-runner/README.md`'s "What `ia`
actually asks of you" for the `(menu)`/`(fly)` qualifier convention.
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# 01 — noPlayer `ia` outliers (quick F10 visual checks)

Status: ✅ done
Type: ia (live DCS, F10 visual confirmation)

## Scenarios

- `tests/dcs/noPlayer/F-046_doubleRefreshIdempotentMenuLooksIdenticalAfterSeco.lua`
- `tests/dcs/noPlayer/F-047_enableFOBMidMissionClearBranchPackVehicles11ItemsP.lua`

Both live in `noPlayer/` but are tagged `ia` because they ask for a one-off visual F10 check the
code itself can't verify — no flight required, just a player slot occupied.

## Resolution (2026-07-12, live with David, UH-1H)

- **F-046**: injected as-is, confirmed live — menu identical after second `refresh()`. PASS.
- **F-047**: originally referenced dead menu paths (`{"CTLD Commands","FOB"}`,
`{"CTLD Commands","Pack Vehicles"}`) — `"CTLD Commands"` and `"Pack Vehicles"` are marked
`STALE` in the i18n files (`CTLD_i18n_en.lua:284-285`), left over from a pre-rename menu
structure. Rewrote to the current tree (`{root, fobSub}` = `{"CTLD","FOBs List"}`,
`{root, cratesSub, packSub}` = `{"CTLD","Crate Commands","Pack Equipt"}`, per
`src/CTLD_fob.lua:519-521` and `src/CTLD_crate.lua:916-918`).
- First re-run: `FOBs List` appeared, but `Pack Equipt` didn't — the node already exists,
created `enabled=false` by production code at slot-in (no packable vehicle nearby), and
`clearBranch` alone doesn't flip `enabled` (see `src/CTLD_crate.lua:941` comment). Added the
missing `menu:setBranchEnabled({root, cratesSub, packSub}, true)` (mirrors
`refreshPackEquiptSection`'s own approach at `CTLD_crate.lua:949`).
- Second re-run: both branches confirmed live — FOBs List visible, Pack Equipt shows 9 items +
Next Page (11 total). PASS.

Note: the pagination behavior itself (9 items + Next Page) is already covered headlessly by
`U-052`; this scenario's remaining value is confirming `setBranchEnabled`/`clearBranch` behave
correctly against a *live-managed* branch, not a synthetic one.

## Acceptance criteria

- [x] Both injected via the `integration-testing` skill loop, verdict read.
- [x] Any FAIL root-caused (stale assertion vs current code, or real bug) and fixed.
41 changes: 41 additions & 0 deletions .backlog/CATCH-UP-PILOT-SCENARIOS/tickets/02-l5-menu-visual.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# 02 — L5 F10 menu visual (foundational)

Status: ✅ done
Type: ia (live DCS, player F10 actions on demand)

## Scenarios

- `tests/dcs/pilotActive/scenario_crate_menu_sol_vol_visual.lua`
- `tests/dcs/pilotActive/scenario_troop_menu_sol_vol_visual.lua`

Confirms the Crate/Troop Commands F10 menu structure on ground / in flight / after landing.
Run first: later groups assume these base menus are correct.

## Progress (2026-07-13, live with David, UH-1H)

- **Crate menu (CMFV-VIS): PASS 5/5.** Fixed a real instruction bug along the way: step C's
checklist listed "Pack Equipt VISIBLE" as expected *after* step B already packed the vehicle
— contradictory (once packed, nothing's left to pack, so it correctly disappears). Corrected
the wording in the scenario file.
- **Troop menu (TMFV): PASS 5/5** (re-confirmed live by David alone via the new
`run_ia_scenario.py`, no AI in the loop). `CTLDTroopManager:refreshMenuSection` always
computed `inAir` live via `_isInAir(unit)`, unlike `CTLDCrateManager:refreshCrateFlightSection`
which accepts an `overrideInAir` param so `onTakeoff`/`onLand` can force the correct state
immediately (`S_EVENT_LAND` fires before `inAir()`'s speed/AGL threshold settles). Fixed:
`refreshMenuSection` now accepts the same `overrideInAir` param, wired through
`onTakeoff`/`onLand`/the flight-state poller (`CTLD_player.lua`) and the scenario's own S3/S5
checks. 3 new busted tests in `tests/ci/unit/menu_gating_spec.lua`
(`refreshMenuSection — overrideInAir forces state`).
- Built `tools/integration-runner/run_ia_scenario.py`: an interactive terminal runner for
`ia`-tier `pilotActive`/`pilotPassive` scenarios that self-verify (most of them) — injects,
mirrors in-game instructions to the terminal, polls to a verdict, no AI needed for the
loop. Re-running the same command resets stuck state (crash recovery) instead of requiring
a DCS restart. Also bumped `HUMAN_TIMEOUT_S` 300→3600s in both scenarios + the pilotActive
template (was a real source of stress/false FAILs, not a useful safety net at 5 min). Use
this for tickets 03–08 instead of the manual `exec_lua` loop.

## Acceptance criteria

- [x] Crate menu (CMFV-VIS): injected, all steps PASS (ground/flight/ground-restored).
- [x] Troop menu (TMFV): re-run live with the `overrideInAir` fix, confirm F-185 PASS.
- [x] Any FAIL root-caused and fixed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# 03 — Troop/JTAC core cycle

Status: ✅ done
Type: **auto-check** (all 4 retagged — see below; none actually need a pilot)

## Scenarios

All 4 were tagged `ia` by the `pilotPassive/` folder-blanket default, but none of them check
real flight state or wait on F10 — they only need a BLUE slot occupied (position/groupId), a
structural precondition, not piloting/judgment. Retagged `auto-check`; runnable via
`run_scenarios.py --no-ai` going forward, not just the interactive runner.

- `tests/dcs/pilotPassive/scenarioTroopsFullCycle_v2.lua` — its stated "TRZ zone in mission"
prerequisite was stale (step 2 builds its own `CTLDTroopGroup` directly, never looks up a
real zone) — verified live no such zone exists in `Test_CTLDNEXT_01.miz`, comment fixed. Used
the `RUNNING: step=N` pattern but returned `STARTED` for incomplete steps (inconsistent with
the return contract) — fixed to return `RUNNING`.
**Second bug found live**: step 7 (destroys 4 targets on a timer, validates the alive JTAC
reacquires a new one each time) had no guard against re-entry — `run_ia_scenario.py`
re-injects the full source every `poll_interval` (2s) while `RUNNING`, and step 7's ~50s
monitoring window has no internal state to stop a second re-injection from installing a
*second* concurrent destroy/snapshot timer, racing the first and corrupting the claim log
(`F-T7.2 FAIL: only 1 distinct target(s) seen`, should be ≥2). Fixed: guard skips reinstalling
the timer if one's already running. Also exposed `_SCN_TFC_CLEANUP` (this scenario had no
external-reset hook at all, unlike the pilotActive templates) since a FAIL inside `check()`
aborts before the step's own state reset, leaving `_G[STEP_N]` stuck re-validating the same
stale data on any re-run.
- `tests/dcs/pilotPassive/scenario_extract_menu.lua` — fully automatic (1 auto step, `STARTED`
pattern), just needs CTLD initialized. Has vestigial "Recette CTLD" F10 menu scaffolding
that's never actually wired to a command — dead code, left as-is (out of scope here).
- `tests/dcs/pilotPassive/scenario_jtac_crate_pack.lua` — spawns its own RED target 800m away
and its own JTAC Hummer 80m away; never reads player position again after the initial spawn.
- `tests/dcs/pilotPassive/scenario_feature_k_jtac_vehicle.lua` — same self-contained spawn
pattern; its header claimed "player must be on the ground" and "targets RED must be present"
but neither is checked in code — comment corrected.

Also fixed `run_scenarios.py`/`run_ia_scenario.py` to re-inject the full source on a `RUNNING`
verdict (previously only `run_ia_scenario.py` did; `run_scenarios.py` failed it outright) — see
`tools/integration-runner/README.md`. And fixed `run_ia_scenario.py` to print progress whenever
the verdict *message* changes, not just when the token itself changes — a multi-step `RUNNING`
scenario's message advances every step while the token stays `RUNNING` the whole time; without
this a long-but-healthy step (like TFC's ~50s step 7) looked indistinguishable from a hang.

Run via `tools/integration-runner/run_ia_scenario.py --scenario <name>` (still works fine post
retag) or `run_scenarios.py --no-ai --scenario <name>` (now that they're `auto-check`).

## Progress

- [x] `scenario_jtac_crate_pack.lua` — PASS (confirmed live, David).
- [x] `scenarioTroopsFullCycle_v2.lua` — PASS (confirmed live, David, after the step-7 race fix).
- [x] `scenario_extract_menu.lua` — PASS 7/7 (confirmed live, David).
- [x] `scenario_feature_k_jtac_vehicle.lua` — PASS (confirmed live, David).

## Acceptance criteria

- [x] All 4 injected, verdicts read.
- [x] Any FAIL root-caused (stale assertion vs current code, or real bug) and fixed.
- [x] Tier mistagging found and corrected (all 4 → `auto-check`).
Loading