Skip to content

RemapEndToEndTests fails on the self-hosted CI runner (simulator returns no mappings) — passes locally with identical engine #896

Description

@malpern

Unmasked by #891's double-fulfill fix. Before that fix, the full coverage lane crashed mid-run and KEYPATH_ALLOW_TEST_RUNNER_CRASH_SUCCESS=1 reported the lane green anyway — RemapEndToEndTests likely never executed on the runner. With the crash fixed, the lane runs further than ever and these two tests fail consistently:

testRuleCollectionRemapOneToTwo: XCTAssertEqual failed: ("nil") is not equal to ("Optional("2")")
testRuleCollectionRemapRoundTrip: Key a must output b / Key 1 must output 2 / Key q must output w — all nil

Ruled out (verified during #891 babysitting)

  • Engine pin / stale simulator: a poisoned workspace cache was real and is fixed (engine-sha stamp, commit on Fix double-fulfill crash in testSingleKeyCaptureLifecycle (unblocks full-coverage lane) #891) — the failing run rebuilt fresh from the correct pin (cache miss: cached=none expected=6a05a4a…) and still failed.
  • Binary identity: CI and Scripts/build-kanata-simulator.sh both deploy kanata_simulated_input (renamed kanata-simulator).
  • The product itself: same engine passes locally (KEYPATH_SIMULATOR_PATH=build/kanata-simulator swift test --filter RemapEndToEndTests → green) and the installed-app smoke suite exercises real remaps end-to-end through real kanata.

Remaining suspects

  1. Full-lane test-order pollution (these pass in isolation; the lane runs ~4300 tests in one process).
  2. Runner env: CI_ENVIRONMENT=true, KEYPATH_BUNDLED_SIMULATOR_OVERRIDE=/opt/homebrew/bin/kanata-simulator, sandbox/exec context for spawning the simulator from the test process.
  3. The simulator's stdout/stderr on the runner (next diagnostic: instrument SimulatorService to dump raw output on parse-to-nil and re-run on CI).

Interim state

The two tests are skipped when CI_ENVIRONMENT=true (same pattern as SKIP_EVENT_TAP_TESTS), with this issue referenced. Local runs and the installed-app smokes keep real coverage of the remap pipeline.

Related cleanup

With the known crasher fixed, consider retiring KEYPATH_ALLOW_TEST_RUNNER_CRASH_SUCCESS=1 — it converted a crashed lane into a green check, which is how this failure stayed invisible.

Metadata

Metadata

Assignees

No one assigned

    Labels

    tech-debttestingTest coverage and test infrastructure

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions