Bots freeze permanently on dm2: one embedded in world solid, one starved of goals

[saworbit]

A dm2 deathmatch tape recorded a single bot frozen for 144.2 s of a 190 s match. That is the longest freeze in the project's history. It reproduces in a second tape at 87.0 s with a different bot, in a different place, on a different build.

Evidence

dm2, current main:   Carmack   t 28.7..172.9  (144.2s) at '2273 -170 -136'  mode 2
dm2, door fix build: Joe Rogan t 104.3..191.3 ( 87.0s) at '1931 -1153 344'  mode 2
                     Romero    t 132.1..139.8 (  7.7s) at '2536 -224 120'   mode 2

Both long ones are mode 2, spd 0, and emit no stall events, which means a stall suppressed hold rather than a wedge. The give up clocks are 8 s for doors and 15 s for trains, so something is holding well past every cap we have.

The tell

Joe Rogan's freeze starts immediately after a vertical jump:

t 103.8  pos '1936 -1221  21.4'  spd 30.2  mode 2
t 104.3  pos '1931 -1153 344.0'  spd  0    mode 2

322 units of z in half a second, onto dm2's east train deck, and ARGEVT Carmack train fires in the same window. So the bot arrived on a mover and then never moved again.

Carmack's 144 s freeze at '2273 -170 -136' starts right after ARGEVT Carmack engage Joe Rogan, from 259 u/s to 0 in one sample.

Why it matters more than the count suggests

A freeze detector counts this as one event, so a tape with a bot doing nothing for 76 per cent of the match still reads as "1 freeze" beside a tape with five short ones. The gate does not weight duration. Worth considering whether the freeze gate should fail on total frozen time as well as on count and worst case.

Suspects, in order

1. ar_liftwait set and never cleared. It is a physics level hold, and the keyed door branch in the door handler sets it with no timeout at all, unlike the ar_doortime branch below it.
2. The train ride hold, given the mover arrival immediately before.
3. Something in the teleport exit coast, given the instantaneous position change.

edicts injected while the bot is frozen would settle this in one command by printing ar_liftwait, ar_door, ar_hoptrain and ar_node directly. See #255.

[saworbit]

Diagnosed with the edict dump from #261. The mover hold hypothesis in the issue body is refuted, and the two long freezes turn out to have different causes.

Method

argus_edicts.py --make-cfg 45 --repeat 4 arms four dumps at 45, 90, 135 and 180 s. Two dm2 deathmatch runs of 205 s caught three freezes, two of them long, with a dump landing inside each window.

frz1  Romero     t 11.2..76.6  (65.4s) at '2352 -103 56'   dumps at t27 and t55 both inside
frz2  Joe Rogan  t 99.5..144.8 (45.3s) at '2556 -414 24'   dump  at t108 inside

Not a mover hold

Neither frozen bot has ar_liftwait, ar_door, ar_doorbtn, ar_hoplift or ar_hoptrain set. All three suspects listed in this issue are wrong.

Joe Rogan is inside the world

His origin reads SOLID in hull 0, at the origin and again at eye height. Every bot that was moving normally in the same dumps reads EMPTY at both.

bot	state	hull 0 at origin	at eye
Joe Rogan	frozen 45.3s	SOLID	SOLID
Romero	frozen 65.4s	EMPTY	EMPTY
Carmack, dump 0	moving	EMPTY	EMPTY
Carmack, dump 1	moving	EMPTY	EMPTY
Joe Rogan, dump 0	moving	EMPTY	EMPTY
Romero, dump 2	moving	EMPTY	EMPTY
Carmack, dump 2	moving	EMPTY	EMPTY

Nearest empty point in the player hull is 32 units away in +y. A bot embedded in geometry cannot walkmove in any direction, which is exactly a permanent freeze with no hold flag set.

This is the case the mre review filed for "the day a bot embeds in a wall", with MOVETYPE_BOUNCE during knockback flight as the suggested cure. Joe Rogan was on 22 health, so he had been taking hits.

Method note: a hull 1 test was tried first and is worthless here. A standing player's origin sits inside hull 1's expanded solid, so every bot reads SOLID including the ones walking around. Only hull 0 separates them.

Romero is not

He is in open space, and stayed at the identical position 2352.5 -103.1 56.0 across two dumps 28 seconds apart. His state:

ar_pending    1.0        route request in flight
ar_goal       (absent)   no goal at all
ar_node       (absent)   no steering node
ar_mode       (absent)   mode 0
ar_hazhold    1.0        hazard steer holding
ar_hazardyaw  190.0
ar_nextai     27.2       the AI tick is running normally
velocity      (absent)   zero

Sixty five seconds with no goal, while Carmack and Joe Rogan routed normally throughout, so the router was not globally blocked. Argus_PickGoal clears ar_pending itself, so a request stuck at 1 means PickGoal was not reached, or was reached and picked nothing repeatedly. The floors around him step 28, 56, 28, -4, 44, -4, -4, 12 across eight headings, so it is broken ground with a hazard hold active.

Suggested split

These want separate work. The embedding is the more tractable and the more damaging: an unstick has precedent in the tree already, since Argus_CoopSafeWarp does spawn_tfog plus setorigin to recover a companion. Detecting it in QC is the interesting part, because pointcontents reads hull 0 and would catch Joe Rogan's case but not a hull 1 wedge.

[saworbit]

Half of this is fixed in #262: the embedding.

A bot inside solid now warps to the nearest nav node after three seconds without moving. Verified recovering a real one on dm4, where Romero fell into the pit at t 33.3, sat motionless inside solid for three seconds, was warped out at t 36.3 and was running at 306 u/s by t 37.2. Without it he would have been there for the rest of the match, which is what Joe Rogan did for 45 s.

Worth recording: the first version of that fix keyed on contents plus a low instantaneous speed, and teleported a healthy bot 192 units into the dm4 pit while it ran the walkway at 320 u/s. Clipping a thin edge for one tick reads solid, and velocity at an apex proves nothing. Only "inside solid AND has not moved" separates the two.

STILL OPEN, and it is the other signature in this issue. Romero's 65 s freeze was in open space, hull 0 EMPTY, no goal, no node, mode 0, with ar_pending stuck at 1 and ar_hazhold set, while the other two bots routed normally throughout. A 27.3 s freeze in the #262 ladder has the same shape and did not trigger the unstick, so it is live on main.

Argus_PickGoal clears ar_pending itself, so a request stuck at 1 means either PickGoal was never reached or it was reached and picked nothing, repeatedly, for over a minute. The ground under him steps 28, 56, 28, -4, 44, -4, -4, 12 across eight headings with a hazard hold active, so a boxed-in hazard steer is the obvious suspect and is not yet evidence.

The instrument for it is the same one: arm dumps with --repeat, catch one inside the window, and read ar_goalstart, ar_failstreak and ar_hazardyaw alongside what is already known.