Fix trading state: settle expired puts, alarm on blind runs, reconcile against broker

[biswajeetdev]

Why

The paper account looked like it was generating no profit. Diagnosis found the opposite, and a set of mechanical faults underneath it.

The Alpaca paper account is at $109,554 (+9.55%), not flat. The bot reports $98,929 because its equity tracking follows the ai4trade.ai sim account while it actually executes through Alpaca — two disjoint accounts.

That gain is not strategy alpha. Essentially all of it is MSFT 114 shares (+$10,527 unrealized) acquired by short-put assignment, which the bot has no record of holding. Net of it, the deliberate strategy is roughly flat-to-negative (realized −$150, tracked unrealized −$185).

What was broken

1. The short-put strategy was hard-blocked for six weeks.
check_exits() computed dte = expiry - today and tested dte <= GAMMA_DTE. A contract expired 40 days ago yields -40, which passes forever — so every run submitted a close order on a contract that no longer trades, failed, and retried. With MAX_OPEN = 3 and all three slots held by dead contracts, every new short-put entry was rejected since Jul 2.

2. 36% of runs were blind and reported as healthy.
209 of 572 runs got no price data, each summarised as a tidy Trades today: 0. Two independent causes, confirmed by bucketing the log: 102 runs DNS-down (Errno 8), 76 runs Yahoo-side empties with no network error, 31 both. Nothing ever surfaced it — which is how two months passed.

3. Local state had drifted completely from the broker.
scripts/reconcile.py reports 12 issues against the live account:

Invisible to the bot	AAPL 10, AMZN −11, ARM 95, MSFT 114 (+$10,527), SOFI −284
Wrong in the bot	META phantom (local qty 4, absent at broker); NVDA local 14 vs broker 29
Stuck	3 expired puts occupying all MAX_OPEN slots
Frozen	equity_history = 30 identical values → the drawdown circuit breaker guards a constant

NVDA is a live risk, not cosmetic: check_stops does its arithmetic on qty 14 while the broker holds 29, so a stop that fires sells 14 and silently orphans 15.

What this changes

• Expired puts settle instead of retrying — resolved from the underlying's close on the expiry date (worthless vs assigned), then released so the slot frees up.
• Degraded-run alarm — a data-health line, a RUN DEGRADED banner past 50% blind, and a Telegram alert through the existing send_error_alert, so it reaches a human instead of dying in cron.log.
• scripts/reconcile.py — read-only local-vs-Alpaca drift report, exits non-zero. Deliberately does not auto-repair; correcting positions is a human decision, not a side effect of a health check.

Correctness note on assignment accounting

The first version recorded an assigned put as credit-minus-intrinsic (MSFT as −$1,771). Assignment realises nothing — the credit is kept and the position converts to stock at a known basis, so that was a mark, not a realised loss.

It mattered because trades flow into trade_history.json, and update_from_trade_history() reads trade["pnl"] straight into regime stats while bootstrap_from_trade_history() feeds the strategy bandit. The old shape would have taught the learning layer that short puts lost ~$2,189 when they in fact collected $4,259 — confidently wrong, which is worse than the stale state being replaced. Fixed in d68facd.

Verification

• All touched files compile; check_exits / settle_expired exercised against the real position file, with live state verified untouched afterwards.
• Settlement cross-checks two ways: realized total $4,259.00 equals credit collected ($680 + $254 + $3,325); computed ARM basis $286.75 matches the broker's actual avg entry of $286.55 on the 95 assigned shares.
• reconcile.py run against the live account: exit 1, 12 issues.

Deliberately out of scope

Trailing stops (checked — correctly held, not broken), the India short-put loop, and reducing the ~40–60 Yahoo calls per run. Also unresolved and left for a human: AMZN −11 and SOFI −284 are short equity positions the bot's schema (action: BUY only) cannot represent — origin unknown.

Also included (earlier commits on this branch)

Debate gate that short-circuits the ~6-call LLM debate on no-catalyst/flat-technical setups, and point-in-time EDGAR catalyst context for the offline eval harness (leak-safe; options-flow excluded for lack of free historical data).

[biswajeetdev]

Added: point-in-time catalyst context in the eval harness (ba5e446)

The eval previously fed the brain only technicals, so it measured a deliberately-cautious technicals-only brain — not the catalyst-driven strategy. Now it reconstructs three top-weighted catalysts from SEC EDGAR filings pinned to filing-date ≤ as-of (leak-free): 8-K events, 13D/G/13F institutional, Form 4 insider. Options-flow has no free point-in-time history → explicitly excluded. Incompatible with --anonymize (filings name the issuer) → anonymize disables catalysts.

Proof-run (NVDA/AAPL, 2 dates each, non-anonymized):

• 4/4 decisions carried a real EDGAR catalyst, 0 gated — the debate gate correctly stood down when catalysts were present, and every date ran the full debate.
• Behavior changed vs. the technicals-only run: the brain issued an actual BUY @75% conf (crossed the 70% threshold) where before it only ever produced low-conviction HOLDs.

Edge numbers themselves are noise at n=4 — this commit is about what gets tested, not a verdict. Leak-safety verified: EDGAR queries return only filings dated ≤ as-of.

[biswajeetdev]

Round 2 — short-side risk, execution guards, atomic state (2dab7f1, 1ce38e7)

Six requested fixes. The short-selling path turned out to be broken in ways that compound, so most of this is that.

SHORT was executing twice

execute_alpaca_trade maps any non-BUY/COVER action to OrderSide.SELL (alpaca_exec.py:88), so it already opened the short — then trader.py called execute_short_sell() for a second identical order. Double the intended size. And record_open() only ran if that second call succeeded, so when it failed the first fill stayed at the broker with no local record. That is the origin of the untracked AMZN -11 / SOFI -284. Removed the duplicate; short_positions.json had no reader, so nothing depended on it.

Shorts were managed backwards

record_open stamped "action": "BUY" on everything with a negative qty and long-side stops, and check_stops only skipped non-BUY rows — so shorts were evaluated with inverted semantics: falling price (a short's profit) fired STOP_LOSS; rising price fired PROFIT_TARGET. The bot closed winning shorts and held losing ones.

Now: record_open takes an explicit action, puts a short's stop above entry and target below, stores qty as a positive size, and rejects anything but BUY/SHORT. check_stops mirrors every comparison and ratchets the trail downward. chandelier_exit gained a short mode. mark_partial_done was left alone — stop_price = entry_price is breakeven in both directions already.

The learning layer was trained backwards on shorts

pnl_pct carried no direction sign, and both consumers read that field (update_from_trade_history → regime stats, bootstrap_from_trade_history → strategy bandit). Fixed in record_close and at both record_signal_outcome call sites.

Execution guards (the fail-safe ask)

• Verbs are normalised to the held direction before anything executes; COVER with no position is skipped; closing orders are clamped to the size actually held.
• 1ce38e7 fixes a blocker found in review: the stop-exit path is a second call site that hardcoded "SELL". On an armed short that would have doubled the position while record_close marked it flat — the same bug through the other door, with unmanaged: true the only thing holding it shut. Both exit sites now derive COVER/SELL from the position's own action.

State integrity

broker/state_io.py: every store used write_text(json.dumps(...)), which truncates before the new bytes land. A crash between the two left a truncated file — and the loaders swallow the parse error and return {}, i.e. "no open positions."

Correction to my earlier commit message: this fixes torn reads, not concurrent interleaving. check_stops and reconcile --fix both do read-modify-write, so a lost update is still possible (cron's trail ratchet could be overwritten). The window is small and the consequence is a stale trail rather than a bad order, so I've left it — but the commit message overstated it.

reconcile.py

Baseline now comes from Alpaca portfolio/history.base_value instead of a hardcoded $100,000, and an unreadable baseline is labelled ASSUMED rather than passed off as fact. Added --fix (adopt broker truth) and --arm SYM (promote to managed). Adopted rows carry unmanaged: true, which check_stops honours: the broker's avg_entry_price is real, but the ATR-derived stop is a guess about intent, and arming one on 114 MSFT shares could liquidate the position at an arbitrary level. Corrections to already-tracked rows deliberately do not recompute an existing stop — moving a live stop is a risk decision, and the log says so explicitly.

Verification

• Long regression re-run: stop, target, partial and close all fire exactly as before.
• Shorts assert stop > entry > target, STOP_LOSS on a rise, PROFIT_TARGET on a fall, PARTIAL_PROFIT on a favourable move, and a cover at 90 from a short at 100 records +$110 / +10% / WIN.
• Armed-short exit driven end-to-end on a --fix-produced SOFI row: verb is COVER, maps to OrderSide.BUY, P&L reads −20.48% on an adverse move.
• unmanaged rows skipped; record_open rejects a bad action; a failed serialise leaves the previous file intact with no temp files left behind; a default reconcile run leaves positions.json byte-identical.

⚠️ Known gap — do not arm the adopted shorts yet

The stop-exit path calls only execute_trade (ai4trade); it never calls execute_alpaca_trade. Longs opened by the bot are covered because BUY gets a native Alpaca bracket, but:

• Bot-opened shorts get no bracket (use_bracket requires side == BUY), so they have no server-side stop at all.
• An armed short hitting its stop would close on ai4trade and locally, while the real Alpaca short stayed open — re-creating the drift this PR exists to fix.

unmanaged: true keeps the adopted shorts inert, which is why this isn't live risk today. Mirroring exits to Alpaca is the right next change, and it needs care so it doesn't double-close bracketed longs. Left out deliberately rather than rushed.