diff --git a/CLAUDE.md b/CLAUDE.md index 7c4734a8..e4d76a28 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -7,6 +7,7 @@ LLM multi-agent 美股量化交易系统，通过 Alpaca 执行交易（默认 p ```bash pytest tests/ -v # 全量测试 python main.py --mode morning|midday|evening|live # 手动跑 +python main.py --mode daily # P&L 历史 CSV → Telegram（无 LLM 无交易） ``` **Prompt/CoT 改动验证（决策重放 harness，2026-06-07 加）**：prompt 改动这里没法回测,所以容易"听起来对就上"。`scripts/replay_decision.py` 把 `agent_logs` 里某次真实调用的 `input_message` 喂回**当前** prompt+model,结构化 diff 新旧决策(PM 比 per-symbol target 权重/conviction 的增删改),把"我觉得这版更好"变成"它在 N 个真实输入上具体怎么改了决策"。`--no-llm` 只列待重放、不烧 token；要 LLM 时需 `.env` key。核心逻辑在 `src/replay.py`(load/replay/diff,有单测),靠 `base.py:BaseAgent._execute(user_message)` 这个 seam(= `run()` 去掉 build_user_message 的部分)。**这是"先验证再改"的工具——P1-P5/#1-#2 这类软层改动应该用它在历史决策点上看效果,而不是只靠论证**。outcome-aware 评分(对比次日/5日真实走势判好坏)是它之上的下一层,尚未建。 @@ -19,12 +20,12 @@ python main.py --mode morning|midday|evening|live # 手动跑 - 8 个日常 LLM agent：tech / news / macro / earnings / portfolio_manager / risk_manager / position_reviewer / evening_analyst。额外一个 **meta_reflector** 每季度末跑一次，负责对 6 个可编辑 agent（tech / news / macro / earnings / PM / evening）**自我画像 + prompt 审计 + 自动修订**。risk_manager 和 position_reviewer 被 **schema + prompt_editor 双层保护**，不允许被 auto-evolved 改动（硬纪律不能被稀释） - 双层风控：硬规则引擎（cash_only / 仓位 / 暴露 / 日损 / 板块 / 相关性 / earnings-queued） + LLM RiskManager 审核 + `_force_delever()` 硬兜底 -- **6 个 session**（ET Mon-Fri）：earnings_preprocess 08:00-09:15（唯一跑 earnings LLM）、morning 09:30-12:00（full team）、intra_check 09:30-16:00 每 30min tick（熔断器，零 LLM）、midday 13:00-14:30（position_reviewer patient）、close 15:30-16:00（position_reviewer act-on-trigger；窗口 ≥ launchd 30min tick 保证任何 phase 都能打中）、evening 20:00-22:00（report + outlook）。**季度末额外一次 meta**：`--mode meta` / `run_quarterly_meta_reflection`，跑 `quarterly_digest` 聚合 90 天事实 + `meta_reflector` LLM 7 步 CoT + `prompt_editor` 4 道保险 apply +- **6 个 session**（ET Mon-Fri）：earnings_preprocess 08:00-09:15（唯一跑 earnings LLM）、morning 09:30-12:00（full team）、intra_check 09:30-16:00 每 30min tick（熔断器，零 LLM）、midday 13:00-14:30（position_reviewer patient）、close 15:30-16:00（position_reviewer act-on-trigger；窗口 ≥ launchd 30min tick 保证任何 phase 都能打中）、evening 20:00-22:00（report + outlook）。**季度末额外一次 meta**：`--mode meta` / `run_quarterly_meta_reflection`，跑 `quarterly_digest` 聚合 90 天事实 + `meta_reflector` LLM 7 步 CoT + `prompt_editor` 4 道保险 apply。**另有 daily CSV 导出**（2026-06-10 PR #99）：`--mode daily` 把全量 NAV/SPY/drawdown 历史发成 Telegram 文档（取代 evening 推送里的 P&L 文本表）；纯数据导出（零 LLM 零交易），**不走** run_if_et_window.sh（wrapper 会拒掉 daily 模式），由独立 timer `quant-agent-daily.timer`（Mon-Fri 09:00 ET）经 `scripts/run_daily_export.sh` 触发，units 在 `scripts/systemd/` - 数据源：yfinance、FRED、RSS、SEC EDGAR - 配置：`config/settings.yaml` + `.env`；按 agent 独立选 OpenAI / Anthropic / DeepSeek 模型。**2026-06-04 起所有 9 个 agent 用 OpenAI `gpt-5.5`**（5-11 曾切 claude-opus-4-7 应对 OpenAI quota，6-04 又切回 OpenAI 并升到 5.5；切 provider 一条 `sed` 命令）。Provider 路由按 model name 前缀判断：`deepseek-` 走 DeepSeek，`gpt-` / `o1-` / `o3-` / `o4-` 走 OpenAI，其它走 Anthropic（`src/agents/base.py` 的 `_DEEPSEEK_PREFIXES` / `_OPENAI_PREFIXES`） - **DeepSeek（OpenAI-compatible，2026-06-05 加）**：走 openai SDK + `base_url=https://api.deepseek.com` + DeepSeek key（`_call_deepseek`）。三个坑都已处理（研究自 api-docs.deepseek.com）：(1) DeepSeek 只认 **`max_tokens`** 不认 OpenAI 的 `max_completion_tokens`（发错会被静默丢弃 → 回落 ~4096 默认截断）；(2) DeepSeek **拒绝**（不裁剪）超 ceiling 的 max_tokens，所以按 `_DEEPSEEK_MAX_OUTPUT` per-model 客户端 clamp（v4-flash/pro/chat/reasoner=384K，未知 deepseek-* 保守 8192）；(3) 402「Insufficient Balance」= 不可重试 → 触发 failover，`insufficient_system_resource` finish_reason 记为 truncated。**`deepseek-chat`/`deepseek-reasoner` 2026-07-24 弃用**（现已 alias `deepseek-v4-flash`），新配置直接用 `deepseek-v4-flash`。cost 用**官方** $0.14/$0.28（LiteLLM 的 $0.28/$0.42 是 V4 前旧值，已 **pin** 在 `cost_table._PRICING_PINNED` 防 cache 刷新覆盖） - **跨 provider 自动 failover**：当**非-Anthropic 主**（OpenAI 或 DeepSeek）调用重试耗尽 / 非可重试错误(quota、DeepSeek 402、死 key、宕机)后，`base.py:run()` 会**自动用 Anthropic 的 `_FALLBACK_MODEL`(=`claude-opus-4-7`)单发一次**(无重试,避免吃穿 session 窗口),成功就用它的结果继续(`AgentResult.model` 记实际用的模型,cost 按实际模型算)、失败就抛出原始错误。只在「主=OpenAI/DeepSeek 且 `.env` 有 ANTHROPIC_API_KEY」时触发;主已是 Claude 则 no-op(同 provider 无意义,且构造时不会因此报错)。截断(max_tokens)不触发 failover。pipeline 给 9 个 agent 都传 `fallback_api_key=config.api_keys.anthropic`。`src/config.py` 的 LLMConfig 默认(settings.yaml 漏配时的兜底)也已从过时的 `*-4-6` 更到 `claude-opus-4-7` -- **Telegram 推送**：开/关由 `.env` 控制，缺 `TELEGRAM_BOT_TOKEN` 或 `TELEGRAM_CHAT_ID` 时 notifier 静默 no-op，trading 不受影响。每个 session 在 `main.py` finally 块里调一次 `notifier.send(format_session_result(...))`；噪声策略：morning/midday/close/evening 总推；earnings_preprocess 只在真分析了 filing 时推；intra_check 只在 emergency 触发时推（14 次/天 OK tick 静默）；meta 只在真季末跑时推；**任何 session 抛异常都强制推**绕过噪声策略。文档见 `src/notifier.py` docstring 和 README "Optional env vars" 段 +- **Telegram 推送**：开/关由 `.env` 控制，缺 `TELEGRAM_BOT_TOKEN` 或 `TELEGRAM_CHAT_ID` 时 notifier 静默 no-op，trading 不受影响。每个 session 在 `main.py` finally 块里调一次 `notifier.send(format_session_result(...))`；噪声策略：morning/midday/close/evening 总推；earnings_preprocess 只在真分析了 filing 时推；intra_check 只在 emergency 触发时推（14 次/天 OK tick 静默）；meta 只在真季末跑时推；daily 的 `sent` 静默（CSV 文档本身就是送达确认，status text 只在 error/skipped 时推且带原因）；**任何 session 抛异常都强制推**绕过噪声策略。文档见 `src/notifier.py` docstring 和 README "Optional env vars" 段 ### Agent CoT 结构（schema-enforced 必填字段数；违反 → ValidationError） | Agent | CoT 步数 | 备注 | @@ -78,7 +79,7 @@ python main.py --mode morning|midday|evening|live # 手动跑 - `src/notifier.py:TelegramNotifier`，在 `main.py` finally 块里调用 `format_session_result(mode, result, elapsed, error=...)` 推送。**Hook 必须在 finally 里**，不能在 try 内部——否则 session 抛异常时收不到 FAILED 推送（这是日志之外操作员唯一的实时信号） - 错误必须 swallow：`notifier.send()` 内部 `except Exception` 兜住所有 HTTP / 网络 / Telegram-端报错；main.py finally 块再包一层 try/except 防 notifier 自己挂。**`notify` 失败永远不能让 session 失败**，这条比"得到通知"重要 - 噪声策略由 `format_session_result` 返回 `None` 实现（caller 看 `None` 就 skip send）。policy 见模块 docstring；改这条策略前先想清楚"这个 silence 是不是把真信号也吞了"——典型反例：earnings_preprocess 当时把 `analysis_error` 也 silence 过，结果 OpenAI quota 耗尽那天 13 个 filing 全 retry 烧 token 但没人收到通知（2026-05-11 修，`analysis_error` 现在推） -- **确定性升级告警**：evening 推送的 🚨 banner 不只看 LLM 的 `risk_rating`——`_append_evening_body` 还独立算"今日亏损 ≥ 80% 日损熔断线"（用 `result["max_daily_loss_pct"]`）触发 `DETERMINISTIC ALERT`，与 LLM 判断 OR。理由：自评风险时 LLM 最容易**低估**，而这正是最该被抓住的情况——镜像交易路径"硬规则 + LLM"两层哲学。`suggested_actions` 已挪到 P&L 历史表**之前**渲染，避免 4000 字尾部截断在高风险日吃掉最该读的行 +- **确定性升级告警**：evening 推送的 🚨 banner 不只看 LLM 的 `risk_rating`——`_append_evening_body` 还独立算"今日亏损 ≥ 80% 日损熔断线"（用 `result["max_daily_loss_pct"]`）触发 `DETERMINISTIC ALERT`，与 LLM 判断 OR。理由：自评风险时 LLM 最容易**低估**，而这正是最该被抓住的情况——镜像交易路径"硬规则 + LLM"两层哲学。`suggested_actions` 渲染在 headline P&L 之后的高位，避免 4000 字尾部截断在高风险日吃掉最该读的行（其后原有的 P&L 历史文本表 2026-06-10 起移除，改由 `--mode daily` 的 CSV 导出承担） - **内部 dead-man's check**：evening（已 gated 在交易日）调 `_expected_sessions_missing_today()`，查 `agent_logs` 今日 ET 是否有 morning(`run-`)/midday/close 的 run_id 前缀；缺了就在推送顶部 🔴(morning)/⚠️(midday·close)。catch "某 session 静默没跑"（timer 挂 / lock 卡 / 半日盘窗口算错）——push-on-completion 观测唯一看不见的失败模式。**不覆盖主机宕机 / evening 本身没跑**——那需要外部 dead-man's switch（healthchecks.io 式，wrapper 成功就 ping、缺席就外部报警），建议补上 - Telegram bot token 等同密码：写 `.env` 用 `chmod 600`，**不要**贴 git / issue / 公开 chat。token 万一外泄（推送的截图 / 误贴 ssh log）马上去 BotFather 发 `/revoke` 生新的 diff --git a/README.md b/README.md index eb11ab74..45ae0ff2 100644 --- a/README.md +++ b/README.md @@ -280,6 +280,9 @@ python main.py --mode midday # Position review + trailing stops python main.py --mode evening # PnL report + insights for tomorrow python main.py --mode live # APScheduler in-process (dev/legacy; production # uses systemd/launchd timers, not this) +python main.py --mode daily # P&L history CSV -> Telegram document + # (no LLM, no trading; full NAV/SPY/drawdown + # history from Alpaca portfolio_history) ``` **Automated scheduling**: the production path is a 30-min OS-level timer (systemd `quant-agent@.timer` on Linux, launchd plist on macOS) that calls `scripts/run_if_et_window.sh ` for each session. The wrapper checks the current **US/Eastern** wall clock against the target window, applies the cross-mode session lock (one heavy LLM session at a time, except `intra_check` which is exempt), and skips if the mode already ran today. Runs the right session at the right ET moment regardless of the host's timezone — handy when traveling. Windows (Mon-Fri ET, authoritative Python table at `src/trading_calendar.py` `SESSION_WINDOWS`, locked to the bash wrapper by `test_trading_calendar.py`): @@ -290,6 +293,8 @@ python main.py --mode live # APScheduler in-process (dev/legacy; productio - `close` 15:30-16:00 ET — position review (act-on-trigger; window ≥ 30-min OS-timer tick so it never misses) - `evening` 20:00-22:00 ET — daily P&L + insights for next morning +The **daily P&L CSV export** (`--mode daily`) is scheduled separately — it is a pure data export (no LLM, no orders), so it skips the window/lock wrapper entirely. A fixed-time systemd timer fires it Mon-Fri 09:00 ET via `scripts/run_daily_export.sh` (sources `.env`, 300s timeout). Units are tracked at `scripts/systemd/quant-agent-daily.{service,timer}`; install with `cp scripts/systemd/quant-agent-daily.* ~/.config/systemd/user/ && systemctl --user daemon-reload && systemctl --user enable --now quant-agent-daily.timer`. The CSV replaced the P&L history text table that the evening push used to embed. + ## Trading Universe 97 symbols (source of truth: `config/settings.yaml:trading.universe`): diff --git a/main.py b/main.py index d0bb9a40..2e2d6100 100644 --- a/main.py +++ b/main.py @@ -50,7 +50,7 @@ def main(): "--mode", choices=[ "live", "once", "morning", "midday", "close", "evening", - "intra_check", "earnings_preprocess", "meta", "weekly", + "intra_check", "earnings_preprocess", "meta", "daily", ], default="once", help="Run mode", ) @@ -65,61 +65,87 @@ def main(): ) args = parser.parse_args() - config_path = Path(args.config) - if not config_path.is_absolute(): - config_path = PROJECT_ROOT / config_path - if not config_path.exists(): - logger.error("Config file not found: %s", config_path) - sys.exit(1) - - config = load_config(config_path) - logger.info("Config loaded. Universe: %s, Paper: %s", config.trading.universe, config.alpaca.paper) - - # Loud startup warning when running against the live Alpaca endpoint. - # Operators flipping `alpaca.paper: false` in the YAML is the single - # action that converts every subsequent BUY/SELL into a real-money - # order — make sure they SEE the change at every startup, not just - # the first one. Telegram operators who never look at logs still see - # the order list itself, but a launchd one-off run is the dangerous - # case (no Telegram, no live tail) where a misconfigured config - # could silently flip paper → live with no human-visible signal. - if not config.alpaca.paper: - logger.warning( - "LIVE TRADING ENABLED (alpaca.paper=false). Real-money orders " - "will be submitted via the Alpaca API key from .env. To revert " - "to paper trading, set `alpaca.paper: true` in your config." - ) - - # Refresh LLM pricing from LiteLLM's public JSON if our cache is - # stale (>24h). Best-effort: fetch failure or no-network falls back - # to the in-memory PRICING dict (cache or hardcoded baseline). - # Cost tracking is observability-only — a stale price table never - # blocks trading. - try: - refresh_pricing() - except Exception as exc: - logger.warning("pricing refresh failed at startup: %s", exc) - + # Construct the notifier and the finally-block state FIRST — before + # anything that can crash (config loading, pricing refresh, pipeline + # construction). A crash in any of those used to be a complete blind + # spot: no notifier existed yet and the protective try/finally hadn't + # started, so nothing could tell the operator the session never ran + # (cf. the missing-Saturday-report incident — a pydantic + # ValidationError thrown by load_config()). + # + # HONEST LIMIT: the notifier reads TELEGRAM_* from os.environ at + # construction, so when the root cause is ".env was never sourced" + # (the Saturday incident's actual trigger) the creds are missing too + # and the FAILED push is silently dropped. This restructure covers + # every other early crash (bad YAML while creds are exported, broken + # pricing cache, TradingPipeline/scheduler construction); the + # no-env case is only catchable by an external dead-man's switch + # (see CLAUDE.md observability section). notifier = TelegramNotifier() - - if args.mode == "live": - # The blocking scheduler runs forever and never reaches the - # finally block below. Per-session Telegram notifications are - # therefore emitted by TradingScheduler._run_safe (its own - # finally hook + format_session_result), which mirrors the - # one-shot path here. audit F6: this used to be only a comment - # claiming parity while _run_safe in fact just logged. - notifier.send("🟢 quant-agent live scheduler starting") - scheduler = TradingScheduler(config) - scheduler.setup() - scheduler.start() - return - - pipeline = TradingPipeline(config) start = time.monotonic() result = None error: BaseException | None = None try: + config_path = Path(args.config) + if not config_path.is_absolute(): + config_path = PROJECT_ROOT / config_path + if not config_path.exists(): + logger.error("Config file not found: %s", config_path) + # Exit with the message, not a bare code: this SystemExit is + # caught below and pushed to Telegram, and str(SystemExit(1)) + # is just "1" — useless from a phone. A string arg keeps the + # non-zero exit code AND gives the push (and stderr) the path. + sys.exit(f"Config file not found: {config_path}") + + config = load_config(config_path) + logger.info("Config loaded. Universe: %s, Paper: %s", config.trading.universe, config.alpaca.paper) + + # Loud startup warning when running against the live Alpaca endpoint. + # Operators flipping `alpaca.paper: false` in the YAML is the single + # action that converts every subsequent BUY/SELL into a real-money + # order — make sure they SEE the change at every startup, not just + # the first one. Telegram operators who never look at logs still see + # the order list itself, but a launchd one-off run is the dangerous + # case (no Telegram, no live tail) where a misconfigured config + # could silently flip paper → live with no human-visible signal. + if not config.alpaca.paper: + logger.warning( + "LIVE TRADING ENABLED (alpaca.paper=false). Real-money orders " + "will be submitted via the Alpaca API key from .env. To revert " + "to paper trading, set `alpaca.paper: true` in your config." + ) + + # Refresh LLM pricing from LiteLLM's public JSON if our cache is + # stale (>24h). Best-effort: fetch failure or no-network falls back + # to the in-memory PRICING dict (cache or hardcoded baseline). + # Cost tracking is observability-only — a stale price table never + # blocks trading. + try: + refresh_pricing() + except Exception as exc: + logger.warning("pricing refresh failed at startup: %s", exc) + + if args.mode == "live": + # The blocking scheduler runs forever in the normal case and + # never reaches the finally block below. Per-session Telegram + # notifications are therefore emitted by TradingScheduler._run_safe + # (its own finally hook + format_session_result), which mirrors + # the one-shot path here. audit F6: this used to be only a + # comment claiming parity while _run_safe in fact just logged. + # If the scheduler itself crashes at startup or exits, the + # finally block below now also catches it (previously silent). + notifier.send("🟢 quant-agent live scheduler starting") + scheduler = TradingScheduler(config) + scheduler.setup() + scheduler.start() + # Reached only if the blocking scheduler returns gracefully + # (no exception). Without a result dict the finally block + # would push the cryptic "⚪ live returned non-dict result / + # type: NoneType" — say what actually happened instead. + result = {"status": "scheduler_exited", "run_id": "live"} + return + + pipeline = TradingPipeline(config) if args.mode == "once" or args.mode == "morning": result = pipeline.run_morning() elif args.mode == "midday": @@ -134,12 +160,13 @@ def main(): result = pipeline.run_earnings_preprocess() elif args.mode == "meta": result = pipeline.run_quarterly_meta_reflection(force=args.force) - elif args.mode == "weekly": - result = pipeline.run_weekly() + elif args.mode == "daily": + result = pipeline.run_daily() except BaseException as exc: # Catch broadly (incl. SystemExit / KeyboardInterrupt) so a - # wrapper-kill or ctrl-C still gets a notification — but - # re-raise so the process exits with the proper status code. + # wrapper-kill, a config-load crash, or ctrl-C still gets a + # notification — but re-raise so the process exits with the + # proper status code. error = exc raise finally: diff --git a/scripts/run_daily_export.sh b/scripts/run_daily_export.sh new file mode 100755 index 00000000..a0825edf --- /dev/null +++ b/scripts/run_daily_export.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# Daily P&L CSV export runner (`main.py --mode daily`) — systemd entry point. +# +# Deliberately NOT routed through run_if_et_window.sh: that wrapper exists +# for the 6 ET-windowed trading sessions (window check, last-run dedup, +# cross-mode session lock) and rejects "daily" as an unknown mode by design. +# The daily export is a pure data read (no LLM, no orders, no trading-DB +# writes) fired by a fixed-time timer (quant-agent-daily.timer, Mon-Fri +# 09:00 America/New_York), so none of that machinery applies. +# +# What it DOES share with the wrapper: .env sourcing (Telegram + Alpaca +# creds — without it the run crashes config validation AND the notifier is +# creds-less, i.e. the original missing-Saturday-report failure mode) and +# an outer timeout so a hung HTTP call can't wedge the unit. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(dirname "$SCRIPT_DIR")" +PYTHON="${PROJECT_ROOT}/.venv/bin/python" +# Linux: /usr/bin/timeout. macOS fallback: brew coreutils via TIMEOUT_OVERRIDE +# (same convention as run_if_et_window.sh). +TIMEOUT="${TIMEOUT_OVERRIDE:-/usr/bin/timeout}" + +cd "$PROJECT_ROOT" + +if [[ -f "${PROJECT_ROOT}/.env" ]]; then + # shellcheck disable=SC1091 + set -a + source "${PROJECT_ROOT}/.env" + set +a +fi + +# 300s is generous: the export is one portfolio_history call, one yfinance +# SPY fetch, one Telegram upload. (Trading sessions use 1200s; not needed.) +exec "$TIMEOUT" --kill-after=30 300 "$PYTHON" main.py --mode daily diff --git a/scripts/systemd/quant-agent-daily.service b/scripts/systemd/quant-agent-daily.service new file mode 100644 index 00000000..2f6da45b --- /dev/null +++ b/scripts/systemd/quant-agent-daily.service @@ -0,0 +1,17 @@ +# Daily P&L CSV export — pairs with quant-agent-daily.timer. +# Install: +# cp scripts/systemd/quant-agent-daily.* ~/.config/systemd/user/ +# systemctl --user daemon-reload +# systemctl --user enable --now quant-agent-daily.timer +[Unit] +Description=quant-agent daily P&L CSV export (no LLM, no trading) + +[Service] +Type=oneshot +WorkingDirectory=/home/yebo/quant-agent +# The runner sources .env and applies its own 300s timeout. +ExecStart=/home/yebo/quant-agent/scripts/run_daily_export.sh +# systemd safety net above the runner's 300+30s kill. +TimeoutStartSec=420 +StandardOutput=journal +StandardError=journal diff --git a/scripts/systemd/quant-agent-daily.timer b/scripts/systemd/quant-agent-daily.timer new file mode 100644 index 00000000..d492e4da --- /dev/null +++ b/scripts/systemd/quant-agent-daily.timer @@ -0,0 +1,17 @@ +# Fires the daily P&L CSV export Mon-Fri at 09:00 ET. +[Unit] +Description=quant-agent daily P&L CSV export — Mon-Fri 09:00 ET + +[Timer] +# systemd resolves the IANA timezone, so this stays 09:00 New York across +# DST — matching the project's everything-in-ET discipline. 09:00 ET also +# guarantees yesterday's portfolio_history bar has settled (the 20:00 ET +# evening run is often a day behind; by the next morning it never is). +# Market holidays: the export still fires and simply re-sends history with +# no new row — harmless, and simpler than teaching systemd the NYSE calendar. +OnCalendar=Mon..Fri 09:00 America/New_York +# Run a missed firing on boot (server was down at 09:00). +Persistent=true + +[Install] +WantedBy=timers.target diff --git a/src/notifier.py b/src/notifier.py index c48210df..14ea076f 100644 --- a/src/notifier.py +++ b/src/notifier.py @@ -12,6 +12,10 @@ - intra_check: notify only on emergency action (skip the 14 silent OK ticks per trading day) - meta: notify on actual run; skip "not_quarter_end" / etc. + - daily (P&L CSV export): the CSV itself goes out as a Telegram + document with a self-describing caption, so the "sent" status + text is suppressed (the document IS the confirmation); "error" + (with the reason) and "skipped" still notify - Any session that raised an exception: always notify """ from __future__ import annotations @@ -181,6 +185,11 @@ def format_session_result( return None if mode == "meta" and status == "skipped": return None # quarter-end check fires daily; silent on non-Q-end + if mode == "daily" and status == "sent": + # The CSV document push (with its self-describing caption) IS + # the delivery confirmation — a second status text every weekday + # would be pure noise. error / skipped still notify below. + return None run_id = result.get("run_id", "?") emoji = _status_emoji(status) @@ -210,10 +219,16 @@ def format_session_result( _append_intra_check_body(lines, result) elif mode == "meta": _append_meta_body(lines, result) - elif mode == "weekly": - rows = result.get("rows", "?") + elif mode == "daily": + # Only error / skipped reach here ("sent" is silenced above). + # Surface the failure reason — a bare '🔴 status: error' is + # undebuggable from a phone. filename = result.get("filename", "") - lines.append(f"📊 {rows} rows → {filename}") + if filename: + lines.append(f"📊 {result.get('rows', '?')} rows → {filename}") + err = result.get("error") + if err: + lines.append(f"error: {err}") lines.append(f"elapsed: {elapsed_str}") return "\n".join(lines) @@ -301,141 +316,6 @@ def _append_trade_session_body(lines: list[str], result: dict) -> None: lines.append(f"⚠️ degraded: {', '.join(sorted(degraded))}") -def _spy_daily_returns(dates: list[str]) -> dict[str, float | None]: - """Return SPY close-to-close daily return (%) keyed by date string. - - Fetches enough history to cover one extra bar before the earliest date - so the first row has a prior close to diff against. Returns an empty - dict on any failure so the caller can degrade gracefully. - """ - if not dates: - return {} - try: - from datetime import datetime, timedelta as _td - from src.trading_calendar import et_today as _et_today - import yfinance as _yf - import pandas as _pd - - earliest = min(dates) - # Extra buffer: fetch 20 calendar days before earliest to guarantee - # at least one prior trading-day close even across holiday gaps. - start = (datetime.strptime(earliest, "%Y-%m-%d") - _td(days=20)).date() - end = _et_today() + _td(days=1) # yfinance end is exclusive; +1 to include today - - def _dl(): - return _yf.download("SPY", start=str(start), end=str(end), progress=False) - - from concurrent.futures import ThreadPoolExecutor, TimeoutError as _FT - with ThreadPoolExecutor(max_workers=1) as ex: - df = ex.submit(_dl).result(timeout=15) - - if df is None or df.empty: - return {} - if isinstance(df.columns, _pd.MultiIndex): - df.columns = df.columns.get_level_values(0) - closes = df["Close"].dropna() - date_strs = [str(d.date()) for d in closes.index] - close_vals = list(closes.values) - - # Build close-to-close return for each date. - spy_map: dict[str, float | None] = {} - for i, ds in enumerate(date_strs): - if i == 0: - spy_map[ds] = None # no prior bar - else: - prev = close_vals[i - 1] - spy_map[ds] = (close_vals[i] - prev) / prev * 100 if prev else None - return spy_map - except Exception as exc: - logger.warning("SPY daily return fetch failed: %s", exc) - return {} - - -def _pnl_history_table(lookback: int = 10) -> str | None: - """Query daily_pnl table and return a formatted text table. - - Returns None when the table is empty or DB is unreachable. - """ - try: - import sqlite3 - if not _DB_PATH.exists(): - return None - conn = sqlite3.connect(str(_DB_PATH)) - try: - rows = conn.execute( - "SELECT date, total_value, daily_pnl, daily_return_pct, equity_close " - "FROM daily_pnl ORDER BY date DESC LIMIT ?", - (lookback,), - ).fetchall() - finally: - conn.close() - except Exception as exc: - logger.warning("pnl history lookup failed: %s", exc) - return None - if not rows: - return None - - rows = list(reversed(rows)) # chronological order - dates = [r[0] for r in rows] - spy_returns = _spy_daily_returns(dates) - - cum = 0.0 - peak_nav: float | None = None - - # Anchor NAV on the official 4pm close (equity_close) whenever it's stored; - # legacy rows that predate that column step by the real-time daily_pnl - # instead. The seed is the first row's prior-day close - # (total_value - daily_pnl == last_equity). Net P&L per row is the NAV - # increment, so a row that HAS equity_close shows the SAME 4pm-to-4pm figure - # as the headline — no headline/table contradiction within one message. - first_tv, first_pnl = rows[0][1], rows[0][2] - running_nav = (first_tv - (first_pnl or 0.0)) if first_tv is not None else 0.0 - - table_lines = ["📊 P&L History (last {} days)".format(len(rows))] - table_lines.append( - f"{'Date':<10} {'Net P&L':>11} {'Dly Ret':>7} {'SPY':>7} " - f"{'NAV':>11} {'Cumul P&L':>9} {'Drawdown':>8}" - ) - table_lines.append("─" * 76) - for date, total_value, daily_pnl, daily_ret, equity_close in rows: - prev_nav = running_nav - if equity_close is not None: - running_nav = equity_close # true 4pm close - else: - running_nav += (daily_pnl or 0.0) # legacy: step by real-time P&L - row_pnl = running_nav - prev_nav # NAV increment = the day's P&L - cum += row_pnl - - if peak_nav is None or running_nav > peak_nav: - peak_nav = running_nav - - pnl_str = f"{row_pnl:+,.2f}" - # Return computed from the same NAV increment so it matches Net P&L; for - # legacy rows (row_pnl == daily_pnl) fall back to the stored figure. - if prev_nav and prev_nav > 0: - ret_str = f"{(row_pnl / prev_nav * 100):+.2f}%" - elif daily_ret is not None: - ret_str = f"{daily_ret:+.2f}%" - else: - ret_str = "?" - spy_ret = spy_returns.get(date) - spy_str = f"{spy_ret:+.2f}%" if spy_ret is not None else " n/a" - nav_str = f"${running_nav:,.2f}" - cum_str = f"{cum:+,.2f}" - - if peak_nav and peak_nav > 0: - dd_pct = (running_nav - peak_nav) / peak_nav * 100 - dd_str = f"{dd_pct:.2f}%" if dd_pct < 0 else "0.00%" - else: - dd_str = "?" - - table_lines.append( - f"{date:<10} {pnl_str:>11} {ret_str:>7} {spy_str:>7} " - f"{nav_str:>11} {cum_str:>9} {dd_str:>8}" - ) - return "\n".join(table_lines) - - def _append_evening_body(lines: list[str], result: dict) -> None: # === Escalation banners (first thing read, before Daily P&L) === analysis = result.get("analysis") @@ -539,10 +419,12 @@ def _fmt_pnl(v: float) -> str: lines.append(f" Equity: ${total_value:,.2f}") # Suggested actions — surfaced HIGH in the message (right after the - # headline P&L, before the ~1000-char history table) so the tail-clip - # truncation in send() can never eat them. On exactly the high-risk days - # where these are populated the message is longest, and these are the - # lines most worth reading. Only shown when risk_rating is elevated/high. + # headline P&L) so the tail-clip truncation in send() can never eat + # them. On exactly the high-risk days where these are populated the + # message is longest, and these are the lines most worth reading. + # Only shown when risk_rating is elevated/high. (The P&L history + # text table that used to follow was replaced by the daily CSV + # export — PR #99.) risk_for_actions = _attr_or_key(analysis, "risk_rating") if isinstance(risk_for_actions, str) and risk_for_actions.lower() in ("elevated", "high"): actions = _attr_or_key(analysis, "suggested_actions") or [] @@ -558,12 +440,6 @@ def _fmt_pnl(v: float) -> str: # evening result dict is constructed. _append_position_snapshot(lines, total_value) - # Historical P&L table — last 10 trading days - pnl_table = _pnl_history_table(lookback=10) - if pnl_table: - lines.append("") - lines.append(pnl_table) - analysis = result.get("analysis") risk = _attr_or_key(analysis, "risk_rating") bias = _attr_or_key(analysis, "tomorrow_bias") @@ -752,6 +628,7 @@ def _append_meta_body(lines: list[str], result: dict) -> None: def _status_emoji(status: str) -> str: if status in ( "executed", "analyzed", "reviewed", "preprocessed", "reflected", + "sent", ): return "🟢" if status in ( @@ -841,7 +718,7 @@ def _fmt_elapsed(seconds: float) -> str: return f"{minutes}m {secs}s" -def build_weekly_csv(closes: list[tuple[str, float]]) -> bytes: +def build_daily_csv(closes: list[tuple[str, float]]) -> bytes: """Build a P&L history CSV from portfolio_history closes. Columns: Date, NAV, Daily P&L, Daily Return %, Drawdown %, SPY Close, @@ -878,7 +755,7 @@ def build_weekly_csv(closes: list[tuple[str, float]]) -> bytes: if math.isfinite(val): spy_closes[str(dt_idx.date())] = val except Exception as exc: - logger.warning("build_weekly_csv: SPY fetch failed: %s", exc) + logger.warning("build_daily_csv: SPY fetch failed: %s", exc) buf = io.StringIO() writer = csv.writer(buf) diff --git a/src/pipeline.py b/src/pipeline.py index 0baed9e7..78ff171d 100644 --- a/src/pipeline.py +++ b/src/pipeline.py @@ -5861,6 +5861,41 @@ def run_evening(self) -> dict: "(API lag?) — evening uses the real-time P&L fallback", closes[-1][0], today_str, ) + # Self-heal: when portfolio_history is a day behind at the + # 20:00 ET evening run (the "API lag?" branch above), that + # evening's equity_close landed NULL — but by a LATER evening + # the API has caught up on those dates, which are still inside + # this lookback window. Backfill any still-NULL rows now. + # today_str is excluded because today's row is owned by the + # branches above + save_evening_snapshot below: when today's + # bar is present the first branch already uses it as the + # official close, and when it's absent there is nothing to + # backfill yet. + for d, close_val in closes: + if d == today_str: + continue + # Mirror the `prev_close > 0` guard above: Alpaca + # portfolio_history can emit 0.0 (pre-funding / account + # reset) or non-finite points, and a backfilled value is + # permanent (the fill targets NULL-only rows, so a bad + # write can never be corrected by a later run) — never + # freeze a corrupt equity in. NaN must be caught here + # anyway: sqlite binds it as NULL, which would make + # backfill report success while storing nothing. + if not (math.isfinite(close_val) and close_val > 0): + logger.warning( + "equity_close backfill skipped for %s: suspect " + "equity value %r", d, close_val, + ) + continue + try: + if self.db.backfill_equity_close(d, close_val): + logger.info( + "equity_close backfilled for %s = %.2f (API lag self-heal)", + d, close_val, + ) + except Exception as exc: + logger.warning("equity_close backfill failed for %s: %s", d, exc) except Exception as e: logger.warning("4pm snapshot fetch failed: %s — using real-time P&L", e) @@ -6223,9 +6258,9 @@ def run_quarterly_meta_reflection( "editor_report": editor_report, } - def run_weekly(self) -> dict: + def run_daily(self) -> dict: """Fetch full portfolio history from Alpaca, build a CSV, and send - via Telegram. No LLM calls — pure data export. Runs on Saturdays. + via Telegram. No LLM calls — pure data export. Runs on weekdays. Returns {"status": "sent", "rows": N, "filename": ...} on delivery, {"status": "skipped", ...} when Telegram is disabled (CSV built but no @@ -6234,14 +6269,14 @@ def run_weekly(self) -> dict: the notifier was disabled, so the operator couldn't tell a delivered export from a silently-dropped one. """ - from src.notifier import build_weekly_csv, TelegramNotifier + from src.notifier import build_daily_csv, TelegramNotifier from src.trading_calendar import et_today try: closes = self.broker.get_full_portfolio_history() if not closes: - logger.warning("run_weekly: no portfolio history returned") + logger.warning("run_daily: no portfolio history returned") return {"status": "error", "error": "no data from portfolio_history"} - csv_bytes = build_weekly_csv(closes) + csv_bytes = build_daily_csv(closes) date_str = et_today().strftime("%Y-%m-%d") filename = f"pnl_history_{date_str}.csv" caption = f"📊 P&L History export — {date_str} ({len(closes)} trading days)" @@ -6249,20 +6284,19 @@ def run_weekly(self) -> dict: delivered = notifier.send_document(csv_bytes, filename, caption) base = {"rows": len(closes), "filename": filename} if delivered: - logger.info("run_weekly: sent %d rows as %s", len(closes), filename) + logger.info("run_daily: sent %d rows as %s", len(closes), filename) return {"status": "sent", **base} if not notifier.enabled: # CSV built fine; Telegram simply isn't configured — not a # failure, just nowhere to deliver it. logger.info( - "run_weekly: built %d-row CSV %s but Telegram is disabled", + "run_daily: built %d-row CSV %s but Telegram is disabled", len(closes), filename, ) return {"status": "skipped", **base} # Enabled but the upload failed (network / API / rate limit). - logger.error("run_weekly: Telegram delivery failed for %s", filename) + logger.error("run_daily: Telegram delivery failed for %s", filename) return {"status": "error", "error": "telegram delivery failed", **base} except Exception as exc: - logger.error("run_weekly failed: %s", exc, exc_info=True) + logger.error("run_daily failed: %s", exc, exc_info=True) return {"status": "error", "error": str(exc)} - diff --git a/src/storage/db.py b/src/storage/db.py index f3b38ef6..17e8ebca 100644 --- a/src/storage/db.py +++ b/src/storage/db.py @@ -973,6 +973,25 @@ def insert_daily_pnl(self, date: str, total_value: float, daily_pnl: float, ) self.conn.commit() + def backfill_equity_close(self, date: str, equity_close: float) -> bool: + """Fill in a still-NULL equity_close on an existing daily_pnl row. + + Self-heal for the API-lag gap: portfolio_history doesn't have a + trading day's official close yet at the 20:00 ET evening run (it + lands hours later), so equity_close is stored NULL that night — but + it's available by the FOLLOWING evening's lookback fetch. Only + touches rows that are still NULL; never overwrites an already- + captured close. Returns True if a row was updated. + """ + with self._lock: + cursor = self.conn.execute( + "UPDATE daily_pnl SET equity_close = ? " + "WHERE date = ? AND equity_close IS NULL", + (equity_close, date), + ) + self.conn.commit() + return cursor.rowcount > 0 + def get_daily_pnl(self, limit: int = 30, before_date: str | None = None) -> list[dict]: conditions = [] params: list = [] diff --git a/tests/test_bugfixes.py b/tests/test_bugfixes.py index bbf8c10d..93a56efa 100644 --- a/tests/test_bugfixes.py +++ b/tests/test_bugfixes.py @@ -1359,3 +1359,151 @@ def test_pm_prompt_example_reasoning_chain_parses_with_premortem(): assert m, "PM prompt no longer has a reasoning_chain JSON example" rc = ReasoningChain(**json.loads(m.group(1))) assert rc.premortem_check # the example populates it (non-empty) + + +# =========================================================================== +# PR #99: evening equity_close backfill — pipeline integration. The db-level +# SQL is covered in test_db.py; these pin the run_evening loop itself: the +# today_str exclusion (the only thing keeping an unsettled same-day bar out +# of a permanent NULL-only fill), the corrupt-value guard, and the per-date +# error isolation. +# =========================================================================== + +def _evening_pipeline_with_closes(closes): + """Minimal run_evening harness (mirrors the evening tests above) with a + real get_recent_daily_closes payload so the backfill loop executes.""" + pipeline = TradingPipeline.__new__(TradingPipeline) + pipeline.broker = MagicMock() + pipeline.db = MagicMock() + pipeline.macro = MagicMock() + pipeline.evening_analyst = MagicMock() + pipeline.config = MagicMock() + pipeline.config.llm.evening_analyst_model = "test-model" + pipeline.broker.is_trading_day.return_value = True + pipeline.broker.get_account.return_value = {"portfolio_value": 10_000.0, "last_equity": 10_000.0} + pipeline.broker.get_positions.return_value = [] + pipeline.db.get_trades.return_value = [] + pipeline.macro.get_macro_summary.return_value = {} + pipeline.evening_analyst.analyze.return_value = ( + EveningReport(reasoning_chain=_valid_evening_rc(), daily_summary="Flat", lessons="n/a", tomorrow_outlook="Watch", risk_rating="low"), + AgentResult(raw_text="{}", tokens_used=10, model="test", user_message="test"), + ) + pipeline.broker.get_recent_daily_closes.return_value = closes + return pipeline + + +def test_evening_backfills_prior_dates_but_never_today(): + """[PR #99] The API-lag self-heal must backfill PRIOR dates and must + NEVER write today's bar — today's row is owned by the 4pm-snapshot + branch + save_evening_snapshot, even when today's bar IS present.""" + from src.trading_calendar import session_date_key + today = session_date_key() + closes = [("2026-01-02", 100_000.0), ("2026-01-05", 100_500.0), (today, 100_700.0)] + pipeline = _evening_pipeline_with_closes(closes) + pipeline.run_evening() + called_dates = [c.args[0] for c in pipeline.db.backfill_equity_close.call_args_list] + assert "2026-01-02" in called_dates and "2026-01-05" in called_dates + assert today not in called_dates + + +def test_evening_backfill_runs_in_lag_case_without_today(): + """[PR #99] The lag case (today's bar absent — the branch that motivates + the self-heal): every prior date is backfilled.""" + closes = [("2026-01-02", 100_000.0), ("2026-01-05", 100_500.0)] + pipeline = _evening_pipeline_with_closes(closes) + pipeline.run_evening() + called_dates = [c.args[0] for c in pipeline.db.backfill_equity_close.call_args_list] + assert called_dates == ["2026-01-02", "2026-01-05"] + + +def test_evening_backfill_skips_zero_and_nonfinite_values(): + """[PR #99 review] A backfilled value is permanent (NULL-only fill), so + 0.0 (pre-funding/reset), NaN (sqlite binds it as NULL → fake success + log forever), inf, and negatives must never reach the DB.""" + closes = [ + ("2026-01-02", 0.0), + ("2026-01-05", float("nan")), + ("2026-01-06", float("inf")), + ("2026-01-07", -5.0), + ("2026-01-08", 100_500.0), # the one legit value + ] + pipeline = _evening_pipeline_with_closes(closes) + pipeline.run_evening() + called_dates = [c.args[0] for c in pipeline.db.backfill_equity_close.call_args_list] + assert called_dates == ["2026-01-08"] + + +def test_evening_backfill_one_bad_date_does_not_abort_the_rest(): + """[PR #99] A DB error on one date must not abort backfill of later + dates (per-date try/except) nor crash the evening run.""" + closes = [("2026-01-02", 100_000.0), ("2026-01-05", 100_500.0)] + pipeline = _evening_pipeline_with_closes(closes) + pipeline.db.backfill_equity_close.side_effect = [RuntimeError("locked"), True] + pipeline.run_evening() # must not raise + assert pipeline.db.backfill_equity_close.call_count == 2 + + +# =========================================================================== +# PR #99: main.py crash-visibility net — early crashes (config load, missing +# config file) must produce a FAILED Telegram push and still exit non-zero. +# =========================================================================== + +def test_main_pushes_failed_notification_when_config_load_crashes(monkeypatch): + """[PR #99] A load_config crash (e.g. pydantic ValidationError) must + produce a FAILED push (when Telegram creds are available) AND re-raise + so the process exits non-zero.""" + import main as main_mod + + sent = [] + fake_notifier = MagicMock() + fake_notifier.send = lambda msg: sent.append(msg) or True + monkeypatch.setattr(main_mod, "TelegramNotifier", lambda: fake_notifier) + monkeypatch.setattr( + main_mod, "load_config", + MagicMock(side_effect=RuntimeError("OPENAI_API_KEY is required")), + ) + monkeypatch.setattr("sys.argv", ["main.py", "--mode", "morning"]) + + with pytest.raises(RuntimeError, match="OPENAI_API_KEY"): + main_mod.main() + + assert any("FAILED" in m and "OPENAI_API_KEY" in m for m in sent) + + +def test_main_missing_config_file_push_carries_the_path(monkeypatch, tmp_path): + """[PR #99] The missing-config sys.exit must carry the path into the + push — str(SystemExit(1)) is just '1', useless from a phone.""" + import main as main_mod + + sent = [] + fake_notifier = MagicMock() + fake_notifier.send = lambda msg: sent.append(msg) or True + monkeypatch.setattr(main_mod, "TelegramNotifier", lambda: fake_notifier) + missing = str(tmp_path / "nope.yaml") + monkeypatch.setattr("sys.argv", ["main.py", "--mode", "morning", "--config", missing]) + + with pytest.raises(SystemExit) as ei: + main_mod.main() + + assert ei.value.code != 0 # non-zero exit preserved + assert any("FAILED" in m and "nope.yaml" in m for m in sent) + + +def test_main_live_mode_graceful_scheduler_exit_notifies_clearly(monkeypatch): + """[PR #99] scheduler.start() returning gracefully must push a clear + 'scheduler_exited' status, not '⚪ live returned non-dict result'.""" + import main as main_mod + + sent = [] + fake_notifier = MagicMock() + fake_notifier.send = lambda msg: sent.append(msg) or True + monkeypatch.setattr(main_mod, "TelegramNotifier", lambda: fake_notifier) + monkeypatch.setattr(main_mod, "load_config", lambda _p: MagicMock()) + monkeypatch.setattr(main_mod, "refresh_pricing", lambda: None) + monkeypatch.setattr(main_mod, "TradingScheduler", lambda _c: MagicMock()) + monkeypatch.setattr("sys.argv", ["main.py", "--mode", "live"]) + + main_mod.main() + + assert any("scheduler_exited" in m for m in sent) + assert not any("non-dict" in m for m in sent) diff --git a/tests/test_weekly_report.py b/tests/test_daily_report.py similarity index 70% rename from tests/test_weekly_report.py rename to tests/test_daily_report.py index 95c64ef6..606e1214 100644 --- a/tests/test_weekly_report.py +++ b/tests/test_daily_report.py @@ -1,9 +1,11 @@ -"""Weekly Saturday P&L CSV export (PR #98). +"""Daily P&L CSV export (weekly export shipped in PR #98; renamed +daily-only in PR #99). -Covers: build_weekly_csv settlement math (close-to-close, drawdown, return), +Covers: build_daily_csv settlement math (close-to-close, drawdown, return), SPY column population + graceful degradation, broker.get_full_portfolio_history -ET-date mapping + pre-funding skip, send_document, run_weekly orchestration, -and the format_session_result weekly body. +ET-date mapping + pre-funding skip, send_document, run_daily orchestration, +and the format_session_result daily noise policy (sent silent; error/skipped +notify with the reason). """ import csv import io @@ -17,7 +19,7 @@ def _parse_csv(b: bytes) -> list[dict]: return list(csv.DictReader(io.StringIO(b.decode("utf-8")))) -def test_build_weekly_csv_close_to_close_pnl_and_drawdown(monkeypatch): +def test_build_daily_csv_close_to_close_pnl_and_drawdown(monkeypatch): """Per-row Daily P&L = consecutive close diff; drawdown vs running peak.""" from src import notifier # No network: force the SPY fetch to fail → SPY columns blank. @@ -27,7 +29,7 @@ def test_build_weekly_csv_close_to_close_pnl_and_drawdown(monkeypatch): ("2026-05-27", 100_500.0), # +500 ("2026-05-28", 100_200.0), # -300, drawdown from 100500 ] - out = _parse_csv(notifier.build_weekly_csv(closes)) + out = _parse_csv(notifier.build_daily_csv(closes)) assert [r["Date"] for r in out] == ["2026-05-26", "2026-05-27", "2026-05-28"] assert out[0]["Daily P&L"] == "+0.00" # first row has no predecessor assert out[1]["Daily P&L"] == "+500.00" @@ -42,12 +44,12 @@ def test_build_weekly_csv_close_to_close_pnl_and_drawdown(monkeypatch): assert out[1]["SPY Close"] == "" and out[1]["SPY Return %"] == "" -def test_build_weekly_csv_empty_returns_empty_bytes(): +def test_build_daily_csv_empty_returns_empty_bytes(): from src import notifier - assert notifier.build_weekly_csv([]) == b"" + assert notifier.build_daily_csv([]) == b"" -def test_build_weekly_csv_populates_spy(monkeypatch): +def test_build_daily_csv_populates_spy(monkeypatch): """SPY Close + Return % populated when yfinance returns data.""" import pandas as pd from src import notifier @@ -55,7 +57,7 @@ def test_build_weekly_csv_populates_spy(monkeypatch): df = pd.DataFrame({"Close": [500.0, 505.0, 503.0]}, index=idx) monkeypatch.setattr("yfinance.download", lambda *a, **k: df) closes = [("2026-05-26", 100_000.0), ("2026-05-27", 100_500.0), ("2026-05-28", 100_200.0)] - out = _parse_csv(notifier.build_weekly_csv(closes)) + out = _parse_csv(notifier.build_daily_csv(closes)) assert out[0]["SPY Close"] == "500.00" assert out[1]["SPY Close"] == "505.00" # SPY return row1 = (505-500)/500 = +1.0000% @@ -103,7 +105,7 @@ def test_send_document_posts_and_swallows(monkeypatch): assert n.send_document(b"x", "x.csv") is False # swallowed -def test_run_weekly_sends_and_reports(monkeypatch): +def test_run_daily_sends_and_reports(monkeypatch): from src.pipeline import TradingPipeline pipe = TradingPipeline.__new__(TradingPipeline) pipe.broker = MagicMock() @@ -114,29 +116,72 @@ def test_run_weekly_sends_and_reports(monkeypatch): sent = {} with patch("src.notifier.TelegramNotifier") as TN: TN.return_value.send_document = lambda b, f, c="": sent.update(filename=f, n=len(b)) or True - res = pipe.run_weekly() + res = pipe.run_daily() assert res["status"] == "sent" assert res["rows"] == 2 assert res["filename"].startswith("pnl_history_") and res["filename"].endswith(".csv") -def test_run_weekly_error_on_no_data(): +def test_run_daily_error_on_no_data(): from src.pipeline import TradingPipeline pipe = TradingPipeline.__new__(TradingPipeline) pipe.broker = MagicMock() pipe.broker.get_full_portfolio_history.return_value = [] - res = pipe.run_weekly() + res = pipe.run_daily() assert res["status"] == "error" -def test_format_session_result_weekly_body(): +def test_format_session_result_daily_sent_is_silent(): + """'sent' → None: the CSV document push (with its caption) IS the + confirmation; a second status text every weekday is pure noise.""" from src.notifier import format_session_result - msg = format_session_result("weekly", {"status": "sent", "run_id": "run-w", "rows": 42, "filename": "pnl_history_2026-05-30.csv"}, 3.0) + msg = format_session_result("daily", {"status": "sent", "run_id": "run-w", "rows": 42, "filename": "pnl_history_2026-05-30.csv"}, 3.0) + assert msg is None + + +def test_format_session_result_daily_error_surfaces_reason(): + """'error' notifies AND carries the reason — a bare '🔴 status: error' + is undebuggable from a phone. No filename → no dangling '📊 ? rows →'.""" + from src.notifier import format_session_result + msg = format_session_result( + "daily", + {"status": "error", "run_id": "run-w", "error": "no data from portfolio_history"}, + 3.0, + ) + assert msg is not None + assert "🔴" in msg and "status: error" in msg + assert "no data from portfolio_history" in msg + assert "📊" not in msg # rows/filename line skipped when absent + + +def test_format_session_result_daily_delivery_failure_keeps_rows_line(): + """Delivery failure includes rows+filename (CSV was built) plus reason.""" + from src.notifier import format_session_result + msg = format_session_result( + "daily", + {"status": "error", "error": "telegram delivery failed", + "rows": 42, "filename": "pnl_history_2026-05-30.csv"}, + 3.0, + ) assert msg is not None assert "42 rows" in msg and "pnl_history_2026-05-30.csv" in msg + assert "telegram delivery failed" in msg + + +def test_format_session_result_daily_skipped_notifies(): + """'skipped' (Telegram unconfigured) still renders a message — moot in + production (send() no-ops without creds) but honest for manual runs.""" + from src.notifier import format_session_result + msg = format_session_result( + "daily", + {"status": "skipped", "rows": 42, "filename": "pnl_history_2026-05-30.csv"}, + 3.0, + ) + assert msg is not None + assert "status: skipped" in msg and "42 rows" in msg -def test_build_weekly_csv_filters_nan_spy(monkeypatch): +def test_build_daily_csv_filters_nan_spy(monkeypatch): """[Bug 1] A NaN SPY close (data gap/halt) must NOT render as '+nan' and must NOT poison prev_spy for later rows — the NaN day is dropped and the next valid day diffs against the last *valid* prior close.""" @@ -146,7 +191,7 @@ def test_build_weekly_csv_filters_nan_spy(monkeypatch): df = pd.DataFrame({"Close": [500.0, float("nan"), 503.0]}, index=idx) monkeypatch.setattr("yfinance.download", lambda *a, **k: df) closes = [("2026-05-26", 100_000.0), ("2026-05-27", 100_500.0), ("2026-05-28", 100_200.0)] - raw = notifier.build_weekly_csv(closes) + raw = notifier.build_daily_csv(closes) assert b"nan" not in raw.lower() # no '+nan' leak anywhere out = _parse_csv(raw) assert out[1]["SPY Close"] == "" and out[1]["SPY Return %"] == "" # NaN day blank @@ -155,7 +200,7 @@ def test_build_weekly_csv_filters_nan_spy(monkeypatch): assert float(out[2]["SPY Return %"]) == pytest.approx(0.6, abs=1e-3) -def test_run_weekly_skipped_when_telegram_disabled(monkeypatch): +def test_run_daily_skipped_when_telegram_disabled(monkeypatch): """[Bug 2] Telegram disabled (no creds) → CSV built but undelivered → honest 'skipped', not 'sent'.""" from src.pipeline import TradingPipeline @@ -168,11 +213,11 @@ def test_run_weekly_skipped_when_telegram_disabled(monkeypatch): with patch("src.notifier.TelegramNotifier") as TN: TN.return_value.enabled = False TN.return_value.send_document.return_value = False - res = pipe.run_weekly() + res = pipe.run_daily() assert res["status"] == "skipped" and res["rows"] == 2 -def test_run_weekly_error_when_delivery_fails(monkeypatch): +def test_run_daily_error_when_delivery_fails(monkeypatch): """[Bug 2] Telegram enabled but the upload failed → 'error', not 'sent'.""" from src.pipeline import TradingPipeline pipe = TradingPipeline.__new__(TradingPipeline) @@ -182,5 +227,5 @@ def test_run_weekly_error_when_delivery_fails(monkeypatch): with patch("src.notifier.TelegramNotifier") as TN: TN.return_value.enabled = True TN.return_value.send_document.return_value = False - res = pipe.run_weekly() + res = pipe.run_daily() assert res["status"] == "error" diff --git a/tests/test_db.py b/tests/test_db.py index a524beda..1dfb851f 100644 --- a/tests/test_db.py +++ b/tests/test_db.py @@ -700,3 +700,27 @@ def test_daily_pnl_reinsert_preserves_equity_close_when_none(db): assert row["daily_pnl"] == -550.0 # other columns still updated db.insert_daily_pnl("2026-05-28", 100_450.0, -550.0, -0.55, equity_close=100_600.0) assert db.get_daily_pnl(limit=1)[0]["equity_close"] == 100_600.0 # real value overwrites + + +def test_backfill_equity_close_fills_null(db): + """[API-lag self-heal] A NULL equity_close (yesterday's portfolio_history + fetch hit the lag gap) gets filled once the API has caught up.""" + db.insert_daily_pnl("2026-05-29", 100_000.0, -50.0, -0.05) # equity_close NULL + assert db.backfill_equity_close("2026-05-29", 100_053.18) is True + row = [r for r in db.get_daily_pnl(limit=5) if r["date"] == "2026-05-29"][0] + assert row["equity_close"] == 100_053.18 + + +def test_backfill_equity_close_never_overwrites_existing(db): + """[API-lag self-heal] Must not clobber an already-captured 4pm close — + backfill is a gap-filler, not a corrector.""" + db.insert_daily_pnl("2026-05-28", 100_400.0, -600.0, -0.59, equity_close=100_500.0) + assert db.backfill_equity_close("2026-05-28", 999_999.0) is False + row = [r for r in db.get_daily_pnl(limit=5) if r["date"] == "2026-05-28"][0] + assert row["equity_close"] == 100_500.0 # untouched + + +def test_backfill_equity_close_no_row_for_date(db): + """[API-lag self-heal] Backfilling a date with no daily_pnl row is a + no-op, not a crash (e.g. lookback window predates the account's history).""" + assert db.backfill_equity_close("2020-01-01", 100_000.0) is False diff --git a/tests/test_notifier.py b/tests/test_notifier.py index 3080a80e..21ff40a2 100644 --- a/tests/test_notifier.py +++ b/tests/test_notifier.py @@ -353,8 +353,8 @@ def test_format_evening_return_pct_na_when_prior_equity_nonpositive(): } msg = format_session_result("evening", result, 10.0) assert msg is not None - # Assert on the Daily P&L line specifically (the P&L history table, - # if a DB is present, legitimately contains "0.00%" elsewhere). + # Assert on the Daily P&L line specifically rather than bare "n/a" + # (other message sections could legitimately contain that text). assert "💰 Daily P&L: -$500.00 (n/a)" in msg, msg @@ -937,9 +937,11 @@ def test_format_evening_missing_morning_session_is_red(): assert "midday" in msg # soft warning for the non-morning miss -def test_format_evening_suggested_actions_precede_history_table(): - """Suggested actions must appear ABOVE the long P&L history table so the - tail-clip truncation can't eat them on high-risk days.""" +def test_format_evening_suggested_actions_render_high_in_message(): + """Suggested actions must appear high in the message (right after the + headline P&L) so the tail-clip truncation can't eat them on high-risk + days. (The P&L history table they used to precede was replaced by the + daily CSV export — PR #99.)""" result = { "status": "analyzed", "run_id": "r", "daily_pnl": -100.0, "total_value": 100_000.0, @@ -952,7 +954,7 @@ def test_format_evening_suggested_actions_precede_history_table(): msg = format_session_result("evening", result, 10.0) assert "⚡ Suggested actions:" in msg assert "Reduce NVDA exposure" in msg - # Appears before the Tomorrow block (which sits after the history table). + # Appears before the Tomorrow block (the tail of the message). assert msg.index("⚡ Suggested actions:") < msg.index("🔮 Tomorrow") @@ -1052,36 +1054,3 @@ def test_deterministic_escalation_ignores_realtime_loss_when_4pm_small(): } msg = format_session_result("evening", result, 10.0) assert "DETERMINISTIC ALERT" not in msg - - -def test_pnl_history_table_uses_equity_close_for_4pm_consistency(tmp_path, monkeypatch): - """[C] The table anchors NAV + per-row P&L on equity_close, so today's row - shows the 4pm-to-4pm P&L (matching the headline) — NOT the real-time - daily_pnl. Regression against the headline/table contradiction.""" - import sqlite3 - from src.notifier import _pnl_history_table - db_dir = tmp_path / "data"; db_dir.mkdir() - dbp = db_dir / "quant_agent.db" - monkeypatch.setattr("src.notifier._DB_PATH", dbp) - monkeypatch.setattr("src.notifier._spy_daily_returns", lambda dates: {}) # no network - conn = sqlite3.connect(str(dbp)) - conn.execute( - "CREATE TABLE daily_pnl (date TEXT PRIMARY KEY, total_value REAL, " - "daily_pnl REAL, daily_return_pct REAL, equity_close REAL)" - ) - conn.executemany( - "INSERT INTO daily_pnl VALUES (?,?,?,?,?)", - [ - # seed prior close = total_value - daily_pnl = 100000 - ("2026-05-27", 100_400.0, 400.0, 0.40, 100_300.0), # 4pm: +300 - ("2026-05-28", 101_200.0, 1200.0, 1.20, 100_500.0), # 4pm: 100500-100300 = +200 - ], - ) - conn.commit(); conn.close() - - table = _pnl_history_table(lookback=10) - assert table is not None - today_line = [ln for ln in table.splitlines() if ln.startswith("2026-05-28")][0] - assert "+200.00" in today_line # 4pm-to-4pm P&L - assert "+1,200" not in today_line # NOT the real-time figure - assert "$100,500.00" in today_line # NAV = today's 4pm close