diff --git a/.kiro/specs/ai-csv-import/design.md b/.kiro/specs/ai-csv-import/design.md new file mode 100644 index 0000000..8fc350f --- /dev/null +++ b/.kiro/specs/ai-csv-import/design.md @@ -0,0 +1,333 @@ +# Design — AI-Assisted CSV Transaction Import (Human-in-the-Loop) + +## Introduction + +Implements **GitHub Issue #9** (`toolpathguy/FinanceApp`): let the user upload a bank +CSV they exported themselves and have the AI map an **arbitrary CSV layout** to +normalized transactions. The AI **proposes** rows; the user reviews/edits them in +a staging table (account, envelope, payee) and **approves**; only then are they +committed via the existing direct journal writer. Nothing is written without +explicit per-row approval. + +This builds directly on the merged **#8 AI budgeting chat** (PR #16): it reuses +`server/utils/anthropic.ts` (shared client + key resolution), `server/utils/aiConfig.ts` +(Settings-page key), `types/ai.ts` patterns, and the **propose → approve → commit** +HITL spine. It introduces no new accounting logic — commits flow through the same +simplified-transaction → `journalWriter` path that `POST /api/transactions` already +uses (extracted to a shared util so neither route duplicates the envelope math). + +### Non-negotiable safety invariant + +> **The parse route never writes to the journal.** `POST /api/import/parse` only +> reads (account list) and calls Anthropic; it returns *proposals*. A write happens +> only in `POST /api/import/commit`, and only for rows the user explicitly approved. +> Covered by a dedicated test (R from requirements.md): a parse call records zero +> `appendTransaction` calls. + +--- + +## Why structured outputs, not a tool loop + +#8's chat is an **agentic tool loop** (read tools + proposed-action tools, paused for +approval). CSV parse is the opposite shape: a **single, one-shot extraction** with no +agency. The right primitive is **structured outputs**, not tools. + +- **`claude-opus-4-8` supports structured outputs** via + `output_config: { format: { type: 'json_schema', schema } }`. The SDK's + `client.messages.parse()` validates the response against the schema and returns a + typed `parsed_output` (null on parse failure) — no manual JSON parsing, no + brittle string handling. +- **Compatible with adaptive thinking** — structured outputs works alongside + `thinking: { type: 'adaptive' }`, so we keep the shared `REQUEST_DEFAULTS` from #8 + unchanged. No forced `tool_choice` is needed (forcing a tool would be the wrong + tool for the job and interacts awkwardly with thinking). +- **Caveats the design accounts for:** + - JSON Schema is constrained: every object needs `additionalProperties: false`, + and `minLength`/`maxLength`/numeric bounds/recursion are unsupported (the SDK + strips unsupported keywords and validates client-side). Our schema stays within + the supported subset; per-field bounds are enforced in our own validation pass. + - **Refusal:** on `stop_reason: 'refusal'` the output may not match the schema — + the route returns a clear error and writes nothing. + - **Incompatible with prefill / citations** (neither is used here). + +--- + +## Architecture & data flow + +``` +pages/import.vue ─────────────────────────┐ + (UFileUpload + egress notice + │ approve → composables/useImport.commit() + ImportReviewTable: edit/approve rows) │ POST /api/import/commit + │ │ │ + ▼ │ ▼ +composables/useImport.ts ──────────────────┘ server/api/import/commit.post.ts + parse(): POST /api/import/parse ├─ dedup vs journal (importDedup.ts) + │ └─ appendSimplifiedTransaction() ← shared write util + ▼ │ (NEW, extracted from transactions.post.ts) +server/api/import/parse.post.ts ← reads only; NEVER writes + ├─ server/utils/anthropic.ts ← shared SDK client + key (reused from #8) + ├─ server/utils/importParse.ts ← system prompt + JSON schema + normalization + validation + └─ server/utils/importContext.ts ← valid real accounts + envelope keys (delegates to filterAccounts) +``` + +The **system prompt + JSON schema** are the stable, cacheable prefix +(`cache_control: { type: 'ephemeral' }`). The **CSV text and the account/envelope +context are volatile** and go in the user message *after* the cached prefix, so the +cache stays warm across imports. + +--- + +## The parse path (`POST /api/import/parse`) + +1. Client uploads CSV text (read client-side from the chosen file). +2. Route resolves the Anthropic client (`getAnthropic()`); 503 + empty state if no key + (same handling as #8's chat route). +3. Route fetches **import context** — the list of valid real accounts + (`assets:` / `liabilities:`) and valid envelope keys (expense categories), via + `getImportContext()` (delegates to `hledgerExecText(['accounts'])` + the existing + `filterRealAccounts` / `filterCategoryAccounts` pure utils). This grounds the + model's `suggestedAccount` / `suggestedEnvelope` in **real** targets. +4. Route calls `client.messages.parse({ ...REQUEST_DEFAULTS, system, messages, output_config })` + with the `IMPORT_SCHEMA` (one-shot, non-streaming). +5. `parsed_output` is normalized + validated by `normalizeProposals()` (pure): + - date → `YYYY-MM-DD` (handle `MM/DD/YYYY`, `DD/MM/YYYY` ambiguity via a hint the + model emits, `D Mon YYYY`, ISO, etc. — the model returns ISO; we re-validate); + - amount → positive magnitude in dollars, sign dropped, direction carried + separately (`inflow` | `outflow`) — covers single signed-amount columns **and** + separate debit/credit columns (the model maps either to one direction + magnitude); + - `suggestedAccount` / `suggestedEnvelope` kept only if they match a real target, + else blank (blank envelope is legal — see Uncategorized below); + - a stable `dedupHash` (see Dedup) is computed per row. +6. Returns `{ proposals: ImportProposal[], context: { accounts, envelopes }, droppedRows }`. + `context` feeds the review table's account/envelope dropdowns; `droppedRows` + reports any rows the model couldn't parse (never silently dropped — R requirement). + +**No iteration loop, no pending-tool protocol.** One request, one structured response. + +### Bounding output size + +Output scales with row count. v1 caps the CSV at **`MAX_IMPORT_ROWS = 200`** rows per +parse and sets `max_tokens` accordingly (non-streaming, well under the SDK HTTP-timeout +threshold). Larger files return a 413-style message asking the user to split the file. +Chunked/streamed parsing of very large statements is a deferred enhancement (see +Alternatives); the wire contract doesn't change when it's added. + +--- + +## The commit path (`POST /api/import/commit`) + +1. Client sends only the **approved, possibly-edited** rows (`CommitRow[]`): final + `date, payee, amount, direction, account, envelope`. +2. Route re-validates each row server-side (date format, amount > 0, account is a real + account, envelope — if present — is a real expense category). Invalid rows are + rejected with per-row errors; the rest still commit (partial success is reported). +3. **Dedup** (`importDedup.ts`): build the set of existing-journal dedup hashes once + (from `getTransactionList()` / `hledger print`), and skip any approved row whose + hash already exists — **reported as skipped, not silently dropped**. Because every + committed row becomes a real journal entry, re-importing the same statement later is + caught by this same journal check — **no separate "imported ledger" file is needed** + (keeps the server stateless, consistent with the rest of the app). + - Dedup is a **safety net, not a hard gate at parse time**: legitimately-identical + rows (two $5 coffees, same day/payee) are surfaced in the review table as + "possible duplicate" so the user decides; the commit-time journal check prevents + accidental *re-import* of an already-committed batch. +4. Each surviving row → `SimplifiedTransactionInput` → **`appendSimplifiedTransaction()`** + (the shared write util), which runs `toTransactionInput` + envelope postings + + `appendTransaction` (validate → format → `fs.appendFile`, integer-cents balancing). +5. Returns `{ committed: number, skippedDuplicates: CommitRow[], failed: {row, error}[] }`. + +### Direction → transaction type mapping + +| CSV direction | Maps to | Postings | +|---|---|---| +| `outflow` (money leaves the account) | `expense` | debit chosen envelope's `expenses:` category, credit the budget sub-account (existing envelope-aware logic) | +| `inflow` (money enters the account) | `income` | debit the real account, credit `income:` — lands in Ready-to-Assign per YNAB Rule 1 | + +### Uncategorized handling (Issue #9 requirement) + +- **Inflow with no envelope** → income → naturally lands in **Ready to Assign** (RTA = + net worth − envelope balances; an income inflow with no assignment raises RTA). This + is the intended resting place; no special-casing needed. +- **Outflow with no envelope** → the review table **requires** an envelope before the + row can be approved (an outflow must hit a category to keep the budget balanced). + The model suggests one; the user can change it. Rows left uncategorized are simply + not approvable — they never reach commit. + +--- + +## Refactor to enable clean reuse (no behavior change) + +`POST /api/transactions` currently inlines `applyEnvelopePostings` + `toTransactionInput` ++ `appendTransaction`. Extract that composition into: + +- **`server/utils/transactionWriter.ts`** — `appendSimplifiedTransaction(input: SimplifiedTransactionInput): Promise`, + moving `applyEnvelopePostings` (which already delegates to `resolveBudgetBase`) out of + the route. `transactions.post.ts` becomes a thin validate-and-call wrapper; the import + commit route calls the same function. This mirrors how #8 extracted `getBudgetReport` + from `budget.get.ts`, and keeps the **accounting logic in one place** (separation-of-concerns: + no envelope math duplicated across two routes). Existing `transactions.post.ts` tests + still pass against the delegate (refactor parity). + +--- + +## Key interfaces / types (`types/import.ts`) + +```ts +export type ImportDirection = 'inflow' | 'outflow' + +/** A normalized transaction the AI proposed from one CSV row. */ +export interface ImportProposal { + id: string // stable per-row id (index-based) for the review table + date: string // YYYY-MM-DD (validated) + payee: string + amount: number // positive magnitude + direction: ImportDirection + suggestedAccount: string // real account path, or '' if unknown + suggestedEnvelope: string // expense category key, or '' (uncategorized) + dedupHash: string // sha256(date|cents|payee) — see Dedup + possibleDuplicate: boolean // hash already present in the journal at parse time + sourceRow: string // raw CSV line, for display + user trust +} + +export interface ImportParseResponse { + proposals: ImportProposal[] + context: { accounts: string[]; envelopes: string[] } // dropdown options + droppedRows: { sourceRow: string; reason: string }[] // never silently dropped +} + +/** A row the user approved (and possibly edited) in the review table. */ +export interface CommitRow { + date: string + payee: string + amount: number + direction: ImportDirection + account: string // chosen real account + envelope: string // chosen expense category ('' only allowed for inflow) + dedupHash: string +} + +export interface ImportCommitRequest { rows: CommitRow[] } +export interface ImportCommitResponse { + committed: number + skippedDuplicates: CommitRow[] + failed: { row: CommitRow; error: string }[] +} +``` + +The JSON schema passed to Anthropic (`IMPORT_SCHEMA` in `importParse.ts`) is the +proposal shape **minus** the purely-server-computed fields (`id`, `dedupHash`, +`possibleDuplicate`) — the model returns `{ date, payee, amount, direction, +suggestedAccount, suggestedEnvelope, sourceRow }[]`, and the server adds the rest. +**Refinement during implementation:** the model echoes the verbatim original CSV line +as `sourceRow` (rather than the server reconstructing it), which gives reliable per-row +provenance for display and lets the normalizer attribute every dropped row to its source. +Every object carries `additionalProperties: false` (structured-outputs requirement). + +--- + +## Dedup hash + +`dedupHash = sha256(`${date}|${Math.round(amount*100)}|${payee.trim().toLowerCase()}`)`. +Date + integer cents + normalized payee. Computed identically at parse time (to flag +`possibleDuplicate` against the existing journal) and at commit time (to skip rows +already in the journal). Collisions between genuinely-distinct same-day/same-amount/same-payee +transactions are treated as *possible duplicates to confirm*, never as silent drops. + +--- + +## Files added / changed + +| File | Change | +|---|---| +| `server/api/import/parse.post.ts` (new) | reads context, calls Anthropic structured output, returns proposals; 503 when key missing; never writes | +| `server/api/import/commit.post.ts` (new) | re-validates, dedups, commits approved rows via the shared write util | +| `server/utils/importParse.ts` (new) | `IMPORT_SYSTEM_PROMPT`, `IMPORT_SCHEMA`, `normalizeProposals()`, per-field validation, `MAX_IMPORT_ROWS` | +| `server/utils/importContext.ts` (new) | `getImportContext()` → `{ accounts, envelopes }` (delegates to `filterAccounts` utils) | +| `server/utils/importDedup.ts` (new) | `computeDedupHash()`, `loadJournalHashes()` (reads existing journal via `getTransactionList`) | +| `server/utils/transactionWriter.ts` (new) | `appendSimplifiedTransaction()` extracted from `transactions.post.ts` | +| `server/api/transactions.post.ts` (edit) | delegate to `appendSimplifiedTransaction` (thin wrapper) | +| `composables/useImport.ts` (new) | reactive state: upload, parse, edit, approve/reject, commit, result summary | +| `components/ImportReviewTable.vue` (new) | UTable staging grid: per-row account/envelope/payee edit, approve toggles, duplicate badge | +| `pages/import.vue` (new) | UFileUpload + persistent egress notice + no-key empty state + review table + commit | +| `layouts/default.vue` (edit) | sidebar nav entry → Import | +| `types/import.ts` (new) | wire + proposal types above | +| `AI-MAP.md` | new route/util/composable/page rows + CSV-egress quirk (main agent, after impl) | + +No new dependency (`@anthropic-ai/sdk` already present from #8). No config changes. + +--- + +## Data egress (must document prominently — Issue #9 risk note) + +CSV rows — transaction descriptions and amounts — are sent to the Anthropic API to +perform the mapping. This is the **one external data flow** in the app (same as #8's +chat). No bank credentials, no aggregator, no stored secrets beyond `ANTHROPIC_API_KEY`. +`pages/import.vue` shows a **persistent, visible notice** before upload ("The contents +of this CSV are sent to Anthropic to extract transactions"), captured as a requirement, +not just a code comment. + +--- + +## Alternatives considered + +- **Forced `tool_choice` to emit transactions** — rejected. Structured outputs + (`output_config.format`) is the purpose-built primitive for one-shot JSON extraction, + returns a validated `parsed_output`, and stays cleanly compatible with adaptive + thinking. A forced single-tool call is the wrong shape and adds tool-protocol + overhead for no benefit. +- **Reuse #8's agentic tool loop** — rejected. Parse has no agency: no reads to chain, + no actions to pause on. A loop would add latency and complexity for a single + extraction. +- **A fixed per-bank CSV parser** — rejected (this is the whole point of #9): bank + layouts vary wildly (column names, date formats, single signed column vs separate + debit/credit, sign conventions). The AI mapping is the feature's value-add. +- **A separate "imported transactions" ledger for dedup** — rejected. Committed rows + are real journal entries, so the journal *is* the dedup source; checking against it + catches re-imports without adding persistent state. +- **Hard-block duplicates at parse time** — rejected. Identical same-day transactions + are legitimate; silently dropping them loses real data. Dedup flags possibles for the + user (HITL) and only the commit-time journal check prevents accidental batch + re-import. +- **Streaming the parse response** — deferred. v1 caps rows and is non-streaming + (replies stay under the timeout threshold). Streaming + chunked parsing of large + statements is a later enhancement; `ImportParseResponse` is unchanged by it. +- **New API-key UI** — avoided. Reuse #8's `aiConfig` + Settings card; the no-key + empty state links there. + +--- + +## Testing strategy (detail in tasks.md) + +- **Load-bearing safety test:** a parse call with a mocked Anthropic client records + **zero** `appendTransaction` / journal-writer calls; the response carries proposals. +- **Parse normalization (pure, `normalizeProposals`):** signed single-column → magnitude + + direction; separate debit/credit columns → one direction; multiple date formats → + `YYYY-MM-DD`; bogus `suggestedEnvelope` not in context → blanked; unparseable row → + surfaced in `droppedRows`, never dropped silently. fast-check property: output amounts + are always ≥ 0 and direction is preserved. +- **Dedup (`importDedup`):** a row whose hash matches an existing journal entry is + flagged `possibleDuplicate` at parse and **skipped** at commit; two identical rows in + one batch are both surfaced (not auto-merged). +- **Commit:** approved rows commit via `appendSimplifiedTransaction`; outflow with empty + envelope is rejected; inflow with empty envelope commits (→ RTA); partial success + reports `committed` / `skippedDuplicates` / `failed`. +- **Refactor parity:** existing `transactions.post.ts` tests pass against the + `appendSimplifiedTransaction` delegate (no behavior change). +- **Missing key:** parse/commit with no `ANTHROPIC_API_KEY` → 503, clear message. +- **Refusal:** mocked `stop_reason: 'refusal'` → parse returns an error, no proposals, + no writes. +- Mock Nitro globals with `vi.stubGlobal()` per project convention; mock the Anthropic + SDK (`any` allowed in tests). Full `npx vitest run` + `npx nuxi typecheck` clean at the + end. + +--- + +## Out of scope + +- Streaming / chunked parsing of very large statements (deferred; v1 caps at + `MAX_IMPORT_ROWS`). +- Direct bank/aggregator connections (the #9 pivot away from Stripe/Plaid — CSV only). +- Auto-creating new envelopes/categories during import (user picks from existing). +- Multi-currency CSVs (single `$` commodity, consistent with the rest of the app). +- Persisting import history / undo (deletes use the existing register delete-by-index). +- Multi-user auth / per-user keys (single-user local app). diff --git a/.kiro/specs/ai-csv-import/requirements.md b/.kiro/specs/ai-csv-import/requirements.md new file mode 100644 index 0000000..f76167d --- /dev/null +++ b/.kiro/specs/ai-csv-import/requirements.md @@ -0,0 +1,180 @@ +# Requirements — AI-Assisted CSV Transaction Import (Human-in-the-Loop) + +Traceable to **GitHub Issue #9**. Builds on the merged #8 AI budgeting chat (PR #16). +Acceptance criteria use EARS form ("WHEN … THE SYSTEM SHALL …"). Each requirement +notes the design section it derives from. + +--- + +## R1 — Upload a CSV and get AI-extracted proposals + +**User story:** As a budgeter, I want to upload a bank CSV and have the app turn its +arbitrary layout into a list of proposed transactions, so I don't hand-enter them. + +- R1.1 — WHEN the user selects a `.csv` file on the import page, THE SYSTEM SHALL read + its text client-side and POST it to `POST /api/import/parse`. +- R1.2 — WHEN `parse` receives CSV text, THE SYSTEM SHALL call the Anthropic API with a + JSON-schema structured-output request and return normalized proposals + (`{ date, payee, amount, direction, suggestedAccount, suggestedEnvelope }` per row, + enriched server-side with `id`, `dedupHash`, `possibleDuplicate`, `sourceRow`). +- R1.3 — THE SYSTEM SHALL ground the model's `suggestedAccount` / `suggestedEnvelope` + in the real account/envelope list fetched from hledger (`getImportContext()`), and + SHALL blank any suggestion that does not match a real target. +- R1.4 — WHEN the model returns rows it could not parse, THE SYSTEM SHALL surface them + in `droppedRows` with a reason; THE SYSTEM SHALL NOT silently discard any input row. +- R1.5 — WHEN the CSV exceeds `MAX_IMPORT_ROWS` (200), THE SYSTEM SHALL reject the + request with a clear message asking the user to split the file, and SHALL NOT + truncate the input. + +*(Design: parse path, "Bounding output size".)* + +--- + +## R2 — Safety: parse never writes + +**User story:** As a user, I want certainty that uploading/previewing never changes my +ledger, so I can review safely. + +- R2.1 — WHEN `POST /api/import/parse` runs for any input, THE SYSTEM SHALL NOT call + `appendTransaction` or otherwise modify the journal file. +- R2.2 — THE SYSTEM SHALL perform all journal writes exclusively in + `POST /api/import/commit`, and only for rows the user approved. + +*(Design: Non-negotiable safety invariant. Load-bearing test in tasks.md.)* + +--- + +## R3 — Review and edit before approving + +**User story:** As a budgeter, I want to review and correct each proposed row before +anything is saved. + +- R3.1 — WHEN proposals are returned, THE SYSTEM SHALL display them in a staging table + with the date, payee, amount, direction, and editable account + envelope dropdowns + populated from the parse `context`. +- R3.2 — THE SYSTEM SHALL let the user edit a row's account, envelope, and payee, and + approve or reject each row individually. +- R3.3 — WHEN a row's direction is `outflow` AND its envelope is empty, THE SYSTEM + SHALL prevent that row from being approved (an outflow must hit a category). +- R3.4 — WHEN a row's direction is `inflow` AND its envelope is empty, THE SYSTEM SHALL + allow approval (it commits as income → Ready to Assign). +- R3.5 — THE SYSTEM SHALL display the original CSV line (`sourceRow`) for each proposal + so the user can verify the mapping. + +*(Design: review/commit paths, Uncategorized handling.)* + +--- + +## R4 — Commit only approved rows + +**User story:** As a user, I want only the rows I approved to be written, as balanced +journal entries. + +- R4.1 — WHEN the user commits, THE SYSTEM SHALL send only approved (possibly-edited) + rows to `POST /api/import/commit`. +- R4.2 — WHEN `commit` receives a row, THE SYSTEM SHALL re-validate it server-side + (valid `YYYY-MM-DD` date; amount > 0; account is a real account; envelope, if + present, is a real expense category) and SHALL reject invalid rows with a per-row + error while still committing the valid ones (partial success). +- R4.3 — WHEN a valid row is committed, THE SYSTEM SHALL write it via the shared + `appendSimplifiedTransaction()` util (`outflow` → `expense`, `inflow` → `income`), + producing a balanced, integer-cents journal entry. +- R4.4 — THE SYSTEM SHALL return a summary `{ committed, skippedDuplicates, failed }` + and the UI SHALL display it. + +*(Design: commit path, Direction → type mapping, refactor.)* + +--- + +## R5 — Duplicate detection (flag, don't silently drop) + +**User story:** As a user re-importing a statement, I don't want duplicate transactions +written — but I also don't want legitimate identical transactions silently lost. + +- R5.1 — THE SYSTEM SHALL compute `dedupHash = sha256(date|cents|payeeLowercased)` for + every proposal and commit row. +- R5.2 — WHEN a proposal's hash matches an existing journal entry at parse time, THE + SYSTEM SHALL mark it `possibleDuplicate: true` and the table SHALL badge it. +- R5.3 — WHEN an approved row's hash matches an existing journal entry at commit time, + THE SYSTEM SHALL skip writing it and report it in `skippedDuplicates`. +- R5.4 — WHEN two approved rows in the same batch share a hash, THE SYSTEM SHALL treat + them as distinct (surface both; do not auto-merge) — only journal-existing matches are + skipped. + +*(Design: Dedup hash, commit dedup.)* + +--- + +## R6 — Messy real-world CSV handling + +**User story:** As a user, my bank's CSV has odd formats; the import should cope. + +- R6.1 — THE SYSTEM SHALL normalize varied date formats (ISO, `MM/DD/YYYY`, + `DD/MM/YYYY`, `D Mon YYYY`) to `YYYY-MM-DD`. +- R6.2 — THE SYSTEM SHALL produce a positive `amount` magnitude plus a separate + `direction`, correctly mapping both a single signed-amount column and separate + debit/credit columns. +- R6.3 — THE SYSTEM SHALL leave `suggestedEnvelope` blank when no confident category + match exists, rather than inventing a category. + +*(Design: parse normalization, Uncategorized.)* + +--- + +## R7 — Reuse the #8 AI plumbing and key config + +**User story:** As a user who already configured my API key for the chat, I want import +to use the same key with no extra setup. + +- R7.1 — THE SYSTEM SHALL resolve the Anthropic key via the existing + `server/utils/anthropic.ts` (`env → config/ai-config.json → none`), with no new key UI. +- R7.2 — WHEN no API key is configured, THE SYSTEM SHALL return 503 from **parse** + and the import page SHALL show a "configure your API key" empty state linking to + Settings. (Commit performs a purely local journal write and needs no key, so it does + not gate on one — implementation correction to the original "parse/commit".) +- R7.3 — WHEN the Anthropic call fails (network/billing/rate-limit/refusal), THE SYSTEM + SHALL surface an actionable message and SHALL NOT write a partial/garbage entry. + +*(Design: reuse, Refusal caveat. Mirrors #8 error handling.)* + +--- + +## R8 — Data-egress transparency + +**User story:** As a privacy-conscious user, I want to know my transaction text leaves +my machine. + +- R8.1 — THE import page SHALL display a persistent, visible notice that the CSV + contents are sent to the Anthropic API, shown before/at upload. +- R8.2 — THE SYSTEM SHALL NOT log CSV contents or the API key. + +*(Design: Data egress section — Issue #9 risk note.)* + +--- + +## Non-functional requirements + +- **NF1 — Separation of concerns.** Pages fetch via the `useImport` composable; the + composable calls `/api/import/*`; routes delegate to `server/utils`; only + `server/utils` (journalWriter / transactionWriter / hledger) touch the journal or + spawn hledger. Account-name shaping / amount formatting use existing `utils/` helpers. +- **NF2 — No new accounting logic.** Commits reuse the existing simplified-transaction → + `journalWriter` path via the extracted `appendSimplifiedTransaction`; no balance math + is reimplemented. +- **NF3 — Types & casts.** No `any` / unnecessary `as` outside validated trust + boundaries (parsing the Anthropic response, reading request bodies). `any` allowed in + tests for SDK mocks. +- **NF4 — Tooling untouched.** No changes to `vitest.config`, `tsconfig`, `nuxt.config`, + or `package.json` scripts/deps (`@anthropic-ai/sdk` already present). +- **NF5 — Windows/CRLF.** hledger output parsed with `split(/\r?\n/)` + trim; account + args passed after `--` (existing utils already do this). +- **NF6 — Verification.** `npx vitest run` and `npx nuxi typecheck` both clean before + the feature is considered done. + +--- + +## Out of scope (restated) + +Streaming/chunked parsing of large statements; direct bank/aggregator connections; +auto-creating envelopes during import; multi-currency CSVs; persisted import history / +undo; multi-user auth. diff --git a/.kiro/specs/ai-csv-import/tasks.md b/.kiro/specs/ai-csv-import/tasks.md new file mode 100644 index 0000000..e0b792e --- /dev/null +++ b/.kiro/specs/ai-csv-import/tasks.md @@ -0,0 +1,136 @@ +# Tasks — AI-Assisted CSV Transaction Import (Human-in-the-Loop) + +Implements [design.md](./design.md) and satisfies [requirements.md](./requirements.md) +(GitHub Issue #9). Ordered by dependency. Each task is small, independently verifiable, +names the files it touches and the tests to add, and cites the requirement(s) it covers. + +Tests live beside source. Run a single file with +`npx vitest run ` while iterating; full suite + typecheck at the end. + +--- + +- [x] **1. Shared types** — `types/import.ts` + - Add `ImportDirection`, `ImportProposal`, `ImportParseResponse`, `CommitRow`, + `ImportCommitRequest`, `ImportCommitResponse` (per design "Key interfaces"). + - Verify: `npx nuxi typecheck` clean (no runtime). + - Covers: R1.2, R4.1, R5.1. + +- [x] **2. Extract the shared write util** — `server/utils/transactionWriter.ts` + edit `server/api/transactions.post.ts` + - Move `applyEnvelopePostings` + `toTransactionInput` + `appendTransaction` + composition into `appendSimplifiedTransaction(input: SimplifiedTransactionInput)`. + - Make `transactions.post.ts` a thin validate-and-call wrapper delegating to it. + - Tests: new `server/utils/__tests__/transactionWriter.test.ts` (asset expense → + 2-posting; liability expense → 4-posting; income → asset+income). Re-run existing + `transactions.post.ts` tests — must still pass (refactor parity). + - Covers: R4.3, NF2; design "Refactor to enable clean reuse". + +- [x] **3. Import context util** — `server/utils/importContext.ts` + - `getImportContext()` → `{ accounts: string[]; envelopes: string[] }` via + `hledgerExecText(['accounts'])` + existing `filterRealAccounts` / + `filterCategoryAccounts`. CRLF-safe split. + - Tests: `server/utils/__tests__/importContext.test.ts` — stub `hledgerExecText`, + assert real accounts vs expense categories partitioned; CRLF trimmed. + - Covers: R1.3, NF5. + +- [x] **4. Dedup util** — `server/utils/importDedup.ts` + - `computeDedupHash({date, amount, payee})` = `sha256(date|cents|payeeLowerTrimmed)`; + `loadJournalHashes()` reads existing entries via `getTransactionList()` and returns + a `Set`. + - Tests: `server/utils/__tests__/importDedup.test.ts` — hash stable across + equivalent inputs (e.g. `$5.00` vs `5`); differs on date/payee/amount; whitespace/ + case-insensitive payee. + - Covers: R5.1, R5.2, R5.3. + +- [x] **5. Parse prompt, schema, normalization** — `server/utils/importParse.ts` + - `IMPORT_SYSTEM_PROMPT` (map arbitrary CSV → normalized rows; positive magnitude + + direction; ISO dates; suggest from provided accounts/envelopes; leave envelope blank + if unsure; cache_control on the stable prefix), `IMPORT_SCHEMA` (json_schema, + `additionalProperties:false`, model-returned fields only), `MAX_IMPORT_ROWS = 200`, + and pure `normalizeProposals(parsed, context, sourceRows, journalHashes)` that + validates dates/amounts, blanks bogus suggestions, computes `dedupHash` + + `possibleDuplicate`, and collects `droppedRows`. + - Tests: `server/utils/__tests__/importParse.test.ts` — signed single column → + magnitude+direction; separate debit/credit → one direction; multi date-format → + `YYYY-MM-DD`; bogus envelope blanked; unparseable row → `droppedRows`. fast-check + property `importParse.property.test.ts`: normalized amount ≥ 0 and direction preserved. + - Covers: R1.2, R1.4, R6.1, R6.2, R6.3. + +- [x] **6. Parse route** — `server/api/import/parse.post.ts` + - Resolve client (`getAnthropic()`, 503 on `MissingApiKeyError`); enforce + `MAX_IMPORT_ROWS`; fetch context; call `client.messages.parse({...REQUEST_DEFAULTS, + system, messages, output_config: { format: { type:'json_schema', schema: IMPORT_SCHEMA }}})`; + handle `stop_reason: 'refusal'` and API errors (mirror #8's chat error mapping); + run `normalizeProposals`; return `ImportParseResponse`. **No journal writes.** + - Tests: `server/api/import/__tests__/parse.post.test.ts` — **(safety, R2.1)** mocked + SDK returns rows → assert `appendTransaction` never called and proposals returned; + missing key → 503; refusal → error + no proposals; over-cap → rejected. + - Covers: R1.1, R1.2, R1.5, R2.1, R7.1, R7.2, R7.3. + +- [x] **7. Commit route** — `server/api/import/commit.post.ts` + - Re-validate each `CommitRow` (date/amount/account/envelope; outflow requires + envelope); load journal hashes once; skip rows whose hash already exists; commit the + rest via `appendSimplifiedTransaction`; return `{ committed, skippedDuplicates, + failed }`. Partial success. + - Tests: `server/api/import/__tests__/commit.post.test.ts` — approved rows committed; + outflow w/ empty envelope rejected; inflow w/ empty envelope → income committed; + existing-journal hash skipped (R5.3); two same-hash batch rows both attempted (R5.4); + one invalid row doesn't block the valid ones (R4.2); writer failure → failed row, not a 500. + (Commit needs no API key — local write; R7.2 correction.) + - Covers: R2.2, R4.1, R4.2, R4.3, R4.4, R5.3, R5.4. + +- [x] **8. Composable** — `composables/useImport.ts` + - Reactive state + actions: `parse(csvText)`, editable proposal rows, approve/reject + toggles, `commit()` (sends approved rows), result summary, loading/error, no-key + flag. Thin `$fetch` client only — no business logic. + - Tests: `composables/__tests__/useImport.test.ts` — parse populates rows; reject + excludes a row from the commit payload; outflow-without-envelope not approvable; + commit surfaces the summary. + - Covers: R3.2, R3.3, R3.4, R4.1, NF1. + +- [x] **9. Review table component** — `components/ImportReviewTable.vue` + - UTable staging grid: date, payee (editable), amount, direction; account + envelope + dropdowns from `context`; per-row approve toggle; `possibleDuplicate` badge; + `sourceRow` shown (expandable/tooltip). Disable approve for outflow w/ empty envelope. + - Verify: renders in the page (task 10); `npx nuxi typecheck` clean. (Light/no unit + test — presentational; logic lives in the composable.) + - Covers: R3.1, R3.2, R3.3, R3.5, R5.2. + +- [x] **10. Import page + nav + egress notice** — `pages/import.vue`, edit `layouts/default.vue` + - UFileUpload (read CSV text client-side) → `useImport.parse`; persistent egress + notice before upload; no-key empty state linking to Settings; mount + `ImportReviewTable`; commit button → `useImport.commit` + result summary + (committed / skipped duplicates / failed). Add sidebar nav entry → Import. + - Verify: `npm run dev`, upload a sample CSV, confirm proposals render, edit a row, + approve, commit, and see the summary; confirm the egress notice is visible and the + no-key state appears when the key is unset. State how it was checked. + - Covers: R1.1, R3.1, R4.4, R7.2, R8.1. + +- [x] **11. AI-MAP.md update** (main agent, after impl) + - Add rows for the two `/api/import/*` routes, the new server utils + (`importParse`, `importContext`, `importDedup`, `transactionWriter`), `useImport`, + `ImportReviewTable`, `pages/import.vue`; note the CSV→Anthropic egress quirk. + - Covers: project AI-map maintenance. + +- [x] **12. Verification checkpoint** + - `npx vitest run` — all tests pass (new + existing, incl. refactor-parity for + `transactions.post.ts`). + - `npx nuxi typecheck` — clean (exit 0). + - `npm run build` — production build succeeds (page/component/routes compile). + - Confirmed no `vitest.config` / `tsconfig` / `nuxt.config` / `package.json` changes. + - NOTE: the live AI round-trip smoke test (upload → real Anthropic parse → commit) + was NOT run — it needs a configured `ANTHROPIC_API_KEY` and a live API call. Static + verification (build + typecheck + unit tests with a mocked SDK) is complete; the + end-to-end UI run is left for the user to confirm with a key configured. + - Covers: NF3, NF4, NF6. + +--- + +### Suggested commit points (batch over noise) + +- After **task 2** (refactor lands, parity green) — safe standalone commit. +- After **task 7** (server side complete: types, utils, both routes, all server tests). +- After **task 10** (UI works end-to-end). +- After **task 12** (map updated, full verification) — final. + +PR (when requested): body starts with `Fixes #9`, conventional-commit title +(`feat: AI-assisted CSV transaction import (human-in-the-loop)`). diff --git a/AI-MAP.md b/AI-MAP.md index 000766e..1615d0c 100644 --- a/AI-MAP.md +++ b/AI-MAP.md @@ -33,6 +33,7 @@ rejected on delete and upload**, since they break the date-line ↔ tindex mappi |---|---|---| | `/` | `index.vue` | Dashboard placeholder (hidden from nav) | | `/budget` | `budget.vue` | Envelope budget — Ready to Assign, groups, Assigned/Activity/Available, inline assign. **AI assistant** in a slideover (Issue #8) | +| `/import` | `import.vue` | **AI CSV import** (Issue #9): upload CSV → AI proposals → review/edit/approve table → commit. Persistent data-egress notice + no-key empty state | | `/reports` | `reports.vue` | Placeholder (hidden) | | `/settings` | `settings.vue` | Journal mgmt (create/upload/export/list/activate) + **AI Assistant** API-key config (Issue #8) | | `/accounts` | `accounts/index.vue` | Add/delete real accounts | @@ -42,6 +43,7 @@ rejected on delete and upload**, since they break the date-line ↔ tindex mappi - `components/AccountRegister.vue` — YNAB register table (Date, Payee, Envelope, Inflow, Outflow, Balance). For a real account the register is **family-aggregated**: rows net the account + its `:budget:*` envelopes, so Balance = the real bank balance and internal moves (assignments, envelope transfers) drop out. - `components/SimplifiedTransactionForm.vue` — Add-transaction modal (Account, Payee, Envelope, Inflow/Outflow). - `components/AiChatPanel.vue` — AI budgeting chat (Issue #8). Nuxt UI chat input + message bubbles + **proposed-action cards** (Approve/Reject) + persistent data-egress notice + no-API-key empty state. Emits `committed` (budget page refreshes). All logic via `useAiChat`. +- `components/ImportReviewTable.vue` — CSV import staging grid (Issue #9). Per-row editable date/payee/account/envelope (USelect dropdowns), approve checkbox, possible-duplicate badge, source-row tooltip. Presentational; mutates the rows passed from `useImport`. - `layouts/default.vue` — UDashboardGroup + sidebar + real-accounts UTree. ## Composables (`composables/`) — data fetch @@ -52,6 +54,11 @@ rejected on delete and upload**, since they break the date-line ↔ tindex mappi assign/transfer endpoints. Holds the opaque Anthropic history; `send`/`approve`/ `reject`. **Money is committed only here on user approval** (chat route never writes); auto-rejects un-acted proposals on a new message. +`useImport({onCommitted?})` (Issue #9) — client for `/api/import/{parse,commit}`. +`parse(csv,name)` → editable `rows` (pre-approves confident, non-duplicate rows); +`canApprove` (outflow needs an envelope); `commit()` sends only approved rows → +`result` summary. Exports `ImportRowState` (proposal + editable account/envelope + +approved). ## API surface (`server/api/`) | Method | Path | Purpose | @@ -59,13 +66,15 @@ writes); auto-rejects un-acted proposals on a new message. | GET | `/api/accounts?type=` | Account names (real/category/all; real excludes `:budget:`) | | GET | `/api/balances` | Balances (transformed) | | GET | `/api/transactions` | RegisterRow[] when account filter, else HledgerTransaction[] | -| POST | `/api/transactions` | Add (envelope-aware; CC expense → 4 postings) | +| POST | `/api/transactions` | Add (envelope-aware; CC expense → 4 postings). Delegates to `appendSimplifiedTransaction` | | DELETE | `/api/transactions?index=N` | Delete by journal index | | GET | `/api/budget?period=` | BudgetEnvelopeReport — Ready to Assign, Assigned/Activity/Available | | POST | `/api/budget/assign` | Assignment txn (**unallocated pool → envelope**; inverse of reduce) | | POST | `/api/budget/transfer` | Move between envelopes | | POST | `/api/ai/chat` | AI budgeting chat tool loop (Issue #8). **Never writes** — read tools run server-side; assign/transfer are *proposed* for HITL approval. Stateless (opaque history round-trips). 503 if no key configured | | GET·POST·DELETE | `/api/ai/config` | AI key status / save / clear (Issue #8). Returns `{configured, source, maskedKey}` — **never the full key**. Save takes effect with no restart | +| POST | `/api/import/parse` | AI CSV import (Issue #9): CSV → Anthropic **structured output** (`messages.parse`, json_schema) → normalized proposals + dropdown context + dropped rows. **Never writes.** 503 if no key; 413 over `MAX_IMPORT_ROWS` (200); 422 on refusal | +| POST | `/api/import/commit` | Write approved import rows via `appendSimplifiedTransaction` (outflow→expense, inflow→income). Re-validates, skips journal-existing duplicates, partial success `{committed,skippedDuplicates,failed}`. No key needed (local write) | | POST | `/api/categories` | Create expense groups/envelopes | | GET·POST | `/api/hidden-envelopes` | List / hide-unhide (zero balance to hide) | | * | `/api/journal/{create,upload,export,activate,list}` | Journal file management | @@ -114,6 +123,20 @@ writes); auto-rejects un-acted proposals on a new message. assign/transfer payload — **builds a proposal, never writes**). - `server/ai/budgetInstructions.ts` — `BUDGET_SYSTEM_PROMPT` (cached system prefix: YNAB Rule 1, envelope conventions, propose-never-execute, tone). +- `transactionWriter.ts` — `appendSimplifiedTransaction(SimplifiedTransactionInput)` + (Issue #9 extraction): the envelope-aware posting build (`toTransactionInput` + + `applyEnvelopePostings` + `appendTransaction`) lifted out of `transactions.post.ts` + so that route AND `import/commit` share one write path (no duplicated accounting). +- `importContext.ts` — `getImportContext()` → `{accounts, envelopes}` (real accounts + + expense-category keys) grounding the CSV-import AI's suggestions + review dropdowns; + delegates to `filterAccounts`. +- `importDedup.ts` — `computeDedupHash({date,amount,payee})` = `sha256(date|cents|payeeLower)`; + `loadJournalHashes()` (existing entries via `getTransactionList`). Dedup is a flag/skip, + never a silent drop (Issue #9 R5). +- `importParse.ts` — CSV import (Issue #9): `IMPORT_SYSTEM_PROMPT`, `IMPORT_SCHEMA` + (json_schema for structured outputs), `MAX_IMPORT_ROWS`, pure `normalizeProposals` + (re-validates dates/amounts/direction, grounds suggestions, computes dedup/possibleDuplicate, + collects dropped rows) + `normalizeDate`. ## Pure utils (`utils/`) — property-tested `formatAmount`, `stripAccountPrefix`, `buildAccountTree`, `filterAccounts` @@ -131,7 +154,9 @@ PostingInput, BalanceQuery, TransactionQuery), `ui.ts` (SimplifiedTransactionInp RegisterRow, BudgetCategory/Group, BudgetEnvelopeReport, RealAccount, AccountTreeItem), `ai.ts` (Issue #8: AssignProposalPayload, TransferProposalPayload, ProposedAction, ChatResolution, AiChatRequest/Response, ChatDisplayMessage — `messages` is opaque -Anthropic `MessageParam[]`, cast at the SDK boundary in `chat.post.ts`). +Anthropic `MessageParam[]`, cast at the SDK boundary in `chat.post.ts`), +`import.ts` (Issue #9: ImportDirection, ImportProposal, ImportParseResponse, CommitRow, +ImportCommitRequest/Response, DroppedRow). ## Known quirks / gotchas - **Windows CRLF:** hledger text output → `split(/\r?\n/)` + trim, else `\r` leaks (`%0D` in URLs). @@ -164,6 +189,15 @@ Anthropic `MessageParam[]`, cast at the SDK boundary in `chat.post.ts`). `claude-opus-4-8`, non-streaming, manual tool loop (capped at 8 iterations). **Data egress:** chat + budget data go to the Anthropic API (the one external flow); the panel shows a persistent notice. +- **AI CSV import (Issue #9):** `/api/import/parse` NEVER writes (guarded by + `parse.post.test.ts`) — it's a one-shot **structured-output** call (`messages.parse` + + json_schema, no tool loop), compatible with adaptive thinking, no forced + `tool_choice`. `/api/import/commit` is the only write path, only for approved rows, + via `appendSimplifiedTransaction`. Dedup hashes against the journal (no separate + import ledger); identical same-day rows are flagged, not silently dropped. + **Data egress:** uploaded CSV rows go to the Anthropic API — the import page shows a + persistent notice. Outflows need an envelope; uncategorized inflows → `income:uncategorized` + (Ready to Assign). Row cap `MAX_IMPORT_ROWS=200`. - **Robustness (Issue #4):** hledger spawns time out / reject (never hang) via `runHledger`; simplified `POST /api/transactions` rejects non-positive/non-finite amounts; the budget base is **derived** (`resolveBudgetBase`), not hardcoded diff --git a/components/ImportReviewTable.vue b/components/ImportReviewTable.vue new file mode 100644 index 0000000..05ab3eb --- /dev/null +++ b/components/ImportReviewTable.vue @@ -0,0 +1,100 @@ + + + diff --git a/composables/__tests__/useImport.test.ts b/composables/__tests__/useImport.test.ts new file mode 100644 index 0000000..97d4934 --- /dev/null +++ b/composables/__tests__/useImport.test.ts @@ -0,0 +1,97 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' +import { ref } from 'vue' +import type { ImportParseResponse, ImportProposal, ImportCommitResponse } from '~/types/import' + +// The composable uses auto-imported `ref` and `$fetch`. +vi.stubGlobal('ref', ref) +const fetchMock = vi.fn() +vi.stubGlobal('$fetch', (...args: any[]) => fetchMock(...args)) + +const { useImport } = await import('../useImport') + +const proposal = (over: Partial = {}): ImportProposal => ({ + id: '0', date: '2026-06-17', payee: 'Store', amount: 40, direction: 'outflow', + suggestedAccount: 'assets:checking', suggestedEnvelope: 'food:groceries', + dedupHash: 'h0', possibleDuplicate: false, sourceRow: 'raw', ...over, +}) + +const parseRes = (proposals: ImportProposal[]): ImportParseResponse => ({ + proposals, + context: { accounts: ['assets:checking'], envelopes: ['food:groceries'] }, + droppedRows: [], +}) + +beforeEach(() => fetchMock.mockReset()) + +describe('useImport.parse', () => { + it('populates editable rows and pre-approves confident non-duplicate rows', async () => { + fetchMock.mockResolvedValueOnce(parseRes([proposal()])) + const imp = useImport() + await imp.parse('date,desc,amt\n...', 'bank.csv') + + expect(imp.fileName.value).toBe('bank.csv') + expect(imp.rows.value).toHaveLength(1) + expect(imp.rows.value[0]!.account).toBe('assets:checking') + expect(imp.rows.value[0]!.envelope).toBe('food:groceries') + expect(imp.rows.value[0]!.approved).toBe(true) + }) + + it('does not pre-approve a possible duplicate or an outflow with no envelope', async () => { + fetchMock.mockResolvedValueOnce(parseRes([ + proposal({ id: '0', possibleDuplicate: true }), + proposal({ id: '1', suggestedEnvelope: '' }), + ])) + const imp = useImport() + await imp.parse('x', 'f.csv') + expect(imp.rows.value[0]!.approved).toBe(false) // duplicate + expect(imp.rows.value[1]!.approved).toBe(false) // outflow without envelope + }) + + it('surfaces a 503 as not-configured', async () => { + fetchMock.mockRejectedValueOnce({ statusCode: 503 }) + const imp = useImport() + await imp.parse('x', 'f.csv') + expect(imp.error.value).toBe('not-configured') + expect(imp.rows.value).toHaveLength(0) + }) +}) + +describe('useImport.canApprove', () => { + it('blocks an outflow with no envelope, allows an inflow with none', async () => { + const imp = useImport() + expect(imp.canApprove({ direction: 'outflow', envelope: '' } as any)).toBe(false) + expect(imp.canApprove({ direction: 'outflow', envelope: 'rent' } as any)).toBe(true) + expect(imp.canApprove({ direction: 'inflow', envelope: '' } as any)).toBe(true) + }) +}) + +describe('useImport.commit', () => { + it('sends only approved+approvable rows and exposes the summary', async () => { + fetchMock.mockResolvedValueOnce(parseRes([ + proposal({ id: '0' }), + proposal({ id: '1', payee: 'Skip' }), + ])) + const onCommitted = vi.fn() + const imp = useImport({ onCommitted }) + await imp.parse('x', 'f.csv') + + imp.rows.value[1]!.approved = false // user rejects the second row + const summary: ImportCommitResponse = { committed: 1, skippedDuplicates: [], failed: [] } + fetchMock.mockResolvedValueOnce(summary) + + await imp.commit() + + const body = fetchMock.mock.calls[1]![1].body.rows + expect(body).toHaveLength(1) + expect(body[0].payee).toBe('Store') + expect(imp.result.value).toEqual(summary) + expect(onCommitted).toHaveBeenCalledOnce() + }) + + it('refuses to commit when nothing is approved', async () => { + const imp = useImport() + await imp.commit() + expect(fetchMock).not.toHaveBeenCalled() + expect(imp.error.value).toMatch(/approve at least one/i) + }) +}) diff --git a/composables/useImport.ts b/composables/useImport.ts new file mode 100644 index 0000000..df96544 --- /dev/null +++ b/composables/useImport.ts @@ -0,0 +1,123 @@ +import { ref } from 'vue' +import type { + ImportProposal, ImportParseResponse, CommitRow, ImportCommitResponse, DroppedRow, +} from '~/types/import' + +/** + * A proposal row with the user's editable selections + approval state for the + * table. `account`/`envelope` are initialized from the model's suggestions and + * edited in place; `date`/`payee`/`amount` (from ImportProposal) are editable too. + */ +export interface ImportRowState extends ImportProposal { + approved: boolean + account: string + envelope: string +} + +function is503(e: unknown): boolean { + const err = e as any + return err?.statusCode === 503 || err?.status === 503 || err?.response?.status === 503 +} + +function errMessage(e: unknown): string { + const err = e as any + return err?.data?.message || err?.data?.statusMessage || err?.statusMessage || err?.message || 'Something went wrong.' +} + +/** + * Client for AI-assisted CSV import (Issue #9). Thin layer over + * `/api/import/parse` and `/api/import/commit` — no business logic, no + * accounting math. Holds the editable proposal rows and the commit result. + * + * @param options.onCommitted called after a commit that wrote ≥1 row, so the + * page can refresh balances/budget. + */ +export function useImport(options?: { onCommitted?: () => void }) { + const rows = ref([]) + const accounts = ref([]) + const envelopes = ref([]) + const droppedRows = ref([]) + const fileName = ref('') + const parsing = ref(false) + const committing = ref(false) + /** 'not-configured' (no API key) | a message | null. */ + const error = ref(null) + const result = ref(null) + + /** An outflow must have an envelope before it can be approved (R3.3). */ + function canApprove(row: ImportRowState): boolean { + return row.direction === 'inflow' || !!row.envelope + } + + async function parse(csvText: string, name: string): Promise { + if (parsing.value || !csvText.trim()) return + parsing.value = true + error.value = null + result.value = null + fileName.value = name + try { + const res = await $fetch('/api/import/parse', { + method: 'POST', + body: { csv: csvText }, + }) + accounts.value = res.context.accounts + envelopes.value = res.context.envelopes + droppedRows.value = res.droppedRows + // Pre-approve confident, non-duplicate rows; leave duplicates and + // unapprovable rows (outflow w/o envelope) for the user to decide. + rows.value = res.proposals.map((p): ImportRowState => ({ + ...p, + approved: !p.possibleDuplicate && (p.direction === 'inflow' || !!p.suggestedEnvelope), + account: p.suggestedAccount, + envelope: p.suggestedEnvelope, + })) + } catch (e) { + error.value = is503(e) ? 'not-configured' : errMessage(e) + rows.value = [] + } finally { + parsing.value = false + } + } + + async function commit(): Promise { + if (committing.value) return + const approved = rows.value.filter(r => r.approved && canApprove(r)) + if (!approved.length) { + error.value = 'Approve at least one row before importing.' + return + } + committing.value = true + error.value = null + try { + const body: CommitRow[] = approved.map(r => ({ + date: r.date, payee: r.payee, amount: r.amount, direction: r.direction, + account: r.account, envelope: r.envelope, dedupHash: r.dedupHash, + })) + result.value = await $fetch('/api/import/commit', { + method: 'POST', + body: { rows: body }, + }) + if (result.value.committed > 0) options?.onCommitted?.() + } catch (e) { + error.value = errMessage(e) + } finally { + committing.value = false + } + } + + function reset(): void { + rows.value = [] + accounts.value = [] + envelopes.value = [] + droppedRows.value = [] + fileName.value = '' + error.value = null + result.value = null + } + + return { + rows, accounts, envelopes, droppedRows, fileName, + parsing, committing, error, result, + canApprove, parse, commit, reset, + } +} diff --git a/layouts/default.vue b/layouts/default.vue index b6f8e3d..99e036a 100644 --- a/layouts/default.vue +++ b/layouts/default.vue @@ -34,6 +34,7 @@ function titleCaseTree(items: AccountTreeItem[]): AccountTreeItem[] { const links: NavigationMenuItem[][] = [[ { label: 'Budget', icon: 'i-lucide-target', to: '/budget' }, + { label: 'Import', icon: 'i-lucide-file-up', to: '/import' }, ]] const settingsLinks: NavigationMenuItem[][] = [[ diff --git a/pages/import.vue b/pages/import.vue new file mode 100644 index 0000000..9cae376 --- /dev/null +++ b/pages/import.vue @@ -0,0 +1,147 @@ + + + diff --git a/server/api/import/__tests__/commit.post.test.ts b/server/api/import/__tests__/commit.post.test.ts new file mode 100644 index 0000000..bebfb9b --- /dev/null +++ b/server/api/import/__tests__/commit.post.test.ts @@ -0,0 +1,87 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' +import { computeDedupHash } from '../../../utils/importDedup' +import type { CommitRow } from '../../../../types/import' + +const h = vi.hoisted(() => ({ + appendMock: vi.fn(), + contextMock: vi.fn(), + hashesMock: vi.fn(), +})) + +vi.mock('../../../utils/importContext', () => ({ getImportContext: (...a: any[]) => h.contextMock(...a) })) +vi.mock('../../../utils/transactionWriter', () => ({ appendSimplifiedTransaction: (...a: any[]) => h.appendMock(...a) })) +// Keep real computeDedupHash; stub only the journal read. +vi.mock('../../../utils/importDedup', async (orig) => ({ + ...(await orig()), + loadJournalHashes: (...a: any[]) => h.hashesMock(...a), +})) + +vi.stubGlobal('defineEventHandler', (fn: Function) => fn) +vi.stubGlobal('readBody', async (event: any) => event.body) + +const { default: commit } = await import('../commit.post') + +const ev = (rows: CommitRow[]) => ({ body: { rows } } as any) +const row = (over: Partial = {}): CommitRow => ({ + date: '2026-06-17', payee: 'Store', amount: 40, direction: 'outflow', + account: 'assets:checking', envelope: 'food:groceries', dedupHash: '', ...over, +}) + +beforeEach(() => { + vi.clearAllMocks() + h.contextMock.mockResolvedValue({ accounts: ['assets:checking'], envelopes: ['food:groceries'] }) + h.hashesMock.mockResolvedValue(new Set()) + h.appendMock.mockResolvedValue(undefined) +}) + +describe('POST /api/import/commit', () => { + it('commits valid approved rows via the shared writer', async () => { + const res = await commit(ev([row(), row({ payee: 'Cafe', amount: 5 })])) + expect(res.committed).toBe(2) + expect(h.appendMock).toHaveBeenCalledTimes(2) + expect(res.failed).toHaveLength(0) + }) + + it('rejects an outflow with no envelope (R3.3) without blocking the valid rows (R4.2)', async () => { + const res = await commit(ev([row({ envelope: '' }), row({ payee: 'OK' })])) + expect(res.committed).toBe(1) + expect(res.failed).toHaveLength(1) + expect(res.failed[0]!.error).toMatch(/envelope/i) + }) + + it('commits an inflow with no envelope as income → Ready to Assign (R3.4)', async () => { + const res = await commit(ev([row({ direction: 'inflow', envelope: '', payee: 'Refund' })])) + expect(res.committed).toBe(1) + const sent = h.appendMock.mock.calls[0]![0] + expect(sent.type).toBe('income') + expect(sent.category).toBe('income:uncategorized') + }) + + it('skips a row whose hash already exists in the journal (R5.3)', async () => { + const existing = computeDedupHash({ date: '2026-06-17', amount: 40, payee: 'Store' }) + h.hashesMock.mockResolvedValue(new Set([existing])) + const res = await commit(ev([row()])) + expect(res.committed).toBe(0) + expect(res.skippedDuplicates).toHaveLength(1) + expect(h.appendMock).not.toHaveBeenCalled() + }) + + it('commits two identical in-batch rows as distinct (R5.4)', async () => { + const res = await commit(ev([row(), row()])) + expect(res.committed).toBe(2) + expect(res.skippedDuplicates).toHaveLength(0) + }) + + it('rejects an unknown account', async () => { + const res = await commit(ev([row({ account: 'assets:nope' })])) + expect(res.committed).toBe(0) + expect(res.failed[0]!.error).toMatch(/account/i) + }) + + it('reports a writer failure as a failed row, not a thrown 500', async () => { + h.appendMock.mockRejectedValueOnce(new Error('unbalanced')) + const res = await commit(ev([row()])) + expect(res.committed).toBe(0) + expect(res.failed[0]!.error).toMatch(/unbalanced/) + }) +}) diff --git a/server/api/import/__tests__/parse.post.test.ts b/server/api/import/__tests__/parse.post.test.ts new file mode 100644 index 0000000..d2483a6 --- /dev/null +++ b/server/api/import/__tests__/parse.post.test.ts @@ -0,0 +1,93 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' + +// --- Hoisted mock state --- +const h = vi.hoisted(() => ({ + parseMock: vi.fn(), + appendMock: vi.fn(), + contextMock: vi.fn(), + hashesMock: vi.fn(), + state: { keyConfigured: true }, +})) + +class FakeMissingApiKeyError extends Error {} + +vi.mock('../../../utils/anthropic', () => ({ + getAnthropic: () => { + if (!h.state.keyConfigured) throw new FakeMissingApiKeyError() + return { messages: { parse: h.parseMock } } + }, + MissingApiKeyError: FakeMissingApiKeyError, + REQUEST_DEFAULTS: { model: 'claude-opus-4-8', max_tokens: 4096, output_config: { effort: 'medium' } }, +})) + +vi.mock('../../../utils/importContext', () => ({ getImportContext: (...a: any[]) => h.contextMock(...a) })) +// Keep the real computeDedupHash (normalizeProposals needs it); stub only the journal read. +vi.mock('../../../utils/importDedup', async (orig) => ({ + ...(await orig()), + loadJournalHashes: (...a: any[]) => h.hashesMock(...a), +})) +// The journal writer — proving parse NEVER writes (R2.1). The graph must not reach it. +vi.mock('../../../utils/journalWriter', () => ({ appendTransaction: (...a: any[]) => h.appendMock(...a) })) + +vi.stubGlobal('defineEventHandler', (fn: Function) => fn) +vi.stubGlobal('readBody', async (event: any) => event.body) +vi.stubGlobal('createError', (opts: any) => Object.assign(new Error(opts.statusMessage || opts.message), opts)) + +const { default: parse } = await import('../parse.post') + +const ev = (body: any) => ({ body } as any) + +beforeEach(() => { + vi.clearAllMocks() + h.state.keyConfigured = true + h.contextMock.mockResolvedValue({ accounts: ['assets:checking'], envelopes: ['rent'] }) + h.hashesMock.mockResolvedValue(new Set()) +}) + +describe('POST /api/import/parse — safety (R2.1)', () => { + it('returns proposals WITHOUT ever writing to the journal', async () => { + h.parseMock.mockResolvedValueOnce({ + stop_reason: 'end_turn', + parsed_output: { transactions: [ + { date: '2026-06-17', payee: 'Rent Co', amount: 1200, direction: 'outflow', + suggestedAccount: 'assets:checking', suggestedEnvelope: 'rent', sourceRow: '06/17/2026,Rent Co,-1200' }, + ] }, + }) + + const res = await parse(ev({ csv: 'date,desc,amount\n06/17/2026,Rent Co,-1200' })) + + expect(h.appendMock).not.toHaveBeenCalled() + expect(res.proposals).toHaveLength(1) + expect(res.proposals[0]!.suggestedEnvelope).toBe('rent') + expect(res.context.accounts).toEqual(['assets:checking']) + }) +}) + +describe('POST /api/import/parse — error handling', () => { + it('returns 503 when no API key is configured', async () => { + h.state.keyConfigured = false + await expect(parse(ev({ csv: 'a,b\n1,2' }))).rejects.toMatchObject({ statusCode: 503 }) + expect(h.parseMock).not.toHaveBeenCalled() + }) + + it('rejects an empty CSV', async () => { + await expect(parse(ev({ csv: ' ' }))).rejects.toMatchObject({ statusCode: 400 }) + }) + + it('rejects a CSV over the row cap', async () => { + const lines = ['header', ...Array.from({ length: 201 }, (_, i) => `row${i}`)].join('\n') + await expect(parse(ev({ csv: lines }))).rejects.toMatchObject({ statusCode: 413 }) + expect(h.parseMock).not.toHaveBeenCalled() + }) + + it('maps a refusal to an actionable error, no proposals, no writes', async () => { + h.parseMock.mockResolvedValueOnce({ stop_reason: 'refusal', parsed_output: null }) + await expect(parse(ev({ csv: 'a,b\n1,2' }))).rejects.toMatchObject({ statusCode: 422 }) + expect(h.appendMock).not.toHaveBeenCalled() + }) + + it('wraps an SDK/network failure as a 502 (no key leakage)', async () => { + h.parseMock.mockRejectedValueOnce(Object.assign(new Error('network down'), { status: 500 })) + await expect(parse(ev({ csv: 'a,b\n1,2' }))).rejects.toMatchObject({ statusCode: 502 }) + }) +}) diff --git a/server/api/import/commit.post.ts b/server/api/import/commit.post.ts new file mode 100644 index 0000000..3434133 --- /dev/null +++ b/server/api/import/commit.post.ts @@ -0,0 +1,99 @@ +import { getImportContext } from '../../utils/importContext' +import { computeDedupHash, loadJournalHashes } from '../../utils/importDedup' +import { normalizeDate } from '../../utils/importParse' +import { appendSimplifiedTransaction } from '../../utils/transactionWriter' +import type { SimplifiedTransactionInput } from '../../../types/ui' +import type { CommitRow, ImportCommitRequest, ImportCommitResponse } from '../../../types/import' + +/** + * POST /api/import/commit — write the user-approved import rows (Issue #9). + * + * This is the ONLY write path for the import feature. Each row is re-validated + * server-side (R4.2), checked against the existing journal for duplicates + * (skipped, not silently dropped — R5.3), and written via the shared + * `appendSimplifiedTransaction` so the envelope accounting matches the rest of + * the app. Partial success: one bad row never blocks the others. + * + * No Anthropic key is required here — committing is a local journal write. + */ + +/** Uncategorized inflows land here, which raises net worth → Ready to Assign. */ +const UNCATEGORIZED_INCOME = 'income:uncategorized' + +function validateRow( + row: CommitRow, + accounts: Set, + envelopes: Set, +): { ok: true; date: string } | { ok: false; error: string } { + const date = normalizeDate(String(row.date ?? '')) + if (!date) return { ok: false, error: 'Invalid date' } + if (typeof row.amount !== 'number' || !Number.isFinite(row.amount) || row.amount <= 0) { + return { ok: false, error: 'Amount must be a positive number' } + } + if (row.direction !== 'inflow' && row.direction !== 'outflow') { + return { ok: false, error: 'Invalid direction' } + } + if (!accounts.has(row.account)) { + return { ok: false, error: `Unknown account: ${row.account || '(none)'}` } + } + if (row.direction === 'outflow') { + // An outflow must hit a category to keep the budget balanced (R3.3 / R4.2). + if (!row.envelope) return { ok: false, error: 'Outflows require an envelope' } + if (!envelopes.has(row.envelope)) return { ok: false, error: `Unknown envelope: ${row.envelope}` } + } + return { ok: true, date } +} + +function toSimplified(row: CommitRow, date: string): SimplifiedTransactionInput { + if (row.direction === 'outflow') { + return { + date, payee: row.payee, account: row.account, type: 'expense', + category: `expenses:${row.envelope}`, amount: row.amount, + } + } + // Inflow → income; uncategorized lands in Ready to Assign (R3.4). + return { + date, payee: row.payee, account: row.account, type: 'income', + category: UNCATEGORIZED_INCOME, amount: row.amount, + } +} + +export default defineEventHandler(async (event): Promise => { + const body = await readBody(event) + const rows = Array.isArray(body.rows) ? body.rows : [] + + const context = await getImportContext() + const accounts = new Set(context.accounts) + const envelopes = new Set(context.envelopes) + const journalHashes = await loadJournalHashes() + + let committed = 0 + const skippedDuplicates: CommitRow[] = [] + const failed: { row: CommitRow; error: string }[] = [] + + for (const row of rows) { + const v = validateRow(row, accounts, envelopes) + if (!v.ok) { + failed.push({ row, error: v.error }) + continue + } + + const hash = computeDedupHash({ date: v.date, amount: row.amount, payee: row.payee }) + if (journalHashes.has(hash)) { + skippedDuplicates.push(row) + continue + } + + try { + await appendSimplifiedTransaction(toSimplified(row, v.date)) + committed++ + // NB: we do NOT add `hash` to the set here. Two identical approved rows in + // one batch are legitimately distinct (R5.4) — both commit. Only hashes that + // already existed in the journal are auto-skipped. + } catch (err) { + failed.push({ row, error: (err as Error).message || 'Failed to write transaction' }) + } + } + + return { committed, skippedDuplicates, failed } +}) diff --git a/server/api/import/parse.post.ts b/server/api/import/parse.post.ts new file mode 100644 index 0000000..54c414e --- /dev/null +++ b/server/api/import/parse.post.ts @@ -0,0 +1,105 @@ +import { getAnthropic, MissingApiKeyError, REQUEST_DEFAULTS } from '../../utils/anthropic' +import { getImportContext } from '../../utils/importContext' +import { loadJournalHashes } from '../../utils/importDedup' +import { + IMPORT_SYSTEM_PROMPT, IMPORT_SCHEMA, MAX_IMPORT_ROWS, normalizeProposals, +} from '../../utils/importParse' +import type { ImportParseResponse } from '../../../types/import' + +/** + * POST /api/import/parse — turn an uploaded CSV into proposed transactions (Issue #9). + * + * SAFETY INVARIANT: this route NEVER writes to the journal. It reads the account + * list, calls Anthropic with a structured-output (json_schema) request, and + * returns normalized proposals for the user to review. Writes happen only in + * commit.post.ts, only for approved rows. (Guarded by parse.post.test.ts.) + */ + +interface ParseBody { + csv?: unknown +} + +export default defineEventHandler(async (event): Promise => { + const body = await readBody(event) + const csv = typeof body.csv === 'string' ? body.csv : '' + if (!csv.trim()) { + throw createError({ statusCode: 400, statusMessage: 'No CSV content provided.' }) + } + + // Bound output size: cap the number of data rows (one header row assumed). + const nonEmptyLines = csv.split(/\r?\n/).filter(l => l.trim()).length + if (nonEmptyLines - 1 > MAX_IMPORT_ROWS) { + throw createError({ + statusCode: 413, + statusMessage: `CSV has too many rows (max ${MAX_IMPORT_ROWS}). Please split the file and import in batches.`, + }) + } + + let client + try { + client = getAnthropic() + } catch (err) { + if (err instanceof MissingApiKeyError) { + throw createError({ + statusCode: 503, + statusMessage: 'AI import is not configured. Set your Anthropic API key in Settings to enable it.', + }) + } + throw err + } + + // Ground the model's suggestions in real targets; also feeds the review dropdowns. + const context = await getImportContext() + + const userMessage = [ + 'Valid real accounts (use exactly one of these or ""):', + context.accounts.join('\n') || '(none)', + '', + 'Valid envelope keys (use exactly one of these or ""):', + context.envelopes.join('\n') || '(none)', + '', + 'CSV to convert:', + csv, + ].join('\n') + + let parsed: unknown + try { + const response = await client.messages.parse({ + ...REQUEST_DEFAULTS, + // Override the chat default (4096): proposal arrays can be large; non-streaming + // stays well under the SDK HTTP-timeout threshold at this size with the row cap. + max_tokens: 16000, + output_config: { ...REQUEST_DEFAULTS.output_config, format: { type: 'json_schema', schema: IMPORT_SCHEMA } }, + system: [{ type: 'text', text: IMPORT_SYSTEM_PROMPT, cache_control: { type: 'ephemeral' } }], + messages: [{ role: 'user', content: userMessage }], + }) + + if (response.stop_reason === 'refusal') { + throw createError({ + statusCode: 422, + statusMessage: 'The assistant declined to process this file. Please check its contents and try again.', + }) + } + parsed = response.parsed_output + } catch (err) { + // Re-throw our own createError (it carries a statusCode); wrap SDK/network failures. + if ((err as { statusCode?: number })?.statusCode) throw err + const status = (err as { status?: number })?.status + const message = (err as { message?: string })?.message ?? 'unknown error' + const name = (err as { name?: string })?.name ?? 'Error' + // Log structure/transport only — never the CSV contents or the key (R8.2). + console.error(`[import/parse] Anthropic request failed: ${name}${status ? ` (status ${status})` : ''}: ${message}`) + + let statusMessage = 'Could not reach the assistant. Please try again.' + if (/credit balance|billing|quota/i.test(message)) statusMessage = 'Your Anthropic account is out of credits. Add credits in the Anthropic Console, then try again.' + else if (status === 401) statusMessage = 'The Anthropic API key was rejected. Check it in Settings.' + else if (status === 403) statusMessage = 'This Anthropic API key is not permitted to use this model. Check it in Settings.' + else if (status === 429) statusMessage = 'Anthropic is rate-limiting requests — please wait a moment and try again.' + throw createError({ statusCode: 502, statusMessage }) + } + + const journalHashes = await loadJournalHashes() + const { proposals, droppedRows } = normalizeProposals(parsed, context, journalHashes) + + return { proposals, context, droppedRows } +}) diff --git a/server/api/transactions.post.ts b/server/api/transactions.post.ts index bbb7d60..5e370d7 100644 --- a/server/api/transactions.post.ts +++ b/server/api/transactions.post.ts @@ -1,7 +1,7 @@ import type { TransactionInput } from '../../types/api' import type { SimplifiedTransactionInput } from '../../types/ui' -import { toTransactionInput } from '../../utils/toTransactionInput' import { appendTransaction } from '../utils/journalWriter' +import { appendSimplifiedTransaction } from '../utils/transactionWriter' function isSimplifiedInput(body: any): body is SimplifiedTransactionInput { return typeof body.payee === 'string' && typeof body.type === 'string' @@ -11,54 +11,6 @@ function isLegacyInput(body: any): body is TransactionInput { return typeof body.description === 'string' && Array.isArray(body.postings) } -/** - * Post-process a simplified expense to use envelope budget sub-accounts. - * - For asset accounts: 2-posting (expense debit, budget sub-account credit) - * - For liability accounts: 4-posting (expense debit, budget sub-account credit, - * pending credit card budget debit, liability credit) - */ -async function applyEnvelopePostings( - txInput: TransactionInput, - body: SimplifiedTransactionInput, -): Promise { - if (body.type !== 'expense' || !body.category) { - return txInput - } - - const commodity = body.commodity ?? '$' - const envelopeCategory = body.category.replace(/^expenses:/, '') - - if (body.account.startsWith('liabilities:')) { - // Credit card expense: 4-posting structure. Derive the budget base from the - // journal (Issue #4 item 3) rather than hardcoding `assets:checking`, so a - // non-default primary account routes its envelope postings correctly. - const budgetBase = await resolveBudgetBase() - const liabilityName = body.account.replace(/^liabilities:/, '') - return { - ...txInput, - postings: [ - { account: body.category, amount: body.amount, commodity }, - { account: `${budgetBase}:budget:${envelopeCategory}`, amount: -body.amount, commodity }, - { account: `${budgetBase}:budget:pending:${liabilityName}`, amount: body.amount, commodity }, - { account: body.account, amount: -body.amount, commodity }, - ], - } - } - - if (body.account.startsWith('assets:')) { - // Regular expense: debit expense, credit budget sub-account - return { - ...txInput, - postings: [ - { account: body.category, amount: body.amount, commodity }, - { account: `${body.account}:budget:${envelopeCategory}`, amount: -body.amount, commodity }, - ], - } - } - - return txInput -} - export default defineEventHandler(async (event) => { const body = await readBody(event) @@ -73,10 +25,8 @@ export default defineEventHandler(async (event) => { if (typeof body.amount !== 'number' || !Number.isFinite(body.amount) || body.amount <= 0) { throw createError({ statusCode: 400, message: 'Amount must be a positive number' }) } - let txInput = toTransactionInput(body) - txInput = await applyEnvelopePostings(txInput, body) try { - await appendTransaction(txInput) + await appendSimplifiedTransaction(body) } catch (err: any) { throw createError({ statusCode: 400, message: err.message || 'Transaction validation failed' }) } diff --git a/server/utils/__tests__/importContext.test.ts b/server/utils/__tests__/importContext.test.ts new file mode 100644 index 0000000..0239ad6 --- /dev/null +++ b/server/utils/__tests__/importContext.test.ts @@ -0,0 +1,35 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' + +const h = vi.hoisted(() => ({ accounts: vi.fn() })) +// hledgerExecText is a Nitro auto-imported global at runtime. +vi.stubGlobal('hledgerExecText', (...a: any[]) => h.accounts(...a)) + +const { getImportContext } = await import('../importContext') + +beforeEach(() => vi.clearAllMocks()) + +describe('getImportContext', () => { + it('partitions real accounts from envelope keys and strips the expenses: prefix', async () => { + h.accounts.mockResolvedValue([ + 'assets:checking', + 'assets:savings', + 'liabilities:visa', + 'assets:checking:budget:food', // budget sub-account — not a real account target + 'expenses:food:groceries', + 'expenses:rent', + 'income:salary', // category but not an expense → not an envelope key + ].join('\n')) + + const { accounts, envelopes } = await getImportContext() + + expect(accounts).toEqual(['assets:checking', 'assets:savings', 'liabilities:visa']) + expect(envelopes).toEqual(['food:groceries', 'rent']) + }) + + it('is CRLF-safe and trims/ignores blank lines (Windows hledger output)', async () => { + h.accounts.mockResolvedValue('assets:checking\r\nexpenses:rent\r\n\r\n') + const { accounts, envelopes } = await getImportContext() + expect(accounts).toEqual(['assets:checking']) + expect(envelopes).toEqual(['rent']) + }) +}) diff --git a/server/utils/__tests__/importDedup.test.ts b/server/utils/__tests__/importDedup.test.ts new file mode 100644 index 0000000..0e97e2a --- /dev/null +++ b/server/utils/__tests__/importDedup.test.ts @@ -0,0 +1,42 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' + +const h = vi.hoisted(() => ({ txList: vi.fn() })) +vi.mock('../transactionList', () => ({ getTransactionList: (...a: any[]) => h.txList(...a) })) + +const { computeDedupHash, loadJournalHashes } = await import('../importDedup') + +beforeEach(() => vi.clearAllMocks()) + +describe('computeDedupHash', () => { + it('is stable across equivalent amounts and payee casing/whitespace', () => { + const a = computeDedupHash({ date: '2026-06-17', amount: 5, payee: 'Coffee Shop' }) + const b = computeDedupHash({ date: '2026-06-17', amount: 5.0, payee: ' coffee shop ' }) + expect(a).toBe(b) + }) + + it('differs on date, amount, or payee', () => { + const base = computeDedupHash({ date: '2026-06-17', amount: 5, payee: 'X' }) + expect(computeDedupHash({ date: '2026-06-18', amount: 5, payee: 'X' })).not.toBe(base) + expect(computeDedupHash({ date: '2026-06-17', amount: 6, payee: 'X' })).not.toBe(base) + expect(computeDedupHash({ date: '2026-06-17', amount: 5, payee: 'Y' })).not.toBe(base) + }) + + it('treats $5.00 and 5 as the same (cents rounding)', () => { + expect(computeDedupHash({ date: '2026-06-17', amount: 5.004, payee: 'X' })) + .toBe(computeDedupHash({ date: '2026-06-17', amount: 5, payee: 'X' })) + }) +}) + +describe('loadJournalHashes', () => { + it('builds a hash set from existing journal entries (sign-insensitive)', async () => { + h.txList.mockResolvedValue([ + { date: '2026-06-01', payee: 'Rent', amount: -1200, account: 'expenses:rent' }, + { date: '2026-06-02', payee: 'Salary', amount: 3000, account: 'income:salary' }, + ]) + const hashes = await loadJournalHashes() + // An outflow proposal (positive magnitude) matches the negative journal leg. + expect(hashes.has(computeDedupHash({ date: '2026-06-01', amount: 1200, payee: 'Rent' }))).toBe(true) + expect(hashes.has(computeDedupHash({ date: '2026-06-02', amount: 3000, payee: 'Salary' }))).toBe(true) + expect(hashes.has(computeDedupHash({ date: '2026-06-03', amount: 1, payee: 'Nope' }))).toBe(false) + }) +}) diff --git a/server/utils/__tests__/importParse.property.test.ts b/server/utils/__tests__/importParse.property.test.ts new file mode 100644 index 0000000..5ce51b2 --- /dev/null +++ b/server/utils/__tests__/importParse.property.test.ts @@ -0,0 +1,34 @@ +import { describe, it, expect } from 'vitest' +import fc from 'fast-check' +import { normalizeProposals } from '../importParse' + +const CONTEXT = { accounts: ['assets:checking'], envelopes: ['rent'] } + +/** + * Property: for any row the normalizer accepts, the output amount is a positive + * magnitude and the direction is preserved verbatim — sign never leaks into the + * amount, and the normalizer never flips inflow/outflow (R6.2). + */ +describe('normalizeProposals — property', () => { + it('accepted rows always have amount >= 0 and preserved direction', () => { + fc.assert( + fc.property( + fc.float({ noNaN: true, min: Math.fround(-100000), max: Math.fround(100000) }), + fc.constantFrom('inflow', 'outflow'), + fc.string(), + (amount, direction, payee) => { + const { proposals } = normalizeProposals( + { transactions: [{ date: '2026-06-17', payee, amount, direction, suggestedAccount: '', suggestedEnvelope: '', sourceRow: 'r' }] }, + CONTEXT, new Set(), + ) + // amount 0 is rejected (dropped); any accepted row obeys the invariant. + for (const p of proposals) { + expect(p.amount).toBeGreaterThanOrEqual(0) + expect(p.amount).toBe(Math.abs(p.amount)) + expect(p.direction).toBe(direction) + } + }, + ), + ) + }) +}) diff --git a/server/utils/__tests__/importParse.test.ts b/server/utils/__tests__/importParse.test.ts new file mode 100644 index 0000000..fdcf070 --- /dev/null +++ b/server/utils/__tests__/importParse.test.ts @@ -0,0 +1,94 @@ +import { describe, it, expect } from 'vitest' +import { normalizeDate, normalizeProposals } from '../importParse' + +const CONTEXT = { + accounts: ['assets:checking', 'liabilities:visa'], + envelopes: ['food:groceries', 'rent'], +} +const NO_HASHES = new Set() + +function wrap(transactions: unknown[]) { + return { transactions } +} + +describe('normalizeDate', () => { + it('passes through ISO dates', () => { + expect(normalizeDate('2026-06-17')).toBe('2026-06-17') + expect(normalizeDate('2026-6-7')).toBe('2026-06-07') + }) + it('converts MM/DD/YYYY (US default) and DD/MM when day > 12', () => { + expect(normalizeDate('06/17/2026')).toBe('2026-06-17') // MM/DD + expect(normalizeDate('17/06/2026')).toBe('2026-06-17') // first part > 12 → DD/MM + }) + it('converts "D Mon YYYY"', () => { + expect(normalizeDate('5 Jun 2026')).toBe('2026-06-05') + expect(normalizeDate('17 December 2026')).toBe('2026-12-17') + }) + it('rejects nonsense and impossible dates', () => { + expect(normalizeDate('not a date')).toBeNull() + expect(normalizeDate('2026-13-01')).toBeNull() + expect(normalizeDate('2026-02-30')).toBeNull() + }) +}) + +describe('normalizeProposals', () => { + it('maps a signed single-column outflow to magnitude + direction', () => { + const { proposals, droppedRows } = normalizeProposals( + wrap([{ date: '2026-06-17', payee: 'Store', amount: 42.5, direction: 'outflow', + suggestedAccount: 'assets:checking', suggestedEnvelope: 'food:groceries', sourceRow: 'raw1' }]), + CONTEXT, NO_HASHES, + ) + expect(droppedRows).toHaveLength(0) + expect(proposals[0]).toMatchObject({ + date: '2026-06-17', amount: 42.5, direction: 'outflow', + suggestedAccount: 'assets:checking', suggestedEnvelope: 'food:groceries', sourceRow: 'raw1', + }) + }) + + it('takes the magnitude of a negative amount (separate debit column case)', () => { + const { proposals } = normalizeProposals( + wrap([{ date: '2026-06-17', payee: 'X', amount: -19.99, direction: 'outflow', + suggestedAccount: '', suggestedEnvelope: '', sourceRow: 'r' }]), + CONTEXT, NO_HASHES, + ) + expect(proposals[0]!.amount).toBe(19.99) + expect(proposals[0]!.direction).toBe('outflow') + }) + + it('blanks suggestions that are not real targets, strips an expenses: prefix', () => { + const { proposals } = normalizeProposals( + wrap([{ date: '2026-06-17', payee: 'X', amount: 5, direction: 'outflow', + suggestedAccount: 'assets:nope', suggestedEnvelope: 'expenses:rent', sourceRow: 'r' }]), + CONTEXT, NO_HASHES, + ) + expect(proposals[0]!.suggestedAccount).toBe('') // not in context + expect(proposals[0]!.suggestedEnvelope).toBe('rent') // prefix stripped, matches context + }) + + it('surfaces unparseable rows in droppedRows (never silently dropped, R1.4)', () => { + const { proposals, droppedRows } = normalizeProposals( + wrap([ + { date: 'garbage', payee: 'X', amount: 5, direction: 'outflow', suggestedAccount: '', suggestedEnvelope: '', sourceRow: 'bad-date' }, + { date: '2026-06-17', payee: 'X', amount: 0, direction: 'outflow', suggestedAccount: '', suggestedEnvelope: '', sourceRow: 'zero-amt' }, + { date: '2026-06-17', payee: 'X', amount: 5, direction: 'sideways', suggestedAccount: '', suggestedEnvelope: '', sourceRow: 'bad-dir' }, + { date: '2026-06-17', payee: 'OK', amount: 5, direction: 'inflow', suggestedAccount: '', suggestedEnvelope: '', sourceRow: 'good' }, + ]), + CONTEXT, NO_HASHES, + ) + expect(proposals).toHaveLength(1) + expect(proposals[0]!.sourceRow).toBe('good') + expect(droppedRows.map(d => d.sourceRow)).toEqual(['bad-date', 'zero-amt', 'bad-dir']) + }) + + it('flags possibleDuplicate when the hash exists in the journal', () => { + const first = normalizeProposals( + wrap([{ date: '2026-06-17', payee: 'Coffee', amount: 5, direction: 'outflow', suggestedAccount: '', suggestedEnvelope: '', sourceRow: 'r' }]), + CONTEXT, NO_HASHES, + ).proposals[0]! + const { proposals } = normalizeProposals( + wrap([{ date: '2026-06-17', payee: 'Coffee', amount: 5, direction: 'outflow', suggestedAccount: '', suggestedEnvelope: '', sourceRow: 'r' }]), + CONTEXT, new Set([first.dedupHash]), + ) + expect(proposals[0]!.possibleDuplicate).toBe(true) + }) +}) diff --git a/server/utils/__tests__/transactionWriter.test.ts b/server/utils/__tests__/transactionWriter.test.ts new file mode 100644 index 0000000..a60e8ca --- /dev/null +++ b/server/utils/__tests__/transactionWriter.test.ts @@ -0,0 +1,67 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' +import type { TransactionInput } from '../../../types/api' + +// Capture what gets handed to the journal writer so we can assert the postings +// the simplified→balanced conversion produced (Issue #9 extraction). +const h = vi.hoisted(() => ({ append: vi.fn() })) +vi.mock('../journalWriter', () => ({ appendTransaction: (...a: any[]) => h.append(...a) })) + +// `resolveBudgetBase` is a Nitro auto-imported global in app runtime. +vi.stubGlobal('resolveBudgetBase', async () => 'assets:checking') + +const { appendSimplifiedTransaction } = await import('../transactionWriter') + +const lastTx = (): TransactionInput => h.append.mock.calls.at(-1)![0] + +beforeEach(() => { + vi.clearAllMocks() + h.append.mockResolvedValue(undefined) +}) + +describe('appendSimplifiedTransaction — envelope-aware postings', () => { + it('expense from an asset account → 2 postings (expense debit, budget sub-account credit)', async () => { + await appendSimplifiedTransaction({ + date: '2026-06-17', payee: 'Store', account: 'assets:checking', + type: 'expense', category: 'expenses:food:groceries', amount: 40, + }) + const tx = lastTx() + expect(tx.postings).toEqual([ + { account: 'expenses:food:groceries', amount: 40, commodity: '$' }, + { account: 'assets:checking:budget:food:groceries', amount: -40, commodity: '$' }, + ]) + }) + + it('expense from a liability account → 4 postings (with pending budget leg)', async () => { + await appendSimplifiedTransaction({ + date: '2026-06-17', payee: 'Store', account: 'liabilities:visa', + type: 'expense', category: 'expenses:food:dining', amount: 25, + }) + const tx = lastTx() + expect(tx.postings).toEqual([ + { account: 'expenses:food:dining', amount: 25, commodity: '$' }, + { account: 'assets:checking:budget:food:dining', amount: -25, commodity: '$' }, + { account: 'assets:checking:budget:pending:visa', amount: 25, commodity: '$' }, + { account: 'liabilities:visa', amount: -25, commodity: '$' }, + ]) + }) + + it('income → asset debit + income credit (no envelope postings)', async () => { + await appendSimplifiedTransaction({ + date: '2026-06-17', payee: 'Employer', account: 'assets:checking', + type: 'income', category: 'income:salary', amount: 1000, + }) + const tx = lastTx() + expect(tx.postings).toEqual([ + { account: 'assets:checking', amount: 1000, commodity: '$' }, + { account: 'income:salary', amount: -1000, commodity: '$' }, + ]) + }) + + it('propagates a validation failure from the journal writer (no swallowing)', async () => { + h.append.mockRejectedValueOnce(new Error('Postings do not sum to zero')) + await expect(appendSimplifiedTransaction({ + date: 'bad', payee: 'x', account: 'assets:checking', + type: 'income', category: 'income:salary', amount: 5, + })).rejects.toThrow(/sum to zero/) + }) +}) diff --git a/server/utils/importContext.ts b/server/utils/importContext.ts new file mode 100644 index 0000000..4ed8c2b --- /dev/null +++ b/server/utils/importContext.ts @@ -0,0 +1,27 @@ +import { filterRealAccounts, filterCategoryAccounts } from '../../utils/filterAccounts' + +/** + * Valid targets the CSV-import AI may suggest (Issue #9). + * + * Returns the real accounts (assets:/liabilities:) the user can attribute a + * transaction to, and the envelope keys (expense categories, "expenses:" prefix + * stripped) it can land in. These ground the model's suggestions in real targets + * and populate the review-table dropdowns. Read-only and delegation-only — it + * reuses `hledgerExecText` (Nitro auto-imported) and the existing pure + * `filterAccounts` helpers; no accounting logic here. + * + * CRLF-safe: hledger emits `\r\n` on Windows, so split on `/\r?\n/` and trim. + */ +export async function getImportContext(): Promise<{ accounts: string[]; envelopes: string[] }> { + const raw = await hledgerExecText(['accounts']) + const accounts = raw.trim().split(/\r?\n/).filter(Boolean).map(s => s.trim()) + + const realAccounts = filterRealAccounts(accounts) + // Envelope keys are expense categories with the "expenses:" prefix stripped, so + // they match the budget sub-account naming the journal writer expects. + const envelopes = filterCategoryAccounts(accounts) + .filter(a => a.startsWith('expenses:')) + .map(a => a.replace(/^expenses:/, '')) + + return { accounts: realAccounts, envelopes } +} diff --git a/server/utils/importDedup.ts b/server/utils/importDedup.ts new file mode 100644 index 0000000..7399e40 --- /dev/null +++ b/server/utils/importDedup.ts @@ -0,0 +1,37 @@ +import { createHash } from 'node:crypto' +import { getTransactionList } from './transactionList' + +/** + * Duplicate detection for CSV import (Issue #9). + * + * A transaction's identity for dedup is (date, integer cents, normalized payee). + * The hash is computed identically for proposals (to flag `possibleDuplicate` + * against the existing journal at parse time) and for commit rows (to skip rows + * already in the journal). Because every committed row becomes a real journal + * entry, re-importing the same statement is caught by checking the journal — no + * separate "imported ledger" file is needed, keeping the server stateless. + * + * Dedup is a SAFETY NET, not a silent drop: identical same-day transactions are + * legitimate, so matches are surfaced for the user to confirm (R5). + */ + +/** Stable identity hash: date + integer cents + lowercased/trimmed payee. */ +export function computeDedupHash(input: { date: string; amount: number; payee: string }): string { + const cents = Math.round(input.amount * 100) + const payee = input.payee.trim().toLowerCase() + return createHash('sha256').update(`${input.date}|${cents}|${payee}`).digest('hex') +} + +/** + * Build the set of dedup hashes already present in the journal. Reads via + * `getTransactionList` (which yields one entry per category leg); the amount is + * taken as a magnitude so the hash matches a proposal regardless of leg sign. + */ +export async function loadJournalHashes(): Promise> { + const entries = await getTransactionList({ limit: 100000 }) + const hashes = new Set() + for (const e of entries) { + hashes.add(computeDedupHash({ date: e.date, amount: Math.abs(e.amount), payee: e.payee })) + } + return hashes +} diff --git a/server/utils/importParse.ts b/server/utils/importParse.ts new file mode 100644 index 0000000..dcd6779 --- /dev/null +++ b/server/utils/importParse.ts @@ -0,0 +1,196 @@ +import type { ImportProposal, ImportDirection, DroppedRow } from '../../types/import' +import { computeDedupHash } from './importDedup' + +/** + * Prompt, structured-output schema, and normalization for CSV import parsing + * (Issue #9). + * + * The model performs a one-shot extraction: arbitrary CSV layout → a normalized + * array of transactions. We use Anthropic structured outputs (json_schema), so + * the SDK validates the response against IMPORT_SCHEMA and we get typed data + * back. `normalizeProposals` is a pure function that re-validates every field + * (dates, amounts, direction) and grounds the model's account/envelope + * suggestions in the real targets — rows that fail validation are surfaced in + * `droppedRows`, never silently discarded (R1.4). + */ + +/** Max CSV rows per parse — bounds output tokens (non-streaming). */ +export const MAX_IMPORT_ROWS = 200 + +export const IMPORT_SYSTEM_PROMPT = `You convert a bank/credit-card CSV export into normalized transactions for a budgeting app. + +For EACH data row in the CSV (skip the header row and blank lines), emit one transaction with: +- date: the transaction date as ISO "YYYY-MM-DD". Infer the source format (MM/DD/YYYY, DD/MM/YYYY, "5 Jun 2026", etc.) and convert. +- payee: who was paid or who paid, from the description/merchant/memo column. Keep it human-readable; strip noise like reference numbers when obvious. +- amount: a POSITIVE number (the magnitude). Never negative. +- direction: "outflow" if money left the account (a debit/purchase/withdrawal), "inflow" if money came in (a credit/deposit/refund). Determine this from the sign, or from separate debit/credit columns, or from the wording. +- suggestedAccount: the best-matching real account from the provided list, or "" if unsure. +- suggestedEnvelope: the best-matching expense category (envelope) from the provided list for an outflow, or "" if unsure. Leave "" for inflows. Do not invent categories that are not in the list. +- sourceRow: the original CSV line, copied verbatim, so the user can verify your mapping. + +Rules: +- Output every data row. If a row is malformed, still emit it with your best guess and the verbatim sourceRow. +- amount is always positive; the sign lives in direction. +- Only suggest accounts/envelopes that appear in the provided lists; otherwise use "".` + +/** + * JSON schema for structured outputs — model-returned fields only. + * Typed as a plain record so it assigns to the SDK's `JSONOutputFormat.schema` + * (`{ [key: string]: unknown }`) without a cast at the route boundary. + */ +export const IMPORT_SCHEMA: Record = { + type: 'object', + additionalProperties: false, + properties: { + transactions: { + type: 'array', + items: { + type: 'object', + additionalProperties: false, + properties: { + date: { type: 'string', description: 'Transaction date, ISO YYYY-MM-DD.' }, + payee: { type: 'string', description: 'Human-readable payee/merchant.' }, + amount: { type: 'number', description: 'Positive magnitude.' }, + direction: { type: 'string', enum: ['inflow', 'outflow'] }, + suggestedAccount: { type: 'string', description: 'Real account from the list, or "".' }, + suggestedEnvelope: { type: 'string', description: 'Envelope key from the list, or "".' }, + sourceRow: { type: 'string', description: 'Verbatim original CSV line.' }, + }, + required: ['date', 'payee', 'amount', 'direction', 'suggestedAccount', 'suggestedEnvelope', 'sourceRow'], + }, + }, + }, + required: ['transactions'], +} + +const MONTHS: Record = { + jan: 1, feb: 2, mar: 3, apr: 4, may: 5, jun: 6, + jul: 7, aug: 8, sep: 9, oct: 10, nov: 11, dec: 12, +} + +function pad(n: number): string { + return String(n).padStart(2, '0') +} + +/** True if (year, month, day) is a real calendar date. */ +function isRealDate(y: number, m: number, d: number): boolean { + if (m < 1 || m > 12 || d < 1 || d > 31) return false + const dt = new Date(y, m - 1, d) + return dt.getFullYear() === y && dt.getMonth() === m - 1 && dt.getDate() === d +} + +/** + * Normalize a date string to YYYY-MM-DD, or null if unparseable. Tolerates ISO, + * slash formats (MM/DD/YYYY US default; DD/MM when the first part is > 12), and + * "D Mon YYYY". A safety net over the model's ISO output (R6.1). + */ +export function normalizeDate(raw: string): string | null { + const s = raw.trim() + + let m = /^(\d{4})-(\d{1,2})-(\d{1,2})$/.exec(s) + if (m) { + const [y, mo, d] = [Number(m[1]), Number(m[2]), Number(m[3])] + return isRealDate(y, mo, d) ? `${y}-${pad(mo)}-${pad(d)}` : null + } + + m = /^(\d{1,2})\/(\d{1,2})\/(\d{4})$/.exec(s) + if (m) { + const a = Number(m[1]); const b = Number(m[2]); const y = Number(m[3]) + // a/b ambiguous: if a > 12 it must be the day (DD/MM), else assume MM/DD (US). + const [mo, d] = a > 12 ? [b, a] : [a, b] + return isRealDate(y, mo, d) ? `${y}-${pad(mo)}-${pad(d)}` : null + } + + m = /^(\d{1,2})\s+([A-Za-z]{3,})\s+(\d{4})$/.exec(s) + if (m) { + const d = Number(m[1]); const mo = MONTHS[m[2]!.slice(0, 3).toLowerCase()]; const y = Number(m[3]) + return mo && isRealDate(y, mo, d) ? `${y}-${pad(mo)}-${pad(d)}` : null + } + + return null +} + +/** A single transaction object as returned by the model (pre-validation). */ +interface RawTransaction { + date?: unknown + payee?: unknown + amount?: unknown + direction?: unknown + suggestedAccount?: unknown + suggestedEnvelope?: unknown + sourceRow?: unknown +} + +function asString(v: unknown): string { + return typeof v === 'string' ? v : '' +} + +/** + * Re-validate and enrich the model's transactions into ImportProposals. + * + * Pure: no I/O. Rejected rows go to `droppedRows` with a reason. The model's + * account/envelope suggestions are kept only if they match a real target. + */ +export function normalizeProposals( + raw: unknown, + context: { accounts: string[]; envelopes: string[] }, + journalHashes: Set, +): { proposals: ImportProposal[]; droppedRows: DroppedRow[] } { + const accountSet = new Set(context.accounts) + const envelopeSet = new Set(context.envelopes) + const list: RawTransaction[] = Array.isArray((raw as { transactions?: unknown })?.transactions) + ? (raw as { transactions: RawTransaction[] }).transactions + : [] + + const proposals: ImportProposal[] = [] + const droppedRows: DroppedRow[] = [] + + list.forEach((row, i) => { + const sourceRow = asString(row.sourceRow) + + const date = normalizeDate(asString(row.date)) + if (!date) { + droppedRows.push({ sourceRow, reason: `Unrecognized date: "${asString(row.date)}"` }) + return + } + + const amountNum = typeof row.amount === 'number' ? row.amount : Number(asString(row.amount)) + const amount = Math.abs(amountNum) + if (!Number.isFinite(amount) || amount <= 0) { + droppedRows.push({ sourceRow, reason: `Invalid amount: "${asString(row.amount)}"` }) + return + } + + const direction = row.direction === 'inflow' || row.direction === 'outflow' + ? (row.direction as ImportDirection) + : null + if (!direction) { + droppedRows.push({ sourceRow, reason: `Invalid direction: "${asString(row.direction)}"` }) + return + } + + const suggestedAccountRaw = asString(row.suggestedAccount).trim() + const suggestedAccount = accountSet.has(suggestedAccountRaw) ? suggestedAccountRaw : '' + + const envKey = asString(row.suggestedEnvelope).trim().replace(/^expenses:/, '') + const suggestedEnvelope = envelopeSet.has(envKey) ? envKey : '' + + const payee = asString(row.payee).trim() + const dedupHash = computeDedupHash({ date, amount, payee }) + + proposals.push({ + id: String(i), + date, + payee, + amount, + direction, + suggestedAccount, + suggestedEnvelope, + dedupHash, + possibleDuplicate: journalHashes.has(dedupHash), + sourceRow, + }) + }) + + return { proposals, droppedRows } +} diff --git a/server/utils/transactionWriter.ts b/server/utils/transactionWriter.ts new file mode 100644 index 0000000..4519174 --- /dev/null +++ b/server/utils/transactionWriter.ts @@ -0,0 +1,79 @@ +import type { TransactionInput } from '../../types/api' +import type { SimplifiedTransactionInput } from '../../types/ui' +import { toTransactionInput } from '../../utils/toTransactionInput' +import { appendTransaction } from './journalWriter' + +/** + * Shared simplified-transaction write path (Issue #9 extraction). + * + * The envelope-aware posting logic used to live inline in + * `server/api/transactions.post.ts`. It is extracted here so both that route and + * the CSV import commit route (`server/api/import/commit.post.ts`) write through + * exactly the same accounting path — no envelope math duplicated across routes + * (separation-of-concerns). Behaviour is unchanged from the original route. + * + * `resolveBudgetBase` is a Nitro auto-imported server util (no import needed), + * matching the original inline code. + */ + +/** + * Post-process a simplified expense to use envelope budget sub-accounts. + * - Asset accounts: 2-posting (expense debit, budget sub-account credit). + * - Liability accounts: 4-posting (expense debit, budget sub-account credit, + * pending credit-card budget debit, liability credit). + * Other transaction types / accounts pass through unchanged. + */ +async function applyEnvelopePostings( + txInput: TransactionInput, + body: SimplifiedTransactionInput, +): Promise { + if (body.type !== 'expense' || !body.category) { + return txInput + } + + const commodity = body.commodity ?? '$' + const envelopeCategory = body.category.replace(/^expenses:/, '') + + if (body.account.startsWith('liabilities:')) { + // Credit card expense: 4-posting structure. Derive the budget base from the + // journal (Issue #4 item 3) rather than hardcoding `assets:checking`, so a + // non-default primary account routes its envelope postings correctly. + const budgetBase = await resolveBudgetBase() + const liabilityName = body.account.replace(/^liabilities:/, '') + return { + ...txInput, + postings: [ + { account: body.category, amount: body.amount, commodity }, + { account: `${budgetBase}:budget:${envelopeCategory}`, amount: -body.amount, commodity }, + { account: `${budgetBase}:budget:pending:${liabilityName}`, amount: body.amount, commodity }, + { account: body.account, amount: -body.amount, commodity }, + ], + } + } + + if (body.account.startsWith('assets:')) { + // Regular expense: debit expense, credit budget sub-account + return { + ...txInput, + postings: [ + { account: body.category, amount: body.amount, commodity }, + { account: `${body.account}:budget:${envelopeCategory}`, amount: -body.amount, commodity }, + ], + } + } + + return txInput +} + +/** + * Convert a SimplifiedTransactionInput to balanced postings (applying envelope + * sub-accounts for expenses) and append it to the active journal. + * + * @throws Error (from `appendTransaction`) with joined validation messages if the + * resulting transaction is invalid; the journal is never modified on failure. + */ +export async function appendSimplifiedTransaction(body: SimplifiedTransactionInput): Promise { + let txInput = toTransactionInput(body) + txInput = await applyEnvelopePostings(txInput, body) + await appendTransaction(txInput) +} diff --git a/types/import.ts b/types/import.ts new file mode 100644 index 0000000..8c292e6 --- /dev/null +++ b/types/import.ts @@ -0,0 +1,75 @@ +// Wire + UI types for AI-assisted CSV transaction import (Issue #9). +// +// Flow: upload CSV → POST /api/import/parse returns ImportProposal[] (the AI's +// normalized mapping) → user reviews/edits in a staging table → POST /api/import/commit +// writes only the approved CommitRow[] via the shared journal writer. The parse +// route never writes; the commit route is the only write path. + +/** Direction of money relative to the chosen real account. */ +export type ImportDirection = 'inflow' | 'outflow' + +/** + * A normalized transaction the AI proposed from one CSV row. The model returns + * date, payee, amount, direction, suggestedAccount, suggestedEnvelope, and the + * verbatim sourceRow; the server enriches each row with id/dedupHash/ + * possibleDuplicate. + */ +export interface ImportProposal { + /** Stable per-row id (index-based) used as the review-table key. */ + id: string + /** Validated YYYY-MM-DD. */ + date: string + payee: string + /** Positive magnitude; sign is carried by `direction`, never here. */ + amount: number + direction: ImportDirection + /** Real account path (assets:/liabilities:), or '' if no confident match. */ + suggestedAccount: string + /** Expense category key (e.g. "food:groceries"), or '' if uncategorized. */ + suggestedEnvelope: string + /** sha256(date|cents|payeeLowercased) — see server/utils/importDedup. */ + dedupHash: string + /** True when this hash already exists in the journal at parse time. */ + possibleDuplicate: boolean + /** The original CSV line, shown so the user can verify the mapping. */ + sourceRow: string +} + +/** A CSV row the model (or our normalizer) could not turn into a proposal. */ +export interface DroppedRow { + sourceRow: string + reason: string +} + +export interface ImportParseResponse { + proposals: ImportProposal[] + /** Valid dropdown options for the review table. */ + context: { accounts: string[]; envelopes: string[] } + /** Rows that couldn't be parsed — surfaced, never silently dropped (R1.4). */ + droppedRows: DroppedRow[] +} + +/** A row the user approved (and possibly edited) in the review table. */ +export interface CommitRow { + date: string + payee: string + amount: number + direction: ImportDirection + /** Chosen real account path. */ + account: string + /** Chosen expense category; '' is allowed only for an inflow (→ Ready to Assign). */ + envelope: string + dedupHash: string +} + +export interface ImportCommitRequest { + rows: CommitRow[] +} + +export interface ImportCommitResponse { + committed: number + /** Rows skipped because their hash already exists in the journal (R5.3). */ + skippedDuplicates: CommitRow[] + /** Rows that failed server-side validation, with the reason. */ + failed: { row: CommitRow; error: string }[] +}