diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 00000000..3b20c24e --- /dev/null +++ b/docs/README.md @@ -0,0 +1,104 @@ +# Fistful Server Documentation + +Fistful is an Erlang/OTP wallet‑processing service built on top of [Vality](https://github.com/valitydev)'s +event‑sourced state‑machine platform (`machinery` + `progressor`). It persists every domain +entity — sources, destinations, deposits, withdrawals, withdrawal sessions — as an ordered +event log in PostgreSQL, exposes a Thrift/Woody RPC API over HTTP port `8022`, and orchestrates +the money‑movement lifecycle by talking to external Vality services (dominant / DMT for domain +config, party‑management, shumway for double‑entry accounting, limiter for turnover limits, +bender for idempotent ID generation, and pluggable withdrawal adapters). + +## Table of Contents + +| File | What it covers | +|------|----------------| +| [architecture.md](architecture.md) | Top‑level architecture, supervision tree, process model | +| [applications.md](applications.md) | The OTP umbrella apps: `ff_core`, `fistful`, `ff_transfer`, `ff_server`, `ff_validator`, `machinery_extra`, `ff_cth` | +| [domain-model.md](domain-model.md) | Core entities and their relationships: party → wallet → source/destination → deposit/withdrawal → session → adjustment | +| [state-machines.md](state-machines.md) | Each machinery namespace, the events it emits, and its state transitions | +| [withdrawal-flow.md](withdrawal-flow.md) | End‑to‑end processing of a withdrawal, activity dispatcher, adapter sessions, route exhaustion | +| [deposit-flow.md](deposit-flow.md) | End‑to‑end processing of a deposit | +| [accounting.md](accounting.md) | Postings transfers, cash flow resolution, fees, accounter integration | +| [routing.md](routing.md) | Routing rulesets, provider/terminal selection, turnover‑limit filtering | +| [adjustments.md](adjustments.md) | Adjustment change types, replay semantics, cash‑flow inversion | +| [adapter-integration.md](adapter-integration.md) | Withdrawal provider protocol, callbacks, adapter host | +| [rpc-api.md](rpc-api.md) | Woody/Thrift service surface exposed on `:8022` | +| [persistence.md](persistence.md) | Progressor backend, machinery schemas, event serialization | +| [configuration.md](configuration.md) | `sys.config`, `vm.args`, `.env`, application environment keys | +| [external-services.md](external-services.md) | DMT, party‑management, shumway, limiter, bender, validator | +| [observability.md](observability.md) | Prometheus, OpenTelemetry, scoper, health checks, internal trace endpoint | +| [development.md](development.md) | Build, test, lint, dialyze, release, docker workflow | +| [operations.md](operations.md) | Deployment model, healthchecks, repair scenarios | +| [glossary.md](glossary.md) | Terminology cheat‑sheet | + +## Top‑level architecture + +```mermaid +flowchart LR + subgraph ext[External clients / adapters] + CL[Client services] + AD[Withdrawal adapters] + end + + subgraph ff[fistful-server :8022] + direction TB + WOODY[Woody HTTP server
ff_server:init/1] + HANDLERS[Thrift handlers
ff_*_handler] + MACHS[Machines
ff_*_machine] + DOMAIN[Domain logic
ff_withdrawal, ff_deposit, ...] + ACCT[Accounting
ff_postings_transfer / ff_cash_flow] + WOODY --> HANDLERS --> MACHS --> DOMAIN + DOMAIN --> ACCT + end + + subgraph svc[Vality services] + DMT[dmt / dominant-v2] + PM[party-management] + SHUM[shumway
double-entry accounter] + LIM[limiter] + BEND[bender
ID generation] + VAL[validator-personal-data] + end + + subgraph store[State storage] + PG[(PostgreSQL
progressor tables)] + end + + CL -->|Thrift| WOODY + AD -->|ProcessCallback
/v1/ff_withdrawal_adapter_host| WOODY + DOMAIN -->|RPC| DMT + DOMAIN -->|RPC| PM + ACCT -->|Hold/Commit/Rollback| SHUM + DOMAIN -->|Hold/Commit/Rollback| LIM + HANDLERS -->|GenerateID| BEND + DOMAIN -->|ValidatePersonalData| VAL + DOMAIN -.->|Adapter RPC| AD + MACHS <-->|events, aux_state| PG +``` + +> [!NOTE] +> Clients speak Thrift‑over‑HTTP (Woody). Every entity lives in its own progressor +> namespace, persisted to PostgreSQL as a strictly ordered event stream. State is +> reconstructed in memory via `ff_machine:collapse/2`. Money movement is effected +> by `shumway` (the double‑entry accounter); fistful only records intent and +> coordinates the lifecycle. + +## Reading order + +For someone new to the codebase: + +1. Read this file, then [architecture.md](architecture.md). +2. Skim [applications.md](applications.md) to map the umbrella. +3. Read [domain-model.md](domain-model.md) for the vocabulary. +4. Follow one flow end‑to‑end: [withdrawal-flow.md](withdrawal-flow.md) is the + most complex and exercises every subsystem. +5. Drill into specific topics as needed. + +## Canonical source references + +This project is written in Erlang/OTP 27.1.2 (see [.env:5](../.env) and +[rebar.config:29](../rebar.config)). The prod release target is built via +`rebar3 as prod release` (see [Makefile:88](../Makefile)) and packaged into a +Docker image by [Dockerfile](../Dockerfile). Local test orchestration uses +[compose.yaml](../compose.yaml) plus the optional tracing overlay +[compose.tracing.yaml](../compose.tracing.yaml). diff --git a/docs/accounting.md b/docs/accounting.md new file mode 100644 index 00000000..1efadc24 --- /dev/null +++ b/docs/accounting.md @@ -0,0 +1,195 @@ +# Accounting + +Fistful records money movement **intent**; the actual double‑entry +bookkeeping lives in `shumway` (the Vality accounter). Every transfer is +modeled as a `final_cash_flow` (a list of postings) that shumway either +holds, commits, or rolls back. + +## Actors + +```mermaid +flowchart LR + WTH[ff_withdrawal] -->|compute cash flow| CF[ff_cash_flow] + DEP[ff_deposit] -->|compute cash flow| CF + ADJ[ff_adjustment] -->|replay cash flow| CF + CF -->|final_cash_flow| PT[ff_postings_transfer] + PT -->|Hold / CommitPlan /
RollbackPlan| ACC[ff_accounting] + ACC -->|Woody RPC| SHUM[(shumway)] + PARTY[ff_party
wallet_log_balance] -->|GetAccountByID| ACC +``` + +## Plan → final + +A **plan** cash flow describes where money flows symbolically. A +**final** cash flow is the plan with every plan‑account resolved to an +actual account ID and every plan‑volume resolved to a concrete cash. + +### Plan accounts + +Defined in [`ff_cash_flow:plan_account/0`](../apps/fistful/src/ff_cash_flow.erl#L73): + +| Tag | Meaning | +|-----|---------| +| `{wallet, sender_source}` | The originating wallet's source account | +| `{wallet, sender_settlement}` | The originating wallet's settlement account (debit side) | +| `{wallet, receiver_settlement}` | The receiving wallet's settlement account (credit side) | +| `{wallet, receiver_destination}` | The receiving destination's account | +| `{system, settlement}` | System settlement (platform's general ledger) | +| `{system, subagent}` | System subagent (the PI's subagent account) | +| `{provider, settlement}` | Provider's own settlement account | + +Account resolution happens in +[`ff_cash_flow:finalize/3`](../apps/fistful/src/ff_cash_flow.erl#L9) +against an `account_mapping :: #{plan_account() => account()}` the caller +builds by combining wallet accounts, PI system accounts and provider +accounts. + +### Plan volumes + +Defined in [`ff_cash_flow:plan_volume/0`](../apps/fistful/src/ff_cash_flow.erl#L25): + +- `{fixed, cash()}` — literal amount. +- `{share, {rational(), plan_constant(), rounding_method()}}` — a + fractional share of a constant (e.g. 1% of `operation_amount`), with an + explicit rounding mode (`default`, `round_half_towards_zero`, + `round_half_away_from_zero`). +- `{product, {min_of | max_of, [plan_volume()]}}` — combinator for caps + or floors. + +The constants are supplied through a `constant_mapping` (typically just +`#{operation_amount => Cash}`) and evaluated recursively by +[`ff_cash_flow:compute_volume/2`](../apps/fistful/src/ff_cash_flow.erl#L15). + +### Final posting + +```erlang +-type final_posting() :: #{ + sender := final_account(), + receiver := final_account(), + volume := cash(), + details => binary() +}. +``` + +Every posting is a transfer of `volume` from `sender` to `receiver`, +both identified by `ff_account:account()` records (account id + currency ++ realm). + +## Postings transfer lifecycle + +[`ff_postings_transfer`](../apps/fistful/src/ff_postings_transfer.erl) wraps +the shumway RPCs into a four‑state lifecycle: + +```mermaid +stateDiagram-v2 + [*] --> created: create/2 + validate accounts + created --> prepared: prepare/1 → shumway Hold + prepared --> committed: commit/1 → shumway CommitPlan + prepared --> cancelled: cancel/1 → shumway RollbackPlan + committed --> [*] + cancelled --> [*] +``` + +### `create/2` + +[ff_postings_transfer.erl:77](../apps/fistful/src/ff_postings_transfer.erl#L77) + +Validates before emitting any events: + +- **Non‑empty** — empty posting list → `{error, empty}`. +- **Currency** — every posting's accounts share the same currency + (`valid = validate_currencies(...)`). +- **Realm** — every posting stays within one realm + (`valid = validate_realms(...)`). +- **Accessibility** — every referenced account is accessible + (`accessible = validate_accessible(...)`). + +Emits `[{created, Transfer}, {status_changed, created}]`. + +### `prepare/1` / `commit/1` / `cancel/1` + +Each emits a single `{status_changed, _}` event. The actual I/O is in +[`ff_accounting`](../apps/fistful/src/ff_accounting.erl): + +| Erlang call | Shumway RPC | +|-------------|-------------| +| `ff_accounting:prepare_trx/2` | `Accounter.Hold(PlanChange)` | +| `ff_accounting:commit_trx/2` | `Accounter.CommitPlan(PlanChange)` | +| `ff_accounting:cancel_trx/2` | `Accounter.RollbackPlan(PlanChange)` | +| `ff_accounting:balance/1,2` | `Accounter.GetAccountByID(id)` | +| `ff_accounting:create_account/2` | `Accounter.CreateAccount(proto)` | + +See [ff_accounting.erl:25‑32](../apps/fistful/src/ff_accounting.erl#L25). + +> [!TIP] +> Hold → Commit/Rollback gives two‑phase commit semantics. If the system +> crashes between Hold and Commit, the hold survives; on the next +> `process_timeout` the machine walks the activity chain again, re‑derives +> the same plan ID, and shumway's idempotency turns the second Hold into +> a no‑op before the Commit is issued. + +## Account creation + +Wallet settlement accounts are not created by fistful; they come with the +wallet configuration in `party-management`. Fistful *uses* them via +[`ff_party:get_wallet_account/1`](../apps/fistful/src/ff_party.erl#L64) and +[`ff_party:build_account_for_wallet/2`](../apps/fistful/src/ff_party.erl#L62). + +System and provider accounts are configured in DMT (domain config) at the +payment‑institution level and fetched via +[`ff_payment_institution:system_accounts/2`](../apps/fistful/src/ff_payment_institution.erl). + +## Balances + +[`ff_accounting:balance/1`](../apps/fistful/src/ff_accounting.erl#L35) +returns `{ok, {ff_indef:indef(amount()), currency_id()}}`. `indef` is a +three‑value arithmetic from +[`ff_indef`](../apps/ff_core/src/ff_indef.erl) that captures `own_amount`, +`min_available`, `max_available` — shumway reports these three figures so +callers can reason about funds in flight (held but not committed). + +The wallet's current balance is also periodically logged out of fistful +via +[`ff_party:wallet_log_balance/2`](../apps/fistful/src/ff_party.erl#L63) +after each successful withdrawal commit. + +## Fees + +A provider/terminal's **cash flow plan** is expressed as a pair of fee +plans plus the base postings. Fistful gathers those via +[`ff_party:compute_provider_terminal_terms/4`](../apps/fistful/src/ff_party.erl#L76) +and applies them with +[`ff_cash_flow:add_fee/2`](../apps/fistful/src/ff_cash_flow.erl#L10) +before finalizing. See +[`ff_fees_plan`](../apps/fistful/src/ff_fees_plan.erl) and +[`ff_fees_final`](../apps/fistful/src/ff_fees_final.erl). + +## Idempotence of the posting plan + +The plan ID (`ff_accounting:id()`) is constructed deterministically from +the withdrawal/deposit ID, the route, and the iteration — see +[`ff_withdrawal:construct_p_transfer_id/1`](../apps/ff_transfer/src/ff_withdrawal.erl#L962) +for the withdrawal case. Shumway uses this ID to dedupe, so even when +progressor retries `process_timeout` after a crash mid‑operation, the net +effect on balances is identical to a single successful walk. + +## Cash flow inversion (for adjustments) + +[`ff_cash_flow:inverse/1`](../apps/fistful/src/ff_cash_flow.erl#L12) +produces the mirror image of a final cash flow — every posting's sender +and receiver are swapped. Adjustments use this to undo a previously +committed cash flow before applying a new one. See +[adjustments.md](adjustments.md). + +## Local limits vs external limiter + +Two separate mechanisms: + +| Kind | Where | Scope | How | +|------|-------|-------|-----| +| Wallet cash‑range limits | [`ff_party:validate_wallet_limits/2`](../apps/fistful/src/ff_party.erl#L72) | Wallet‑local balance bounds | One‑shot check against shumway balance | +| Turnover limits | [`ff_limiter`](../apps/ff_transfer/src/ff_limiter.erl) + external `limiter` service | Per provider / terminal / wallet, over a timespan | Reserve‑then‑commit; idempotent | +| Process‑internal limits | [`ff_limit`](../apps/fistful/src/ff_limit.erl) | Local machinery for counting (unused in prod paths) | Machinery namespace per limit | + +The external `limiter` service is the load‑bearing one for withdrawal +path. See [external-services.md](external-services.md). diff --git a/docs/adapter-integration.md b/docs/adapter-integration.md new file mode 100644 index 00000000..9f6bbe78 --- /dev/null +++ b/docs/adapter-integration.md @@ -0,0 +1,195 @@ +# Adapter Integration + +Withdrawal providers (the external systems that actually move money to +bank cards, wallets, crypto, etc.) speak Thrift to fistful over the +`dmsl_wthd_provider_thrift:Adapter` and `AdapterHost` services. Fistful +calls the *Adapter* outbound; the adapter calls *AdapterHost* back to +notify of asynchronous outcomes. + +## Two services + +```mermaid +sequenceDiagram + participant S as ff_withdrawal_session_machine + participant A as ff_adapter_withdrawal + participant P as Provider Adapter + participant H as ff_withdrawal_adapter_host + + S->>A: process_withdrawal(Adapter, Withdrawal, State, Opts) + A->>P: Adapter.ProcessWithdrawal(withdrawal, state, opts) + P-->>A: #wthd_provider_ProcessResult{intent, next_state, trx_info} + A-->>S: {ok, #{intent, next_state, transaction_info}} + + alt intent = {finish, Status} + S-->>S: {finished, Status} + else intent = {sleep, #{timer, callback_tag}} + S-->>S: schedule timer + register tag + Note over P,H: later ... + P->>H: AdapterHost.ProcessCallback(callback) + H->>S: process_callback(callback) + S->>A: handle_callback(Adapter, Callback, Withdrawal, State, Opts) + A->>P: Adapter.HandleCallback(callback, withdrawal, state, opts) + P-->>A: #wthd_provider_CallbackResult{intent, response, next_state, trx_info} + A-->>S: {ok, #{intent, response, next_state, transaction_info}} + S-->>H: response + H-->>P: marshal response + end +``` + +## Outbound: `ff_adapter_withdrawal` + +[`ff_adapter_withdrawal`](../apps/ff_transfer/src/ff_adapter_withdrawal.erl) is +the single client. Entry points: + +- `process_withdrawal/4` — first call and every wake‑up. +- `handle_callback/5` — when a tagged callback arrives. +- `get_quote/3` — price discovery prior to Create. + +### The withdrawal payload + +```erlang +-type withdrawal() :: #{ + id => binary(), + session_id => binary(), + resource => ff_destination:resource(), + dest_auth_data => ff_destination:auth_data(), + cash => ff_accounting:body(), + sender => ff_party:id(), + receiver => ff_party:id(), + quote => quote(), + contact_info => ff_withdrawal:contact_info() +}. +``` + +The payload is marshalled to +`dmsl_wthd_domain_thrift:'Withdrawal'` via +[`ff_adapter_withdrawal_codec`](../apps/ff_transfer/src/ff_adapter_withdrawal_codec.erl). + +### Intent types + +```erlang +-type intent() :: + {finish, finish_status()} + | {sleep, #{timer := machinery:timer(), + callback_tag => ff_withdrawal_callback:tag(), + user_interaction => user_interaction()}}. + +-type finish_status() :: success | {failed, failure()}. +``` + +`finish` ends the session. `sleep` parks the session until either: + +- The timer expires (→ `ProcessWithdrawal` is re‑invoked with the current + `AdapterState`), **or** +- A callback with the registered `callback_tag` arrives at + `ff_withdrawal_adapter_host`. + +### Transaction info + +The adapter can bind a `TransactionInfo` (provider's own trx ID, +extra metadata) to the session by returning `trx_info` in the result. +Session records it as `{transaction_bound, TrxInfo}` so later callbacks +can be reconciled. + +## Inbound: `ff_withdrawal_adapter_host` + +[`ff_withdrawal_adapter_host`](../apps/ff_server/src/ff_withdrawal_adapter_host.erl) +is a `ff_woody_wrapper`‑based handler for the +`dmsl_wthd_provider_thrift:'AdapterHost'` service. It's bound to the HTTP +path `/v1/ff_withdrawal_adapter_host` (see +[`ff_services:get_service_path(ff_withdrawal_adapter_host)`](../apps/ff_server/src/ff_services.erl#L47)). + +Single RPC: `ProcessCallback(Callback)`. The handler: + +1. Unmarshals the callback via `ff_adapter_withdrawal_codec:unmarshal/2`. +2. Dispatches to + [`ff_withdrawal_session_machine:process_callback/1`](../apps/ff_transfer/src/ff_withdrawal_session_machine.erl#L27). +3. Translates outcomes: + - `{ok, Response}` → `{succeeded, Response}`. + - `{error, {session_already_finished, Context}}` → `{finished, Context}` + (the provider asks for the current finished state, they don't + resurrect the session). + - `{error, {unknown_session, _}}` → raise `#wthd_provider_SessionNotFound{}`. + +## Tag → session lookup + +Callbacks are routed to sessions by the **tag** the session registered +when it went to sleep. The mapping is maintained by +[`ff_machine_tag`](../apps/fistful/src/ff_machine_tag.erl) (a separate +machinery namespace dedicated to tag bookkeeping) and wrapped per session +in +[`ff_withdrawal_callback_utils`](../apps/ff_transfer/src/ff_withdrawal_callback_utils.erl): + +```erlang +-opaque index() :: #{callbacks := #{tag() => callback()}}. +``` + +Helper functions: + +- `new_index/0` — empty. +- `wrap_event/2` / `unwrap_event/1` — bridge callback events into the + session event stream. +- `get_by_tag/2` — find callback state by tag. +- `process_response/2` — apply the response returned by the adapter. + +## Session‑side callback processing + +[`ff_withdrawal_session:process_callback/2`](../apps/ff_transfer/src/ff_withdrawal_session.erl#L20): + +1. Look up callback by tag. +2. Emit `{callback, {status_changed, pending}}` (via + `ff_withdrawal_callback_utils`). +3. Invoke `ff_adapter_withdrawal:handle_callback/5` with the current + adapter state. +4. Apply the returned `intent` exactly as for `ProcessWithdrawal`. +5. Emit `{callback, {finished, #{response => Response}}}`. +6. Return the `Response` payload to the caller (the adapter host), which + marshals it back to the provider. + +## Retries and timeouts + +Defined in [`ff_withdrawal_session_machine`](../apps/ff_transfer/src/ff_withdrawal_session_machine.erl): + +- `?SESSION_RETRY_TIME_LIMIT` — total clock time the session may spend + retrying before giving up (24 hours). +- `?MAX_SESSION_RETRY_TIMEOUT` — cap on the sleep interval between retries + (4 hours). + +Transient provider errors (e.g. HTTP 503) do not fail the session — the +adapter returns `{failed, Failure}` with a *retryable* failure reason, +configured through +[`ff_transfer.withdrawal.default_transient_errors`](../config/sys.config#L222) +and per‑party +[`ff_transfer.withdrawal.party_transient_errors`](../config/sys.config#L227). +On a retryable failure the session schedules another attempt; on a +terminal failure it emits `{finished, {failed, _}}` and notifies the +withdrawal. + +## Quote flow + +`ff_adapter_withdrawal:get_quote/3` returns: + +```erlang +-type quote() :: #{ + cash_from := cash(), + cash_to := cash(), + quote_data := term() %% opaque, provider-specific +}. +``` + +The `quote_data` is an opaque blob that the provider supplies now and +wants echoed back later during `ProcessWithdrawal` — so the quote can be +honoured at exchange. Fistful doesn't interpret it; it stores it in the +withdrawal (`ff_withdrawal:quote/1`) and forwards it verbatim. + +## Provider/adapter registry + +Providers are looked up through +[`ff_payouts_provider`](../apps/fistful/src/ff_payouts_provider.erl) and +their terminals through +[`ff_payouts_terminal`](../apps/fistful/src/ff_payouts_terminal.erl) — both +return a `ProviderConfig`/`TerminalConfig` from DMT. Fistful expects each +provider to expose an **adapter URL** and an **options map**; the adapter +URL is what +[`ff_adapter_withdrawal:get_adapter_with_opts/1,2`](../apps/ff_transfer/src/ff_withdrawal_session.erl#L23) +uses to build a Woody client. diff --git a/docs/adjustments.md b/docs/adjustments.md new file mode 100644 index 00000000..2f2f3bea --- /dev/null +++ b/docs/adjustments.md @@ -0,0 +1,175 @@ +# Adjustments + +An adjustment corrects an already‑finished deposit or withdrawal. It can +either flip the terminal status (`succeeded` ↔ `failed`) or replay the +cash flow against a fresher domain revision (e.g. because fees changed and +we want to reprice an old transfer). + +## Shape + +[`ff_adjustment:adjustment/0`](../apps/ff_transfer/src/ff_adjustment.erl): + +```erlang +-type adjustment() :: #{ + version := 1, + id := id(), + status := pending | succeeded, + created_at := ff_time:timestamp_ms(), + changes_plan := changes(), + domain_revision := ff_domain_config:revision(), + operation_timestamp := ff_time:timestamp_ms(), + external_id => id(), + p_transfer => ff_postings_transfer:transfer() | undefined +}. + +-type changes() :: #{ + new_cash_flow => #{old_cash_flow_inverted, new_cash_flow}, + new_status => #{new_status := status()}, + new_domain_revision => #{new_domain_revision := domain_revision()} +}. +``` + +## Kinds of change + +From [`ff_withdrawal:adjustment_change/0`](../apps/ff_transfer/src/ff_withdrawal.erl#L163): + +```erlang +-type adjustment_change() :: + {change_status, status()} + | {change_cash_flow, domain_revision()}. +``` + +1. **`change_status`** — override the terminal status. Valid targets are + the other terminal state (`succeeded` → `{failed, F}` or vice versa). + Attempting to change to `pending`, or to a status equal to the current + one, is rejected as + [`invalid_status_change_error`](../apps/ff_transfer/src/ff_withdrawal.erl#L178). +2. **`change_cash_flow`** — rebuild the cash flow at the supplied domain + revision and apply the delta: inverse the old one, apply the new one. + Rejected if the withdrawal is in a status that doesn't support it, or + if the withdrawal's current domain revision already equals the + requested one + ([`start_adjustment_error`](../apps/ff_transfer/src/ff_withdrawal.erl#L167)). + +Only one adjustment can be in flight at a time +(`{another_adjustment_in_progress, adjustment_id()}` otherwise). + +## Lifecycle + +```mermaid +stateDiagram-v2 + [*] --> pending: create + pending --> p_start: build changes plan + p_start --> p_prepared: ff_postings_transfer:prepare
(inverse + new flow) + p_prepared --> p_committed: ff_postings_transfer:commit + p_committed --> succeeded: apply new status / revision + succeeded --> [*] +``` + +Adjustments never emit `failed` — they always commit. The business +guards at creation time are what keep things safe. + +## Index + +[`ff_adjustment_utils`](../apps/ff_transfer/src/ff_adjustment_utils.erl) +holds every adjustment on the owning entity: + +```erlang +-opaque index() :: #{ + adjustments := #{id() => adjustment()}, + inversed_order := [id()], + active => id(), + cash_flow => final_cash_flow(), + domain_revision => domain_revision() +}. +``` + +- `active` is the ID of the adjustment currently being processed (if + any); `is_active/1` returns whether one is in flight. +- `cash_flow` is the **effective** final cash flow after all committed + adjustments. `ff_withdrawal:effective_final_cash_flow/1` + ([ff_withdrawal.erl:514](../apps/ff_transfer/src/ff_withdrawal.erl#L514)) + exposes this to the outside world. +- `domain_revision` is the effective revision (useful for auditing which + DMT snapshot was actually applied). +- Events are wrapped and unwrapped via + [`ff_adjustment_utils:wrap_event/2, unwrap_event/1`](../apps/ff_transfer/src/ff_adjustment_utils.erl) + so the parent entity can treat adjustment events as opaque. + +## Dispatch from the withdrawal machine + +```mermaid +sequenceDiagram + participant C as Client + participant H as ff_withdrawal_handler + participant M as ff_withdrawal_machine + participant W as ff_withdrawal + participant AU as ff_adjustment_utils + participant A as ff_adjustment + participant SH as shumway + + C->>H: CreateAdjustment(withdrawal_id, params) + H->>M: start_adjustment(id, params) + M->>W: do_start_adjustment/2 + W-->>M: [wrapped_adjustment_event()] + Note over M: machine persists events, returns + H->>C: adjustment state + Note over M: process_timeout + M->>W: process_transfer/1 (activity=adjustment) + W->>AU: process_adjustments/1 + AU->>A: process_transfer/1 + A->>SH: Hold (inverse old + new cash flow) + A->>SH: CommitPlan + A-->>AU: succeeded + AU-->>W: update index, emit wrapped events +``` + +Entry points: + +- Thrift: `CreateAdjustment` on the withdrawal management service → + [`ff_withdrawal_handler:handle_function_('CreateAdjustment', ...)`](../apps/ff_server/src/ff_withdrawal_handler.erl#L142). +- Erlang: [`ff_withdrawal_machine:start_adjustment/2`](../apps/ff_transfer/src/ff_withdrawal_machine.erl#L60). +- Domain: [`ff_withdrawal:start_adjustment/2`](../apps/ff_transfer/src/ff_withdrawal.erl#L494). + +During `process_timeout`, the activity dispatcher returns `adjustment` +whenever `ff_adjustment_utils:is_active/1` returns `true`. The adjustment +then walks its own mini activity chain (start → prepare → commit → finish) +embedded inside the parent machine's event stream. + +## Cash‑flow inversion detail + +When `change_cash_flow` is applied, the new final cash flow is NOT simply +the newly computed one — it's the **delta**: + +1. Start from the current effective flow. +2. Append its [inverse](../apps/fistful/src/ff_cash_flow.erl#L12) (senders + and receivers swapped, same volumes). +3. Append the newly computed flow at the fresh domain revision. + +That concatenation is the cash flow of the adjustment's own posting +transfer. Shumway sees it as a single balanced plan; net effect on the +books is "replace old with new". + +## Operation timestamp + +Every adjustment stores an `operation_timestamp` (the original +withdrawal's timestamp, *not* the adjustment's creation time). This is +the timestamp shumway and the limiter get — adjustments are a posteriori +corrections, not new operations in the current period. +[`ff_withdrawal:operation_timestamp/1`](../apps/ff_transfer/src/ff_withdrawal.erl#L669). + +## Errors + +From +[`ff_withdrawal:start_adjustment_error/0`](../apps/ff_transfer/src/ff_withdrawal.erl#L167): + +- `{invalid_withdrawal_status, status()}` — adjustment only valid on + finished withdrawals. +- `{invalid_status_change, {unavailable_status | already_has_status, _}}`. +- `{another_adjustment_in_progress, id()}`. +- `{invalid_cash_flow_change, {already_has_domain_revision, _} | + {unavailable_status, _}}`. +- `ff_adjustment:create_error()` — e.g. posting transfer rejections. + +These are mapped to Thrift exceptions in +[ff_withdrawal_handler.erl:142‑](../apps/ff_server/src/ff_withdrawal_handler.erl#L142). diff --git a/docs/applications.md b/docs/applications.md new file mode 100644 index 00000000..127e6ca2 --- /dev/null +++ b/docs/applications.md @@ -0,0 +1,159 @@ +# Applications + +The project is an OTP umbrella laid out under `apps/` (see +[rebar.config:69](../rebar.config#L69) — note that `ff_claim` and `w2w` are +listed as project dirs but are not present in this working copy). Only the +seven directories below exist; each is its own OTP application. + +```mermaid +flowchart TD + ff_core --> genlib[[genlib]] + fistful --> ff_core + fistful --> progressor[[progressor]] + fistful --> machinery[[machinery]] + fistful --> machinery_extra + fistful --> damsel[[damsel]] + fistful --> party_client[[party_client]] + fistful --> dmt_client[[dmt_client]] + fistful --> bender_client[[bender_client]] + ff_transfer --> fistful + ff_transfer --> limiter_proto[[limiter_proto]] + ff_validator --> woody[[woody]] + ff_server --> fistful + ff_server --> ff_transfer + ff_server --> ff_validator + ff_server --> fistful_proto[[fistful_proto]] + ff_server --> woody + ff_server --> erl_health[[erl_health]] + ff_cth -.test.-> ff_server +``` + +## `ff_core` + +Pure utility library with **no** Vality deps — see +[ff_core.app.src](../apps/ff_core/src/ff_core.app.src). + +| Module | Role | +|--------|------| +| [`ff_pipeline`](../apps/ff_core/src/ff_pipeline.erl) | `do/1`, `unwrap/1`, `valid/2`, `with/3` — the error‑propagation monad used everywhere | +| [`ff_maybe`](../apps/ff_core/src/ff_maybe.erl) | Option helpers | +| [`ff_map`](../apps/ff_core/src/ff_map.erl) | Map helpers | +| [`ff_range`](../apps/ff_core/src/ff_range.erl) | Inclusive/exclusive bounded ranges (used for cash ranges) | +| [`ff_time`](../apps/ff_core/src/ff_time.erl) | `timestamp_ms/0`, RFC3339 helpers | +| [`ff_indef`](../apps/ff_core/src/ff_indef.erl) | Indefinite arithmetic (min/max/avg) for balances | +| [`ff_string`](../apps/ff_core/src/ff_string.erl) | String utilities | +| [`ff_random`](../apps/ff_core/src/ff_random.erl) | Randomness helpers | +| [`ff_failure`](../apps/ff_core/src/ff_failure.erl) | Common failure representation | + +## `fistful` + +The domain core: parties, wallets, accounts, cash, cash flows, the machinery +wrapper. See [fistful.app.src](../apps/fistful/src/fistful.app.src). + +Key modules: + +| Module | Role | +|--------|------| +| [`fistful`](../apps/fistful/src/fistful.erl) | `machinery` + `machinery_backend` façade — dispatches progressor callbacks to per‑namespace handlers and installs the `ff_context` | +| [`ff_machine`](../apps/fistful/src/ff_machine.erl) | Timestamped events, `collapse/2`, `emit_event/1`, `init/process_*` wrappers | +| [`ff_context`](../apps/fistful/src/ff_context.erl) | Process‑dictionary storage of `{woody_ctx, party_client}` | +| [`ff_entity_context`](../apps/fistful/src/ff_entity_context.erl) | Arbitrary JSON‑ish metadata keyed by namespace | +| [`ff_party`](../apps/fistful/src/ff_party.erl) | Party/wallet lookup & validation via `party-management` + DMT; computes payment institutions, routing rulesets, provider/terminal terms | +| [`ff_account`](../apps/fistful/src/ff_account.erl) | Account identity, accessibility | +| [`ff_cash`](../apps/fistful/src/ff_cash.erl) | `{Amount, CurrencyID}` algebra | +| [`ff_currency`](../apps/fistful/src/ff_currency.erl) | Currency lookups | +| [`ff_cash_flow`](../apps/fistful/src/ff_cash_flow.erl) | Plan/final cash flows, volume computation, inversion | +| [`ff_postings_transfer`](../apps/fistful/src/ff_postings_transfer.erl) | Hold/commit/cancel lifecycle backed by `ff_accounting` | +| [`ff_accounting`](../apps/fistful/src/ff_accounting.erl) | RPC to `shumway` (`Hold`, `CommitPlan`, `RollbackPlan`, `CreateAccount`, `GetAccountByID`) | +| [`ff_limit`](../apps/fistful/src/ff_limit.erl) | Local limit‑tracker machinery (`account/4`, `confirm/4`, `reject/4`) — separate from the external `limiter` service | +| [`ff_fees_plan`](../apps/fistful/src/ff_fees_plan.erl), [`ff_fees_final`](../apps/fistful/src/ff_fees_final.erl) | Fee schemas and evaluation | +| [`ff_domain_config`](../apps/fistful/src/ff_domain_config.erl) | Thin wrapper around `dmt_client` | +| [`ff_payment_institution`](../apps/fistful/src/ff_payment_institution.erl) | PI lookup & realm resolution | +| [`ff_payouts_provider`](../apps/fistful/src/ff_payouts_provider.erl), [`ff_payouts_terminal`](../apps/fistful/src/ff_payouts_terminal.erl) | Provider/terminal configuration from DMT | +| [`ff_routing_rule`](../apps/fistful/src/ff_routing_rule.erl) | Routing‑ruleset evaluation with policies + prohibitions | +| [`ff_resource`](../apps/fistful/src/ff_resource.erl) | Bank card / crypto wallet / digital wallet / generic resource definitions | +| [`ff_bin_data`](../apps/fistful/src/ff_bin_data.erl) | Card BIN lookup | +| [`ff_repair`](../apps/fistful/src/ff_repair.erl) | Repair‑scenario dispatcher | +| [`ff_varset`](../apps/fistful/src/ff_varset.erl) | Damsel varset builder (used to reduce selectors) | +| [`ff_dmsl_codec`](../apps/fistful/src/ff_dmsl_codec.erl) | Marshalling between `ff_*` types and `dmsl_domain_thrift` types | +| [`ff_woody_client`](../apps/fistful/src/ff_woody_client.erl) | Outbound Woody RPC helper | +| [`ff_machine_tag`](../apps/fistful/src/ff_machine_tag.erl) | Tag helpers used to resolve session callbacks | +| [`ff_clock`](../apps/fistful/src/ff_clock.erl) | Monotonic clock / operation timestamps | +| [`hg_cash_range`](../apps/fistful/src/hg_cash_range.erl) | Cash‑range arithmetic (legacy `hg_` prefix) | + +## `ff_transfer` + +The business processes: sources, destinations, deposits, withdrawals and +their sessions, adjustments, routing, adapter and limiter integration. See +[ff_transfer.app.src](../apps/ff_transfer/src/ff_transfer.app.src). + +| Module | Role | +|--------|------| +| [`ff_source`](../apps/ff_transfer/src/ff_source.erl), [`ff_source_machine`](../apps/ff_transfer/src/ff_source_machine.erl) | Source entity + its machinery | +| [`ff_destination`](../apps/ff_transfer/src/ff_destination.erl), [`ff_destination_machine`](../apps/ff_transfer/src/ff_destination_machine.erl) | Destination entity + its machinery | +| [`ff_deposit`](../apps/ff_transfer/src/ff_deposit.erl), [`ff_deposit_machine`](../apps/ff_transfer/src/ff_deposit_machine.erl) | Deposit domain + machinery | +| [`ff_withdrawal`](../apps/ff_transfer/src/ff_withdrawal.erl), [`ff_withdrawal_machine`](../apps/ff_transfer/src/ff_withdrawal_machine.erl) | Withdrawal domain + machinery | +| [`ff_withdrawal_session`](../apps/ff_transfer/src/ff_withdrawal_session.erl), [`ff_withdrawal_session_machine`](../apps/ff_transfer/src/ff_withdrawal_session_machine.erl) | Per‑attempt adapter session + machinery | +| [`ff_withdrawal_routing`](../apps/ff_transfer/src/ff_withdrawal_routing.erl) | Prepare / gather / filter routes; commit or rollback their limits | +| [`ff_withdrawal_route_attempt_utils`](../apps/ff_transfer/src/ff_withdrawal_route_attempt_utils.erl) | Per‑attempt index of route + posting + session data, for multi‑attempt routing | +| [`ff_adjustment`](../apps/ff_transfer/src/ff_adjustment.erl), [`ff_adjustment_utils`](../apps/ff_transfer/src/ff_adjustment_utils.erl) | Adjustment entity + its index | +| [`ff_limiter`](../apps/ff_transfer/src/ff_limiter.erl) | Turnover limits via the external `limiter` service | +| [`ff_adapter`](../apps/ff_transfer/src/ff_adapter.erl), [`ff_adapter_withdrawal`](../apps/ff_transfer/src/ff_adapter_withdrawal.erl), [`ff_adapter_withdrawal_codec`](../apps/ff_transfer/src/ff_adapter_withdrawal_codec.erl) | Outbound withdrawal‑provider adapter protocol | +| [`ff_withdrawal_callback`](../apps/ff_transfer/src/ff_withdrawal_callback.erl), [`ff_withdrawal_callback_utils`](../apps/ff_transfer/src/ff_withdrawal_callback_utils.erl) | Adapter‑initiated callback bookkeeping | + +## `ff_server` + +The I/O layer: Thrift service handlers, Thrift ↔ domain codecs, machinery +schemas, repair service handlers, the Cowboy trace handler, the release +boot entrypoint. See [ff_server.app.src](../apps/ff_server/src/ff_server.app.src). + +| Module | Role | +|--------|------| +| [`ff_server`](../apps/ff_server/src/ff_server.erl) | Application + root supervisor + service wire‑up | +| [`ff_services`](../apps/ff_server/src/ff_services.erl) | The catalogue mapping service names to Thrift service + HTTP path | +| [`ff_woody_wrapper`](../apps/ff_server/src/ff_woody_wrapper.erl) | Woody‑handler adapter installing `ff_context` | +| [`ff_woody_event_handler`](../apps/ff_server/src/ff_woody_event_handler.erl) | Logstash/scoper‑compatible Woody event handler | +| `ff_*_handler` (5× entity + 3× repair + `ff_withdrawal_adapter_host`) | Thrift service handlers | +| `ff_*_codec` | Thrift ↔ domain type converters — one per entity, plus [`ff_codec`](../apps/ff_server/src/ff_codec.erl) for common types, [`ff_cash_flow_codec`](../apps/ff_server/src/ff_cash_flow_codec.erl), [`ff_p_transfer_codec`](../apps/ff_server/src/ff_p_transfer_codec.erl), [`ff_limit_check_codec`](../apps/ff_server/src/ff_limit_check_codec.erl), [`ff_withdrawal_adjustment_codec`](../apps/ff_server/src/ff_withdrawal_adjustment_codec.erl), [`ff_withdrawal_status_codec`](../apps/ff_server/src/ff_withdrawal_status_codec.erl), [`ff_deposit_status_codec`](../apps/ff_server/src/ff_deposit_status_codec.erl), [`ff_entity_context_codec`](../apps/ff_server/src/ff_entity_context_codec.erl), [`ff_msgpack_codec`](../apps/ff_server/src/ff_msgpack_codec.erl) | +| `ff_*_machinery_schema` | Event + aux_state marshalling for progressor ([`ff_withdrawal_machinery_schema`](../apps/ff_server/src/ff_withdrawal_machinery_schema.erl) etc.) | +| `ff_*_repair` | Thrift `Repair` handlers | +| [`ff_machine_handler`](../apps/ff_server/src/ff_machine_handler.erl) | Cowboy HTTP handler for `/traces/internal/…` | +| [`ff_proto_utils`](../apps/ff_server/src/ff_proto_utils.erl) | Serialize/deserialize arbitrary Thrift structs with Erlang's thrift client | + +## `ff_validator` + +A one‑module wrapper around the `validator-personal-data` Thrift service. +[`ff_validator:validate_personal_data/1`](../apps/ff_validator/src/ff_validator.erl#L18) +is used by withdrawal contact‑info validation. + +## `machinery_extra` + +Optional machinery extensions: an in‑memory `gen_server`‑based backend +([`machinery_gensrv_backend`](../apps/machinery_extra/src/machinery_gensrv_backend.erl), +[`machinery_gensrv_backend_sup`](../apps/machinery_extra/src/machinery_gensrv_backend_sup.erl)) +used by tests that don't want to pay for PostgreSQL, and a +[`machinery_time`](../apps/machinery_extra/src/machinery_time.erl) helper. + +## `ff_cth` + +Common‑test helper library (NOT included in the production release — +see the `profiles.test.project_app_dirs = "apps/*"` in +[rebar.config:115](../rebar.config#L115)). + +| Module | Role | +|--------|------| +| [`ct_helper`](../apps/ff_cth/src/ct_helper.erl) | Config slot helpers, `start_apps/1`, tracing utilities, `await/2,3` | +| [`ct_domain`](../apps/ff_cth/src/ct_domain.erl) + [include/ct_domain.hrl](../apps/ff_cth/include/ct_domain.hrl) | Macros + builders for DMT domain objects (providers, terminals, rulesets, PIs, wallet configs) | +| [`ct_domain_config`](../apps/ff_cth/src/ct_domain_config.erl) | Full test domain fixture | +| [`ct_objects`](../apps/ff_cth/src/ct_objects.erl) | Domain‑object builders | +| [`ct_payment_system`](../apps/ff_cth/src/ct_payment_system.erl) | Predefined payment systems | +| [`ct_cardstore`](../apps/ff_cth/src/ct_cardstore.erl) | In‑memory card fixture | +| [`ct_sup`](../apps/ff_cth/src/ct_sup.erl) | Test supervisor helpers | + +## What's not here (but is referenced) + +[rebar.config:69](../rebar.config#L69) lists `apps/ff_claim` and `apps/w2w` +as project apps, but neither directory exists in this working tree. The +`ff_claim_committer` service in [`ff_services`](../apps/ff_server/src/ff_services.erl#L39) +has no registered handler — it is only a type alias, not a currently +exposed endpoint. diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 00000000..b23e5b91 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,190 @@ +# Architecture + +Fistful is a single OTP release (`fistful-server`) composed of an umbrella of +applications. At runtime it boots a Woody/Thrift HTTP server on port `8022`, +loads a set of `machinery` backends, and wires them into `progressor` — +an Erlang framework that stores state‑machine histories in PostgreSQL and +drives machine processing (timeouts, calls, notifications, repairs) from +worker pools. + +## Mental model + +Every domain entity is an **event‑sourced state machine**. The authoritative +state is the ordered event log; the in‑memory model (the `Model` part of +[`ff_machine:st/1`](../apps/fistful/src/ff_machine.erl#L17)) is a fold of that +log, reconstructed each time the machine is woken. State transitions happen +as a result of three external drivers: + +1. **Calls** — synchronous RPCs from clients (`Create`, `GetQuote`, + `CreateAdjustment`, …). +2. **Timeouts** — the processor decides a machine should advance on its own + (e.g. push a withdrawal along its activity chain). +3. **Notifications** — another machine or another service reports something + interesting (e.g. `ff_withdrawal_session_machine` tells + `ff_withdrawal_machine` that the session finished). + +Money is never held or moved inside fistful itself. `shumway` (the Vality +double‑entry accounter) is the book of record; fistful describes *what* it +wants to post via `final_cash_flow` and calls `Hold` / `CommitPlan` / +`RollbackPlan` on that service. + +## Supervision tree + +```mermaid +flowchart TD + APP[ff_server application
apps/ff_server/src/ff_server.erl] --> SUP[ff_server supervisor
one_for_one] + SUP --> PC[party_client child
party_client:child_spec/2] + SUP --> WS[Woody HTTP server
woody_server:child_spec/2] + WS --> H1[ff_withdrawal_adapter_host] + WS --> H2[destination_management] + WS --> H3[source_management] + WS --> H4[withdrawal_management] + WS --> H5[withdrawal_session_management] + WS --> H6[deposit_management] + WS --> H7[withdrawal_session_repairer] + WS --> H8[withdrawal_repairer] + WS --> H9[deposit_repairer] + WS --> H10[/metrics Prometheus/] + WS --> H11[ff_machine_handler
/traces/internal/...] + WS --> H12[health readiness+liveness] +``` + +The top‑level supervisor is defined in +[ff_server.erl:50](../apps/ff_server/src/ff_server.erl#L50). It starts +exactly two children: a party‑client pool and the Woody server. The Woody +server multiplexes **all** Thrift services, the Prometheus metrics scrape +endpoint, the `ff_machine_handler` JSON trace endpoint, and the readiness / +liveness probes onto one HTTP listener. + +The handler list is built from a fixed table +[ff_server.erl:77](../apps/ff_server/src/ff_server.erl#L77) and dispatched to +its service path by [`ff_services:get_service_path/1`](../apps/ff_server/src/ff_services.erl#L46). + +## Progressor backends + +`progressor` is configured in [config/sys.config:34‑134](../config/sys.config#L34) +with seven namespaces, each pointing at a machinery processor and a +serialization schema: + +| Namespace | Handler module | Schema module | +|-----------|----------------|---------------| +| `ff/identity` | `ff_identity_machine` | `ff_identity_machinery_schema` | +| `ff/wallet_v2` | `ff_wallet_machine` | `ff_wallet_machinery_schema` | +| `ff/source_v1` | [`ff_source_machine`](../apps/ff_transfer/src/ff_source_machine.erl) | [`ff_source_machinery_schema`](../apps/ff_server/src/ff_source_machinery_schema.erl) | +| `ff/destination_v2` | [`ff_destination_machine`](../apps/ff_transfer/src/ff_destination_machine.erl) | [`ff_destination_machinery_schema`](../apps/ff_server/src/ff_destination_machinery_schema.erl) | +| `ff/deposit_v1` | [`ff_deposit_machine`](../apps/ff_transfer/src/ff_deposit_machine.erl) | [`ff_deposit_machinery_schema`](../apps/ff_server/src/ff_deposit_machinery_schema.erl) | +| `ff/withdrawal_v2` | [`ff_withdrawal_machine`](../apps/ff_transfer/src/ff_withdrawal_machine.erl) | [`ff_withdrawal_machinery_schema`](../apps/ff_server/src/ff_withdrawal_machinery_schema.erl) | +| `ff/withdrawal/session_v2` | [`ff_withdrawal_session_machine`](../apps/ff_transfer/src/ff_withdrawal_session_machine.erl) | [`ff_withdrawal_session_machinery_schema`](../apps/ff_server/src/ff_withdrawal_session_machinery_schema.erl) | + +> [!NOTE] +> The `ff/identity` and `ff/wallet_v2` handler modules are referenced in +> `sys.config` but their `.erl` sources are not present in the current +> working copy. Wallet and party configuration is instead fetched live from +> `party-management` (see [`ff_party`](../apps/fistful/src/ff_party.erl#L57)) and +> the Vality DMT. These legacy namespaces remain configured for backwards +> compatibility. + +All project namespaces share one storage pool, `default_pool`, backed by +`prg_pg_backend` pointed at PostgreSQL database `fistful` with user +`fistful` ([sys.config:17‑32](../config/sys.config#L17)). Each namespace gets +its own progressor pool with `worker_pool_size => 100`, +`process_step_timeout => 30`, and a retry policy of up to 3 attempts with +exponential backoff capped at 180 s ([sys.config:43‑53](../config/sys.config#L43)). + +The conversion from progressor's configuration maps into machinery backends +happens during `ff_server`'s supervisor init — see +[`get_namespaces_params/0`](../apps/ff_server/src/ff_server.erl#L139) and +[`contruct_backend_childspec/4`](../apps/ff_server/src/ff_server.erl#L150), +which register one backend per namespace under +`application:set_env(fistful, backends, #{...})`. Domain code later resolves +a backend with [`fistful:backend/1`](../apps/fistful/src/fistful.erl#L36). + +## Request lifecycle + +```mermaid +sequenceDiagram + autonumber + participant C as Client + participant W as woody_server + participant WR as ff_woody_wrapper + participant H as ff_*_handler + participant M as ff_*_machine + participant D as ff_* (domain) + participant PG as PostgreSQL / progressor + participant X as External services + + C->>W: POST /v1/withdrawal (Thrift Create) + W->>WR: dispatch, attach woody context + WR->>H: handle_function('Create', Args, Opts) + H->>H: ff_*_codec:unmarshal_* + H->>M: create(Params, Ctx) + M->>D: ff_withdrawal:create(Params) + D->>X: compute_payment_institution / get_wallet / ... + D-->>M: [events] + M->>PG: machinery:start(NS, ID, Args) + PG-->>M: ok + M-->>H: ok + H->>H: Get(ID) -> marshal state + H-->>WR: {ok, Thrift response} + WR-->>W: {ok, reply} + W-->>C: HTTP 200 Thrift body + Note over PG,M: Later, progressor fires process_timeout;
fistful:process_timeout walks the activity chain +``` + +The wrapping happens in +[`ff_woody_wrapper`](../apps/ff_server/src/ff_woody_wrapper.erl) — each Thrift +call is handed to `ff_woody_wrapper` with a static options map that contains +the `party_client` pool handle and the default handling timeout. The wrapper +pushes a scoper scope and a [`ff_context`](../apps/fistful/src/ff_context.erl) +with the woody context for downstream RPC calls. + +All domain modules use a single error‑propagation idiom: a `do/1` block from +[`ff_pipeline`](../apps/ff_core/src/ff_pipeline.erl) that catches thrown +exceptions and turns them into `{error, Reason}`. Individual handler clauses +then translate each `Reason` into a business Thrift exception via +`woody_error:raise(business, #fistful_*{})` — see the big case in +[`ff_withdrawal_handler`](../apps/ff_server/src/ff_withdrawal_handler.erl#L36). + +## Processing model + +There is **no Erlang gen_server per entity**. Machines are stateless: on every +event progressor reads the relevant events from PostgreSQL, hands them to +`machinery_prg_backend` → `fistful:process_*/3`, which does the work and +returns a `machinery:result`. Writes (appended events and updated +`aux_state`) land back in PostgreSQL in the same transaction. + +Concurrency is controlled by progressor: at most one worker per +`(namespace, id)` runs at a time (serialized with a row lock); worker pools +are sized per namespace (default 100, see +[sys.config:52](../config/sys.config#L52)). Timeouts and retries on errors +are also driven by progressor. + +> [!TIP] +> The pull‑based (`process_timeout`) model means a withdrawal doesn't finish +> "instantly" on Create. The create call returns as soon as the initial event +> is persisted; the entity then walks its activity chain asynchronously — +> routing, posting, session, limit check, commit, status — each iteration a +> fresh machinery step. See [state-machines.md](state-machines.md) for the +> activity dispatcher. + +## Internal endpoints + +In addition to the Thrift services, the HTTP server exposes: + +- `GET /metrics/[:registry]` — Prometheus scrape + ([ff_server.erl:119](../apps/ff_server/src/ff_server.erl#L119)). +- `GET /health/liveness`, `GET /health/readiness` — health checks from + `erl_health` driven by the `health_check_liveness` and + `health_check_readiness` maps in + [sys.config:252‑270](../config/sys.config#L252). +- `GET /traces/internal/{namespace}/:process_id` — a JSON dump of the + machine's current trace, dispatched by + [`ff_machine_handler`](../apps/ff_server/src/ff_machine_handler.erl#L7); + under the hood it calls `ff_machine:trace/2` which in turn goes through + `machinery:trace/3`. Useful for post‑mortems. + +## See also + +- [applications.md](applications.md) — what each umbrella app provides. +- [state-machines.md](state-machines.md) — namespace‑by‑namespace. +- [persistence.md](persistence.md) — how events are serialized and stored. diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 00000000..fb59de6f --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,208 @@ +# Configuration + +Fistful reads its configuration from four sources: + +1. **`.env`** — pinned tool versions, used for Dockerfile builds and + Makefile targets. See [.env](../.env). +2. **`config/sys.config`** — OTP application environment at release time. +3. **`config/vm.args`** — Erlang VM arguments. +4. **Process environment** (set by docker / the wrapping platform) — + OpenTelemetry and a few switches. + +The whole file `config/sys.config` is baked into the release (see +[rebar.config:103](../rebar.config#L103)), but is replaceable at runtime +by mounting a different `sys.config` over `releases/0.1/sys.config`. + +## `.env` + +[.env:4‑7](../.env#L4) + +``` +SERVICE_NAME=fistful-server +OTP_VERSION=27.1.2 +REBAR_VERSION=3.24 +THRIFT_VERSION=0.14.2.3 +``` + +These are interpolated by the Makefile and Dockerfile. The Thrift +version matters because every dep that ships IDL needs compatible +generated code. + +## `config/vm.args` + +Defaults for the Erlang VM (node name, cookie, distribution config). + +## `config/sys.config` + +A flat proplist of `{application, [{key, value}, ...]}` tuples. The +important groups: + +### `kernel` + +[sys.config:2‑14](../config/sys.config#L2). Configures logger: + +- Global level `info`. +- Single handler `default` writing JSON to + `/var/log/fistful-server/console.json` via + `logger_logstash_formatter`. +- `sync_mode_qlen => 20` — switches the handler into sync mode once the + queue grows beyond 20 messages (back‑pressure on log floods). + +### `epg_connector` + +[sys.config:16‑32](../config/sys.config#L16). PostgreSQL connection pools +used by `prg_pg_backend`: + +| Key | Value | +|-----|-------| +| `databases.default_db.host` | `db` | +| `databases.default_db.port` | `5432` | +| `databases.default_db.database` | `fistful` | +| `databases.default_db.username` | `fistful` | +| `databases.default_db.password` | `postgres` | +| `pools.default_pool.database` | `default_db` | +| `pools.default_pool.size` | `10` | + +### `progressor` + +[sys.config:34‑134](../config/sys.config#L34). See +[persistence.md](persistence.md) for structure. Namespaces and their +associated handler/schema modules are declared here. Each namespace is +registered as a separate progressor pool with `worker_pool_size => 100`. + +### `fistful` + +[sys.config:199‑218](../config/sys.config#L199). + +- `provider` — a map of *legacy* provider aliases (`<<"ncoeps">>`, + `<<"test">>`) to their `payment_institution_id`, `contract_template_id`, + and `contractor_level`. Only referenced in legacy identity/claim paths. +- `services` — outbound Woody endpoints: + - `accounter => "http://shumway:8022/accounter"` + - `limiter => "http://limiter:8022/v1/limiter"` + - `validator => "http://validator:8022/v1/validator_personal_data"` + - `party_config => "http://party-management:8022/v1/processing/partycfg"` + +### `ff_transfer` + +[sys.config:220‑234](../config/sys.config#L220). + +| Key | Meaning | +|-----|---------| +| `max_session_poll_timeout` | Max wait (seconds) between session adapter polls. Default 14400 (4 h). | +| `withdrawal.default_transient_errors` | Failure reason strings treated as retryable for all parties | +| `withdrawal.party_transient_errors` | Per‑party override map of failure reasons → retryable | + +### `ff_server` + +[sys.config:236‑271](../config/sys.config#L236). + +| Key | Default | Notes | +|-----|---------|-------| +| `ip` | `"::"` | Bind address (IPv6 any) | +| `port` | `8022` | HTTP listener | +| `default_woody_handling_timeout` | `30000` | ms; inherited by inbound handlers | +| `net_opts` | `[{timeout, 60000}]` | Keep‑alive timeout bumped to 60 s | +| `scoper_event_handler_options` | — | Log‑formatter size caps | +| `health_check_liveness` | `{disk, memory, service}` | 99% disk usage, 99% cgroup memory, service alive | +| `health_check_readiness` | `{dmt_client, progressor}` | DMT reachable, all progressor NS registered | + +### `dmt_client` + +[sys.config:140‑162](../config/sys.config#L140). + +| Key | Default | +|-----|---------| +| `cache_update_interval` | `5000` ms | +| `max_cache_size` | 20 elements / 50 MiB | +| `service_urls.AuthorManagement` | `http://dmt:8022/v1/domain/author` | +| `service_urls.Repository` | `http://dmt:8022/v1/domain/repository` | +| `service_urls.RepositoryClient` | `http://dmt:8022/v1/domain/repository_client` | + +### `party_client` + +[sys.config:164‑184](../config/sys.config#L164). + +| Key | Default | +|-----|---------| +| `services.party_management` | `http://party_management:8022/v1/processing/partymgmt` | +| `woody.cache_mode` | `safe` (other options: `disabled`, `aggressive`) | + +### `bender_client` + +[sys.config:186‑197](../config/sys.config#L186). + +| Key | Default | +|-----|---------| +| `services.Bender` | `http://bender:8022/v1/bender` | +| `services.Generator` | `http://bender:8022/v1/generator` | +| `deadline` | `60000` ms | + +### Miscellaneous + +- `scoper` — `{storage, scoper_storage_logger}` (logs scopes instead of + collecting them) ([sys.config:136](../config/sys.config#L136)). +- `snowflake` — `machine_id` optionally pinned + ([sys.config:273](../config/sys.config#L273)). +- `prometheus` — `{collectors, [default]}` + ([sys.config:277](../config/sys.config#L277)). +- `hackney` — `mod_metrics => woody_hackney_prometheus` + ([sys.config:281](../config/sys.config#L281)). + +## Environment variables + +### OpenTelemetry + +Set only when the tracing overlay compose file is used +([compose.tracing.yaml](../compose.tracing.yaml)): + +| Variable | Value | +|----------|-------| +| `OTEL_SERVICE_NAME` | `fistful_testrunner` (for the testrunner container) | +| `OTEL_TRACES_EXPORTER` | `otlp` | +| `OTEL_TRACES_SAMPLER` | `parentbased_always_on` (testrunner) / `parentbased_always_off` (other services) | +| `OTEL_EXPORTER_OTLP_PROTOCOL` | `http_protobuf` | +| `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://jaeger:4318` | + +The `opentelemetry`, `opentelemetry_api` and `opentelemetry_exporter` +applications are pulled in as deps +([rebar.config:47‑49](../rebar.config#L47)) and loaded into the release +as `temporary` (release layout at +[rebar.config:95](../rebar.config#L95)). + +### Docker build args + +Passed to the Dockerfile by +[Makefile](../Makefile): + +| Variable | Source | +|----------|--------| +| `OTP_VERSION`, `REBAR_VERSION`, `THRIFT_VERSION` | `.env` | +| `SERVICE_NAME` | `.env` | +| `TARGETARCH` | docker buildx | +| `USER_UID` / `USER_GID` | `1001` (default in Dockerfile) | + +## Fistful's own reads + +Code that queries its own application env (pattern: +`genlib_app:env(ff_*, Key, Default)`): + +| Location | Key | +|----------|-----| +| [`ff_server:init/1`](../apps/ff_server/src/ff_server.erl#L52) | `ff_server.{ip, port, woody_opts, default_woody_handling_timeout, health_check_*}` | +| [`fistful:backend/1`](../apps/fistful/src/fistful.erl#L36) | `fistful.backends` (set by `ff_server`) | +| [`ff_withdrawal_session_machine`](../apps/ff_transfer/src/ff_withdrawal_session_machine.erl) | `ff_transfer.max_session_poll_timeout`, `withdrawal.default_transient_errors`, etc. | + +## Putting it together + +```mermaid +flowchart TD + DOTENV[".env
pinned versions"] -->|docker build --arg| DF[Dockerfile] + DF --> REL["_build/prod/rel/
fistful-server"] + SC[config/sys.config] -->|cp| REL + VA[config/vm.args] -->|cp| REL + REL -->|foreground| RUN[[Running OTP node]] + RUN -->|application_controller| APPS[applications] + APPS -->|env| SC + ENV[process env
OTEL_*] --> RUN +``` diff --git a/docs/deposit-flow.md b/docs/deposit-flow.md new file mode 100644 index 00000000..ceb043b2 --- /dev/null +++ b/docs/deposit-flow.md @@ -0,0 +1,145 @@ +# Deposit Flow + +A deposit moves money **into** a wallet from a source. It is the +simpler counterpart of a withdrawal — no routing, no provider adapter, no +sessions, no callbacks. Just a double‑entry posting transfer with an +optional limit check. + +## Mental model + +Source → wallet transfer via shumway. The deposit machine records the +intent and drives the posting transfer to commit (or compensates with a +rollback on failure). There is no external communication beyond the +accounter (`shumway`) and, if the wallet has a configured limit, a +balance check. + +## Sequence + +```mermaid +sequenceDiagram + autonumber + participant C as Client + participant H as ff_deposit_handler + participant M as ff_deposit_machine + participant D as ff_deposit + participant SRC as ff_source_machine + participant SH as shumway + + C->>H: Create(params, ctx) + H->>D: create(params) + D->>SRC: get(source_id) + D->>D: validate currencies / realms / amount + D->>M: machinery:start [{created, Deposit}] + M-->>H: ok + H-->>C: Get(id) + + Note over M: process_timeout + M->>D: process_transfer/1 + D->>D: build final_cash_flow + D->>SH: Hold (accounter:prepare_trx) + SH-->>D: ok + D->>D: check receiver wallet limit + D->>SH: CommitPlan + SH-->>D: ok + D-->>M: [{status_changed, succeeded}] +``` + +## State transitions + +```mermaid +stateDiagram-v2 + [*] --> pending: {created, Deposit} + pending --> p_created: {p_transfer, {created, _}} + p_created --> p_prepared: {p_transfer, {status_changed, prepared}} + p_prepared --> limit_ok: {limit_check, {wallet_receiver, ok}} + p_prepared --> limit_fail: {limit_check, {wallet_receiver, {failed, _}}} + limit_ok --> p_committed: {p_transfer, {status_changed, committed}} + p_committed --> succeeded: {status_changed, succeeded} + limit_fail --> p_cancelled: {p_transfer, {status_changed, cancelled}} + p_cancelled --> failed: {status_changed, {failed, _}} + succeeded --> [*] + failed --> [*] +``` + +## Step detail + +### 1. Create + +Entry: [`ff_deposit_handler:handle_function('Create', ...)`](../apps/ff_server/src/ff_deposit_handler.erl#L30). + +[`ff_deposit:create/1`](../apps/ff_transfer/src/ff_deposit.erl) runs inside +a `ff_pipeline:do/1`: + +1. Fetch source via `ff_source_machine:get/1`. +2. Fetch party + wallet via `ff_party`. +3. Validate the wallet is accessible (not blocked/suspended). +4. Validate that source currency = wallet currency = deposit body currency. +5. Validate source and wallet realms match. +6. `ff_party:validate_deposit_creation/2` — enforces cash policies from + the wallet's terms (allowed currency, bad amount, etc.). +7. Emit `[{created, Deposit}]`. + +Possible errors — see +[`ff_deposit:create_error/0`](../apps/ff_transfer/src/ff_deposit.erl#L75): + +- `{source, notfound | unauthorized}` +- `{wallet, notfound}` +- `{party, notfound}` +- `ff_party:validate_deposit_creation_error()` — `{currency_validation, _}` + or `{bad_deposit_amount, Cash}` +- `{inconsistent_currency, {Deposit, Source, Wallet}}` +- `{realms_mismatch, {SourceRealm, WalletRealm}}` + +### 2. Posting transfer + +`process_transfer/1` dispatches to: + +1. **p_transfer_start** — `ff_deposit` builds a cash flow with postings: + - `{wallet, sender_source}` → `{wallet, receiver_settlement}` for the + principal amount. + - Optional system/subagent postings for fees (deposits typically have + none; check domain config). +2. **p_transfer_prepare** — `ff_postings_transfer:prepare/1` calls + `shumway:Hold`. +3. **limit_check** — applies only the receiver side: + `{limit_check, {wallet_receiver, ok | {failed, _}}}`. Failure here + means the wallet's balance after receiving would overflow its + configured upper bound. +4. **p_transfer_commit** or **p_transfer_cancel** — depending on limit + outcome. +5. **finish** — `{status_changed, succeeded}` or `{status_changed, {failed, _}}`. + +### 3. Negative deposits + +A deposit created with a *negative* body (an `ff_accounting:body()` whose +amount is negative) is flagged as +[`is_negative`](../apps/ff_transfer/src/ff_deposit.erl#L99). Functionally +this inverts the cash flow: money flows from wallet **back** into the +source. This is used for corrections and manual reversals without creating +a true withdrawal. + +## Thrift handler surface + +`fistful_deposit_thrift:'Management'` — four RPCs: + +| RPC | Module | Purpose | +|-----|--------|---------| +| `Create` | [`ff_deposit_handler`](../apps/ff_server/src/ff_deposit_handler.erl#L30) | Create & persist | +| `Get` | [`ff_deposit_handler`](../apps/ff_server/src/ff_deposit_handler.erl#L68) | Current state | +| `GetContext` | [`ff_deposit_handler`](../apps/ff_server/src/ff_deposit_handler.erl#L79) | Stored `ff_entity_context` | +| `GetEvents` | [`ff_deposit_handler`](../apps/ff_server/src/ff_deposit_handler.erl#L88) | Raw event stream | + +There's also `fistful_deposit_thrift:'Repairer'`:`Repair` served by +[`ff_deposit_repair`](../apps/ff_server/src/ff_deposit_repair.erl). + +## Idempotence + +The deposit ID is the machine ID — a second `Create` with the same ID is +a no‑op. The posting transfer ID is the deposit ID, so shumway +deduplicates there as well. + +> [!NOTE] +> Unlike withdrawals, deposits do **not** use the external `limiter` +> service — there are no turnover limits on receiving money into a wallet. +> Only the wallet's own cash‑range terms are enforced via +> `ff_party:validate_wallet_limits/2`. diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 00000000..25e784fd --- /dev/null +++ b/docs/development.md @@ -0,0 +1,191 @@ +# Development + +Fistful is built with `rebar3` and ships a production release via +`rebar3 as prod release`. Most day‑to‑day tasks are wrapped in the +[Makefile](../Makefile), which has two modes: + +1. **Local** — run `rebar3` directly on the host (`make compile`, + `make eunit`, `rebar3 shell`). +2. **Containerised** — `wc-%` targets run the task in a dev image; + `wdeps-%` targets bring the full dependency compose stack up first. + +## Toolchain versions + +Pinned in [.env](../.env): + +| Tool | Version | +|------|---------| +| Erlang/OTP | `27.1.2` | +| rebar3 | `3.24` | +| Thrift (Vality fork) | `0.14.2.3` | + +`rebar.config` uses `debug_info`, `warnings_as_errors`, and an extensive +warning set — treat every warning as a compile break (see +[rebar.config:2‑25](../rebar.config#L2)). + +## Project layout + +Umbrella project with apps in `apps/`. See +[applications.md](applications.md) for the app breakdown and +[rebar.config:69](../rebar.config#L69) for the list. + +``` +fistful-server/ +├── apps/ # OTP umbrella +│ ├── ff_core/ # Pure utilities +│ ├── fistful/ # Domain core +│ ├── ff_transfer/ # Transfer processes +│ ├── ff_server/ # I/O, handlers, release app +│ ├── ff_validator/ # Personal-data validator wrapper +│ ├── machinery_extra/ # Optional machinery backends +│ └── ff_cth/ # Common-test helpers +├── config/ # sys.config + vm.args +├── test/ # CT env config (DB, DMT, bender, party) +├── compose.yaml # Dev stack +├── compose.tracing.yaml # Tracing overlay +├── Dockerfile # Prod image (multi-stage) +├── Dockerfile.dev # Dev image for CI + local containerised tests +├── Makefile # Task runner +├── rebar.config # Build + deps +└── elvis.config # Lint rules (erlang-elvis) +``` + +## Common tasks + +All of these live in the [Makefile](../Makefile): + +| Target | Description | +|--------|-------------| +| `make compile` | `rebar3 compile` | +| `make xref` | Cross‑reference check | +| `make lint` | Run `rebar3 lint` (elvis rules in [elvis.config](../elvis.config)) | +| `make check-format` | `rebar3 fmt -c` (erlfmt, 120‑column) | +| `make format` | `rebar3 fmt -w` (auto‑fix) | +| `make dialyze` | `rebar3 as test dialyzer` (test profile adds `eunit`, `common_test`, `meck`, `jose` to the PLT) | +| `make static-check` | `check-format` + `lint` + `xref` + `dialyze` | +| `make eunit` | Unit tests with coverage | +| `make common-test` | All CT suites | +| `make common-test.` | One suite, optional `CT_CASE=` | +| `make test` | `eunit` + `common-test` | +| `make cover` / `make cover-report` | Coverage aggregation | +| `make release` | `rebar3 as prod release` | +| `make clean` / `make distclean` | Clean build artifacts | + +> [!TIP] +> The user CLAUDE.md convention is: after edits, run `make check` or +> `make lint`, then `rebar3 as test dialyzer`, and verify both pass +> before declaring done. There's no `make check` target — the closest +> equivalent is `make static-check`. + +## Container‑based workflow + +The `wc-*` (workspace‑container) and `wdeps-*` (workspace‑with‑deps) +prefix targets run make inside a dev image. + +| Target | What it does | +|--------|--------------| +| `make dev-image` | Build the `Dockerfile.dev` image (cached in `.image.dev`) | +| `make wc-shell` | Interactive shell inside the dev image | +| `make wc-` | e.g. `make wc-dialyze` — run `dialyze` inside the container | +| `make wdeps-shell` | `compose up` the full stack + interactive shell | +| `make wdeps-` | e.g. `make wdeps-common-test` — run a target against live dependencies | +| `make wdeps-common-test.` | Run one CT suite inside the stack | + +The containerised path is what CI uses, so reproducing a failure +locally is `make wdeps-common-test. CT_CASE=`. + +## Running the compose stack manually + +```mermaid +flowchart LR + DB[(postgres:15)] --> DMT[dmt / dominant-v2] + DB --> PM[party-management] + DB --> BN[bender] + DB --> SH[shumway] + DB --> LIM[liminator] + LIM --> LI[limiter] + DMT --> PM + SH --> PM + DB --> TR[testrunner
fistful-server dev image] + DMT --> TR + PM --> TR + LI --> TR + BN --> TR + SH --> TR +``` + +[compose.yaml](../compose.yaml) brings everything up with healthchecks +gating dependencies. `compose.yaml:14‑27` has the explicit +`depends_on` matrix — the `testrunner` waits on `db`, `dmt`, +`party-management`, `limiter`, `shumway`, and `bender`. The +[compose.tracing.yaml](../compose.tracing.yaml) overlay adds a Jaeger +all‑in‑one container and sets OpenTelemetry env vars on every service. + +## REPL + +A local shell that doesn't talk to any dependencies: + +``` +make rebar-shell # rebar3 shell (runs ff_server as a library) +``` + +For an interactive shell *inside* a running stack, use +`make wdeps-shell` — this gives you `rebar3 shell` in the `testrunner` +container with all deps reachable on the docker network. + +## Test suites + +Common Test suites live next to the app under test: + +- [apps/ff_transfer/test/](../apps/ff_transfer/test/) — domain‑level + suites (`ff_withdrawal_SUITE`, `ff_withdrawal_adjustment_SUITE`, + `ff_withdrawal_limits_SUITE`, `ff_withdrawal_routing_SUITE`, + `ff_deposit_SUITE`, `ff_source_SUITE`, `ff_destination_SUITE`, + `ff_transfer_SUITE`). +- [apps/ff_server/test/](../apps/ff_server/test/) — RPC‑level + integration (`ff_withdrawal_handler_SUITE`, + `ff_deposit_handler_SUITE`, `ff_source_handler_SUITE`, + `ff_destination_handler_SUITE`, `ff_withdrawal_session_repair_SUITE`). +- [apps/fistful/test/](../apps/fistful/test/) — unit‑level + (`ff_limit_SUITE`, `ff_routing_rule_SUITE`). + +Suites use the helpers in [apps/ff_cth](../apps/ff_cth/) — see +[ct_helper](../apps/ff_cth/src/ct_helper.erl) for `cfg/2,3`, +`start_apps/1`, `await/2,3`, and the `ct_domain` / `ct_objects` / +`ct_domain_config` fixtures for setting up a test domain state in DMT. + +> [!TIP] +> A single case: `make common-test.ff_withdrawal_SUITE CT_CASE=my_case`. +> A single suite: `make common-test.ff_withdrawal_SUITE`. + +## Release + +[Makefile:88‑89](../Makefile#L88) calls `rebar3 as prod release`. The +`prod` profile is declared in [rebar.config:81‑108](../rebar.config#L81): + +- Adds `recon` and `logger_logstash_formatter`. +- Release name `fistful-server` version `0.1`. +- Apps loaded: `runtime_tools`, `tools`, `recon`, `opentelemetry` + (temporary), `logger_logstash_formatter`, `canal` (both `load`), plus + the full OTP set (`sasl`, `prometheus*`) and `ff_server`. +- `mode => minimal` keeps the release small. +- `extended_start_script => true` enables the usual `foreground`, + `console`, `remote_console`, `ping` commands. + +The Dockerfile uses a two‑stage build with `erlang:27.1.2` for the +builder and `erlang:27.1.2-slim` for the runtime image — final image +runs as UID 1001, ENTRYPOINT starts the release in `foreground` mode, +and exposes `:8022`. + +## Linting + +- **erlfmt** — `rebar3 fmt`. Print width 120. +- **rebar3_lint** (elvis) — rules in [elvis.config](../elvis.config). +- **xref** — checks for undefined function calls, deprecated calls, + etc. ([rebar.config:52](../rebar.config#L52)). +- **dialyzer** — `unmatched_returns`, `error_handling`, `unknown` + warnings enabled on all deps + ([rebar.config:59‑67](../rebar.config#L59)). + +All of these run in CI (via `make static-check`) and are treated as +gates. diff --git a/docs/domain-model.md b/docs/domain-model.md new file mode 100644 index 00000000..e4d16669 --- /dev/null +++ b/docs/domain-model.md @@ -0,0 +1,232 @@ +# Domain Model + +Fistful orchestrates money movement between **wallets** (balances a party +holds with fistful) and external **sources**/**destinations** (bank cards, +crypto wallets, digital wallets, generic rails). The two money‑moving +primitives are **deposit** (external → wallet) and **withdrawal** (wallet → +external). A withdrawal may spawn one or more **sessions**, each +representing an attempt through a specific provider/terminal. Already +finalized transfers can be corrected through **adjustments**. + +## Entity map + +```mermaid +erDiagram + PARTY ||--o{ WALLET : "owns" + WALLET ||--o{ ACCOUNT : "holds" + PARTY ||--o{ SOURCE : "owns" + PARTY ||--o{ DESTINATION : "owns" + SOURCE ||--o{ DEPOSIT : "funds" + WALLET ||--o{ DEPOSIT : "receives" + WALLET ||--o{ WITHDRAWAL : "debits" + DESTINATION ||--o{ WITHDRAWAL : "targets" + WITHDRAWAL ||--o{ SESSION : "attempts" + WITHDRAWAL ||--o{ ADJUSTMENT : "adjusted by" + DEPOSIT ||--o{ ADJUSTMENT : "adjusted by" + WITHDRAWAL ||--|| ROUTE : "effective" + ROUTE }o--|| PROVIDER : "uses" + ROUTE }o--|| TERMINAL : "via" + PAYMENT_INSTITUTION ||--o{ ROUTING_RULESET : "owns" + ROUTING_RULESET ||--o{ ROUTE : "emits" + PROVIDER }o--|| PAYMENT_SYSTEM : "speaks" + DESTINATION ||--|| RESOURCE : "has" + RESOURCE ||--o{ BIN_DATA : "enriched by" +``` + +The `PARTY`, `WALLET`, `PAYMENT_INSTITUTION`, `PROVIDER`, `TERMINAL`, +`ROUTING_RULESET`, and `PAYMENT_SYSTEM` boxes live in external systems +(`party-management` and Vality DMT). The other entities are local to +fistful and persisted as event streams. + +## Party and wallet + +| Field | Type | Source | +|-------|------|--------| +| id (party) | [`dmsl_base_thrift:'ID'()`](../apps/fistful/src/ff_party.erl#L14) | party‑management | +| wallet | [`dmsl_domain_thrift:'WalletConfig'()`](../apps/fistful/src/ff_party.erl#L17) | party‑management | +| terms | [`dmsl_domain_thrift:'TermSet'()`](../apps/fistful/src/ff_party.erl#L17) | DMT (via party‑management) | + +Parties and wallets are **not** machines in this service — fistful fetches +them on demand through +[`ff_party:get_party/1`](../apps/fistful/src/ff_party.erl#L57) and +[`ff_party:get_wallet/2,3`](../apps/fistful/src/ff_party.erl#L60) and uses +them to compute applicable terms, allowed currencies, cash ranges, and so +on. Accessibility is checked up front; a wallet in `blocked` or `suspended` +state causes `{wallet, {inaccessible, blocked | suspended}}`. See +[`ff_party:inaccessibility/0`](../apps/fistful/src/ff_party.erl#L52). + +A wallet has a primary settlement account (retrieved via +[`ff_party:get_wallet_account/1`](../apps/fistful/src/ff_party.erl#L64)) — this +is where deposits credit and withdrawals debit. + +## Source (`ff_source`) + +An external funding entity (a bank card, a generic resource) that can +originate deposits. Persisted in namespace `ff/source_v1`. + +```erlang +-type source() :: #{ + version := pos_integer(), + id := binary(), + name := binary(), + party_id := ff_party:id(), + resource := ff_resource:resource(), + realm := ff_payment_institution:realm(), + ... +}. +``` + +Sources are lightweight — their machine only records creation and an +authorization step; there is no active processing. See +[`ff_source`](../apps/ff_transfer/src/ff_source.erl) and +[`ff_source_machine`](../apps/ff_transfer/src/ff_source_machine.erl). + +## Destination (`ff_destination`) + +An external payout target. Persisted in namespace `ff/destination_v2`. +Structurally similar to sources but has an `auth_data` slot for one‑time +push tokens when the destination is a tokenised resource (see +[`ff_destination:auth_data`](../apps/ff_transfer/src/ff_destination.erl)). + +## Resource (`ff_resource`) + +The typed value inside a source or destination. +[`ff_resource:resource/0`](../apps/fistful/src/ff_resource.erl) is a tagged +union: + +- `{bank_card, resource_bank_card()}` — includes BIN data enriched by + [`ff_bin_data`](../apps/fistful/src/ff_bin_data.erl) (payment system, + card type, issuer country). +- `{crypto_wallet, resource_crypto_wallet()}` — address + currency. +- `{digital_wallet, resource_digital_wallet()}` — e.g. a wallet at a + specific provider. +- `{generic, resource_generic()}` — opaque binary payload for provider‑ + defined schemas. + +A `resource_descriptor` is a compact reference (e.g. +`{bank_card, bin_data_id()}`) used in withdrawal quotes for replay. + +## Cash and cash flow + +- [`ff_cash:cash()`](../apps/fistful/src/ff_cash.erl) — `{Amount, CurrencyID}`, + where `Amount :: integer()` is in minor units. +- [`ff_cash_flow:plan_account/0`](../apps/fistful/src/ff_cash_flow.erl#L73) — + one of: + - `{wallet, sender_source|sender_settlement|receiver_settlement|receiver_destination}` + - `{system, settlement|subagent}` + - `{provider, settlement}` +- A **plan** cash flow ([`cash_flow_plan`](../apps/fistful/src/ff_cash_flow.erl#L46)) + is a template of postings with symbolic volumes (`{fixed, Cash}`, + `{share, {Ratio, operation_amount, Rounding}}`, `{product, min/max_of}`). +- [`ff_cash_flow:finalize/3`](../apps/fistful/src/ff_cash_flow.erl#L9) resolves + plan accounts to real account IDs and plan volumes to concrete amounts — + the resulting `final_cash_flow` is what postings transfers commit. + +## Fees + +[`ff_fees_plan`](../apps/fistful/src/ff_fees_plan.erl) is a map of plan +constants → plan volumes (`operation_amount`, `surplus`, etc.). It's +evaluated into a [`ff_fees_final`](../apps/fistful/src/ff_fees_final.erl) +against a concrete cash; the final fees are then folded into the cash flow +via [`ff_cash_flow:add_fee/2`](../apps/fistful/src/ff_cash_flow.erl#L10). + +## Payment institution, provider, terminal + +From the Vality domain (DMT). Fetched and reduced via +[`ff_payment_institution`](../apps/fistful/src/ff_payment_institution.erl), +[`ff_payouts_provider`](../apps/fistful/src/ff_payouts_provider.erl) and +[`ff_payouts_terminal`](../apps/fistful/src/ff_payouts_terminal.erl). A +payment institution has a **realm** (`test` or `live`) that's propagated to +accounts and checked against wallet/destination compatibility — see the +`{realms_mismatch, {_, _}}` errors in +[`ff_withdrawal:create_error/0`](../apps/ff_transfer/src/ff_withdrawal.erl#L93) +and [`ff_deposit:create_error/0`](../apps/ff_transfer/src/ff_deposit.erl#L75). + +## Route + +A choice of `(provider_id, terminal_id)` for a single withdrawal attempt. +Defined in [`ff_withdrawal_routing:route/0`](../apps/ff_transfer/src/ff_withdrawal_routing.erl#L22): + +```erlang +-type route() :: #{ + version := 1, + provider_id := pos_integer(), + terminal_id := pos_integer(), + provider_id_legacy => pos_integer() +}. +``` + +Routes are produced by +[`ff_withdrawal_routing:prepare_routes/3`](../apps/ff_transfer/src/ff_withdrawal_routing.erl#L77) +after +[`ff_routing_rule:gather_routes/4`](../apps/fistful/src/ff_routing_rule.erl) +evaluates the payment institution's routing rulesets (policies applied then +prohibitions subtracted). See [routing.md](routing.md) for the full story. + +## Deposit (`ff_deposit`) + +Source → wallet credit. Single posting transfer, no sessions, no routing. +See [deposit-flow.md](deposit-flow.md). + +Status progression: `pending → succeeded | {failed, failure()}` +([`ff_deposit:status/0`](../apps/ff_transfer/src/ff_deposit.erl#L52)). + +A deposit can carry a `negative_body` flag +([`ff_deposit:is_negative/1`](../apps/ff_transfer/src/ff_deposit.erl#L99)) — +used for refunds/corrections that credit the source. + +## Withdrawal (`ff_withdrawal`) + +Wallet → destination debit. Involves routing, limits, a posting transfer, +zero or more sessions, and optionally adjustments. See +[withdrawal-flow.md](withdrawal-flow.md). + +Status progression: `pending → succeeded | {failed, failure()}` +([`ff_withdrawal:status/0`](../apps/ff_transfer/src/ff_withdrawal.erl#L55)). + +## Session (`ff_withdrawal_session`) + +A single attempt to push a withdrawal through a provider. Stored in its +own namespace `ff/withdrawal/session_v2`. Carries adapter state between +invocations, handles callbacks tagged via +[`ff_machine_tag`](../apps/fistful/src/ff_machine_tag.erl), and emits a +`session_result` back to the owning withdrawal. + +```erlang +-type status() :: active | {finished, success | {failed, failure()}} +-type session_result() :: success | {success, transaction_info()} | {failed, failure()} +``` + +([`ff_withdrawal_session`](../apps/ff_transfer/src/ff_withdrawal_session.erl#L59).) + +A withdrawal may run through several sessions (one per attempt/route) — +see [`ff_withdrawal_route_attempt_utils`](../apps/ff_transfer/src/ff_withdrawal_route_attempt_utils.erl). + +## Adjustment (`ff_adjustment`) + +A corrective operation on an already‑finished withdrawal or deposit. It can +either change the status or replay the cash flow against a newer domain +revision. See [adjustments.md](adjustments.md). + +## Entity context + +Every machine has an associated `ctx :: ff_entity_context:context()` — +a map keyed by an arbitrary namespace, holding opaque JSON‑ish metadata. +Clients use this to stash their own bookkeeping (e.g. idempotency keys, +external IDs, display names) without touching fistful's own schema. See +[`ff_entity_context`](../apps/fistful/src/ff_entity_context.erl). + +## Summary of event shapes + +| Entity | Event type (first field) | File | +|--------|--------------------------|------| +| Source | `{created, source()}`, `{auth_data_changed, ...}` | [ff_source.erl](../apps/ff_transfer/src/ff_source.erl) | +| Destination | `{created, destination()}`, `{auth_data_changed, ...}` | [ff_destination.erl](../apps/ff_transfer/src/ff_destination.erl) | +| Deposit | `{created, ...}`, `{limit_check, ...}`, `{p_transfer, ...}`, `{status_changed, ...}` | [ff_deposit.erl:57](../apps/ff_transfer/src/ff_deposit.erl#L57) | +| Withdrawal | `{created, ...}`, `{resource_got, ...}`, `{route_changed, ...}`, `{p_transfer, ...}`, `{limit_check, ...}`, `{validation, ...}`, `{session_started, ...}`, `{session_finished, ...}`, `{status_changed, ...}`, `wrapped_adjustment_event()` | [ff_withdrawal.erl:81](../apps/ff_transfer/src/ff_withdrawal.erl#L81) | +| Session | `{created, ...}`, `{next_state, ...}`, `{transaction_bound, ...}`, `{finished, ...}`, `wrapped_callback_event()` | [ff_withdrawal_session.erl:63](../apps/ff_transfer/src/ff_withdrawal_session.erl#L63) | +| Adjustment | produced inside `wrapped_adjustment_event()` | [ff_adjustment.erl](../apps/ff_transfer/src/ff_adjustment.erl) | + +Every event is actually wrapped in a `{ev, Timestamp, Event}` tuple by +[`ff_machine:emit_event/1`](../apps/fistful/src/ff_machine.erl#L63) before +it hits the log. diff --git a/docs/external-services.md b/docs/external-services.md new file mode 100644 index 00000000..4e626da0 --- /dev/null +++ b/docs/external-services.md @@ -0,0 +1,184 @@ +# External Services + +Fistful is a coordinator, not a source of truth. It depends on six +external services — all Vality open‑source components reachable over +Woody/Thrift — plus pluggable **provider adapters** for actual money +movement. + +```mermaid +flowchart LR + FF[fistful-server] -->|"domain config
(dmt_client)"| DMT[dmt / dominant-v2] + FF -->|"party + wallet
(party_client)"| PM[party-management] + FF -->|"Hold/Commit/Rollback
(ff_accounting)"| SH[shumway] + PM -->|inherits domain from| DMT + PM -->|"accounts via"| SH + FF -->|"turnover limits
(ff_limiter)"| LI[limiter] + LI -->|underlying store| LIM[liminator] + FF -->|"GenerateID
(bender_client)"| BN[bender] + FF -->|"ValidatePersonalData
(ff_validator)"| VL[validator-personal-data] + FF -.->|ProcessWithdrawal
HandleCallback
GetQuote| AD[(Provider adapters)] + AD -->|ProcessCallback| FF + FF --> PG[(PostgreSQL)] + LIM --> PG + SH --> PG + PM --> PG + DMT --> PG + BN --> PG +``` + +## DMT — Vality domain config + +- **Repo**: [dominant-v2](https://github.com/valitydev/dominant-v2). +- **Thrift service**: `dmsl_domain_conf_v2_thrift:'Repository'`, + `'RepositoryClient'`, `'AuthorManagement'`. +- **Erlang client**: `dmt_client` (bundled into the release at + [rebar.config:39](../rebar.config#L39)). +- **Endpoints used**: configured in + [sys.config:157‑161](../config/sys.config#L157) — + `AuthorManagement`, `Repository`, `RepositoryClient` all at + `http://dmt:8022/v1/domain/...`. + +DMT holds every domain object fistful reads: + +- Payment institutions (`PaymentInstitution`, `PaymentInstitutionRef`). +- Routing rulesets (`RoutingRuleset`). +- Providers (`Provider`, `ProviderRef`) with their `WithdrawalProvisionTerms`. +- Terminals (`Terminal`, `TerminalRef`). +- Currencies, payment systems, payment methods. +- Wallet configs (`WalletConfig`) — bridged through party‑management. + +Accessed via [`ff_domain_config:object/1,2`](../apps/fistful/src/ff_domain_config.erl) +(thin wrapper around `dmt_client`) and revision helpers +[`head/0`](../apps/fistful/src/ff_domain_config.erl). + +The `dmt_client` health check is bound into readiness at +[sys.config:258](../config/sys.config#L258). + +## party-management + +- **Repo**: [party-management](https://github.com/valitydev/party-management). +- **Thrift service**: `dmsl_payproc_thrift:'PartyManagement'`. +- **Erlang client**: `party_client` ([rebar.config:42](../rebar.config#L42)) + using a `safe` cache mode + ([sys.config:170](../config/sys.config#L170)). +- **Endpoint**: `http://party_management:8022/v1/processing/partymgmt` + ([sys.config:166](../config/sys.config#L166)). + +Authoritative for every party and wallet. Fistful fetches: + +- Party existence and revision + ([`ff_party:get_party/1`](../apps/fistful/src/ff_party.erl#L57)). +- Wallet configs ([`ff_party:get_wallet/2,3`](../apps/fistful/src/ff_party.erl#L60)). +- Effective term sets ([`ff_party:get_terms/3`](../apps/fistful/src/ff_party.erl#L73)). +- Computed routing rulesets + ([`ff_party:compute_routing_ruleset/3`](../apps/fistful/src/ff_party.erl#L75)). +- Computed provider/terminal terms + ([`ff_party:compute_provider_terminal_terms/4`](../apps/fistful/src/ff_party.erl#L76)). + +## shumway — double‑entry accounter + +- **Repo**: [shumway](https://github.com/valitydev/shumway) (Spring Boot, + Java). +- **Thrift service**: `dmsl_accounter_thrift:'Accounter'`. +- **Endpoint**: `http://shumway:8022/accounter` + ([sys.config:213](../config/sys.config#L213)). + +Called by [`ff_accounting`](../apps/fistful/src/ff_accounting.erl): + +| RPC | Purpose | +|-----|---------| +| `CreateAccount` | Called if an account doesn't exist (rare — accounts are usually provisioned by party‑management) | +| `GetAccountByID` | Balance lookups | +| `Hold` | Reserve funds (prepare a posting plan) | +| `CommitPlan` | Apply a previously held plan | +| `RollbackPlan` | Release a held plan | + +Idempotency is keyed by the plan ID (`ff_accounting:id()`) fistful +supplies — identical IDs across retries are deduped by shumway. + +## limiter + liminator — turnover limits + +- **Repos**: [limiter](https://github.com/valitydev/limiter), + [liminator](https://github.com/valitydev/liminator) (underlying + counter store). +- **Thrift service**: `limproto_limiter_thrift:'LimiterService'` (proto + at `limiter_proto`). +- **Endpoint**: `http://limiter:8022/v1/limiter` + ([sys.config:214](../config/sys.config#L214)). + +Called by [`ff_limiter`](../apps/ff_transfer/src/ff_limiter.erl): + +| Operation | RPC | Fistful entry | +|-----------|-----|---------------| +| Read limit values | `GetValues` | [`check_limits/4`](../apps/ff_transfer/src/ff_limiter.erl#L39) | +| Hold capacity | `Hold` | [`hold_withdrawal_limits/4`](../apps/ff_transfer/src/ff_limiter.erl#L27) | +| Commit capacity | `Commit` | [`commit_withdrawal_limits/4`](../apps/ff_transfer/src/ff_limiter.erl#L28) | +| Rollback | `Rollback` | [`rollback_withdrawal_limits/4`](../apps/ff_transfer/src/ff_limiter.erl#L29) | + +Used only on the withdrawal path. + +## bender — ID generation + +- **Repo**: [bender](https://github.com/valitydev/bender). +- **Thrift services**: `Bender` and `Generator` (separate schemas). +- **Erlang client**: `bender_client` + ([rebar.config:43](../rebar.config#L43)). +- **Endpoints**: + - `http://bender:8022/v1/bender` ([sys.config:188](../config/sys.config#L188)) + - `http://bender:8022/v1/generator` ([sys.config:189](../config/sys.config#L189)) + +bender gives fistful **idempotent ID generation**: given an external +correlation key, it returns a stable internal ID. Used by upstream APIs +(e.g. wAPI / anapi) more than by fistful itself; from fistful's +perspective IDs are handed in by the client as part of each `Create` +payload. + +## validator-personal-data + +- **Repo**: + [validator-personal-data](https://github.com/valitydev/validator-personal-data). +- **Thrift service**: + `validator_personal_data_validator_personal_data_thrift:'ValidatorPersonalDataService'`. +- **Endpoint**: `http://validator:8022/v1/validator_personal_data` + ([sys.config:215](../config/sys.config#L215)). + +Called by [`ff_validator:validate_personal_data/1`](../apps/ff_validator/src/ff_validator.erl#L18) +— the withdrawal sender/receiver contact info is checked against a +personal‑data token. Result is stored on the withdrawal as +`{validation, {sender | receiver, {personal, ...}}}` event. + +## Provider adapters (pluggable) + +Adapters are *not* part of the fistful release — they are separate +processes that fistful calls outbound for each withdrawal session. + +- **Thrift services**: `dmsl_wthd_provider_thrift:'Adapter'` (outbound, + fistful → adapter), `'AdapterHost'` (inbound, adapter → fistful). +- **Endpoint**: per‑provider, configured in DMT as part of each + `Provider` object's proxy definition. +- **Fistful side**: + [`ff_adapter_withdrawal`](../apps/ff_transfer/src/ff_adapter_withdrawal.erl) + for outbound calls, + [`ff_withdrawal_adapter_host`](../apps/ff_server/src/ff_withdrawal_adapter_host.erl) + at `/v1/ff_withdrawal_adapter_host` for inbound callbacks. + +See [adapter-integration.md](adapter-integration.md) for the protocol. + +## Shared database + +Every service named above (except the adapters) runs against the same +PostgreSQL instance in the compose stack; each owns a distinct database. +[compose.yaml:134](../compose.yaml#L134): + +``` +POSTGRES_MULTIPLE_DATABASES: "fistful,bender,dmt,party_management,shumway,liminator" +``` + +The init scripts under +[test/postgres/docker-entrypoint-initdb.d/](../test/postgres/) create +each user and database. + +> [!NOTE] +> In production deployments these services typically have their own +> managed PostgreSQL instances — the single‑db compose file is a local +> convenience only. diff --git a/docs/glossary.md b/docs/glossary.md new file mode 100644 index 00000000..d26f0a13 --- /dev/null +++ b/docs/glossary.md @@ -0,0 +1,185 @@ +# Glossary + +A quick reference for terms that recur across the codebase and docs. +Most come either from the banking/payments domain, from Vality's +platform vocabulary, or from the machinery/progressor runtime. + +## Domain terms + +**Party** — an owner of wallets, sources, and destinations. Lives in +`party-management`; fistful only reads. See +[`ff_party`](../apps/fistful/src/ff_party.erl). + +**Wallet** — a balance a party holds within Vality. One currency per +wallet, one primary settlement account per wallet. Lives in +`party-management` and DMT. + +**Realm** — `test | live`. Attached to each payment institution and +propagated to accounts. Withdrawal and deposit creation fail with +`{realms_mismatch, _}` if the wallet and destination/source realms +don't match. +[`ff_payment_institution:realm/0`](../apps/fistful/src/ff_payment_institution.erl). + +**Source** — external funding, e.g. a card or generic rail that can +credit a wallet via a deposit. Fistful namespace `ff/source_v1`. + +**Destination** — external payout target. Fistful namespace +`ff/destination_v2`. Carries a `resource` and optional `auth_data`. + +**Resource** — typed payload inside a source/destination: +`{bank_card, …}`, `{crypto_wallet, …}`, `{digital_wallet, …}`, +`{generic, …}`. See [`ff_resource`](../apps/fistful/src/ff_resource.erl). + +**Deposit** — external source → wallet credit. Single posting +transfer, no routing. Namespace `ff/deposit_v1`. + +**Withdrawal** — wallet → external destination debit. Involves +routing, posting transfer, sessions, and possibly adjustments. +Namespace `ff/withdrawal_v2`. + +**Session** — a single attempt to push a withdrawal through a +provider. Namespace `ff/withdrawal/session_v2`. Can sleep waiting for +a provider callback. + +**Adjustment** — a corrective operation on a finished deposit or +withdrawal. Can change status or replay cash flow at a new domain +revision. See [`ff_adjustment`](../apps/ff_transfer/src/ff_adjustment.erl). + +**Quote** — a provider‑supplied price for a withdrawal, valid for a +limited time and redeemable by passing `quote_data` back into `Create`. + +**Route** — `(provider_id, terminal_id)` pair chosen for a withdrawal +attempt. See [routing.md](routing.md). + +**Payment institution (PI)** — a DMT object that groups providers, +routing rulesets, system accounts, and payment methods. Every wallet +is tied to a PI via its terms. + +**Provider / terminal** — the concrete payout channel (acquirer / +terminal). Each terminal has its own `WithdrawalProvisionTerms` (cash +range, supported methods, fees). + +**Routing ruleset** — a DMT object encoding selection logic as +selectors reduced against a varset. A PI has two: policies (positive) +and prohibitions (negative). + +**Varset** — the set of values against which selectors are reduced +during routing (currency, cash, method, party ID, wallet ID, etc.). +Built via [`ff_varset`](../apps/fistful/src/ff_varset.erl). + +## Accounting terms + +**Cash** — `{Amount :: integer(), CurrencyID}`, amount in minor units. + +**Cash range** — inclusive/exclusive bounded range of cash values. Used +for wallet balance limits and per‑route cash caps. + +**Plan cash flow** — a template of postings with symbolic accounts +(`{wallet, sender_source}` etc.) and symbolic volumes (`{fixed, _}`, +`{share, _}`, `{product, _}`). + +**Final cash flow** — plan cash flow with concrete account IDs and +concrete amounts. What's sent to shumway. + +**Posting** — a single credit/debit pair: `{Sender, Receiver, Volume}`. + +**Posting transfer** — a named set of postings with a state machine +(`created → prepared → committed | cancelled`). Backed by shumway's +`Hold` / `CommitPlan` / `RollbackPlan`. + +**Plan ID / transaction ID** — the deterministic identifier fistful +supplies to shumway. Derived from withdrawal/deposit ID + route + +iteration. Used for idempotency. + +**Turnover limit** — a per‑provider/terminal/wallet rolling limit +(daily, monthly, etc.) tracked in the external `limiter` service. + +**Operation ID (limiter)** — the segmented ID fistful gives the +limiter to dedupe holds: `[provider, terminal, withdrawal, iteration?]`. +See [`ff_limiter:make_operation_segments/3`](../apps/ff_transfer/src/ff_limiter.erl#L52). + +## Machinery / platform terms + +**machinery** — Vality's state‑machine library. Provides the +`machinery:machine/2` / `machine:result/2` shapes, `dispatch_signal`, +`dispatch_call`, etc. + +**progressor** — Vality's machine runtime. Persists histories in +PostgreSQL, owns worker pools, schedules timeouts, serializes +concurrent operations on the same `(namespace, id)`. + +**machinery_prg_backend** — the progressor‑backed implementation of the +`machinery_backend` behaviour. + +**Namespace** — a unique string identifying a machine family, e.g. +`ff/withdrawal_v2`. Each namespace has its own progressor pool, handler +module, and schema module. + +**Aux state** — per‑machine blob persisted alongside the event log. +In fistful, this is always `#{ctx => ff_entity_context:context()}`. + +**Entity context** — per‑machine user‑defined metadata. Namespaced +map; see [`ff_entity_context`](../apps/fistful/src/ff_entity_context.erl). + +**Timestamped event** — `{ev, Timestamp, Change}`. Every fistful event +is wrapped this way by +[`ff_machine:emit_event/1`](../apps/fistful/src/ff_machine.erl#L63). + +**Collapse** — the fold‑over‑history that reconstructs the in‑memory +model from events. See +[`ff_machine:collapse/2`](../apps/fistful/src/ff_machine.erl#L60). + +**Repair** — a special machine operation that appends arbitrary +events to unstick a machine in a faulted state. + +## Networking / RPC terms + +**Woody** — Vality's Thrift‑over‑HTTP RPC framework. Uses normal +Thrift binary protocol but over HTTP with a specific header vocabulary +(deadline, parent ID, trace ID). + +**Thrift service** — a Thrift IDL interface, e.g. +`fistful_wthd_thrift:'Management'`. Fistful's IDL lives in the +[fistful‑proto](https://github.com/valitydev/fistful-proto) repo. + +**DMT** — Vality's *Domain Management Thing*, a versioned store of +domain configuration (payment institutions, providers, routing rules, +etc.). Exposed over Thrift; client lib is `dmt_client`. + +**Damsel** — the Thrift IDL repository where DMT's types are defined +(`dmsl_*` namespaces). Erlang dep at +[rebar.config:38](../rebar.config#L38). + +**Adapter** — an external process that speaks the +`dmsl_wthd_provider_thrift:Adapter` service. Each withdrawal provider +is an adapter. + +**Adapter host** — the opposite direction: fistful serves +`dmsl_wthd_provider_thrift:AdapterHost` so adapters can call back with +asynchronous results. + +## Error‑handling idioms + +**`do/1` block** — [`ff_pipeline:do/1`](../apps/ff_core/src/ff_pipeline.erl#L28) +wraps a function, catches thrown exceptions, returns `{ok, Value}` or +`{error, Reason}`. The universal error‑propagation idiom. + +**`unwrap/1,2`** — extract the `ok` value from a result, or throw its +error. Used inside `do/1`. + +**Business error** — a `woody_error:raise(business, #*{...})` invocation +that surfaces a typed Thrift exception to the client. Distinct from +**system errors**, which become Woody system errors (5xx‑ish). + +## Config / vocabulary + +**sys.config** — OTP application environment baked into the release. + +**vm.args** — Erlang VM flags. + +**Scoper** — Vality's structured‑logging scope tracker. Every machine +operation and every handler opens a scope; loggers see the merged +scope of the call chain. + +**Opentelemetry** — W3C tracing. Optional, enabled by +[compose.tracing.yaml](../compose.tracing.yaml). diff --git a/docs/observability.md b/docs/observability.md new file mode 100644 index 00000000..967a04bf --- /dev/null +++ b/docs/observability.md @@ -0,0 +1,186 @@ +# Observability + +Four pillars: + +- **Logs** — structured JSON via `logger_logstash_formatter`, enriched by + `scoper`. +- **Metrics** — Prometheus, exported on `GET /metrics`. +- **Traces** — OpenTelemetry, optional, enabled by the + [compose.tracing.yaml](../compose.tracing.yaml) overlay. +- **Machine introspection** — JSON event dumps over + `/traces/internal/…`. + +## Logging + +Configured in [sys.config:2‑14](../config/sys.config#L2): + +```erlang +{kernel, [ + {logger_level, info}, + {logger, [ + {handler, default, logger_std_h, #{ + level => debug, + config => #{ + type => {file, "/var/log/fistful-server/console.json"}, + sync_mode_qlen => 20 + }, + formatter => {logger_logstash_formatter, #{}} + }} + ]} +]} +``` + +Every log line is one JSON object ready for Logstash / Loki / any +JSON‑ingesting log backend. The file directory is created by the +Dockerfile +([Dockerfile:43‑46](../Dockerfile#L43)). `sync_mode_qlen => 20` forces +synchronous logging if the handler queue exceeds 20 messages — backpressure +on log floods. + +### Scoper + +[`scoper`](https://github.com/valitydev/scoper) accumulates structured +key/value context across nested scopes. Fistful opens scopes everywhere: + +- At the machine boundary in + [`fistful:scope/3`](../apps/fistful/src/fistful.erl#L168) with + `{machine, namespace, id, activity}`. +- At the handler boundary — e.g. + [`ff_withdrawal_handler:handle_function/3`](../apps/ff_server/src/ff_withdrawal_handler.erl#L17) + opens `withdrawal` and adds `{id, wallet_id, destination_id, external_id}` + via `scoper:add_meta/1`. +- At the adapter host + ([`ff_withdrawal_adapter_host:handle_function/3`](../apps/ff_server/src/ff_withdrawal_adapter_host.erl#L19)). + +The `scoper_storage_logger` storage +([sys.config:137](../config/sys.config#L137)) attaches the current scope +to every log message automatically, so you see `namespace=ff/withdrawal_v2` +`id=...` on every line without having to pass it around. + +### Woody event handler + +[`ff_woody_event_handler`](../apps/ff_server/src/ff_woody_event_handler.erl) +logs inbound and outbound Woody calls. The `scoper_woody_event_handler` +variant is used for `dmt_client` and `party_client` +([sys.config:149, 172](../config/sys.config#L149)) with +`formatter_opts.max_length => 1000` to cap payload dumps. + +## Metrics + +### Exposure + +`GET /metrics[/:registry]` on port `:8022` +([`ff_server:get_prometheus_routes/0`](../apps/ff_server/src/ff_server.erl#L119)) +serves the default Prometheus registry. + +Configured collectors: +[sys.config:277‑279](../config/sys.config#L277): `{collectors, [default]}`. + +### Instrumentation + +Two metric families are explicitly set up at boot +([`ff_server:setup_metrics/0`](../apps/ff_server/src/ff_server.erl#L164)): + +- `woody_ranch_prometheus_collector:setup/0` — Ranch (Cowboy listener) + metrics for inbound Woody/HTTP. +- `woody_hackney_prometheus_collector:setup/0` — Hackney metrics for + outbound HTTP. + +Hackney is also wired to report request metrics via `hackney.mod_metrics +=> woody_hackney_prometheus` ([sys.config:282](../config/sys.config#L282)). + +Out of the box you get: + +- Inbound: request count, duration histogram, in‑flight counts, per‑status‑code. +- Outbound: request count, duration, connection‑pool states. +- VM: the default Prometheus Erlang collector publishes memory, process + count, reductions, GC, etc. + +## OpenTelemetry + +Disabled by default. Enable by adding +[compose.tracing.yaml](../compose.tracing.yaml) to your `docker compose` +invocation (the Makefile does this via `DOCKERCOMPOSE_W_ENV`): + +``` +docker compose -f compose.yaml -f compose.tracing.yaml up -d +``` + +With tracing enabled: + +- `OTEL_TRACES_EXPORTER=otlp` +- `OTEL_EXPORTER_OTLP_PROTOCOL=http_protobuf` +- `OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4318` +- Sampler: `parentbased_always_on` for `testrunner`, `parentbased_always_off` + for the DMT/bender/limiter/party‑management containers (so trace + propagation works if the upstream samples, but they don't start traces + on their own). + +A Jaeger all‑in‑one container is added at +[compose.tracing.yaml:28](../compose.tracing.yaml#L28): + +- UI: `:16686` +- OTLP gRPC receiver: `:4317` +- OTLP HTTP receiver: `:4318` + +Tracing libraries are pulled in as rebar deps and loaded as `temporary` +applications ([rebar.config:95](../rebar.config#L95)) — the tracer only +runs if the opentelemetry application is actually started (which it is, +via `application:ensure_all_started`). + +## Health checks + +From [sys.config:252‑270](../config/sys.config#L252): + +**Liveness** (`GET /health/liveness`): + +| Check | Module | Runner | +|-------|--------|--------| +| `disk` | `erl_health` | `disk("/", 99)` — raise if > 99% full | +| `memory` | `erl_health` | `cg_memory(99)` — raise if cgroup > 99% | +| `service` | `erl_health` | `service(<<"fistful-server">>)` — sanity check | + +**Readiness** (`GET /health/readiness`): + +| Check | Runner | +|-------|--------| +| `dmt_client` | `dmt_client:health_check/0` — DMT reachable and caches initialized | +| `progressor` | `progressor:health_check/1` across the five active namespaces | + +All checks are wrapped by +[`ff_server:enable_health_logging/1`](../apps/ff_server/src/ff_server.erl#L114) +which attaches the `erl_health_event_handler` so state transitions +(healthy ↔ failed) land in the log. + +> [!WARNING] +> `ff/identity` and `ff/wallet_v2` are deliberately **not** part of the +> progressor readiness check. If your deployment still relies on those +> legacy namespaces, add them to the readiness list in `sys.config`. + +## Machine trace endpoint + +[`ff_machine_handler`](../apps/ff_server/src/ff_machine_handler.erl) +registers five Cowboy routes for JSON trace dumps: + +``` +GET /traces/internal/source_v1/:process_id +GET /traces/internal/destination_v2/:process_id +GET /traces/internal/deposit_v1/:process_id +GET /traces/internal/withdrawal_v2/:process_id +GET /traces/internal/withdrawal_session_v2/:process_id +``` + +Response: `200 OK` with a JSON serialization of the machine's full event +log (from `machinery:trace/3`). Status codes: + +- `404 Unknown process` if the ID is missing. +- `400 Invalid ProcessID` if the URL segment is malformed. +- `405 Method Not Allowed` for anything other than `GET`. + +Use this for post‑mortem debugging — you can rebuild the timeline of any +withdrawal, deposit, or session without opening a shell into the +database. + +> [!NOTE] +> The trace endpoint is **not** authenticated. Keep it off the public +> internet. diff --git a/docs/operations.md b/docs/operations.md new file mode 100644 index 00000000..f9db8a83 --- /dev/null +++ b/docs/operations.md @@ -0,0 +1,176 @@ +# Operations + +This document covers what running fistful in production looks like: +container image, release artifacts, health model, and the repair +vocabulary used to unstick broken machines. + +## Release image + +Built by [Dockerfile](../Dockerfile). Two stages: + +1. **Builder** — `erlang:27.1.2` with the Vality thrift compiler + installed, runs `rebar3 compile && rebar3 as prod release`. +2. **Runtime** — `erlang:27.1.2-slim` with: + - `/opt/fistful-server/` — release tree, owned by UID 1001. + - `/var/log/fistful-server/` — log directory. + - `/entrypoint.sh` — invokes `/opt/fistful-server/bin/fistful-server foreground`. + - Exposes `:8022`. + +Released via `rebar3 as prod release`; see the prod profile at +[rebar.config:81‑108](../rebar.config#L81). `mode => minimal` and +`extended_start_script => true` are set, so the standard release +commands (`console`, `foreground`, `ping`, `remote_console`, +`rpc`, `attach`) are all available from +`bin/fistful-server`. + +## Runtime topology + +```mermaid +flowchart TB + subgraph pod[fistful-server pod] + FF[fistful-server
:8022] + LOGS[/var/log/fistful-server/
console.json/] + end + LB[ingress / LB] -->|Thrift| FF + FF -->|OTLP :4318| JAEGER[jaeger / OTEL collector] + FF -->|Prometheus scrape| PROM[prometheus] + FF -->|logs| LOGCOLL[log collector
filebeat / vector] + FF -->|Woody RPC| DMT[DMT] + FF -->|Woody RPC| PM[party-management] + FF -->|Woody RPC| SH[shumway] + FF -->|Woody RPC| LI[limiter] + FF -->|Woody RPC| BN[bender] + FF -->|Woody RPC| VL[validator-personal-data] + FF --> DB[(PostgreSQL
fistful schema)] +``` + +- The HTTP listener on `:8022` serves Thrift RPCs, the Prometheus + scrape endpoint, the health probes, and the internal trace dump. Lock + this port down to trusted callers — there is no authentication + except whatever the ingress enforces. +- Logs are written to `/var/log/fistful-server/console.json` (JSON / + logstash format). Mount or tail this to your log collector. +- If OTEL env vars are set, traces are pushed to the configured OTLP + endpoint. + +## Health model + +Two endpoints, independently significant: + +| Endpoint | What it checks | Suggested use | +|----------|----------------|---------------| +| `GET /health/liveness` | disk < 99% full, cgroup memory < 99%, service registered | Kubernetes livenessProbe | +| `GET /health/readiness` | `dmt_client:health_check/0`, `progressor:health_check/[namespaces]` | Kubernetes readinessProbe | + +Configured in [sys.config:252‑270](../config/sys.config#L252). Both +check specs go through +[`ff_server:enable_health_logging/1`](../apps/ff_server/src/ff_server.erl#L114), +so transitions between healthy and unhealthy are also logged. + +Readiness covers: + +- DMT reachability — if DMT is down, fistful will misroute; returning + unready keeps traffic off the pod. +- Progressor namespaces — `ff/source_v1`, `ff/destination_v2`, + `ff/deposit_v1`, `ff/withdrawal_v2`, `ff/withdrawal/session_v2`. + +> [!WARNING] +> The readiness probe does **not** check `party-management`, `shumway`, +> `limiter`, `bender`, or `validator`. A failing dependency will manifest +> as business‑error responses, not readiness flips. + +## Scaling + +Fistful is stateless in the sense that all durable state lives in +PostgreSQL via progressor. Multiple replicas can share the same +database; progressor's worker pool serializes work on `(namespace, id)` +with PostgreSQL advisory locking (see the progressor source for +details). + +- **Vertical**: `progressor.defaults.worker_pool_size` (default 100) is + the knob for CPU utilization per node. +- **Horizontal**: add replicas. No sticky sessions required — any + replica can serve any RPC. + +## Database + +One PostgreSQL database per service. Fistful uses `fistful` with user +`fistful`. Migrations are applied automatically by progressor on startup. + +- **Backups**: standard PostgreSQL backup is fine. Progressor's tables + are append‑only for events; `aux_state` is mutated in place. +- **Retention**: none automatic. The entire event history of every + machine is kept indefinitely. Cleanup is a business decision. + +## Repair scenarios + +When a machine ends up stuck (e.g. a provider adapter returns +nonsense, a posting transfer got into a bad state, a session lost its +callback tag), fistful exposes **repair** RPCs that inject events into +the machine to nudge it into a valid state. + +Three services: + +| Thrift path | Handler | Typical scenarios | +|-------------|---------|-------------------| +| `/v1/repair/withdrawal` | [`ff_withdrawal_repair`](../apps/ff_server/src/ff_withdrawal_repair.erl) | Force `{status_changed, {failed, _}}`, set session result | +| `/v1/repair/withdrawal/session` | [`ff_withdrawal_session_repair`](../apps/ff_server/src/ff_withdrawal_session_repair.erl) | Force session result, override adapter state | +| `/v1/repair/deposit` | [`ff_deposit_repair`](../apps/ff_server/src/ff_deposit_repair.erl) | Force completion status | + +The Thrift scenario is unmarshalled by +[`ff_withdrawal_codec:unmarshal(repair_scenario, ...)`](../apps/ff_server/src/ff_withdrawal_codec.erl) +and then dispatched through +[`ff_repair:apply_scenario/3`](../apps/fistful/src/ff_repair.erl): + +- `add_events` — the generic scenario: append an arbitrary list of + domain events to the machine. The events must type‑check against the + entity's `apply_event/2` or the machine will remain stuck. + +Repair emits `{error, working}` if the machine is *not* in a faulted +state — progressor will not let you inject events into a machine that +is currently executing. Mapped to `#fistful_MachineAlreadyWorking{}` in +the handler. + +## Operational runbook snippets + +**Inspect a withdrawal's full timeline** (no shell needed): + +``` +curl -s http://:8022/traces/internal/withdrawal_v2/ | jq +``` + +**Check how many machines in each namespace are stuck**: no built‑in +metric; progressor's metrics tables expose this directly (see the +`progressor.*` Prometheus series once exported). + +**Force a failed withdrawal to terminal state** — via Thrift `Repair`: + +```thrift +Repair( + withdrawal_id, + {add_events: [ + {status_changed: {failed: {code: "forced_by_ops"}}} + ]} +) +``` + +**Graceful shutdown**: send `SIGTERM`; the release's OTP shutdown +sequence stops the Cowboy listener first (refusing new requests) +before tearing down supervision trees. + +## Known operational gaps + +Pulled from TODOs in the codebase: + +- There is **no manual sweep** for asynchronous machines — if a + machine has no scheduled timeout, it will stay asleep forever. + Stuck machines must currently be poked by a `Repair` call or a + synthetic notification. See the TODO in + [README.md](../README.md#L32) ("добавить ручную прополку для всех + асинхронных процессов"). +- The progressor readiness check doesn't cover the legacy + `ff/identity` / `ff/wallet_v2` namespaces; if they're still in use in + your deployment, add them to [sys.config:262‑268](../config/sys.config#L262) + to extend readiness. +- Withdrawal session retries are capped at 24 h; if an adapter genuinely + needs longer, you must repair the session. diff --git a/docs/persistence.md b/docs/persistence.md new file mode 100644 index 00000000..16940b52 --- /dev/null +++ b/docs/persistence.md @@ -0,0 +1,208 @@ +# Persistence + +Fistful uses [progressor](https://github.com/valitydev/progressor) as its +state‑machine runtime, with PostgreSQL as the sole backing store. Every +domain entity is written as an append‑only event stream plus an +`aux_state` blob; state is reconstructed in memory from the event log on +demand. + +## Layers + +```mermaid +flowchart LR + DOMAIN[fistful domain code
ff_withdrawal, ...] + FFM[ff_machine] + FF[fistful
machinery_backend] + PRG[machinery_prg_backend] + PRG_APP[progressor
worker pools] + PG_BACKEND[prg_pg_backend] + PG[(PostgreSQL
database=fistful)] + + DOMAIN --> FFM + FFM --> FF + FF --> PRG + PRG --> PRG_APP + PRG_APP --> PG_BACKEND + PG_BACKEND --> PG +``` + +## Progressor configuration + +[config/sys.config:34‑134](../config/sys.config#L34) configures progressor +with: + +- A shared PostgreSQL connection pool `default_pool` of size 10 + ([sys.config:27‑30](../config/sys.config#L27)). +- A common default block: + + ```erlang + #{ + storage => #{client => prg_pg_backend, + options => #{pool => default_pool}}, + retry_policy => #{initial_timeout => 5, + backoff_coefficient => 1.0, + max_timeout => 180, + max_attempts => 3, + non_retryable_errors => []}, + task_scan_timeout => 1, + worker_pool_size => 100, + process_step_timeout => 30 + } + ``` + +- One `namespaces` map entry per machinery namespace, each specifying a + `processor` (the machinery backend) with: + - `namespace` — e.g. `'ff/withdrawal_v2'`. + - `handler` — `{fistful, #{handler => ff_withdrawal_machine, party_client => #{}}}`. + - `schema` — a `machinery_mg_schema` implementation (see below). + +At startup, +[`ff_server:get_namespaces_params/0`](../apps/ff_server/src/ff_server.erl#L139) +reads this configuration back out and registers a `machinery_prg_backend` +child spec under the application environment key `fistful.backends` (one +per namespace). Domain code later obtains a backend with +[`fistful:backend/1`](../apps/fistful/src/fistful.erl#L36). + +## Schema modules + +Each namespace has a schema module under +[apps/ff_server/src](../apps/ff_server/src/) that implements +`machinery_mg_schema`: + +| Namespace | Schema module | +|-----------|---------------| +| `ff/source_v1` | [`ff_source_machinery_schema`](../apps/ff_server/src/ff_source_machinery_schema.erl) | +| `ff/destination_v2` | [`ff_destination_machinery_schema`](../apps/ff_server/src/ff_destination_machinery_schema.erl) | +| `ff/deposit_v1` | [`ff_deposit_machinery_schema`](../apps/ff_server/src/ff_deposit_machinery_schema.erl) | +| `ff/withdrawal_v2` | [`ff_withdrawal_machinery_schema`](../apps/ff_server/src/ff_withdrawal_machinery_schema.erl) | +| `ff/withdrawal/session_v2` | [`ff_withdrawal_session_machinery_schema`](../apps/ff_server/src/ff_withdrawal_session_machinery_schema.erl) | + +A schema exposes: + +```erlang +-spec get_version(value_type()) -> machinery_mg_schema:version(). +-spec marshal(type(), value(data()), context()) -> {machinery_msgpack:t(), context()}. +-spec unmarshal(type(), machinery_msgpack:t(), context()) -> {value(data()), context()}. +``` + +The implementation pattern is: convert the Erlang event to a Thrift +struct via the corresponding `ff_*_codec` (e.g. `TimestampedChange`), +serialize with `ff_proto_utils:serialize/2`, wrap in a `{bin, Binary}` +msgpack value. `aux_state` is handled similarly for the `ff_entity_context`. + +### Versioning + +Each schema tags events with a version number. On unmarshal, an older +version is either accepted directly (if the Thrift struct is +backwards‑compatible) or passed through a migration step in the domain +module — see each entity's `maybe_migrate/2` callback (e.g. +[`ff_withdrawal:maybe_migrate/2`](../apps/ff_transfer/src/ff_withdrawal.erl)). + +## PostgreSQL database + +Database name: `fistful`; owner: `fistful` (see +[sys.config:17‑25](../config/sys.config#L17)). Created as part of the +[compose.yaml](../compose.yaml) stack — the `db` service runs a Postgres +15 image with a multi‑database init script +(`POSTGRES_MULTIPLE_DATABASES="fistful,bender,dmt,party_management,shumway,liminator"` +in [compose.yaml:134](../compose.yaml#L134)). The init SQL lives under +[test/postgres/docker-entrypoint-initdb.d/](../test/postgres/). + +Schema migrations for progressor are applied automatically by the library +when the first namespace registers. + +## Event append path + +```mermaid +sequenceDiagram + autonumber + participant D as ff_withdrawal (domain) + participant M as ff_machine + participant MP as machinery_prg_backend + participant P as progressor + participant S as ff_*_machinery_schema + participant PG as prg_pg_backend + + D->>M: emit_event/1 for each change + M-->>M: {ev, Timestamp, Change} + D-->>MP: {response, [events], action} + MP->>P: record_step(namespace, id, events, aux_state) + P->>S: marshal({event, V}, {ev, T, E}) + S-->>P: {bin, Binary} + P->>PG: append to events, update aux_state + PG-->>P: ok + P-->>MP: ok +``` + +## Read path + +```mermaid +sequenceDiagram + autonumber + participant C as caller (handler) + participant F as fistful:get + participant M as ff_machine:collapse + participant P as progressor + participant S as ff_*_machinery_schema + participant PG as prg_pg_backend + participant D as entity module (apply_event) + + C->>F: machinery:get(NS, ID, Range) + F->>P: read events + aux_state + P->>PG: SELECT + PG-->>P: events, aux_state + P->>S: unmarshal each + S-->>P: {ev, Timestamp, Change} + P-->>F: machine{} + F->>M: collapse(ff_entity, Machine) + M->>D: apply_event(Change, Model) for each event + D-->>M: Model' + M-->>F: st{model, ctx} + F-->>C: st() +``` + +## Health and readiness + +The readiness probe in +[sys.config:257‑269](../config/sys.config#L257) includes a progressor +check: + +```erlang +progressor => {progressor, health_check, [[ + 'ff/source_v1', 'ff/destination_v2', 'ff/deposit_v1', + 'ff/withdrawal_v2', 'ff/withdrawal/session_v2' +]]} +``` + +This confirms each configured namespace is registered and its worker +pool is alive. Note that `ff/identity` and `ff/wallet_v2` are **not** in +the readiness list — consistent with those namespaces no longer being +served by local machines. + +## Contexts (`aux_state`) + +The `aux_state` blob holds the `ff_entity_context:context()` — arbitrary +client‑defined metadata stored alongside the machine. Every +`ff_machine:emit_event/1` write also rewrites the aux state if the +context changed. + +## Trace / introspection + +`machinery:trace/3` (wrapped by +[`ff_machine:trace/2`](../apps/fistful/src/ff_machine.erl#L58)) returns a +JSON‑serializable dump of the full event history. Exposed over HTTP via +[`ff_machine_handler`](../apps/ff_server/src/ff_machine_handler.erl) for +post‑mortem inspection. + +> [!WARNING] +> The trace endpoint is **not** authenticated. It is only exposed on the +> internal `:8022` port; lock the port down at the network layer. + +## The `machinery_extra` in‑memory backend + +[`machinery_gensrv_backend`](../apps/machinery_extra/src/machinery_gensrv_backend.erl) +provides an in‑memory alternative to progressor for tests that don't +need to exercise the storage layer (or for experimental hot‑path +measurements). It is registered by name rather than namespace, and +events live in the `gen_server` state — no migrations, no persistence. +Production does **not** use this backend. diff --git a/docs/routing.md b/docs/routing.md new file mode 100644 index 00000000..1eec5113 --- /dev/null +++ b/docs/routing.md @@ -0,0 +1,195 @@ +# Routing + +Routing selects the `(provider_id, terminal_id)` pair a withdrawal will +attempt. The logic is split across three modules: + +- [`ff_withdrawal_routing`](../apps/ff_transfer/src/ff_withdrawal_routing.erl) — orchestration. +- [`ff_routing_rule`](../apps/fistful/src/ff_routing_rule.erl) — ruleset evaluation. +- [`ff_limiter`](../apps/ff_transfer/src/ff_limiter.erl) — turnover‑limit filtering. + +## Mental model + +A payment institution (PI) has two routing rulesets attached: + +1. **Policies** — positive rules that accept routes. +2. **Prohibitions** — negative rules that reject routes. + +The effective candidate set is *policies minus prohibitions*. Each +candidate is then filtered by its provider/terminal's withdrawal +provision terms (currency, methods, cash range, payment system) and +finally by turnover limits held through the external `limiter` service. +The first accepted route is chosen; the rest are kept in an attempt index +so that a failed session can fall forward to the next candidate. + +## Flow + +```mermaid +flowchart TD + PI[payment institution
ff_party:compute_payment_institution] + POL[policies ruleset] + PROH[prohibitions ruleset] + PI --> POL + PI --> PROH + POL -->|accepted routes| A[Candidate pool] + PROH -->|subtract rejected| A + A --> TF[per-route term filter
ff_party:compute_provider_terminal_terms] + TF --> LF[limit filter
ff_withdrawal_routing:filter_limit_overflow_routes] + LF --> R{non-empty?} + R -- yes --> PICK[pick first
remember rest] + R -- no --> FAIL[route_not_found
+ reject_context] +``` + +## Entry point + +[`ff_withdrawal_routing:prepare_routes/2,3`](../apps/ff_transfer/src/ff_withdrawal_routing.erl#L72): + +```erlang +-spec prepare_routes(party_varset(), routing_context()) -> + {ok, [route()]} | {error, route_not_found()}. +-type routing_context() :: #{ + domain_revision := domain_revision(), + wallet := wallet(), + iteration := pos_integer(), + withdrawal => withdrawal() +}. +``` + +It builds a `routing_state` = `#{routes, reject_context}` by: + +1. Fetching the PI for the wallet via + [`ff_party:compute_payment_institution/3`](../apps/fistful/src/ff_party.erl#L74). +2. Calling + [`ff_routing_rule:gather_routes/4`](../apps/fistful/src/ff_routing_rule.erl) + with the PI's `withdrawal_routing_rules` and the party varset. The + return is a `{Accepted, RejectContext}` pair — the reject context is + an accumulator that lists every route that was dropped and why. +3. Filtering the accepted list by terminal‑level terms (ever‑present in + each provider/terminal config). +4. If `withdrawal` is present in the context (i.e. we're creating a real + withdrawal, not just computing a quote), calling + [`filter_limit_overflow_routes/3`](../apps/ff_transfer/src/ff_withdrawal_routing.erl#L10) + to drop routes whose `limiter` turnover limits would overflow. +5. Returning `{ok, Routes}` or `{error, {route_not_found, Rejected}}`. + +## Routing rules + +[`ff_routing_rule:route/0`](../apps/fistful/src/ff_routing_rule.erl): + +```erlang +-type route() :: #{ + provider_ref := provider_ref(), + terminal_ref := terminal_ref(), + priority := integer(), + weight => integer() +}. + +-type reject_context() :: #{ + varset := varset(), + accepted_routes := [route()], + rejected_routes := [rejected_route()] +}. +``` + +Evaluation reduces DMT selectors against the supplied party varset. When +a route fails evaluation (e.g. a selector demands `Currency = USD` but +the varset has `Currency = RUB`), the route is appended to +`rejected_routes` with the reason, and +[`ff_routing_rule:log_reject_context/1`](../apps/fistful/src/ff_routing_rule.erl) +is later called to surface the reasons in logs — invaluable when +debugging "why did my withdrawal not route". + +## Per‑route term filter + +For every candidate route, a dedicated +[`ff_party:compute_provider_terminal_terms/4`](../apps/fistful/src/ff_party.erl#L76) +is reduced to produce the provider/terminal's +`WithdrawalProvisionTerms`. The candidate is rejected if: + +- Currency is not in `currencies` selector. +- Withdrawal method (e.g. `{bank_card, VISA}`) is not in `methods` + selector. +- Cash amount is outside `cash_limit` selector. +- Payment system doesn't match `payment_system` selector. + +The `process_route_fun` type +([ff_withdrawal_routing.erl:63](../apps/ff_transfer/src/ff_withdrawal_routing.erl#L63)) +describes the validator signature. + +## Turnover limit filter + +Each provider/terminal can declare **turnover limits** in DMT (a +`TurnoverLimitSelector` of `[TurnoverLimit]`). For each limit fistful +holds capacity in the external `limiter` service. See +[`ff_limiter:check_limits/4`](../apps/ff_transfer/src/ff_limiter.erl#L39): + +```erlang +-spec check_limits([turnover_limit()], withdrawal(), route(), pos_integer()) -> + {ok, [limit()]} | + {error, {overflow, [{limit_id(), limit_amount(), upper_boundary()}]}}. +``` + +The request carries a context built by +[`gen_limit_context/2`](../apps/ff_transfer/src/ff_limiter.erl) that +includes a marshalled domain withdrawal, the wallet identity, and an +**operation segment list** from +[`make_operation_segments/3`](../apps/ff_transfer/src/ff_limiter.erl#L52): + +``` +[provider_id, terminal_id, withdrawal_id, iteration?] +``` + +The operation ID is the concatenation of these segments — this is what +the limiter uses to make hold/commit/rollback idempotent per attempt. On +a retry with the same iteration, the limiter no‑ops; a new iteration +produces a distinct operation ID and a distinct reservation. + +## Commit and rollback + +Once a posting transfer commits successfully, +[`ff_withdrawal_routing:commit_routes_limits/3`](../apps/ff_transfer/src/ff_withdrawal_routing.erl#L11) +finalises the limit holds. On failure, +[`ff_withdrawal_routing:rollback_routes_limits/3`](../apps/ff_transfer/src/ff_withdrawal_routing.erl#L11) +releases them. Both are called from +[`ff_withdrawal:do_process_transfer/2`](../apps/ff_transfer/src/ff_withdrawal.erl#L745) +next to their posting‑transfer counterparts. + +## Multi‑attempt routing + +When a session fails on one route, the withdrawal doesn't have to fail +globally. The set of available routes is captured during the first +`process_routing` step. Each attempt: + +- Picks the next not‑yet‑attempted route. +- Rebuilds the cash flow with that provider's plan and fees. +- Starts a *new* posting transfer (new ID including iteration) and a new + session. +- Holds fresh turnover limits under a new operation ID. + +[`ff_withdrawal_route_attempt_utils`](../apps/ff_transfer/src/ff_withdrawal_route_attempt_utils.erl) +holds the per‑attempt data: route, transfer, session. The withdrawal +`attempts` field tracks this: + +```erlang +-type attempts() :: ff_withdrawal_route_attempt_utils:attempts(). +``` + +When all candidate routes are exhausted, the withdrawal transitions to +`{failed, Failure}`. The final iteration is also used when rolling back +routing limits (so holds from *every* attempted route are released). + +## Edge cases + +> [!WARNING] +> Transient errors (e.g. +> `authorization_failed:temporarily_unavailable`) are treated specially — +> the limit hold is **not** released and the route is kept for retry. +> Configured via +> [`ff_transfer.withdrawal.default_transient_errors`](../config/sys.config#L222) +> and the per‑party override +> [`ff_transfer.withdrawal.party_transient_errors`](../config/sys.config#L227). + +> [!NOTE] +> Routing terms failures have been a source of outages: a prior bug +> crashed the machine when a provider had no withdrawal terms; now +> `ff_withdrawal_routing` guards against missing terms (see commit +> `abee5cd — fix withdrawal routing crash when terms not found`). diff --git a/docs/rpc-api.md b/docs/rpc-api.md new file mode 100644 index 00000000..6b0d465d --- /dev/null +++ b/docs/rpc-api.md @@ -0,0 +1,174 @@ +# RPC API + +Fistful exposes a **Woody/Thrift** API over HTTP on port `8022` (see +[sys.config:238](../config/sys.config#L238)). Every endpoint is defined +in the [fistful‑proto](https://github.com/valitydev/fistful-proto) +IDL (pulled in as a rebar dep, version `v2.0.2` — see +[rebar.config:40](../rebar.config#L40)). The adapter‑host endpoint uses +[damsel](https://github.com/valitydev/damsel)'s `dmsl_wthd_provider_thrift`. + +## Endpoint catalogue + +The service table lives in +[`ff_services`](../apps/ff_server/src/ff_services.erl). Paths are mounted +by [`ff_server`](../apps/ff_server/src/ff_server.erl#L77). + +| HTTP path | Thrift service | Handler | +|-----------|----------------|---------| +| `/v1/source` | `fistful_source_thrift:'Management'` | [`ff_source_handler`](../apps/ff_server/src/ff_source_handler.erl) | +| `/v1/destination` | `fistful_destination_thrift:'Management'` | [`ff_destination_handler`](../apps/ff_server/src/ff_destination_handler.erl) | +| `/v1/deposit` | `fistful_deposit_thrift:'Management'` | [`ff_deposit_handler`](../apps/ff_server/src/ff_deposit_handler.erl) | +| `/v1/withdrawal` | `fistful_wthd_thrift:'Management'` | [`ff_withdrawal_handler`](../apps/ff_server/src/ff_withdrawal_handler.erl) | +| `/v1/withdrawal_session` | `fistful_wthd_session_thrift:'Management'` | [`ff_withdrawal_session_handler`](../apps/ff_server/src/ff_withdrawal_session_handler.erl) | +| `/v1/repair/deposit` | `fistful_deposit_thrift:'Repairer'` | [`ff_deposit_repair`](../apps/ff_server/src/ff_deposit_repair.erl) | +| `/v1/repair/withdrawal` | `fistful_wthd_thrift:'Repairer'` | [`ff_withdrawal_repair`](../apps/ff_server/src/ff_withdrawal_repair.erl) | +| `/v1/repair/withdrawal/session` | `fistful_wthd_session_thrift:'Repairer'` | [`ff_withdrawal_session_repair`](../apps/ff_server/src/ff_withdrawal_session_repair.erl) | +| `/v1/ff_withdrawal_adapter_host` | `dmsl_wthd_provider_thrift:'AdapterHost'` | [`ff_withdrawal_adapter_host`](../apps/ff_server/src/ff_withdrawal_adapter_host.erl) | + +## Handler pattern + +Every handler implements the [`ff_woody_wrapper`](../apps/ff_server/src/ff_woody_wrapper.erl) +behaviour: + +```erlang +-spec handle_function(woody:func(), woody:args(), woody:options()) -> + {ok, woody:result()} | no_return(). +``` + +The uniform recipe inside each clause: + +```erlang +handle_function_('Create', {MarshaledParams, MarshaledContext}, _Opts) -> + Params = ff_*_codec:unmarshal(params, MarshaledParams), + Context = ff_*_codec:unmarshal(ctx, MarshaledContext), + ok = scoper:add_meta(#{ ... }), + case ff_*_machine:create(Params, Context) of + ok -> handle_function_('Get', {Id, Range}, _); + {error, exists} -> handle_function_('Get', {Id, Range}, _); + {error, Reason} -> woody_error:raise(business, #fistful_*{...}) + end. +``` + +Create is idempotent — a duplicate call to the same ID re‑reads the +current state rather than failing. + +## Source management — `/v1/source` + +Handler: [`ff_source_handler`](../apps/ff_server/src/ff_source_handler.erl). + +| RPC | Signature | Notes | +|-----|-----------|-------| +| `Create` | `(SourceParams, Ctx) → SourceState` | Idempotent on ID | +| `Get` | `(ID, EventRange) → SourceState` | Throws `#fistful_SourceNotFound{}` | +| `GetContext` | `(ID) → Context` | | +| `GetEvents` | `(ID, EventRange) → [Event]` | | + +## Destination management — `/v1/destination` + +Handler: [`ff_destination_handler`](../apps/ff_server/src/ff_destination_handler.erl). + +Same shape as sources: `Create`, `Get`, `GetContext`, `GetEvents`. + +## Deposit management — `/v1/deposit` + +Handler: [`ff_deposit_handler`](../apps/ff_server/src/ff_deposit_handler.erl). + +Four RPCs as above (`Create`, `Get`, `GetContext`, `GetEvents`). Business +errors from Create include `#fistful_SourceNotFound{}`, +`#fistful_WalletNotFound{}`, `#fistful_ForbiddenOperationCurrency{}`, +`#fistful_ForbiddenOperationAmount{}`, `#fistful_RealmsMismatch{}`, +`#wthd_InconsistentDepositCurrency{}` (see +[ff_deposit_handler.erl:30‑](../apps/ff_server/src/ff_deposit_handler.erl#L30)). + +## Withdrawal management — `/v1/withdrawal` + +Handler: [`ff_withdrawal_handler`](../apps/ff_server/src/ff_withdrawal_handler.erl). + +| RPC | Signature | Notes | +|-----|-----------|-------| +| `GetQuote` | `(QuoteParams) → Quote` | Doesn't start a machine; one‑shot | +| `Create` | `(WithdrawalParams, Ctx) → WithdrawalState` | Idempotent on ID | +| `Get` | `(ID, EventRange) → WithdrawalState` | | +| `GetContext` | `(ID) → Context` | | +| `GetEvents` | `(ID, EventRange) → [Event]` | | +| `CreateAdjustment` | `(ID, AdjParams) → AdjustmentState` | See [adjustments.md](adjustments.md) | + +Full error mapping is in +[ff_withdrawal_handler.erl:36‑](../apps/ff_server/src/ff_withdrawal_handler.erl#L36). + +## Withdrawal session management — `/v1/withdrawal_session` + +Handler: [`ff_withdrawal_session_handler`](../apps/ff_server/src/ff_withdrawal_session_handler.erl). + +| RPC | Signature | +|-----|-----------| +| `Get` | `(ID, EventRange) → SessionState` | +| `GetEvents` | `(ID, EventRange) → [Event]` | +| `GetContext` | `(ID) → Context` | + +Sessions are not directly created over the API — they're spawned by +the withdrawal machine. The management service is read‑only. + +## Repairers — `/v1/repair/*` + +Three services, one RPC each: + +| Service | Handler | RPC | +|---------|---------|-----| +| `/v1/repair/withdrawal` | [`ff_withdrawal_repair`](../apps/ff_server/src/ff_withdrawal_repair.erl) | `Repair(ID, Scenario) → ok` | +| `/v1/repair/withdrawal/session` | [`ff_withdrawal_session_repair`](../apps/ff_server/src/ff_withdrawal_session_repair.erl) | `Repair(ID, Scenario) → ok` | +| `/v1/repair/deposit` | [`ff_deposit_repair`](../apps/ff_server/src/ff_deposit_repair.erl) | `Repair(ID, Scenario) → ok` | + +Errors: `#fistful_*NotFound{}` when the machine is missing, +`#fistful_MachineAlreadyWorking{}` when a repair is attempted against a +healthy machine. See [operations.md](operations.md#repair-scenarios) for +the scenario vocabulary. + +## Adapter host — `/v1/ff_withdrawal_adapter_host` + +Handler: [`ff_withdrawal_adapter_host`](../apps/ff_server/src/ff_withdrawal_adapter_host.erl). +Single RPC: `ProcessCallback(Callback) → ProcessCallbackResult`. + +This is the only endpoint *called by external parties into* fistful — +provider adapters push asynchronous results back here. See +[adapter-integration.md](adapter-integration.md). + +## Codecs + +Codec modules marshal/unmarshal between the Thrift wire types and the +fistful domain types. One codec per entity plus several shared ones: + +| Codec | File | +|-------|------| +| [`ff_codec`](../apps/ff_server/src/ff_codec.erl) | Common: ID, cash, currency, cash range, context, event range | +| [`ff_dmsl_codec`](../apps/fistful/src/ff_dmsl_codec.erl) | Damsel types used by domain code | +| [`ff_source_codec`](../apps/ff_server/src/ff_source_codec.erl) | Source params / state / events | +| [`ff_destination_codec`](../apps/ff_server/src/ff_destination_codec.erl) | Destination | +| [`ff_deposit_codec`](../apps/ff_server/src/ff_deposit_codec.erl) | Deposit params / state / events | +| [`ff_withdrawal_codec`](../apps/ff_server/src/ff_withdrawal_codec.erl) | Withdrawal params / state / events / quote / repair scenario | +| [`ff_withdrawal_session_codec`](../apps/ff_server/src/ff_withdrawal_session_codec.erl) | Session state / events | +| [`ff_adapter_withdrawal_codec`](../apps/ff_transfer/src/ff_adapter_withdrawal_codec.erl) | Provider adapter types | +| [`ff_cash_flow_codec`](../apps/ff_server/src/ff_cash_flow_codec.erl) | Final cash flow | +| [`ff_p_transfer_codec`](../apps/ff_server/src/ff_p_transfer_codec.erl) | Posting transfer events | +| [`ff_limit_check_codec`](../apps/ff_server/src/ff_limit_check_codec.erl) | Limit‑check details | +| [`ff_withdrawal_adjustment_codec`](../apps/ff_server/src/ff_withdrawal_adjustment_codec.erl) | Adjustment change + state | +| [`ff_withdrawal_status_codec`](../apps/ff_server/src/ff_withdrawal_status_codec.erl) / [`ff_deposit_status_codec`](../apps/ff_server/src/ff_deposit_status_codec.erl) | Status enums | +| [`ff_entity_context_codec`](../apps/ff_server/src/ff_entity_context_codec.erl) | `ff_entity_context:context()` | +| [`ff_msgpack_codec`](../apps/ff_server/src/ff_msgpack_codec.erl) | Msgpack ↔ Erlang for entity context | + +All codecs follow a symmetrical `marshal/2 + unmarshal/2` shape. +`undefined` fields are stripped via `genlib_map:compact/1` on the way +out and defaulted on the way in. + +## Internal HTTP endpoints + +Exposed on the same `:8022` listener but outside the Thrift dispatch: + +| Path | Purpose | Source | +|------|---------|--------| +| `GET /metrics[/:registry]` | Prometheus scrape | [ff_server.erl:119](../apps/ff_server/src/ff_server.erl#L119) | +| `GET /health/liveness` | Liveness probe | [ff_server.erl:103](../apps/ff_server/src/ff_server.erl#L103) | +| `GET /health/readiness` | Readiness probe | [ff_server.erl:103](../apps/ff_server/src/ff_server.erl#L103) | +| `GET /traces/internal/{source_v1\|destination_v2\|deposit_v1\|withdrawal_v2\|withdrawal_session_v2}/:process_id` | Machine trace JSON dump | [ff_machine_handler.erl](../apps/ff_server/src/ff_machine_handler.erl) | + +See [observability.md](observability.md) for what these surface. diff --git a/docs/state-machines.md b/docs/state-machines.md new file mode 100644 index 00000000..7b6ccce4 --- /dev/null +++ b/docs/state-machines.md @@ -0,0 +1,200 @@ +# State Machines + +Every processable domain entity is a `machinery:machine`. The common +scaffolding lives in [`ff_machine`](../apps/fistful/src/ff_machine.erl) +and the progressor ↔ handler glue lives in +[`fistful`](../apps/fistful/src/fistful.erl). + +## Machinery callbacks + +The [`ff_machine`](../apps/fistful/src/ff_machine.erl#L76) behaviour pins +down six callbacks each entity module must implement: + +```erlang +-callback init(machinery:args(_)) -> [event()]. +-callback apply_event(event(), model()) -> model(). +-callback maybe_migrate(event(), migrate_params()) -> event(). %% optional +-callback process_call(machinery:args(_), st()) -> {machinery:response(_), [event()]}. +-callback process_repair(machinery:args(_), st()) -> + {ok, machinery:response(_), [event()]} | {error, machinery:error(_)}. +-callback process_timeout(st()) -> [event()]. +``` + +`process_notification/2` is defined on a per‑entity basis — e.g. the +withdrawal machine uses it to react to session completion. + +Event emission is funneled through `ff_machine:emit_event/1`, which wraps +the raw change in a `{ev, CurrentTimestamp, Change}` triple — giving every +logged event its own timestamp. + +## Namespace catalogue + +```mermaid +flowchart LR + SRC[ff/source_v1
ff_source_machine] + DST[ff/destination_v2
ff_destination_machine] + DEP[ff/deposit_v1
ff_deposit_machine] + WTH[ff/withdrawal_v2
ff_withdrawal_machine] + SES[ff/withdrawal/session_v2
ff_withdrawal_session_machine] + SRC -->|source_id| DEP + DEP --> WAL[(wallet balance
shumway)] + WTH -->|session_id| SES + SES -->|session_finished
notify| WTH + WTH --> DST + WAL --> WTH +``` + +## Source machine — `ff/source_v1` + +Module: [`ff_source_machine`](../apps/ff_transfer/src/ff_source_machine.erl). + +- Public API: `create/2`, `get/1,2`, `events/2`. +- `init/1` emits `[{created, Source}]` plus an optional `{auth_data_changed, _}`. +- `process_timeout/1` is effectively a no‑op; sources have no ongoing processing. +- `process_repair` delegates to `ff_repair:apply_scenario/3` with + `add_events` support. + +## Destination machine — `ff/destination_v2` + +Module: [`ff_destination_machine`](../apps/ff_transfer/src/ff_destination_machine.erl). + +Symmetric to the source machine. Destinations additionally carry +`auth_data` for tokenised card payouts, set on creation via +`{auth_data_changed, ...}`. + +## Deposit machine — `ff/deposit_v1` + +Module: [`ff_deposit_machine`](../apps/ff_transfer/src/ff_deposit_machine.erl). + +```mermaid +stateDiagram-v2 + [*] --> pending: {created, Deposit} + pending --> limit_check: {limit_check, {wallet_receiver, ok}} + pending --> limit_failed: {limit_check, {wallet_receiver, {failed, _}}} + limit_check --> p_prepared: {p_transfer, {status_changed, prepared}} + p_prepared --> p_committed: {p_transfer, {status_changed, committed}} + p_committed --> succeeded: {status_changed, succeeded} + limit_failed --> p_cancelled: {p_transfer, {status_changed, cancelled}} + p_cancelled --> failed: {status_changed, {failed, _}} + succeeded --> [*] + failed --> [*] +``` + +The activity dispatcher lives in +[`ff_deposit`](../apps/ff_transfer/src/ff_deposit.erl) (`process_transfer/1`) +and walks: receiver limit check → posting transfer → commit → finish, with +compensation on failure. Unlike withdrawals, there is no routing and no +session. + +## Withdrawal machine — `ff/withdrawal_v2` + +Module: [`ff_withdrawal_machine`](../apps/ff_transfer/src/ff_withdrawal_machine.erl). +Driven by [`ff_withdrawal:process_transfer/1`](../apps/ff_transfer/src/ff_withdrawal.erl#L557). + +The activity for a given state is computed by +[`deduce_activity/1`](../apps/ff_transfer/src/ff_withdrawal.erl#L685) from a +tuple of sub‑statuses — see the table: + +| `status` | `route` | `p_transfer` | `limit_check` | `session` | → activity | +|----------|---------|--------------|---------------|-----------|------------| +| pending | unknown | undefined | — | — | `routing` | +| pending | found | undefined | — | — | `p_transfer_start` | +| pending | — | created | — | — | `p_transfer_prepare` | +| pending | — | prepared | unknown | — | `limit_check` | +| pending | — | prepared | ok | undefined | `session_starting` | +| pending | — | prepared | — | pending | `session_sleeping` | +| pending | — | prepared | — | succeeded | `p_transfer_commit` | +| pending | — | committed | — | succeeded | `finish` | +| pending | — | prepared | — | failed | `p_transfer_cancel` | +| pending | — | prepared | failed | — | `p_transfer_cancel` | +| pending | — | cancelled | failed | — | `{fail, limit_check}` | +| pending | — | cancelled | — | failed | `{fail, session}` | +| succeeded/failed | — | — | — | — | `adjustment` or `rollback_routing` | + +Each activity is then dispatched by +[`do_process_transfer/2`](../apps/ff_transfer/src/ff_withdrawal.erl#L735). +See [withdrawal-flow.md](withdrawal-flow.md) for the full narrative. + +> [!NOTE] +> Calls: the withdrawal machine implements `process_call` for +> `start_adjustment`. Notifications: it implements `process_notification` +> to receive `{session_finished, SessionID, Result}` messages from a +> session machine that just completed. + +## Withdrawal session machine — `ff/withdrawal/session_v2` + +Module: [`ff_withdrawal_session_machine`](../apps/ff_transfer/src/ff_withdrawal_session_machine.erl). + +```mermaid +stateDiagram-v2 + [*] --> active: {created, Session} + active --> sleeping: {next_state, AdapterState} + sleeping --> active: process_timeout + active --> await_callback: setup_callback tag+timer + await_callback --> active: process_callback + active --> finished_ok: {finished, success} + active --> finished_failed: {finished, {failed, _}} + finished_ok --> [*] + finished_failed --> [*] +``` + +`process_session/1` in +[`ff_withdrawal_session`](../apps/ff_transfer/src/ff_withdrawal_session.erl) +calls the configured adapter's +[`ProcessWithdrawal`](../apps/ff_transfer/src/ff_adapter_withdrawal.erl) +RPC. The adapter returns an `intent`: + +- `{finish, Status}` → session emits `{finished, Status}` and finishes. +- `{sleep, #{timer, callback_tag => ..., user_interaction => ...}}` → + the machine schedules a timeout (and optionally registers a tag so a + callback from the adapter can find this session). + +Callbacks arrive via the +[`ff_withdrawal_adapter_host`](../apps/ff_server/src/ff_withdrawal_adapter_host.erl) +Thrift endpoint (`ProcessCallback`) and are dispatched to the session by +`ff_withdrawal_session_machine:process_callback/1` — see +[adapter-integration.md](adapter-integration.md). + +### Retries + +Defined in [`ff_withdrawal_session_machine`](../apps/ff_transfer/src/ff_withdrawal_session_machine.erl) +as: + +- total retry time limit: 24 h +- max sleep between retries: 4 h + +When an adapter `ProcessWithdrawal` returns a transient failure, the +session schedules a retry rather than immediately failing out. + +## Identity and wallet machines — legacy + +`ff/identity` and `ff/wallet_v2` are still configured in +[sys.config:56‑77](../config/sys.config#L56) but their `.erl` sources +(`ff_identity_machine`, `ff_wallet_machine`) are **not** present in this +working tree. Live party/wallet data is served from `party-management` via +[`ff_party`](../apps/fistful/src/ff_party.erl). Historical identity/wallet +machines in existing PostgreSQL stores remain readable by the progressor +for backwards compatibility. + +## Serialization + +Each namespace has a paired **schema** module under +[apps/ff_server/src/](../apps/ff_server/src/) that implements +`machinery_mg_schema`: + +- Marshals `{event, Version}` changes through per‑entity Thrift structs + (`TimestampedChange`, etc.). +- Stores the `ctx :: ff_entity_context:context()` in `aux_state`. +- Handles migrations when an older event format is read back. + +See [persistence.md](persistence.md) for how that interacts with progressor. + +## Repair + +Every machine implements `process_repair/4` through +[`ff_repair:apply_scenario/3`](../apps/fistful/src/ff_repair.erl). Built‑in +support: the `add_events` scenario which appends arbitrary events to fix up +broken state. Entity‑specific handling lives in each Thrift repair handler +([`ff_withdrawal_repair`](../apps/ff_server/src/ff_withdrawal_repair.erl), +[`ff_deposit_repair`](../apps/ff_server/src/ff_deposit_repair.erl), +[`ff_withdrawal_session_repair`](../apps/ff_server/src/ff_withdrawal_session_repair.erl)). diff --git a/docs/withdrawal-flow.md b/docs/withdrawal-flow.md new file mode 100644 index 00000000..a88cf700 --- /dev/null +++ b/docs/withdrawal-flow.md @@ -0,0 +1,294 @@ +# Withdrawal Flow + +A withdrawal moves money *out* of a wallet to an external destination +through a provider. It is the most complex flow in fistful: it touches +routing, double‑entry accounting, external limiter reservations, an +adapter session machine, optional callbacks from the provider, retry +with route exhaustion, and — after the fact — optional adjustments. + +## Mental model + +1. A client calls `Create` on the withdrawal management service. +2. The withdrawal machine is persisted in `pending` status with minimal + fields — just enough to re‑derive everything deterministically. +3. Progressor wakes the machine via `process_timeout`. The machine walks + an **activity chain** until it either reaches `succeeded`, `failed`, + or is suspended waiting for a session / adapter callback. +4. Each activity is a deterministic function of the current + `ff_withdrawal:withdrawal_state()`. Idempotency is built in — the same + input events always produce the same next action. +5. Every money‑moving step is double‑booked: the *intent* is recorded as + a fistful event, and the actual debit/credit is delegated to + `shumway` via `ff_postings_transfer`. + +## Flowchart + +```mermaid +flowchart TD + Create[ff_withdrawal:create/1] -->|events| M[ff_withdrawal_machine] + M -->|process_timeout| A{deduce_activity/1} + A -->|routing| R[process_routing
ff_withdrawal_routing:prepare_routes] + R -->|route_changed
+ resource_got| A + A -->|p_transfer_start| PS[make_final_cash_flow
→ ff_postings_transfer:create] + PS --> A + A -->|p_transfer_prepare| PP[ff_postings_transfer:prepare
→ shumway Hold] + PP --> A + A -->|limit_check| LC[ff_limiter:check_limits
ff_party:validate_wallet_limits] + LC --> A + A -->|session_starting| SS[ff_withdrawal_session_machine:create] + SS --> A + A -->|session_sleeping| SSL[wait for session_finished notify] + SSL --> A + A -->|p_transfer_commit| PC[ff_postings_transfer:commit
→ shumway CommitPlan] + PC -->|limiter commit| A + A -->|p_transfer_cancel| PX[ff_postings_transfer:cancel
→ shumway RollbackPlan] + PX -->|limiter rollback| A + A -->|finish| OK[status_changed → succeeded] + A -->|fail| Fail[status_changed → failed] + A -->|rollback_routing| RR[rollback route limits] + RR --> Fail + A -->|adjustment| Adj[ff_adjustment_utils:process] +``` + +## Sequence diagram + +```mermaid +sequenceDiagram + autonumber + participant C as Client + participant H as ff_withdrawal_handler + participant M as ff_withdrawal_machine + participant W as ff_withdrawal + participant R as ff_withdrawal_routing + participant S as ff_withdrawal_session_machine + participant AD as Adapter provider + participant SH as shumway + participant LI as limiter + + C->>H: Create(params, ctx) + H->>M: machinery:start + M-->>H: ok + H-->>C: Get(id) + + Note over M: process_timeout loop starts + M->>W: process_transfer/1 + W->>R: prepare_routes(varset, context) + R->>LI: hold_withdrawal_limits + R-->>W: [routes] + W->>SH: accounter:Hold (posting plan) + SH-->>W: ok + W->>LI: refine / re-check + LI-->>W: ok + W->>S: machinery:start new session + S->>AD: ProcessWithdrawal + AD-->>S: {sleep, callback_tag, timer} + Note over AD,S: later ... + AD->>H: ProcessCallback(callback) + H->>S: process_callback + S->>AD: handle_callback + AD-->>S: {finish, success} + S-->>M: notify {session_finished, id, success} + M->>W: process_transfer/1 + W->>SH: accounter:CommitPlan + W->>LI: commit_withdrawal_limits + W-->>M: [{status_changed, succeeded}] +``` + +## Step detail + +### 1. Create + +Entry: [`ff_withdrawal_handler:handle_function('Create', ...)`](../apps/ff_server/src/ff_withdrawal_handler.erl#L69). + +Steps performed before the machine is started: + +1. Unmarshal params via + [`ff_withdrawal_codec:unmarshal_withdrawal_params/1`](../apps/ff_server/src/ff_withdrawal_codec.erl). +2. Scoper: `withdrawal` + params (`id`, `wallet_id`, `destination_id`, + `external_id`). +3. `ff_withdrawal_machine:create(Params, Context)` → + [`ff_withdrawal:create/1`](../apps/ff_transfer/src/ff_withdrawal.erl#L423) + inside `do/1`: + - Fetch party + wallet via `ff_party:get_party/1` and + `ff_party:get_wallet/3`. + - Fetch destination via `ff_destination_machine:get/1`. + - Ensure wallet accessibility (`is_wallet_accessible/1`). + - Validate currencies match (withdrawal vs wallet vs destination). + - Validate realms match (test/live). + - Call `ff_party:validate_withdrawal_creation/3` against domain terms: + allowed currency, cash range, allowed withdrawal method. + - Produce `[{created, Withdrawal}]` + `{resource_got, Resource}` if the + destination's resource needs BIN enrichment. + +The returned errors are mapped 1:1 onto Thrift business exceptions in +[ff_withdrawal_handler.erl:69‑113](../apps/ff_server/src/ff_withdrawal_handler.erl#L69). + +### 2. Routing (`process_routing`) + +[`ff_withdrawal:process_routing/1`](../apps/ff_transfer/src/ff_withdrawal.erl#L773) +delegates to +[`ff_withdrawal_routing:prepare_routes/2`](../apps/ff_transfer/src/ff_withdrawal_routing.erl#L72). + +1. Build a **varset** via `ff_varset` that includes the wallet, cash, + destination method, etc. — this is what selectors get reduced against. +2. Compute the **payment institution** for the wallet via + `ff_party:compute_payment_institution/3`. +3. Evaluate the PI's `withdrawal_routing_rules`: + - Apply policies routing ruleset → accepted routes. + - Apply prohibitions routing ruleset → subtract forbidden routes. +4. For each candidate route, evaluate the provider/terminal terms + (`ff_party:compute_provider_terminal_terms/4`) against the varset — + this filters by currency, method, cash ranges. +5. Use [`ff_limiter:check_limits/4`](../apps/ff_transfer/src/ff_limiter.erl#L39) + to filter out routes whose turnover limits would overflow. +6. Emit `{route_changed, Route}`. The first accepted route wins; the rest + are remembered via + [`ff_withdrawal_route_attempt_utils`](../apps/ff_transfer/src/ff_withdrawal_route_attempt_utils.erl) + so subsequent attempts can pick the next candidate. + +If nothing matches, `{error, {route, {route_not_found, [RejectedRoutes]}}}` +propagates back to the handler as a `WithdrawalSessionFailure`. The reject +context (why each route was dropped) is logged by +[`ff_routing_rule:log_reject_context/1`](../apps/fistful/src/ff_routing_rule.erl). + +### 3. Posting transfer start and prepare + +[`ff_withdrawal:make_final_cash_flow/1,2`](../apps/ff_transfer/src/ff_withdrawal.erl#L1009) +builds the plan cash flow from the provider's cash‑flow plan in DMT plus +provider/terminal fees, then calls `ff_cash_flow:finalize/3` to resolve +plan accounts (wallet's account ID on the debit side, provider settlement +account on the credit side, plus system settlement / subagent accounts for +fees). + +[`ff_postings_transfer:create/2`](../apps/fistful/src/ff_postings_transfer.erl#L77) +validates the accounts (same currency, same realm, accessible) and emits +`{created, Transfer}` + `{status_changed, created}`. + +`p_transfer_prepare` calls +[`ff_postings_transfer:prepare/1`](../apps/fistful/src/ff_postings_transfer.erl#L44) +which invokes +[`ff_accounting:prepare_trx/2`](../apps/fistful/src/ff_accounting.erl#L58) — +this is the `Hold` Thrift call on `shumway`. On success the status +transitions to `prepared`. **At this point** funds are reserved but not +yet released on either side. + +### 4. Limit check + +[`ff_withdrawal:process_limit_check/1`](../apps/ff_transfer/src/ff_withdrawal.erl#L882) +emits a `{limit_check, {wallet_sender, ok | {failed, _}}}` event. The +actual balance check is done by +[`ff_party:validate_wallet_limits/2`](../apps/fistful/src/ff_party.erl#L72): +the wallet's current balance (from shumway) plus the held amount must not +drop below the wallet's configured cash range. + +If the limit check fails, the activity table jumps to `p_transfer_cancel` +and then `{fail, limit_check}`; the resulting withdrawal status is +`{failed, {inner_failure, ...}}`. + +### 5. Session creation + +[`ff_withdrawal:process_session_creation/1`](../apps/ff_transfer/src/ff_withdrawal.erl#L919) +constructs a deterministic session ID +([`construct_session_id/1`](../apps/ff_transfer/src/ff_withdrawal.erl#L955)) +and starts a new session machine with +`ff_withdrawal_session_machine:create(ID, Data, Params)`. + +The session runs in its own machinery namespace and — importantly — in its +own machinery worker. It processes adapter RPCs, sleeps between retries, +handles callbacks, and notifies the owning withdrawal on finish. See +[adapter-integration.md](adapter-integration.md) for the RPC protocol. + +### 6. Session sleeping + +[`ff_withdrawal:process_session_sleep/1`](../apps/ff_transfer/src/ff_withdrawal.erl#L977) +returns `{sleep, []}` — the withdrawal machine stops processing and waits +for a notification. The session ultimately calls back through +[`ff_withdrawal_machine:notify/2`](../apps/ff_transfer/src/ff_withdrawal_machine.erl#L58) +with `{session_finished, SessionID, Result}`. `process_notification` in +the withdrawal machine feeds that into +[`ff_withdrawal:finalize_session/3`](../apps/ff_transfer/src/ff_withdrawal.erl#L573) +which appends a `{session_finished, ...}` event and wakes the withdrawal. + +### 7. Commit / cancel + +On session success: `p_transfer_commit` calls +[`ff_postings_transfer:commit/1`](../apps/fistful/src/ff_postings_transfer.erl#L45) +(shumway `CommitPlan`), then +[`ff_withdrawal_routing:commit_routes_limits/3`](../apps/ff_transfer/src/ff_withdrawal_routing.erl#L11) +for the external turnover limiter, then refreshes the wallet balance via +[`ff_party:wallet_log_balance/2`](../apps/fistful/src/ff_party.erl#L63). + +On session failure (or limit failure): `p_transfer_cancel` calls +[`ff_postings_transfer:cancel/1`](../apps/fistful/src/ff_postings_transfer.erl#L46) +(shumway `RollbackPlan`) and +[`ff_withdrawal_routing:rollback_routes_limits/3`](../apps/ff_transfer/src/ff_withdrawal_routing.erl#L11). + +If one route failed but other candidates remain, the machine enters a new +`routing` iteration (see +[`update_attempts/2`](../apps/ff_transfer/src/ff_withdrawal.erl#L631)) and +picks the next route — a fresh `{route_changed, _}`, a new session with a +distinct ID (the route iteration is folded into the session ID), fresh +hold/commit. + +### 8. Terminal state + +`{status_changed, succeeded}` or `{status_changed, {failed, failure()}}` +is emitted by `process_transfer_finish/1` or `process_transfer_fail/2` +([ff_withdrawal.erl:981](../apps/ff_transfer/src/ff_withdrawal.erl#L981)). + +Once finished, the only further activities are adjustments (see +[adjustments.md](adjustments.md)). + +## Quote flow (`GetQuote`) + +A withdrawal can be *quoted* without being created — useful to show the +user the expected exchange rate / fees before committing. + +[`ff_withdrawal:get_quote/1`](../apps/ff_transfer/src/ff_withdrawal.erl#L236) +runs the same party/wallet/destination/terms checks as `Create`, selects a +route, and asks the provider's adapter for a quote via +[`ff_adapter_withdrawal:get_quote/3`](../apps/ff_transfer/src/ff_adapter_withdrawal.erl). +The result is a sealed `quote()` (with `created_at`, `expires_on`, +`quote_data`, `route`, `resource_descriptor`) that can later be passed back +into `Create` — the withdrawal will bypass routing and session‑ID +derivation, instead using the quote's frozen route and resource. + +## Error paths and exceptions + +| Domain error | Thrift exception raised | +|--------------|-------------------------| +| `{party, notfound}` | `#fistful_PartyNotFound{}` | +| `{wallet, notfound}` | `#fistful_WalletNotFound{}` | +| `{wallet, {inaccessible, _}}` | `#fistful_WalletInaccessible{id}` | +| `{destination, notfound}` | `#fistful_DestinationNotFound{}` | +| `{destination, unauthorized}` | `#fistful_DestinationUnauthorized{}` | +| `{terms, {terms_violation, {not_allowed_currency, _}}}` | `#fistful_ForbiddenOperationCurrency{}` | +| `{terms, {terms_violation, {cash_range, _}}}` | `#fistful_ForbiddenOperationAmount{}` | +| `{terms, {terms_violation, {not_allowed_withdrawal_method, _}}}` | `#fistful_ForbiddenWithdrawalMethod{}` | +| `{inconsistent_currency, {...}}` | `#wthd_InconsistentWithdrawalCurrency{}` | +| `{realms_mismatch, {...}}` | `#fistful_RealmsMismatch{}` | + +The full mapping lives in +[ff_withdrawal_handler.erl:69‑113](../apps/ff_server/src/ff_withdrawal_handler.erl#L69). + +## Idempotence + +- **Create** — a second call with the same `id` returns the existing + withdrawal's state (`{error, exists}` is treated as success by the + handler; see [ff_withdrawal_handler.erl:77](../apps/ff_server/src/ff_withdrawal_handler.erl#L77)). +- **Posting transfer** — the transfer ID is derived deterministically from + the withdrawal's route and iteration via + [`construct_p_transfer_id/1`](../apps/ff_transfer/src/ff_withdrawal.erl#L962). + shumway deduplicates Hold/Commit on this ID. +- **Session** — session ID is constructed from withdrawal ID + route + + iteration. A crashed worker restarting does not create a second session. +- **Limiter holds** — idempotent on `(limit_id, operation_id)` where + `operation_id` embeds provider, terminal, withdrawal, iteration (see + [`ff_limiter:make_operation_segments/3`](../apps/ff_transfer/src/ff_limiter.erl#L52)). + +> [!WARNING] +> There's a known gap: `GetQuote` has no handler clause for +> `{error, {route, {route_not_found, _}}}` (see TODO at +> [ff_withdrawal_handler.erl:35](../apps/ff_server/src/ff_withdrawal_handler.erl#L35)). +> A quote request against an unroutable pair will surface as a generic +> Woody system error rather than a typed business error.