From 6fd4727bf7e413b00dd1770b644bf3f83a4d7bf6 Mon Sep 17 00:00:00 2001 From: Diffora Date: Tue, 7 Jul 2026 15:31:40 +0200 Subject: [PATCH] feat(credstore): stateful credential-storage gateway with registry-driven secret types MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Port the credstore gear: a tenant-scoped, PDP-enforced credential gateway over pluggable value-store backends, with hierarchical resolution, a crash-safe saga lifecycle, and dynamic GTS secret typing. Gateway & domain - Stateful metadata gateway (credstore_secrets) over a trait-agnostic value-store plugin SPI; hierarchical read resolves the closest accessible secret up the barrier-respecting tenant ancestor chain (private > tenant > shared), reading the backend once for the winner. - Crash-safe write/delete sagas (provisioning / active / deprovisioning) with a background reaper that reconciles orphaned backend values, releases wedged references, and sweeps expired secrets; optimistic concurrency via If-Match / version. Authorization - Single PDP evaluation per operation on the secret's full concrete GTS type (incl. generic), gated on the caller's tenant; hierarchical visibility decided by the resolver, not the PDP. GET denial/miss is an anti-enumeration 404; PDP outage is 503. Registry-driven secret types - Secret types are GTS types derived from gts.cf.core.credstore.secret.v1~; the types-registry is the runtime source of truth (SecretTypeResolver resolves a type's id + effective traits per operation, no local cache), so custom types register a GTS schema with no credstore release. The compiled-in catalog only seeds the built-in type schemas. - Enforced traits (allow_sharing, value_schema, max_size_bytes, expirable, utf8_only, advisory rotation_period_secs) validated on write against the resolved traits; type is immutable for a secret's lifetime. - Storage: secret_type_uuid UUID (deterministic v5 of the GTS id), default pinned to the generic type. SDK/REST accept a type as catalog short name, full GTS id, or UUID (SecretTypeRef); responses carry the resolved id. - Fail-closed: unknown/non-secret type -> 400 UNKNOWN_SECRET_TYPE on create, 503 for a stored row; registry outage/timeout/malformed traits -> 503. Value-fingerprint fence & optimistic-concurrency hardening - Each row carries value_fp = HMAC-SHA256(fence_key, value), stamped in the same atomic write as the sharing label; reads verify it against the backend value and fail closed (anti-enumeration 404) on mismatch, so two crosswise concurrent last-writer-wins PUTs can never serve one writer's value under another writer's sharing label (closes the cross-tenant disclosure). The fence key is auto-generated and stored in the backend under an API-unreachable reserved entry (split knowledge; the fingerprint never leaves the gateway, in responses or logs). - Strong ETag is generation-bound ".": a validator from a deleted-and-recreated secret's earlier generation never matches even when the version counters coincide (no ABA lost update). - Out-of-band seeded rows (value_fp NULL) are served on trust and backfilled lazily on read and by the reaper sweep. See DESIGN §4.10 and ADR-0003. Integration & docs - OAGW and mini-chat consume credstore through the SDK client; e2e and integration coverage; DESIGN.md, generated OpenAPI spec, and ADRs. Signed-off-by: Diffora --- .github/workflows/cfs.yml | 10 +- Cargo.lock | 19 +- config/e2e-local.yaml | 9 + config/e2e-scope-enforcement.yaml | 9 + config/e2e-tr-authz.yaml | 9 + config/mini-chat.yaml | 9 + config/quickstart-windows.yaml | 9 + config/quickstart.yaml | 9 + config/static-tenants.yaml | 9 + docs/api/api.json | 557 +++ gears/credstore/QUICKSTART.md | 2 +- gears/credstore/credstore-sdk/Cargo.toml | 6 + gears/credstore/credstore-sdk/README.md | 8 +- gears/credstore/credstore-sdk/src/api.rs | 115 +- gears/credstore/credstore-sdk/src/error.rs | 196 +- .../credstore-sdk/src/error_tests.rs | 22 - gears/credstore/credstore-sdk/src/gts.rs | 275 +- .../credstore/credstore-sdk/src/gts_tests.rs | 106 + gears/credstore/credstore-sdk/src/lib.rs | 42 +- gears/credstore/credstore-sdk/src/models.rs | 198 +- .../credstore-sdk/src/models_tests.rs | 114 - .../credstore/credstore-sdk/src/plugin_api.rs | 33 +- .../credstore/credstore-sdk/src/test_util.rs | 153 + gears/credstore/credstore-sdk/src/types.rs | 357 ++ .../credstore-sdk/src/types_tests.rs | 117 + gears/credstore/credstore/Cargo.toml | 51 +- gears/credstore/credstore/README.md | 54 +- gears/credstore/credstore/src/api.rs | 2 + gears/credstore/credstore/src/api/rest.rs | 6 + gears/credstore/credstore/src/api/rest/dto.rs | 203 + .../credstore/src/api/rest/dto_tests.rs | 112 + .../credstore/src/api/rest/handlers.rs | 267 ++ .../credstore/src/api/rest/routes.rs | 136 + .../credstore/src/api/rest/routes_tests.rs | 484 +++ gears/credstore/credstore/src/client.rs | 145 + gears/credstore/credstore/src/client_tests.rs | 249 ++ gears/credstore/credstore/src/config.rs | 175 +- gears/credstore/credstore/src/config_tests.rs | 24 - gears/credstore/credstore/src/domain.rs | 5 + gears/credstore/credstore/src/domain/authz.rs | 66 + gears/credstore/credstore/src/domain/error.rs | 149 +- .../credstore/src/domain/error_tests.rs | 162 - .../credstore/src/domain/local_client.rs | 58 - .../src/domain/local_client_tests.rs | 93 - gears/credstore/credstore/src/domain/mod.rs | 11 - gears/credstore/credstore/src/domain/ports.rs | 2 + .../credstore/src/domain/ports/metrics.rs | 195 + .../credstore/src/domain/ports/plugin.rs | 12 + .../credstore/src/domain/resolver.rs | 17 + .../credstore/credstore/src/domain/secret.rs | 8 + .../credstore/src/domain/secret/fence.rs | 128 + .../credstore/src/domain/secret/model.rs | 186 + .../credstore/src/domain/secret/repo.rs | 149 + .../credstore/src/domain/secret/service.rs | 1725 +++++++++ .../src/domain/secret/service_tests.rs | 3418 +++++++++++++++++ .../src/domain/secret/test_support.rs | 1135 ++++++ .../src/domain/secret/type_resolver.rs | 59 + .../credstore/src/domain/secret/typing.rs | 162 + .../src/domain/secret/typing_tests.rs | 158 + .../credstore/credstore/src/domain/service.rs | 130 - .../credstore/src/domain/service_tests.rs | 274 -- .../credstore/src/domain/test_support.rs | 85 - gears/credstore/credstore/src/gear.rs | 234 +- gears/credstore/credstore/src/infra.rs | 9 + .../credstore/src/infra/canonical_mapping.rs | 103 + .../credstore/src/infra/error_conv.rs | 95 + .../credstore/credstore/src/infra/metrics.rs | 332 ++ .../credstore/src/infra/metrics_tests.rs | 156 + .../credstore/src/infra/plugin_select.rs | 97 + .../credstore/src/infra/sdk_error_mapping.rs | 150 + .../credstore/credstore/src/infra/storage.rs | 4 + .../credstore/src/infra/storage/entity.rs | 3 + .../src/infra/storage/entity/secrets.rs | 48 + .../infra/storage/entity/tenant_closure.rs | 36 + .../credstore/src/infra/storage/migrations.rs | 20 + .../migrations/m0001_initial_schema.rs | 117 + .../credstore/src/infra/storage/repo_impl.rs | 219 ++ .../src/infra/storage/repo_impl/helpers.rs | 58 + .../src/infra/storage/repo_impl/reads.rs | 338 ++ .../src/infra/storage/repo_impl/repo_tests.rs | 1298 +++++++ .../src/infra/storage/repo_impl/writes.rs | 384 ++ .../credstore/src/infra/tenant_resolver.rs | 141 + .../src/infra/tenant_resolver_tests.rs | 202 + .../credstore/src/infra/types_registry.rs | 187 + .../src/infra/types_registry_tests.rs | 328 ++ gears/credstore/credstore/src/lib.rs | 15 +- ...0001-cpt-cf-credstore-adr-stateful-gear.md | 115 + ...pt-cf-credstore-adr-deprovisioning-saga.md | 110 + ...f-credstore-adr-value-fingerprint-fence.md | 153 + gears/credstore/docs/DESIGN.md | 1546 ++++---- gears/credstore/docs/PRD.md | 524 +-- gears/credstore/docs/api/openapi.yaml | 432 +++ .../plugins/static-credstore-plugin/README.md | 75 +- .../static-credstore-plugin/src/config.rs | 2 +- .../src/domain/client.rs | 57 +- .../src/domain/client_tests.rs | 254 +- .../src/domain/service.rs | 200 +- .../src/domain/service_tests.rs | 874 +---- gears/mini-chat/mini-chat/src/config.rs | 92 +- gears/mini-chat/mini-chat/src/gear.rs | 142 +- .../mini-chat/src/infra/oagw_provisioning.rs | 856 ++++- gears/system/oagw/oagw/Cargo.toml | 2 + .../oagw/oagw/src/domain/test_support.rs | 74 +- .../oagw/oagw/src/infra/plugin/apikey_auth.rs | 32 +- .../infra/plugin/oauth2_client_cred_auth.rs | 32 +- .../oagw/oagw/src/infra/proxy/service.rs | 17 +- testing/e2e/gears/credstore/__init__.py | 0 testing/e2e/gears/credstore/conftest.py | 118 + .../e2e/gears/credstore/test_concurrency.py | 342 ++ testing/e2e/gears/credstore/test_crud.py | 204 + testing/e2e/gears/credstore/test_sharing.py | 250 ++ testing/e2e/gears/credstore/test_types.py | 292 ++ testing/e2e/gears/mini_chat/config/base.yaml | 9 + testing/e2e/gears/mini_chat/conftest.py | 134 + testing/e2e/gears/oagw/conftest.py | 67 + 115 files changed, 20864 insertions(+), 3403 deletions(-) delete mode 100644 gears/credstore/credstore-sdk/src/error_tests.rs create mode 100644 gears/credstore/credstore-sdk/src/gts_tests.rs delete mode 100644 gears/credstore/credstore-sdk/src/models_tests.rs create mode 100644 gears/credstore/credstore-sdk/src/test_util.rs create mode 100644 gears/credstore/credstore-sdk/src/types.rs create mode 100644 gears/credstore/credstore-sdk/src/types_tests.rs create mode 100644 gears/credstore/credstore/src/api.rs create mode 100644 gears/credstore/credstore/src/api/rest.rs create mode 100644 gears/credstore/credstore/src/api/rest/dto.rs create mode 100644 gears/credstore/credstore/src/api/rest/dto_tests.rs create mode 100644 gears/credstore/credstore/src/api/rest/handlers.rs create mode 100644 gears/credstore/credstore/src/api/rest/routes.rs create mode 100644 gears/credstore/credstore/src/api/rest/routes_tests.rs create mode 100644 gears/credstore/credstore/src/client.rs create mode 100644 gears/credstore/credstore/src/client_tests.rs delete mode 100644 gears/credstore/credstore/src/config_tests.rs create mode 100644 gears/credstore/credstore/src/domain.rs create mode 100644 gears/credstore/credstore/src/domain/authz.rs delete mode 100644 gears/credstore/credstore/src/domain/error_tests.rs delete mode 100644 gears/credstore/credstore/src/domain/local_client.rs delete mode 100644 gears/credstore/credstore/src/domain/local_client_tests.rs delete mode 100644 gears/credstore/credstore/src/domain/mod.rs create mode 100644 gears/credstore/credstore/src/domain/ports.rs create mode 100644 gears/credstore/credstore/src/domain/ports/metrics.rs create mode 100644 gears/credstore/credstore/src/domain/ports/plugin.rs create mode 100644 gears/credstore/credstore/src/domain/resolver.rs create mode 100644 gears/credstore/credstore/src/domain/secret.rs create mode 100644 gears/credstore/credstore/src/domain/secret/fence.rs create mode 100644 gears/credstore/credstore/src/domain/secret/model.rs create mode 100644 gears/credstore/credstore/src/domain/secret/repo.rs create mode 100644 gears/credstore/credstore/src/domain/secret/service.rs create mode 100644 gears/credstore/credstore/src/domain/secret/service_tests.rs create mode 100644 gears/credstore/credstore/src/domain/secret/test_support.rs create mode 100644 gears/credstore/credstore/src/domain/secret/type_resolver.rs create mode 100644 gears/credstore/credstore/src/domain/secret/typing.rs create mode 100644 gears/credstore/credstore/src/domain/secret/typing_tests.rs delete mode 100644 gears/credstore/credstore/src/domain/service.rs delete mode 100644 gears/credstore/credstore/src/domain/service_tests.rs delete mode 100644 gears/credstore/credstore/src/domain/test_support.rs create mode 100644 gears/credstore/credstore/src/infra.rs create mode 100644 gears/credstore/credstore/src/infra/canonical_mapping.rs create mode 100644 gears/credstore/credstore/src/infra/error_conv.rs create mode 100644 gears/credstore/credstore/src/infra/metrics.rs create mode 100644 gears/credstore/credstore/src/infra/metrics_tests.rs create mode 100644 gears/credstore/credstore/src/infra/plugin_select.rs create mode 100644 gears/credstore/credstore/src/infra/sdk_error_mapping.rs create mode 100644 gears/credstore/credstore/src/infra/storage.rs create mode 100644 gears/credstore/credstore/src/infra/storage/entity.rs create mode 100644 gears/credstore/credstore/src/infra/storage/entity/secrets.rs create mode 100644 gears/credstore/credstore/src/infra/storage/entity/tenant_closure.rs create mode 100644 gears/credstore/credstore/src/infra/storage/migrations.rs create mode 100644 gears/credstore/credstore/src/infra/storage/migrations/m0001_initial_schema.rs create mode 100644 gears/credstore/credstore/src/infra/storage/repo_impl.rs create mode 100644 gears/credstore/credstore/src/infra/storage/repo_impl/helpers.rs create mode 100644 gears/credstore/credstore/src/infra/storage/repo_impl/reads.rs create mode 100644 gears/credstore/credstore/src/infra/storage/repo_impl/repo_tests.rs create mode 100644 gears/credstore/credstore/src/infra/storage/repo_impl/writes.rs create mode 100644 gears/credstore/credstore/src/infra/tenant_resolver.rs create mode 100644 gears/credstore/credstore/src/infra/tenant_resolver_tests.rs create mode 100644 gears/credstore/credstore/src/infra/types_registry.rs create mode 100644 gears/credstore/credstore/src/infra/types_registry_tests.rs create mode 100644 gears/credstore/docs/ADR/0001-cpt-cf-credstore-adr-stateful-gear.md create mode 100644 gears/credstore/docs/ADR/0002-cpt-cf-credstore-adr-deprovisioning-saga.md create mode 100644 gears/credstore/docs/ADR/0003-cpt-cf-credstore-adr-value-fingerprint-fence.md create mode 100644 gears/credstore/docs/api/openapi.yaml create mode 100644 testing/e2e/gears/credstore/__init__.py create mode 100644 testing/e2e/gears/credstore/conftest.py create mode 100644 testing/e2e/gears/credstore/test_concurrency.py create mode 100644 testing/e2e/gears/credstore/test_crud.py create mode 100644 testing/e2e/gears/credstore/test_sharing.py create mode 100644 testing/e2e/gears/credstore/test_types.py diff --git a/.github/workflows/cfs.yml b/.github/workflows/cfs.yml index 7d3075a48..b998e716f 100644 --- a/.github/workflows/cfs.yml +++ b/.github/workflows/cfs.yml @@ -65,8 +65,14 @@ jobs: base_ref="${{ github.base_ref }}" if [ "${{ github.event_name }}" = "pull_request" ] && [ -n "$base_ref" ]; then - git fetch --no-tags --depth=1 origin "$base_ref" - changed_files="$(git diff --name-only "origin/$base_ref"...HEAD)" + # Full (not --depth=1) fetch so the three-dot diff can find a merge + # base: checkout uses fetch-depth: 0 (full HEAD), so a shallow base + # ref has no reachable common ancestor and the diff fails with + # "no merge base" (notably on fork PRs). Tolerate a failed fetch and + # a failed diff so this detection step never hard-fails the job (the + # run shell is `bash -eo pipefail`, so an unguarded fetch would exit). + git fetch --no-tags origin "$base_ref" || true + changed_files="$(git diff --name-only "origin/$base_ref"...HEAD 2>/dev/null || true)" else changed_files="$( git diff --name-only HEAD~1 HEAD 2>/dev/null || true diff --git a/Cargo.lock b/Cargo.lock index 9774503ad..281988076 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1681,20 +1681,35 @@ version = "0.1.24" dependencies = [ "anyhow", "async-trait", + "aws-lc-rs", + "axum", + "cf-gears-authz-resolver-sdk", "cf-gears-credstore-sdk", + "cf-gears-tenant-resolver-sdk", "cf-gears-toolkit", "cf-gears-toolkit-canonical-errors", + "cf-gears-toolkit-db", "cf-gears-toolkit-gts", "cf-gears-toolkit-macros", "cf-gears-toolkit-security", "cf-gears-types-registry-sdk", - "inventory", + "jsonschema", + "opentelemetry", + "opentelemetry_sdk", + "rand 0.10.1", + "sea-orm", + "sea-orm-migration", "serde", "serde_json", "thiserror 2.0.18", + "time", "tokio", + "tokio-util", + "tower", "tracing", + "utoipa", "uuid", + "zeroize", ] [[package]] @@ -1707,10 +1722,12 @@ dependencies = [ "cf-gears-toolkit-gts", "cf-gears-toolkit-security", "gts", + "gts-macros", "schemars 1.2.1", "serde", "serde_json", "thiserror 2.0.18", + "time", "uuid", "zeroize", ] diff --git a/config/e2e-local.yaml b/config/e2e-local.yaml index 3a77bd992..107e12e42 100644 --- a/config/e2e-local.yaml +++ b/config/e2e-local.yaml @@ -196,6 +196,15 @@ gears: config: vendor: "constructorfabric" + credstore: + database: + server: "sqlite_users" + file: "credstore.db" + config: + # Gateway selects its value-store backend plugin by GTS vendor; + # must match static-credstore-plugin (default "constructorfabric"). + vendor: "constructorfabric" + static-credstore-plugin: config: secrets: diff --git a/config/e2e-scope-enforcement.yaml b/config/e2e-scope-enforcement.yaml index 18c52a0fa..9edb3fd96 100644 --- a/config/e2e-scope-enforcement.yaml +++ b/config/e2e-scope-enforcement.yaml @@ -25,6 +25,15 @@ logging: file_level: debug gears: + credstore: + database: + server: "sqlite_users" + file: "credstore.db" + config: + # Gateway selects its value-store backend plugin by GTS vendor; + # must match static-credstore-plugin (default "constructorfabric"). + vendor: "constructorfabric" + api-gateway: config: bind_addr: "0.0.0.0:8086" diff --git a/config/e2e-tr-authz.yaml b/config/e2e-tr-authz.yaml index 73d294e7c..22cd63926 100644 --- a/config/e2e-tr-authz.yaml +++ b/config/e2e-tr-authz.yaml @@ -177,6 +177,15 @@ gears: config: vendor: "constructorfabric" + credstore: + database: + server: "sqlite_users" + file: "credstore.db" + config: + # Gateway selects its value-store backend plugin by GTS vendor; + # must match static-credstore-plugin (default "constructorfabric"). + vendor: "constructorfabric" + static-credstore-plugin: config: secrets: diff --git a/config/mini-chat.yaml b/config/mini-chat.yaml index e4147e691..d13ac5a3e 100644 --- a/config/mini-chat.yaml +++ b/config/mini-chat.yaml @@ -101,6 +101,15 @@ gears: config: max_field_length: 100 + credstore: + database: + server: "sqlite_mini_chat" + file: "credstore.db" + config: + # Gateway selects its value-store backend plugin by GTS vendor; + # must match static-credstore-plugin (default "constructorfabric"). + vendor: "constructorfabric" + static-credstore-plugin: config: secrets: diff --git a/config/quickstart-windows.yaml b/config/quickstart-windows.yaml index 4d14fa6b5..e53667a7c 100644 --- a/config/quickstart-windows.yaml +++ b/config/quickstart-windows.yaml @@ -211,6 +211,15 @@ gears: parent_id: "00000000-0000-0000-0000-000000000011" status: active + credstore: + database: + server: "sqlite_users" + file: "credstore.db" + config: + # Gateway selects its value-store backend plugin by GTS vendor; + # must match static-credstore-plugin (default "constructorfabric"). + vendor: "constructorfabric" + static-credstore-plugin: config: secrets: diff --git a/config/quickstart.yaml b/config/quickstart.yaml index d14da84f6..72bdf9418 100644 --- a/config/quickstart.yaml +++ b/config/quickstart.yaml @@ -330,6 +330,15 @@ gears: parent_id: "00000000-0000-0000-0000-000000000005" status: active + credstore: + database: + server: "sqlite_users" + file: "credstore.db" + config: + # Gateway selects its value-store backend plugin by GTS vendor; + # must match static-credstore-plugin (default "constructorfabric"). + vendor: "constructorfabric" + static-credstore-plugin: config: secrets: diff --git a/config/static-tenants.yaml b/config/static-tenants.yaml index 30b3e579f..7a83e44ad 100644 --- a/config/static-tenants.yaml +++ b/config/static-tenants.yaml @@ -19,6 +19,15 @@ database: max_conns: 5 gears: + credstore: + database: + server: "sqlite_test" + file: "credstore.db" + config: + # Gateway selects its value-store backend plugin by GTS vendor; + # must match static-credstore-plugin (default "constructorfabric"). + vendor: "constructorfabric" + api-gateway: config: bind_addr: "127.0.0.1:8087" diff --git a/docs/api/api.json b/docs/api/api.json index 5bac53c9f..8ea91ec53 100644 --- a/docs/api/api.json +++ b/docs/api/api.json @@ -790,6 +790,47 @@ ], "type": "object" }, + "CreateSecretRequestDto": { + "additionalProperties": false, + "description": "Request body for `POST /credstore/v1/secrets`.\n\n`Debug` is hand-written to redact `value` — a derived `Debug` would expose\nthe plaintext secret if this DTO is ever `{:?}`-logged by a future layer.", + "properties": { + "expires_at": { + "description": "Expiry instant (RFC 3339); only for expirable types.", + "format": "date-time", + "type": [ + "string", + "null" + ] + }, + "reference": { + "description": "Secret reference key — `[a-zA-Z0-9_-]+`, max 255 characters.", + "maxLength": 255, + "minLength": 1, + "pattern": "^[A-Za-z0-9_-]+$", + "type": "string" + }, + "sharing": { + "$ref": "#/components/schemas/SharingModeDto", + "description": "Sharing mode for the secret." + }, + "type": { + "description": "Secret type as a full GTS type id (built-in or custom); defaults to\nthe generic type. Resolved and existence-checked against the\ntypes-registry.", + "type": [ + "string", + "null" + ] + }, + "value": { + "description": "Secret value as a UTF-8 string.", + "type": "string" + } + }, + "required": [ + "reference", + "value" + ], + "type": "object" + }, "CreateTypeDto": { "description": "REST DTO for creating a new GTS type.", "properties": { @@ -1240,6 +1281,24 @@ ], "type": "object" }, + "GetSecretResponseDto": { + "description": "Response body for `GET /credstore/v1/secrets/{ref}`.\n\n`Debug` is hand-written to redact `value` (see [`CreateSecretRequestDto`]).", + "properties": { + "metadata": { + "$ref": "#/components/schemas/SecretMetadataDto", + "description": "Access metadata for the resolved secret." + }, + "value": { + "description": "Secret value as a UTF-8 string.", + "type": "string" + } + }, + "required": [ + "value", + "metadata" + ], + "type": "object" + }, "GpuInfoDto": { "properties": { "cores": { @@ -4084,6 +4143,49 @@ ], "type": "string" }, + "SecretMetadataDto": { + "description": "Access metadata returned alongside the secret value.", + "properties": { + "expires_at": { + "description": "Expiry instant (RFC 3339), when set.", + "format": "date-time", + "type": [ + "string", + "null" + ] + }, + "is_inherited": { + "description": "Whether the secret came from an ancestor tenant.", + "type": "boolean" + }, + "owner_tenant_id": { + "description": "The tenant that owns this secret.", + "format": "uuid", + "type": "string" + }, + "sharing": { + "$ref": "#/components/schemas/SharingModeDto", + "description": "The sharing mode that governed the lookup result." + }, + "type": { + "description": "Secret type as its full GTS type id.", + "type": "string" + }, + "version": { + "description": "Monotonic version of the resolved secret (also returned as `ETag`).", + "format": "int64", + "type": "integer" + } + }, + "required": [ + "owner_tenant_id", + "sharing", + "is_inherited", + "version", + "type" + ], + "type": "object" + }, "Server": { "properties": { "endpoints": { @@ -4144,6 +4246,15 @@ ], "type": "string" }, + "SharingModeDto": { + "description": "Sharing mode for the REST transport layer.", + "enum": [ + "private", + "tenant", + "shared" + ], + "type": "string" + }, "SimpleUserSettingsDto": { "properties": { "language": { @@ -4903,6 +5014,46 @@ ], "type": "object" }, + "UpdateSecretRequestDto": { + "additionalProperties": false, + "description": "Request body for `PUT /credstore/v1/secrets/{ref}`.\n\n`Debug` is hand-written to redact `value` (see [`CreateSecretRequestDto`]).", + "properties": { + "expires_at": { + "description": "Expiry instant (RFC 3339); only for expirable types. A PUT is a\nwhole-value replace: omitting `expires_at` clears a stored expiry.", + "format": "date-time", + "type": [ + "string", + "null" + ] + }, + "sharing": { + "oneOf": [ + { + "type": "null" + }, + { + "$ref": "#/components/schemas/SharingModeDto", + "description": "Sharing mode for the secret. **Optional: when omitted on an overwrite\nthe secret's current sharing is preserved**, so rotating a `shared`\nsecret's value with `{\"value\": \"...\"}` alone no longer silently narrows\nit back to `tenant` (review finding #8). On create-via-upsert an omitted\n`sharing` defaults to `tenant`. Send an explicit value to change the\nsharing mode." + } + ] + }, + "type": { + "description": "Secret type as a full GTS type id. Optional; when present must match\nthe existing secret's type (the type is immutable). On create via\nupsert it selects the new secret's type (default: the generic type).", + "type": [ + "string", + "null" + ] + }, + "value": { + "description": "Secret value as a UTF-8 string.", + "type": "string" + } + }, + "required": [ + "value" + ], + "type": "object" + }, "UpdateSimpleUserSettingsRequest": { "properties": { "language": { @@ -11217,6 +11368,412 @@ ] } }, + "/credstore/v1/secrets": { + "post": { + "description": "Create a new secret for the authenticated tenant.", + "operationId": "credstore.create_secret", + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/CreateSecretRequestDto" + } + } + }, + "description": "Secret reference, value, and sharing mode", + "required": true + }, + "responses": { + "201": { + "description": "Secret created (see Location header)" + }, + "400": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Bad Request" + }, + "401": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Unauthorized" + }, + "403": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Forbidden" + }, + "409": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Conflict" + }, + "500": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Internal Server Error" + }, + "503": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Service Unavailable" + } + }, + "security": [ + { + "bearerAuth": [] + } + ], + "summary": "Create a secret", + "tags": [ + "Credential Store" + ] + } + }, + "/credstore/v1/secrets/{ref}": { + "delete": { + "description": "Delete a secret owned by the authenticated tenant.", + "operationId": "credstore.delete_secret", + "parameters": [ + { + "description": "Secret reference (`[a-zA-Z0-9_-]+`, maximum length 255 characters)", + "in": "path", + "name": "ref", + "required": true, + "schema": { + "type": "string" + } + }, + { + "description": "Optimistic-concurrency precondition (RFC 7232). `*` requires the secret to exist; a quoted `\".\"` ETag requires the current version to match, otherwise the request fails with `409 OPTIMISTIC_LOCK_FAILURE`.", + "in": "header", + "name": "If-Match", + "required": false, + "schema": { + "type": "string" + } + } + ], + "responses": { + "204": { + "description": "Secret deleted" + }, + "400": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Bad Request" + }, + "401": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Unauthorized" + }, + "403": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Forbidden" + }, + "404": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Not Found" + }, + "409": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Conflict" + }, + "500": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Internal Server Error" + }, + "503": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Service Unavailable" + } + }, + "security": [ + { + "bearerAuth": [] + } + ], + "summary": "Delete a secret by reference", + "tags": [ + "Credential Store" + ] + }, + "get": { + "description": "Retrieve a secret for the authenticated tenant, with walk-up resolution.", + "operationId": "credstore.get_secret", + "parameters": [ + { + "description": "Secret reference (`[a-zA-Z0-9_-]+`, maximum length 255 characters)", + "in": "path", + "name": "ref", + "required": true, + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/GetSecretResponseDto" + } + } + }, + "description": "Resolved secret value and metadata" + }, + "400": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Bad Request" + }, + "401": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Unauthorized" + }, + "403": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Forbidden" + }, + "404": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Not Found" + }, + "500": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Internal Server Error" + }, + "503": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Service Unavailable" + } + }, + "security": [ + { + "bearerAuth": [] + } + ], + "summary": "Get a secret by reference", + "tags": [ + "Credential Store" + ] + }, + "put": { + "description": "Store a secret for the authenticated tenant.", + "operationId": "credstore.put_secret", + "parameters": [ + { + "description": "Secret reference (`[a-zA-Z0-9_-]+`, maximum length 255 characters)", + "in": "path", + "name": "ref", + "required": true, + "schema": { + "type": "string" + } + }, + { + "description": "Optimistic-concurrency precondition (RFC 7232). `*` requires the secret to exist; a quoted `\".\"` ETag requires the current version to match, otherwise the request fails with `409 OPTIMISTIC_LOCK_FAILURE`.", + "in": "header", + "name": "If-Match", + "required": false, + "schema": { + "type": "string" + } + } + ], + "requestBody": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/UpdateSecretRequestDto" + } + } + }, + "description": "Secret value and sharing mode", + "required": true + }, + "responses": { + "204": { + "description": "Secret stored" + }, + "400": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Bad Request" + }, + "401": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Unauthorized" + }, + "403": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Forbidden" + }, + "409": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Conflict" + }, + "500": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Internal Server Error" + }, + "503": { + "content": { + "application/problem+json": { + "schema": { + "$ref": "#/components/schemas/Problem" + } + } + }, + "description": "Service Unavailable" + } + }, + "security": [ + { + "bearerAuth": [] + } + ], + "summary": "Create or update a secret by reference", + "tags": [ + "Credential Store" + ] + } + }, "/file-parser/v1/info": { "get": { "operationId": "file_parser.get_parser_info", diff --git a/gears/credstore/QUICKSTART.md b/gears/credstore/QUICKSTART.md index 9ee6fb5d7..fb967c410 100644 --- a/gears/credstore/QUICKSTART.md +++ b/gears/credstore/QUICKSTART.md @@ -15,7 +15,7 @@ Stores, retrieves, and deletes secrets scoped to tenants and owners. Secrets are Full API documentation: -The example server uses the gateway prefix `/cf`. This comes from `gearss.api-gateway.config.prefix_path` and is configurable. +The example server uses the gear prefix `/cf`. This comes from `gears.api-gateway.config.prefix_path` and is configurable. ## Configuration diff --git a/gears/credstore/credstore-sdk/Cargo.toml b/gears/credstore/credstore-sdk/Cargo.toml index a36a8c79b..6a5386a8c 100644 --- a/gears/credstore/credstore-sdk/Cargo.toml +++ b/gears/credstore/credstore-sdk/Cargo.toml @@ -25,10 +25,12 @@ uuid = { workspace = true } zeroize = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } +time = { workspace = true } schemars = { workspace = true } # GTS types gts = { workspace = true } +gts-macros = { workspace = true } toolkit-gts = { workspace = true } # ToolKit dependencies @@ -37,3 +39,7 @@ toolkit-security = { workspace = true } # Domain types owned by tenant-resolver tenant-resolver-sdk = { workspace = true } + +[features] +# Centralized `CredStoreClientV1` test double for downstream gears' tests. +test-util = [] diff --git a/gears/credstore/credstore-sdk/README.md b/gears/credstore/credstore-sdk/README.md index b23344489..b9259bd7b 100644 --- a/gears/credstore/credstore-sdk/README.md +++ b/gears/credstore/credstore-sdk/README.md @@ -6,8 +6,12 @@ SDK crate for the CredStore gear, providing public API contracts for credential This crate defines the transport-agnostic interface for the CredStore gear: -- **`CredStoreClientV1`** — Async trait for consumers (get/put/delete secrets) -- **`CredStorePluginClientV1`** — Async trait for backend storage plugin implementations +- **`CredStoreClientV1`** — consumer-facing trait (`get`/`put`/`create`/`delete`); + `get` returns the value plus metadata (`owner_tenant_id`, `sharing`, + `is_inherited`, `id`, `version`, `secret_type`, `expires_at`) +- **`CredStorePluginClientV1`** — backend trait: a pure per-tenant value store + (`get`/`put`/`delete` keyed by `tenant_id` + `key` + optional `owner_id`); it + holds no sharing/hierarchy/policy — that lives in the gear - **`SecretRef`** / **`SecretValue`** / **`SharingMode`** / **`GetSecretResponse`** — Domain models - **`CredStoreError`** — Error types for all operations - **`CredStorePluginSpecV1`** — GTS schema for plugin registration diff --git a/gears/credstore/credstore-sdk/src/api.rs b/gears/credstore/credstore-sdk/src/api.rs index 88c2b6b07..aea9fd1ab 100644 --- a/gears/credstore/credstore-sdk/src/api.rs +++ b/gears/credstore/credstore-sdk/src/api.rs @@ -2,27 +2,116 @@ use async_trait::async_trait; use toolkit_security::SecurityContext; use crate::error::CredStoreError; -use crate::models::{GetSecretResponse, SecretRef}; +use crate::models::{GetSecretResponse, SecretRef, SecretValue, SharingMode, WriteOptions}; /// Consumer-facing API trait for credential storage operations. -/// -/// Obtained from `ClientHub` as `Arc`. All methods -/// accept a `SecurityContext` from which the gateway derives tenant and -/// owner identity. Access denial is expressed as `Ok(None)` from `get`, -/// not as an error. #[async_trait] pub trait CredStoreClientV1: Send + Sync { - /// Retrieves a secret by reference. + /// Retrieves a secret by reference, applying hierarchical resolution. /// - /// Returns `Ok(Some(response))` if the secret exists and is accessible, - /// `Ok(None)` if not found or inaccessible (prevents enumeration), - /// or `Err` for infrastructure failures. - /// - /// The response includes the decrypted value and metadata (owning tenant, - /// sharing mode, whether the secret was inherited via hierarchical resolution). + /// Returns `Ok(Some(_))` with the value and metadata when an accessible + /// secret is found, `Ok(None)` when none exists or is inaccessible (a + /// single 404 surface that prevents enumeration), and + /// `Err(AccessDenied)` only when the caller lacks read permission. async fn get( &self, ctx: &SecurityContext, key: &SecretRef, ) -> Result, CredStoreError>; + + /// Stores or updates a secret (upsert) with default [`WriteOptions`]: + /// the type is preserved on overwrite (`generic` on create) and the expiry + /// is preserved ([`ExpiryWrite::Preserve`](crate::ExpiryWrite::Preserve)), + /// so a value rotation never strips an existing expiry. Use + /// [`Self::put_opts`] to set or clear it explicitly. + async fn put( + &self, + ctx: &SecurityContext, + key: &SecretRef, + value: SecretValue, + sharing: SharingMode, + ) -> Result<(), CredStoreError> { + self.put_opts(ctx, key, value, sharing, WriteOptions::default()) + .await + } + + /// Stores or updates a secret (upsert) with explicit [`WriteOptions`] + /// (secret type, expiry, optimistic-concurrency precondition). The type is + /// immutable: an `opts.secret_type` differing from an existing secret's + /// type is rejected. A failed `opts.precondition` + /// ([`WritePrecondition`](crate::WritePrecondition)) yields + /// [`CredStoreError::Conflict`]. + /// + /// The default implementation reports the operation as unsupported so + /// value-store test doubles that only override [`Self::put`] stay valid; + /// real gear clients override it. + async fn put_opts( + &self, + ctx: &SecurityContext, + key: &SecretRef, + value: SecretValue, + sharing: SharingMode, + opts: WriteOptions, + ) -> Result<(), CredStoreError> { + let _ = (ctx, key, value, sharing, opts); + Err(CredStoreError::internal( + "put_opts is not supported by this CredStoreClientV1 implementation", + )) + } + + /// Creates a secret, failing with [`CredStoreError::Conflict`] if one of the + /// same sharing class already exists (create-only — the 409 path behind the + /// REST `POST`). Use [`Self::put`] for upsert semantics. + async fn create( + &self, + ctx: &SecurityContext, + key: &SecretRef, + value: SecretValue, + sharing: SharingMode, + ) -> Result<(), CredStoreError> { + self.create_opts(ctx, key, value, sharing, WriteOptions::default()) + .await + } + + /// Create-only variant of [`Self::put_opts`]. See it for the default + /// implementation contract. + async fn create_opts( + &self, + ctx: &SecurityContext, + key: &SecretRef, + value: SecretValue, + sharing: SharingMode, + opts: WriteOptions, + ) -> Result<(), CredStoreError> { + let _ = (ctx, key, value, sharing, opts); + Err(CredStoreError::internal( + "create_opts is not supported by this CredStoreClientV1 implementation", + )) + } + + /// Deletes a secret unconditionally. Convenience for + /// [`Self::delete_opts`] with no precondition. + async fn delete(&self, ctx: &SecurityContext, key: &SecretRef) -> Result<(), CredStoreError> { + self.delete_opts(ctx, key, None).await + } + + /// Deletes a secret, optionally guarded by an optimistic-concurrency + /// [`WritePrecondition`](crate::WritePrecondition) (the in-process + /// equivalent of a REST `If-Match` delete). A failed precondition yields + /// [`CredStoreError::Conflict`]. + /// + /// The default implementation reports the operation as unsupported so + /// value-store test doubles that only override [`Self::delete`] stay valid; + /// real gear clients override it. + async fn delete_opts( + &self, + ctx: &SecurityContext, + key: &SecretRef, + precondition: Option, + ) -> Result<(), CredStoreError> { + let _ = (ctx, key, precondition); + Err(CredStoreError::internal( + "delete_opts is not supported by this CredStoreClientV1 implementation", + )) + } } diff --git a/gears/credstore/credstore-sdk/src/error.rs b/gears/credstore/credstore-sdk/src/error.rs index 2c46a1523..44ef654b5 100644 --- a/gears/credstore/credstore-sdk/src/error.rs +++ b/gears/credstore/credstore-sdk/src/error.rs @@ -1,4 +1,5 @@ -// Updated: 2026-04-07 by Constructor Tech +use std::time::Duration; + use thiserror::Error; /// Errors that can occur during credential store operations. @@ -6,16 +7,26 @@ use thiserror::Error; pub enum CredStoreError { #[error("invalid secret reference: {reason}")] InvalidSecretRef { reason: String }, - + #[error("access denied")] + AccessDenied, #[error("secret not found")] NotFound, - + #[error("secret already exists")] + Conflict, #[error("no plugin available")] NoPluginAvailable, - - #[error("service unavailable: {0}")] - ServiceUnavailable(String), - + #[error("service unavailable: {detail}")] + ServiceUnavailable { + detail: String, + retry_after: Option, + }, + #[error("unsupported sharing transition: {detail}")] + UnsupportedTransition { detail: String }, + /// A write violated the secret type's traits (unknown type, disallowed + /// sharing mode, schema/size/format violation, expiry on a + /// non-expirable type). `reason` is a stable machine-readable code. + #[error("secret type violation ({reason}): {detail}")] + TypeViolation { reason: String, detail: String }, #[error("internal error: {0}")] Internal(String), } @@ -27,19 +38,176 @@ impl CredStoreError { reason: reason.into(), } } - #[must_use] - pub fn service_unavailable(msg: impl Into) -> Self { - Self::ServiceUnavailable(msg.into()) + pub fn unsupported_transition(detail: impl Into) -> Self { + Self::UnsupportedTransition { + detail: detail.into(), + } + } + #[must_use] + pub fn service_unavailable(detail: impl Into) -> Self { + Self::ServiceUnavailable { + detail: detail.into(), + retry_after: None, + } + } + #[must_use] + pub fn service_unavailable_with_retry( + detail: impl Into, + retry_after: Duration, + ) -> Self { + Self::ServiceUnavailable { + detail: detail.into(), + retry_after: Some(retry_after), + } } - #[must_use] pub fn internal(msg: impl Into) -> Self { Self::Internal(msg.into()) } + + // ── category predicates (collapse variants for call-site handling) ────────── + + /// `true` for the not-found condition. + #[must_use] + pub fn is_not_found(&self) -> bool { + matches!(self, Self::NotFound) + } + + /// `true` for any transient infrastructure outage where retry is appropriate. + #[must_use] + pub fn is_unavailable(&self) -> bool { + matches!( + self, + Self::ServiceUnavailable { .. } | Self::NoPluginAvailable + ) + } + + /// `true` if the operation may succeed on a future retry. + /// + /// Narrower than [`Self::is_unavailable`]: `NoPluginAvailable` is an + /// operator misconfiguration (no backend plugin registered), not a + /// transient outage, so it is reported as unavailable but **not** + /// retryable — retrying without a config change cannot succeed. + #[must_use] + pub fn is_retryable(&self) -> bool { + matches!(self, Self::ServiceUnavailable { .. }) + } + + /// `true` for request-shape rejections (invalid secret reference, + /// secret-type trait violations). + #[must_use] + pub fn is_validation_error(&self) -> bool { + matches!( + self, + Self::InvalidSecretRef { .. } | Self::TypeViolation { .. } + ) + } + + /// `true` for state-precondition failures (unsupported sharing transition). + #[must_use] + pub fn is_precondition_failed(&self) -> bool { + matches!(self, Self::UnsupportedTransition { .. }) + } + + /// `true` for duplicate-on-create failures. + #[must_use] + pub fn is_already_exists(&self) -> bool { + matches!(self, Self::Conflict) + } + + /// `true` for authorization denials. + #[must_use] + pub fn is_permission_denied(&self) -> bool { + matches!(self, Self::AccessDenied) + } + + /// Retry-after hint (seconds) for transient outages that carry one. Only + /// [`Self::ServiceUnavailable`] populates it; other variants return `None`. + #[must_use] + pub fn retry_after_seconds(&self) -> Option { + match self { + Self::ServiceUnavailable { retry_after, .. } => { + retry_after.and_then(|d| u32::try_from(d.as_secs()).ok()) + } + _ => None, + } + } } #[cfg(test)] -#[cfg_attr(coverage_nightly, coverage(off))] -#[path = "error_tests.rs"] -mod error_tests; +mod error_tests { + use super::*; + + #[test] + fn constructors_build_expected_variants() { + assert!(matches!( + CredStoreError::invalid_ref("x"), + CredStoreError::InvalidSecretRef { .. } + )); + assert!(matches!( + CredStoreError::service_unavailable("down"), + CredStoreError::ServiceUnavailable { + retry_after: None, + .. + } + )); + assert!(matches!( + CredStoreError::service_unavailable_with_retry("down", Duration::from_secs(5)), + CredStoreError::ServiceUnavailable { + retry_after: Some(_), + .. + } + )); + assert!(matches!( + CredStoreError::internal("boom"), + CredStoreError::Internal(_) + )); + } + + #[test] + fn display_redacts_nothing_but_is_stable() { + assert_eq!(CredStoreError::NotFound.to_string(), "secret not found"); + assert_eq!(CredStoreError::AccessDenied.to_string(), "access denied"); + assert_eq!( + CredStoreError::Conflict.to_string(), + "secret already exists" + ); + } + + #[test] + fn category_predicates_classify_variants() { + assert!(CredStoreError::NotFound.is_not_found()); + assert!(CredStoreError::Conflict.is_already_exists()); + assert!(CredStoreError::AccessDenied.is_permission_denied()); + assert!(CredStoreError::invalid_ref("x").is_validation_error()); + assert!( + CredStoreError::TypeViolation { + reason: "SHARING_NOT_ALLOWED_FOR_TYPE".to_owned(), + detail: "x".to_owned(), + } + .is_validation_error() + ); + assert!(CredStoreError::unsupported_transition("x").is_precondition_failed()); + assert!(CredStoreError::NoPluginAvailable.is_unavailable()); + assert!(CredStoreError::service_unavailable("down").is_unavailable()); + assert!(CredStoreError::service_unavailable("down").is_retryable()); + assert!(!CredStoreError::NotFound.is_retryable()); + // Misconfiguration: unavailable, but not retryable without a config change. + assert!(!CredStoreError::NoPluginAvailable.is_retryable()); + } + + #[test] + fn retry_after_seconds_only_for_service_unavailable_with_hint() { + assert_eq!( + CredStoreError::service_unavailable_with_retry("x", Duration::from_secs(7)) + .retry_after_seconds(), + Some(7) + ); + assert_eq!( + CredStoreError::service_unavailable("x").retry_after_seconds(), + None + ); + assert_eq!(CredStoreError::NotFound.retry_after_seconds(), None); + } +} diff --git a/gears/credstore/credstore-sdk/src/error_tests.rs b/gears/credstore/credstore-sdk/src/error_tests.rs deleted file mode 100644 index 642857f1f..000000000 --- a/gears/credstore/credstore-sdk/src/error_tests.rs +++ /dev/null @@ -1,22 +0,0 @@ -// Created: 2026-04-07 by Constructor Tech -use super::*; - -#[test] -fn invalid_ref_constructor_sets_reason() { - let e = CredStoreError::invalid_ref("must not be empty"); - assert_eq!(e.to_string(), "invalid secret reference: must not be empty"); -} - -#[test] -fn service_unavailable_constructor_sets_message() { - let e = CredStoreError::service_unavailable("backend down"); - assert!(matches!(e, CredStoreError::ServiceUnavailable(ref m) if m == "backend down")); - assert_eq!(e.to_string(), "service unavailable: backend down"); -} - -#[test] -fn internal_constructor_sets_message() { - let e = CredStoreError::internal("unexpected state"); - assert!(matches!(e, CredStoreError::Internal(ref m) if m == "unexpected state")); - assert_eq!(e.to_string(), "internal error: unexpected state"); -} diff --git a/gears/credstore/credstore-sdk/src/gts.rs b/gears/credstore/credstore-sdk/src/gts.rs index f90be4764..ee9b12460 100644 --- a/gears/credstore/credstore-sdk/src/gts.rs +++ b/gears/credstore/credstore-sdk/src/gts.rs @@ -1,5 +1,33 @@ +//! GTS (Global Type System) declarations for the credstore SDK. +//! +//! Credstore type/resource ids follow the platform convention +//! `gts.cf.core.credstore..v1~`: the trailing `~` terminates the type +//! URI and `.v1` is the schema version. Plugin specs nest under the toolkit base, +//! `gts.cf.toolkit.plugins.plugin.v1~cf.core.credstore..v1~`. +//! +//! Each `#[gts_type_schema(...)]`-annotated struct submits its schema to the +//! link-time inventory the types-registry loads at boot. Registration is what +//! makes a resource type authorizable: RBAC validates a role's `target_type` +//! against the registry and the PDP compiles a tenant-scoped grant into an +//! `InTenantSubtree` constraint on `owner_tenant_id`. The same string is the +//! single source of truth for the canonical-error `resource_type` — see +//! [`SECRET_RESOURCE_TYPE`], pinned to the impl crate's `#[resource_error(...)]` +//! marker by a unit test. + +use gts_macros::GtsTraitsSchema; +use schemars::JsonSchema; use toolkit::gts::PluginV1; -use toolkit_gts::gts_type_schema; +use toolkit_gts::{gts_id, gts_type_schema}; + +/// GTS resource-type id for the credstore **secret** base type — the single +/// source of truth for this string. Mirrored by [`SecretV1`]'s `type_id` and +/// the `#[resource_error(...)]` marker in `infra::sdk_error_mapping` (a unit +/// test there pins that marker to this constant). Enforcement authorizes +/// against the secret's full *concrete* type, not this base: the impl crate's +/// `domain::authz::secret_type_resource(gts_id)` builds the per-operation PEP +/// `ResourceType` from the type resolved out of the types-registry, and the +/// resolver checks a type descends from this base id. +pub const SECRET_RESOURCE_TYPE: &str = gts_id!("cf.core.credstore.secret.v1~"); #[derive(Default)] #[gts_type_schema( @@ -10,3 +38,248 @@ use toolkit_gts::gts_type_schema; properties = "", )] pub struct CredStorePluginSpecV1; + +/// GTS type-schema for the credstore **secret** resource. +/// +/// This is the base of the PEP `ResourceType`s credstore enforces on: every +/// operation authorizes against the secret's full concrete type via the impl +/// crate's `domain::authz::secret_type_resource(gts_id)`, where `gts_id` is +/// resolved from the types-registry and checked to descend from this base. +/// Registering it (and its derived types) with the types-registry is what +/// makes the resource types authorizable: the RBAC role-definitions API +/// validates `target_type` against the registry, and the authz-resolver PDP +/// compiles a tenant-scoped grant into an `InTenantSubtree` constraint on +/// `owner_tenant_id`. Without this registration every credstore operation is +/// denied (403) because no role can name the type. +/// +/// Modelled as a GTS **type base** (mirroring model-registry's `ModelInfoV1`): +/// the derived secret types (`GenericSecretV1`, `ApiKeySecretV1`, …) chain off +/// it via `#[gts_type_schema(base = SecretV1, …)]`, so the macro emits and +/// registers their schemas. The base carries only the contract-mandated fields +/// — a `gts_type` discriminator and one generic `payload` field — because a +/// secret *type* has no payload: types differ purely by their `x-gts-traits`, +/// so `payload` is always `()`. Authorization itself only needs the type *id* +/// known to the registry (RBAC `target_type` validation) and the PDP; the +/// tenant-scope binding comes from the impl crate's `ResourceType` +/// (`OWNER_TENANT_ID`), not from the schema body. +#[derive(Debug)] +#[gts_type_schema( + dir_path = "schemas", + type_id = gts_id!("cf.core.credstore.secret.v1~"), + description = "CredStore secret resource — tenant-scoped, RBAC/PEP protected", + properties = "gts_type", + traits_schema = inline(SecretTypeTraits), + // `x-gts-traits` defaults for the base type itself: generic semantics (all + // sharing modes, opaque value). gts's `validate_entity_traits` requires the + // base of a traits-schema chain to carry values. Supplied as the typed + // `SecretTypeTraits` carrier (via the catalog's `generic` entry) rather than + // an untyped `json!`, so the base default is compile-checked and can't drift + // from the catalog — the idiom from gts-rust's `traits_struct_literal`. + traits = builtin_traits("generic"), + base = true +)] +pub struct SecretV1 { + /// GTS type discriminator (the base-type contract's type field, mirroring + /// `ModelInfoV1`). Secret types are types, not instances. + pub gts_type: gts::GtsTypeId, + /// Required generic payload field (the base-type contract mandates exactly + /// one). Empty for secret types — they differ by traits, not payload. + pub payload: P, +} + +/// Trait schema (`x-gts-traits-schema`) carried by the secret base type, +/// and the runtime carrier of a secret type's enforceable traits. +/// +/// Derived secret-type schemas (below and any registered later) declare +/// their `x-gts-traits` values against this shape; the registry validates +/// them at registration (`deny_unknown_fields` ⇒ `additionalProperties: +/// false`, so no stray trait keys). At operation time the gear resolves +/// a type's effective traits (chain-merged, leaf wins, base fills the rest) +/// from the types-registry and deserializes them into this struct — the +/// registry is the source of truth for enforcement; +/// [`crate::types::SECRET_TYPE_CATALOG`] only seeds the built-in schemas. +#[derive( + Debug, Clone, PartialEq, JsonSchema, serde::Serialize, serde::Deserialize, GtsTraitsSchema, +)] +#[serde(deny_unknown_fields)] +pub struct SecretTypeTraits { + /// Sharing modes permitted for secrets of this type. + #[serde(default)] + #[schemars(schema_with = "sharing_modes_schema")] + pub allow_sharing: Vec, + /// Whether secrets of this type may carry an expiry. + #[serde(default)] + pub expirable: bool, + /// Upper bound on the raw value size in bytes. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub max_size_bytes: Option, + /// Advisory rotation cadence in seconds. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub rotation_period_secs: Option, + /// Whether the value must be valid UTF-8. + #[serde(default)] + pub utf8_only: bool, + /// JSON Schema the (UTF-8, JSON) secret value must satisfy, if any. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub value_schema: Option, +} + +impl SecretTypeTraits { + /// `true` when `mode` is permitted for this type. + #[must_use] + pub fn allows_sharing(&self, mode: crate::models::SharingMode) -> bool { + self.allow_sharing.contains(&mode) + } +} + +/// Inline `{"type": "array", "items": {"type": "string", "enum": [...]}}` +/// schema for `allow_sharing`. Spelled out (instead of schemars' derived +/// `$ref` to a `oneOf`-of-`const` definition) because the GTS trait +/// validator resolves neither `$defs` references nor `oneOf` branch +/// shapes; `sharing_modes_schema_matches_enum` pins the literals to the +/// serde labels of [`crate::models::SharingMode`]. +fn sharing_modes_schema(_gen: &mut schemars::SchemaGenerator) -> schemars::Schema { + schemars::json_schema!({ + "type": "array", + "items": {"type": "string", "enum": ["private", "tenant", "shared"]}, + "default": [] + }) +} + +// ── Secret types (derived from `SecretV1`) ────────────────────────────────── +// +// One registered GTS type schema per catalog entry in +// [`crate::types::SECRET_TYPE_CATALOG`]. Registration makes each secret type +// discoverable in the types-registry and addressable as a PDP resource type +// (per-type RBAC) without new authorization machinery, and the schema's +// `x-gts-traits` mirror the catalog traits for GTS tooling. +// +// Each derived type is a `#[gts_type_schema(base = SecretV1, …)]` unit struct, +// so the macro emits the schema and submits it to the GTS inventory +// automatically. The `x-gts-traits` value is sourced from the catalog +// descriptor (single source of truth), so a registered schema cannot drift. + +#[allow( + clippy::expect_used, + reason = "built-in type names are compile-time constants proven present by catalog unit tests" +)] +fn builtin_traits(name: &str) -> SecretTypeTraits { + crate::types::SecretType::from_name(name) + .expect("built-in secret type name must be in the catalog") + .descriptor() + .traits() +} + +#[derive(Default)] +#[gts_type_schema( + dir_path = "schemas", + base = SecretV1, + type_id = gts_id!("cf.core.credstore.secret.v1~cf.core.credstore.generic.v1~"), + description = "CredStore secret type: generic", + properties = "", + traits = builtin_traits("generic"), +)] +pub struct GenericSecretV1; + +#[derive(Default)] +#[gts_type_schema( + dir_path = "schemas", + base = SecretV1, + type_id = gts_id!("cf.core.credstore.secret.v1~cf.core.credstore.api_key.v1~"), + description = "CredStore secret type: api-key", + properties = "", + traits = builtin_traits("api-key"), +)] +pub struct ApiKeySecretV1; + +#[derive(Default)] +#[gts_type_schema( + dir_path = "schemas", + base = SecretV1, + type_id = gts_id!("cf.core.credstore.secret.v1~cf.core.credstore.personal_token.v1~"), + description = "CredStore secret type: personal-token", + properties = "", + traits = builtin_traits("personal-token"), +)] +pub struct PersonalTokenSecretV1; + +#[derive(Default)] +#[gts_type_schema( + dir_path = "schemas", + base = SecretV1, + type_id = gts_id!("cf.core.credstore.secret.v1~cf.core.credstore.oauth2_client.v1~"), + description = "CredStore secret type: oauth2-client", + properties = "", + traits = builtin_traits("oauth2-client"), +)] +pub struct Oauth2ClientSecretV1; + +#[derive(Default)] +#[gts_type_schema( + dir_path = "schemas", + base = SecretV1, + type_id = gts_id!("cf.core.credstore.secret.v1~cf.core.credstore.basic_auth.v1~"), + description = "CredStore secret type: basic-auth", + properties = "", + traits = builtin_traits("basic-auth"), +)] +pub struct BasicAuthSecretV1; + +#[derive(Default)] +#[gts_type_schema( + dir_path = "schemas", + base = SecretV1, + type_id = gts_id!("cf.core.credstore.secret.v1~cf.core.credstore.bearer_token.v1~"), + description = "CredStore secret type: bearer-token", + properties = "", + traits = builtin_traits("bearer-token"), +)] +pub struct BearerTokenSecretV1; + +#[derive(Default)] +#[gts_type_schema( + dir_path = "schemas", + base = SecretV1, + type_id = gts_id!("cf.core.credstore.secret.v1~cf.core.credstore.certificate.v1~"), + description = "CredStore secret type: certificate", + properties = "", + traits = builtin_traits("certificate"), +)] +pub struct CertificateSecretV1; + +#[derive(Default)] +#[gts_type_schema( + dir_path = "schemas", + base = SecretV1, + type_id = gts_id!("cf.core.credstore.secret.v1~cf.core.credstore.ssh_key.v1~"), + description = "CredStore secret type: ssh-key", + properties = "", + traits = builtin_traits("ssh-key"), +)] +pub struct SshKeySecretV1; + +#[derive(Default)] +#[gts_type_schema( + dir_path = "schemas", + base = SecretV1, + type_id = gts_id!("cf.core.credstore.secret.v1~cf.core.credstore.webhook_hmac.v1~"), + description = "CredStore secret type: webhook-hmac", + properties = "", + traits = builtin_traits("webhook-hmac"), +)] +pub struct WebhookHmacSecretV1; + +#[derive(Default)] +#[gts_type_schema( + dir_path = "schemas", + base = SecretV1, + type_id = gts_id!("cf.core.credstore.secret.v1~cf.core.credstore.connection_string.v1~"), + description = "CredStore secret type: connection-string", + properties = "", + traits = builtin_traits("connection-string"), +)] +pub struct ConnectionStringSecretV1; + +#[cfg(test)] +#[path = "gts_tests.rs"] +mod secret_type_gts_tests; diff --git a/gears/credstore/credstore-sdk/src/gts_tests.rs b/gears/credstore/credstore-sdk/src/gts_tests.rs new file mode 100644 index 000000000..fddfdc1ca --- /dev/null +++ b/gears/credstore/credstore-sdk/src/gts_tests.rs @@ -0,0 +1,106 @@ +//! Unit tests for the registered secret-type GTS seed schemas. + +use crate::models::SharingMode; +use crate::types::SECRET_TYPE_CATALOG; + +/// The registered base traits schema must stay free of `$ref`/`oneOf` +/// shapes (the GTS trait validator resolves neither — a `oneOf` enum +/// breaks types-registry boot validation) and its `allow_sharing` enum +/// must match the serde labels of [`SharingMode`]. +#[test] +fn sharing_modes_schema_matches_enum() { + let registered = toolkit_gts::all_inventory_type_schemas().expect("inventory schemas parse"); + let want_id = serde_json::json!(format!("gts://{}", super::SECRET_RESOURCE_TYPE)); + let base = registered + .iter() + .find(|s| s["$id"] == want_id) + .expect("secret base type registered"); + let traits_schema = &base["x-gts-traits-schema"]; + let rendered = traits_schema.to_string(); + assert!( + !rendered.contains("$ref") && !rendered.contains("oneOf"), + "traits schema must stay $ref/oneOf-free for the GTS validator: {rendered}" + ); + let labels: Vec = [ + SharingMode::Private, + SharingMode::Tenant, + SharingMode::Shared, + ] + .iter() + .map(|m| serde_json::to_value(m).expect("serializes")) + .collect(); + assert_eq!( + traits_schema["properties"]["allow_sharing"]["items"]["enum"], + serde_json::Value::Array(labels), + "allow_sharing enum drifted from SharingMode's serde labels" + ); +} + +/// Every catalog entry must be backed by a registered GTS type schema +/// whose `x-gts-traits` mirror the catalog descriptor — a divergence +/// between the compiled-in traits and the registered types trips here. +#[test] +fn every_catalog_type_has_a_registered_schema_with_matching_traits() { + let registered = toolkit_gts::all_inventory_type_schemas().expect("inventory schemas parse"); + for d in SECRET_TYPE_CATALOG { + let want_id = serde_json::json!(format!("gts://{}", d.gts_id)); + let schema = registered + .iter() + .find(|s| s["$id"] == want_id) + .unwrap_or_else(|| panic!("no registered schema for {}", d.name)); + let traits = &schema["x-gts-traits"]; + assert_eq!(traits["expirable"], serde_json::json!(d.expirable)); + assert_eq!(traits["utf8_only"], serde_json::json!(d.utf8_only)); + assert_eq!( + traits["allow_sharing"].as_array().map(Vec::len), + Some(d.allow_sharing.len()) + ); + match d.value_schema { + Some(src) => { + let want: serde_json::Value = + serde_json::from_str(src).expect("catalog value_schema parses"); + assert_eq!( + traits["value_schema"], want, + "{}: value_schema trait", + d.name + ); + } + None => assert!( + traits.get("value_schema").is_none(), + "{}: unexpected value_schema trait", + d.name + ), + } + } +} + +/// The registered trait values must deserialize into the runtime traits +/// carrier — this is exactly what the gear's registry-driven resolver +/// does with `effective_traits()`, so a shape drift trips here first. +#[test] +fn registered_traits_deserialize_into_secret_type_traits() { + let registered = toolkit_gts::all_inventory_type_schemas().expect("inventory schemas parse"); + for d in SECRET_TYPE_CATALOG { + let want_id = serde_json::json!(format!("gts://{}", d.gts_id)); + let schema = registered + .iter() + .find(|s| s["$id"] == want_id) + .unwrap_or_else(|| panic!("no registered schema for {}", d.name)); + let traits: super::SecretTypeTraits = + serde_json::from_value(schema["x-gts-traits"].clone()) + .unwrap_or_else(|e| panic!("{}: traits must deserialize: {e}", d.name)); + for mode in d.allow_sharing { + assert!(traits.allows_sharing(*mode), "{}: {mode:?}", d.name); + } + assert_eq!(traits.expirable, d.expirable, "{}", d.name); + assert_eq!(traits.utf8_only, d.utf8_only, "{}", d.name); + assert_eq!( + traits.max_size_bytes, + d.max_size_bytes + .map(|n| u64::try_from(n).expect("catalog size fits u64")), + "{}", + d.name + ); + assert_eq!(traits.value_schema.is_some(), d.value_schema.is_some()); + } +} diff --git a/gears/credstore/credstore-sdk/src/lib.rs b/gears/credstore/credstore-sdk/src/lib.rs index a844a592c..c79d925a2 100644 --- a/gears/credstore/credstore-sdk/src/lib.rs +++ b/gears/credstore/credstore-sdk/src/lib.rs @@ -1,44 +1,20 @@ -//! `CredStore` SDK -//! -//! This crate provides the public API for the `credstore` gear: -//! -//! - [`CredStoreClientV1`] — Consumer API trait for storing/retrieving secrets -//! - [`CredStorePluginClientV1`] — Plugin API trait for backend storage adapters -//! - [`SecretRef`], [`SecretValue`], [`SharingMode`], [`GetSecretResponse`], [`SecretMetadata`] — Domain models -//! - [`CredStoreError`] — Error types -//! - [`CredStorePluginSpecV1`] — GTS schema for plugin discovery -//! -//! # Usage -//! -//! ```rust,ignore -//! use credstore_sdk::{CredStoreClientV1, SecretRef, SecretValue, SharingMode}; -//! -//! async fn store_secret(client: &dyn CredStoreClientV1, ctx: &SecurityContext) { -//! let key = SecretRef::new("partner-openai-key").unwrap(); -//! let value = SecretValue::from("sk-abc123"); -//! -//! client.put(ctx, &key, value, SharingMode::Tenant).await.unwrap(); -//! -//! if let Some(resp) = client.get(ctx, &key).await.unwrap() { -//! // Use resp.value.as_bytes() -//! // Check resp.is_inherited, resp.sharing, resp.owner_tenant_id -//! } -//! } -//! ``` - -#![cfg_attr(coverage_nightly, feature(coverage_attribute))] - +//! credstore SDK — public API traits, models, errors, GTS schema, secret types. pub mod api; pub mod error; pub mod gts; pub mod models; pub mod plugin_api; +#[cfg(feature = "test-util")] +pub mod test_util; +pub mod types; -// Re-export main types at crate root +pub use ::gts::GtsId; pub use api::CredStoreClientV1; pub use error::CredStoreError; -pub use gts::CredStorePluginSpecV1; +pub use gts::{CredStorePluginSpecV1, SECRET_RESOURCE_TYPE, SecretTypeTraits, SecretV1}; pub use models::{ - GetSecretResponse, OwnerId, SecretMetadata, SecretRef, SecretValue, SharingMode, TenantId, + ExpiryWrite, GetSecretResponse, OwnerId, SecretRef, SecretValue, SharingMode, TenantId, + WriteOptions, WritePrecondition, }; pub use plugin_api::CredStorePluginClientV1; +pub use types::{SECRET_TYPE_CATALOG, SecretType, SecretTypeDescriptor}; diff --git a/gears/credstore/credstore-sdk/src/models.rs b/gears/credstore/credstore-sdk/src/models.rs index ab2f820c7..f72c8620a 100644 --- a/gears/credstore/credstore-sdk/src/models.rs +++ b/gears/credstore/credstore-sdk/src/models.rs @@ -7,6 +7,8 @@ use serde::{Deserialize, Serialize}; use uuid::Uuid; use zeroize::Zeroize; +use gts::GtsId; + use crate::error::CredStoreError; /// Re-export from tenant-resolver-sdk for cross-gear type consistency. @@ -151,7 +153,14 @@ impl fmt::Display for SecretValue { } /// Controls the visibility scope of a stored secret. -#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize)] +/// +/// Also part of the GTS trait vocabulary: `SecretTypeTraits::allow_sharing` +/// lists the modes a secret type permits, so the derived `x-gts-traits-schema` +/// constrains trait values to this enum (schemars follows the serde +/// `snake_case` renames). +#[derive( + Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize, schemars::JsonSchema, +)] #[serde(rename_all = "snake_case")] pub enum SharingMode { /// Only the owner can access the secret. @@ -163,12 +172,106 @@ pub enum SharingMode { Shared, } +/// How a write should treat the secret's expiry. +/// +/// Modelled as an explicit tri-state rather than `Option` so that "leave the +/// stored expiry untouched" and "remove the stored expiry" are distinct: with a +/// plain `Option`, `None` conflates the two and an ordinary +/// value rotation (`put`) would silently strip an existing expiry. Expiry is +/// only permitted for types whose traits are `expirable`. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub enum ExpiryWrite { + /// Keep the currently stored expiry on overwrite; no expiry on create. + /// This is the default, so the convenience + /// [`CredStoreClientV1::put`](crate::CredStoreClientV1::put) never clears an + /// expiry as a side effect of a value rotation. + #[default] + Preserve, + /// Set the expiry to this instant (must be in the future at write time). + Set(time::OffsetDateTime), + /// Remove any stored expiry. + Clear, +} + +impl ExpiryWrite { + /// The instant this write introduces as a *new* expiry, if any. Only + /// [`Self::Set`] introduces one; [`Self::Preserve`]/[`Self::Clear`] do not, + /// so they never trigger expiry validation against the secret's type. + #[must_use] + pub fn requested(self) -> Option { + match self { + Self::Set(at) => Some(at), + Self::Preserve | Self::Clear => None, + } + } + + /// Resolve the value to persist against the currently stored expiry + /// (`current` — `None` on create). `Preserve` keeps `current`, `Set` + /// overrides it, `Clear` drops it. + #[must_use] + pub fn resolve(self, current: Option) -> Option { + match self { + Self::Preserve => current, + Self::Set(at) => Some(at), + Self::Clear => None, + } + } +} + +/// Optimistic-concurrency precondition for a guarded write or delete — the +/// in-process equivalent of the REST `If-Match` header. Read the current +/// generation from a prior [`GetSecretResponse`] (`id` + `version`). +/// +/// A failed precondition surfaces as [`CredStoreError::Conflict`]; a write with +/// `None` is unconditional. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum WritePrecondition { + /// The target secret must already exist (REST `If-Match: *`). + Exists, + /// Compare-and-set: the current generation must still be `(id, version)` + /// (REST `If-Match: "."`). `id` is the row UUID — fresh per + /// recreated secret — so a validator from an earlier generation never + /// matches after a delete/recreate even if the version counters coincide. + Matches { + /// Row (generation) UUID from the observed [`GetSecretResponse::id`]. + id: uuid::Uuid, + /// Version counter from the observed [`GetSecretResponse::version`]. + version: i64, + }, +} + +/// Options for typed writes ([`CredStoreClientV1::put_opts`](crate::CredStoreClientV1::put_opts) / +/// [`create_opts`](crate::CredStoreClientV1::create_opts)). +#[derive(Debug, Clone, Default)] +pub struct WriteOptions { + /// Full GTS type id of the secret type (e.g. via + /// [`SecretType::gts_id`](crate::SecretType::gts_id) for a built-in, or a + /// custom type's id). `None` keeps the existing secret's type on overwrite + /// and defaults to the generic type on create. The gear resolves it to the + /// type's deterministic UUID and validates existence against the + /// types-registry. + pub secret_type: Option, + /// Expiry write intent — see [`ExpiryWrite`]. Defaults to + /// [`ExpiryWrite::Preserve`], so a write that does not mention expiry + /// leaves any stored expiry unchanged. + pub expires_at: ExpiryWrite, + /// Optional optimistic-concurrency guard — see [`WritePrecondition`]. + /// `None` (default) writes unconditionally. + pub precondition: Option, +} + /// Response returned by [`CredStoreClientV1::get`](crate::CredStoreClientV1::get) /// containing the secret value and access metadata. #[derive(Debug)] pub struct GetSecretResponse { /// The decrypted secret value. pub value: SecretValue, + /// Generation id of the resolved secret — the metadata row's UUID, minted + /// fresh for every recreated secret. Combined with `version` it forms the + /// gear's strong `ETag` (`"."`), so a validator from a + /// deleted-and-recreated secret's earlier generation can never match the + /// current one even when the version counters coincide. + pub id: uuid::Uuid, /// The tenant that owns this secret (may differ from the requesting tenant /// when the secret is inherited via hierarchical resolution). pub owner_tenant_id: TenantId, @@ -177,17 +280,88 @@ pub struct GetSecretResponse { /// `true` if the secret was retrieved from an ancestor tenant via /// hierarchical resolution, `false` if owned by the requesting tenant. pub is_inherited: bool, -} - -/// Metadata returned by plugins alongside the secret value. -#[derive(Debug)] -pub struct SecretMetadata { - pub value: SecretValue, - pub owner_id: OwnerId, - pub sharing: SharingMode, - pub owner_tenant_id: TenantId, + /// Monotonic version of the resolved secret within its generation; + /// surfaced as the HTTP `ETag` together with `id`. + pub version: i64, + /// The secret's full GTS type id, as resolved from the types-registry + /// (covers dynamically registered custom types; use + /// [`crate::types::SecretType::from_gts_id`] to recover a catalog type + /// when needed). + pub secret_type: String, + /// Expiry instant, when the type is expirable and one was set. + pub expires_at: Option, } #[cfg(test)] -#[path = "models_tests.rs"] -mod models_tests; +mod models_tests { + use super::*; + + #[test] + fn secret_ref_accepts_valid_shapes() { + for ok in ["partner-openai-key", "api_key_v2", "ABC123", "ok-key_1"] { + assert!(SecretRef::new(ok).is_ok(), "{ok} should be valid"); + } + } + + #[test] + fn secret_ref_rejects_invalid_chars_and_empty() { + assert!(SecretRef::new("").is_err()); + for bad in ["has:colon", "my key", "key/path"] { + assert!(SecretRef::new(bad).is_err(), "{bad} should be rejected"); + } + } + + #[test] + fn secret_ref_length_boundary() { + // 255 is the inclusive max; 256 is rejected (boundary both sides). + assert!(SecretRef::new("a".repeat(255)).is_ok()); + assert!(SecretRef::new("a".repeat(256)).is_err()); + } + + #[test] + fn secret_ref_deserialize_validates() { + // The custom Deserialize is the real wire path (a raw JSON string must + // go through the same validation as `SecretRef::new`). + let valid: Result = serde_json::from_str("\"valid-key_1\""); + assert_eq!(valid.expect("valid").as_ref(), "valid-key_1"); + assert!(serde_json::from_str::("\"my:evil/key\"").is_err()); + assert!(serde_json::from_str::("\"\"").is_err()); + assert!(serde_json::from_str::(&format!("\"{}\"", "a".repeat(256))).is_err()); + } + + #[test] + fn secret_ref_serde_round_trips() { + let r = SecretRef::new("round-trip").expect("valid"); + let json = serde_json::to_string(&r).expect("serialize"); + assert_eq!(json, "\"round-trip\""); + let back: SecretRef = serde_json::from_str(&json).expect("deserialize"); + assert_eq!(back.as_ref(), "round-trip"); + } + + #[test] + fn secret_value_redacts() { + let v = SecretValue::from("supersecret"); + assert_eq!(format!("{v:?}"), "[REDACTED]"); + assert_eq!(format!("{v}"), "[REDACTED]"); + assert_eq!(v.as_bytes(), b"supersecret"); + } + + #[test] + fn sharing_mode_default_is_tenant() { + assert_eq!(SharingMode::default(), SharingMode::Tenant); + } + + #[test] + fn sharing_mode_serde_round_trips() { + for (mode, expected) in [ + (SharingMode::Private, "\"private\""), + (SharingMode::Tenant, "\"tenant\""), + (SharingMode::Shared, "\"shared\""), + ] { + let json = serde_json::to_string(&mode).expect("serialize"); + assert_eq!(json, expected); + let back: SharingMode = serde_json::from_str(&json).expect("deserialize"); + assert_eq!(back, mode); + } + } +} diff --git a/gears/credstore/credstore-sdk/src/models_tests.rs b/gears/credstore/credstore-sdk/src/models_tests.rs deleted file mode 100644 index 24c464b0b..000000000 --- a/gears/credstore/credstore-sdk/src/models_tests.rs +++ /dev/null @@ -1,114 +0,0 @@ -// Created: 2026-04-07 by Constructor Tech -use super::*; - -#[test] -fn secret_ref_valid() { - assert!(SecretRef::new("partner-openai-key").is_ok()); - assert!(SecretRef::new("api_key_v2").is_ok()); - assert!(SecretRef::new("ABC123").is_ok()); -} - -#[test] -fn secret_ref_invalid_chars() { - assert!(SecretRef::new("my:key").is_err()); - assert!(SecretRef::new("my key").is_err()); - assert!(SecretRef::new("key/path").is_err()); -} - -#[test] -fn secret_ref_empty() { - assert!(SecretRef::new("").is_err()); -} - -#[test] -fn secret_ref_too_long() { - let long = "a".repeat(256); - assert!(SecretRef::new(long).is_err()); -} - -#[test] -fn secret_ref_max_length() { - let max = "a".repeat(255); - assert!(SecretRef::new(max).is_ok()); -} - -#[test] -fn secret_ref_deserialize_validates() { - let valid: Result = serde_json::from_str("\"valid-key_1\""); - assert!(valid.is_ok()); - assert_eq!(valid.unwrap().as_ref(), "valid-key_1"); - - let with_colon: Result = serde_json::from_str("\"my:evil/key\""); - assert!(with_colon.is_err()); - - let empty: Result = serde_json::from_str("\"\""); - assert!(empty.is_err()); -} - -#[test] -fn secret_value_debug_redacted() { - let val = SecretValue::new(b"super-secret".to_vec()); - assert_eq!(format!("{val:?}"), "[REDACTED]"); -} - -#[test] -fn secret_value_display_redacted() { - let val = SecretValue::new(b"super-secret".to_vec()); - assert_eq!(format!("{val}"), "[REDACTED]"); -} - -#[test] -fn secret_value_as_bytes() { - let val = SecretValue::from("hello"); - assert_eq!(val.as_bytes(), b"hello"); -} - -#[test] -fn get_secret_response_debug_redacts_value() { - let resp = GetSecretResponse { - value: SecretValue::from("secret"), - owner_tenant_id: TenantId::nil(), - sharing: SharingMode::Shared, - is_inherited: true, - }; - let debug = format!("{resp:?}"); - assert!(debug.contains("[REDACTED]")); - assert!(!debug.contains("secret")); - assert!(debug.contains("is_inherited: true")); -} - -#[test] -fn secret_metadata_debug_redacts_value() { - let meta = SecretMetadata { - value: SecretValue::from("secret"), - owner_id: OwnerId::nil(), - sharing: SharingMode::Tenant, - owner_tenant_id: TenantId::nil(), - }; - let debug = format!("{meta:?}"); - assert!(debug.contains("[REDACTED]")); - assert!(!debug.contains("secret")); -} - -#[test] -fn sharing_mode_serde_roundtrip() { - for (mode, expected_json) in [ - (SharingMode::Private, "\"private\""), - (SharingMode::Tenant, "\"tenant\""), - (SharingMode::Shared, "\"shared\""), - ] { - let json = serde_json::to_string(&mode).unwrap(); - assert_eq!(json, expected_json); - let back: SharingMode = serde_json::from_str(&json).unwrap(); - assert_eq!(back, mode); - } -} - -#[test] -fn secret_ref_serialize_roundtrip() { - let r = SecretRef::new("round-trip").unwrap(); - let json = serde_json::to_string(&r).unwrap(); - assert_eq!(json, "\"round-trip\""); - let back: SecretRef = serde_json::from_str(&json).unwrap(); - assert_eq!(back.as_ref(), "round-trip"); -} diff --git a/gears/credstore/credstore-sdk/src/plugin_api.rs b/gears/credstore/credstore-sdk/src/plugin_api.rs index 7752f3e25..33cf1bd89 100644 --- a/gears/credstore/credstore-sdk/src/plugin_api.rs +++ b/gears/credstore/credstore-sdk/src/plugin_api.rs @@ -2,18 +2,37 @@ use async_trait::async_trait; use toolkit_security::SecurityContext; use crate::error::CredStoreError; -use crate::models::{SecretMetadata, SecretRef}; +use crate::models::{OwnerId, SecretRef, SecretValue, TenantId}; -/// Backend storage adapter trait implemented by credential store plugins. -/// -/// Plugins operate at the single-tenant level with explicit parameters -/// decomposed by the gateway. Authorization is the gateway's responsibility. +/// Pure per-tenant value store. `owner_id=Some` selects the owner's private +/// key class; `None` selects the tenant key class. No sharing/hierarchy/policy. #[async_trait] pub trait CredStorePluginClientV1: Send + Sync { - /// Retrieves a secret with full metadata from the backend. + /// Retrieves a secret from the backend. async fn get( &self, ctx: &SecurityContext, + tenant_id: &TenantId, key: &SecretRef, - ) -> Result, CredStoreError>; + owner_id: Option<&OwnerId>, + ) -> Result, CredStoreError>; + + /// Stores a secret in the backend. + async fn put( + &self, + ctx: &SecurityContext, + tenant_id: &TenantId, + key: &SecretRef, + value: SecretValue, + owner_id: Option<&OwnerId>, + ) -> Result<(), CredStoreError>; + + /// Deletes a secret from the backend. + async fn delete( + &self, + ctx: &SecurityContext, + tenant_id: &TenantId, + key: &SecretRef, + owner_id: Option<&OwnerId>, + ) -> Result<(), CredStoreError>; } diff --git a/gears/credstore/credstore-sdk/src/test_util.rs b/gears/credstore/credstore-sdk/src/test_util.rs new file mode 100644 index 000000000..589935325 --- /dev/null +++ b/gears/credstore/credstore-sdk/src/test_util.rs @@ -0,0 +1,153 @@ +//! Test-only [`CredStoreClientV1`] doubles, behind the `test-util` feature. +//! +//! A single configurable [`MockCredStoreClient`] covering the shapes consumers +//! exercise in tests, so each gear no longer hand-rolls its own: +//! +//! * [`MockCredStoreClient::empty`] — every `get` resolves to `Ok(None)`; +//! * [`MockCredStoreClient::with_secrets`] — a keyed `(reference, value)` store; +//! * [`MockCredStoreClient::returning_raw_value`] — a fixed raw value for any +//! reference (e.g. non-UTF-8 bytes to drive malformed-value paths); +//! * [`MockCredStoreClient::always_failing`] — every operation fails with +//! [`CredStoreError::Internal`]. +//! +//! Only `get` carries behaviour; the write half is a no-op that succeeds (or +//! fails, in the always-failing mode) to match. + +use std::collections::HashMap; + +use async_trait::async_trait; +use toolkit_security::SecurityContext; + +use crate::{ + CredStoreClientV1, CredStoreError, GetSecretResponse, SecretRef, SecretType, SecretValue, + SharingMode, TenantId, WriteOptions, WritePrecondition, +}; + +enum Behavior { + /// `get` returns the mapped value for a known reference, else `Ok(None)`. + Store(HashMap>), + /// `get` returns this raw value for *any* reference. + AnyValue(Vec), + /// Every operation fails with [`CredStoreError::Internal`]. + Failing, +} + +/// Configurable in-process [`CredStoreClientV1`] test double. See the module +/// docs for the available modes. +pub struct MockCredStoreClient { + behavior: Behavior, +} + +impl MockCredStoreClient { + /// Empty store — every `get` resolves to `Ok(None)`. + #[must_use] + pub fn empty() -> Self { + Self { + behavior: Behavior::Store(HashMap::new()), + } + } + + /// Store seeded with `(reference, value)` pairs; an unknown reference + /// resolves to `Ok(None)`. A leading `cred://` scheme prefix on a key is + /// stripped, so callers using either the bare reference or the consumer's + /// `cred://` spelling resolve identically. + #[must_use] + pub fn with_secrets(creds: Vec<(String, String)>) -> Self { + let store = creds + .into_iter() + .map(|(k, v)| { + let key = k.strip_prefix("cred://").unwrap_or(&k).to_owned(); + (key, v.into_bytes()) + }) + .collect(); + Self { + behavior: Behavior::Store(store), + } + } + + /// Resolve *any* reference to this raw value — useful for exercising + /// non-UTF-8 / malformed-value paths. + #[must_use] + pub fn returning_raw_value(value: Vec) -> Self { + Self { + behavior: Behavior::AnyValue(value), + } + } + + /// Every operation fails with [`CredStoreError::Internal`] — for + /// error-handling paths. + #[must_use] + pub fn always_failing() -> Self { + Self { + behavior: Behavior::Failing, + } + } + + /// Build a canned response wrapping `value` with placeholder metadata + /// (nil id/tenant, `generic` type, version 1, not inherited, no expiry). + fn response(value: Vec) -> GetSecretResponse { + GetSecretResponse { + value: SecretValue::new(value), + id: uuid::Uuid::nil(), + owner_tenant_id: TenantId::nil(), + sharing: SharingMode::default(), + is_inherited: false, + version: 1, + secret_type: SecretType::generic().gts_id().to_owned(), + expires_at: None, + } + } + + fn write_result(&self) -> Result<(), CredStoreError> { + match self.behavior { + Behavior::Failing => Err(CredStoreError::Internal("backend failure".into())), + Behavior::Store(_) | Behavior::AnyValue(_) => Ok(()), + } + } +} + +#[async_trait] +impl CredStoreClientV1 for MockCredStoreClient { + async fn get( + &self, + _ctx: &SecurityContext, + key: &SecretRef, + ) -> Result, CredStoreError> { + match &self.behavior { + Behavior::Store(store) => Ok(store.get(key.as_ref()).cloned().map(Self::response)), + Behavior::AnyValue(value) => Ok(Some(Self::response(value.clone()))), + Behavior::Failing => Err(CredStoreError::Internal("backend failure".into())), + } + } + + async fn put_opts( + &self, + _ctx: &SecurityContext, + _key: &SecretRef, + _value: SecretValue, + _sharing: SharingMode, + _opts: WriteOptions, + ) -> Result<(), CredStoreError> { + self.write_result() + } + + async fn create_opts( + &self, + _ctx: &SecurityContext, + _key: &SecretRef, + _value: SecretValue, + _sharing: SharingMode, + _opts: WriteOptions, + ) -> Result<(), CredStoreError> { + self.write_result() + } + + async fn delete_opts( + &self, + _ctx: &SecurityContext, + _key: &SecretRef, + _precondition: Option, + ) -> Result<(), CredStoreError> { + self.write_result() + } +} diff --git a/gears/credstore/credstore-sdk/src/types.rs b/gears/credstore/credstore-sdk/src/types.rs new file mode 100644 index 000000000..4039ddb38 --- /dev/null +++ b/gears/credstore/credstore-sdk/src/types.rs @@ -0,0 +1,357 @@ +//! GTS-based secret types and their enforceable traits. +//! +//! A *secret type* classifies a secret and binds the handling rules the +//! gear enforces uniformly — most importantly which [`SharingMode`]s the +//! type permits. Every type is a GTS type derived from the credstore secret +//! base type ([`crate::SECRET_RESOURCE_TYPE`]): +//! +//! ```text +//! gts.cf.core.credstore.secret.v1~cf.core.credstore..v1~ +//! ``` +//! +//! The catalog below is the single source of truth for the traits; each +//! entry's GTS schema is also registered with the types-registry via the +//! link-time inventory (see [`crate::gts`]), which makes the types +//! discoverable and per-type PDP targeting possible without new machinery. +//! Adding a type means adding a catalog entry (and its schema struct); the +//! enforcement code is trait-driven and needs no change unless a new *trait* +//! is introduced. +//! +//! The type of a secret is chosen at creation (default [`SecretType::generic`]) +//! and is immutable for the secret's lifetime. + +use std::fmt; + +use serde::{Deserialize, Serialize}; +use toolkit_gts::gts_id; +use uuid::Uuid; + +use crate::error::CredStoreError; +use crate::models::SharingMode; + +/// Enforceable traits of a secret type. +/// +/// All fields are enforced by the gear on write except +/// `rotation_period_secs` (advisory) and `expirable`, which additionally +/// gates reads (an expired secret resolves as not-found). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct SecretTypeDescriptor { + /// Short, stable label used on the REST transport (e.g. `"api-key"`). + pub name: &'static str, + /// Full derived GTS type id (stored in the gear metadata table). + pub gts_id: &'static str, + /// Sharing modes permitted for secrets of this type. + pub allow_sharing: &'static [SharingMode], + /// Embedded JSON Schema the (UTF-8, JSON) value must satisfy, if any. + pub value_schema: Option<&'static str>, + /// Upper bound on the raw value size; `None` = platform default only. + pub max_size_bytes: Option, + /// Whether secrets of this type may carry `expires_at`; expired secrets + /// resolve as not-found and are swept by the reaper. + pub expirable: bool, + /// Advisory rotation cadence; surfaced via metadata only. + pub rotation_period_secs: Option, + /// Whether the value must be valid UTF-8 (all current types; a future + /// binary type would clear this). + pub utf8_only: bool, +} + +impl SecretTypeDescriptor { + /// `true` when `mode` is permitted for this type. + #[must_use] + pub fn allows_sharing(&self, mode: SharingMode) -> bool { + self.allow_sharing.contains(&mode) + } + + /// The descriptor's traits in the runtime trait-carrier shape — the same + /// view the gear resolves from the types-registry for a registered + /// type. Catalog `value_schema` constants are pinned valid JSON by + /// `embedded_value_schemas_are_valid_json`; on the impossible parse + /// failure the trait is omitted. + #[must_use] + pub fn traits(&self) -> crate::gts::SecretTypeTraits { + crate::gts::SecretTypeTraits { + allow_sharing: self.allow_sharing.to_vec(), + expirable: self.expirable, + max_size_bytes: self.max_size_bytes.and_then(|n| u64::try_from(n).ok()), + rotation_period_secs: self.rotation_period_secs, + utf8_only: self.utf8_only, + value_schema: self.value_schema.and_then(|s| serde_json::from_str(s).ok()), + } + } +} + +const ALL: &[SharingMode] = &[ + SharingMode::Private, + SharingMode::Tenant, + SharingMode::Shared, +]; +const PRIVATE_ONLY: &[SharingMode] = &[SharingMode::Private]; +const PRIVATE_TENANT: &[SharingMode] = &[SharingMode::Private, SharingMode::Tenant]; +const TENANT_SHARED: &[SharingMode] = &[SharingMode::Tenant, SharingMode::Shared]; +const TENANT_ONLY: &[SharingMode] = &[SharingMode::Tenant]; + +const OAUTH2_CLIENT_SCHEMA: &str = r#"{ + "type": "object", + "required": ["client_id", "client_secret"], + "properties": { + "client_id": {"type": "string", "minLength": 1}, + "client_secret": {"type": "string", "minLength": 1}, + "token_url": {"type": "string", "minLength": 1}, + "scopes": {"type": "array", "items": {"type": "string"}} + }, + "additionalProperties": false +}"#; + +const BASIC_AUTH_SCHEMA: &str = r#"{ + "type": "object", + "required": ["username", "password"], + "properties": { + "username": {"type": "string", "minLength": 1}, + "password": {"type": "string"} + }, + "additionalProperties": false +}"#; + +/// The full compiled-in type catalog. Order is presentation-only. +pub const SECRET_TYPE_CATALOG: &[SecretTypeDescriptor] = &[ + SecretTypeDescriptor { + name: "generic", + gts_id: gts_id!("cf.core.credstore.secret.v1~cf.core.credstore.generic.v1~"), + allow_sharing: ALL, + value_schema: None, + max_size_bytes: None, + expirable: false, + rotation_period_secs: None, + utf8_only: false, + }, + SecretTypeDescriptor { + name: "api-key", + gts_id: gts_id!("cf.core.credstore.secret.v1~cf.core.credstore.api_key.v1~"), + allow_sharing: ALL, + value_schema: None, + max_size_bytes: Some(8 * 1024), + expirable: false, + rotation_period_secs: None, + utf8_only: true, + }, + SecretTypeDescriptor { + name: "personal-token", + gts_id: gts_id!("cf.core.credstore.secret.v1~cf.core.credstore.personal_token.v1~"), + allow_sharing: PRIVATE_ONLY, + value_schema: None, + max_size_bytes: Some(8 * 1024), + expirable: true, + rotation_period_secs: None, + utf8_only: true, + }, + SecretTypeDescriptor { + name: "oauth2-client", + gts_id: gts_id!("cf.core.credstore.secret.v1~cf.core.credstore.oauth2_client.v1~"), + allow_sharing: TENANT_SHARED, + value_schema: Some(OAUTH2_CLIENT_SCHEMA), + max_size_bytes: Some(16 * 1024), + expirable: false, + rotation_period_secs: None, + utf8_only: true, + }, + SecretTypeDescriptor { + name: "basic-auth", + gts_id: gts_id!("cf.core.credstore.secret.v1~cf.core.credstore.basic_auth.v1~"), + allow_sharing: ALL, + value_schema: Some(BASIC_AUTH_SCHEMA), + max_size_bytes: Some(16 * 1024), + expirable: false, + rotation_period_secs: None, + utf8_only: true, + }, + SecretTypeDescriptor { + name: "bearer-token", + gts_id: gts_id!("cf.core.credstore.secret.v1~cf.core.credstore.bearer_token.v1~"), + allow_sharing: PRIVATE_TENANT, + value_schema: None, + max_size_bytes: Some(64 * 1024), + expirable: true, + rotation_period_secs: None, + utf8_only: true, + }, + SecretTypeDescriptor { + name: "certificate", + gts_id: gts_id!("cf.core.credstore.secret.v1~cf.core.credstore.certificate.v1~"), + allow_sharing: TENANT_SHARED, + value_schema: None, + max_size_bytes: Some(256 * 1024), + expirable: true, + rotation_period_secs: Some(90 * 24 * 3600), + utf8_only: true, + }, + SecretTypeDescriptor { + name: "ssh-key", + gts_id: gts_id!("cf.core.credstore.secret.v1~cf.core.credstore.ssh_key.v1~"), + allow_sharing: PRIVATE_TENANT, + value_schema: None, + max_size_bytes: Some(64 * 1024), + expirable: false, + rotation_period_secs: None, + utf8_only: true, + }, + SecretTypeDescriptor { + name: "webhook-hmac", + gts_id: gts_id!("cf.core.credstore.secret.v1~cf.core.credstore.webhook_hmac.v1~"), + allow_sharing: TENANT_SHARED, + value_schema: None, + max_size_bytes: Some(8 * 1024), + expirable: false, + rotation_period_secs: None, + utf8_only: true, + }, + SecretTypeDescriptor { + name: "connection-string", + gts_id: gts_id!("cf.core.credstore.secret.v1~cf.core.credstore.connection_string.v1~"), + allow_sharing: TENANT_ONLY, + value_schema: None, + max_size_bytes: Some(4 * 1024), + expirable: false, + rotation_period_secs: None, + utf8_only: true, + }, +]; + +/// A validated secret type — always resolves to a catalog descriptor. +#[derive(Clone, Copy, PartialEq, Eq)] +pub struct SecretType(&'static SecretTypeDescriptor); + +impl SecretType { + /// The default type: fully backward-compatible opaque secrets. + #[must_use] + pub fn generic() -> Self { + Self(&SECRET_TYPE_CATALOG[0]) + } + + /// Resolve a type by its REST label (e.g. `"api-key"`). + /// + /// # Errors + /// + /// Returns [`CredStoreError::TypeViolation`] with reason + /// `UNKNOWN_SECRET_TYPE` when no catalog entry matches. + pub fn from_name(name: &str) -> Result { + SECRET_TYPE_CATALOG + .iter() + .find(|d| d.name == name) + .map(Self) + .ok_or_else(|| CredStoreError::TypeViolation { + reason: "UNKNOWN_SECRET_TYPE".to_owned(), + detail: format!("unknown secret type: {name}"), + }) + } + + /// Resolve a type by its full GTS id (the stored representation). + #[must_use] + pub fn from_gts_id(id: &str) -> Option { + SECRET_TYPE_CATALOG + .iter() + .find(|d| d.gts_id == id) + .map(Self) + } + + /// Short REST label. + #[must_use] + pub fn name(self) -> &'static str { + self.0.name + } + + /// Full derived GTS type id. + #[must_use] + pub fn gts_id(self) -> &'static str { + self.0.gts_id + } + + /// Deterministic types-registry UUID (v5) of this type — the stored + /// representation (`credstore_secrets.secret_type_uuid`) and the key for + /// resolving the type's schema/traits from the registry + /// ([`type_uuid`] of [`Self::gts_id`]). + /// + /// # Panics + /// + /// Panics if a catalog `gts_id` is not a valid GTS type id — an impossible + /// state guarded by `catalog_names_and_ids_are_unique_and_well_formed` and + /// `type_uuid_is_deterministic_and_matches_registry_v5`. + #[must_use] + #[allow( + clippy::expect_used, + reason = "catalog gts ids are compile-time constants proven valid by unit tests" + )] + pub fn uuid(self) -> Uuid { + type_uuid(self.0.gts_id).expect("catalog gts id must be a valid GTS type id") + } + + /// Trait descriptor. + #[must_use] + pub fn descriptor(self) -> &'static SecretTypeDescriptor { + self.0 + } +} + +/// Deterministic types-registry UUID (v5) of a GTS type id — matches the id the +/// registry assigns (`GtsId::to_uuid`), so credstore can compute a type's +/// storage/lookup key locally without a registry round-trip. Returns `None` +/// when `gts_id` is not a valid GTS type id. +#[must_use] +pub fn type_uuid(gts_id: &str) -> Option { + gts::GtsId::try_new(gts_id) + .ok() + .map(|parsed| parsed.to_uuid()) +} + +/// Deterministic v5 UUID of the generic (default) secret type, as a string — +/// the value the `credstore_secrets.secret_type_uuid` column DEFAULT uses. +/// Pinned by `type_uuid_is_deterministic_and_matches_registry_v5` so the +/// migration default and the computed id can never drift. +pub const GENERIC_TYPE_UUID_STR: &str = "2a8aac98-cf09-58ed-acd6-f599f35cb5bf"; + +impl Default for SecretType { + fn default() -> Self { + Self::generic() + } +} + +impl From for gts::GtsId { + /// A built-in secret type's full GTS type id. Ergonomic for setting + /// [`crate::WriteOptions::secret_type`] to a catalog type. + #[allow( + clippy::expect_used, + reason = "catalog gts ids are compile-time constants proven valid by unit tests" + )] + fn from(t: SecretType) -> Self { + gts::GtsId::try_new(t.gts_id()).expect("catalog gts id must be a valid GTS type id") + } +} + +impl fmt::Debug for SecretType { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_tuple("SecretType").field(&self.0.name).finish() + } +} + +impl fmt::Display for SecretType { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(self.0.name) + } +} + +impl Serialize for SecretType { + fn serialize(&self, serializer: S) -> Result { + serializer.serialize_str(self.0.name) + } +} + +impl<'de> Deserialize<'de> for SecretType { + fn deserialize>(deserializer: D) -> Result { + let s = String::deserialize(deserializer)?; + Self::from_name(&s).map_err(serde::de::Error::custom) + } +} + +#[cfg(test)] +#[path = "types_tests.rs"] +mod types_tests; diff --git a/gears/credstore/credstore-sdk/src/types_tests.rs b/gears/credstore/credstore-sdk/src/types_tests.rs new file mode 100644 index 000000000..cc63e4d84 --- /dev/null +++ b/gears/credstore/credstore-sdk/src/types_tests.rs @@ -0,0 +1,117 @@ +//! Unit tests for the secret-type catalog and type references. + +use toolkit_gts::GTS_ID_PREFIX; + +use super::*; + +#[test] +fn catalog_names_and_ids_are_unique_and_well_formed() { + let mut names: Vec<_> = SECRET_TYPE_CATALOG.iter().map(|d| d.name).collect(); + names.sort_unstable(); + names.dedup(); + assert_eq!(names.len(), SECRET_TYPE_CATALOG.len(), "duplicate names"); + + let mut ids: Vec<_> = SECRET_TYPE_CATALOG.iter().map(|d| d.gts_id).collect(); + ids.sort_unstable(); + ids.dedup(); + assert_eq!(ids.len(), SECRET_TYPE_CATALOG.len(), "duplicate gts ids"); + + for d in SECRET_TYPE_CATALOG { + assert!( + d.gts_id.starts_with(&format!( + "{GTS_ID_PREFIX}cf.core.credstore.secret.v1~cf.core.credstore." + )), + "{} not derived from the secret base type", + d.name + ); + assert!(d.gts_id.ends_with(".v1~")); + assert!( + !d.allow_sharing.is_empty(), + "{}: a type must allow at least one sharing mode", + d.name + ); + } +} + +#[test] +fn embedded_value_schemas_are_valid_json() { + for d in SECRET_TYPE_CATALOG { + if let Some(schema) = d.value_schema { + let parsed: serde_json::Value = + serde_json::from_str(schema).expect("schema must be valid JSON"); + assert!(parsed.is_object(), "{}: schema must be an object", d.name); + } + } +} + +#[test] +fn type_uuid_is_deterministic_and_matches_registry_v5() { + // Every catalog type has a resolvable deterministic UUID, and it is + // stable across calls (the stored key must never drift). + for d in SECRET_TYPE_CATALOG { + let a = type_uuid(d.gts_id).expect("catalog gts id resolves to a uuid"); + let b = type_uuid(d.gts_id).unwrap(); + assert_eq!(a, b, "{}: uuid must be deterministic", d.name); + assert_eq!(a.get_version_num(), 5, "{}: must be a v5 uuid", d.name); + } + // UUIDs are distinct per type. + let mut uuids: Vec<_> = SECRET_TYPE_CATALOG + .iter() + .map(|d| type_uuid(d.gts_id)) + .collect(); + uuids.sort(); + uuids.dedup(); + assert_eq!(uuids.len(), SECRET_TYPE_CATALOG.len(), "type uuids collide"); + + // Pin the generic default — this is the value the m0001 column DEFAULT + // must use; a drift here means the migration default is stale. + assert_eq!( + SecretType::generic().uuid().to_string(), + GENERIC_TYPE_UUID_STR, + "generic type uuid drifted; update the m0001 DEFAULT and this pin" + ); +} + +#[test] +fn resolution_by_name_and_gts_id_round_trips() { + for d in SECRET_TYPE_CATALOG { + let by_name = SecretType::from_name(d.name).expect("known name"); + assert_eq!(by_name.gts_id(), d.gts_id); + let by_id = SecretType::from_gts_id(d.gts_id).expect("known id"); + assert_eq!(by_id.name(), d.name); + } + assert!(SecretType::from_name("no-such-type").is_err()); + assert!(SecretType::from_gts_id(&format!("{GTS_ID_PREFIX}nope~")).is_none()); +} + +#[test] +fn generic_is_default_and_allows_everything() { + let g = SecretType::default(); + assert_eq!(g.name(), "generic"); + for m in [ + SharingMode::Private, + SharingMode::Tenant, + SharingMode::Shared, + ] { + assert!(g.descriptor().allows_sharing(m)); + } + assert!(!g.descriptor().expirable); +} + +#[test] +fn personal_token_is_private_only_flagship_restriction() { + let t = SecretType::from_name("personal-token").expect("known"); + assert!(t.descriptor().allows_sharing(SharingMode::Private)); + assert!(!t.descriptor().allows_sharing(SharingMode::Tenant)); + assert!(!t.descriptor().allows_sharing(SharingMode::Shared)); +} + +#[test] +fn serde_round_trip_uses_short_name() { + let t = SecretType::from_name("api-key").expect("known"); + let json = serde_json::to_string(&t).expect("serialize"); + assert_eq!(json, "\"api-key\""); + let back: SecretType = serde_json::from_str(&json).expect("deserialize"); + assert_eq!(back, t); + assert!(serde_json::from_str::("\"bogus\"").is_err()); +} diff --git a/gears/credstore/credstore/Cargo.toml b/gears/credstore/credstore/Cargo.toml index daab6ea78..41cc3d7cc 100644 --- a/gears/credstore/credstore/Cargo.toml +++ b/gears/credstore/credstore/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "cf-gears-credstore" -description = "credstore gateway module" +description = "credstore gear module" version = "0.1.24" edition.workspace = true license.workspace = true @@ -19,25 +19,54 @@ name = "credstore" workspace = true [dependencies] -# Trait boundaries of consumed SDKs are `CanonicalError` (ADR 0005). -toolkit-canonical-errors = { workspace = true } credstore-sdk = { workspace = true } +toolkit = { workspace = true } +toolkit-macros = { workspace = true } +toolkit-security = { workspace = true } +toolkit-db = { workspace = true } +toolkit-canonical-errors = { workspace = true } +authz-resolver-sdk = { workspace = true } +tenant-resolver-sdk = { workspace = true } types-registry-sdk = { workspace = true } -anyhow = { workspace = true } async-trait = { workspace = true } -tokio = { workspace = true } -tracing = { workspace = true } -inventory = { workspace = true } +axum = { workspace = true } +# Value-fingerprint fence (docs/features/001-value-fingerprint-fence.md): +# HMAC-SHA256 over the backend value + fence-key generation via AWS-LC +# (aws-lc-rs) — the workspace's FIPS-capable crypto backend, so the fence runs +# through the validated module under a `--features fips` build (RustCrypto's +# pure-Rust sha2/hmac, banned by the DE0708 FIPS-hasher lint, cannot). +aws-lc-rs = { workspace = true } +# Non-crypto retry jitter only (fence-key bootstrap de-sync); not key material. +rand = { workspace = true } +# Zeroize the in-process cached fence key on drop (highest-value single secret). +zeroize = { workspace = true } +sea-orm = { workspace = true } +sea-orm-migration = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } +jsonschema = { workspace = true } +tokio = { workspace = true } +tokio-util = { workspace = true } +tracing = { workspace = true } uuid = { workspace = true } thiserror = { workspace = true } - -toolkit = { workspace = true } -toolkit-security = { workspace = true } -toolkit-macros = { workspace = true } +anyhow = { workspace = true } +time = { workspace = true } +utoipa = { workspace = true } +opentelemetry = { workspace = true, features = ["metrics"] } [dev-dependencies] toolkit-gts = { workspace = true } +tower = { workspace = true } +toolkit-db = { workspace = true, features = ["sqlite"] } +tokio = { workspace = true, features = ["rt", "macros", "test-util"] } types-registry-sdk = { workspace = true, features = ["test-util"] } + +[features] +test-support = ["dep:opentelemetry_sdk"] + +[dependencies.opentelemetry_sdk] +workspace = true +features = ["metrics", "testing"] +optional = true diff --git a/gears/credstore/credstore/README.md b/gears/credstore/credstore/README.md index bd6fbe7ef..2f1f14bc0 100644 --- a/gears/credstore/credstore/README.md +++ b/gears/credstore/credstore/README.md @@ -1,17 +1,39 @@ # CredStore -Credential storage gateway gear. Discovers storage backend plugins via the types registry and routes secret operations through the selected plugin with hierarchical tenant resolution. +Stateful credential-storage gear module. Owns per-secret metadata in its own +database, enforces authorization in SQL, resolves secrets hierarchically across +the tenant tree, and stores the secret **value** in a backend plugin discovered +via the types registry. + +> Design: [`docs/DESIGN.md`](../docs/DESIGN.md) is the baseline; the shipped +> implementation is described in [`docs/DESIGN-ADDENDUM.md`](../docs/DESIGN-ADDENDUM.md) +> (stateful gear, `credstore_secrets` table, PDP-scope authz, versioning/ETag, +> write saga, tenant-isolation barriers). ## Overview -The `cf-gears-credstore` gear provides: +The `cf-gears-credstore` module provides: -- **Plugin discovery** — finds storage backend plugins via the types registry using a configured vendor -- **Secret routing** — delegates `get`/`put`/`delete` to the active plugin -- **Hierarchical resolution** — walks the tenant hierarchy to resolve inherited secrets -- **ClientHub integration** — registers `CredStoreClientV1` for inter-gear use +- **Local metadata** — a gear-owned `credstore_secrets` table (SecureORM / + sea-orm, migration `m0001`) holding sharing, owner, status, `version`, and + the value-fingerprint fence +- **PDP authorization** — `AccessScope` enforced in SQL via SecureORM clamps; + out-of-scope access is fail-closed (canonical 404, anti-enumeration) +- **Hierarchical resolution** — a single indexed query over the ancestor chain + (TTL+LRU cached, barrier-aware); the backend is read once for the winner's value +- **Value-fingerprint fence** — every read verifies the backend value against a + per-row `HMAC-SHA256` (key auto-stored in the backend, never on the wire), so + a metadata/value desync from a concurrent write fails closed instead of + disclosing a value under a foreign sharing label (DESIGN §4.10, ADR-0003) +- **Versioning** — strong generation-bound `ETag` (`"."`) on `GET`, + `If-Match` optimistic concurrency on `PUT`/`DELETE` (no ABA across recreation) +- **Crash-safe writes** — provisioning→backend→active saga with rollback and a reaper +- **Backend plugin** — value-only store discovered via the types registry (vendor) +- **ClientHub + REST** — registers `CredStoreClientV1`; exposes `/credstore/v1/secrets` -This gear depends on `types-registry`. All storage logic lives in the plugin (e.g. `cf-gears-static-credstore-plugin`). +This module depends on `types-registry`, `tenant-resolver`, and `authz-resolver`, +and **requires a database**. The secret value is stored in a plugin (e.g. +`cf-gears-static-credstore-plugin`, or an OpenBao-backed plugin). ## Usage @@ -29,9 +51,21 @@ if let Some(resp) = credstore.get(&ctx, &SecretRef::new("my-api-key")?).await? { ## Configuration -```toml -[credstore] -vendor = "x" # GTS vendor used to discover the storage plugin +The module requires a `database:` section (it is stateful). Gear config: + +```yaml +credstore: + database: + server: "sqlite_users" # a database server template; module gets its own file + file: "credstore.db" + config: + vendor: "virtuozzo" # GTS vendor used to discover the value-store plugin + hierarchy: + ancestor_cache_ttl_secs: 300 + tenant_closure_colocated: false # opt in only where the closure is co-located + reaper: + tick_secs: 60 + provisioning_timeout_secs: 300 ``` ## License diff --git a/gears/credstore/credstore/src/api.rs b/gears/credstore/credstore/src/api.rs new file mode 100644 index 000000000..404d99ed3 --- /dev/null +++ b/gears/credstore/credstore/src/api.rs @@ -0,0 +1,2 @@ +//! HTTP API surface for the credstore module. +pub mod rest; diff --git a/gears/credstore/credstore/src/api/rest.rs b/gears/credstore/credstore/src/api/rest.rs new file mode 100644 index 000000000..2bf3237c6 --- /dev/null +++ b/gears/credstore/credstore/src/api/rest.rs @@ -0,0 +1,6 @@ +//! REST API for the credstore module. +pub mod dto; +pub mod handlers; +pub mod routes; + +pub use routes::register_routes; diff --git a/gears/credstore/credstore/src/api/rest/dto.rs b/gears/credstore/credstore/src/api/rest/dto.rs new file mode 100644 index 000000000..f7b783099 --- /dev/null +++ b/gears/credstore/credstore/src/api/rest/dto.rs @@ -0,0 +1,203 @@ +//! REST DTOs for the credstore module. + +use credstore_sdk::{GetSecretResponse, SharingMode}; +use uuid::Uuid; + +use crate::domain::error::DomainError; + +/// Sharing mode for the REST transport layer. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +#[toolkit_macros::api_dto(response, request)] +pub enum SharingModeDto { + /// Only the owner can access the secret. + Private, + /// Any actor inside the owning tenant can access the secret. + #[default] + Tenant, + /// Descendant tenants can inherit the secret. + Shared, +} + +impl From for SharingModeDto { + fn from(value: SharingMode) -> Self { + match value { + SharingMode::Private => Self::Private, + SharingMode::Tenant => Self::Tenant, + SharingMode::Shared => Self::Shared, + } + } +} + +impl From for SharingMode { + fn from(value: SharingModeDto) -> Self { + match value { + SharingModeDto::Private => Self::Private, + SharingModeDto::Tenant => Self::Tenant, + SharingModeDto::Shared => Self::Shared, + } + } +} + +/// Request body for `POST /credstore/v1/secrets`. +/// +/// `Debug` is hand-written to redact `value` — a derived `Debug` would expose +/// the plaintext secret if this DTO is ever `{:?}`-logged by a future layer. +#[derive(Clone, PartialEq, Eq)] +#[toolkit_macros::api_dto(request)] +#[serde(deny_unknown_fields)] +pub struct CreateSecretRequestDto { + /// Secret reference key — `[a-zA-Z0-9_-]+`, max 255 characters. + #[schema(pattern = "^[A-Za-z0-9_-]+$", min_length = 1, max_length = 255)] + pub reference: String, + /// Secret value as a UTF-8 string. + pub value: String, + /// Sharing mode for the secret. + #[serde(default)] + pub sharing: SharingModeDto, + /// Secret type as a full GTS type id (built-in or custom); defaults to + /// the generic type. Resolved and existence-checked against the + /// types-registry. + #[serde(default, rename = "type")] + pub secret_type: Option, + /// Expiry instant (RFC 3339); only for expirable types. + #[serde(default)] + #[schema(format = DateTime)] + pub expires_at: Option, +} + +impl std::fmt::Debug for CreateSecretRequestDto { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("CreateSecretRequestDto") + .field("reference", &self.reference) + .field("value", &"[REDACTED]") + .field("sharing", &self.sharing) + .field("secret_type", &self.secret_type) + .field("expires_at", &self.expires_at) + .finish() + } +} + +/// Request body for `PUT /credstore/v1/secrets/{ref}`. +/// +/// `Debug` is hand-written to redact `value` (see [`CreateSecretRequestDto`]). +#[derive(Clone, PartialEq, Eq)] +#[toolkit_macros::api_dto(request)] +#[serde(deny_unknown_fields)] +pub struct UpdateSecretRequestDto { + /// Secret value as a UTF-8 string. + pub value: String, + /// Sharing mode for the secret. **Optional: when omitted on an overwrite + /// the secret's current sharing is preserved**, so rotating a `shared` + /// secret's value with `{"value": "..."}` alone no longer silently narrows + /// it back to `tenant` (review finding #8). On create-via-upsert an omitted + /// `sharing` defaults to `tenant`. Send an explicit value to change the + /// sharing mode. + #[serde(default)] + pub sharing: Option, + /// Secret type as a full GTS type id. Optional; when present must match + /// the existing secret's type (the type is immutable). On create via + /// upsert it selects the new secret's type (default: the generic type). + #[serde(default, rename = "type")] + pub secret_type: Option, + /// Expiry instant (RFC 3339); only for expirable types. A PUT is a + /// whole-value replace: omitting `expires_at` clears a stored expiry. + #[serde(default)] + #[schema(format = DateTime)] + pub expires_at: Option, +} + +impl std::fmt::Debug for UpdateSecretRequestDto { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("UpdateSecretRequestDto") + .field("value", &"[REDACTED]") + .field("sharing", &self.sharing) + .field("secret_type", &self.secret_type) + .field("expires_at", &self.expires_at) + .finish() + } +} + +/// Access metadata returned alongside the secret value. +#[derive(Debug, Clone, PartialEq, Eq)] +#[toolkit_macros::api_dto(response)] +pub struct SecretMetadataDto { + /// The tenant that owns this secret. + pub owner_tenant_id: Uuid, + /// The sharing mode that governed the lookup result. + pub sharing: SharingModeDto, + /// Whether the secret came from an ancestor tenant. + pub is_inherited: bool, + /// Monotonic version of the resolved secret (also returned as `ETag`). + pub version: i64, + /// Secret type as its full GTS type id. + #[serde(rename = "type")] + pub secret_type: String, + /// Expiry instant (RFC 3339), when set. + #[serde(skip_serializing_if = "Option::is_none")] + #[schema(format = DateTime)] + pub expires_at: Option, +} + +/// Response body for `GET /credstore/v1/secrets/{ref}`. +/// +/// `Debug` is hand-written to redact `value` (see [`CreateSecretRequestDto`]). +#[derive(Clone, PartialEq, Eq)] +#[toolkit_macros::api_dto(response)] +pub struct GetSecretResponseDto { + /// Secret value as a UTF-8 string. + pub value: String, + /// Access metadata for the resolved secret. + pub metadata: SecretMetadataDto, +} + +impl std::fmt::Debug for GetSecretResponseDto { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("GetSecretResponseDto") + .field("value", &"[REDACTED]") + .field("metadata", &self.metadata) + .finish() + } +} + +impl GetSecretResponseDto { + /// Convert the domain [`GetSecretResponse`] into the REST DTO shape. + /// + /// # Errors + /// + /// Returns [`DomainError::Internal`] when the secret value is not valid + /// UTF-8 (e.g. binary written via the SDK). The JSON/string transport cannot + /// represent it, and lossy decoding would silently corrupt the secret, so we + /// reject rather than mangle. + pub fn try_from_response(resp: &GetSecretResponse) -> Result { + let value = String::from_utf8(resp.value.as_bytes().to_vec()).map_err(|_| { + DomainError::internal( + "secret value is not valid UTF-8 and cannot be encoded for the REST transport", + ) + })?; + let expires_at = resp + .expires_at + .map(|at| { + at.format(&time::format_description::well_known::Rfc3339) + .map_err(|e| DomainError::internal(format!("expires_at failed to format: {e}"))) + }) + .transpose()?; + // The type is named by its full GTS id — symmetric with the + // create/update contract, which accepts a full GTS type id. + let secret_type = resp.secret_type.clone(); + Ok(Self { + value, + metadata: SecretMetadataDto { + owner_tenant_id: resp.owner_tenant_id.0, + sharing: resp.sharing.into(), + is_inherited: resp.is_inherited, + version: resp.version, + secret_type, + expires_at, + }, + }) + } +} + +#[cfg(test)] +#[path = "dto_tests.rs"] +mod tests; diff --git a/gears/credstore/credstore/src/api/rest/dto_tests.rs b/gears/credstore/credstore/src/api/rest/dto_tests.rs new file mode 100644 index 000000000..fd46f3e58 --- /dev/null +++ b/gears/credstore/credstore/src/api/rest/dto_tests.rs @@ -0,0 +1,112 @@ +//! Unit tests for the credstore REST DTOs. + +use credstore_sdk::{SecretType, SecretValue, TenantId}; +use uuid::Uuid; + +use super::*; + +#[test] +fn get_secret_response_dto_debug_redacts_value() { + let dto = GetSecretResponseDto { + value: "super-secret-value".to_owned(), + metadata: SecretMetadataDto { + owner_tenant_id: Uuid::nil(), + sharing: SharingModeDto::Tenant, + is_inherited: false, + version: 1, + secret_type: "generic".to_owned(), + expires_at: None, + }, + }; + let debug = format!("{dto:?}"); + assert!( + debug.contains("[REDACTED]"), + "debug must contain [REDACTED]" + ); + assert!( + !debug.contains("super-secret-value"), + "debug must not contain the plaintext value" + ); +} + +#[test] +fn get_secret_response_maps_to_documented_rest_shape() { + let resp = GetSecretResponse { + value: SecretValue::from("my-value"), + id: Uuid::new_v4(), + owner_tenant_id: TenantId(Uuid::nil()), + sharing: SharingMode::Shared, + is_inherited: true, + version: 7, + secret_type: SecretType::generic().gts_id().to_owned(), + expires_at: None, + }; + let dto = GetSecretResponseDto::try_from_response(&resp).expect("utf-8 value"); + assert_eq!(dto.value, "my-value"); + assert_eq!(dto.metadata.owner_tenant_id, Uuid::nil()); + assert_eq!(dto.metadata.sharing, SharingModeDto::Shared); + assert!(dto.metadata.is_inherited); + assert_eq!(dto.metadata.version, 7); +} + +#[test] +fn try_from_response_rejects_non_utf8_value() { + // A binary value (written via the SDK) must not be silently corrupted by + // lossy decoding — it is rejected with a typed error instead. + let resp = GetSecretResponse { + value: SecretValue::new(vec![0xff, 0xfe, 0x00]), + id: Uuid::new_v4(), + owner_tenant_id: TenantId(Uuid::nil()), + sharing: SharingMode::Tenant, + is_inherited: false, + version: 1, + secret_type: SecretType::generic().gts_id().to_owned(), + expires_at: None, + }; + let err = GetSecretResponseDto::try_from_response(&resp) + .expect_err("non-UTF-8 value must be rejected, not lossily decoded"); + assert!(matches!( + err, + crate::domain::error::DomainError::Internal { .. } + )); +} + +#[test] +fn sharing_mode_roundtrip() { + for (dto, sdk) in [ + (SharingModeDto::Private, SharingMode::Private), + (SharingModeDto::Tenant, SharingMode::Tenant), + (SharingModeDto::Shared, SharingMode::Shared), + ] { + assert_eq!(SharingModeDto::from(sdk), dto); + assert_eq!(SharingMode::from(dto), sdk); + } +} + +#[test] +fn create_request_debug_redacts_value() { + let dto = CreateSecretRequestDto { + reference: "my-ref".to_owned(), + value: "super-secret-value".to_owned(), + sharing: SharingModeDto::default(), + secret_type: None, + expires_at: None, + }; + let debug = format!("{dto:?}"); + assert!(debug.contains("[REDACTED]")); + assert!(!debug.contains("super-secret-value")); + assert!(debug.contains("my-ref")); +} + +#[test] +fn update_request_debug_redacts_value() { + let dto = UpdateSecretRequestDto { + value: "another-secret".to_owned(), + sharing: Some(SharingModeDto::default()), + secret_type: None, + expires_at: None, + }; + let debug = format!("{dto:?}"); + assert!(debug.contains("[REDACTED]")); + assert!(!debug.contains("another-secret")); +} diff --git a/gears/credstore/credstore/src/api/rest/handlers.rs b/gears/credstore/credstore/src/api/rest/handlers.rs new file mode 100644 index 000000000..084ba3e1c --- /dev/null +++ b/gears/credstore/credstore/src/api/rest/handlers.rs @@ -0,0 +1,267 @@ +//! REST handlers for the credstore module. + +use std::sync::Arc; + +use axum::Json; +use axum::extract::{Extension, Path}; +use axum::http::StatusCode; +use axum::response::IntoResponse; +use credstore_sdk::{ExpiryWrite, GtsId, SecretRef, SecretValue, SharingMode, WriteOptions}; +use toolkit::api::canonical_prelude::*; +use toolkit_security::SecurityContext; + +use super::dto::{CreateSecretRequestDto, GetSecretResponseDto, UpdateSecretRequestDto}; +use crate::domain::error::DomainError; +use crate::domain::secret::model::{WritePrecondition, WriteSpec}; +use crate::domain::secret::service::Service; + +/// Concrete service alias for the handlers. +pub(crate) type ConcreteService = Service; + +/// Parse an optional `If-Match` precondition (RFC 7232 §3.1). The GET handler +/// emits a strong, generation-bound `"."` validator (`id` = the +/// row UUID, fresh per recreated secret — no ABA reuse across generations). +/// +/// `If-Match` is `"*" / 1#entity-tag`: it may span multiple header lines and +/// each line may carry a comma-separated list, and the list matches if **any** +/// validator matches. We accept `*` (target must exist) or one-or-more strong +/// `"."` validators; a single one yields [`WritePrecondition::Version`], +/// several yield [`WritePrecondition::AnyVersion`]. Weak validators (`W/"…"`) +/// and any other shape are a typed 400. +fn parse_if_match( + headers: &axum::http::HeaderMap, +) -> Result, DomainError> { + let mut lines = headers + .get_all(axum::http::header::IF_MATCH) + .iter() + .peekable(); + if lines.peek().is_none() { + return Ok(None); + } + let malformed = || DomainError::InvalidPrecondition { + detail: "If-Match must be `*` or quoted `.` ETag(s)".to_owned(), + }; + let mut validators: Vec<(uuid::Uuid, i64)> = Vec::new(); + for raw in lines { + let line = raw.to_str().map_err(|_| DomainError::InvalidPrecondition { + detail: "If-Match header is not valid ASCII".to_owned(), + })?; + for tag in line.split(',').map(str::trim).filter(|t| !t.is_empty()) { + // `*` matches any current representation → an existence check; it + // subsumes any other member, so return immediately. + if tag == "*" { + return Ok(Some(WritePrecondition::Exists)); + } + // Strong validator only: `"."` (a UUID contains no + // `.`, so the split is unambiguous). + let parsed = tag + .strip_prefix('"') + .and_then(|s| s.strip_suffix('"')) + .and_then(|inner| inner.split_once('.')) + .and_then(|(id, version)| { + Some(( + uuid::Uuid::parse_str(id).ok()?, + version.parse::().ok()?, + )) + }) + .ok_or_else(malformed)?; + validators.push(parsed); + } + } + match validators.as_slice() { + [] => Err(malformed()), + [(id, version)] => Ok(Some(WritePrecondition::Version { + id: *id, + version: *version, + })), + _ => Ok(Some(WritePrecondition::AnyVersion(validators))), + } +} + +/// Parse the optional typed-write fields shared by the create/update DTOs +/// into [`WriteOptions`]: a full GTS type id (`GtsId`) and an RFC 3339 +/// expiry. A `type` that is not a well-formed GTS type id and malformed +/// timestamps are typed 400s; whether a well-formed custom type actually +/// exists is decided by the service against the types-registry. +/// +/// Expiry follows REST whole-value-replace semantics: a present `expires_at` +/// sets it ([`ExpiryWrite::Set`]) and an omitted one clears any stored expiry +/// ([`ExpiryWrite::Clear`]). (This differs from the SDK convenience +/// [`put`](credstore_sdk::CredStoreClientV1::put), which defaults to +/// [`ExpiryWrite::Preserve`](credstore_sdk::ExpiryWrite::Preserve).) +fn parse_write_options( + secret_type: Option<&str>, + expires_at: Option<&str>, +) -> Result { + let secret_type = secret_type + .map(|input| { + GtsId::try_new(input).map_err(|_| DomainError::TypeViolation { + field: "type", + reason: crate::domain::secret::typing::reasons::UNKNOWN_SECRET_TYPE, + detail: format!("secret type must be a full GTS type id: {input}"), + }) + }) + .transpose()?; + let expires_at = expires_at + .map(|raw| { + time::OffsetDateTime::parse(raw, &time::format_description::well_known::Rfc3339) + .map(ExpiryWrite::Set) + .map_err(|_| DomainError::TypeViolation { + field: "expires_at", + reason: "INVALID_EXPIRES_AT", + detail: "expires_at must be an RFC 3339 timestamp".to_owned(), + }) + }) + .transpose()? + .unwrap_or(ExpiryWrite::Clear); + Ok(WriteOptions { + secret_type, + expires_at, + // REST carries the optimistic-concurrency precondition in the `If-Match` + // header (parsed into `WriteSpec::precondition`), not in `WriteOptions`. + precondition: None, + }) +} + +/// `POST /credstore/v1/secrets` +/// +/// # Errors +/// +/// Returns a canonical `Problem` envelope on invalid reference (400), access denied (403), +/// conflict (409), or service unavailable (503). +pub async fn create_secret( + uri: axum::http::Uri, + Extension(ctx): Extension, + Extension(svc): Extension>, + Json(body): Json, +) -> ApiResult { + let key = SecretRef::new(body.reference).map_err(|e| { + CanonicalError::from(DomainError::InvalidSecretRef { + detail: e.to_string(), + }) + })?; + let opts = parse_write_options(body.secret_type.as_deref(), body.expires_at.as_deref()) + .map_err(CanonicalError::from)?; + svc.put( + &ctx, + &key, + SecretValue::from(body.value), + WriteSpec::create(body.sharing.into()).with_opts(opts), + ) + .await?; + let location = format!("{}/{}", uri.path().trim_end_matches('/'), key.as_ref()); + Ok(( + StatusCode::CREATED, + [(axum::http::header::LOCATION, location)], + ) + .into_response()) +} + +/// `PUT /credstore/v1/secrets/{ref}` +/// +/// Honours an optional `If-Match` precondition: a stale version yields a +/// canonical `Aborted` (409, `OPTIMISTIC_LOCK_FAILURE`). +/// +/// # Errors +/// +/// Returns a canonical `Problem` envelope on invalid reference / malformed +/// `If-Match` (400), access denied (403), unsupported transition (400), +/// version precondition failure (409), or service unavailable (503). +pub async fn put_secret( + Extension(ctx): Extension, + Extension(svc): Extension>, + Path(reference): Path, + headers: axum::http::HeaderMap, + Json(body): Json, +) -> ApiResult { + let precondition = parse_if_match(&headers)?; + let key = SecretRef::new(reference).map_err(|e| { + CanonicalError::from(DomainError::InvalidSecretRef { + detail: e.to_string(), + }) + })?; + let opts = parse_write_options(body.secret_type.as_deref(), body.expires_at.as_deref()) + .map_err(CanonicalError::from)?; + // An omitted `sharing` preserves the existing secret's mode on overwrite + // (finding #8); `tenant` is only the create-via-upsert / class default. + let (sharing, preserve_sharing) = match body.sharing { + Some(s) => (s.into(), false), + None => (SharingMode::Tenant, true), + }; + svc.put( + &ctx, + &key, + SecretValue::from(body.value), + WriteSpec::upsert(sharing) + .preserve_sharing(preserve_sharing) + .with_precondition(precondition) + .with_opts(opts), + ) + .await?; + Ok(StatusCode::NO_CONTENT.into_response()) +} + +/// `GET /credstore/v1/secrets/{ref}` +/// +/// # Errors +/// +/// Returns a canonical `Problem` envelope on invalid reference (400), access denied (403), +/// not found (404), or service unavailable (503). +pub async fn get_secret( + Extension(ctx): Extension, + Extension(svc): Extension>, + Path(reference): Path, +) -> ApiResult { + let key = SecretRef::new(reference).map_err(|e| { + CanonicalError::from(DomainError::InvalidSecretRef { + detail: e.to_string(), + }) + })?; + match svc.get(&ctx, &key).await? { + Some(ref resp) => { + // Reject (don't lossily decode) a non-UTF-8 value before building headers. + let dto = GetSecretResponseDto::try_from_response(resp)?; + // Strong, generation-bound validator: the row UUID (fresh per + // recreated secret) plus the per-generation monotonic counter, so + // validators never repeat across delete+recreate (no ABA). + let etag = format!("\"{}.{}\"", resp.id, resp.version); + Ok(( + StatusCode::OK, + [ + (axum::http::header::ETAG, etag), + // Secret material must never be cached by intermediaries. + (axum::http::header::CACHE_CONTROL, "no-store".to_owned()), + ], + Json(dto), + ) + .into_response()) + } + None => Err(CanonicalError::from(DomainError::NotFound)), + } +} + +/// `DELETE /credstore/v1/secrets/{ref}` +/// +/// Honours an optional `If-Match` precondition: a stale version yields a +/// canonical `Aborted` (409, `OPTIMISTIC_LOCK_FAILURE`). +/// +/// # Errors +/// +/// Returns a canonical `Problem` envelope on invalid reference / malformed +/// `If-Match` (400), access denied (403), not found (404), version precondition +/// failure (409), or service unavailable (503). +pub async fn delete_secret( + Extension(ctx): Extension, + Extension(svc): Extension>, + Path(reference): Path, + headers: axum::http::HeaderMap, +) -> ApiResult { + let precondition = parse_if_match(&headers)?; + let key = SecretRef::new(reference).map_err(|e| { + CanonicalError::from(DomainError::InvalidSecretRef { + detail: e.to_string(), + }) + })?; + svc.delete(&ctx, &key, precondition).await?; + Ok(StatusCode::NO_CONTENT.into_response()) +} diff --git a/gears/credstore/credstore/src/api/rest/routes.rs b/gears/credstore/credstore/src/api/rest/routes.rs new file mode 100644 index 000000000..427083623 --- /dev/null +++ b/gears/credstore/credstore/src/api/rest/routes.rs @@ -0,0 +1,136 @@ +//! REST route registration for the credstore module. + +use std::sync::Arc; + +use axum::Router; +use axum::http::StatusCode; +use toolkit::api::{OpenApiRegistry, OperationBuilder, ParamLocation, ParamSpec}; + +use super::dto::{CreateSecretRequestDto, GetSecretResponseDto, UpdateSecretRequestDto}; +use super::handlers::{self, ConcreteService}; + +const TAG: &str = "Credential Store"; + +/// Optional `If-Match` precondition header for guarded writes/deletes (RFC 7232 +/// §3.1). `*` requires the secret to exist; a quoted `"."` `ETag` +/// requires the current version to match, otherwise the operation fails with +/// `409 OPTIMISTIC_LOCK_FAILURE`. +fn if_match_param() -> ParamSpec { + ParamSpec { + name: "If-Match".to_owned(), + location: ParamLocation::Header, + required: false, + description: Some( + "Optimistic-concurrency precondition (RFC 7232). `*` requires the secret to \ + exist; a quoted `\".\"` ETag requires the current version to \ + match, otherwise the request fails with `409 OPTIMISTIC_LOCK_FAILURE`." + .to_owned(), + ), + param_type: "string".to_owned(), + } +} + +/// Register all REST routes for the credstore module. +pub fn register_routes( + router: Router, + openapi: &dyn OpenApiRegistry, + svc: Arc, +) -> Router { + let router = OperationBuilder::post("/credstore/v1/secrets") + .operation_id("credstore.create_secret") + .summary("Create a secret") + .description("Create a new secret for the authenticated tenant.") + .tag(TAG) + .authenticated() + .no_license_required() + .json_request::( + openapi, + "Secret reference, value, and sharing mode", + ) + .handler(handlers::create_secret) + .no_content_response(StatusCode::CREATED, "Secret created (see Location header)") + .error_400(openapi) + .error_401(openapi) + .error_403(openapi) + .error_409(openapi) + .error_500(openapi) + .error_503(openapi) + .register(router, openapi); + + let router = OperationBuilder::put("/credstore/v1/secrets/{ref}") + .operation_id("credstore.put_secret") + .summary("Create or update a secret by reference") + .description("Store a secret for the authenticated tenant.") + .tag(TAG) + .authenticated() + .no_license_required() + .path_param( + "ref", + "Secret reference (`[a-zA-Z0-9_-]+`, maximum length 255 characters)", + ) + .param(if_match_param()) + .json_request::(openapi, "Secret value and sharing mode") + .handler(handlers::put_secret) + .no_content_response(StatusCode::NO_CONTENT, "Secret stored") + .error_400(openapi) + .error_401(openapi) + .error_403(openapi) + .error_409(openapi) + .error_500(openapi) + .error_503(openapi) + .register(router, openapi); + + let router = OperationBuilder::get("/credstore/v1/secrets/{ref}") + .operation_id("credstore.get_secret") + .summary("Get a secret by reference") + .description("Retrieve a secret for the authenticated tenant, with walk-up resolution.") + .tag(TAG) + .authenticated() + .no_license_required() + .path_param( + "ref", + "Secret reference (`[a-zA-Z0-9_-]+`, maximum length 255 characters)", + ) + .handler(handlers::get_secret) + .json_response_with_schema::( + openapi, + StatusCode::OK, + "Resolved secret value and metadata", + ) + .error_400(openapi) + .error_401(openapi) + .error_403(openapi) + .error_404(openapi) + .error_500(openapi) + .error_503(openapi) + .register(router, openapi); + + let router = OperationBuilder::delete("/credstore/v1/secrets/{ref}") + .operation_id("credstore.delete_secret") + .summary("Delete a secret by reference") + .description("Delete a secret owned by the authenticated tenant.") + .tag(TAG) + .authenticated() + .no_license_required() + .path_param( + "ref", + "Secret reference (`[a-zA-Z0-9_-]+`, maximum length 255 characters)", + ) + .param(if_match_param()) + .handler(handlers::delete_secret) + .no_content_response(StatusCode::NO_CONTENT, "Secret deleted") + .error_400(openapi) + .error_401(openapi) + .error_403(openapi) + .error_404(openapi) + .error_409(openapi) + .error_500(openapi) + .error_503(openapi) + .register(router, openapi); + + router.layer(axum::Extension(svc)) +} + +#[cfg(test)] +#[path = "routes_tests.rs"] +mod tests; diff --git a/gears/credstore/credstore/src/api/rest/routes_tests.rs b/gears/credstore/credstore/src/api/rest/routes_tests.rs new file mode 100644 index 000000000..8714d2c7c --- /dev/null +++ b/gears/credstore/credstore/src/api/rest/routes_tests.rs @@ -0,0 +1,484 @@ +#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] + +use std::sync::Arc; + +use axum::Router; +use axum::body::{Body, to_bytes}; +use axum::http::{Request, StatusCode}; +use toolkit::api::OpenApiRegistryImpl; +use toolkit_gts::gts_id; +use toolkit_security::SecurityContext; +use tower::ServiceExt; +use uuid::Uuid; + +use crate::domain::ports::metrics::CredStoreMetricsPort; +use crate::domain::ports::plugin::PluginSelector; +use crate::domain::resolver::TenantDirectory; +use crate::domain::secret::model::{SecretRow, SecretStatus}; +use crate::domain::secret::repo::SecretRepo; +use crate::domain::secret::service::{ReaperSettings, Service}; +use crate::domain::secret::test_support::{ + FakeDir, FakeMetrics, FakePlugin, FakePluginSelector, FakeSecretRepo, catalog_type_resolver, + make_ctx, mock_enforcer, +}; +use credstore_sdk::{OwnerId, SecretRef, SecretType, SecretValue, SharingMode, TenantId}; + +use super::register_routes; + +// ── Harness helpers ────────────────────────────────────────────────────────── + +fn test_subject() -> Uuid { + Uuid::from_u128(0xAAAA) +} + +fn test_tenant() -> Uuid { + Uuid::from_u128(0xBBBB) +} + +fn test_ctx() -> SecurityContext { + make_ctx(test_subject(), test_tenant()) +} + +struct TestHarness { + router: Router, + repo: Arc, + plugin: Arc, +} + +fn build_harness() -> TestHarness { + let repo = Arc::new(FakeSecretRepo::new()); + let plugin = FakePlugin::new(); + let selector = Arc::new(FakePluginSelector::new(Arc::clone(&plugin))); + let enforcer = mock_enforcer(); + let dir = Arc::new(FakeDir::single(test_tenant())); + let metrics = FakeMetrics::new(); + let svc = Arc::new(Service::new( + Arc::clone(&repo) as Arc, + dir as Arc, + enforcer, + selector as Arc, + catalog_type_resolver(), + metrics as Arc, + ReaperSettings { + tick_secs: 60, + provisioning_timeout_secs: 300, + deprovisioning_timeout_secs: 300, + }, + )); + let openapi = OpenApiRegistryImpl::new(); + let router = register_routes(Router::new(), &openapi, svc); + TestHarness { + router, + repo, + plugin, + } +} + +/// Build a JSON request with the `SecurityContext` injected as an extension. +fn json_request( + method: &str, + uri: &str, + body: Option, + ctx: SecurityContext, +) -> Request { + let mut builder = Request::builder().method(method).uri(uri); + if body.is_some() { + builder = builder.header("content-type", "application/json"); + } + let body_bytes = match body { + Some(json) => Body::from(serde_json::to_vec(&json).unwrap()), + None => Body::empty(), + }; + let mut req = builder.body(body_bytes).unwrap(); + req.extensions_mut().insert(ctx); + req +} + +async fn body_json(resp: axum::response::Response) -> serde_json::Value { + let bytes = to_bytes(resp.into_body(), 1024 * 64).await.unwrap(); + serde_json::from_slice(&bytes).unwrap_or(serde_json::Value::Null) +} + +// ── Seed helpers ───────────────────────────────────────────────────────────── + +/// Seed an active `Tenant`-shared row directly into the fake repo AND plugin. +/// Returns the row id — the generation half of the `"."` +/// validator the `If-Match` tests build. +async fn seed_secret(harness: &TestHarness, reference: &str, value: &str) -> Uuid { + use credstore_sdk::CredStorePluginClientV1; + let key = SecretRef::new(reference).expect("valid ref"); + let tenant = TenantId(test_tenant()); + let owner = OwnerId(test_subject()); + let row_id = Uuid::new_v4(); + harness.repo.seed(SecretRow { + id: row_id, + tenant_id: tenant, + reference: reference.to_owned(), + sharing: SharingMode::Tenant, + owner_id: owner, + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + // Also prime the plugin store so `svc.get` fetches a real value. + harness + .plugin + .put(&test_ctx(), &tenant, &key, SecretValue::from(value), None) + .await + .expect("plugin put"); + row_id +} + +// ── Tests ───────────────────────────────────────────────────────────────────── + +#[tokio::test] +async fn post_create_returns_201_with_location() { + let h = build_harness(); + let req = json_request( + "POST", + "/credstore/v1/secrets", + Some(serde_json::json!({ + "reference": "mykey", + "value": "mysecret", + "sharing": "tenant" + })), + test_ctx(), + ); + let resp = h.router.oneshot(req).await.expect("router"); + assert_eq!(resp.status(), StatusCode::CREATED); + let location = resp + .headers() + .get(axum::http::header::LOCATION) + .expect("Location header") + .to_str() + .expect("ascii") + .to_owned(); + assert!( + location.ends_with("/mykey"), + "Location must end with /mykey, got {location}" + ); +} + +#[tokio::test] +async fn post_type_is_full_gts_id_only() { + // The `type` field is a full GTS type id only (PR #4204 review C9): the + // catalog short name and the type's raw UUID are no longer accepted, and + // the response echoes the full GTS id. + let api_key = SecretType::from_name("api-key").expect("known"); + + let h = build_harness(); + let req = json_request( + "POST", + "/credstore/v1/secrets", + Some(serde_json::json!({ + "reference": "byid", + "value": "v", + "sharing": "tenant", + "type": api_key.gts_id() + })), + test_ctx(), + ); + let resp = h.router.clone().oneshot(req).await.expect("router"); + assert_eq!(resp.status(), StatusCode::CREATED); + + let get = json_request("GET", "/credstore/v1/secrets/byid", None, test_ctx()); + let resp = h.router.oneshot(get).await.expect("router"); + assert_eq!(resp.status(), StatusCode::OK); + let body = body_json(resp).await; + assert_eq!(body["metadata"]["type"], api_key.gts_id()); + + // Short name and raw UUID are not GTS type ids → rejected at the transport. + for bad in ["api-key".to_owned(), api_key.uuid().to_string()] { + let h = build_harness(); + let req = json_request( + "POST", + "/credstore/v1/secrets", + Some(serde_json::json!({ + "reference": "badtype", + "value": "v", + "sharing": "tenant", + "type": bad + })), + test_ctx(), + ); + let resp = h.router.clone().oneshot(req).await.expect("router"); + assert_eq!(resp.status(), StatusCode::BAD_REQUEST, "{bad}"); + } +} + +#[tokio::test] +async fn post_unknown_custom_type_returns_400_unknown_secret_type() { + // A well-formed but unregistered custom GTS type id passes transport + // parsing and is rejected by the (catalog-backed test) resolver with + // the same 400 the registry-driven resolver produces. + let h = build_harness(); + let req = json_request( + "POST", + "/credstore/v1/secrets", + Some(serde_json::json!({ + "reference": "customkey", + "value": "v", + "sharing": "tenant", + "type": gts_id!("cf.core.credstore.secret.v1~acme.connectors.creds.db_password.v1~") + })), + test_ctx(), + ); + let resp = h.router.oneshot(req).await.expect("router"); + assert_eq!(resp.status(), StatusCode::BAD_REQUEST); + let body = body_json(resp).await; + assert_eq!( + body["context"]["field_violations"][0]["reason"], "UNKNOWN_SECRET_TYPE", + "body: {body}" + ); +} + +#[tokio::test] +async fn post_duplicate_returns_409() { + let h = build_harness(); + // First create + let req1 = json_request( + "POST", + "/credstore/v1/secrets", + Some(serde_json::json!({ + "reference": "dupkey", + "value": "v1", + "sharing": "tenant" + })), + test_ctx(), + ); + let r1 = h.router.clone().oneshot(req1).await.expect("router"); + assert_eq!(r1.status(), StatusCode::CREATED); + + // Second create — should conflict + let req2 = json_request( + "POST", + "/credstore/v1/secrets", + Some(serde_json::json!({ + "reference": "dupkey", + "value": "v2", + "sharing": "tenant" + })), + test_ctx(), + ); + let r2 = h.router.oneshot(req2).await.expect("router"); + assert_eq!(r2.status(), StatusCode::CONFLICT); +} + +#[tokio::test] +async fn get_existing_returns_200_with_body() { + let h = build_harness(); + seed_secret(&h, "getkey", "hello-world").await; + + let req = json_request("GET", "/credstore/v1/secrets/getkey", None, test_ctx()); + let resp = h.router.oneshot(req).await.expect("router"); + assert_eq!(resp.status(), StatusCode::OK); + let body = body_json(resp).await; + assert_eq!(body["value"], "hello-world"); + assert_eq!(body["metadata"]["sharing"], "tenant"); + assert_eq!(body["metadata"]["is_inherited"], false); +} + +#[tokio::test] +async fn get_response_is_not_cacheable() { + let h = build_harness(); + seed_secret(&h, "cachekey", "topsecret").await; + + let req = json_request("GET", "/credstore/v1/secrets/cachekey", None, test_ctx()); + let resp = h.router.oneshot(req).await.expect("router"); + assert_eq!(resp.status(), StatusCode::OK); + let cc = resp + .headers() + .get(axum::http::header::CACHE_CONTROL) + .expect("GET secret must set Cache-Control") + .to_str() + .expect("ascii"); + assert!( + cc.contains("no-store"), + "secret material must not be cached; Cache-Control was {cc:?}" + ); +} + +#[tokio::test] +async fn get_missing_returns_404() { + let h = build_harness(); + let req = json_request("GET", "/credstore/v1/secrets/nokey", None, test_ctx()); + let resp = h.router.oneshot(req).await.expect("router"); + assert_eq!(resp.status(), StatusCode::NOT_FOUND); +} + +/// Build a request with an `If-Match` header set. +fn json_request_if_match( + method: &str, + uri: &str, + body: Option, + if_match: &str, + ctx: SecurityContext, +) -> Request { + let mut req = json_request(method, uri, body, ctx); + req.headers_mut().insert( + axum::http::header::IF_MATCH, + axum::http::HeaderValue::from_str(if_match).expect("ascii"), + ); + req +} + +#[tokio::test] +async fn put_with_matching_if_match_returns_204() { + let h = build_harness(); + let row_id = seed_secret(&h, "ocp", "old").await; // seeded at version 1 + + let req = json_request_if_match( + "PUT", + "/credstore/v1/secrets/ocp", + Some(serde_json::json!({ "value": "new", "sharing": "tenant" })), + &format!("\"{row_id}.1\""), + test_ctx(), + ); + let resp = h.router.oneshot(req).await.expect("router"); + assert_eq!(resp.status(), StatusCode::NO_CONTENT); +} + +#[tokio::test] +async fn put_with_stale_if_match_returns_409() { + let h = build_harness(); + let row_id = seed_secret(&h, "ocp", "old").await; // version 1 + + let req = json_request_if_match( + "PUT", + "/credstore/v1/secrets/ocp", + Some(serde_json::json!({ "value": "new", "sharing": "tenant" })), + &format!("\"{row_id}.999\""), + test_ctx(), + ); + let resp = h.router.oneshot(req).await.expect("router"); + assert_eq!(resp.status(), StatusCode::CONFLICT); +} + +#[tokio::test] +async fn put_with_malformed_if_match_returns_400() { + let h = build_harness(); + seed_secret(&h, "ocp", "old").await; + + let req = json_request_if_match( + "PUT", + "/credstore/v1/secrets/ocp", + Some(serde_json::json!({ "value": "new", "sharing": "tenant" })), + "not-a-valid-etag", + test_ctx(), + ); + let resp = h.router.oneshot(req).await.expect("router"); + assert_eq!(resp.status(), StatusCode::BAD_REQUEST); +} + +#[tokio::test] +async fn put_with_multivalued_if_match_matches_any_validator() { + // RFC 7232 §3.1: a multi-valued If-Match matches if ANY listed validator + // matches. A stale one alongside the current one must still commit. + let h = build_harness(); + let row_id = seed_secret(&h, "ocp", "old").await; // version 1 + + let req = json_request_if_match( + "PUT", + "/credstore/v1/secrets/ocp", + Some(serde_json::json!({ "value": "new", "sharing": "tenant" })), + &format!("\"{}.1\", \"{row_id}.1\"", uuid::Uuid::new_v4()), + test_ctx(), + ); + let resp = h.router.oneshot(req).await.expect("router"); + assert_eq!(resp.status(), StatusCode::NO_CONTENT); +} + +#[tokio::test] +async fn put_with_multivalued_if_match_none_matching_returns_409() { + // A list where no validator matches the current row is a failed + // precondition, not a spurious 400. + let h = build_harness(); + let row_id = seed_secret(&h, "ocp", "old").await; // version 1 + + let req = json_request_if_match( + "PUT", + "/credstore/v1/secrets/ocp", + Some(serde_json::json!({ "value": "new", "sharing": "tenant" })), + &format!("\"{row_id}.99\", \"{}.1\"", uuid::Uuid::new_v4()), + test_ctx(), + ); + let resp = h.router.oneshot(req).await.expect("router"); + assert_eq!(resp.status(), StatusCode::CONFLICT); +} + +#[tokio::test] +async fn delete_with_stale_if_match_returns_409() { + let h = build_harness(); + let row_id = seed_secret(&h, "ocd", "bye").await; // version 1 + + let req = json_request_if_match( + "DELETE", + "/credstore/v1/secrets/ocd", + None, + &format!("\"{row_id}.999\""), + test_ctx(), + ); + let resp = h.router.oneshot(req).await.expect("router"); + assert_eq!(resp.status(), StatusCode::CONFLICT); +} + +#[tokio::test] +async fn put_existing_returns_204() { + let h = build_harness(); + seed_secret(&h, "putkey", "old-value").await; + + let req = json_request( + "PUT", + "/credstore/v1/secrets/putkey", + Some(serde_json::json!({ + "value": "new-value", + "sharing": "tenant" + })), + test_ctx(), + ); + let resp = h.router.oneshot(req).await.expect("router"); + assert_eq!(resp.status(), StatusCode::NO_CONTENT); +} + +#[tokio::test] +async fn delete_existing_returns_204() { + let h = build_harness(); + seed_secret(&h, "delkey", "bye").await; + + let req = json_request("DELETE", "/credstore/v1/secrets/delkey", None, test_ctx()); + let resp = h.router.oneshot(req).await.expect("router"); + assert_eq!(resp.status(), StatusCode::NO_CONTENT); +} + +#[tokio::test] +async fn delete_missing_returns_404() { + let h = build_harness(); + let req = json_request("DELETE", "/credstore/v1/secrets/ghost", None, test_ctx()); + let resp = h.router.oneshot(req).await.expect("router"); + assert_eq!(resp.status(), StatusCode::NOT_FOUND); +} + +#[tokio::test] +async fn invalid_ref_returns_400() { + let h = build_harness(); + // "has:colon" contains a colon which is invalid per SecretRef::new + let req = json_request("GET", "/credstore/v1/secrets/has%3Acolon", None, test_ctx()); + let resp = h.router.oneshot(req).await.expect("router"); + assert_eq!(resp.status(), StatusCode::BAD_REQUEST); +} + +#[tokio::test] +async fn invalid_ref_on_delete_returns_400() { + let h = build_harness(); + let req = json_request( + "DELETE", + "/credstore/v1/secrets/has%3Acolon", + None, + test_ctx(), + ); + let resp = h.router.oneshot(req).await.expect("router"); + assert_eq!(resp.status(), StatusCode::BAD_REQUEST); +} diff --git a/gears/credstore/credstore/src/client.rs b/gears/credstore/credstore/src/client.rs new file mode 100644 index 000000000..cf928766c --- /dev/null +++ b/gears/credstore/credstore/src/client.rs @@ -0,0 +1,145 @@ +use std::sync::Arc; + +use async_trait::async_trait; +use credstore_sdk::{ + CredStoreClientV1, CredStoreError, GetSecretResponse, SecretRef, SecretValue, SharingMode, + WriteOptions, +}; +use toolkit_security::SecurityContext; + +use crate::domain::error::DomainError; +use crate::domain::secret::model::{WritePrecondition, WriteSpec}; +use crate::domain::secret::service::Service; + +/// Map the SDK's optimistic-concurrency precondition onto the domain one. The +/// typed `ClientHub` precondition only expresses the single-generation cases +/// (`Exists` / `Matches`); the multi-validator `AnyVersion` is REST-only. +fn to_domain_precondition(p: credstore_sdk::WritePrecondition) -> WritePrecondition { + match p { + credstore_sdk::WritePrecondition::Exists => WritePrecondition::Exists, + credstore_sdk::WritePrecondition::Matches { id, version } => { + WritePrecondition::Version { id, version } + } + } +} + +impl From for CredStoreError { + fn from(err: DomainError) -> Self { + match err { + DomainError::NotFound => CredStoreError::NotFound, + // Both are 409-class; the SDK has no distinct optimistic-lock variant. + DomainError::Conflict | DomainError::VersionConflict => CredStoreError::Conflict, + DomainError::InvalidSecretRef { detail } => CredStoreError::invalid_ref(detail), + DomainError::UnsupportedTransition { detail } => { + CredStoreError::unsupported_transition(detail) + } + DomainError::TypeViolation { reason, detail, .. } => CredStoreError::TypeViolation { + reason: reason.to_owned(), + detail, + }, + DomainError::AccessDenied { .. } => CredStoreError::AccessDenied, + DomainError::ServiceUnavailable { + detail, + retry_after, + .. + } => CredStoreError::ServiceUnavailable { + detail, + retry_after, + }, + DomainError::Internal { diagnostic, .. } => CredStoreError::internal(diagnostic), + // The ClientHub API sends only typed, always-valid preconditions + // (`Exists`/`Matches`); a malformed one can originate only from + // REST `If-Match` parsing, so `InvalidPrecondition` crossing the + // in-process boundary is a gear-internal invariant breach. + DomainError::InvalidPrecondition { detail } => { + CredStoreError::internal(format!("invalid precondition: {detail}")) + } + #[allow(unreachable_patterns)] + other => CredStoreError::internal(format!("unmapped DomainError variant: {other}")), + } + } +} + +/// In-process [`CredStoreClientV1`] that delegates to the domain [`Service`]. +pub struct CredStoreLocalClient { + svc: Arc, +} + +impl CredStoreLocalClient { + #[must_use] + pub fn new(svc: Arc) -> Self { + Self { svc } + } +} + +#[async_trait] +impl CredStoreClientV1 for CredStoreLocalClient { + async fn get( + &self, + ctx: &SecurityContext, + key: &SecretRef, + ) -> Result, CredStoreError> { + self.svc.get(ctx, key).await.map_err(Into::into) + } + + async fn put_opts( + &self, + ctx: &SecurityContext, + key: &SecretRef, + value: SecretValue, + sharing: SharingMode, + opts: WriteOptions, + ) -> Result<(), CredStoreError> { + let precondition = opts.precondition.map(to_domain_precondition); + self.svc + .put( + ctx, + key, + value, + WriteSpec::upsert(sharing) + .with_opts(opts) + .with_precondition(precondition), + ) + .await + .map_err(Into::into) + } + + async fn create_opts( + &self, + ctx: &SecurityContext, + key: &SecretRef, + value: SecretValue, + sharing: SharingMode, + opts: WriteOptions, + ) -> Result<(), CredStoreError> { + // create-only: Conflict if a secret of this sharing class exists. + let precondition = opts.precondition.map(to_domain_precondition); + self.svc + .put( + ctx, + key, + value, + WriteSpec::create(sharing) + .with_opts(opts) + .with_precondition(precondition), + ) + .await + .map_err(Into::into) + } + + async fn delete_opts( + &self, + ctx: &SecurityContext, + key: &SecretRef, + precondition: Option, + ) -> Result<(), CredStoreError> { + self.svc + .delete(ctx, key, precondition.map(to_domain_precondition)) + .await + .map_err(Into::into) + } +} + +#[cfg(test)] +#[path = "client_tests.rs"] +mod client_tests; diff --git a/gears/credstore/credstore/src/client_tests.rs b/gears/credstore/credstore/src/client_tests.rs new file mode 100644 index 000000000..ad496e217 --- /dev/null +++ b/gears/credstore/credstore/src/client_tests.rs @@ -0,0 +1,249 @@ +//! Unit tests for [`CredStoreLocalClient`] and the `DomainError` → +//! `CredStoreError` SDK-error conversion. +#![allow(clippy::expect_used, clippy::unwrap_used)] + +use std::sync::Arc; +use std::time::Duration; + +use credstore_sdk::{ + CredStoreClientV1, CredStoreError, SecretRef, SecretValue, SharingMode, WriteOptions, + WritePrecondition, +}; +use uuid::Uuid; + +use super::{CredStoreLocalClient, DomainError, Service}; +use crate::domain::ports::metrics::NoopMetrics; +use crate::domain::secret::service::ReaperSettings; +use crate::domain::secret::test_support::{ + FakeDir, FakePlugin, FakePluginSelector, FakeSecretRepo, catalog_type_resolver, make_ctx, + mock_enforcer, +}; + +#[test] +fn domain_error_maps_to_sdk_error() { + assert!(matches!( + CredStoreError::from(DomainError::NotFound), + CredStoreError::NotFound + )); + assert!(matches!( + CredStoreError::from(DomainError::Conflict), + CredStoreError::Conflict + )); + assert!(matches!( + CredStoreError::from(DomainError::InvalidSecretRef { + detail: "x".to_owned() + }), + CredStoreError::InvalidSecretRef { .. } + )); + assert!(matches!( + CredStoreError::from(DomainError::UnsupportedTransition { + detail: "x".to_owned() + }), + CredStoreError::UnsupportedTransition { .. } + )); + assert!(matches!( + CredStoreError::from(DomainError::AccessDenied { cause: None }), + CredStoreError::AccessDenied + )); + assert!(matches!( + CredStoreError::from(DomainError::ServiceUnavailable { + detail: "x".to_owned(), + retry_after: Some(Duration::from_secs(1)), + cause: None, + }), + CredStoreError::ServiceUnavailable { .. } + )); + assert!(matches!( + CredStoreError::from(DomainError::internal("x")), + CredStoreError::Internal(_) + )); +} + +#[tokio::test] +async fn local_client_round_trips_through_service() { + let tenant = Uuid::new_v4(); + let ctx = make_ctx(Uuid::new_v4(), tenant); + let k = SecretRef::new("client-key").expect("ref"); + + let repo = Arc::new(FakeSecretRepo::new()); + let dir = Arc::new(FakeDir::single(tenant)); + let selector = Arc::new(FakePluginSelector::new(FakePlugin::new())); + let svc = Arc::new(Service::new( + repo, + dir, + mock_enforcer(), + selector, + catalog_type_resolver(), + Arc::new(NoopMetrics), + ReaperSettings { + tick_secs: 60, + provisioning_timeout_secs: 300, + deprovisioning_timeout_secs: 300, + }, + )); + let client = CredStoreLocalClient::new(svc); + + client + .put( + &ctx, + &k, + SecretValue::new(b"v".to_vec()), + SharingMode::Tenant, + ) + .await + .expect("put"); + assert!(client.get(&ctx, &k).await.expect("get").is_some()); + client.delete(&ctx, &k).await.expect("delete"); +} + +#[tokio::test] +async fn create_is_create_only_put_is_upsert() { + let tenant = Uuid::new_v4(); + let ctx = make_ctx(Uuid::new_v4(), tenant); + let k = SecretRef::new("create-key").expect("ref"); + + let repo = Arc::new(FakeSecretRepo::new()); + let dir = Arc::new(FakeDir::single(tenant)); + let selector = Arc::new(FakePluginSelector::new(FakePlugin::new())); + let svc = Arc::new(Service::new( + repo, + dir, + mock_enforcer(), + selector, + catalog_type_resolver(), + Arc::new(NoopMetrics), + ReaperSettings { + tick_secs: 60, + provisioning_timeout_secs: 300, + deprovisioning_timeout_secs: 300, + }, + )); + let client = CredStoreLocalClient::new(svc); + + // First create succeeds. + client + .create( + &ctx, + &k, + SecretValue::new(b"v1".to_vec()), + SharingMode::Tenant, + ) + .await + .expect("first create"); + // Second create of the same sharing class → Conflict (create-only). + let err = client + .create( + &ctx, + &k, + SecretValue::new(b"v2".to_vec()), + SharingMode::Tenant, + ) + .await + .expect_err("second create conflicts"); + assert!(matches!(err, CredStoreError::Conflict)); + // `put` still upserts the existing secret (no conflict). + client + .put( + &ctx, + &k, + SecretValue::new(b"v3".to_vec()), + SharingMode::Tenant, + ) + .await + .expect("put upserts"); +} + +#[tokio::test] +async fn precondition_guards_in_process_write_and_delete() { + // The in-process client can now carry an optimistic-concurrency + // precondition (the ClientHub equivalent of a REST `If-Match`), not just + // the REST surface. + let tenant = Uuid::new_v4(); + let ctx = make_ctx(Uuid::new_v4(), tenant); + let k = SecretRef::new("guarded-key").expect("ref"); + + let repo = Arc::new(FakeSecretRepo::new()); + let dir = Arc::new(FakeDir::single(tenant)); + let selector = Arc::new(FakePluginSelector::new(FakePlugin::new())); + let svc = Arc::new(Service::new( + repo, + dir, + mock_enforcer(), + selector, + catalog_type_resolver(), + Arc::new(NoopMetrics), + ReaperSettings { + tick_secs: 60, + provisioning_timeout_secs: 300, + deprovisioning_timeout_secs: 300, + }, + )); + let client = CredStoreLocalClient::new(svc); + + client + .put( + &ctx, + &k, + SecretValue::new(b"v1".to_vec()), + SharingMode::Tenant, + ) + .await + .expect("put"); + let observed = client.get(&ctx, &k).await.expect("get").expect("present"); + let stale = WritePrecondition::Matches { + id: observed.id, + version: observed.version, + }; + + // A guarded update against the observed generation succeeds and bumps the + // version, so the same validator is now stale. + client + .put_opts( + &ctx, + &k, + SecretValue::new(b"v2".to_vec()), + SharingMode::Tenant, + WriteOptions { + precondition: Some(stale), + ..Default::default() + }, + ) + .await + .expect("guarded put matches current generation"); + + // Re-using the stale validator is rejected as a conflict. + let err = client + .put_opts( + &ctx, + &k, + SecretValue::new(b"v3".to_vec()), + SharingMode::Tenant, + WriteOptions { + precondition: Some(stale), + ..Default::default() + }, + ) + .await + .expect_err("stale precondition conflicts"); + assert!(matches!(err, CredStoreError::Conflict), "got: {err:?}"); + + // A guarded delete: the stale validator conflicts, the current one succeeds. + let current = client.get(&ctx, &k).await.expect("get").expect("present"); + let err = client + .delete_opts(&ctx, &k, Some(stale)) + .await + .expect_err("stale delete conflicts"); + assert!(matches!(err, CredStoreError::Conflict), "got: {err:?}"); + client + .delete_opts( + &ctx, + &k, + Some(WritePrecondition::Matches { + id: current.id, + version: current.version, + }), + ) + .await + .expect("guarded delete matches current generation"); + assert!(client.get(&ctx, &k).await.expect("get").is_none()); +} diff --git a/gears/credstore/credstore/src/config.rs b/gears/credstore/credstore/src/config.rs index 02a6648e3..c4689b2d5 100644 --- a/gears/credstore/credstore/src/config.rs +++ b/gears/credstore/credstore/src/config.rs @@ -1,28 +1,181 @@ -// Updated: 2026-04-07 by Constructor Tech -//! Configuration for the credstore gear. - use serde::Deserialize; -/// Gear configuration. #[derive(Debug, Clone, Deserialize)] #[serde(default, deny_unknown_fields)] pub struct CredStoreConfig { - /// Vendor selector used to pick a plugin implementation. - /// - /// The gear queries types-registry for plugin instances matching - /// this vendor and selects the one with lowest priority number. pub vendor: String, + pub hierarchy: HierarchyCfg, + pub reaper: ReaperCfg, } impl Default for CredStoreConfig { fn default() -> Self { Self { vendor: "constructorfabric".to_owned(), + hierarchy: HierarchyCfg::default(), + reaper: ReaperCfg::default(), + } + } +} + +#[derive(Debug, Clone, Deserialize)] +#[serde(default, deny_unknown_fields)] +pub struct HierarchyCfg { + pub ancestor_cache_ttl_secs: u64, + /// Whether the shared tenant-closure table is co-located in this module's + /// database. When true, advertise the tenant-hierarchy PDP capability so the + /// PDP emits a structured subtree predicate resolved by a closure subquery. + /// When false, the PDP pre-expands the subtree to a flat membership list, + /// enforced without any local closure access. Defaults to false — opt in + /// explicitly only where the closure is known to be co-located. + pub tenant_closure_colocated: bool, +} + +impl Default for HierarchyCfg { + fn default() -> Self { + Self { + ancestor_cache_ttl_secs: 300, + tenant_closure_colocated: false, // conservative: degraded unless opted in } } } +#[derive(Debug, Clone, Deserialize)] +#[serde(default, deny_unknown_fields)] +#[allow( + clippy::struct_field_names, + reason = "serialized config keys; renaming would break existing configs" +)] +pub struct ReaperCfg { + pub tick_secs: u64, + pub provisioning_timeout_secs: u64, + /// Age after which a stuck `deprovisioning` row is completed by the + /// reaper (backend value deleted, row removed). + pub deprovisioning_timeout_secs: u64, +} + +impl Default for ReaperCfg { + fn default() -> Self { + Self { + tick_secs: 60, + provisioning_timeout_secs: 300, + deprovisioning_timeout_secs: 300, + } + } +} + +impl CredStoreConfig { + /// # Errors + /// Returns `Err` with a description if any field is invalid. + pub fn validate(&self) -> Result<(), String> { + if self.vendor.trim().is_empty() { + return Err("vendor must be non-empty".to_owned()); + } + if self.reaper.tick_secs == 0 { + return Err("reaper.tick_secs must be > 0".to_owned()); + } + if self.reaper.provisioning_timeout_secs == 0 { + return Err("reaper.provisioning_timeout_secs must be > 0".to_owned()); + } + if self.reaper.deprovisioning_timeout_secs == 0 { + return Err("reaper.deprovisioning_timeout_secs must be > 0".to_owned()); + } + if self.hierarchy.ancestor_cache_ttl_secs == 0 { + return Err("hierarchy.ancestor_cache_ttl_secs must be > 0".to_owned()); + } + Ok(()) + } +} + #[cfg(test)] -#[cfg_attr(coverage_nightly, coverage(off))] -#[path = "config_tests.rs"] -mod config_tests; +mod tests { + use super::CredStoreConfig; + + #[test] + fn default_config_is_valid() { + let cfg = CredStoreConfig::default(); + // Must match the backend plugin's default vendor (static-credstore-plugin + // defaults to "constructorfabric"); otherwise a default-config deployment + // resolves no backend plugin and 503s on every secret op. + assert_eq!(cfg.vendor, "constructorfabric"); + assert_eq!(cfg.hierarchy.ancestor_cache_ttl_secs, 300); + // Co-location defaults off: degraded (flat-In) mode unless opted in. + assert!(!cfg.hierarchy.tenant_closure_colocated); + assert_eq!(cfg.reaper.tick_secs, 60); + assert_eq!(cfg.reaper.provisioning_timeout_secs, 300); + assert_eq!(cfg.reaper.deprovisioning_timeout_secs, 300); + assert!(cfg.validate().is_ok()); + } + + #[test] + fn deserializes_partial_config_with_defaults() { + let cfg: CredStoreConfig = + serde_json::from_str(r#"{"vendor":"acme","reaper":{"tick_secs":5}}"#) + .expect("deserialize"); + assert_eq!(cfg.vendor, "acme"); + assert_eq!(cfg.reaper.tick_secs, 5); + // Unspecified fields fall back to defaults. + assert_eq!(cfg.reaper.provisioning_timeout_secs, 300); + assert_eq!(cfg.reaper.deprovisioning_timeout_secs, 300); + assert_eq!(cfg.hierarchy.ancestor_cache_ttl_secs, 300); + assert!(!cfg.hierarchy.tenant_closure_colocated); + } + + #[test] + fn deserializes_explicit_tenant_closure_colocated_true() { + let cfg: CredStoreConfig = + serde_json::from_str(r#"{"hierarchy":{"tenant_closure_colocated":true}}"#) + .expect("deserialize"); + // Explicit opt-in overrides the conservative default. + assert!(cfg.hierarchy.tenant_closure_colocated); + // Co-location is independent of validation (a bool is always valid). + assert!(cfg.validate().is_ok()); + } + + #[test] + fn validate_rejects_each_invalid_field() { + use super::{HierarchyCfg, ReaperCfg}; + + let empty_vendor = CredStoreConfig { + vendor: String::new(), + ..Default::default() + }; + assert!(empty_vendor.validate().is_err()); + + let zero_tick = CredStoreConfig { + reaper: ReaperCfg { + tick_secs: 0, + ..Default::default() + }, + ..Default::default() + }; + assert!(zero_tick.validate().is_err()); + + let zero_timeout = CredStoreConfig { + reaper: ReaperCfg { + provisioning_timeout_secs: 0, + ..Default::default() + }, + ..Default::default() + }; + assert!(zero_timeout.validate().is_err()); + + let zero_deprov_timeout = CredStoreConfig { + reaper: ReaperCfg { + deprovisioning_timeout_secs: 0, + ..Default::default() + }, + ..Default::default() + }; + assert!(zero_deprov_timeout.validate().is_err()); + + let zero_ttl = CredStoreConfig { + hierarchy: HierarchyCfg { + ancestor_cache_ttl_secs: 0, + ..Default::default() + }, + ..Default::default() + }; + assert!(zero_ttl.validate().is_err()); + } +} diff --git a/gears/credstore/credstore/src/config_tests.rs b/gears/credstore/credstore/src/config_tests.rs deleted file mode 100644 index ef915f967..000000000 --- a/gears/credstore/credstore/src/config_tests.rs +++ /dev/null @@ -1,24 +0,0 @@ -// Created: 2026-04-07 by Constructor Tech -use super::*; - -#[test] -fn vendor_can_be_overridden_via_serde() { - let json = r#"{"vendor": "acme"}"#; - let cfg: CredStoreConfig = serde_json::from_str(json).unwrap(); - assert_eq!(cfg.vendor, "acme"); -} - -#[test] -fn serde_default_applies_default_vendor() { - let cfg: CredStoreConfig = serde_json::from_str("{}").unwrap(); - assert_eq!( - cfg.vendor, "constructorfabric", - "serde(default) must use Default impl" - ); -} - -#[test] -fn rejects_unknown_fields() { - let json = r#"{"vendor": "x", "unexpected": true}"#; - assert!(serde_json::from_str::(json).is_err()); -} diff --git a/gears/credstore/credstore/src/domain.rs b/gears/credstore/credstore/src/domain.rs new file mode 100644 index 000000000..2d8a7cdfe --- /dev/null +++ b/gears/credstore/credstore/src/domain.rs @@ -0,0 +1,5 @@ +pub mod authz; +pub mod error; +pub mod ports; +pub mod resolver; +pub mod secret; diff --git a/gears/credstore/credstore/src/domain/authz.rs b/gears/credstore/credstore/src/domain/authz.rs new file mode 100644 index 000000000..80d34ef89 --- /dev/null +++ b/gears/credstore/credstore/src/domain/authz.rs @@ -0,0 +1,66 @@ +use authz_resolver_sdk::pep::{AccessRequest, EnforcerError, PolicyEnforcer, ResourceType}; +use toolkit_security::{AccessScope, SecurityContext, pep_properties}; + +use crate::domain::error::DomainError; + +/// PDP resource type for a concrete secret type: the full derived GTS id +/// (design §5.4), e.g. +/// `gts.cf.core.credstore.secret.v1~cf.core.credstore.api_key.v1~`. +/// +/// This is the **only** resource type credstore evaluates: every operation +/// authorizes against the secret's full concrete type (including `generic`), +/// so policies can target any type without a separate base-type gate. The id +/// comes from the per-operation types-registry resolution +/// ([`crate::domain::secret::type_resolver::ResolvedSecretType::gts_id`]), +/// so dynamically registered types are addressable without a release. +#[must_use] +pub fn secret_type_resource(gts_id: &str) -> ResourceType { + ResourceType::new(gts_id.to_owned(), &[pep_properties::OWNER_TENANT_ID]) +} + +pub mod actions { + pub const READ: &str = "read"; + pub const WRITE: &str = "write"; + pub const DELETE: &str = "delete"; +} + +/// Map a PEP enforcement failure to a domain error (fail-closed). +/// `Denied` / `CompileFailed` → `AccessDenied` (403); `EvaluationFailed` → `ServiceUnavailable` (503). +#[must_use] +pub fn map_enforcer_err(err: EnforcerError) -> DomainError { + match err { + EnforcerError::Denied { .. } | EnforcerError::CompileFailed(_) => { + DomainError::AccessDenied { + cause: Some(Box::new(err)), + } + } + EnforcerError::EvaluationFailed(source) => DomainError::ServiceUnavailable { + detail: "authorization evaluation failed".to_owned(), + retry_after: None, + cause: Some(Box::new(EnforcerError::EvaluationFailed(source))), + }, + } +} + +/// Returns the PDP `AccessScope` for `action` on `resource` for the +/// caller's tenant. +/// +/// # Errors +/// +/// Returns `DomainError::AccessDenied` if the PDP denies access or fails to compile constraints. +/// Returns `DomainError::ServiceUnavailable` if the PDP evaluation call fails. +pub async fn scope_for( + enforcer: &PolicyEnforcer, + ctx: &SecurityContext, + resource: &ResourceType, + action: &str, +) -> Result { + let tenant = ctx.subject_tenant_id(); + let request = AccessRequest::new() + .resource_property(pep_properties::OWNER_TENANT_ID, tenant) + .require_constraints(true); + enforcer + .access_scope_with(ctx, resource, action, None, &request) + .await + .map_err(map_enforcer_err) +} diff --git a/gears/credstore/credstore/src/domain/error.rs b/gears/credstore/credstore/src/domain/error.rs index c4e547551..ade7bc2c0 100644 --- a/gears/credstore/credstore/src/domain/error.rs +++ b/gears/credstore/credstore/src/domain/error.rs @@ -1,111 +1,62 @@ -// Updated: 2026-04-07 by Constructor Tech -//! Domain errors for the credstore gear. +use std::time::Duration; -use credstore_sdk::CredStoreError; +use thiserror::Error; use toolkit_macros::domain_model; -/// Internal domain errors. +type BoxError = Box; + #[domain_model] -#[derive(thiserror::Error, Debug)] +#[derive(Debug, Error)] +#[non_exhaustive] pub enum DomainError { - #[error("types registry is not available: {0}")] - TypesRegistryUnavailable(String), - - #[error("no plugin instances found for vendor '{vendor}'")] - PluginNotFound { vendor: String }, - - #[error("invalid plugin instance content for '{gts_id}': {reason}")] - InvalidPluginInstance { gts_id: String, reason: String }, - - #[error("plugin not available for '{gts_id}': {reason}")] - PluginUnavailable { gts_id: String, reason: String }, - + #[error("invalid secret reference: {detail}")] + InvalidSecretRef { detail: String }, #[error("secret not found")] NotFound, - - #[error("internal error: {0}")] - Internal(String), -} - -// TODO(DE1302): `DomainError::Internal` only carries a String, so these From -// impls drop the source error. Extend the variant to hold a boxed source (or -// introduce typed variants) so `.source()` returns the original error, then -// remove these allows. -#[allow(unknown_lints, de1302_error_from_to_string)] -impl From for DomainError { - fn from(e: toolkit_canonical_errors::CanonicalError) -> Self { - // Prefer the in-process diagnostic for `Internal`/`Unknown` (the wire - // strips it); other categories carry their message in `detail()` via - // `to_string()`. - Self::Internal(e.diagnostic().map_or_else(|| e.to_string(), str::to_owned)) - } -} - -#[allow(unknown_lints, de1302_error_from_to_string)] -impl From for DomainError { - fn from(e: toolkit::client_hub::ClientHubError) -> Self { - Self::Internal(e.to_string()) - } + #[error("secret already exists")] + Conflict, + #[error("version precondition failed")] + VersionConflict, + #[error("invalid precondition: {detail}")] + InvalidPrecondition { detail: String }, + #[error("unsupported sharing transition: {detail}")] + UnsupportedTransition { detail: String }, + /// A write violated the secret type's traits. `reason` is the stable + /// machine-readable code surfaced on the wire (e.g. + /// `SHARING_NOT_ALLOWED_FOR_TYPE`); `field` names the offending request + /// field for the canonical field violation. + #[error("secret type violation ({reason}): {detail}")] + TypeViolation { + field: &'static str, + reason: &'static str, + detail: String, + }, + #[error("access denied")] + AccessDenied { + #[source] + cause: Option, + }, + #[error("service unavailable: {detail}")] + ServiceUnavailable { + detail: String, + retry_after: Option, + #[source] + cause: Option, + }, + #[error("internal error")] + Internal { + diagnostic: String, + #[source] + cause: Option, + }, } -#[allow(unknown_lints, de1302_error_from_to_string)] -impl From for DomainError { - fn from(e: serde_json::Error) -> Self { - Self::Internal(e.to_string()) - } -} - -impl From for DomainError { - fn from(e: toolkit::plugins::ChoosePluginError) -> Self { - match e { - toolkit::plugins::ChoosePluginError::InvalidPluginInstance { gts_id, reason } => { - Self::InvalidPluginInstance { gts_id, reason } - } - toolkit::plugins::ChoosePluginError::PluginNotFound { vendor, .. } => { - Self::PluginNotFound { vendor } - } - } - } -} - -impl From for DomainError { - fn from(e: CredStoreError) -> Self { - match e { - CredStoreError::NotFound => Self::NotFound, - // CredStoreError variants don't carry vendor/gts_id, so these - // fields cannot be populated from the error alone. - CredStoreError::NoPluginAvailable => Self::PluginNotFound { - vendor: "unknown".to_owned(), - }, - CredStoreError::ServiceUnavailable(msg) => Self::PluginUnavailable { - gts_id: "unknown".to_owned(), - reason: msg, - }, - CredStoreError::InvalidSecretRef { reason } => Self::Internal(reason), - CredStoreError::Internal(msg) => Self::Internal(msg), - } - } -} - -impl From for CredStoreError { - fn from(e: DomainError) -> Self { - match e { - DomainError::PluginNotFound { .. } => Self::NoPluginAvailable, - DomainError::InvalidPluginInstance { gts_id, reason } => { - Self::Internal(format!("invalid plugin instance '{gts_id}': {reason}")) - } - DomainError::PluginUnavailable { gts_id, reason } => { - Self::ServiceUnavailable(format!("plugin not available for '{gts_id}': {reason}")) - } - DomainError::NotFound => Self::NotFound, - DomainError::TypesRegistryUnavailable(reason) | DomainError::Internal(reason) => { - Self::Internal(reason) - } +impl DomainError { + #[must_use] + pub fn internal(diagnostic: impl Into) -> Self { + Self::Internal { + diagnostic: diagnostic.into(), + cause: None, } } } - -#[cfg(test)] -#[cfg_attr(coverage_nightly, coverage(off))] -#[path = "error_tests.rs"] -mod error_tests; diff --git a/gears/credstore/credstore/src/domain/error_tests.rs b/gears/credstore/credstore/src/domain/error_tests.rs deleted file mode 100644 index 5543bf7aa..000000000 --- a/gears/credstore/credstore/src/domain/error_tests.rs +++ /dev/null @@ -1,162 +0,0 @@ -// Created: 2026-04-07 by Constructor Tech -use toolkit::plugins::ChoosePluginError; -use toolkit_gts::gts_id; - -use super::*; - -// ── From ───────────────────────────────────────────────── - -#[test] -fn from_canonical_error_becomes_internal() { - // The types-registry trait boundary is now `CanonicalError` (ADR 0005); - // credstore folds any such error into its opaque `Internal`. - let src = types_registry_sdk::testing::internal("oops"); - let dst = DomainError::from(src); - assert!(matches!(dst, DomainError::Internal(_))); -} - -// ── From ───────────────────────────────────────────────── - -#[test] -fn from_client_hub_error_becomes_internal() { - // Trigger a real ClientHubError by requesting an unregistered type. - let hub = toolkit::client_hub::ClientHub::default(); - let src = hub - .get::() - .err() - .unwrap(); - let dst = DomainError::from(src); - assert!(matches!(dst, DomainError::Internal(_))); -} - -// ── From ────────────────────────────────────────────── - -#[test] -fn from_serde_json_error_becomes_internal() { - let src: serde_json::Error = serde_json::from_str::("not-json").unwrap_err(); - let dst = DomainError::from(src); - assert!(matches!(dst, DomainError::Internal(_))); -} - -// ── From ────────────────────────────────────────────── - -#[test] -fn from_choose_plugin_error_not_found_becomes_plugin_not_found() { - let src = ChoosePluginError::PluginNotFound { - type_id: gts_id!("cf.core.test.plugin.v1~").into(), - vendor: "acme".into(), - }; - let dst = DomainError::from(src); - assert!(matches!(dst, DomainError::PluginNotFound { vendor } if vendor == "acme")); -} - -#[test] -fn from_choose_plugin_error_invalid_instance_becomes_invalid_plugin_instance() { - let src = ChoosePluginError::InvalidPluginInstance { - gts_id: gts_id!("cf.core.test.error.v1~").into(), - reason: "bad content".into(), - }; - let dst = DomainError::from(src); - assert!( - matches!(dst, DomainError::InvalidPluginInstance { gts_id, reason } - if gts_id == gts_id!("cf.core.test.error.v1~") && reason == "bad content") - ); -} - -// ── From for DomainError ───────────────────────────────── - -#[test] -fn from_credstore_error_not_found_becomes_not_found() { - let dst = DomainError::from(CredStoreError::NotFound); - assert!(matches!(dst, DomainError::NotFound)); -} - -#[test] -fn from_credstore_error_no_plugin_available_becomes_plugin_not_found() { - let dst = DomainError::from(CredStoreError::NoPluginAvailable); - assert!(matches!(dst, DomainError::PluginNotFound { vendor } if vendor == "unknown")); -} - -#[test] -fn from_credstore_error_service_unavailable_becomes_plugin_unavailable() { - let dst = DomainError::from(CredStoreError::ServiceUnavailable("down".into())); - assert!( - matches!(dst, DomainError::PluginUnavailable { gts_id, reason } - if gts_id == "unknown" && reason == "down") - ); -} - -#[test] -fn from_credstore_error_invalid_secret_ref_becomes_internal() { - let dst = DomainError::from(CredStoreError::InvalidSecretRef { - reason: "bad".into(), - }); - assert!(matches!(dst, DomainError::Internal(msg) if msg == "bad")); -} - -#[test] -fn from_credstore_error_internal_becomes_internal() { - let dst = DomainError::from(CredStoreError::Internal("boom".into())); - assert!(matches!(dst, DomainError::Internal(msg) if msg == "boom")); -} - -// ── From for CredStoreError ──────────────────────────────── - -#[test] -fn domain_plugin_not_found_becomes_no_plugin_available() { - let src = DomainError::PluginNotFound { - vendor: "acme".into(), - }; - let dst = CredStoreError::from(src); - assert!(matches!(dst, CredStoreError::NoPluginAvailable)); -} - -#[test] -fn domain_invalid_plugin_instance_becomes_internal() { - const TEST_ERROR_ID: &str = gts_id!("cf.core.test.error.v1~"); - let src = DomainError::InvalidPluginInstance { - gts_id: TEST_ERROR_ID.into(), - reason: "bad".into(), - }; - let dst = CredStoreError::from(src); - assert!( - matches!(dst, CredStoreError::Internal(ref msg) - if msg.contains(TEST_ERROR_ID) && msg.contains("bad")), - "expected Internal with gts_id and reason, got: {dst:?}" - ); -} - -#[test] -fn domain_plugin_unavailable_becomes_service_unavailable() { - const TEST_ERROR_ID: &str = gts_id!("cf.core.test.error.v1~"); - let src = DomainError::PluginUnavailable { - gts_id: TEST_ERROR_ID.into(), - reason: "not ready".into(), - }; - let dst = CredStoreError::from(src); - assert!( - matches!(dst, CredStoreError::ServiceUnavailable(ref msg) - if msg.contains(TEST_ERROR_ID) && msg.contains("not ready")), - "expected ServiceUnavailable with gts_id and reason, got: {dst:?}" - ); -} - -#[test] -fn domain_not_found_becomes_not_found() { - let dst = CredStoreError::from(DomainError::NotFound); - assert!(matches!(dst, CredStoreError::NotFound)); -} - -#[test] -fn domain_types_registry_unavailable_becomes_internal() { - let src = DomainError::TypesRegistryUnavailable("gone".into()); - let dst = CredStoreError::from(src); - assert!(matches!(dst, CredStoreError::Internal(msg) if msg == "gone")); -} - -#[test] -fn domain_internal_becomes_internal() { - let src = DomainError::Internal("err".into()); - let dst = CredStoreError::from(src); - assert!(matches!(dst, CredStoreError::Internal(msg) if msg == "err")); -} diff --git a/gears/credstore/credstore/src/domain/local_client.rs b/gears/credstore/credstore/src/domain/local_client.rs deleted file mode 100644 index 992bd6843..000000000 --- a/gears/credstore/credstore/src/domain/local_client.rs +++ /dev/null @@ -1,58 +0,0 @@ -// Updated: 2026-04-07 by Constructor Tech -//! Local (in-process) client for the credstore gear. - -use std::sync::Arc; - -use async_trait::async_trait; -use credstore_sdk::{CredStoreClientV1, CredStoreError, GetSecretResponse, SecretRef}; -use toolkit_macros::domain_model; -use toolkit_security::SecurityContext; - -use super::{DomainError, Service}; - -/// Local client wrapping the credstore service. -/// -/// Registered in `ClientHub` by the credstore gear during `init()`. -#[domain_model] -pub struct CredStoreLocalClient { - svc: Arc, -} - -impl CredStoreLocalClient { - /// Creates a new local client wrapping the given service. - #[must_use] - pub fn new(svc: Arc) -> Self { - Self { svc } - } -} - -fn log_and_convert(op: &str, e: DomainError) -> CredStoreError { - match &e { - DomainError::NotFound => { - tracing::debug!(operation = op, "credstore secret not found"); - } - _ => { - tracing::error!(operation = op, error = ?e, "credstore call failed"); - } - } - e.into() -} - -#[async_trait] -impl CredStoreClientV1 for CredStoreLocalClient { - async fn get( - &self, - ctx: &SecurityContext, - key: &SecretRef, - ) -> Result, CredStoreError> { - self.svc - .get(ctx, key) - .await - .map_err(|e| log_and_convert("get", e)) - } -} - -#[cfg(test)] -#[cfg_attr(coverage_nightly, coverage(off))] -#[path = "local_client_tests.rs"] -mod local_client_tests; diff --git a/gears/credstore/credstore/src/domain/local_client_tests.rs b/gears/credstore/credstore/src/domain/local_client_tests.rs deleted file mode 100644 index bbfb2e5a8..000000000 --- a/gears/credstore/credstore/src/domain/local_client_tests.rs +++ /dev/null @@ -1,93 +0,0 @@ -// Created: 2026-04-07 by Constructor Tech -use std::sync::Arc; - -use credstore_sdk::{ - CredStorePluginClientV1, CredStorePluginSpecV1, OwnerId, SecretMetadata, SecretValue, - SharingMode, TenantId, -}; -use toolkit::client_hub::{ClientHub, ClientScope}; -use types_registry_sdk::TypesRegistryClient; -use types_registry_sdk::testing::{MockTypesRegistryClient, make_test_instance}; - -use super::*; -use crate::domain::Service; -use crate::domain::test_support::{MockPlugin, test_ctx}; - -fn make_client() -> CredStoreLocalClient { - let hub = Arc::new(ClientHub::default()); - let svc = Arc::new(Service::new(hub, "constructorfabric".into())); - CredStoreLocalClient::new(svc) -} - -fn make_wired_client(plugin: Arc) -> CredStoreLocalClient { - let instance_id = format!( - "{}test.credstore.mock.local_client.v1", - CredStorePluginSpecV1::gts_type_id() - ); - let hub = Arc::new(ClientHub::default()); - - let instance = make_test_instance( - &instance_id, - serde_json::json!({ - "id": instance_id, - "vendor": "constructorfabric", - "priority": 0, - "properties": {} - }), - ); - let reg: Arc = - Arc::new(MockTypesRegistryClient::new().with_instances([instance])); - hub.register::(reg); - hub.register_scoped::(ClientScope::gts_id(&instance_id), plugin); - - let svc = Arc::new(Service::new(hub, "constructorfabric".into())); - CredStoreLocalClient::new(svc) -} - -// ── CredStoreClientV1::get — error path ────────────────────────────────── - -#[tokio::test] -async fn get_trait_impl_propagates_service_error() { - let client = make_client(); - let key = SecretRef::new("test-key").unwrap(); - // Hub is empty → TypesRegistryUnavailable → CredStoreError::Internal - let result = client.get(&test_ctx(), &key).await; - assert!(matches!(result.unwrap_err(), CredStoreError::Internal(_))); -} - -#[tokio::test] -async fn get_trait_impl_converts_not_found_from_plugin() { - let client = make_wired_client(MockPlugin::errors_not_found()); - let key = SecretRef::new("missing-key").unwrap(); - let result = client.get(&test_ctx(), &key).await; - assert!( - matches!(result.unwrap_err(), CredStoreError::NotFound), - "DomainError::NotFound must map to CredStoreError::NotFound" - ); -} - -// ── CredStoreClientV1::get — happy paths ───────────────────────────────── - -#[tokio::test] -async fn get_trait_impl_returns_some_on_success() { - let meta = SecretMetadata { - value: SecretValue::from("val"), - owner_id: OwnerId::nil(), - sharing: SharingMode::Tenant, - owner_tenant_id: TenantId::nil(), - }; - let client = make_wired_client(MockPlugin::returns(Some(&meta))); - let key = SecretRef::new("key").unwrap(); - let resp = client.get(&test_ctx(), &key).await.unwrap(); - let resp = resp.expect("expected Some"); - assert_eq!(resp.value.as_bytes(), b"val"); - assert!(!resp.is_inherited); -} - -#[tokio::test] -async fn get_trait_impl_returns_none_when_plugin_returns_none() { - let client = make_wired_client(MockPlugin::returns(None)); - let key = SecretRef::new("missing").unwrap(); - let resp = client.get(&test_ctx(), &key).await.unwrap(); - assert!(resp.is_none()); -} diff --git a/gears/credstore/credstore/src/domain/mod.rs b/gears/credstore/credstore/src/domain/mod.rs deleted file mode 100644 index 2e5fc59c0..000000000 --- a/gears/credstore/credstore/src/domain/mod.rs +++ /dev/null @@ -1,11 +0,0 @@ -//! Domain layer for the credstore gear. - -pub mod error; -pub mod local_client; -pub mod service; -#[cfg(test)] -pub mod test_support; - -pub use error::DomainError; -pub use local_client::CredStoreLocalClient; -pub use service::Service; diff --git a/gears/credstore/credstore/src/domain/ports.rs b/gears/credstore/credstore/src/domain/ports.rs new file mode 100644 index 000000000..ee870366d --- /dev/null +++ b/gears/credstore/credstore/src/domain/ports.rs @@ -0,0 +1,2 @@ +pub mod metrics; +pub mod plugin; diff --git a/gears/credstore/credstore/src/domain/ports/metrics.rs b/gears/credstore/credstore/src/domain/ports/metrics.rs new file mode 100644 index 000000000..e29465f68 --- /dev/null +++ b/gears/credstore/credstore/src/domain/ports/metrics.rs @@ -0,0 +1,195 @@ +use toolkit_macros::domain_model; + +#[domain_model] +#[derive(Debug, Default, Clone, Copy)] +pub struct SecretCounts { + pub private: i64, + pub tenant: i64, + pub shared: i64, + pub provisioning: i64, + pub deprovisioning: i64, + pub tenants: i64, +} + +#[domain_model] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ReadOutcome { + HitOwn, + HitInherited, + Miss, +} +impl ReadOutcome { + #[must_use] + pub const fn as_str(self) -> &'static str { + match self { + Self::HitOwn => "hit_own", + Self::HitInherited => "hit_inherited", + Self::Miss => "miss", + } + } +} + +#[domain_model] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Dep { + TenantResolver, + Plugin, + Pdp, + TypesRegistry, +} +impl Dep { + #[must_use] + pub const fn as_str(self) -> &'static str { + match self { + Self::TenantResolver => "tenant_resolver", + Self::Plugin => "plugin", + Self::Pdp => "pdp", + Self::TypesRegistry => "types_registry", + } + } +} + +#[domain_model] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum DepOp { + GetAncestors, + PluginGet, + PluginPut, + PluginDelete, + Evaluate, + GetTypeSchemaByUuid, +} +impl DepOp { + #[must_use] + pub const fn as_str(self) -> &'static str { + match self { + Self::GetAncestors => "get_ancestors", + Self::PluginGet => "plugin_get", + Self::PluginPut => "plugin_put", + Self::PluginDelete => "plugin_delete", + Self::Evaluate => "evaluate", + Self::GetTypeSchemaByUuid => "get_type_schema_by_uuid", + } + } +} + +/// Value-fingerprint fence verdict for a read (see +/// `docs/features/001-value-fingerprint-fence.md`). `Legacy` is an +/// out-of-band seeded row served on trust (no fingerprint yet); `Mismatch` +/// is the fail-closed anti-enumeration miss — the alertable signal. +#[domain_model] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum FenceVerify { + Ok, + Legacy, + Mismatch, +} +impl FenceVerify { + #[must_use] + pub const fn as_str(self) -> &'static str { + match self { + Self::Ok => "ok", + Self::Legacy => "legacy", + Self::Mismatch => "mismatch", + } + } +} + +#[domain_model] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Outcome { + Success, + NotFound, + Error, +} +impl Outcome { + #[must_use] + pub const fn as_str(self) -> &'static str { + match self { + Self::Success => "success", + Self::NotFound => "not_found", + Self::Error => "error", + } + } +} + +pub trait CredStoreMetricsPort: Send + Sync + 'static { + fn record_inventory(&self, counts: SecretCounts); + fn read_outcome(&self, outcome: ReadOutcome); + fn walkup_depth(&self, depth: u64); + fn dependency(&self, dep: Dep, op: DepOp, outcome: Outcome, secs: f64); + fn provisioning_reaped(&self, n: u64); + /// Stuck delete-saga rows completed by the reaper (backend value deleted, + /// row removed). + fn deprovisioning_reaped(&self, n: u64); + /// Records a create-saga provisioning-row rollback after a failed backend + /// write. `outcome` is `Error` when the rollback itself failed (the + /// reference stays wedged until reaped) — the signal worth alerting on. + fn provisioning_rollback(&self, outcome: Outcome); + fn cross_tenant_denied(&self); + /// Records the fence verdict of a read (`ok`/`legacy`/`mismatch`); + /// `mismatch` is the fail-closed 404 worth alerting on. + fn fence_verify(&self, outcome: FenceVerify); + /// Records a lazy fingerprint backfill attempt: `Success` = stamped, + /// `NotFound` = CAS no-op (a concurrent PUT already stamped), `Error` = + /// the best-effort UPDATE failed (retried on a later read/sweep). + fn fence_backfill(&self, outcome: Outcome); +} + +#[domain_model] +#[derive(Debug, Default, Clone, Copy)] +pub struct NoopMetrics; +impl CredStoreMetricsPort for NoopMetrics { + fn record_inventory(&self, _: SecretCounts) {} + fn read_outcome(&self, _: ReadOutcome) {} + fn walkup_depth(&self, _: u64) {} + fn dependency(&self, _: Dep, _: DepOp, _: Outcome, _: f64) {} + fn provisioning_reaped(&self, _: u64) {} + fn deprovisioning_reaped(&self, _: u64) {} + fn provisioning_rollback(&self, _: Outcome) {} + fn cross_tenant_denied(&self) {} + fn fence_verify(&self, _: FenceVerify) {} + fn fence_backfill(&self, _: Outcome) {} +} + +#[cfg(test)] +mod tests { + use super::*; + #[test] + fn labels_snake_case() { + assert_eq!(ReadOutcome::HitInherited.as_str(), "hit_inherited"); + assert_eq!(Dep::TenantResolver.as_str(), "tenant_resolver"); + assert_eq!(DepOp::PluginGet.as_str(), "plugin_get"); + assert_eq!(Outcome::NotFound.as_str(), "not_found"); + } + + #[test] + fn all_label_variants_render() { + assert_eq!(ReadOutcome::HitOwn.as_str(), "hit_own"); + assert_eq!(ReadOutcome::Miss.as_str(), "miss"); + assert_eq!(Dep::Plugin.as_str(), "plugin"); + assert_eq!(Dep::Pdp.as_str(), "pdp"); + assert_eq!(Dep::TypesRegistry.as_str(), "types_registry"); + assert_eq!(DepOp::GetAncestors.as_str(), "get_ancestors"); + assert_eq!(DepOp::PluginPut.as_str(), "plugin_put"); + assert_eq!(DepOp::PluginDelete.as_str(), "plugin_delete"); + assert_eq!(DepOp::Evaluate.as_str(), "evaluate"); + assert_eq!( + DepOp::GetTypeSchemaByUuid.as_str(), + "get_type_schema_by_uuid" + ); + assert_eq!(Outcome::Success.as_str(), "success"); + assert_eq!(Outcome::Error.as_str(), "error"); + } + + #[test] + fn noop_metrics_port_is_inert() { + let noop = NoopMetrics; + noop.record_inventory(SecretCounts::default()); + noop.read_outcome(ReadOutcome::Miss); + noop.walkup_depth(3); + noop.dependency(Dep::Pdp, DepOp::Evaluate, Outcome::Success, 0.1); + noop.provisioning_reaped(2); + noop.cross_tenant_denied(); + } +} diff --git a/gears/credstore/credstore/src/domain/ports/plugin.rs b/gears/credstore/credstore/src/domain/ports/plugin.rs new file mode 100644 index 000000000..d96df1e95 --- /dev/null +++ b/gears/credstore/credstore/src/domain/ports/plugin.rs @@ -0,0 +1,12 @@ +use std::sync::Arc; + +use async_trait::async_trait; +use credstore_sdk::CredStorePluginClientV1; + +use crate::domain::error::DomainError; + +/// Selects the active backend storage plugin (one per deployment). +#[async_trait] +pub trait PluginSelector: Send + Sync { + async fn resolve(&self) -> Result, DomainError>; +} diff --git a/gears/credstore/credstore/src/domain/resolver.rs b/gears/credstore/credstore/src/domain/resolver.rs new file mode 100644 index 000000000..ad30e6f74 --- /dev/null +++ b/gears/credstore/credstore/src/domain/resolver.rs @@ -0,0 +1,17 @@ +use async_trait::async_trait; +use credstore_sdk::TenantId; +use toolkit_security::SecurityContext; +use uuid::Uuid; + +use crate::domain::error::DomainError; + +/// Resolves a tenant's ancestor chain (self first, root last). +#[async_trait] +pub trait TenantDirectory: Send + Sync { + /// Returns `[req, parent, …, root]` (req first), self included. + async fn ancestor_chain( + &self, + ctx: &SecurityContext, + req: TenantId, + ) -> Result, DomainError>; +} diff --git a/gears/credstore/credstore/src/domain/secret.rs b/gears/credstore/credstore/src/domain/secret.rs new file mode 100644 index 000000000..77b3f4bd4 --- /dev/null +++ b/gears/credstore/credstore/src/domain/secret.rs @@ -0,0 +1,8 @@ +pub mod fence; +pub mod model; +pub mod repo; +pub mod service; +#[cfg(test)] +pub mod test_support; +pub mod type_resolver; +pub mod typing; diff --git a/gears/credstore/credstore/src/domain/secret/fence.rs b/gears/credstore/credstore/src/domain/secret/fence.rs new file mode 100644 index 000000000..8163d6377 --- /dev/null +++ b/gears/credstore/credstore/src/domain/secret/fence.rs @@ -0,0 +1,128 @@ +//! Value-fingerprint fence — the crypto and constants binding a secret's +//! backend value to the metadata row that governs its visibility. +//! +//! Every API write stamps `credstore_secrets.value_fp` with +//! `HMAC-SHA256(fence_key, value)`; every read recomputes it from the value +//! the backend returned and serves the value only when the row agrees. The +//! gear performs a dual write (backend value + DB metadata) with no +//! transaction spanning both stores, so two concurrent unconditional PUTs +//! can interleave crosswise; the fingerprint makes the poisoned combination +//! unreadable (fail-closed 404) instead of a cross-tenant disclosure. See +//! `docs/features/001-value-fingerprint-fence.md`. +//! +//! The fence key is deployment state, not configuration: auto-generated and +//! stored in the value-store backend itself under [`FENCE_KEY_REF`] (nil +//! tenant, no metadata row — unreachable through the API by construction). +//! Split knowledge: fingerprints live in the gear DB, the key lives with +//! the values, so a read-only DB compromise alone cannot dictionary-test +//! fingerprints, and backend compromise yields the plaintexts anyway. +//! The fingerprint itself never leaves the gear (no API field, header, +//! or log line). + +use aws_lc_rs::hmac; + +use crate::domain::error::DomainError; + +/// Reserved backend reference the auto-generated fence key is stored under +/// (tenant = nil UUID, owner = `None`). It has no `credstore_secrets` row, +/// so no API path can resolve, overwrite, or delete it: resolution always +/// starts from a metadata row, and external callers always carry a real +/// tenant, never nil. +pub const FENCE_KEY_REF: &str = "cfs-internal-fence-key"; + +/// Fence-key id stamped into `fp_key_id` alongside every fingerprint. +/// v1 uses a single key; the id column plus the reserved-reference naming +/// are the keyring groundwork for rotation (out of scope — see the feature +/// spec §3, `algo-fp-compute-verify`). +pub const CURRENT_FENCE_KEY_ID: i16 = 1; + +/// Fence-key length in bytes. +pub const FENCE_KEY_LEN: usize = 32; + +/// Max jitter (ms) before the fence-key bootstrap read/generate. De-synchronises +/// replicas that cold-start together so a create collision is unlikely. The +/// plugin port has no atomic create-if-absent, so this is a probabilistic +/// defence, not mutual exclusion; residual races stay fail-closed (mismatch → +/// 404 → self-heal on rewrite). One-time cost on a replica's first fence op. +/// Zero under unit tests: the jitter is a timing-only knob, not logic under +/// test, and real sleeps would make the suite slow and time-sensitive. (Lives +/// here, not in the service module, so its test override does not put a +/// `#[cfg(test)]` item in a file that has a companion `_tests.rs` — DE1101.) +#[cfg(not(test))] +pub const BOOTSTRAP_JITTER_MAX_MS: u64 = 500; +#[cfg(test)] +pub const BOOTSTRAP_JITTER_MAX_MS: u64 = 0; + +/// `HMAC-SHA256(fence_key, value)` — the full 32-byte fingerprint stored in +/// `credstore_secrets.value_fp`. Never truncated: the value is internal-only, +/// and truncation would only add fence-collision surface. +/// +/// Backed by AWS-LC (`aws-lc-rs`), the same crypto library as the workspace TLS +/// stack; under a `--features fips` build its `fips` feature unifies on, so the +/// MAC runs through the FIPS-validated module (`RustCrypto`'s pure-Rust +/// `sha2`/`hmac`, banned by the DE0708 FIPS-hasher lint, never could). +#[must_use] +pub fn compute_fp(key: &[u8], value: &[u8]) -> Vec { + let key = hmac::Key::new(hmac::HMAC_SHA256, key); + hmac::sign(&key, value).as_ref().to_vec() +} + +/// Constant-time fingerprint verification (`aws_lc_rs::hmac::verify`). +#[must_use] +pub fn verify_fp(key: &[u8], value: &[u8], fp: &[u8]) -> bool { + let key = hmac::Key::new(hmac::HMAC_SHA256, key); + hmac::verify(&key, value, fp).is_ok() +} + +/// Generate a fresh [`FENCE_KEY_LEN`]-byte fence key from the platform CSPRNG +/// (AWS-LC; FIPS-validated under a `--features fips` build, as above). +/// +/// # Errors +/// +/// Returns [`DomainError::Internal`] if the system RNG fails. +pub fn generate_key() -> Result, DomainError> { + let mut buf = [0u8; FENCE_KEY_LEN]; + aws_lc_rs::rand::fill(&mut buf) + .map_err(|_| DomainError::internal("fence: system RNG failed to generate a fence key"))?; + Ok(buf.to_vec()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn fp_is_deterministic_and_key_and_value_sensitive() { + let k1 = vec![1u8; FENCE_KEY_LEN]; + let k2 = vec![2u8; FENCE_KEY_LEN]; + let a = compute_fp(&k1, b"value"); + assert_eq!(a, compute_fp(&k1, b"value")); + assert_eq!(a.len(), 32); + assert_ne!(a, compute_fp(&k2, b"value")); + assert_ne!(a, compute_fp(&k1, b"other")); + } + + #[test] + fn verify_accepts_matching_and_rejects_foreign_fp() { + let key = generate_key().expect("key"); + let fp = compute_fp(&key, b"secret"); + assert!(verify_fp(&key, b"secret", &fp)); + assert!(!verify_fp(&key, b"tampered", &fp)); + assert!(!verify_fp(&generate_key().expect("key"), b"secret", &fp)); + // Truncated/foreign-length fingerprints must not verify. + assert!(!verify_fp(&key, b"secret", &fp[..8])); + } + + #[test] + fn generated_keys_are_distinct_and_sized() { + let a = generate_key().expect("key"); + let b = generate_key().expect("key"); + assert_eq!(a.len(), FENCE_KEY_LEN); + assert_ne!(a, b); + } + + #[test] + fn fence_key_ref_is_a_valid_secret_ref() { + assert!(credstore_sdk::SecretRef::new(FENCE_KEY_REF).is_ok()); + } +} diff --git a/gears/credstore/credstore/src/domain/secret/model.rs b/gears/credstore/credstore/src/domain/secret/model.rs new file mode 100644 index 000000000..3bfa2c7c0 --- /dev/null +++ b/gears/credstore/credstore/src/domain/secret/model.rs @@ -0,0 +1,186 @@ +use credstore_sdk::{OwnerId, SecretRef, SharingMode, TenantId, WriteOptions}; +use time::OffsetDateTime; +use toolkit_macros::domain_model; +use uuid::Uuid; + +#[domain_model] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum SecretStatus { + Provisioning, + Active, + /// Delete saga in flight: invisible to resolution, still holds the + /// partial unique index until backend cleanup completes. + Deprovisioning, +} +impl SecretStatus { + #[must_use] + pub fn as_smallint(self) -> i16 { + match self { + Self::Provisioning => 1, + Self::Active => 2, + Self::Deprovisioning => 3, + } + } + #[must_use] + pub fn from_smallint(v: i16) -> Option { + match v { + 1 => Some(Self::Provisioning), + 2 => Some(Self::Active), + 3 => Some(Self::Deprovisioning), + _ => None, + } + } +} + +#[domain_model] +#[derive(Debug, Clone)] +pub struct SecretRow { + pub id: Uuid, + pub tenant_id: TenantId, + pub reference: String, + pub sharing: SharingMode, + pub owner_id: OwnerId, + pub status: SecretStatus, + /// Monotonic version (optimistic-locking groundwork); 1 on create. + pub version: i64, + /// Deterministic v5 UUID of the secret's GTS type id (the stored + /// representation); immutable for the row's lifetime. Resolved to the + /// type id + traits via the types-registry per operation. + pub secret_type_uuid: Uuid, + /// Expiry instant for expirable types; expired rows do not resolve. + pub expires_at: Option, + /// Value-fingerprint fence (`HMAC-SHA256(fence_key, value)`) binding this + /// row's metadata to the backend value it was written for. `None` only + /// for out-of-band seeded rows, which are served on trust and backfilled + /// on first read / reaper sweep. Internal-only: never serialized to any + /// API response or log. + pub value_fp: Option>, + /// Fence-key id `value_fp` was computed under; `None` iff `value_fp` is. + pub fp_key_id: Option, +} + +/// Everything a write needs beyond identity and value: sharing mode, +/// create-only vs upsert, optimistic-concurrency precondition, and typed +/// options (secret type + expiry). +#[domain_model] +#[derive(Debug, Clone, Default)] +pub struct WriteSpec { + pub sharing: SharingMode, + pub create_only: bool, + pub precondition: Option, + pub opts: WriteOptions, + /// When overwriting an existing secret, keep the stored sharing mode + /// instead of applying `sharing`. Set for a PUT that omitted `sharing`, so + /// a value rotation (`{"value": "..."}`) never silently narrows a `shared` + /// secret back to `tenant`. `sharing` is still the class/create default. + pub preserve_sharing: bool, +} + +impl WriteSpec { + /// Upsert with default options. + #[must_use] + pub fn upsert(sharing: SharingMode) -> Self { + Self { + sharing, + ..Self::default() + } + } + + /// Create-only (409 on same-class duplicate) with default options. + #[must_use] + pub fn create(sharing: SharingMode) -> Self { + Self { + sharing, + create_only: true, + ..Self::default() + } + } + + /// Attach an `If-Match` precondition. + #[must_use] + pub fn with_precondition(mut self, precondition: Option) -> Self { + self.precondition = precondition; + self + } + + /// Attach typed write options (secret type, expiry). + #[must_use] + pub fn with_opts(mut self, opts: WriteOptions) -> Self { + self.opts = opts; + self + } + + /// Preserve the existing secret's sharing on overwrite (see field docs). + #[must_use] + pub fn preserve_sharing(mut self, preserve: bool) -> Self { + self.preserve_sharing = preserve; + self + } +} + +/// Optimistic-concurrency precondition for a write, parsed from `If-Match`. +#[domain_model] +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum WritePrecondition { + /// `If-Match: *` — the target secret must already exist. + Exists, + /// `If-Match: "."` — the generation-bound strong validator. + /// `id` is the row UUID (a fresh one per recreated secret), so a + /// validator from a deleted-and-recreated secret's earlier generation can + /// never match the current row even when the version counters coincide + /// (no ABA); `version` is the per-row monotonic counter. + Version { + /// Row (generation) UUID the caller's validator was minted for. + id: Uuid, + /// Version counter the caller last observed. + version: i64, + }, + /// `If-Match: ".", ".", …` — a multi-valued list (RFC 7232 + /// §3.1). The precondition is satisfied if the current row matches **any** + /// listed `(id, version)` validator. + AnyVersion(Vec<(Uuid, i64)>), +} + +#[domain_model] +#[derive(Debug, Clone)] +pub struct NewSecret { + pub id: Uuid, + pub tenant_id: TenantId, + pub reference: SecretRef, + pub sharing: SharingMode, + pub owner_id: OwnerId, + /// Deterministic v5 UUID of the (registry-validated) GTS type id. + pub secret_type_uuid: Uuid, + pub expires_at: Option, + /// Fence fingerprint of the value this create will write to the backend + /// (API creates always stamp; only out-of-band seeding leaves it NULL). + pub value_fp: Vec, + /// Fence-key id `value_fp` was computed under. + pub fp_key_id: i16, +} + +#[cfg(test)] +mod tests { + use super::SecretStatus; + + #[test] + fn secret_status_smallint_round_trips() { + for s in [ + SecretStatus::Provisioning, + SecretStatus::Active, + SecretStatus::Deprovisioning, + ] { + assert_eq!(SecretStatus::from_smallint(s.as_smallint()), Some(s)); + } + assert_eq!(SecretStatus::Provisioning.as_smallint(), 1); + assert_eq!(SecretStatus::Active.as_smallint(), 2); + assert_eq!(SecretStatus::Deprovisioning.as_smallint(), 3); + } + + #[test] + fn secret_status_from_smallint_rejects_out_of_domain() { + assert_eq!(SecretStatus::from_smallint(0), None); + assert_eq!(SecretStatus::from_smallint(4), None); + assert_eq!(SecretStatus::from_smallint(-1), None); + } +} diff --git a/gears/credstore/credstore/src/domain/secret/repo.rs b/gears/credstore/credstore/src/domain/secret/repo.rs new file mode 100644 index 000000000..83e00609a --- /dev/null +++ b/gears/credstore/credstore/src/domain/secret/repo.rs @@ -0,0 +1,149 @@ +use async_trait::async_trait; +use credstore_sdk::{OwnerId, SecretRef, SharingMode, TenantId}; +use time::OffsetDateTime; +use toolkit_security::AccessScope; +use uuid::Uuid; + +use crate::domain::error::DomainError; +use crate::domain::ports::metrics::SecretCounts; +use crate::domain::secret::model::{NewSecret, SecretRow, SecretStatus}; + +#[async_trait] +pub trait SecretRepo: Send + Sync { + /// Resolve the winning active secret for `req_tenant` walking the ordered + /// `chain` (req first, root last), applying two-phase priority + sharing. + async fn resolve_for_get( + &self, + req_tenant: TenantId, + subject: OwnerId, + key: &SecretRef, + chain: &[Uuid], + ) -> Result, DomainError>; + + /// Insert a `provisioning` row (scope-checked); maps UNIQUE → Conflict. + async fn insert_provisioning( + &self, + scope: &AccessScope, + new: &NewSecret, + ) -> Result<(), DomainError>; + + /// Flip provisioning → active (scope-checked). 0 rows → Conflict. + async fn mark_active(&self, scope: &AccessScope, id: Uuid) -> Result<(), DomainError>; + + /// Bump the version (and set sharing + expiry + fence fingerprint) of an + /// existing active row, keyed by id. Atomic `version = version + 1`. When + /// `expected_version` is `Some`, the bump is gated on `version = expected` + /// (optimistic concurrency). `expires_at` fully replaces the stored expiry + /// (a PUT is a whole-value replace, expiry included). `value_fp` is the + /// fingerprint of the value this write puts to the backend; it is set in + /// the same atomic UPDATE as `sharing`, which is what makes a fingerprint + /// match on read prove that value and metadata came from one writer. + /// Returns the post-write row, or `None` if no active row matched (row gone, + /// or — under `expected_version` — the version no longer matches). + async fn touch( + &self, + scope: &AccessScope, + id: Uuid, + sharing: SharingMode, + expected_version: Option, + expires_at: Option, + value_fp: Vec, + ) -> Result, DomainError>; + + /// Stamp the fence fingerprint onto an out-of-band seeded row (CAS on + /// `value_fp IS NULL`; a concurrent PUT that already stamped wins → `false`). + /// Does NOT bump `version`/`updated_at` — nothing client-visible changed. + async fn backfill_fp( + &self, + id: Uuid, + value_fp: Vec, + fp_key_id: i16, + ) -> Result; + + /// Active rows still missing a fence fingerprint (bounded batch; reaper + /// backfill sweep). + async fn list_unfenced(&self, limit: u64) -> Result, DomainError>; + + /// Flip active → deprovisioning (scope-checked), stamping `updated_at` + /// (the deprovisioning-timeout clock). When `expected_version` is `Some`, + /// the flip is gated on `version = expected` (optimistic concurrency). + /// The version itself is NOT bumped — an `If-Match` retry of the same + /// delete must still match. Returns `false` if no active row matched. + async fn mark_deprovisioning( + &self, + scope: &AccessScope, + id: Uuid, + expected_version: Option, + ) -> Result; + + /// Find the caller's own-tenant row (two-phase: private-for-subject, else + /// tenant/shared). Returns `active` rows and — so a `DELETE` retry can + /// resume a stuck delete saga — `deprovisioning` rows; never + /// `provisioning` ones. + async fn find_own( + &self, + scope: &AccessScope, + tenant: TenantId, + subject: OwnerId, + key: &SecretRef, + ) -> Result, DomainError>; + + /// Find the row a write of `sharing` would target, by sharing-class identity + /// (mirrors the partial unique indexes): `Private` → `(tenant, ref, owner)`, + /// `Tenant`/`Shared` → `(tenant, ref)` among non-private. Unlike [`find_own`] + /// this never crosses the private boundary, so a private write does not see a + /// coexisting tenant/shared secret (and vice-versa) — they coexist per design. + async fn find_for_write( + &self, + scope: &AccessScope, + tenant: TenantId, + subject: OwnerId, + key: &SecretRef, + sharing: SharingMode, + ) -> Result, DomainError>; + + /// Delete a row by id (scope-checked). When `expected_version` is `Some`, + /// the delete is gated on `version = expected` (optimistic concurrency). + /// 0 rows → `NotFound`. + async fn delete_by_id( + &self, + scope: &AccessScope, + id: Uuid, + expected_version: Option, + ) -> Result<(), DomainError>; + + /// List stale saga rows for the reaper (unscoped, bounded by `limit`): + /// `provisioning` rows whose `updated_at` is older than + /// `provisioning_older_than_secs` and `deprovisioning` rows older than + /// `deprovisioning_older_than_secs`. Provisioning rows are never updated + /// after insert, so `updated_at` equals `created_at` for them. + async fn list_stale_pending( + &self, + provisioning_older_than_secs: u64, + deprovisioning_older_than_secs: u64, + limit: u64, + ) -> Result, DomainError>; + + /// Delete a saga row by id, but only while it still holds `expected` + /// status (unscoped; reaper cleanup). The status guard fences the delete + /// against a concurrent saga transition (notably a slow create's + /// `mark_active`), so a secret that became active after the reaper listed + /// it is never removed. Returns `true` when this call removed the row, + /// `false` when it was already gone or has moved on (both benign). + async fn reap_by_id(&self, id: Uuid, expected: SecretStatus) -> Result; + + /// Flip expired `active` rows (`expires_at <= now`) to `deprovisioning` + /// (unscoped; reaper), stamping `updated_at`. The rows then complete + /// through the ordinary deprovisioning sweep. Returns rows flipped. + async fn mark_expired_deprovisioning(&self) -> Result; + + /// Inventory counts by sharing + provisioning + distinct tenants (for gauges). + async fn inventory(&self) -> Result; + + /// True iff `tenant` is within the read `scope` (closure-backed for subtree). + async fn scope_includes_tenant( + &self, + scope: &AccessScope, + tenant: Uuid, + ) -> Result; +} diff --git a/gears/credstore/credstore/src/domain/secret/service.rs b/gears/credstore/credstore/src/domain/secret/service.rs new file mode 100644 index 000000000..c446e1011 --- /dev/null +++ b/gears/credstore/credstore/src/domain/secret/service.rs @@ -0,0 +1,1725 @@ +use std::sync::Arc; +use std::time::{Duration, Instant}; + +use authz_resolver_sdk::PolicyEnforcer; +use credstore_sdk::{ + CredStoreError, CredStorePluginClientV1, GetSecretResponse, OwnerId, SecretRef, SecretValue, + SharingMode, TenantId, +}; +use tokio::time::sleep; +use toolkit_macros::domain_model; +use toolkit_security::{AccessScope, SecurityContext}; +use uuid::Uuid; + +use authz_resolver_sdk::pep::ResourceType; + +use crate::domain::authz::{self, actions, scope_for}; +use crate::domain::error::DomainError; +use crate::domain::ports::metrics::{ + CredStoreMetricsPort, Dep, DepOp, FenceVerify, Outcome, ReadOutcome, +}; +use crate::domain::ports::plugin::PluginSelector; +use crate::domain::resolver::TenantDirectory; +use credstore_sdk::SecretType; +use time::OffsetDateTime; + +use crate::domain::secret::fence; +use crate::domain::secret::model::{ + NewSecret, SecretRow, SecretStatus, WritePrecondition, WriteSpec, +}; +use crate::domain::secret::repo::SecretRepo; +use crate::domain::secret::type_resolver::{ResolvedSecretType, SecretTypeResolver}; +use crate::domain::secret::typing; + +/// Maps a [`CredStoreError`] from the plugin layer to a [`DomainError`]. +#[must_use] +pub fn map_plugin_err(e: CredStoreError) -> DomainError { + match e { + CredStoreError::NotFound => DomainError::NotFound, + CredStoreError::AccessDenied => DomainError::AccessDenied { cause: None }, + CredStoreError::ServiceUnavailable { + detail, + retry_after, + } => { + // The plugin's own detail may embed backend/infra specifics (a + // future vault-backed plugin's error text is not curated for the + // credstore boundary the way a CF-internal sibling's is, and could + // in principle carry sensitive material — e.g. echo a token being + // written). The wire already redacts it; for the same reason we do + // NOT log the raw detail either (a credential store must not risk + // secret material in its logs — review finding #4). We record only + // its length so operators can still tell an empty detail from a + // populated one when correlating an outage. + tracing::warn!( + detail_len = detail.len(), + "credstore: storage plugin reported unavailable (detail redacted)" + ); + DomainError::ServiceUnavailable { + detail: "storage backend unavailable".to_owned(), + retry_after, + cause: None, + } + } + // Operator misconfiguration, not a transient outage: keep a stable, + // distinguishable detail and no `retry_after` so callers don't retry. + CredStoreError::NoPluginAvailable => DomainError::ServiceUnavailable { + detail: "no storage plugin registered".into(), + retry_after: None, + cause: None, + }, + CredStoreError::Conflict => DomainError::Conflict, + // These are plugin contract violations (a plugin should never return + // them). Their free-text payloads originate in the plugin, and — like + // the `ServiceUnavailable` detail above — a future non-CF backend's + // error text is not curated for the credstore boundary and could carry + // secret material (e.g. echo a token being written). A credential store + // must not risk that in its own logs, so we drop the raw text and keep + // only its length as an internal diagnostic (review finding #4). + CredStoreError::InvalidSecretRef { reason } => DomainError::Internal { + diagnostic: format!( + "plugin returned InvalidSecretRef (detail redacted, {} bytes)", + reason.len() + ), + cause: None, + }, + CredStoreError::UnsupportedTransition { detail } => DomainError::Internal { + diagnostic: format!( + "plugin returned UnsupportedTransition (detail redacted, {} bytes)", + detail.len() + ), + cause: None, + }, + CredStoreError::TypeViolation { reason, detail } => DomainError::Internal { + diagnostic: format!( + "plugin returned TypeViolation (detail redacted, {} bytes)", + reason.len() + detail.len() + ), + cause: None, + }, + CredStoreError::Internal(s) => DomainError::Internal { + diagnostic: format!( + "plugin returned Internal error (detail redacted, {} bytes)", + s.len() + ), + cause: None, + }, + } +} + +/// Bounded retries when a PUT loses the create race (the winner holds a +/// provisioning row invisible to `find_for_write`); the winner marks active +/// within ms, so a small bounded budget resolves the race. +const CREATE_RACE_MAX_RETRIES: u32 = 3; + +/// Small backoff between create-race retries; the winner marks active within ms. +const CREATE_RACE_RETRY_BACKOFF_MS: u64 = 25; + +/// Upper bound on stale saga rows processed per reaper tick, so one tick's +/// backend-reconciliation work stays bounded; the remainder waits for the +/// next tick. +const REAP_BATCH_LIMIT: u64 = 256; + +/// Reaper cadence and saga-timeout settings (from `ReaperCfg`). +#[domain_model] +#[derive(Debug, Clone, Copy)] +#[allow( + clippy::struct_field_names, + reason = "field names mirror the serialized ReaperCfg config keys" +)] +pub struct ReaperSettings { + pub tick_secs: u64, + pub provisioning_timeout_secs: u64, + pub deprovisioning_timeout_secs: u64, +} + +/// Re-read the cached fence key at least this often. Bounds the window in which +/// a replica that adopted a losing key during a bootstrap race keeps using it: +/// it converges on the stored key within this TTL even if it never hits a +/// verify mismatch. The backend read is cheap and the key almost never changes. +const FENCE_KEY_CACHE_TTL: Duration = Duration::from_mins(1); + +/// Minimum spacing between fingerprint-mismatch–triggered backend re-reads of +/// the fence key. A crosswise last-writer-wins interleave produces a row whose +/// value fingerprint *never* matches (a by-design, fail-closed poison), so +/// without this every read of such a row would re-read the key from the +/// backend. The cooldown bounds those forced reloads to one per window per +/// replica. It gates on the *last forced reload* (not cache age), so the first +/// mismatch after a quiet window always re-reads and heals a genuinely stale +/// key; only the repeated, never-healing poison reads are suppressed. +const FENCE_KEY_REFRESH_COOLDOWN: Duration = Duration::from_secs(15); + +/// The in-process fence key, shared out of the cache. Zeroizing so the +/// process's single highest-value secret is wiped from the heap on drop. +type FenceKey = Arc>>; + +/// Cached fence key plus when it was loaded, so [`Service::fence_key`] can +/// re-read it from the backend after [`FENCE_KEY_CACHE_TTL`] and converge on +/// the stored key even without a verify mismatch to trigger a refresh. +#[domain_model] +struct CachedFenceKey { + key: FenceKey, + loaded_at: Instant, +} + +/// `CredStore` domain service — get / put / delete with walk-up, saga, and `AuthZ`. +#[domain_model] +pub struct Service { + repo: Arc, + dir: Arc, + enforcer: PolicyEnforcer, + plugins: Arc, + types: Arc, + metrics: Arc, + reaper: ReaperSettings, + /// In-process cache of the value-fingerprint fence key (auto-generated, + /// stored in the value-store backend under [`fence::FENCE_KEY_REF`]). + /// A fingerprint mismatch triggers a cooldown-gated, single-flighted + /// backend re-read (swapped in place, never evicted to `None`) so a replica + /// holding a stale key self-heals without a poisoned row thrashing the + /// cache. Never logged, never on the wire. + /// Wrapped in [`zeroize::Zeroizing`] so the process's single highest-value + /// secret (it fingerprints every stored value) is wiped from the heap when + /// the cache is replaced or the process exits, matching how every + /// `SecretValue` in the system zeroizes on drop. + fence_key: std::sync::RwLock>, + /// Serialises fence-key (re)load on a single replica so concurrent first + /// writers don't each run the bootstrap and race one another locally; the + /// jitter protocol in [`Service::load_fence_key`] then only has to defend + /// against *inter*-replica races. + bootstrap_lock: tokio::sync::Mutex<()>, + /// When the last fingerprint-mismatch–triggered *forced* reload of the + /// fence key ran, gating the next one to [`FENCE_KEY_REFRESH_COOLDOWN`]. + /// `None` until the first forced reload, so the first mismatch always + /// re-reads (healing a stale key); thereafter a poisoned row cannot drive + /// more than one backend re-read per window. Independent of the cache's own + /// load time. + last_fence_refresh: std::sync::Mutex>, +} + +/// Arguments for [`Service::overwrite_existing`] — bundled so the backend-first +/// overwrite doesn't carry a long positional argument list. +#[domain_model] +struct OverwriteArgs<'a> { + ctx: &'a SecurityContext, + scope: &'a AccessScope, + plugin: &'a Arc, + key: &'a SecretRef, + value: SecretValue, + sharing: SharingMode, + existing_id: Uuid, + expected_version: Option, + expires_at: Option, +} + +impl Service { + /// Creates a new [`Service`]. + #[must_use] + pub fn new( + repo: Arc, + dir: Arc, + enforcer: PolicyEnforcer, + plugins: Arc, + types: Arc, + metrics: Arc, + reaper: ReaperSettings, + ) -> Self { + Self { + repo, + dir, + enforcer, + plugins, + types, + metrics, + reaper, + fence_key: std::sync::RwLock::new(None), + bootstrap_lock: tokio::sync::Mutex::new(()), + last_fence_refresh: std::sync::Mutex::new(None), + } + } + + /// Returns the configured reaper tick interval in seconds. + #[must_use] + pub fn reaper_tick_secs(&self) -> u64 { + self.reaper.tick_secs + } + + /// Evaluate the PDP scope for `action`, recording it as a timed dependency. + /// + /// The PDP call gates every get/put/delete and drives 503s, so it is timed + /// like the plugin and tenant-resolver dependencies. + async fn scope_for_timed( + &self, + ctx: &SecurityContext, + resource: &ResourceType, + action: &str, + ) -> Result { + let t0 = Instant::now(); + let result = scope_for(&self.enforcer, ctx, resource, action).await; + // A PDP *denial* (`AccessDenied`) is a normal authorization decision — + // the dependency answered — not a health signal; counting it as an + // error would inflate the PDP error rate and risk false outage alerts. + // Only an evaluation failure/outage is a dependency error. Mirrors the + // domain/transport split the type resolver makes. + let outcome = match &result { + Ok(_) | Err(DomainError::AccessDenied { .. }) => Outcome::Success, + Err(_) => Outcome::Error, + }; + self.metrics.dependency( + Dep::Pdp, + DepOp::Evaluate, + outcome, + t0.elapsed().as_secs_f64(), + ); + result + } + + /// Resolve the type of an **existing** row (`secret_type_uuid` was + /// validated when the row was written). An `UNKNOWN_SECRET_TYPE` + /// violation here means the type was deregistered while rows persist — + /// an operational inconsistency, not a caller error — so it is remapped + /// onto a retryable 503; registry outages propagate as 503 already. + async fn resolve_stored(&self, type_uuid: Uuid) -> Result { + match self.types.resolve(type_uuid).await { + Err(DomainError::TypeViolation { detail, .. }) => { + tracing::warn!( + uuid = %type_uuid, + detail = %detail, + "stored secret type no longer resolves in the types-registry" + ); + Err(DomainError::ServiceUnavailable { + detail: "secret type is not resolvable".to_owned(), + retry_after: None, + cause: None, + }) + } + other => other, + } + } + + // ── Value-fingerprint fence (docs/features/001-value-fingerprint-fence.md) ── + + /// Internal service context for fence-key backend operations. Nil subject + /// and nil tenant: plugins are pure value stores keyed by the explicit + /// arguments, and no external caller ever carries the nil tenant, which is + /// what keeps the reserved key unreachable through the API. + fn fence_ctx() -> Result<(SecurityContext, TenantId, SecretRef), DomainError> { + let ctx = SecurityContext::builder() + .subject_id(Uuid::nil()) + .subject_tenant_id(Uuid::nil()) + .build() + .map_err(|e| { + DomainError::internal(format!("fence: failed to build service context: {e}")) + })?; + let key_ref = SecretRef::new(fence::FENCE_KEY_REF).map_err(|e| { + DomainError::internal(format!("fence: reserved key reference invalid: {e}")) + })?; + Ok((ctx, TenantId(Uuid::nil()), key_ref)) + } + + /// The cached fence key, loading (and on first boot generating) it from the + /// value-store backend when the cache is cold or older than + /// [`FENCE_KEY_CACHE_TTL`]. The TTL re-read lets a replica that adopted a + /// losing key during a bootstrap race converge on the stored key without + /// waiting for a verify mismatch. + async fn fence_key( + &self, + plugin: &Arc, + ) -> Result { + { + let guard = self + .fence_key + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if let Some(cached) = guard.as_ref() + && cached.loaded_at.elapsed() < FENCE_KEY_CACHE_TTL + { + return Ok(Arc::clone(&cached.key)); + } + } + self.load_fence_key(plugin, false).await + } + + /// Re-read the fence key from the backend — the self-heal a fingerprint + /// mismatch triggers, so a replica whose cache went stale (key re-created + /// under it) converges without restart. + /// + /// Unlike a naive "evict + reload", this must not turn a *poisoned* row (a + /// permanent, by-design mismatch from a crosswise LWW interleave) into a + /// backend read per request nor evict the shared key for every concurrent + /// operation. Two guards, both delegated to [`Self::load_fence_key`] under + /// its `bootstrap_lock` (which also single-flights concurrent callers): + /// the reload is skipped when the cache was (re)loaded within + /// [`FENCE_KEY_REFRESH_COOLDOWN`] (a re-read cannot help inside the window), + /// and the cache is swapped in place rather than cleared to `None`. + async fn refresh_fence_key( + &self, + plugin: &Arc, + ) -> Result { + self.load_fence_key(plugin, true).await + } + + /// (Re)load the fence key from the backend's reserved entry. + /// + /// The plugin port has no atomic create-if-absent, so first-boot generation + /// cannot be a true single-writer operation. Instead we make a bootstrap + /// race both **unlikely** and **low-impact**: + /// + /// * A per-replica [`Self::bootstrap_lock`] serialises this so concurrent + /// first writers on one replica don't race each other locally. + /// * When the key is absent we jitter, re-check, and only then generate and + /// `put`, de-synchronising replicas that cold-start together. The + /// publishing `put` is unconditional (the port has no create-if-absent), + /// so it can still clobber a peer whose write landed in the tiny window + /// between our re-check and our `put` — that residual race is the + /// fail-closed case below. + /// * After publishing we jitter again and **adopt whatever is stored** — + /// ours if we won, a racing writer's if theirs landed last — rather than + /// re-`put`ting. The cache is populated only from this settle read, so no + /// value is ever stamped with a candidate that hasn't survived it. + /// + /// The key has no metadata row, so it is invisible to every API path. Any + /// residual divergence is fail-closed: a wrong key only ever yields a + /// fingerprint mismatch (404 + self-heal on rewrite), never a value served + /// under foreign metadata. + /// + /// `force` distinguishes the two entry points. A normal load (`false`) is + /// satisfied by any cache younger than [`FENCE_KEY_CACHE_TTL`]. A refresh + /// (`true`, from a fingerprint mismatch) re-reads the backend unless a prior + /// forced reload ran within [`FENCE_KEY_REFRESH_COOLDOWN`] — so the first + /// mismatch always re-reads and heals a genuinely stale key, while a + /// poisoned row (a permanent, never-healing mismatch) is bounded to one + /// re-read per window. The `bootstrap_lock` single-flights either path, so a + /// burst of concurrent callers yields at most one backend read. + /// Whether a forced fence-key reload ran within [`FENCE_KEY_REFRESH_COOLDOWN`]. + /// `false` until the first forced reload, so an initial mismatch is never + /// suppressed. + fn fence_refresh_within_cooldown(&self) -> bool { + self.last_fence_refresh + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .is_some_and(|at| at.elapsed() < FENCE_KEY_REFRESH_COOLDOWN) + } + + async fn load_fence_key( + &self, + plugin: &Arc, + force: bool, + ) -> Result { + let _bootstrap = self.bootstrap_lock.lock().await; + // Reuse the cache rather than re-reading the backend when: a non-forced + // load finds it still within the TTL, or a forced refresh finds a prior + // forced reload within the cooldown (re-reading sooner cannot heal a + // poisoned row). A concurrent caller may also have just reloaded while + // we waited on the lock — this same check picks that up. + { + let guard = self + .fence_key + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if let Some(cached) = guard.as_ref() { + let reuse = if force { + self.fence_refresh_within_cooldown() + } else { + cached.loaded_at.elapsed() < FENCE_KEY_CACHE_TTL + }; + if reuse { + return Ok(Arc::clone(&cached.key)); + } + } + } + // About to re-read on a mismatch: stamp the forced-reload clock now so + // repeated poisoned mismatches (and concurrent ones behind the lock) + // coalesce, even if the read below fails. + if force { + *self + .last_fence_refresh + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) = Some(Instant::now()); + } + + let (ctx, tenant, key_ref) = Self::fence_ctx()?; + // Fast path: the key already exists (every start after the first, and + // most concurrent cold starts once one replica has written it). + if let Some(v) = plugin + .get(&ctx, &tenant, &key_ref, None) + .await + .map_err(map_plugin_err)? + { + return Ok(self.store_fence_key(zeroize::Zeroizing::new(v.as_bytes().to_vec()))); + } + + // Cold-start bootstrap. Jitter, then re-check: a peer may have won. + Self::bootstrap_jitter().await; + if let Some(v) = plugin + .get(&ctx, &tenant, &key_ref, None) + .await + .map_err(map_plugin_err)? + { + return Ok(self.store_fence_key(zeroize::Zeroizing::new(v.as_bytes().to_vec()))); + } + + // Still absent: publish a candidate, settle, then adopt what landed. + // Keep the material in zeroizing buffers so no plain `Vec` copy + // lingers on the heap after this scope. + let candidate = zeroize::Zeroizing::new(fence::generate_key()?); + plugin + .put( + &ctx, + &tenant, + &key_ref, + SecretValue::new(candidate.to_vec()), + None, + ) + .await + .map_err(map_plugin_err)?; + Self::bootstrap_jitter().await; + let stored = plugin + .get(&ctx, &tenant, &key_ref, None) + .await + .map_err(map_plugin_err)? + // Our own write vanishing is not expected; fall back to the + // candidate so bootstrap still makes progress (fail-closed). + .map_or(candidate, |v| { + zeroize::Zeroizing::new(v.as_bytes().to_vec()) + }); + Ok(self.store_fence_key(stored)) + } + + /// Publish `key_bytes` as the cached fence key, timestamped for the TTL + /// re-read, and hand back the shared handle. + fn store_fence_key(&self, key_bytes: zeroize::Zeroizing>) -> FenceKey { + let key: FenceKey = Arc::new(key_bytes); + *self + .fence_key + .write() + .unwrap_or_else(std::sync::PoisonError::into_inner) = Some(CachedFenceKey { + key: Arc::clone(&key), + loaded_at: Instant::now(), + }); + key + } + + /// Sleep a random `0..=FENCE_BOOTSTRAP_JITTER_MAX_MS` to de-synchronise + /// replicas racing the first-boot fence-key generation. + async fn bootstrap_jitter() { + use rand::RngExt as _; + let ms = rand::rng().random_range(0..=fence::BOOTSTRAP_JITTER_MAX_MS); + sleep(Duration::from_millis(ms)).await; + } + + /// Fingerprint `value` under the current fence key (write-path stamping). + async fn stamp_fp( + &self, + plugin: &Arc, + value: &SecretValue, + ) -> Result, DomainError> { + let key = self.fence_key(plugin).await?; + Ok(fence::compute_fp(key.as_slice(), value.as_bytes())) + } + + /// Best-effort lazy fingerprint backfill of an out-of-band seeded row + /// (`value_fp IS NULL`), using the value just served. CAS on NULL in the + /// repo, so a concurrent PUT that already stamped wins; never bumps the + /// version (the caller's `ETag` stays stable). Failures only log + count — + /// the read that triggered the backfill already succeeded. + async fn backfill_row_fp( + &self, + plugin: &Arc, + row: &SecretRow, + value: &SecretValue, + ) { + let fp = match self.stamp_fp(plugin, value).await { + Ok(fp) => fp, + Err(e) => { + tracing::warn!( + id = %row.id, + err = %e, + "credstore: fence backfill skipped (fence key unavailable); retried on a later read/sweep" + ); + self.metrics.fence_backfill(Outcome::Error); + return; + } + }; + let outcome = match self + .repo + .backfill_fp(row.id, fp, fence::CURRENT_FENCE_KEY_ID) + .await + { + Ok(true) => Outcome::Success, + // CAS matched 0 rows: a concurrent PUT already stamped — done. + Ok(false) => Outcome::NotFound, + Err(e) => { + tracing::warn!( + id = %row.id, + err = %e, + "credstore: fence backfill failed; retried on a later read/sweep" + ); + Outcome::Error + } + }; + self.metrics.fence_backfill(outcome); + } + + /// Retrieve a secret, walking up the tenant hierarchy. + /// + /// # Errors + /// + /// Returns [`DomainError::AccessDenied`] if the caller is out of scope. + /// Returns [`DomainError::NotFound`] if the plugin has no value for a resolved row. + pub async fn get( + &self, + ctx: &SecurityContext, + key: &SecretRef, + ) -> Result, DomainError> { + let req = TenantId(ctx.subject_tenant_id()); + let subject = OwnerId(ctx.subject_id()); + let chain = self.dir.ancestor_chain(ctx, req).await?; + + // Resolve first (prefetch — AUTHZ_USAGE_SCENARIOS S09): a secret that does + // not resolve is a 404 without consulting the PDP; there is nothing to + // authorize. + let Some(row) = self.repo.resolve_for_get(req, subject, key, &chain).await? else { + self.metrics.read_outcome(ReadOutcome::Miss); + return Ok(None); + }; + + // Single PDP evaluation, on the secret's full concrete type (including + // `generic`) as resolved from the types-registry, gated on the + // *caller's* tenant. Hierarchical visibility (a shared secret + // inherited from an ancestor) is decided by the resolver above, so + // the gate uses `req`, not the row's owner tenant — this is what lets + // inherited reads work. A PDP denial or an out-of-scope tenant is + // indistinguishable from a missing secret (anti-enumeration 404); a + // PDP or registry *outage* propagates as 503. + // + // The type resolve necessarily precedes the PDP (the PDP resource *is* + // the resolved concrete type), so an outage window is the one case + // where an unauthorized caller can distinguish an existing secret (503) + // from a missing one (404). This is an accepted, bounded consequence of + // failing closed, symmetric across the PDP and registry dependencies — + // do NOT "fix" it by returning 404 on an outage, which would mask a real + // outage from authorized callers. + let resolved = self.resolve_stored(row.secret_type_uuid).await?; + let scope = match self + .scope_for_timed( + ctx, + &authz::secret_type_resource(&resolved.gts_id), + actions::READ, + ) + .await + { + Ok(scope) => scope, + Err(DomainError::AccessDenied { .. }) => { + self.metrics.read_outcome(ReadOutcome::Miss); + return Ok(None); + } + Err(e) => return Err(e), + }; + if !self.repo.scope_includes_tenant(&scope, req.0).await? { + self.metrics.cross_tenant_denied(); + self.metrics.read_outcome(ReadOutcome::Miss); + return Ok(None); + } + + let depth = chain + .iter() + .position(|c| *c == row.tenant_id.0) + .unwrap_or(chain.len()); + self.metrics.walkup_depth(depth as u64); + + let owner = if row.sharing == SharingMode::Private { + Some(row.owner_id) + } else { + None + }; + + let plugin = self.plugins.resolve().await?; + let t0 = Instant::now(); + let result = plugin + .get(ctx, &row.tenant_id, key, owner.as_ref()) + .await + .map_err(map_plugin_err); + let secs = t0.elapsed().as_secs_f64(); + let outcome = match &result { + Ok(Some(_)) => Outcome::Success, + Ok(None) => Outcome::NotFound, + Err(_) => Outcome::Error, + }; + self.metrics + .dependency(Dep::Plugin, DepOp::PluginGet, outcome, secs); + let value: SecretValue = match result { + Ok(Some(v)) => v, + // The row resolved but the backend has no value — a 404 (rare; a + // storage artifact of a mid-saga row). + Ok(None) => return Err(DomainError::NotFound), + // A backend-plugin denial on read must NOT distinguish an existing + // secret from a missing one: fold it into the anti-enumeration miss, + // exactly like the PDP denial above. The bundled value-store never + // denies; this guards a future backend whose own ACLs could deny + // after the gear's PDP pass. + Err(DomainError::AccessDenied { .. }) => { + self.metrics.read_outcome(ReadOutcome::Miss); + return Ok(None); + } + Err(e) => return Err(e), + }; + + // Fence verification (feature spec `flow-get-fenced-read`): the value + // is served only when its fingerprint matches the row that authorized + // it. A mismatch means the backend value and the metadata row are not + // from the same writer (crosswise LWW interleave, a mid-saga artifact, + // or a stale out-of-band re-seed) — serving it could put one writer's + // value under another writer's sharing label, so fail closed as an + // anti-enumeration miss. A row without a fingerprint is out-of-band + // seeded: served on trust and backfilled best-effort. + if let Some(stored_fp) = &row.value_fp { + // TODO(rotation): verify under the key selected by `row.fp_key_id`, + // not the single current key. Equivalent for v1 (one key, id + // `CURRENT_FENCE_KEY_ID`), but once a keyring exists every + // pre-rotation row would fail closed here — see feature spec §3 + // `inst-fcv-2` / `algo-fp-compute-verify`. + let fkey = self.fence_key(&plugin).await?; + let mut fp_ok = fence::verify_fp(fkey.as_slice(), value.as_bytes(), stored_fp); + if !fp_ok { + // One-shot key refresh: a replica whose cached key went stale + // (key re-created under it) self-heals before failing closed. + let fkey = self.refresh_fence_key(&plugin).await?; + fp_ok = fence::verify_fp(fkey.as_slice(), value.as_bytes(), stored_fp); + } + if fp_ok { + self.metrics.fence_verify(FenceVerify::Ok); + } else { + self.metrics.fence_verify(FenceVerify::Mismatch); + self.metrics.read_outcome(ReadOutcome::Miss); + tracing::warn!( + tenant = %row.tenant_id.0, + key = %key.as_ref(), + "credstore get: value fingerprint mismatch; failing closed \ + (backend value does not match the metadata row; heals on the next successful put)" + ); + return Ok(None); + } + } else { + self.metrics.fence_verify(FenceVerify::Legacy); + self.backfill_row_fp(&plugin, &row, &value).await; + } + + let is_inherited = row.tenant_id != req; + self.metrics.read_outcome(if is_inherited { + ReadOutcome::HitInherited + } else { + ReadOutcome::HitOwn + }); + + Ok(Some(GetSecretResponse { + value, + id: row.id, + owner_tenant_id: row.tenant_id, + sharing: row.sharing, + is_inherited, + version: row.version, + secret_type: resolved.gts_id, + expires_at: row.expires_at, + })) + } + + /// Create or update a secret. + /// + /// A write targets the row of its own sharing class — `private` → + /// `(tenant, ref, owner)`, `tenant`/`shared` → `(tenant, ref)` — so a private + /// and a tenant/shared secret coexist under one reference (per design §4.1); + /// a write of one class never affects the other. + /// + /// Secret-type traits (design §5.4) are enforced before any side effect: + /// the type is immutable for an existing secret (`spec.opts.secret_type` + /// must be absent or equal), defaults to `generic` on create, and the + /// value/sharing/expiry are validated against the type's traits as + /// resolved from the types-registry. + /// + /// # Errors + /// + /// Returns [`DomainError::Conflict`] if `spec.create_only` and a secret of + /// the same sharing class already exists. + /// Returns [`DomainError::TypeViolation`] on a trait violation. + // Saga orchestration (validate -> scope -> resolve -> backend -> commit) is + // inherently branchy; kept as one function for readability of the flow. + #[allow(clippy::cognitive_complexity, clippy::too_many_lines)] + pub async fn put( + &self, + ctx: &SecurityContext, + key: &SecretRef, + value: SecretValue, + spec: WriteSpec, + ) -> Result<(), DomainError> { + let WriteSpec { + sharing, + create_only, + precondition, + opts, + preserve_sharing, + } = spec; + let tenant = TenantId(ctx.subject_tenant_id()); + let owner = OwnerId(ctx.subject_id()); + + // Fail fast if no plugin is available before touching metadata. + let plugin = self.plugins.resolve().await?; + + // When `sharing` was omitted (`preserve_sharing`), a value rotation + // targets the secret the owner would GET: a `private` secret is + // owner-scoped and wins over a coexisting non-private one, so rotate the + // private row if the caller has one under this reference; otherwise fall + // through to the non-private (tenant/shared) class. Without this an + // omitted-sharing PUT over a private-only reference would silently + // create a tenant row that GET never returns (the PUT 204s but the + // owner's GET still sees the old private value). + let sharing = if preserve_sharing + && self + .repo + .find_for_write( + &AccessScope::allow_all(), + tenant, + owner, + key, + SharingMode::Private, + ) + .await? + .is_some() + { + SharingMode::Private + } else { + sharing + }; + + // Prefetch the target row of this sharing class (own-tenant, keyed by + // tenant+owner+key+sharing) with `allow_all`; the single PDP evaluation + // runs once the secret's concrete type is known — on create from the + // requested/default type, on overwrite from the existing row (per + // AUTHZ_USAGE_SCENARIOS S10/S11). + if let Some(existing) = self + .repo + .find_for_write(&AccessScope::allow_all(), tenant, owner, key, sharing) + .await? + { + // Traits + PDP resource come from the registry resolution of the + // row's (immutable) type. + let resolved = self.resolve_stored(existing.secret_type_uuid).await?; + // Authorize on the concrete type BEFORE any 4xx that would reveal + // the row exists: an unauthorized caller must not be able to tell a + // create-only conflict (409) or a trait/immutability violation (400) + // apart from a plain denial (the own-tenant reference-enumeration + // oracle). The single PDP evaluation is gated on the caller's tenant. + let scope = self + .scope_for_timed( + ctx, + &authz::secret_type_resource(&resolved.gts_id), + actions::WRITE, + ) + .await?; + if !self.repo.scope_includes_tenant(&scope, tenant.0).await? { + self.metrics.cross_tenant_denied(); + return Err(DomainError::AccessDenied { cause: None }); + } + + if create_only { + return Err(DomainError::Conflict); + } + // A PUT that omitted `sharing` (`preserve_sharing`) keeps the stored + // mode, so a value rotation never silently narrows a `shared` secret + // back to `tenant` (review finding #8). An explicit `sharing` still + // takes effect. `sharing` remains the class selector used above. + let effective_sharing = if preserve_sharing { + existing.sharing + } else { + sharing + }; + // find_for_write addresses the target sharing class, so a write never + // crosses the private boundary here — a private and a tenant/shared + // secret coexist under one ref (per design §4.1). The guard stays as a + // defensive invariant; tenant↔shared is the only real in-place change. + if (existing.sharing == SharingMode::Private) + != (effective_sharing == SharingMode::Private) + { + return Err(DomainError::UnsupportedTransition { + detail: "cannot move between private and tenant/shared".into(), + }); + } + // The type is immutable: an explicit differing type is rejected; + // an absent one inherits the row's type for trait validation. + if let Some(requested) = opts.secret_type.as_ref() + && requested.to_uuid() != existing.secret_type_uuid + { + return Err(DomainError::TypeViolation { + field: "type", + reason: typing::reasons::TYPE_IMMUTABLE, + detail: format!( + "secret is of type '{}'; changing it to '{}' is not supported", + resolved.gts_id, requested + ), + }); + } + typing::validate_write( + &resolved.gts_id, + &resolved.traits, + effective_sharing, + &value, + opts.expires_at.requested(), + )?; + let expected_version = Self::precheck_version(precondition.as_ref(), &existing)?; + return self + .overwrite_existing(OverwriteArgs { + ctx, + scope: &scope, + plugin: &plugin, + key, + value, + sharing: effective_sharing, + existing_id: existing.id, + expected_version, + expires_at: opts.expires_at.resolve(existing.expires_at), + }) + .await; + } + + // Create path: the target does not exist. An If-Match precondition + // requires an existing target, so any precondition fails here. + // + // Note (own-tenant existence): this 409 lands before the create-path + // PDP evaluation below, so an unauthorized caller sending `If-Match` + // can tell "reference absent" (409 here) from "reference present" + // (the existing-row branch above runs PDP first → 403). That is the + // same own-tenant existence signal DELETE exposes by design (see the + // delete-path note); it is an accepted part of the module threat model + // (a tenant member may enumerate their own tenant's references), not a + // cross-tenant leak — the read path still folds cross-tenant existence + // into an anti-enumeration 404. The existing-row branch keeps PDP-first + // because there a 409/400 would otherwise distinguish a create-only + // conflict or a trait violation on a reference the caller cannot write. + if precondition.is_some() { + return Err(DomainError::VersionConflict); + } + + // Create path: the type defaults to generic; traits + per-type access + // validated before any side effect. The caller named the type, so an + // unresolvable one propagates as 400 UNKNOWN_SECRET_TYPE. + let type_uuid = opts + .secret_type + .map_or_else(|| SecretType::generic().uuid(), |id| id.to_uuid()); + let resolved = self.types.resolve(type_uuid).await?; + typing::validate_write( + &resolved.gts_id, + &resolved.traits, + sharing, + &value, + opts.expires_at.requested(), + )?; + // Single PDP evaluation on the concrete type being created, gated on the + // caller's tenant. + let scope = self + .scope_for_timed( + ctx, + &authz::secret_type_resource(&resolved.gts_id), + actions::WRITE, + ) + .await?; + if !self.repo.scope_includes_tenant(&scope, tenant.0).await? { + self.metrics.cross_tenant_denied(); + return Err(DomainError::AccessDenied { cause: None }); + } + + // Create saga: insert provisioning → plugin.put → mark_active. The + // fence fingerprint of the value this saga will write travels in the + // INSERT, so the row is fenced from its first readable instant. + let value_fp = self.stamp_fp(&plugin, &value).await?; + let id = Uuid::new_v4(); + let new = NewSecret { + id, + tenant_id: tenant, + reference: key.clone(), + sharing, + owner_id: owner, + secret_type_uuid: type_uuid, + expires_at: opts.expires_at.resolve(None), + value_fp, + fp_key_id: fence::CURRENT_FENCE_KEY_ID, + }; + match self.repo.insert_provisioning(&scope, &new).await { + Ok(()) => {} + Err(DomainError::Conflict) => { + // Lost the create race. A create-only POST surfaces the 409 + // directly; a PUT resolves to an update once the winner (whose + // provisioning row is invisible to find_for_write) marks active. + if create_only { + return Err(DomainError::Conflict); + } + for _ in 0..CREATE_RACE_MAX_RETRIES { + if let Some(existing) = self + .repo + .find_for_write(&scope, tenant, owner, key, sharing) + .await? + { + // Losing the race to a different type is a conflict on + // the immutable type, same as the explicit-mismatch path. + if existing.secret_type_uuid != type_uuid { + // Render the winner's type as its GTS id (like the + // overwrite path), not a raw UUID, so both sides of + // the message are in the same namespace. + let existing_resolved = + self.resolve_stored(existing.secret_type_uuid).await?; + return Err(DomainError::TypeViolation { + field: "type", + reason: typing::reasons::TYPE_IMMUTABLE, + detail: format!( + "secret is of type '{}'; changing it to '{}' is not supported", + existing_resolved.gts_id, resolved.gts_id + ), + }); + } + // Preserve the winner's sharing when the caller omitted + // it, exactly like the direct overwrite branch — else a + // rotation that lost the create race could still narrow + // a `shared` secret back to `tenant` (finding #8). + let effective_sharing = if preserve_sharing { + existing.sharing + } else { + sharing + }; + let expected_version = + Self::precheck_version(precondition.as_ref(), &existing)?; + return self + .overwrite_existing(OverwriteArgs { + ctx, + scope: &scope, + plugin: &plugin, + key, + value, + sharing: effective_sharing, + existing_id: existing.id, + expected_version, + expires_at: opts.expires_at.resolve(existing.expires_at), + }) + .await; + } + sleep(Duration::from_millis(CREATE_RACE_RETRY_BACKOFF_MS)).await; + } + tracing::warn!( + tenant = %tenant.0, + key = %key.as_ref(), + "credstore put: create-race retry budget exhausted; winner appears stuck mid-saga, returning retry-safe Conflict" + ); + return Err(DomainError::Conflict); + } + Err(e) => return Err(e), + } + + let plugin_owner = if sharing == SharingMode::Private { + Some(owner) + } else { + None + }; + let t0 = Instant::now(); + let result = plugin + .put(ctx, &tenant, key, value, plugin_owner.as_ref()) + .await + .map_err(map_plugin_err); + let secs = t0.elapsed().as_secs_f64(); + let outcome = if result.is_ok() { + Outcome::Success + } else { + Outcome::Error + }; + self.metrics + .dependency(Dep::Plugin, DepOp::PluginPut, outcome, secs); + if let Err(e) = result { + // Compensate the saga: the backend write failed, so roll back the + // provisioning row. The partial unique index ignores status, so a + // lingering provisioning row wedges the reference until the reaper + // runs — a misleading 409 for POST, exhausted retries for PUT. + // Best-effort: a failed cleanup just defers to the reaper. + let rollback = match self.repo.delete_by_id(&scope, id, None).await { + Ok(()) => Outcome::Success, + // Row already gone (concurrent reap/delete): not wedged. + Err(DomainError::NotFound) => Outcome::NotFound, + Err(cleanup_err) => { + tracing::warn!( + tenant = %tenant.0, + key = %key.as_ref(), + err = %cleanup_err, + "credstore put: provisioning rollback after backend write failure failed; reference stays wedged until reaped" + ); + Outcome::Error + } + }; + self.metrics.provisioning_rollback(rollback); + return Err(e); + } + + // Transient orphaned backend value: a mark_active failure after a + // successful plugin.put leaves the value in the backend while the row + // stays provisioning. It is never readable (no active row) — a storage + // artifact, not a disclosure — and it is self-healing: a client retry + // overwrites it, and absent a retry the reaper reaps the stale + // provisioning row *and* issues a best-effort plugin.delete for it (see + // `reap_row` → `reap_backend_delete`), so the value is reconciled after + // `provisioning_timeout_secs` + one reaper tick. Surfaced operationally + // via the warn below and the provisioning_reaped metric. + if let Err(e) = self.repo.mark_active(&scope, id).await { + tracing::warn!( + tenant = %tenant.0, + key = %key.as_ref(), + err = %e, + "credstore put: mark_active failed after backend write; backend value orphaned until reaped/retried" + ); + return Err(e); + } + Ok(()) + } + + /// Optimistic-concurrency pre-check before any backend write: returns the + /// `version = ?` filter to gate `touch` on, or `VersionConflict` if the + /// caller's `If-Match` validator disagrees with the current row. The + /// validator is generation-bound (`"."`): a mismatched row + /// id means the caller's validator is from a deleted-and-recreated + /// secret's earlier generation and must never match, even when the + /// per-generation version counters coincide (no ABA). The row-level CAS + /// keeps gating on `version` alone — `id` is already the UPDATE key. + fn precheck_version( + precondition: Option<&WritePrecondition>, + existing: &SecretRow, + ) -> Result, DomainError> { + match precondition { + Some(WritePrecondition::Version { id, version }) => { + if *id != existing.id || *version != existing.version { + return Err(DomainError::VersionConflict); + } + Ok(Some(*version)) + } + // Multi-valued If-Match: satisfied if any listed validator matches + // the current row's (id, version). The CAS still gates on the row's + // own version (the matched one equals `existing.version`). + Some(WritePrecondition::AnyVersion(validators)) => { + if validators + .iter() + .any(|(id, version)| *id == existing.id && *version == existing.version) + { + Ok(Some(existing.version)) + } else { + Err(DomainError::VersionConflict) + } + } + Some(WritePrecondition::Exists) | None => Ok(None), + } + } + + /// Overwrite an existing secret. The ordering of the backend write vs the + /// metadata version bump depends on whether the caller supplied `If-Match`; + /// both orders stamp the fence fingerprint in the same `touch`, so any + /// half-completed overwrite reads fail-closed instead of serving a value + /// under metadata written for a different one: + /// + /// * **No precondition** (last-writer-wins): backend-first — `plugin.put` + /// then a `touch` version bump. A plugin failure leaves metadata + /// untouched (the previous value still verifies); a `touch` failure + /// after a committed backend write leaves the row fingerprinting the + /// previous value, so reads 404 (fence mismatch) until the next put + /// retries. Two crosswise concurrent LWW puts can end with one writer's + /// value under the other's row — the fence turns that from a + /// cross-tenant disclosure into a fail-closed 404 healed by any + /// subsequent successful put. + /// * **`If-Match`** (optimistic concurrency): version-claim-first — the + /// version-gated `touch` runs BEFORE `plugin.put`, so a losing concurrent + /// writer (`touch` matches 0 rows) never reaches the backend and cannot + /// clobber the winner's value. The tradeoff inverts: if the backend write + /// then fails, the row already fingerprints the value that never landed, + /// so reads 404 (fence mismatch, fail-closed — not the previous value) + /// until the caller retries against the new version. + /// + /// Shared by the normal overwrite path and the create-race resolution path. + #[allow(clippy::cognitive_complexity)] + async fn overwrite_existing(&self, args: OverwriteArgs<'_>) -> Result<(), DomainError> { + let OverwriteArgs { + ctx, + scope, + plugin, + key, + value, + sharing, + existing_id, + expected_version, + expires_at, + } = args; + let tenant = TenantId(ctx.subject_tenant_id()); + let owner = OwnerId(ctx.subject_id()); + let plugin_owner = if sharing == SharingMode::Private { + Some(owner) + } else { + None + }; + + // Fence stamp of the value this overwrite writes: `touch` persists it + // in the same atomic UPDATE as the sharing label, so metadata and + // fingerprint always come from one writer. On any interleaving where + // the backend value ends up from a different writer than the row, the + // read-side verification fails closed instead of serving the value + // under foreign metadata. + let value_fp = self.stamp_fp(plugin, &value).await?; + + // If-Match: claim the version bump first (see the fn doc). A 0-row + // `touch` means the version moved or the row vanished — the caller lost + // the CAS and, crucially, has NOT written the backend, so a concurrent + // writer's value can never be clobbered. + if expected_version.is_some() { + match self + .repo + .touch( + scope, + existing_id, + sharing, + expected_version, + expires_at, + value_fp, + ) + .await + { + Ok(Some(_)) => {} + Ok(None) => { + tracing::warn!( + tenant = %tenant.0, + key = %key.as_ref(), + "credstore put: If-Match version precondition lost before the backend write (concurrent write/delete); no backend value written" + ); + return Err(DomainError::VersionConflict); + } + Err(e) => return Err(e), + } + let t0 = Instant::now(); + let result = plugin + .put(ctx, &tenant, key, value, plugin_owner.as_ref()) + .await + .map_err(map_plugin_err); + let secs = t0.elapsed().as_secs_f64(); + self.metrics.dependency( + Dep::Plugin, + DepOp::PluginPut, + if result.is_ok() { + Outcome::Success + } else { + Outcome::Error + }, + secs, + ); + if let Err(e) = &result { + tracing::warn!( + tenant = %tenant.0, + key = %key.as_ref(), + err = %e, + "credstore put: backend write failed after the version was claimed; reads fail closed (fence mismatch) until a retried put lands the value" + ); + } + return result; + } + + // No precondition: last-writer-wins, backend-first. + let t0 = Instant::now(); + let result = plugin + .put(ctx, &tenant, key, value, plugin_owner.as_ref()) + .await + .map_err(map_plugin_err); + let secs = t0.elapsed().as_secs_f64(); + let outcome = if result.is_ok() { + Outcome::Success + } else { + Outcome::Error + }; + self.metrics + .dependency(Dep::Plugin, DepOp::PluginPut, outcome, secs); + result?; + + // Version bump second. A failure here (after the backend write committed) + // leaves the row fingerprinting the previous value, so reads fail closed + // (fence mismatch → 404) until the next put retries. + match self + .repo + .touch( + scope, + existing_id, + sharing, + expected_version, + expires_at, + value_fp, + ) + .await + { + Ok(Some(_)) => Ok(()), + // Row concurrently deleted/reaped: the backend write already committed + // but no active metadata row exists, so the secret would be unreadable + // despite an acknowledged write. Surface the lost race as a retryable + // conflict (canonical Aborted/409) so the caller re-runs the put, which + // recreates the metadata row deterministically. + // + // Orphaned backend value (accepted, review finding #3): the value + // this put wrote now sits in the backend under `(tenant, ref[, + // owner])` with no metadata row. It is NOT readable (resolution + // needs a row) and it is NOT a disclosure, but if the caller does + // not retry it lingers as plaintext-at-rest until a future create + // of the same reference overwrites it — the reaper works off rows, + // so it never sees a row-less backend value. A compensating delete + // here is unsafe (it could erase a concurrent successor's value at + // the same backend key), so this is accept-and-document: the retry + // (the common case) reconciles it; the no-retry orphan is bounded + // by the next create at that reference. + Ok(None) => { + tracing::warn!( + tenant = %tenant.0, + key = %key.as_ref(), + "credstore put: row vanished before version bump (concurrent delete/reap); returning conflict so the caller retries" + ); + Err(DomainError::VersionConflict) + } + Err(e) => { + tracing::warn!( + tenant = %tenant.0, + key = %key.as_ref(), + err = %e, + "credstore put: version bump (touch) failed after backend write; version/sharing metadata lags the backend until the next put retries" + ); + Err(e) + } + } + } + + /// Delete an owned secret via the deprovisioning saga: + /// mark `deprovisioning` (the secret atomically stops resolving, the row + /// keeps holding the unique index) → backend delete → row delete. + /// + /// A retry of `DELETE` resumes a stuck saga: `find_own` returns the + /// `deprovisioning` row and the backend/row steps re-run idempotently. + /// Absent a retry, the reaper completes the saga (`reap_and_refresh`). + /// + /// # Errors + /// + /// Returns [`DomainError::NotFound`] if no own-tenant row exists for the key. + /// Returns [`DomainError::VersionConflict`] if `precondition` is set and the + /// current version does not satisfy it. + // Saga orchestration (scope -> gate -> mark -> backend -> commit) is + // inherently branchy; kept as one function for readability of the flow. + #[allow(clippy::cognitive_complexity)] + pub async fn delete( + &self, + ctx: &SecurityContext, + key: &SecretRef, + precondition: Option, + ) -> Result<(), DomainError> { + let tenant = TenantId(ctx.subject_tenant_id()); + let owner = OwnerId(ctx.subject_id()); + + // Prefetch the caller's own row (allow_all, keyed by tenant+owner+key); + // a missing row is 404 without consulting the PDP. + let Some(row) = self + .repo + .find_own(&AccessScope::allow_all(), tenant, owner, key) + .await? + else { + return Err(DomainError::NotFound); + }; + + // Optimistic-concurrency gate, in two layers mirroring the put path. + // `Exists` is satisfied by find_own; a `Version`/`AnyVersion` + // precondition becomes: + // 1. a generation-bound pre-check here, before any side effect — + // both the row id (a validator minted for a recreated secret's + // earlier generation must never match, no ABA) and the version; + // 2. the same `version = ?` filter on mark_deprovisioning below + // (mark_deprovisioning does not bump the version, so a saga + // resume still sees the version the client knew; the id is + // already the UPDATE key). + let expected_version = Self::precheck_version(precondition.as_ref(), &row)?; + + // Single PDP evaluation on the secret's full concrete type, gated on the + // caller's tenant. Delete reveals reference existence within the caller's + // own tenant — the 404-before-PDP (no row) vs 403-after-PDP (row present, + // denied) split is an own-tenant existence signal available to any member + // of the tenant (find_own matches any subject for tenant/shared rows, not + // only the private owner). This is an accepted part of the module's + // threat model (a tenant member may enumerate their own tenant's + // references); cross-tenant existence stays hidden by the anti-enumeration + // 404 on the read path. A denial is therefore a plain operation-level 403. + let resolved = self.resolve_stored(row.secret_type_uuid).await?; + let scope = self + .scope_for_timed( + ctx, + &authz::secret_type_resource(&resolved.gts_id), + actions::DELETE, + ) + .await?; + if !self.repo.scope_includes_tenant(&scope, tenant.0).await? { + self.metrics.cross_tenant_denied(); + return Err(DomainError::AccessDenied { cause: None }); + } + + // Fail fast if no plugin is available BEFORE marking deprovisioning — + // symmetric with the put saga (which resolves the plugin before + // touching metadata). Otherwise a misconfigured/absent plugin would + // flip the row to deprovisioning (the secret instantly stops + // resolving) and only then 503, wedging the reference: the caller is + // told the delete failed, yet it visibly took effect, and the reaper + // cannot finish it either (no plugin to reconcile the backend). + let plugin = self.plugins.resolve().await?; + + // Step 1 — mark deprovisioning (skip when resuming a stuck saga). + // From this instant the secret is invisible to resolution while the + // row keeps holding the partial unique index, so a concurrent create + // of the same reference stays a retryable Conflict until cleanup + // finishes — releasing the name earlier would let this saga's lagging + // backend delete erase the new secret's value (same backend key). + if row.status == SecretStatus::Active + && !self + .repo + .mark_deprovisioning(&scope, row.id, expected_version) + .await? + { + // 0 rows: the row moved or vanished between find_own and the gated + // flip. Under a version precondition that is an optimistic-lock + // conflict; otherwise a concurrent delete won — report NotFound, + // matching the pre-saga behavior. + return if expected_version.is_some() { + Err(DomainError::VersionConflict) + } else { + Err(DomainError::NotFound) + }; + } + + // Step 2 — backend delete. A missing backend value is success + // (idempotent). A failure leaves the deprovisioning row for a client + // retry or the reaper; the caller sees a retryable error while the + // secret already no longer resolves. + let plugin_owner = if row.sharing == SharingMode::Private { + Some(row.owner_id) + } else { + None + }; + let t0 = Instant::now(); + let plugin_result = plugin + .delete(ctx, &tenant, key, plugin_owner.as_ref()) + .await; + let secs = t0.elapsed().as_secs_f64(); + let plugin_result = match plugin_result { + Ok(()) | Err(CredStoreError::NotFound) => Ok(()), + Err(e) => Err(map_plugin_err(e)), + }; + self.metrics.dependency( + Dep::Plugin, + DepOp::PluginDelete, + if plugin_result.is_ok() { + Outcome::Success + } else { + Outcome::Error + }, + secs, + ); + if let Err(e) = plugin_result { + tracing::warn!( + tenant = %tenant.0, + key = %key.as_ref(), + err = %e, + "credstore delete: backend delete failed; row stays deprovisioning until retried or reaped" + ); + return Err(e); + } + + // Step 3 — remove the metadata row, releasing the reference. The mark + // was the version-gated step, so this delete is unconditional; a row + // already gone (concurrent resume/reap finished first) is success. + match self.repo.delete_by_id(&scope, row.id, None).await { + Ok(()) | Err(DomainError::NotFound) => Ok(()), + Err(e) => { + tracing::warn!( + tenant = %tenant.0, + key = %key.as_ref(), + err = %e, + "credstore delete: row delete after backend cleanup failed; reference stays held until retried or reaped" + ); + Err(e) + } + } + } + + /// Complete stale saga rows and refresh inventory gauges. + /// + /// Called by the reaper loop on a periodic tick. For every stale + /// non-active row a best-effort backend delete reconciles the value store + /// (closing the create-saga orphaned-value debt), then: + /// - `provisioning` rows are removed unconditionally (the write never + /// became visible; the reference must not stay wedged); + /// - `deprovisioning` rows are removed only after a successful backend + /// delete — otherwise the row (and the reference) is kept for the next + /// tick, because releasing the name before backend cleanup could let + /// this saga erase a successor secret's value. + /// + /// Errors are logged, not propagated. + pub async fn reap_and_refresh(&self) { + self.expire_secrets().await; + self.reap_stale().await; + self.backfill_unfenced().await; + match self.repo.inventory().await { + Ok(counts) => self.metrics.record_inventory(counts), + Err(e) => { + tracing::warn!(err = %e, "inventory failed"); + } + } + } + + /// Fence-backfill sweep: stamp out-of-band seeded rows (`value_fp IS + /// NULL`) that nobody reads, so the fleet converges to fully fenced + /// without traffic (feature spec `algo-reaper-fp-sweep`). One bounded + /// batch per tick; every failure logs and continues — the reaper must + /// survive transient backend/DB errors, and an unstamped row is only ever + /// served on trust, never wrongly rejected. + // Per-row: read value from backend -> stamp -> CAS; branchy but one flow. + #[allow(clippy::cognitive_complexity)] + async fn backfill_unfenced(&self) { + let rows = match self.repo.list_unfenced(REAP_BATCH_LIMIT).await { + Ok(rows) => rows, + Err(e) => { + tracing::warn!(err = %e, "reaper: list_unfenced failed"); + return; + } + }; + if rows.is_empty() { + return; + } + let plugin = match self.plugins.resolve().await { + Ok(p) => p, + Err(e) => { + tracing::warn!(err = %e, "reaper: no plugin for fence backfill this tick"); + return; + } + }; + for row in rows { + let Some((ctx, key)) = Self::reaper_ctx_and_key(&row) else { + continue; + }; + let owner = if row.sharing == SharingMode::Private { + Some(row.owner_id) + } else { + None + }; + let value = match plugin.get(&ctx, &row.tenant_id, &key, owner.as_ref()).await { + Ok(Some(v)) => v, + // No backend value: nothing to stamp (the row 404s on read + // anyway); leave it for a future seed/put. + Ok(None) => continue, + Err(e) => { + tracing::warn!(id = %row.id, err = %e, "reaper: fence backfill read failed"); + continue; + } + }; + self.backfill_row_fp(&plugin, &row, &value).await; + } + } + + /// Move expired active rows into the ordinary deprovisioning saga: + /// invisible immediately, backend value + row cleaned by the pending sweep. + async fn expire_secrets(&self) { + match self.repo.mark_expired_deprovisioning().await { + Ok(n) if n > 0 => { + tracing::info!(count = n, "reaper: expired secrets marked deprovisioning"); + } + Ok(_) => {} + Err(e) => { + tracing::warn!(err = %e, "mark_expired_deprovisioning failed"); + } + } + } + + /// Reap one batch of stale saga rows and emit the reap counters. + async fn reap_stale(&self) { + let stale = self.list_stale().await; + if stale.is_empty() { + return; + } + + // Backend reconciliation needs the plugin; without one, still remove + // provisioning rows (pre-reconciliation behavior — un-wedge the + // reference) but keep deprovisioning rows for the next tick. + let plugin = match self.plugins.resolve().await { + Ok(p) => Some(p), + Err(e) => { + tracing::warn!(err = %e, "reaper: no plugin for backend reconciliation this tick"); + None + } + }; + + let (prov_reaped, deprov_reaped) = self.reap_batch(plugin.as_ref(), stale).await; + if prov_reaped > 0 { + self.metrics.provisioning_reaped(prov_reaped); + } + if deprov_reaped > 0 { + self.metrics.deprovisioning_reaped(deprov_reaped); + } + } + + /// One bounded batch of stale saga rows, or empty (with a warning) when + /// the listing itself fails — the reaper must survive transient DB errors. + async fn list_stale(&self) -> Vec { + self.repo + .list_stale_pending( + self.reaper.provisioning_timeout_secs, + self.reaper.deprovisioning_timeout_secs, + REAP_BATCH_LIMIT, + ) + .await + .unwrap_or_else(|e| { + tracing::warn!(err = %e, "list_stale_pending failed"); + Vec::new() + }) + } + + /// Reap every row in the batch; returns + /// `(provisioning_reaped, deprovisioning_reaped)`. + async fn reap_batch( + &self, + plugin: Option<&Arc>, + stale: Vec, + ) -> (u64, u64) { + let mut prov_reaped = 0u64; + let mut deprov_reaped = 0u64; + for row in stale { + match self.reap_row(plugin, &row).await { + Some(SecretStatus::Provisioning) => prov_reaped += 1, + Some(SecretStatus::Deprovisioning) => deprov_reaped += 1, + _ => {} + } + } + (prov_reaped, deprov_reaped) + } + + /// Reap a single stale saga row. Ordering differs by status so the reaper + /// can never delete a value out from under a create that has just + /// succeeded: + /// + /// * `provisioning` — claim the row with a status-gated delete *first*. The + /// delete loses to a concurrent `mark_active` (the slow create finally + /// landing): 0 rows removed means the saga won the race and the now-active + /// secret, plus its backend value, must be left alone. Only once we own + /// the row do we issue the best-effort backend cleanup — so a reported + /// create success can never lose its row and backend value to the reaper. + /// * `deprovisioning` — the reference name is held until the backend value + /// is gone, so the backend delete precedes the (still status-gated) row + /// delete. + /// + /// Returns the row's status when this call actually removed it. + async fn reap_row( + &self, + plugin: Option<&Arc>, + row: &SecretRow, + ) -> Option { + match row.status { + SecretStatus::Provisioning => { + // Claim first: if the status-gated delete loses to a concurrent + // mark_active/delete, leave the row (and its value) alone. + if !self.reap_delete_gated(row).await { + return None; + } + if let Some(p) = plugin { + self.reap_backend_delete(p, row).await; + } + Some(SecretStatus::Provisioning) + } + SecretStatus::Deprovisioning => { + let backend_deleted = match plugin { + Some(p) => self.reap_backend_delete(p, row).await, + None => false, + }; + if !backend_deleted { + return None; + } + self.reap_delete_gated(row) + .await + .then_some(SecretStatus::Deprovisioning) + } + // list_stale_pending never returns active rows; ignore defensively. + SecretStatus::Active => None, + } + } + + /// Status-gated row delete for the reaper: removes the row only while it + /// still holds the status the reaper observed, swallowing (but logging) a + /// delete error. Returns whether this call actually removed the row. + async fn reap_delete_gated(&self, row: &SecretRow) -> bool { + match self.repo.reap_by_id(row.id, row.status).await { + Ok(removed) => removed, + Err(e) => { + tracing::warn!(id = %row.id, status = ?row.status, err = %e, "reaper: row delete failed"); + false + } + } + } + + /// Best-effort backend delete for a reaped row. Returns `true` when the + /// backend no longer holds the value (deleted or was never there). + async fn reap_backend_delete( + &self, + plugin: &Arc, + row: &SecretRow, + ) -> bool { + let Some((ctx, key)) = Self::reaper_ctx_and_key(row) else { + return false; + }; + let owner = if row.sharing == SharingMode::Private { + Some(row.owner_id) + } else { + None + }; + let t0 = Instant::now(); + let result = plugin + .delete(&ctx, &row.tenant_id, &key, owner.as_ref()) + .await; + let secs = t0.elapsed().as_secs_f64(); + let (outcome, deleted) = match result { + Ok(()) => (Outcome::Success, true), + Err(CredStoreError::NotFound) => (Outcome::NotFound, true), + Err(e) => { + tracing::warn!( + id = %row.id, + tenant = %row.tenant_id.0, + err = %e, + "reaper: backend delete failed" + ); + (Outcome::Error, false) + } + }; + self.metrics + .dependency(Dep::Plugin, DepOp::PluginDelete, outcome, secs); + deleted + } + + /// The reaper has no caller: build a minimal service context for the row's + /// tenant plus the validated reference. Plugins are pure value stores keyed + /// by the explicit tenant/key/owner arguments and must not rely on caller + /// identity. `None` (with a warning) on the never-expected invalid row. + fn reaper_ctx_and_key(row: &SecretRow) -> Option<(SecurityContext, SecretRef)> { + let ctx = match SecurityContext::builder() + .subject_id(Uuid::nil()) + .subject_tenant_id(row.tenant_id.0) + .build() + { + Ok(ctx) => ctx, + Err(e) => { + tracing::warn!(id = %row.id, err = %e, "reaper: failed to build service context"); + return None; + } + }; + // The DB CHECK guarantees the stored reference is well-formed. + match SecretRef::new(row.reference.clone()) { + Ok(key) => Some((ctx, key)), + Err(e) => { + tracing::warn!(id = %row.id, err = %e, "reaper: stored reference is invalid"); + None + } + } + } +} + +#[cfg(test)] +#[path = "service_tests.rs"] +mod service_tests; diff --git a/gears/credstore/credstore/src/domain/secret/service_tests.rs b/gears/credstore/credstore/src/domain/secret/service_tests.rs new file mode 100644 index 000000000..df9259a0d --- /dev/null +++ b/gears/credstore/credstore/src/domain/secret/service_tests.rs @@ -0,0 +1,3418 @@ +//! Unit tests for [`Service`] using domain fakes. + +use std::sync::Arc; + +use authz_resolver_sdk::PolicyEnforcer; +use credstore_sdk::{ + CredStorePluginClientV1, OwnerId, SecretRef, SecretType, SecretValue, SharingMode, TenantId, +}; +use uuid::Uuid; + +use crate::domain::error::DomainError; +use crate::domain::secret::model::{SecretRow, SecretStatus, WriteSpec}; +use crate::domain::secret::repo::SecretRepo; +use crate::domain::secret::service::{ReaperSettings, Service}; +use toolkit_security::AccessScope; + +use crate::domain::secret::test_support::{ + FakeDir, FakeMetrics, FakePlugin, FakePluginSelector, FakeSecretRepo, NoopMetrics, + catalog_type_resolver, deny_enforcer, failing_enforcer, make_ctx, mock_enforcer, +}; + +fn key(s: &str) -> SecretRef { + SecretRef::new(s).expect("valid key") +} + +fn make_service( + repo: Arc, + dir: Arc, + enforcer: PolicyEnforcer, + plugin: Arc, + metrics: Arc, +) -> Service { + let selector = Arc::new(FakePluginSelector::new(plugin)); + Service::new( + repo, + dir, + enforcer, + selector, + catalog_type_resolver(), + metrics, + test_reaper_settings(), + ) +} + +fn test_reaper_settings() -> ReaperSettings { + ReaperSettings { + tick_secs: 60, + provisioning_timeout_secs: 300, + deprovisioning_timeout_secs: 300, + } +} + +fn make_service_noop( + repo: Arc, + dir: Arc, + enforcer: PolicyEnforcer, + plugin: Arc, +) -> Service { + let selector = Arc::new(FakePluginSelector::new(plugin)); + Service::new( + repo, + dir, + enforcer, + selector, + catalog_type_resolver(), + Arc::new(NoopMetrics), + test_reaper_settings(), + ) +} + +// ───────────────────────────────────────────────────────────────────────────── + +#[tokio::test] +async fn get_own_tenant_secret_returns_hit_own() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("mykey"); + + let plugin = FakePlugin::new(); + let repo = Arc::new(FakeSecretRepo::new()); + let metrics = FakeMetrics::new(); + + // Pre-seed the plugin with a value. + let plugin_key_owner: Option<&OwnerId> = None; + plugin + .put( + &ctx, + &TenantId(tenant), + &k, + SecretValue::from("v1"), + plugin_key_owner, + ) + .await + .expect("put"); + + // Seed repo with active row. + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "mykey".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + + let svc = make_service( + repo, + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin, + metrics.clone(), + ); + + let result = svc.get(&ctx, &k).await.expect("get ok"); + assert!(result.is_some()); + let resp = result.unwrap(); + assert!(!resp.is_inherited); + assert_eq!(resp.owner_tenant_id.0, tenant); + assert_eq!( + metrics.last_read_outcome(), + Some(crate::domain::ports::metrics::ReadOutcome::HitOwn) + ); +} + +#[tokio::test] +async fn get_records_pdp_dependency_metric() { + use crate::domain::ports::metrics::{Dep, DepOp, Outcome}; + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("mykey"); + + let plugin = FakePlugin::new(); + let repo = Arc::new(FakeSecretRepo::new()); + let metrics = FakeMetrics::new(); + plugin + .put(&ctx, &TenantId(tenant), &k, SecretValue::from("v1"), None) + .await + .expect("put"); + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "mykey".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + let svc = make_service( + repo, + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin, + metrics.clone(), + ); + + svc.get(&ctx, &k).await.expect("get ok"); + + assert!( + metrics + .deps() + .contains(&(Dep::Pdp, DepOp::Evaluate, Outcome::Success)), + "get must record a PDP dependency metric, got {:?}", + metrics.deps() + ); +} + +#[tokio::test] +async fn put_records_pdp_dependency_metric() { + use crate::domain::ports::metrics::{Dep, DepOp, Outcome}; + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("putkey"); + + let metrics = FakeMetrics::new(); + let svc = make_service( + Arc::new(FakeSecretRepo::new()), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + metrics.clone(), + ); + + svc.put( + &ctx, + &k, + SecretValue::from("v"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("put ok"); + + assert!( + metrics + .deps() + .contains(&(Dep::Pdp, DepOp::Evaluate, Outcome::Success)), + "put must record a PDP dependency metric, got {:?}", + metrics.deps() + ); +} + +#[tokio::test] +async fn delete_records_pdp_dependency_metric() { + use crate::domain::ports::metrics::{Dep, DepOp, Outcome}; + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("delkey"); + + let metrics = FakeMetrics::new(); + // The PDP is consulted only for a resolved row (prefetch model), so seed one. + let repo = Arc::new(FakeSecretRepo::new()); + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "delkey".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + let svc = make_service( + repo, + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + metrics.clone(), + ); + + svc.delete(&ctx, &k, None).await.expect("delete"); + + assert!( + metrics + .deps() + .contains(&(Dep::Pdp, DepOp::Evaluate, Outcome::Success)), + "delete must record a PDP dependency metric, got {:?}", + metrics.deps() + ); +} + +#[tokio::test] +async fn get_inherited_shared_from_parent_sets_is_inherited() { + let child = Uuid::new_v4(); + let parent = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, child); + let k = key("shared-key"); + + let plugin = FakePlugin::new(); + let repo = Arc::new(FakeSecretRepo::new()); + + // Seed plugin with value at parent tenant, no owner (shared). + plugin + .put( + &ctx, + &TenantId(parent), + &k, + SecretValue::from("parent-val"), + None, + ) + .await + .expect("put"); + + // Seed repo: parent has a shared active row. + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(parent), + reference: "shared-key".into(), + sharing: SharingMode::Shared, + owner_id: OwnerId(Uuid::nil()), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + + // chain: [child, parent] + let dir = Arc::new(FakeDir::new(vec![child, parent])); + let svc = make_service_noop(repo, dir, mock_enforcer(), plugin); + + let result = svc.get(&ctx, &k).await.expect("get ok"); + assert!(result.is_some()); + let resp = result.unwrap(); + assert!(resp.is_inherited, "must be inherited from parent"); + assert_eq!(resp.owner_tenant_id.0, parent); + assert_eq!(resp.sharing, SharingMode::Shared); +} + +#[tokio::test] +async fn get_tenant_mode_not_inherited_by_child() { + let child = Uuid::new_v4(); + let parent = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, child); + let k = key("tenant-key"); + + let plugin = FakePlugin::new(); + let repo = Arc::new(FakeSecretRepo::new()); + + // Parent has a tenant-mode row; tenant-mode is NOT inherited by children. + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(parent), + reference: "tenant-key".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(Uuid::nil()), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + + let dir = Arc::new(FakeDir::new(vec![child, parent])); + let svc = make_service_noop(repo, dir, mock_enforcer(), plugin); + + let result = svc.get(&ctx, &k).await.expect("get ok"); + // Tenant-mode secret at parent → not visible from child → Miss. + assert!(result.is_none()); +} + +#[tokio::test] +async fn get_private_owner_match_only() { + let tenant = Uuid::new_v4(); + let owner_a = Uuid::new_v4(); + let owner_b = Uuid::new_v4(); + // Subject is B; row belongs to A (private). + let ctx = make_ctx(owner_b, tenant); + let k = key("private-key"); + + let plugin = FakePlugin::new(); + let repo = Arc::new(FakeSecretRepo::new()); + + // A has a private row. + let row_id = Uuid::new_v4(); + repo.seed(SecretRow { + id: row_id, + tenant_id: TenantId(tenant), + reference: "private-key".into(), + sharing: SharingMode::Private, + owner_id: OwnerId(owner_a), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + // Seed plugin for A. + plugin + .put( + &make_ctx(owner_a, tenant), + &TenantId(tenant), + &k, + SecretValue::from("secret-a"), + Some(&OwnerId(owner_a)), + ) + .await + .expect("put"); + + let svc = make_service_noop( + repo, + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin, + ); + + // B can't see A's private secret. + let result = svc.get(&ctx, &k).await.expect("get ok"); + assert!( + result.is_none(), + "private row for A must not be visible to B" + ); +} + +#[tokio::test] +async fn get_shadowing_private_beats_inherited() { + let child = Uuid::new_v4(); + let parent = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, child); + let k = key("shadow-key"); + + let plugin = FakePlugin::new(); + let repo = Arc::new(FakeSecretRepo::new()); + + // Child has a private row for subject. + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(child), + reference: "shadow-key".into(), + sharing: SharingMode::Private, + owner_id: OwnerId(subject), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + // Parent has a shared row. + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(parent), + reference: "shadow-key".into(), + sharing: SharingMode::Shared, + owner_id: OwnerId(Uuid::nil()), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + + // Seed plugin: child private, parent shared. + plugin + .put( + &ctx, + &TenantId(child), + &k, + SecretValue::from("child-private"), + Some(&OwnerId(subject)), + ) + .await + .expect("put child private"); + plugin + .put( + &ctx, + &TenantId(parent), + &k, + SecretValue::from("parent-shared"), + None, + ) + .await + .expect("put parent shared"); + + let dir = Arc::new(FakeDir::new(vec![child, parent])); + let svc = make_service_noop(repo, dir, mock_enforcer(), plugin); + + let result = svc.get(&ctx, &k).await.expect("get ok"); + assert!(result.is_some()); + let resp = result.unwrap(); + assert!( + !resp.is_inherited, + "child-private must shadow parent-shared" + ); + assert_eq!(resp.owner_tenant_id.0, child); + assert_eq!(resp.value.as_bytes(), b"child-private"); +} + +#[tokio::test] +async fn put_create_conflict_returns_conflict() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("existing-key"); + + let plugin = FakePlugin::new(); + let repo = Arc::new(FakeSecretRepo::new()); + + // Seed existing active row. + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "existing-key".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + + let svc = make_service_noop( + repo, + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin, + ); + + let result = svc + .put( + &ctx, + &k, + SecretValue::from("val"), + WriteSpec::create(SharingMode::Tenant), + ) + .await; + assert!(matches!(result, Err(DomainError::Conflict))); +} + +#[tokio::test] +async fn put_shared_coexists_with_private() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("priv-key"); + + let plugin = FakePlugin::new(); + let repo = Arc::new(FakeSecretRepo::new()); + + // Existing private row. + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "priv-key".into(), + sharing: SharingMode::Private, + owner_id: OwnerId(subject), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + + let svc = make_service_noop( + repo.clone(), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin, + ); + + // PUT Shared for the same ref CREATES a coexisting non-private secret — it + // does not "transition" the private one. Per design §4.1 a tenant/shared and + // a private secret coexist under one reference (distinct partial-unique keys). + let result = svc + .put( + &ctx, + &k, + SecretValue::from("val"), + WriteSpec::upsert(SharingMode::Shared), + ) + .await; + assert!(result.is_ok(), "expected coexistence Ok, got {result:?}"); + + let counts = repo.inventory().await.expect("inventory"); + assert_eq!(counts.private, 1, "private secret must remain"); + assert_eq!(counts.shared, 1, "shared secret created alongside private"); +} + +#[tokio::test] +async fn put_omitting_sharing_preserves_existing_shared_mode() { + // A value rotation that omits `sharing` (handler sets preserve_sharing) + // must NOT narrow a `shared` secret back to `tenant` (review finding #8). + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("rotate-me"); + + let plugin = FakePlugin::new(); + plugin.seed_fence_key(); + let repo = Arc::new(FakeSecretRepo::new()); + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "rotate-me".into(), + sharing: SharingMode::Shared, + owner_id: OwnerId(Uuid::nil()), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + + let svc = make_service_noop( + repo.clone(), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin, + ); + + // Rotate the value with sharing omitted: class default `Tenant`, but + // `preserve_sharing(true)`. The old default-narrowing bug flipped Shared + // to Tenant here. + svc.put( + &ctx, + &k, + SecretValue::from("rotated"), + WriteSpec::upsert(SharingMode::Tenant).preserve_sharing(true), + ) + .await + .expect("rotate ok"); + + let rows = repo.rows(); + let row = rows + .iter() + .find(|r| r.reference == "rotate-me") + .expect("row present"); + assert_eq!( + row.sharing, + SharingMode::Shared, + "omitted sharing must preserve the stored Shared mode, not narrow to Tenant" + ); +} + +#[tokio::test] +async fn put_omitting_sharing_rotates_existing_private_secret() { + // A rotation with omitted sharing on a reference that only has the caller's + // PRIVATE secret must rotate THAT secret, not silently create a tenant row + // GET never returns (the PUT would 204 while GET still shows the old value). + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("rotate-priv"); + + let plugin = FakePlugin::new(); + plugin.seed_fence_key(); + let repo = Arc::new(FakeSecretRepo::new()); + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "rotate-priv".into(), + sharing: SharingMode::Private, + owner_id: OwnerId(subject), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + + let svc = make_service_noop( + repo.clone(), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin.clone(), + ); + + // Handler translates an omitted `sharing` to (Tenant default, preserve). + svc.put( + &ctx, + &k, + SecretValue::from("rotated"), + WriteSpec::upsert(SharingMode::Tenant).preserve_sharing(true), + ) + .await + .expect("rotate private ok"); + + let counts = repo.inventory().await.expect("inventory"); + assert_eq!(counts.private, 1, "the private secret is rotated in place"); + assert_eq!( + counts.tenant, 0, + "no tenant row is created by an omitted-sharing rotation" + ); + // The new value landed under the private class, not the tenant class. + assert!( + plugin.contains(&TenantId(tenant), &k, Some(&OwnerId(subject))), + "rotated value written to the private key class" + ); + assert!( + !plugin.contains(&TenantId(tenant), &k, None), + "no value written to the tenant key class" + ); +} + +/// Regression (fence-key cache thrash): a permanently-poisoned row (a +/// by-design, fail-closed fingerprint mismatch, e.g. from a crosswise LWW +/// interleave) must NOT re-read the fence key from the backend on every get. +/// The mismatch-triggered refresh is cooldown-gated and reuses the cache, so +/// repeated poisoned reads collapse to a single cold load rather than a backend +/// round-trip (and a global cache eviction) per request. +#[tokio::test] +async fn poisoned_reads_do_not_rethread_the_fence_key_each_time() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("poisoned"); + + let plugin = FakePlugin::new(); + plugin.seed_fence_key(); + // Backend value present, but the row's stored fingerprint can never match + // it under the fence key — the permanent poison the fence produces. + plugin.seed_value(&TenantId(tenant), &k, None, b"backend-value"); + + let repo = Arc::new(FakeSecretRepo::new()); + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "poisoned".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(Uuid::nil()), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: Some(vec![0u8; 32]), + fp_key_id: Some(crate::domain::secret::fence::CURRENT_FENCE_KEY_ID), + }); + + let svc = make_service_noop( + repo.clone(), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin.clone(), + ); + + for _ in 0..8 { + let got = svc.get(&ctx, &k).await.expect("get"); + assert!(got.is_none(), "poisoned value fails closed as a miss"); + } + + // Bounded regardless of read count: the cold load plus a single forced + // reload on the first mismatch (which heals a genuinely stale key). Every + // later poisoned read reuses the cache within the cooldown. Without the + // guard this would be ~2 backend reads *per* get (16 here) and evict the + // shared cache each time. + assert!( + plugin.fence_key_gets() <= 2, + "poisoned reads must not re-thread the fence key on every request (got {})", + plugin.fence_key_gets() + ); +} + +#[tokio::test] +async fn delete_only_own_tenant_404_when_inherited_only() { + let child = Uuid::new_v4(); + let parent = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, child); + let k = key("ancestor-key"); + + let plugin = FakePlugin::new(); + let repo = Arc::new(FakeSecretRepo::new()); + + // Only the parent has the row (shared). + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(parent), + reference: "ancestor-key".into(), + sharing: SharingMode::Shared, + owner_id: OwnerId(Uuid::nil()), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + + let svc = make_service_noop( + repo, + Arc::new(FakeDir::single(child)), // child-only chain for find_own + mock_enforcer(), + plugin, + ); + + let result = svc.delete(&ctx, &k, None).await; + assert!( + matches!(result, Err(DomainError::NotFound)), + "must return NotFound for inherited-only key" + ); +} + +#[tokio::test] +async fn read_gate_denied_when_tenant_out_of_scope() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("some-key"); + + let plugin = FakePlugin::new(); + // A resolvable row whose PDP scope excludes the caller's tenant: the read + // is an anti-enumeration 404, and the cross-tenant denial is recorded. + let repo = Arc::new(FakeSecretRepo::with_scope_allows(false)); + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "some-key".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + let metrics = FakeMetrics::new(); + + let svc = make_service( + repo, + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin, + metrics.clone(), + ); + + let result = svc.get(&ctx, &k).await; + assert!( + matches!(result, Ok(None)), + "out-of-scope read must be an anti-enumeration 404, got {result:?}" + ); + assert_eq!( + metrics.cross_tenant_denied_count(), + 1, + "cross_tenant_denied metric must be recorded" + ); +} + +// ── H2: UnsupportedTransition maps to CredStoreError::UnsupportedTransition ── + +#[test] +fn unsupported_transition_domain_error_maps_to_sdk_variant() { + use credstore_sdk::CredStoreError; + + use crate::domain::error::DomainError; + + let domain_err = DomainError::UnsupportedTransition { + detail: "cannot move between private and tenant/shared".to_owned(), + }; + let sdk_err = CredStoreError::from(domain_err); + assert!( + matches!(sdk_err, CredStoreError::UnsupportedTransition { .. }), + "expected UnsupportedTransition, got {sdk_err:?}" + ); + assert!( + !matches!(sdk_err, CredStoreError::InvalidSecretRef { .. }), + "must not map to InvalidSecretRef" + ); +} + +// ── M3: PDP-denied / eval-failed after Authorizer-port removal ─────────────── + +#[tokio::test] +async fn get_returns_not_found_when_pdp_denies() { + // Prefetch model: the PDP is consulted only for a resolved secret, and a + // denial is indistinguishable from a missing one (anti-enumeration 404). + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("some-key"); + + let repo = Arc::new(FakeSecretRepo::new()); + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "some-key".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + let svc = make_service_noop( + repo, + Arc::new(FakeDir::single(tenant)), + deny_enforcer(), + FakePlugin::new(), + ); + + let result = svc.get(&ctx, &k).await; + assert!( + matches!(result, Ok(None)), + "PDP denial on a resolved secret must be a 404, got {result:?}" + ); +} + +#[tokio::test] +async fn get_returns_service_unavailable_when_pdp_fails() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("some-key"); + + // A PDP *outage* (distinct from a denial) must surface as 503 — so the row + // must resolve first for the PDP to be consulted at all. + let repo = Arc::new(FakeSecretRepo::new()); + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "some-key".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + let svc = make_service_noop( + repo, + Arc::new(FakeDir::single(tenant)), + failing_enforcer(), + FakePlugin::new(), + ); + + let result = svc.get(&ctx, &k).await; + assert!( + matches!(result, Err(DomainError::ServiceUnavailable { .. })), + "expected ServiceUnavailable from failing_enforcer, got {result:?}" + ); +} + +#[tokio::test] +async fn operations_return_service_unavailable_when_type_resolver_fails() { + use crate::domain::secret::test_support::FailingTypeResolver; + + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("some-key"); + + // Registry outage (resolver → 503) must fail every operation closed: + // the resolved row's type gates get/delete, the requested type gates put. + let repo = Arc::new(FakeSecretRepo::new()); + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "some-key".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + let selector = Arc::new(FakePluginSelector::new(FakePlugin::new())); + let svc = Service::new( + repo, + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + selector, + Arc::new(FailingTypeResolver), + Arc::new(NoopMetrics), + test_reaper_settings(), + ); + + let got = svc.get(&ctx, &k).await; + assert!( + matches!(got, Err(DomainError::ServiceUnavailable { .. })), + "get must fail closed on a registry outage, got {got:?}" + ); + let put = svc + .put( + &ctx, + &k, + SecretValue::from("v"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await; + assert!( + matches!(put, Err(DomainError::ServiceUnavailable { .. })), + "put must fail closed on a registry outage, got {put:?}" + ); + let del = svc.delete(&ctx, &k, None).await; + assert!( + matches!(del, Err(DomainError::ServiceUnavailable { .. })), + "delete must fail closed on a registry outage, got {del:?}" + ); +} + +#[tokio::test] +async fn put_returns_access_denied_when_pdp_denies() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("some-key"); + + let svc = make_service_noop( + Arc::new(FakeSecretRepo::new()), + Arc::new(FakeDir::single(tenant)), + deny_enforcer(), + FakePlugin::new(), + ); + + let result = svc + .put( + &ctx, + &k, + SecretValue::from("v"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await; + assert!( + matches!(result, Err(DomainError::AccessDenied { .. })), + "expected AccessDenied from deny_enforcer, got {result:?}" + ); +} + +#[tokio::test] +async fn delete_returns_access_denied_when_pdp_denies() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("some-key"); + + // Prefetch model: the row must resolve for the PDP to be consulted. Delete + // reveals existence to the owner, so a PDP denial is a plain 403. + let repo = Arc::new(FakeSecretRepo::new()); + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "some-key".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + let svc = make_service_noop( + repo, + Arc::new(FakeDir::single(tenant)), + deny_enforcer(), + FakePlugin::new(), + ); + + let result = svc.delete(&ctx, &k, None).await; + assert!( + matches!(result, Err(DomainError::AccessDenied { .. })), + "expected AccessDenied from deny_enforcer, got {result:?}" + ); +} + +// ── map_plugin_err (pure mapping) ───────────────────────────────────────────── + +#[test] +fn map_plugin_err_covers_all_variants() { + use credstore_sdk::CredStoreError; + + use crate::domain::secret::service::map_plugin_err; + + assert!(matches!( + map_plugin_err(CredStoreError::NotFound), + DomainError::NotFound + )); + assert!(matches!( + map_plugin_err(CredStoreError::AccessDenied), + DomainError::AccessDenied { .. } + )); + assert!(matches!( + map_plugin_err(CredStoreError::ServiceUnavailable { + detail: "x".to_owned(), + retry_after: Some(std::time::Duration::from_secs(2)), + }), + DomainError::ServiceUnavailable { .. } + )); + assert!(matches!( + map_plugin_err(CredStoreError::NoPluginAvailable), + DomainError::ServiceUnavailable { .. } + )); + assert!(matches!( + map_plugin_err(CredStoreError::Conflict), + DomainError::Conflict + )); + assert!(matches!( + map_plugin_err(CredStoreError::InvalidSecretRef { + reason: "r".to_owned() + }), + DomainError::Internal { .. } + )); + assert!(matches!( + map_plugin_err(CredStoreError::UnsupportedTransition { + detail: "d".to_owned() + }), + DomainError::Internal { .. } + )); + assert!(matches!( + map_plugin_err(CredStoreError::Internal("boom".to_owned())), + DomainError::Internal { .. } + )); +} + +#[test] +fn no_plugin_available_maps_to_distinct_non_retryable_unavailable() { + use credstore_sdk::CredStoreError; + + use crate::domain::secret::service::map_plugin_err; + + let mapped = map_plugin_err(CredStoreError::NoPluginAvailable); + // Operator misconfiguration: a distinct, stable detail and no `retry_after`, + // distinguishable from a transient backend outage. + assert!( + matches!( + &mapped, + DomainError::ServiceUnavailable { detail, retry_after, .. } + if detail.contains("no storage plugin registered") && retry_after.is_none() + ), + "NoPluginAvailable must map to a distinct non-retryable ServiceUnavailable, got {mapped:?}" + ); +} + +// ── review follow-up regressions ────────────────────────────────────────────── + +#[test] +fn plugin_unavailable_detail_is_curated_off_the_wire() { + use crate::domain::secret::service::map_plugin_err; + use credstore_sdk::CredStoreError; + // A plugin's raw ServiceUnavailable detail (which a future vault-backed + // plugin could populate with backend specifics or secret material) must + // not reach the domain error / 503 wire response — a fixed, safe detail is + // substituted; the raw text stays in the server log only. + let mapped = map_plugin_err(CredStoreError::ServiceUnavailable { + detail: "KMS failed while storing s3cr3t-plaintext".to_owned(), + retry_after: None, + }); + match mapped { + DomainError::ServiceUnavailable { detail, .. } => { + assert_eq!(detail, "storage backend unavailable"); + assert!( + !detail.contains("s3cr3t"), + "plugin detail must not leak: {detail}" + ); + } + other => panic!("expected ServiceUnavailable, got {other:?}"), + } +} + +#[tokio::test] +async fn get_folds_plugin_access_denied_into_anti_enumeration_miss() { + // A resolved row whose backend denies the read must surface as the + // anti-enumeration 404 (Ok(None)), never a 403 that reveals existence. + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("backend-denied"); + let repo = Arc::new(FakeSecretRepo::new()); + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "backend-denied".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + let svc = make_service_noop( + repo, + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::with_get_denied(), + ); + let res = svc.get(&ctx, &k).await; + assert!( + matches!(res, Ok(None)), + "backend AccessDenied on GET must be a 404 miss, got {res:?}" + ); +} + +#[tokio::test] +async fn pdp_denial_is_not_a_dependency_health_error() { + use crate::domain::ports::metrics::{Dep, DepOp, Outcome}; + // A PDP *denial* is a normal authorization decision — the Pdp/Evaluate + // dependency-health metric must record Success, not Error, so routine + // denials don't inflate the PDP error rate / trip false outage alerts. + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("denied"); + let repo = Arc::new(FakeSecretRepo::new()); + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "denied".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + let metrics = FakeMetrics::new(); + let svc = make_service( + repo, + Arc::new(FakeDir::single(tenant)), + deny_enforcer(), + FakePlugin::new(), + metrics.clone(), + ); + assert!(svc.get(&ctx, &k).await.expect("denial is a miss").is_none()); + assert!( + metrics + .deps() + .iter() + .any(|(dep, op, outcome)| *dep == Dep::Pdp + && *op == DepOp::Evaluate + && *outcome == Outcome::Success), + "a PDP denial must record the Pdp/Evaluate dependency as Success, got {:?}", + metrics.deps() + ); +} + +#[tokio::test] +async fn if_match_loser_does_not_clobber_backend_value() { + use crate::domain::secret::model::WritePrecondition; + // #1: under If-Match the version CAS is claimed BEFORE the backend write, + // so a losing writer (touch matches 0 rows) never calls plugin.put and + // cannot overwrite the winner's value. `with_touch_not_found` forces the + // 0-row touch that models losing the race. + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("cas"); + let repo = Arc::new(FakeSecretRepo::with_touch_not_found()); + let svc = make_service( + repo, + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + FakeMetrics::new(), + ); + // Seed v1 (create path uses insert_provisioning + mark_active, not touch). + svc.put( + &ctx, + &k, + SecretValue::from("v1"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("create seeds v1"); + let generation = svc.get(&ctx, &k).await.expect("get").expect("present").id; + // If-Match:".1" write that loses the CAS (touch → 0 rows). + let res = svc + .put( + &ctx, + &k, + SecretValue::from("v2-loser"), + WriteSpec::upsert(SharingMode::Tenant).with_precondition(Some( + WritePrecondition::Version { + id: generation, + version: 1, + }, + )), + ) + .await; + assert!( + matches!(res, Err(DomainError::VersionConflict)), + "losing the CAS must be a VersionConflict, got {res:?}" + ); + // The loser never reached plugin.put: the backend still holds v1 and the + // version never advanced (its own touch matched 0 rows). + let got = svc + .get(&ctx, &k) + .await + .expect("get") + .expect("still present"); + assert_eq!( + got.value.as_bytes(), + b"v1", + "loser must not have clobbered the value" + ); + assert_eq!(got.version, 1, "loser must not have bumped the version"); +} + +#[tokio::test] +async fn create_only_conflict_is_authorized_before_it_leaks_existence() { + use credstore_sdk::SecretType as SdkSecretType; + // #4: an unauthorized caller must not distinguish a create-only conflict + // (409, the secret exists) from a plain denial — the PDP gate runs before + // the 409. A caller denied the type gets 403 whether or not the row exists. + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let api_key = SdkSecretType::from_name("api-key").expect("known"); + let (enforcer, _resolver) = + crate::domain::secret::test_support::type_deny_enforcer(vec![api_key.gts_id().to_owned()]); + let repo = Arc::new(FakeSecretRepo::new()); + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "existing".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: api_key.uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + let svc = make_service_noop( + repo, + Arc::new(FakeDir::single(tenant)), + enforcer, + FakePlugin::new(), + ); + let err = svc + .put( + &ctx, + &key("existing"), + SecretValue::from("v"), + WriteSpec::create(SharingMode::Tenant), + ) + .await + .expect_err("create-only over an existing denied-type secret"); + assert!( + matches!(err, DomainError::AccessDenied { .. }), + "must be a 403 denial (not a 409 revealing the secret exists), got {err:?}" + ); +} + +// ── getter + reaper loop ────────────────────────────────────────────────────── + +#[test] +fn reaper_tick_secs_reflects_config() { + let svc = make_service_noop( + Arc::new(FakeSecretRepo::new()), + Arc::new(FakeDir::single(Uuid::new_v4())), + mock_enforcer(), + FakePlugin::new(), + ); + assert_eq!(svc.reaper_tick_secs(), 60); +} + +#[tokio::test] +async fn reap_and_refresh_reaps_provisioning_and_refreshes_inventory() { + let tenant = Uuid::new_v4(); + let owner = Uuid::new_v4(); + let repo = Arc::new(FakeSecretRepo::new()); + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "prov".to_owned(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(owner), + status: SecretStatus::Provisioning, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "act".to_owned(), + sharing: SharingMode::Shared, + owner_id: OwnerId(owner), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + let svc = make_service( + repo.clone(), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + FakeMetrics::new(), + ); + + svc.reap_and_refresh().await; + assert_eq!( + repo.inventory().await.expect("inventory").provisioning, + 0, + "the stuck provisioning row was reaped" + ); +} + +// ── private-mode + backend-miss branches ────────────────────────────────────── + +#[tokio::test] +async fn put_create_then_update_private_secret() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("priv-key"); + let svc = make_service( + Arc::new(FakeSecretRepo::new()), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + FakeMetrics::new(), + ); + + // Create saga with Private sharing (plugin_owner = Some). + svc.put( + &ctx, + &k, + SecretValue::new(b"s".to_vec()), + WriteSpec::upsert(SharingMode::Private), + ) + .await + .expect("create private"); + // Update path on the existing private row (still private). + svc.put( + &ctx, + &k, + SecretValue::new(b"s2".to_vec()), + WriteSpec::upsert(SharingMode::Private), + ) + .await + .expect("update private"); + + let got = svc.get(&ctx, &k).await.expect("get").expect("some"); + assert_eq!(got.sharing, SharingMode::Private); +} + +#[tokio::test] +async fn get_resolved_row_without_backend_value_is_not_found() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("ghost"); + let repo = Arc::new(FakeSecretRepo::new()); + // Active row exists, but the plugin never received a value for it. + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "ghost".to_owned(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + let svc = make_service( + repo, + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + FakeMetrics::new(), + ); + + let err = svc + .get(&ctx, &k) + .await + .expect_err("row resolves but backend has no value"); + assert!(matches!(err, DomainError::NotFound)); +} + +#[tokio::test] +async fn delete_private_secret_removes_row() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("del-priv"); + let svc = make_service( + Arc::new(FakeSecretRepo::new()), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + FakeMetrics::new(), + ); + + svc.put( + &ctx, + &k, + SecretValue::new(b"x".to_vec()), + WriteSpec::upsert(SharingMode::Private), + ) + .await + .expect("create"); + svc.delete(&ctx, &k, None).await.expect("delete private"); + assert!(svc.get(&ctx, &k).await.expect("get").is_none()); +} + +// ── versioning (option a) ───────────────────────────────────────────────────── + +#[tokio::test] +async fn create_starts_at_version_one_then_overwrite_bumps() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("ver"); + let repo = Arc::new(FakeSecretRepo::new()); + let svc = make_service( + repo.clone(), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + FakeMetrics::new(), + ); + + svc.put( + &ctx, + &k, + SecretValue::from("v1"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("create"); + let row = repo + .find_for_write( + &AccessScope::for_tenant(tenant), + TenantId(tenant), + OwnerId(subject), + &k, + SharingMode::Tenant, + ) + .await + .expect("find") + .expect("row"); + assert_eq!(row.version, 1, "create seeds version 1"); + + svc.put( + &ctx, + &k, + SecretValue::from("v2"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("overwrite"); + let row = repo + .find_for_write( + &AccessScope::for_tenant(tenant), + TenantId(tenant), + OwnerId(subject), + &k, + SharingMode::Tenant, + ) + .await + .expect("find") + .expect("row"); + assert_eq!(row.version, 2, "overwrite bumps to 2"); +} + +#[tokio::test] +async fn put_create_race_resolves_to_update() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("race"); + + // The winner's row exists but is still Provisioning (invisible to + // find_for_write). The fake promotes it to Active when our insert conflicts, + // simulating the winner finishing its saga — so the bounded retry resolves + // to the update path deterministically. + let repo = Arc::new(FakeSecretRepo::with_promote_on_conflict(true)); + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "race".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Provisioning, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + let svc = make_service_noop( + repo.clone(), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + ); + + // PUT (create_only = false) loses the create race, then resolves to update. + svc.put( + &ctx, + &k, + SecretValue::from("v"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("race resolves to update"); + let row = repo + .find_for_write( + &AccessScope::for_tenant(tenant), + TenantId(tenant), + OwnerId(subject), + &k, + SharingMode::Tenant, + ) + .await + .expect("find") + .expect("row"); + assert_eq!(row.version, 2, "the loser touched the winner's row: 1 -> 2"); +} + +#[tokio::test] +async fn put_create_race_exhausted_returns_conflict() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("stuck"); + + // Winner stuck mid-saga: a Provisioning row that never promotes. PUT's + // bounded retry never sees it Active -> retry-safe 409. + let repo = Arc::new(FakeSecretRepo::new()); + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "stuck".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Provisioning, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + let svc = make_service_noop( + repo, + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + ); + + let result = svc + .put( + &ctx, + &k, + SecretValue::from("v"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await; + assert!( + matches!(result, Err(DomainError::Conflict)), + "exhausted retry -> 409, got {result:?}" + ); +} + +#[tokio::test] +async fn failed_create_does_not_wedge_the_reference() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("wedge"); + + // Backend hiccup: the first plugin.put (mid create-saga) fails, then recovers. + let plugin = FakePlugin::with_put_failures(1); + // Pre-seed the fence key so the injected failure hits the VALUE write + // (mid create-saga), not the fence-key bootstrap put. + plugin.seed_fence_key(); + let repo = Arc::new(FakeSecretRepo::new()); + let svc = make_service( + repo.clone(), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin, + FakeMetrics::new(), + ); + + // Create: the provisioning row is inserted, then the backend put fails. + let first = svc + .put( + &ctx, + &k, + SecretValue::from("v"), + WriteSpec::create(SharingMode::Tenant), + ) + .await; + assert!( + first.is_err(), + "backend put failure must surface, got {first:?}" + ); + + // The failed create must compensate its provisioning row rather than leave + // it to wedge the reference until the reaper runs. + assert_eq!( + repo.inventory().await.expect("inventory").provisioning, + 0, + "failed create must roll back the provisioning row" + ); + + // A retry now succeeds instead of hitting a misleading 409. + svc.put( + &ctx, + &k, + SecretValue::from("v"), + WriteSpec::create(SharingMode::Tenant), + ) + .await + .expect("retry after a failed create must succeed, not 409"); +} + +#[tokio::test] +async fn put_denied_when_tenant_out_of_scope_inserts_no_row() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("oos"); + + // PDP allow-with-constraints scope that excludes the caller's own tenant. + let repo = Arc::new(FakeSecretRepo::with_scope_allows(false)); + let metrics = FakeMetrics::new(); + let svc = make_service( + repo.clone(), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + metrics.clone(), + ); + + let result = svc + .put( + &ctx, + &k, + SecretValue::from("v"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await; + + assert!( + matches!(result, Err(DomainError::AccessDenied { .. })), + "out-of-scope write must be denied, got {result:?}" + ); + // The scope_unchecked insert must never run: no orphan provisioning row. + assert_eq!( + repo.inventory().await.expect("inventory").provisioning, + 0, + "denied write must not insert an orphan provisioning row" + ); + assert_eq!( + metrics.cross_tenant_denied_count(), + 1, + "cross_tenant_denied metric must be recorded on the write path" + ); +} + +#[tokio::test] +async fn failed_create_records_rollback_success_metric() { + use crate::domain::ports::metrics::Outcome; + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("rollback-ok"); + + let plugin = FakePlugin::with_put_failures(1); + // Pre-seed the fence key so the injected failure hits the VALUE write + // (mid create-saga), not the fence-key bootstrap put. + plugin.seed_fence_key(); + let metrics = FakeMetrics::new(); + let svc = make_service( + Arc::new(FakeSecretRepo::new()), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin, + metrics.clone(), + ); + + _ = svc + .put( + &ctx, + &k, + SecretValue::from("v"), + WriteSpec::create(SharingMode::Tenant), + ) + .await; + + assert_eq!( + metrics.provisioning_rollbacks(), + vec![Outcome::Success], + "a compensated create must record one successful rollback" + ); +} + +#[tokio::test] +async fn failed_rollback_records_error_metric() { + use crate::domain::ports::metrics::Outcome; + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("rollback-fail"); + + // Backend put fails, then the rollback delete also fails — the reference + // stays wedged, the case worth alerting on. + let plugin = FakePlugin::with_put_failures(1); + // Pre-seed the fence key so the injected failure hits the VALUE write + // (mid create-saga), not the fence-key bootstrap put. + plugin.seed_fence_key(); + let metrics = FakeMetrics::new(); + let svc = make_service( + Arc::new(FakeSecretRepo::with_delete_failure()), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin, + metrics.clone(), + ); + + _ = svc + .put( + &ctx, + &k, + SecretValue::from("v"), + WriteSpec::create(SharingMode::Tenant), + ) + .await; + + assert_eq!( + metrics.provisioning_rollbacks(), + vec![Outcome::Error], + "a failed rollback must record an Error-outcome metric" + ); +} + +#[tokio::test] +async fn post_create_only_race_returns_conflict_immediately() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("conly"); + + // A Provisioning winner row; create_only POST must 409 without retrying. + // (No promote_on_conflict needed: create_only returns before the retry loop.) + let repo = Arc::new(FakeSecretRepo::new()); + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "conly".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Provisioning, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + let svc = make_service_noop( + repo, + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + ); + + let result = svc + .put( + &ctx, + &k, + SecretValue::from("v"), + WriteSpec::create(SharingMode::Tenant), + ) + .await; + assert!( + matches!(result, Err(DomainError::Conflict)), + "create-only race -> 409, got {result:?}" + ); +} + +// ── optimistic concurrency (If-Match precondition) ─────────────────────────── + +#[tokio::test] +async fn put_if_match_matching_version_overwrites_and_bumps() { + use crate::domain::secret::model::WritePrecondition; + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("oc"); + let repo = Arc::new(FakeSecretRepo::new()); + let svc = make_service( + repo.clone(), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + FakeMetrics::new(), + ); + + // Create at version 1, then overwrite with the matching If-Match. + svc.put( + &ctx, + &k, + SecretValue::from("v1"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("create"); + let generation = svc.get(&ctx, &k).await.expect("get").expect("present").id; + svc.put( + &ctx, + &k, + SecretValue::from("v2"), + WriteSpec::upsert(SharingMode::Tenant).with_precondition(Some( + WritePrecondition::Version { + id: generation, + version: 1, + }, + )), + ) + .await + .expect("matching If-Match must succeed"); + + let got = svc.get(&ctx, &k).await.expect("get").expect("some"); + assert_eq!(got.version, 2, "matching If-Match bumps the version"); + assert_eq!(got.value.as_bytes(), b"v2"); +} + +#[tokio::test] +async fn put_overwrite_touch_zero_rows_maps_to_version_conflict() { + // Overwrite path: the backend `plugin.put` commits, but the gated `touch` + // matches 0 rows because the row was concurrently deleted/reaped between + // `find_for_write` and the version bump. Even without a precondition this + // must surface a retryable VersionConflict (canonical Aborted/409) rather + // than acknowledging a write no active row makes readable. + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("oc-put-race"); + // `touch_not_found` only forces `touch` to return 0 rows; create (which uses + // insert_provisioning + mark_active) still seeds the row, so the second put + // takes the overwrite path. + let repo = Arc::new(FakeSecretRepo::with_touch_not_found()); + let svc = make_service( + repo, + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + FakeMetrics::new(), + ); + + svc.put( + &ctx, + &k, + SecretValue::from("v1"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("create seeds the row"); + + let res = svc + .put( + &ctx, + &k, + SecretValue::from("v2"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await; + assert!( + matches!(res, Err(DomainError::VersionConflict)), + "row-vanished-on-commit (no precondition) -> VersionConflict, got {res:?}" + ); +} + +#[tokio::test] +async fn put_if_match_stale_version_conflicts_without_writing() { + use crate::domain::secret::model::WritePrecondition; + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("oc-stale"); + let svc = make_service( + Arc::new(FakeSecretRepo::new()), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + FakeMetrics::new(), + ); + + svc.put( + &ctx, + &k, + SecretValue::from("v1"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("create"); + let generation = svc.get(&ctx, &k).await.expect("get").expect("present").id; + + let result = svc + .put( + &ctx, + &k, + SecretValue::from("v2"), + WriteSpec::upsert(SharingMode::Tenant).with_precondition(Some( + WritePrecondition::Version { + id: generation, + version: 99, + }, + )), + ) + .await; + assert!( + matches!(result, Err(DomainError::VersionConflict)), + "stale If-Match must conflict, got {result:?}" + ); + + // The stale write must NOT have touched the backend value or the version. + let got = svc.get(&ctx, &k).await.expect("get").expect("some"); + assert_eq!(got.version, 1, "stale write must not bump the version"); + assert_eq!( + got.value.as_bytes(), + b"v1", + "stale write must not overwrite" + ); +} + +#[tokio::test] +async fn put_if_match_on_missing_secret_conflicts() { + use crate::domain::secret::model::WritePrecondition; + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("oc-missing"); + let repo = Arc::new(FakeSecretRepo::new()); + let svc = make_service( + repo.clone(), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + FakeMetrics::new(), + ); + + let result = svc + .put( + &ctx, + &k, + SecretValue::from("v"), + WriteSpec::upsert(SharingMode::Tenant).with_precondition(Some( + WritePrecondition::Version { + id: Uuid::new_v4(), + version: 1, + }, + )), + ) + .await; + assert!( + matches!(result, Err(DomainError::VersionConflict)), + "If-Match on a non-existent secret must conflict, got {result:?}" + ); + assert_eq!( + repo.inventory().await.expect("inventory").provisioning, + 0, + "conflicting create must not insert a row" + ); +} + +#[tokio::test] +async fn delete_if_match_stale_version_conflicts() { + use crate::domain::secret::model::WritePrecondition; + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("oc-del"); + let svc = make_service( + Arc::new(FakeSecretRepo::new()), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + FakeMetrics::new(), + ); + + svc.put( + &ctx, + &k, + SecretValue::from("v1"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("create"); + let generation = svc.get(&ctx, &k).await.expect("get").expect("present").id; + + let result = svc + .delete( + &ctx, + &k, + Some(WritePrecondition::Version { + id: generation, + version: 99, + }), + ) + .await; + assert!( + matches!(result, Err(DomainError::VersionConflict)), + "stale If-Match delete must conflict, got {result:?}" + ); + // The secret must still be there. + assert!( + svc.get(&ctx, &k).await.expect("get").is_some(), + "stale delete must not remove the secret" + ); + + // Matching validator deletes. + svc.delete( + &ctx, + &k, + Some(WritePrecondition::Version { + id: generation, + version: 1, + }), + ) + .await + .expect("matching If-Match delete must succeed"); + assert!(svc.get(&ctx, &k).await.expect("get").is_none()); +} + +#[tokio::test] +async fn delete_if_match_race_maps_zero_rows_to_version_conflict() { + use crate::domain::secret::model::WritePrecondition; + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("oc-del-race"); + + // Row exists at v1 (find_own + pre-check pass), but the gated + // mark_deprovisioning matches 0 rows — the row moved/vanished between the + // pre-check and the commit. + let repo = Arc::new(FakeSecretRepo::with_mark_not_found()); + let row_id = Uuid::new_v4(); + repo.seed(SecretRow { + id: row_id, + tenant_id: TenantId(tenant), + reference: "oc-del-race".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + let svc = make_service_noop( + repo, + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + ); + + // Under a version precondition, the 0-row flip is an optimistic-lock + // conflict, not a misleading 404. + let conflict = svc + .delete( + &ctx, + &k, + Some(WritePrecondition::Version { + id: row_id, + version: 1, + }), + ) + .await; + assert!( + matches!(conflict, Err(DomainError::VersionConflict)), + "0-row mark under precondition -> VersionConflict, got {conflict:?}" + ); + + // Without a precondition, the same 0-row flip stays a NotFound. + let plain = svc.delete(&ctx, &k, None).await; + assert!( + matches!(plain, Err(DomainError::NotFound)), + "0-row mark without precondition -> NotFound, got {plain:?}" + ); +} + +// ── deprovisioning saga ─────────────────────────────────────────────────────── + +#[tokio::test] +async fn delete_backend_failure_leaves_deprovisioning_row_and_hides_secret() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("dp-fail"); + let repo = Arc::new(FakeSecretRepo::new()); + let plugin = FakePlugin::with_delete_failures(1); + let svc = make_service_noop( + repo.clone(), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin, + ); + + svc.put( + &ctx, + &k, + SecretValue::from("v1"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("create"); + + // Backend delete fails: the caller gets the error, the row stays + // deprovisioning, and the secret already no longer resolves. + let err = svc.delete(&ctx, &k, None).await.expect_err("backend fails"); + assert!(matches!(err, DomainError::Internal { .. }), "got {err:?}"); + let rows = repo.rows(); + assert_eq!(rows.len(), 1); + assert_eq!(rows[0].status, SecretStatus::Deprovisioning); + assert!( + svc.get(&ctx, &k).await.expect("get").is_none(), + "deprovisioning secret must not resolve" + ); + + // A DELETE retry resumes the saga (plugin recovered) and finishes cleanup. + svc.delete(&ctx, &k, None) + .await + .expect("retry resumes saga"); + assert!(repo.rows().is_empty(), "row removed after resumed saga"); +} + +#[tokio::test] +async fn delete_while_deprovisioning_recreate_conflicts_until_cleanup() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("dp-held"); + let repo = Arc::new(FakeSecretRepo::new()); + let plugin = FakePlugin::with_delete_failures(1); + let svc = make_service_noop( + repo.clone(), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin, + ); + + svc.put( + &ctx, + &k, + SecretValue::from("v1"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("create"); + svc.delete(&ctx, &k, None) + .await + .expect_err("backend delete fails; row wedges in deprovisioning"); + + // The deprovisioning row still holds the unique index: a re-create of the + // same reference conflicts (retryable) until cleanup completes. + let err = svc + .put( + &ctx, + &k, + SecretValue::from("v2"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect_err("name still held"); + assert!(matches!(err, DomainError::Conflict), "got {err:?}"); +} + +#[tokio::test] +async fn reaper_completes_stuck_deprovisioning_and_reconciles_backend() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("dp-reap"); + let repo = Arc::new(FakeSecretRepo::new()); + let plugin = FakePlugin::with_delete_failures(1); + let metrics = FakeMetrics::new(); + let svc = make_service( + repo.clone(), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin.clone(), + metrics.clone(), + ); + + svc.put( + &ctx, + &k, + SecretValue::from("v1"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("create"); + svc.delete(&ctx, &k, None) + .await + .expect_err("backend delete fails once"); + + // Value still in the backend, row stuck deprovisioning. + assert!(plugin.contains(&TenantId(tenant), &k, None)); + + // The reaper retries the backend delete and removes the row. + svc.reap_and_refresh().await; + assert!(repo.rows().is_empty(), "stuck deprovisioning row reaped"); + assert!( + !plugin.contains(&TenantId(tenant), &k, None), + "backend value reconciled by the reaper" + ); + assert_eq!(metrics.deprovisioning_reaped_total(), 1); +} + +#[tokio::test] +async fn reaper_keeps_deprovisioning_row_while_backend_still_fails() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("dp-keep"); + let repo = Arc::new(FakeSecretRepo::new()); + // Fails the saga's delete AND the reaper's retry. + let plugin = FakePlugin::with_delete_failures(2); + let metrics = FakeMetrics::new(); + let svc = make_service( + repo.clone(), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin, + metrics.clone(), + ); + + svc.put( + &ctx, + &k, + SecretValue::from("v1"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("create"); + svc.delete(&ctx, &k, None) + .await + .expect_err("backend delete fails"); + + svc.reap_and_refresh().await; + let rows = repo.rows(); + assert_eq!( + rows.len(), + 1, + "row must be kept while the backend still holds the value" + ); + assert_eq!(rows[0].status, SecretStatus::Deprovisioning); + assert_eq!(metrics.deprovisioning_reaped_total(), 0); + + // Next tick, the backend recovered: cleanup completes. + svc.reap_and_refresh().await; + assert!(repo.rows().is_empty()); + assert_eq!(metrics.deprovisioning_reaped_total(), 1); +} + +#[tokio::test] +async fn reaper_reconciles_backend_for_stuck_provisioning_rows() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let k = key("pv-orphan"); + let repo = Arc::new(FakeSecretRepo::new()); + let plugin = FakePlugin::new(); + let metrics = FakeMetrics::new(); + let svc = make_service( + repo.clone(), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin.clone(), + metrics.clone(), + ); + + // Simulate a crash between plugin.put and mark_active: provisioning row + // plus an orphaned backend value. + let id = Uuid::new_v4(); + repo.seed(SecretRow { + id, + tenant_id: TenantId(tenant), + reference: "pv-orphan".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Provisioning, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + let ctx = make_ctx(subject, tenant); + plugin + .put( + &ctx, + &TenantId(tenant), + &k, + SecretValue::from("orphan"), + None, + ) + .await + .expect("seed backend value"); + + svc.reap_and_refresh().await; + assert!(repo.rows().is_empty(), "stuck provisioning row reaped"); + assert!( + !plugin.contains(&TenantId(tenant), &k, None), + "orphaned backend value reconciled" + ); + assert_eq!(metrics.provisioning_reaped_total(), 1); +} + +/// Regression: a slow create whose `mark_active` lands *between* the reaper's +/// list and its delete must survive — the reaper must not delete the row nor +/// the backend value out from under a create the client saw succeed. The +/// status-gated delete loses the race and leaves the now-active secret intact. +#[tokio::test] +async fn reaper_does_not_reap_provisioning_row_that_raced_to_active() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let k = key("pv-won-race"); + // list_stale_pending returns the row as Provisioning but flips the stored + // row to Active — exactly the mark_active-after-list interleaving. + let repo = Arc::new(FakeSecretRepo::with_provisioning_promoted_on_list()); + let plugin = FakePlugin::new(); + let metrics = FakeMetrics::new(); + let svc = make_service( + repo.clone(), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin.clone(), + metrics.clone(), + ); + + let id = Uuid::new_v4(); + repo.seed(SecretRow { + id, + tenant_id: TenantId(tenant), + reference: "pv-won-race".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Provisioning, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + let ctx = make_ctx(subject, tenant); + plugin + .put( + &ctx, + &TenantId(tenant), + &k, + SecretValue::from("winner"), + None, + ) + .await + .expect("seed backend value"); + + svc.reap_and_refresh().await; + + let rows = repo.rows(); + assert_eq!(rows.len(), 1, "the raced-to-active row must survive"); + assert_eq!(rows[0].status, SecretStatus::Active); + assert!( + plugin.contains(&TenantId(tenant), &k, None), + "backend value of the succeeded create must NOT be deleted" + ); + assert_eq!(metrics.provisioning_reaped_total(), 0, "nothing was reaped"); +} + +// ── secret types ────────────────────────────────────────────────────────────── + +#[tokio::test] +async fn typed_create_enforces_allow_sharing_and_returns_type() { + use credstore_sdk::{ExpiryWrite, SecretType, WriteOptions}; + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("typed"); + let svc = make_service_noop( + Arc::new(FakeSecretRepo::new()), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + ); + + let personal = WriteOptions { + secret_type: Some( + SecretType::from_name("personal-token") + .expect("known") + .into(), + ), + expires_at: ExpiryWrite::Preserve, + precondition: None, + }; + // personal-token is private-only: tenant sharing is a type violation. + let err = svc + .put( + &ctx, + &k, + SecretValue::from("t"), + WriteSpec::upsert(SharingMode::Tenant).with_opts(personal.clone()), + ) + .await + .expect_err("sharing not allowed for type"); + assert!( + matches!(err, DomainError::TypeViolation { reason, .. } if reason == "SHARING_NOT_ALLOWED_FOR_TYPE"), + ); + + // Private is fine; the read reports the type. + svc.put( + &ctx, + &k, + SecretValue::from("t"), + WriteSpec::upsert(SharingMode::Private).with_opts(personal), + ) + .await + .expect("private personal token ok"); + let got = svc.get(&ctx, &k).await.expect("get").expect("resolves"); + assert_eq!( + got.secret_type, + SecretType::from_name("personal-token") + .expect("known") + .gts_id() + ); +} + +#[tokio::test] +async fn secret_type_is_immutable_on_overwrite() { + use credstore_sdk::{ExpiryWrite, SecretType, WriteOptions}; + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("immutable"); + let svc = make_service_noop( + Arc::new(FakeSecretRepo::new()), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + ); + + svc.put( + &ctx, + &k, + SecretValue::from("k"), + WriteSpec::upsert(SharingMode::Tenant).with_opts(WriteOptions { + secret_type: Some(SecretType::from_name("api-key").expect("known").into()), + expires_at: ExpiryWrite::Preserve, + precondition: None, + }), + ) + .await + .expect("create api-key"); + + // Explicit differing type on overwrite is rejected. + let err = svc + .put( + &ctx, + &k, + SecretValue::from("k2"), + WriteSpec::upsert(SharingMode::Tenant).with_opts(WriteOptions { + secret_type: Some(SecretType::generic().into()), + expires_at: ExpiryWrite::Preserve, + precondition: None, + }), + ) + .await + .expect_err("type immutable"); + assert!(matches!(err, DomainError::TypeViolation { reason, .. } if reason == "TYPE_IMMUTABLE")); + + // Absent type inherits the existing one — overwrite succeeds, type kept. + svc.put( + &ctx, + &k, + SecretValue::from("k3"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("untyped overwrite keeps type"); + let got = svc.get(&ctx, &k).await.expect("get").expect("resolves"); + assert_eq!( + got.secret_type, + SecretType::from_name("api-key").expect("known").gts_id() + ); +} + +#[tokio::test] +async fn expired_secret_stops_resolving_and_is_swept_via_deprovisioning() { + use credstore_sdk::{ExpiryWrite, SecretType, WriteOptions}; + use time::{Duration as TimeDuration, OffsetDateTime}; + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("expiring"); + let repo = Arc::new(FakeSecretRepo::new()); + let plugin = FakePlugin::new(); + let metrics = FakeMetrics::new(); + let svc = make_service( + repo.clone(), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin.clone(), + metrics.clone(), + ); + + // The window must comfortably exceed one create+get. It is generous + // because nextest runs each test in its own process, so this test pays the + // crypto backend's one-time initialization (AWS-LC, on the first fence op) + // inside the timed section — a tight 20ms budget flaked on slow CI runners. + let ttl = TimeDuration::milliseconds(500); + svc.put( + &ctx, + &k, + SecretValue::from("short-lived"), + WriteSpec::upsert(SharingMode::Tenant).with_opts(WriteOptions { + secret_type: Some(SecretType::from_name("bearer-token").expect("known").into()), + expires_at: ExpiryWrite::Set(OffsetDateTime::now_utc() + ttl), + precondition: None, + }), + ) + .await + .expect("create expiring secret"); + assert!(svc.get(&ctx, &k).await.expect("get").is_some()); + + tokio::time::sleep(std::time::Duration::from_millis(600)).await; + // Expired: resolution misses even before the reaper runs. + assert!( + svc.get(&ctx, &k).await.expect("get").is_none(), + "expired secret must not resolve" + ); + + // The reaper flips it to deprovisioning and completes the cleanup + // (fake list_stale_pending treats every non-active row as stale). + svc.reap_and_refresh().await; + assert!(repo.rows().is_empty(), "expired row fully reaped"); + assert!( + !plugin.contains(&TenantId(tenant), &k, None), + "backend value reconciled" + ); + assert_eq!(metrics.deprovisioning_reaped_total(), 1); +} + +#[tokio::test] +async fn expiry_rejected_for_non_expirable_type() { + use credstore_sdk::{ExpiryWrite, WriteOptions}; + use time::{Duration as TimeDuration, OffsetDateTime}; + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let svc = make_service_noop( + Arc::new(FakeSecretRepo::new()), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + ); + + // generic is not expirable: expires_at is a type violation. + let err = svc + .put( + &ctx, + &key("no-expiry"), + SecretValue::from("v"), + WriteSpec::upsert(SharingMode::Tenant).with_opts(WriteOptions { + secret_type: None, + expires_at: ExpiryWrite::Set(OffsetDateTime::now_utc() + TimeDuration::hours(1)), + precondition: None, + }), + ) + .await + .expect_err("generic has no expiry"); + assert!( + matches!(err, DomainError::TypeViolation { reason, .. } if reason == "EXPIRY_NOT_SUPPORTED_FOR_TYPE"), + ); +} + +// ── per-type authorization ──────────────────────────────────────────────────── + +#[tokio::test] +async fn per_type_pdp_denial_hides_reads_and_forbids_writes() { + use credstore_sdk::{ExpiryWrite, SecretType, WriteOptions}; + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("typed-authz"); + let api_key = SecretType::from_name("api-key").expect("known"); + + // PDP allows the base type but denies the concrete api-key type. + let (enforcer, resolver) = + crate::domain::secret::test_support::type_deny_enforcer(vec![api_key.gts_id().to_owned()]); + let repo = Arc::new(FakeSecretRepo::new()); + // Seed an existing active api-key secret directly (writes are denied). + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "typed-authz".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: api_key.uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + let svc = make_service_noop( + repo, + Arc::new(FakeDir::single(tenant)), + enforcer, + FakePlugin::new(), + ); + + // Read: per-type denial is the anti-enumeration 404, not a 403. + assert!( + svc.get(&ctx, &k).await.expect("get ok").is_none(), + "type-denied secret must resolve as not-found" + ); + + // Overwrite: operation-level denial. + let err = svc + .put( + &ctx, + &k, + SecretValue::from("v2"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect_err("typed write denied"); + assert!( + matches!(err, DomainError::AccessDenied { .. }), + "got {err:?}" + ); + + // Create of a new typed secret: denied before any side effect. + let err = svc + .put( + &ctx, + &key("new-typed"), + SecretValue::from("v"), + WriteSpec::upsert(SharingMode::Tenant).with_opts(WriteOptions { + secret_type: Some(api_key.into()), + expires_at: ExpiryWrite::Preserve, + precondition: None, + }), + ) + .await + .expect_err("typed create denied"); + assert!( + matches!(err, DomainError::AccessDenied { .. }), + "got {err:?}" + ); + + // Delete: operation-level denial. + let err = svc + .delete(&ctx, &k, None) + .await + .expect_err("typed delete denied"); + assert!( + matches!(err, DomainError::AccessDenied { .. }), + "got {err:?}" + ); + + // The PEP targeted the concrete type id (not only the base type). + let seen = resolver.seen_resource_types.lock().expect("lock").clone(); + assert!( + seen.iter().any(|t| t == api_key.gts_id()), + "PDP must be evaluated against the concrete type id, saw: {seen:?}" + ); +} + +#[tokio::test] +async fn generic_secrets_evaluate_the_full_concrete_type() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("plain"); + + let (enforcer, resolver) = crate::domain::secret::test_support::type_deny_enforcer(Vec::new()); + let svc = make_service_noop( + Arc::new(FakeSecretRepo::new()), + Arc::new(FakeDir::single(tenant)), + enforcer, + FakePlugin::new(), + ); + + svc.put( + &ctx, + &k, + SecretValue::from("v"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("generic write"); + assert!(svc.get(&ctx, &k).await.expect("get").is_some()); + + // Every credstore operation authorizes against the secret's full concrete + // type — including `generic` — never the bare base type. This is what lets + // a policy target `...generic.v1~` specifically. + let generic_id = credstore_sdk::SecretType::generic().gts_id(); + let seen = resolver.seen_resource_types.lock().expect("lock").clone(); + assert!(!seen.is_empty(), "the PDP must be consulted"); + assert!( + seen.iter().all(|t| t == generic_id), + "generic operations must evaluate the full generic type id {generic_id:?}, saw: {seen:?}" + ); +} + +// ── value-fingerprint fence (docs/features/001-value-fingerprint-fence.md) ─── + +/// The fence key `FakePlugin::seed_fence_key` installs. +const TEST_FENCE_KEY: [u8; 32] = [42u8; 32]; + +#[tokio::test] +async fn crosswise_lww_desync_reads_fail_closed_not_disclosed() { + use crate::domain::ports::metrics::{FenceVerify, ReadOutcome}; + use crate::domain::secret::fence; + // The poisoned end-state of two crosswise unconditional PUTs (finding #2): + // Alice touched the row last (sharing=shared, fp of HER value), but Bob's + // tenant-intent value landed last in the backend. Without the fence a + // child-tenant GET would serve Bob's value under Alice's `shared` label. + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("crosswise"); + + let plugin = FakePlugin::new(); + plugin.seed_fence_key(); + let fp_alice = fence::compute_fp(&TEST_FENCE_KEY, b"value-alice"); + let repo = Arc::new(FakeSecretRepo::new()); + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "crosswise".into(), + sharing: SharingMode::Shared, + owner_id: OwnerId(subject), + status: SecretStatus::Active, + version: 2, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: Some(fp_alice), + fp_key_id: Some(fence::CURRENT_FENCE_KEY_ID), + }); + // Bob's value is what the backend actually holds. + plugin + .put( + &ctx, + &TenantId(tenant), + &k, + SecretValue::from("value-bob"), + None, + ) + .await + .expect("backend value"); + + let metrics = FakeMetrics::new(); + let svc = make_service( + repo, + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin, + metrics.clone(), + ); + + // Fail closed: no value under a foreign sharing label, ever. + let got = svc.get(&ctx, &k).await.expect("get is Ok"); + assert!( + got.is_none(), + "a fingerprint mismatch must be an anti-enumeration miss, got a value" + ); + assert!( + metrics.fence_verifies().contains(&FenceVerify::Mismatch), + "the mismatch must be observable, got {:?}", + metrics.fence_verifies() + ); + assert_eq!(metrics.last_read_outcome(), Some(ReadOutcome::Miss)); + + // Any subsequent successful PUT heals the reference. + svc.put( + &ctx, + &k, + SecretValue::from("value-heal"), + WriteSpec::upsert(SharingMode::Shared), + ) + .await + .expect("healing put"); + let healed = svc.get(&ctx, &k).await.expect("get").expect("healed"); + assert_eq!(healed.value.as_bytes(), b"value-heal"); +} + +#[tokio::test] +async fn clean_write_then_read_verifies_ok() { + use crate::domain::ports::metrics::FenceVerify; + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("clean"); + let metrics = FakeMetrics::new(); + let svc = make_service( + Arc::new(FakeSecretRepo::new()), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + metrics.clone(), + ); + + svc.put( + &ctx, + &k, + SecretValue::from("v1"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("create"); + let got = svc.get(&ctx, &k).await.expect("get").expect("present"); + assert_eq!(got.value.as_bytes(), b"v1"); + assert_eq!( + metrics.fence_verifies(), + vec![FenceVerify::Ok], + "an API-created secret must verify Ok (stamped on create)" + ); +} + +#[tokio::test] +async fn overwrite_restamps_the_fingerprint() { + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("restamp"); + let repo = Arc::new(FakeSecretRepo::new()); + let svc = make_service_noop( + repo.clone(), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + ); + + svc.put( + &ctx, + &k, + SecretValue::from("v1"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("create"); + let fp1 = repo.rows()[0].value_fp.clone().expect("stamped on create"); + + svc.put( + &ctx, + &k, + SecretValue::from("v2"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("overwrite"); + let fp2 = repo.rows()[0].value_fp.clone().expect("still stamped"); + assert_ne!(fp1, fp2, "a new value must get a new fingerprint"); + // And the overwritten secret still reads back (fp matches the new value). + let got = svc.get(&ctx, &k).await.expect("get").expect("present"); + assert_eq!(got.value.as_bytes(), b"v2"); +} + +#[tokio::test] +async fn aba_recreate_rejects_stale_generation_validator() { + use crate::domain::secret::model::WritePrecondition; + // Finding #1: delete + recreate restarts the version counter at 1, so a + // bare-version validator from the earlier generation would match again. + // The generation-bound validator must reject it. + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("aba"); + let svc = make_service_noop( + Arc::new(FakeSecretRepo::new()), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + FakePlugin::new(), + ); + + // Generation 1. + svc.put( + &ctx, + &k, + SecretValue::from("gen1"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("create gen1"); + let gen1 = svc.get(&ctx, &k).await.expect("get").expect("gen1"); + let stale_validator = WritePrecondition::Version { + id: gen1.id, + version: gen1.version, + }; + + // Delete + recreate: generation 2, version counter restarts at 1. + svc.delete(&ctx, &k, None).await.expect("delete gen1"); + svc.put( + &ctx, + &k, + SecretValue::from("gen2"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("create gen2"); + let gen2 = svc.get(&ctx, &k).await.expect("get").expect("gen2"); + assert_eq!( + gen2.version, gen1.version, + "the ABA precondition: version counters coincide across generations" + ); + assert_ne!(gen2.id, gen1.id, "a recreated secret is a new generation"); + + // The stale validator (same version number, old generation) must NOT match. + let res = svc + .put( + &ctx, + &k, + SecretValue::from("stale-overwrite"), + WriteSpec::upsert(SharingMode::Tenant).with_precondition(Some(stale_validator)), + ) + .await; + assert!( + matches!(res, Err(DomainError::VersionConflict)), + "a stale generation's validator must conflict, got {res:?}" + ); + let unchanged = svc.get(&ctx, &k).await.expect("get").expect("present"); + assert_eq!(unchanged.value.as_bytes(), b"gen2", "no lost update"); + + // The current generation's validator works. + svc.put( + &ctx, + &k, + SecretValue::from("gen2-v2"), + WriteSpec::upsert(SharingMode::Tenant).with_precondition(Some( + WritePrecondition::Version { + id: gen2.id, + version: gen2.version, + }, + )), + ) + .await + .expect("current validator must match"); +} + +#[tokio::test] +async fn legacy_seeded_row_serves_and_backfills_without_version_bump() { + use crate::domain::ports::metrics::{FenceVerify, Outcome}; + use crate::domain::secret::fence; + // Out-of-band seeding contract: row inserted with value_fp = NULL, value + // placed directly in the backend. Served on trust, stamped on first read, + // ETag/version untouched by the backfill. + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("seeded"); + + let plugin = FakePlugin::new(); + plugin.seed_fence_key(); + plugin.seed_value(&TenantId(tenant), &k, None, b"seeded-value"); + let repo = Arc::new(FakeSecretRepo::new()); + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "seeded".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }); + let metrics = FakeMetrics::new(); + let svc = make_service( + repo.clone(), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin, + metrics.clone(), + ); + + let got = svc.get(&ctx, &k).await.expect("get").expect("served"); + assert_eq!(got.value.as_bytes(), b"seeded-value"); + assert_eq!(got.version, 1, "backfill must not bump the version"); + assert_eq!(metrics.fence_verifies(), vec![FenceVerify::Legacy]); + assert_eq!(metrics.fence_backfills(), vec![Outcome::Success]); + + // The row is now fenced with the fp of the seeded value under key id 1. + let row = &repo.rows()[0]; + let expected_fp = fence::compute_fp(&TEST_FENCE_KEY, b"seeded-value"); + assert_eq!(row.value_fp.as_deref(), Some(expected_fp.as_slice())); + assert_eq!(row.fp_key_id, Some(fence::CURRENT_FENCE_KEY_ID)); + assert_eq!(row.version, 1); + + // A second read verifies Ok (no longer legacy, no second backfill). + let again = svc.get(&ctx, &k).await.expect("get").expect("served"); + assert_eq!(again.value.as_bytes(), b"seeded-value"); + assert_eq!( + metrics.fence_verifies(), + vec![FenceVerify::Legacy, FenceVerify::Ok] + ); + assert_eq!(metrics.fence_backfills(), vec![Outcome::Success]); +} + +#[tokio::test] +async fn fence_key_bootstrap_persists_the_key_in_the_backend() { + use crate::domain::secret::fence; + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("boot"); + let plugin = FakePlugin::new(); + let svc = make_service_noop( + Arc::new(FakeSecretRepo::new()), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin.clone(), + ); + + let fence_ref = key(fence::FENCE_KEY_REF); + assert!( + !plugin.contains(&TenantId(Uuid::nil()), &fence_ref, None), + "virgin deployment: no fence key yet" + ); + svc.put( + &ctx, + &k, + SecretValue::from("v"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("first write bootstraps the key"); + assert!( + plugin.contains(&TenantId(Uuid::nil()), &fence_ref, None), + "the generated fence key must be persisted under the reserved nil-tenant entry" + ); + // And the write it fenced reads back. + assert!(svc.get(&ctx, &k).await.expect("get").is_some()); +} + +#[tokio::test] +async fn fence_key_bootstrap_adopts_existing_key_without_overwriting() { + // Bootstrap must ADOPT a fence key already present in the backend (as if a + // peer replica won the first-boot race), never regenerate/overwrite it — + // overwriting would restart the race and poison peers' fingerprints. + use crate::domain::secret::fence; + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("boot"); + let plugin = FakePlugin::new(); + + // Pre-seed a distinctive fence key under the reserved nil-tenant entry. + let seeded = vec![7u8; fence::FENCE_KEY_LEN]; + let fence_ref = key(fence::FENCE_KEY_REF); + plugin.seed_value(&TenantId(Uuid::nil()), &fence_ref, None, &seeded); + + let svc = make_service_noop( + Arc::new(FakeSecretRepo::new()), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin.clone(), + ); + + svc.put( + &ctx, + &k, + SecretValue::from("v"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("write adopts the existing fence key"); + + let stored = plugin + .get(&ctx, &TenantId(Uuid::nil()), &fence_ref, None) + .await + .expect("plugin get ok") + .expect("fence key still present"); + assert_eq!( + stored.as_bytes(), + seeded.as_slice(), + "bootstrap must adopt the existing fence key, never overwrite it" + ); + // The write, fenced under the adopted key, verifies and reads back. + assert!(svc.get(&ctx, &k).await.expect("get").is_some()); +} + +#[tokio::test] +async fn stale_cached_key_self_heals_via_refresh_on_mismatch() { + use crate::domain::ports::metrics::FenceVerify; + use crate::domain::secret::fence; + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + + let plugin = FakePlugin::new(); + let repo = Arc::new(FakeSecretRepo::new()); + let metrics = FakeMetrics::new(); + let svc = make_service( + repo.clone(), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin.clone(), + metrics.clone(), + ); + + // Warm this replica's cache: the first write generates + caches a key. + let warm = key("warm"); + svc.put( + &ctx, + &warm, + SecretValue::from("w"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("warm the key cache"); + + // Another replica re-created the backend key entry (K2) and wrote a + // secret stamped under it; this replica still caches the old key. + let k2 = [7u8; 32]; + let fence_ref = key(fence::FENCE_KEY_REF); + plugin.seed_value(&TenantId(Uuid::nil()), &fence_ref, None, &k2); + let k = key("foreign"); + plugin.seed_value(&TenantId(tenant), &k, None, b"vx"); + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "foreign".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: Some(fence::compute_fp(&k2, b"vx")), + fp_key_id: Some(fence::CURRENT_FENCE_KEY_ID), + }); + + // The stale cached key fails the first verify; the one-shot refresh + // re-reads K2 from the backend and the read succeeds. + let got = svc.get(&ctx, &k).await.expect("get").expect("self-healed"); + assert_eq!(got.value.as_bytes(), b"vx"); + assert!( + metrics.fence_verifies().contains(&FenceVerify::Ok), + "refresh-on-mismatch must converge to Ok, got {:?}", + metrics.fence_verifies() + ); + assert!( + !metrics.fence_verifies().contains(&FenceVerify::Mismatch), + "a healed read must not count as a mismatch" + ); +} + +#[tokio::test] +async fn reaper_backfill_sweep_stamps_unread_seeded_rows() { + use crate::domain::ports::metrics::Outcome; + use crate::domain::secret::fence; + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + + let plugin = FakePlugin::new(); + plugin.seed_fence_key(); + let repo = Arc::new(FakeSecretRepo::new()); + let seed_row = |reference: &str| SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: reference.to_owned(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: None, + fp_key_id: None, + }; + repo.seed(seed_row("swept-a")); + repo.seed(seed_row("swept-b")); + // A seeded row whose backend value is missing: nothing to stamp. + repo.seed(seed_row("valueless")); + plugin.seed_value(&TenantId(tenant), &key("swept-a"), None, b"va"); + plugin.seed_value(&TenantId(tenant), &key("swept-b"), None, b"vb"); + let _ = ctx; // rows seeded directly; no API traffic in this test + + let metrics = FakeMetrics::new(); + let svc = make_service( + repo.clone(), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin, + metrics.clone(), + ); + + svc.reap_and_refresh().await; + + let rows = repo.rows(); + let fp_of = |reference: &str| { + rows.iter() + .find(|r| r.reference == reference) + .expect("row") + .value_fp + .clone() + }; + assert_eq!( + fp_of("swept-a").as_deref(), + Some(fence::compute_fp(&TEST_FENCE_KEY, b"va").as_slice()) + ); + assert!(fp_of("swept-b").is_some()); + assert!( + fp_of("valueless").is_none(), + "a row with no backend value stays unfenced (served-on-trust semantics)" + ); + assert_eq!( + metrics.fence_backfills(), + vec![Outcome::Success, Outcome::Success] + ); +} + +#[tokio::test] +async fn fence_key_reference_is_unreachable_through_the_api() { + use credstore_sdk::CredStorePluginClientV1; + + use crate::domain::secret::fence; + // The reserved entry has no metadata row, so no API path resolves it; a + // tenant writing the same reference gets an ordinary tenant-scoped secret + // that never collides with the nil-tenant key entry. + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let fence_ref = key(fence::FENCE_KEY_REF); + + let plugin = FakePlugin::new(); + plugin.seed_fence_key(); + let svc = make_service_noop( + Arc::new(FakeSecretRepo::new()), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + plugin.clone(), + ); + + // Not resolvable: no row exists for it. + assert!( + svc.get(&ctx, &fence_ref) + .await + .expect("get is Ok") + .is_none(), + "the reserved fence-key entry must not resolve through the API" + ); + + // A tenant PUT to the same reference is an ordinary tenant secret… + svc.put( + &ctx, + &fence_ref, + SecretValue::from("own-secret"), + WriteSpec::upsert(SharingMode::Tenant), + ) + .await + .expect("tenant-scoped write under the same name is fine"); + let got = svc + .get(&ctx, &fence_ref) + .await + .expect("get") + .expect("their own secret"); + assert_eq!(got.value.as_bytes(), b"own-secret"); + + // …and the nil-tenant key entry is untouched. + let nil_ctx = make_ctx(Uuid::nil(), Uuid::nil()); + let stored = plugin + .get(&nil_ctx, &TenantId(Uuid::nil()), &fence_ref, None) + .await + .expect("plugin get") + .expect("key entry present"); + assert_eq!( + stored.as_bytes(), + TEST_FENCE_KEY, + "a tenant write must never clobber the reserved key entry" + ); +} + +// ── review-finding follow-ups (#9) ─────────────────────────────────────────── + +#[tokio::test] +async fn delete_with_no_plugin_fails_without_marking_deprovisioning() { + use crate::domain::secret::test_support::NoPluginSelector; + // #9: DELETE must resolve the plugin BEFORE flipping the row to + // deprovisioning (symmetric with put's fail-fast). With no plugin the + // delete 503s and the row stays Active — the secret keeps resolving and + // the reference is not wedged. + let tenant = Uuid::new_v4(); + let subject = Uuid::new_v4(); + let ctx = make_ctx(subject, tenant); + let k = key("np-del"); + + let repo = Arc::new(FakeSecretRepo::new()); + repo.seed(SecretRow { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: "np-del".into(), + sharing: SharingMode::Tenant, + owner_id: OwnerId(subject), + status: SecretStatus::Active, + version: 1, + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: Some(vec![1u8; 32]), + fp_key_id: Some(1), + }); + let svc = Service::new( + repo.clone(), + Arc::new(FakeDir::single(tenant)), + mock_enforcer(), + Arc::new(NoPluginSelector), + catalog_type_resolver(), + Arc::new(NoopMetrics), + test_reaper_settings(), + ); + + let res = svc.delete(&ctx, &k, None).await; + assert!( + matches!(res, Err(DomainError::ServiceUnavailable { .. })), + "no plugin -> 503, got {res:?}" + ); + // The row must NOT have been flipped to deprovisioning. + let rows = repo.rows(); + assert_eq!(rows.len(), 1); + assert_eq!( + rows[0].status, + SecretStatus::Active, + "delete must not mark deprovisioning when no plugin is available" + ); +} diff --git a/gears/credstore/credstore/src/domain/secret/test_support.rs b/gears/credstore/credstore/src/domain/secret/test_support.rs new file mode 100644 index 000000000..71a78a895 --- /dev/null +++ b/gears/credstore/credstore/src/domain/secret/test_support.rs @@ -0,0 +1,1135 @@ +//! Shared test infrastructure for domain-layer unit tests. + +use std::collections::HashMap; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex}; + +use async_trait::async_trait; +use authz_resolver_sdk::constraints::{Constraint, InTenantSubtreePredicate, Predicate}; +use authz_resolver_sdk::models::{ + Capability, EvaluationRequest, EvaluationResponse, EvaluationResponseContext, +}; +use authz_resolver_sdk::{AuthZResolverClient, AuthZResolverError, PolicyEnforcer}; +use credstore_sdk::{ + CredStoreError, CredStorePluginClientV1, OwnerId, SecretRef, SecretValue, SharingMode, TenantId, +}; +use toolkit_security::{AccessScope, SecurityContext, pep_properties}; +use uuid::Uuid; + +use crate::domain::error::DomainError; +pub use crate::domain::ports::metrics::NoopMetrics; +use crate::domain::ports::metrics::{ + CredStoreMetricsPort, Dep, DepOp, FenceVerify, Outcome, ReadOutcome, SecretCounts, +}; +use crate::domain::ports::plugin::PluginSelector; +use crate::domain::resolver::TenantDirectory; +use crate::domain::secret::model::{NewSecret, SecretRow, SecretStatus}; +use crate::domain::secret::repo::SecretRepo; +use crate::domain::secret::type_resolver::{ResolvedSecretType, SecretTypeResolver}; +use crate::domain::secret::typing::reasons; +use time::OffsetDateTime; + +// ── SecurityContext helpers ─────────────────────────────────────────────────── + +/// Build a minimal [`SecurityContext`] for unit tests with custom subject and tenant. +/// +/// # Panics +/// +/// Panics if the builder fails (only possible on missing fields which we always supply). +#[must_use] +pub fn make_ctx(subject_id: Uuid, tenant_id: Uuid) -> SecurityContext { + SecurityContext::builder() + .subject_id(subject_id) + .subject_tenant_id(tenant_id) + .build() + .expect("test ctx") +} + +// ── mock PolicyEnforcer ─────────────────────────────────────────────────────── + +/// Permissive PDP fake: emits one `InTenantSubtree` permit rooted at the +/// caller's `subject_tenant_id` (the slot the PEP populates). The repo fakes +/// ignore scope contents, so this only has to compile to a valid scope under +/// `require_constraints(true)`. +struct MockAuthZResolver; + +#[async_trait] +impl AuthZResolverClient for MockAuthZResolver { + async fn evaluate( + &self, + request: EvaluationRequest, + ) -> Result { + let root = request + .subject + .properties + .get("tenant_id") + .and_then(serde_json::Value::as_str) + .and_then(|s| Uuid::parse_str(s).ok()) + .expect("MockAuthZResolver: subject.properties[\"tenant_id\"]; build ctx via make_ctx"); + Ok(EvaluationResponse { + decision: true, + context: EvaluationResponseContext { + constraints: vec![Constraint { + predicates: vec![Predicate::InTenantSubtree(InTenantSubtreePredicate::new( + pep_properties::OWNER_TENANT_ID, + root, + ))], + }], + ..Default::default() + }, + }) + } +} + +/// Permissive [`PolicyEnforcer`] for `Service` unit tests. +#[must_use] +pub fn mock_enforcer() -> PolicyEnforcer { + PolicyEnforcer::new(Arc::new(MockAuthZResolver)) + .with_capabilities(vec![Capability::TenantHierarchy]) +} + +/// PDP fake that always denies (`decision: false`). +struct DenyAuthZResolver; + +#[async_trait] +impl AuthZResolverClient for DenyAuthZResolver { + async fn evaluate( + &self, + _request: EvaluationRequest, + ) -> Result { + Ok(EvaluationResponse { + decision: false, + context: EvaluationResponseContext::default(), + }) + } +} + +/// Denying [`PolicyEnforcer`] — drives `scope_for` → `DomainError::AccessDenied`. +#[must_use] +pub fn deny_enforcer() -> PolicyEnforcer { + PolicyEnforcer::new(Arc::new(DenyAuthZResolver)) + .with_capabilities(vec![Capability::TenantHierarchy]) +} + +/// Type-aware PDP fake: permissive (like [`mock_enforcer`]) except for the +/// listed resource-type ids, which are denied. Records every evaluated +/// resource type so tests can assert what the PEP targeted. +pub struct TypeAwareAuthZResolver { + denied_types: Vec, + pub seen_resource_types: Mutex>, +} + +#[async_trait] +impl AuthZResolverClient for TypeAwareAuthZResolver { + async fn evaluate( + &self, + request: EvaluationRequest, + ) -> Result { + self.seen_resource_types + .lock() + .expect("lock") + .push(request.resource.resource_type.clone()); + if self.denied_types.contains(&request.resource.resource_type) { + return Ok(EvaluationResponse { + decision: false, + context: EvaluationResponseContext::default(), + }); + } + MockAuthZResolver.evaluate(request).await + } +} + +/// Enforcer denying exactly the given resource-type ids; returns the +/// resolver too so tests can inspect the evaluated types. +#[must_use] +pub fn type_deny_enforcer( + denied_types: Vec, +) -> (PolicyEnforcer, Arc) { + let resolver = Arc::new(TypeAwareAuthZResolver { + denied_types, + seen_resource_types: Mutex::new(Vec::new()), + }); + ( + PolicyEnforcer::new(resolver.clone()).with_capabilities(vec![Capability::TenantHierarchy]), + resolver, + ) +} + +/// PDP fake that always returns a transport failure. +struct FailAuthZResolver; + +#[async_trait] +impl AuthZResolverClient for FailAuthZResolver { + async fn evaluate( + &self, + _request: EvaluationRequest, + ) -> Result { + Err(AuthZResolverError::ServiceUnavailable( + "failing_enforcer: simulated PDP transport failure".to_owned(), + )) + } +} + +/// Failing [`PolicyEnforcer`] — drives `scope_for` → `DomainError::ServiceUnavailable`. +#[must_use] +pub fn failing_enforcer() -> PolicyEnforcer { + PolicyEnforcer::new(Arc::new(FailAuthZResolver)) + .with_capabilities(vec![Capability::TenantHierarchy]) +} + +// ── Fake type resolvers ─────────────────────────────────────────────────────── + +/// Catalog-backed [`SecretTypeResolver`]: resolves the built-in catalog +/// types by their deterministic UUID, mirroring what the production +/// resolver returns for the registry-seeded schemas. Unknown UUIDs map to +/// `UNKNOWN_SECRET_TYPE`, like an unregistered type. +pub struct CatalogTypeResolver; + +#[async_trait] +impl SecretTypeResolver for CatalogTypeResolver { + async fn resolve(&self, type_uuid: Uuid) -> Result { + credstore_sdk::SECRET_TYPE_CATALOG + .iter() + .find(|d| credstore_sdk::types::type_uuid(d.gts_id) == Some(type_uuid)) + .map(|d| ResolvedSecretType { + gts_id: d.gts_id.to_owned(), + traits: d.traits(), + }) + .ok_or_else(|| DomainError::TypeViolation { + field: "type", + reason: reasons::UNKNOWN_SECRET_TYPE, + detail: format!("secret type {type_uuid} is not registered"), + }) + } +} + +/// Catalog-backed resolver as an `Arc` — the +/// default for `Service` unit tests. +#[must_use] +pub fn catalog_type_resolver() -> Arc { + Arc::new(CatalogTypeResolver) +} + +/// [`SecretTypeResolver`] that always fails with `ServiceUnavailable` — +/// drives the registry-outage (503) paths. +pub struct FailingTypeResolver; + +#[async_trait] +impl SecretTypeResolver for FailingTypeResolver { + async fn resolve(&self, _type_uuid: Uuid) -> Result { + Err(DomainError::ServiceUnavailable { + detail: "types-registry unavailable".to_owned(), + retry_after: None, + cause: None, + }) + } +} + +// ── FakeDir ─────────────────────────────────────────────────────────────────── + +/// Returns a preset ancestor chain (self first, root last). +pub struct FakeDir { + chain: Vec, +} + +impl FakeDir { + #[must_use] + pub fn new(chain: Vec) -> Self { + Self { chain } + } + + /// Single-tenant chain (only self). + #[must_use] + pub fn single(id: Uuid) -> Self { + Self { chain: vec![id] } + } +} + +#[async_trait] +impl TenantDirectory for FakeDir { + async fn ancestor_chain( + &self, + _ctx: &SecurityContext, + _req: TenantId, + ) -> Result, DomainError> { + Ok(self.chain.clone()) + } +} + +// ── FakePlugin ──────────────────────────────────────────────────────────────── + +/// Key: `(tenant_id, reference, owner_id)`. +type PluginKey = (Uuid, String, Option); + +/// In-memory plugin store. +pub struct FakePlugin { + store: Mutex>>, + /// Number of upcoming `put` calls that fail before the plugin recovers, + /// simulating a transient backend outage mid create-saga. + put_failures: Mutex, + /// Number of upcoming `delete` calls that fail before the plugin + /// recovers, simulating a transient backend outage mid delete-saga. + delete_failures: Mutex, + /// When set, every `get` returns [`CredStoreError::AccessDenied`], + /// modelling a backend whose own ACLs reject a read the gear's PDP has + /// already allowed. + get_denied: bool, + /// Count of backend reads of the reserved fence-key entry — lets tests + /// assert the mismatch-triggered refresh does not re-read the key on every + /// poisoned get (the cache-thrash guard). + fence_key_gets: AtomicUsize, +} + +impl FakePlugin { + #[must_use] + pub fn new() -> Arc { + Arc::new(Self { + store: Mutex::new(HashMap::new()), + put_failures: Mutex::new(0), + delete_failures: Mutex::new(0), + get_denied: false, + fence_key_gets: AtomicUsize::new(0), + }) + } + + /// Plugin that fails the next `n` `put` calls, then behaves normally. + #[must_use] + pub fn with_put_failures(n: usize) -> Arc { + Arc::new(Self { + store: Mutex::new(HashMap::new()), + put_failures: Mutex::new(n), + delete_failures: Mutex::new(0), + get_denied: false, + fence_key_gets: AtomicUsize::new(0), + }) + } + + /// Plugin that fails the next `n` `delete` calls, then behaves normally. + #[must_use] + pub fn with_delete_failures(n: usize) -> Arc { + Arc::new(Self { + store: Mutex::new(HashMap::new()), + put_failures: Mutex::new(0), + delete_failures: Mutex::new(n), + get_denied: false, + fence_key_gets: AtomicUsize::new(0), + }) + } + + /// Plugin whose every `get` denies — models a backend ACL rejecting a read + /// the gear's PDP already allowed. + #[must_use] + pub fn with_get_denied() -> Arc { + Arc::new(Self { + store: Mutex::new(HashMap::new()), + put_failures: Mutex::new(0), + delete_failures: Mutex::new(0), + get_denied: true, + fence_key_gets: AtomicUsize::new(0), + }) + } + + /// Number of backend reads of the reserved fence-key entry so far. + /// + /// # Panics + /// + /// Never panics. + #[must_use] + pub fn fence_key_gets(&self) -> usize { + self.fence_key_gets.load(Ordering::Relaxed) + } + + /// True when the store holds a value for `(tenant, key, owner)`. + /// + /// # Panics + /// + /// Panics if the internal mutex is poisoned. + #[must_use] + pub fn contains( + &self, + tenant_id: &TenantId, + key: &SecretRef, + owner_id: Option<&OwnerId>, + ) -> bool { + let k = Self::key(tenant_id, key, owner_id); + self.store.lock().expect("lock").contains_key(&k) + } + + /// Insert a value directly into the store, bypassing the failure + /// injection — used to pre-seed the reserved fence-key entry so tests + /// that inject `put` failures exercise the *value* write, not the fence + /// key bootstrap. + /// + /// # Panics + /// + /// Panics if the internal mutex is poisoned. + pub fn seed_value( + &self, + tenant_id: &TenantId, + key: &SecretRef, + owner_id: Option<&OwnerId>, + bytes: &[u8], + ) { + let k = Self::key(tenant_id, key, owner_id); + self.store.lock().expect("lock").insert(k, bytes.to_vec()); + } + + /// Pre-seed the reserved fence-key entry (32 fixed bytes), so the fence + /// bootstrap is a pure read and never consumes an injected `put` failure. + /// + /// # Panics + /// + /// Panics if the fence key reference is invalid (it is a tested constant). + pub fn seed_fence_key(&self) { + let key_ref = SecretRef::new(crate::domain::secret::fence::FENCE_KEY_REF) + .expect("valid fence key ref"); + self.seed_value( + &TenantId(Uuid::nil()), + &key_ref, + None, + &[42u8; crate::domain::secret::fence::FENCE_KEY_LEN], + ); + } + + fn key(tenant_id: &TenantId, key: &SecretRef, owner_id: Option<&OwnerId>) -> PluginKey { + (tenant_id.0, key.as_ref().to_owned(), owner_id.map(|o| o.0)) + } +} + +impl Default for FakePlugin { + fn default() -> Self { + Self { + store: Mutex::new(HashMap::new()), + put_failures: Mutex::new(0), + delete_failures: Mutex::new(0), + get_denied: false, + fence_key_gets: AtomicUsize::new(0), + } + } +} + +#[async_trait] +impl CredStorePluginClientV1 for FakePlugin { + async fn get( + &self, + _ctx: &SecurityContext, + tenant_id: &TenantId, + key: &SecretRef, + owner_id: Option<&OwnerId>, + ) -> Result, CredStoreError> { + if key.as_ref() == crate::domain::secret::fence::FENCE_KEY_REF { + self.fence_key_gets.fetch_add(1, Ordering::Relaxed); + } + if self.get_denied { + return Err(CredStoreError::AccessDenied); + } + let k = Self::key(tenant_id, key, owner_id); + let guard = self.store.lock().expect("lock"); + Ok(guard.get(&k).map(|v| SecretValue::new(v.clone()))) + } + + async fn put( + &self, + _ctx: &SecurityContext, + tenant_id: &TenantId, + key: &SecretRef, + value: SecretValue, + owner_id: Option<&OwnerId>, + ) -> Result<(), CredStoreError> { + { + let mut remaining = self.put_failures.lock().expect("lock"); + if *remaining > 0 { + *remaining -= 1; + return Err(CredStoreError::Internal( + "simulated backend put failure".to_owned(), + )); + } + } + let k = Self::key(tenant_id, key, owner_id); + self.store + .lock() + .expect("lock") + .insert(k, value.as_bytes().to_vec()); + Ok(()) + } + + async fn delete( + &self, + _ctx: &SecurityContext, + tenant_id: &TenantId, + key: &SecretRef, + owner_id: Option<&OwnerId>, + ) -> Result<(), CredStoreError> { + { + let mut remaining = self.delete_failures.lock().expect("lock"); + if *remaining > 0 { + *remaining -= 1; + return Err(CredStoreError::Internal( + "simulated backend delete failure".to_owned(), + )); + } + } + let k = Self::key(tenant_id, key, owner_id); + self.store.lock().expect("lock").remove(&k); + Ok(()) + } +} + +// ── FakePluginSelector ──────────────────────────────────────────────────────── + +pub struct FakePluginSelector { + plugin: Arc, +} + +impl FakePluginSelector { + #[must_use] + pub fn new(plugin: Arc) -> Self { + Self { plugin } + } +} + +#[async_trait] +impl PluginSelector for FakePluginSelector { + async fn resolve(&self) -> Result, DomainError> { + Ok(self.plugin.clone()) + } +} + +/// [`PluginSelector`] that always fails to resolve a plugin — models the +/// `NoPluginAvailable` (misconfigured / unregistered backend) 503 path. +pub struct NoPluginSelector; + +#[async_trait] +impl PluginSelector for NoPluginSelector { + async fn resolve(&self) -> Result, DomainError> { + Err(DomainError::ServiceUnavailable { + detail: "no storage plugin registered".to_owned(), + retry_after: None, + cause: None, + }) + } +} + +// ── FakeSecretRepo ──────────────────────────────────────────────────────────── + +/// In-memory [`SecretRepo`]. +/// +/// `scope_allows` controls the result of [`SecretRepo::scope_includes_tenant`]. +// Independent failure-injection toggles for a test double, not a state machine. +#[allow(clippy::struct_excessive_bools)] +pub struct FakeSecretRepo { + rows: Mutex>, + pub scope_allows: bool, + /// When set, a unique-violation in `insert_provisioning` first promotes the + /// conflicting Provisioning row(s) to Active before returning Conflict — + /// simulating the create-race winner finishing its saga, so the service's + /// bounded retry can resolve to the update path deterministically. + promote_on_conflict: bool, + /// When set, `delete_by_id` returns an error — simulating a DB failure on + /// the create-saga rollback path (the reference then stays wedged). + fail_delete: bool, + /// When set, `delete_by_id` matches 0 rows (`NotFound`) — simulating a row + /// that moved/vanished between `find_own` and the conditional delete. + delete_not_found: bool, + /// When set, `mark_deprovisioning` matches 0 rows — simulating a row that + /// moved/vanished between `find_own` and the gated status flip. + mark_not_found: bool, + /// When set, `touch` matches 0 rows (`Ok(None)`) — simulating a row that was + /// concurrently deleted/reaped between `find_for_write` and the version bump + /// on the overwrite path. + touch_not_found: bool, + /// When set, `list_stale_pending` returns each provisioning row as a stale + /// snapshot but atomically flips the stored row to `Active` — simulating a + /// slow create saga's `mark_active` landing between the reaper's list and + /// its status-gated delete. The reaper must then leave the now-active row + /// (and its backend value) alone. + promote_provisioning_on_list: bool, +} + +impl FakeSecretRepo { + #[must_use] + pub fn new() -> Self { + Self { + rows: Mutex::new(Vec::new()), + scope_allows: true, + promote_on_conflict: false, + fail_delete: false, + delete_not_found: false, + mark_not_found: false, + touch_not_found: false, + promote_provisioning_on_list: false, + } + } + + /// Repo whose `list_stale_pending` flips each returned provisioning row to + /// `Active` — simulates a slow create saga winning the race against the + /// reaper between the list and the status-gated delete. + #[must_use] + pub fn with_provisioning_promoted_on_list() -> Self { + Self { + promote_provisioning_on_list: true, + ..Self::new() + } + } + + #[must_use] + pub fn with_scope_allows(scope_allows: bool) -> Self { + Self { + scope_allows, + ..Self::new() + } + } + + #[must_use] + pub fn with_promote_on_conflict(promote_on_conflict: bool) -> Self { + Self { + promote_on_conflict, + ..Self::new() + } + } + + /// Repo whose `delete_by_id` always fails — exercises the rollback-failed path. + #[must_use] + pub fn with_delete_failure() -> Self { + Self { + fail_delete: true, + ..Self::new() + } + } + + /// Repo whose `delete_by_id` matches 0 rows — simulates a row that moved or + /// vanished between `find_own` and the conditional delete. + #[must_use] + pub fn with_delete_not_found() -> Self { + Self { + delete_not_found: true, + ..Self::new() + } + } + + /// Repo whose `touch` matches 0 rows (`Ok(None)`) — simulates a row that was + /// concurrently deleted/reaped between `find_for_write` and the version bump + /// on the overwrite path. + #[must_use] + pub fn with_touch_not_found() -> Self { + Self { + touch_not_found: true, + ..Self::new() + } + } + + /// Repo whose `mark_deprovisioning` matches 0 rows — simulates a row that + /// moved or vanished between `find_own` and the gated status flip. + #[must_use] + pub fn with_mark_not_found() -> Self { + Self { + mark_not_found: true, + ..Self::new() + } + } + + /// Seed rows directly (for pre-seeding parent/inherited state). + /// + /// # Panics + /// + /// Panics if the internal mutex is poisoned. + pub fn seed(&self, row: SecretRow) { + self.rows.lock().expect("lock").push(row); + } + + /// Snapshot of all rows (for asserting saga state). + /// + /// # Panics + /// + /// Panics if the internal mutex is poisoned. + #[must_use] + pub fn rows(&self) -> Vec { + self.rows.lock().expect("lock").clone() + } +} + +impl Default for FakeSecretRepo { + fn default() -> Self { + Self::new() + } +} + +#[async_trait] +impl SecretRepo for FakeSecretRepo { + async fn resolve_for_get( + &self, + req_tenant: TenantId, + subject: OwnerId, + key: &SecretRef, + chain: &[Uuid], + ) -> Result, DomainError> { + let rows = self.rows.lock().expect("lock"); + let key_str = key.as_ref(); + + // Winner: closest tenant position; private beats non-private at same level. + // Candidates: active, reference matches, tenant in chain, sharing rules. + let pos = |t: Uuid| chain.iter().position(|c| *c == t).unwrap_or(usize::MAX); + let best = rows + .iter() + .filter(|r| { + r.status == SecretStatus::Active + && r.expires_at.is_none_or(|at| at > OffsetDateTime::now_utc()) + && r.reference == key_str + && chain.contains(&r.tenant_id.0) + && match r.sharing { + SharingMode::Private => r.owner_id.0 == subject.0, + SharingMode::Tenant => r.tenant_id == req_tenant, + SharingMode::Shared => true, + } + }) + .min_by(|a, b| { + pos(a.tenant_id.0).cmp(&pos(b.tenant_id.0)).then( + (a.sharing != SharingMode::Private).cmp(&(b.sharing != SharingMode::Private)), + ) + }); + Ok(best.cloned()) + } + + async fn insert_provisioning( + &self, + _scope: &AccessScope, + new: &NewSecret, + ) -> Result<(), DomainError> { + let mut rows = self.rows.lock().expect("lock"); + // Enforce uniqueness: (tenant_id, reference, sharing class). + // For private: (tenant_id, reference, owner_id) must be unique. + // For tenant/shared: (tenant_id, reference) must be unique among non-private. + let conflict = rows.iter().any(|r| { + r.tenant_id == new.tenant_id + && r.reference == new.reference.as_ref() + && match new.sharing { + SharingMode::Private => { + r.sharing == SharingMode::Private && r.owner_id == new.owner_id + } + _ => r.sharing != SharingMode::Private, + } + }); + if conflict { + if self.promote_on_conflict { + for r in rows.iter_mut().filter(|r| { + r.tenant_id == new.tenant_id + && r.reference == new.reference.as_ref() + && r.status == SecretStatus::Provisioning + }) { + r.status = SecretStatus::Active; + } + } + return Err(DomainError::Conflict); + } + rows.push(SecretRow { + id: new.id, + tenant_id: new.tenant_id, + reference: new.reference.as_ref().to_owned(), + sharing: new.sharing, + owner_id: new.owner_id, + status: SecretStatus::Provisioning, + version: 1, + secret_type_uuid: new.secret_type_uuid, + expires_at: new.expires_at, + value_fp: Some(new.value_fp.clone()), + fp_key_id: Some(new.fp_key_id), + }); + Ok(()) + } + + async fn mark_active(&self, _scope: &AccessScope, id: Uuid) -> Result<(), DomainError> { + let mut rows = self.rows.lock().expect("lock"); + let row = rows.iter_mut().find(|r| r.id == id); + match row { + Some(r) => { + r.status = SecretStatus::Active; + Ok(()) + } + None => Err(DomainError::Conflict), + } + } + + async fn touch( + &self, + _scope: &AccessScope, + id: Uuid, + sharing: SharingMode, + expected_version: Option, + expires_at: Option, + value_fp: Vec, + ) -> Result, DomainError> { + if self.touch_not_found { + return Ok(None); + } + let mut rows = self.rows.lock().expect("lock"); + let row = rows.iter_mut().find(|r| { + r.id == id + && r.status == SecretStatus::Active + && expected_version.is_none_or(|v| r.version == v) + }); + match row { + Some(r) => { + r.version += 1; + r.sharing = sharing; + r.expires_at = expires_at; + r.value_fp = Some(value_fp); + r.fp_key_id = Some(crate::domain::secret::fence::CURRENT_FENCE_KEY_ID); + Ok(Some(r.clone())) + } + None => Ok(None), + } + } + + async fn backfill_fp( + &self, + id: Uuid, + value_fp: Vec, + fp_key_id: i16, + ) -> Result { + let mut rows = self.rows.lock().expect("lock"); + let row = rows.iter_mut().find(|r| r.id == id && r.value_fp.is_none()); + match row { + Some(r) => { + r.value_fp = Some(value_fp); + r.fp_key_id = Some(fp_key_id); + Ok(true) + } + None => Ok(false), + } + } + + async fn list_unfenced(&self, limit: u64) -> Result, DomainError> { + let rows = self.rows.lock().expect("lock"); + Ok(rows + .iter() + .filter(|r| r.status == SecretStatus::Active && r.value_fp.is_none()) + .take(usize::try_from(limit).unwrap_or(usize::MAX)) + .cloned() + .collect()) + } + + async fn find_own( + &self, + _scope: &AccessScope, + tenant: TenantId, + subject: OwnerId, + key: &SecretRef, + ) -> Result, DomainError> { + let rows = self.rows.lock().expect("lock"); + let key_str = key.as_ref(); + // Prefer private row. Active + deprovisioning (saga resume), never + // provisioning — mirrors the SQL implementation. + let best = rows + .iter() + .filter(|r| { + r.tenant_id == tenant + && r.reference == key_str + && matches!( + r.status, + SecretStatus::Active | SecretStatus::Deprovisioning + ) + && match r.sharing { + SharingMode::Private => r.owner_id.0 == subject.0, + _ => true, + } + }) + .min_by_key(|r| i32::from(r.sharing != SharingMode::Private)); + Ok(best.cloned()) + } + + async fn find_for_write( + &self, + _scope: &AccessScope, + tenant: TenantId, + subject: OwnerId, + key: &SecretRef, + sharing: SharingMode, + ) -> Result, DomainError> { + let rows = self.rows.lock().expect("lock"); + let key_str = key.as_ref(); + // Address only the target sharing class — private and non-private secrets + // coexist under one (tenant, ref); a write of one class ignores the other. + let row = rows.iter().find(|r| { + r.tenant_id == tenant + && r.reference == key_str + && r.status == SecretStatus::Active + && match sharing { + SharingMode::Private => { + r.sharing == SharingMode::Private && r.owner_id == subject + } + _ => r.sharing != SharingMode::Private, + } + }); + Ok(row.cloned()) + } + + async fn delete_by_id( + &self, + _scope: &AccessScope, + id: Uuid, + expected_version: Option, + ) -> Result<(), DomainError> { + if self.fail_delete { + return Err(DomainError::internal("simulated delete failure")); + } + if self.delete_not_found { + return Err(DomainError::NotFound); + } + let mut rows = self.rows.lock().expect("lock"); + let len_before = rows.len(); + rows.retain(|r| !(r.id == id && expected_version.is_none_or(|v| r.version == v))); + if rows.len() == len_before { + Err(DomainError::NotFound) + } else { + Ok(()) + } + } + + async fn mark_deprovisioning( + &self, + _scope: &AccessScope, + id: Uuid, + expected_version: Option, + ) -> Result { + if self.mark_not_found { + return Ok(false); + } + let mut rows = self.rows.lock().expect("lock"); + let row = rows.iter_mut().find(|r| { + r.id == id + && r.status == SecretStatus::Active + && expected_version.is_none_or(|v| r.version == v) + }); + match row { + Some(r) => { + r.status = SecretStatus::Deprovisioning; + Ok(true) + } + None => Ok(false), + } + } + + async fn list_stale_pending( + &self, + _provisioning_older_than_secs: u64, + _deprovisioning_older_than_secs: u64, + limit: u64, + ) -> Result, DomainError> { + // The fake has no timestamps: every non-active row counts as stale. + let mut rows = self.rows.lock().expect("lock"); + let stale: Vec = rows + .iter() + .filter(|r| r.status != SecretStatus::Active) + .take(usize::try_from(limit).unwrap_or(usize::MAX)) + .cloned() + .collect(); + if self.promote_provisioning_on_list { + // Simulate mark_active landing between list and reap: the returned + // snapshot still reads Provisioning, but the stored row is now Active. + for r in rows.iter_mut() { + if r.status == SecretStatus::Provisioning { + r.status = SecretStatus::Active; + } + } + } + Ok(stale) + } + + async fn reap_by_id(&self, id: Uuid, expected: SecretStatus) -> Result { + if self.fail_delete { + return Err(DomainError::internal("simulated reap failure")); + } + let mut rows = self.rows.lock().expect("lock"); + let before = rows.len(); + // Status-gated like the real repo: only remove the row if it still + // holds the status the reaper observed. + rows.retain(|r| !(r.id == id && r.status == expected)); + Ok(rows.len() != before) + } + + async fn mark_expired_deprovisioning(&self) -> Result { + let now = OffsetDateTime::now_utc(); + let mut rows = self.rows.lock().expect("lock"); + let mut flipped = 0u64; + for r in rows.iter_mut() { + if r.status == SecretStatus::Active && r.expires_at.is_some_and(|at| at <= now) { + r.status = SecretStatus::Deprovisioning; + flipped += 1; + } + } + Ok(flipped) + } + + async fn inventory(&self) -> Result { + let rows = self.rows.lock().expect("lock"); + let mut counts = SecretCounts::default(); + let tenants: std::collections::HashSet = rows + .iter() + .filter(|r| r.status == SecretStatus::Active) + .map(|r| r.tenant_id.0) + .collect(); + #[allow(clippy::cast_possible_wrap)] + let tenant_count = tenants.len() as i64; + counts.tenants = tenant_count; + for r in rows.iter() { + match r.status { + SecretStatus::Provisioning => counts.provisioning += 1, + SecretStatus::Deprovisioning => counts.deprovisioning += 1, + SecretStatus::Active => match r.sharing { + SharingMode::Private => counts.private += 1, + SharingMode::Tenant => counts.tenant += 1, + SharingMode::Shared => counts.shared += 1, + }, + } + } + Ok(counts) + } + + async fn scope_includes_tenant( + &self, + _scope: &AccessScope, + _tenant: Uuid, + ) -> Result { + Ok(self.scope_allows) + } +} + +// ── FakeMetrics ─────────────────────────────────────────────────────────────── + +/// Recording metrics fake for assertions in tests. +pub struct FakeMetrics { + pub cross_tenant_denied_count: Mutex, + pub read_outcomes: Mutex>, + pub deps: Mutex>, + pub provisioning_rollbacks: Mutex>, + pub provisioning_reaped_total: Mutex, + pub deprovisioning_reaped_total: Mutex, + pub fence_verifies: Mutex>, + pub fence_backfills: Mutex>, +} + +impl FakeMetrics { + #[must_use] + pub fn new() -> Arc { + Arc::new(Self::default()) + } + + /// Total secrets counted by `provisioning_reaped`. + /// + /// # Panics + /// + /// Panics if the internal mutex is poisoned. + pub fn provisioning_reaped_total(&self) -> u64 { + *self.provisioning_reaped_total.lock().expect("lock") + } + + /// Total secrets counted by `deprovisioning_reaped`. + /// + /// # Panics + /// + /// Panics if the internal mutex is poisoned. + pub fn deprovisioning_reaped_total(&self) -> u64 { + *self.deprovisioning_reaped_total.lock().expect("lock") + } + + /// Returns all recorded provisioning-rollback outcomes. + /// + /// # Panics + /// + /// Panics if the internal mutex is poisoned. + pub fn provisioning_rollbacks(&self) -> Vec { + self.provisioning_rollbacks.lock().expect("lock").clone() + } + + /// Returns all recorded dependency `(dep, op, outcome)` tuples. + /// + /// # Panics + /// + /// Panics if the internal mutex is poisoned. + pub fn deps(&self) -> Vec<(Dep, DepOp, Outcome)> { + self.deps.lock().expect("lock").clone() + } + + /// Returns the number of times [`CredStoreMetricsPort::cross_tenant_denied`] was called. + /// + /// # Panics + /// + /// Panics if the internal mutex is poisoned (only possible after a panic in another thread). + pub fn cross_tenant_denied_count(&self) -> u64 { + *self.cross_tenant_denied_count.lock().expect("lock") + } + + /// Returns the last recorded [`ReadOutcome`], if any. + /// + /// # Panics + /// + /// Panics if the internal mutex is poisoned. + pub fn last_read_outcome(&self) -> Option { + self.read_outcomes.lock().expect("lock").last().copied() + } + + /// Returns all recorded fence-verify verdicts. + /// + /// # Panics + /// + /// Panics if the internal mutex is poisoned. + pub fn fence_verifies(&self) -> Vec { + self.fence_verifies.lock().expect("lock").clone() + } + + /// Returns all recorded fence-backfill outcomes. + /// + /// # Panics + /// + /// Panics if the internal mutex is poisoned. + pub fn fence_backfills(&self) -> Vec { + self.fence_backfills.lock().expect("lock").clone() + } +} + +impl Default for FakeMetrics { + fn default() -> Self { + Self { + cross_tenant_denied_count: Mutex::new(0), + read_outcomes: Mutex::new(Vec::new()), + deps: Mutex::new(Vec::new()), + provisioning_rollbacks: Mutex::new(Vec::new()), + provisioning_reaped_total: Mutex::new(0), + deprovisioning_reaped_total: Mutex::new(0), + fence_verifies: Mutex::new(Vec::new()), + fence_backfills: Mutex::new(Vec::new()), + } + } +} + +impl CredStoreMetricsPort for FakeMetrics { + fn record_inventory(&self, _counts: SecretCounts) {} + fn read_outcome(&self, outcome: ReadOutcome) { + self.read_outcomes.lock().expect("lock").push(outcome); + } + fn walkup_depth(&self, _depth: u64) {} + fn dependency(&self, dep: Dep, op: DepOp, outcome: Outcome, _secs: f64) { + self.deps.lock().expect("lock").push((dep, op, outcome)); + } + fn provisioning_reaped(&self, n: u64) { + *self.provisioning_reaped_total.lock().expect("lock") += n; + } + fn deprovisioning_reaped(&self, n: u64) { + *self.deprovisioning_reaped_total.lock().expect("lock") += n; + } + fn provisioning_rollback(&self, outcome: Outcome) { + self.provisioning_rollbacks + .lock() + .expect("lock") + .push(outcome); + } + fn cross_tenant_denied(&self) { + *self.cross_tenant_denied_count.lock().expect("lock") += 1; + } + fn fence_verify(&self, outcome: FenceVerify) { + self.fence_verifies.lock().expect("lock").push(outcome); + } + fn fence_backfill(&self, outcome: Outcome) { + self.fence_backfills.lock().expect("lock").push(outcome); + } +} diff --git a/gears/credstore/credstore/src/domain/secret/type_resolver.rs b/gears/credstore/credstore/src/domain/secret/type_resolver.rs new file mode 100644 index 000000000..e42fb3838 --- /dev/null +++ b/gears/credstore/credstore/src/domain/secret/type_resolver.rs @@ -0,0 +1,59 @@ +//! Registry-driven secret-type resolution — the runtime source of truth +//! for a secret type's PDP resource id and enforceable traits. +//! +//! Secrets store their type as the deterministic v5 UUID of the type's GTS +//! id (`credstore_sdk::type_uuid`). Every operation resolves that UUID +//! against the GTS types-registry to recover the full type id (the PDP +//! resource the single authz evaluation targets) and the effective traits +//! the write path enforces. New secret types are added by registering a +//! GTS schema derived from `gts.cf.core.credstore.secret.v1~` — no +//! credstore release involved; the compiled-in catalog only seeds the +//! built-in schemas. +//! +//! This module owns the **trait abstraction**; the production +//! implementation ([`crate::infra::types_registry::GtsSecretTypeResolver`]) +//! wraps `types_registry_sdk::TypesRegistryClient` resolved from +//! `ClientHub`. The resolver deliberately holds no cache of its own: the +//! registry's local client already keeps a TTL cache of `Arc` +//! keyed by id and UUID, so a per-operation resolve is one cached lookup. + +use async_trait::async_trait; +use credstore_sdk::SecretTypeTraits; +use toolkit_macros::domain_model; +use uuid::Uuid; + +use crate::domain::error::DomainError; + +/// A secret type resolved from the types-registry by its deterministic +/// v5 UUID (the stored representation). +#[domain_model] +#[derive(Debug, Clone, PartialEq)] +pub struct ResolvedSecretType { + /// Full chained GTS type id — the PDP resource type and the + /// wire-visible type name (e.g. + /// `gts.cf.core.credstore.secret.v1~cf.core.credstore.api_key.v1~`). + pub gts_id: String, + /// Effective enforcement traits (chain-merged; leaf-declared values + /// win, the secret base type fills the rest). + pub traits: SecretTypeTraits, +} + +/// Per-operation secret-type resolution barrier. +/// +/// Implementations MUST fail closed: a type that cannot be positively +/// resolved to a registered schema descending from the credstore secret +/// base type never reaches trait validation or the PDP. +#[async_trait] +pub trait SecretTypeResolver: Send + Sync { + /// Resolve `type_uuid` to its GTS type id and effective traits. + /// + /// # Errors + /// + /// * [`DomainError::TypeViolation`] (reason `UNKNOWN_SECRET_TYPE`) — + /// no type-schema is registered under this UUID, or the registered + /// schema does not descend from the credstore secret base type. + /// * [`DomainError::ServiceUnavailable`] — the types-registry is + /// unreachable, times out, or the registered schema's effective + /// traits fail to resolve into [`SecretTypeTraits`]. + async fn resolve(&self, type_uuid: Uuid) -> Result; +} diff --git a/gears/credstore/credstore/src/domain/secret/typing.rs b/gears/credstore/credstore/src/domain/secret/typing.rs new file mode 100644 index 000000000..290064a4b --- /dev/null +++ b/gears/credstore/credstore/src/domain/secret/typing.rs @@ -0,0 +1,162 @@ +//! Secret-type trait enforcement (design §5.4, write-path checks). +//! +//! Validates a write against the type's registry-resolved traits +//! ([`credstore_sdk::SecretTypeTraits`], see +//! [`crate::domain::secret::type_resolver`]): permitted sharing modes, +//! value size, UTF-8 requirement, the `value_schema` JSON Schema trait, +//! and the expiry gate. Violations map to [`DomainError::TypeViolation`] +//! with a stable reason code (canonical 400 on the wire). + +use credstore_sdk::{SecretTypeTraits, SecretValue, SharingMode}; +use time::OffsetDateTime; + +use crate::domain::error::DomainError; + +/// Stable reason codes (wire-visible in the canonical field violation). +pub mod reasons { + pub const UNKNOWN_SECRET_TYPE: &str = "UNKNOWN_SECRET_TYPE"; + pub const SHARING_NOT_ALLOWED_FOR_TYPE: &str = "SHARING_NOT_ALLOWED_FOR_TYPE"; + pub const VALUE_TOO_LARGE: &str = "VALUE_TOO_LARGE"; + pub const VALUE_NOT_UTF8: &str = "VALUE_NOT_UTF8"; + pub const VALUE_SCHEMA_VIOLATION: &str = "VALUE_SCHEMA_VIOLATION"; + pub const EXPIRY_NOT_SUPPORTED_FOR_TYPE: &str = "EXPIRY_NOT_SUPPORTED_FOR_TYPE"; + pub const EXPIRY_IN_THE_PAST: &str = "EXPIRY_IN_THE_PAST"; + pub const TYPE_IMMUTABLE: &str = "TYPE_IMMUTABLE"; +} + +fn violation(field: &'static str, reason: &'static str, detail: String) -> DomainError { + DomainError::TypeViolation { + field, + reason, + detail, + } +} + +/// Validate a write against the type's resolved traits. `type_id` is the +/// type's full GTS id, used in wire-visible violation details. +/// +/// `expires_at` semantics: permitted only for `expirable` types; a value in +/// the past is rejected (it would create a secret that never resolves). +/// +/// # Errors +/// +/// Returns [`DomainError::TypeViolation`] with a stable reason on the first +/// violated trait, or [`DomainError::ServiceUnavailable`] when the type's +/// registered `value_schema` trait fails to compile (a broken registration, +/// not a caller error — fail closed). +pub fn validate_write( + type_id: &str, + traits: &SecretTypeTraits, + sharing: SharingMode, + value: &SecretValue, + expires_at: Option, +) -> Result<(), DomainError> { + if !traits.allows_sharing(sharing) { + return Err(violation( + "sharing", + reasons::SHARING_NOT_ALLOWED_FOR_TYPE, + format!("sharing mode {sharing:?} is not permitted for secret type '{type_id}'"), + )); + } + + // A value too large for u64 is definitely over any declared limit. + let len = u64::try_from(value.as_bytes().len()).unwrap_or(u64::MAX); + if let Some(max) = traits.max_size_bytes + && len > max + { + return Err(violation( + "value", + reasons::VALUE_TOO_LARGE, + format!("value of {len} bytes exceeds the {max}-byte limit of secret type '{type_id}'"), + )); + } + + if traits.utf8_only && std::str::from_utf8(value.as_bytes()).is_err() { + return Err(violation( + "value", + reasons::VALUE_NOT_UTF8, + format!("secret type '{type_id}' requires a valid UTF-8 value"), + )); + } + + if let Some(schema) = traits.value_schema.as_ref() { + validate_value_schema(type_id, schema, value)?; + } + + match expires_at { + Some(_) if !traits.expirable => { + return Err(violation( + "expires_at", + reasons::EXPIRY_NOT_SUPPORTED_FOR_TYPE, + format!("secret type '{type_id}' does not support expiry"), + )); + } + Some(at) if at <= OffsetDateTime::now_utc() => { + return Err(violation( + "expires_at", + reasons::EXPIRY_IN_THE_PAST, + "expires_at must be in the future".to_owned(), + )); + } + _ => {} + } + + Ok(()) +} + +/// Validate the value against the type's `value_schema` trait. The value +/// must parse as JSON; violation details never echo the value itself (only +/// schema paths), preserving the no-secret-logging posture. +/// +/// The validator is compiled per call: schemas are dynamic (they arrive +/// with the registry resolution, which the registry client TTL-caches), +/// small, and only types that declare one pay the cost. +fn validate_value_schema( + type_id: &str, + schema: &serde_json::Value, + value: &SecretValue, +) -> Result<(), DomainError> { + let parsed: serde_json::Value = serde_json::from_slice(value.as_bytes()).map_err(|_| { + violation( + "value", + reasons::VALUE_SCHEMA_VIOLATION, + format!("secret type '{type_id}' requires a JSON value matching its schema"), + ) + })?; + + // A non-compiling schema is a broken type registration (the registry + // validated the trait as JSON, not as a JSON Schema): operator-fixable + // by re-registering the type, so 503, not 400/500. + let validator = jsonschema::validator_for(schema).map_err(|e| { + tracing::warn!(type_id = %type_id, err = %e, "secret type value_schema failed to compile"); + DomainError::ServiceUnavailable { + detail: format!("secret type '{type_id}' has a malformed value schema"), + retry_after: None, + cause: None, + } + })?; + if let Err(first) = validator.validate(&parsed) { + // instance_path only — never the offending value. + return Err(violation( + "value", + reasons::VALUE_SCHEMA_VIOLATION, + format!( + "value does not match the '{type_id}' schema at '{}': {}", + first.instance_path(), + redact_schema_error(&first) + ), + )); + } + Ok(()) +} + +/// Keep only the violation *kind* out of a schema error — jsonschema's +/// Display can embed the offending instance fragment, which may be secret +/// material. +fn redact_schema_error(err: &jsonschema::ValidationError<'_>) -> String { + format!("violated `{}`", err.schema_path()) +} + +#[cfg(test)] +#[path = "typing_tests.rs"] +mod typing_tests; diff --git a/gears/credstore/credstore/src/domain/secret/typing_tests.rs b/gears/credstore/credstore/src/domain/secret/typing_tests.rs new file mode 100644 index 000000000..d8fd05c5d --- /dev/null +++ b/gears/credstore/credstore/src/domain/secret/typing_tests.rs @@ -0,0 +1,158 @@ +//! Unit tests for secret-type trait enforcement. +//! +//! Traits come from the catalog descriptors via +//! [`credstore_sdk::SecretTypeDescriptor::traits`] — the same shape the +//! registry-driven resolver produces for the seeded built-in schemas. + +use credstore_sdk::{SecretType, SecretTypeTraits, SecretValue, SharingMode}; +use time::{Duration, OffsetDateTime}; + +use super::{reasons, validate_write}; +use crate::domain::error::DomainError; + +fn reason_of(err: &DomainError) -> &'static str { + match err { + DomainError::TypeViolation { reason, .. } => reason, + other => panic!("expected TypeViolation, got {other:?}"), + } +} + +/// `(gts_id, traits)` of a catalog type, as the resolver would return them. +fn resolved(name: &str) -> (&'static str, SecretTypeTraits) { + let t = SecretType::from_name(name).expect("known type"); + (t.gts_id(), t.descriptor().traits()) +} + +#[test] +fn generic_allows_everything_including_binary() { + let (id, traits) = resolved("generic"); + for mode in [ + SharingMode::Private, + SharingMode::Tenant, + SharingMode::Shared, + ] { + validate_write(id, &traits, mode, &SecretValue::new(vec![0xFF, 0x00]), None) + .expect("generic ok"); + } +} + +#[test] +fn personal_token_rejects_non_private_sharing() { + let (id, traits) = resolved("personal-token"); + validate_write( + id, + &traits, + SharingMode::Private, + &SecretValue::from("t"), + None, + ) + .expect("private ok"); + for mode in [SharingMode::Tenant, SharingMode::Shared] { + let err = + validate_write(id, &traits, mode, &SecretValue::from("t"), None).expect_err("denied"); + assert_eq!(reason_of(&err), reasons::SHARING_NOT_ALLOWED_FOR_TYPE); + } +} + +#[test] +fn size_limit_enforced() { + let (id, traits) = resolved("connection-string"); + let big = SecretValue::new(vec![b'x'; 4 * 1024 + 1]); + let err = validate_write(id, &traits, SharingMode::Tenant, &big, None).expect_err("too large"); + assert_eq!(reason_of(&err), reasons::VALUE_TOO_LARGE); +} + +#[test] +fn utf8_only_rejects_binary() { + let (id, traits) = resolved("api-key"); + let err = validate_write( + id, + &traits, + SharingMode::Tenant, + &SecretValue::new(vec![0xFF, 0xFE]), + None, + ) + .expect_err("binary rejected"); + assert_eq!(reason_of(&err), reasons::VALUE_NOT_UTF8); +} + +#[test] +fn oauth2_client_schema_validated_without_echoing_value() { + let (id, traits) = resolved("oauth2-client"); + + let ok = SecretValue::from(r#"{"client_id":"cid","client_secret":"s3cr3t"}"#); + validate_write(id, &traits, SharingMode::Tenant, &ok, None).expect("valid payload"); + + // Missing required field. + let missing = SecretValue::from(r#"{"client_id":"cid"}"#); + let err = validate_write(id, &traits, SharingMode::Tenant, &missing, None).expect_err("schema"); + assert_eq!(reason_of(&err), reasons::VALUE_SCHEMA_VIOLATION); + assert!( + !err.to_string().contains("cid"), + "schema violation detail must not echo the value: {err}" + ); + + // Not JSON at all. + let not_json = SecretValue::from("just-a-string"); + let err = + validate_write(id, &traits, SharingMode::Tenant, ¬_json, None).expect_err("not json"); + assert_eq!(reason_of(&err), reasons::VALUE_SCHEMA_VIOLATION); +} + +#[test] +fn malformed_value_schema_fails_closed_as_service_unavailable() { + // A `value_schema` trait that is JSON but not a valid JSON Schema — + // possible only through a broken registration, so 503, not 400. + let (id, mut traits) = resolved("generic"); + traits.value_schema = Some(serde_json::json!({"type": 42})); + let err = validate_write( + id, + &traits, + SharingMode::Tenant, + &SecretValue::from("{}"), + None, + ) + .expect_err("bad schema fails closed"); + assert!( + matches!(err, DomainError::ServiceUnavailable { .. }), + "got: {err:?}" + ); +} + +#[test] +fn expiry_gated_by_expirable_trait() { + let future = OffsetDateTime::now_utc() + Duration::hours(1); + + // Non-expirable type rejects expires_at. + let (api_key_id, api_key) = resolved("api-key"); + let err = validate_write( + api_key_id, + &api_key, + SharingMode::Tenant, + &SecretValue::from("k"), + Some(future), + ) + .expect_err("no expiry for api-key"); + assert_eq!(reason_of(&err), reasons::EXPIRY_NOT_SUPPORTED_FOR_TYPE); + + // Expirable type accepts a future expiry, rejects a past one. + let (bearer_id, bearer) = resolved("bearer-token"); + validate_write( + bearer_id, + &bearer, + SharingMode::Tenant, + &SecretValue::from("t"), + Some(future), + ) + .expect("future expiry ok"); + let past = OffsetDateTime::now_utc() - Duration::hours(1); + let err = validate_write( + bearer_id, + &bearer, + SharingMode::Tenant, + &SecretValue::from("t"), + Some(past), + ) + .expect_err("past expiry rejected"); + assert_eq!(reason_of(&err), reasons::EXPIRY_IN_THE_PAST); +} diff --git a/gears/credstore/credstore/src/domain/service.rs b/gears/credstore/credstore/src/domain/service.rs deleted file mode 100644 index 57fe42e71..000000000 --- a/gears/credstore/credstore/src/domain/service.rs +++ /dev/null @@ -1,130 +0,0 @@ -// Updated: 2026-04-07 by Constructor Tech -//! Domain service for the credstore gear. -//! -//! Plugin discovery is lazy: resolved on first API call after -//! types-registry is ready. - -use std::sync::Arc; -use std::time::Duration; - -use credstore_sdk::{CredStorePluginClientV1, CredStorePluginSpecV1, GetSecretResponse, SecretRef}; -use toolkit::client_hub::{ClientHub, ClientScope}; -use toolkit::plugins::{GtsPluginSelector, choose_plugin_instance}; -use toolkit::telemetry::ThrottledLog; -use toolkit_macros::domain_model; -use toolkit_security::SecurityContext; -use tracing::info; -use types_registry_sdk::{InstanceQuery, TypesRegistryClient}; - -use super::error::DomainError; - -/// Throttle interval for plugin unavailable warnings. -const UNAVAILABLE_LOG_THROTTLE: Duration = Duration::from_secs(10); - -/// `CredStore` domain service. -/// -/// Discovers plugins via types-registry and delegates storage operations. -#[domain_model] -pub struct Service { - hub: Arc, - vendor: String, - selector: GtsPluginSelector, - unavailable_log_throttle: ThrottledLog, -} - -impl Service { - /// Creates a new service with lazy plugin resolution. - #[must_use] - pub fn new(hub: Arc, vendor: String) -> Self { - Self { - hub, - vendor, - selector: GtsPluginSelector::new(), - unavailable_log_throttle: ThrottledLog::new(UNAVAILABLE_LOG_THROTTLE), - } - } - - /// Lazily resolves and returns the plugin client. - /// - /// # Errors - /// - /// Returns `DomainError::PluginNotFound` if no plugin is registered for the configured vendor. - /// Returns `DomainError::PluginUnavailable` if the plugin client is not yet registered. - async fn get_plugin(&self) -> Result, DomainError> { - let instance_id = self.selector.get_or_init(|| self.resolve_plugin()).await?; - let scope = ClientScope::gts_id(instance_id.as_ref()); - - if let Some(client) = self - .hub - .try_get_scoped::(&scope) - { - Ok(client) - } else { - if self.unavailable_log_throttle.should_log() { - tracing::warn!( - plugin_gts_id = %instance_id, - vendor = %self.vendor, - "CredStore plugin client not registered yet" - ); - } - Err(DomainError::PluginUnavailable { - gts_id: instance_id.to_string(), - reason: "client not registered yet".into(), - }) - } - } - - /// Resolves the plugin instance from types-registry. - #[tracing::instrument(skip_all, fields(vendor = %self.vendor))] - async fn resolve_plugin(&self) -> Result { - info!("Resolving credstore plugin"); - - let registry = self - .hub - .get::() - .map_err(|e| DomainError::TypesRegistryUnavailable(e.to_string()))?; - - let plugin_type_id = CredStorePluginSpecV1::gts_type_id().clone(); - - let instances = registry - .list_instances(InstanceQuery::new().with_pattern(format!("{plugin_type_id}*"))) - .await?; - - let gts_id = choose_plugin_instance::( - &self.vendor, - instances.iter().map(|e| (e.id.as_ref(), &e.object)), - )?; - info!(plugin_gts_id = %gts_id, "Selected credstore plugin instance"); - - Ok(gts_id) - } - - /// Retrieves a secret from the plugin. - /// - /// Returns `Ok(None)` if the secret is not found (anti-enumeration). - /// - /// # Errors - /// - /// Returns a `DomainError` for plugin resolution or backend failures. - #[tracing::instrument(skip_all, fields(key = ?key))] - pub async fn get( - &self, - ctx: &SecurityContext, - key: &SecretRef, - ) -> Result, DomainError> { - let plugin = self.get_plugin().await?; - - let result = plugin.get(ctx, key).await?; - Ok(result.map(|meta| GetSecretResponse { - value: meta.value, - owner_tenant_id: meta.owner_tenant_id, - sharing: meta.sharing, - is_inherited: false, - })) - } -} - -#[cfg(test)] -#[cfg_attr(coverage_nightly, coverage(off))] -#[path = "service_tests.rs"] -mod service_tests; diff --git a/gears/credstore/credstore/src/domain/service_tests.rs b/gears/credstore/credstore/src/domain/service_tests.rs deleted file mode 100644 index d13532a7f..000000000 --- a/gears/credstore/credstore/src/domain/service_tests.rs +++ /dev/null @@ -1,274 +0,0 @@ -// Created: 2026-04-07 by Constructor Tech -use std::sync::Arc; - -use credstore_sdk::{OwnerId, SecretMetadata, SecretValue, SharingMode, TenantId}; -use toolkit::client_hub::{ClientHub, ClientScope}; -use types_registry_sdk::testing::{self, MockTypesRegistryClient, make_test_instance}; - -use super::*; -use crate::domain::test_support::{MockPlugin, test_ctx}; - -// ── helpers ────────────────────────────────────────────────────────────── - -fn empty_hub() -> Arc { - Arc::new(ClientHub::default()) -} - -/// Build the GTS instance ID string for a credstore plugin test instance. -fn test_instance_id() -> String { - // schema prefix + instance suffix (5-token: vendor.package.namespace.type.vMAJOR) - format!( - "{}test.credstore.mock.instance.v1", - CredStorePluginSpecV1::gts_type_id() - ) -} - -/// Build the JSON content for a `PluginV1` -/// instance that `choose_plugin_instance` can successfully parse. -fn plugin_content(gts_id: &str, vendor: &str) -> serde_json::Value { - serde_json::json!({ - "id": gts_id, - "vendor": vendor, - "priority": 0, - "properties": {} - }) -} - -// ── helper to build a fully-wired hub ──────────────────────────────────── - -/// Wires a counting `MockTypesRegistryClient` and a scoped plugin into a `ClientHub`. -/// Returns `(hub, registry_arc)` so tests can inspect `list_instance_calls()`. -fn hub_with_counting_registry_and_plugin( - instance_id: &str, - vendor: &str, - plugin: Arc, -) -> (Arc, Arc) { - let hub = Arc::new(ClientHub::default()); - - let instance = make_test_instance(instance_id, plugin_content(instance_id, vendor)); - let registry = Arc::new(MockTypesRegistryClient::new().with_instances([instance])); - hub.register::(registry.clone() as Arc); - - hub.register_scoped::(ClientScope::gts_id(instance_id), plugin); - - (hub, registry) -} - -fn hub_with_registry_and_plugin( - instance_id: &str, - vendor: &str, - plugin: Arc, -) -> Arc { - hub_with_counting_registry_and_plugin(instance_id, vendor, plugin).0 -} - -#[tokio::test] -async fn get_returns_registry_unavailable_when_hub_empty() { - let svc = Service::new(empty_hub(), "cf-gears".into()); - let key = SecretRef::new("my-key").unwrap(); - let err = svc.get(&test_ctx(), &key).await.unwrap_err(); - assert!( - matches!(err, DomainError::TypesRegistryUnavailable(_)), - "expected TypesRegistryUnavailable, got: {err:?}" - ); -} - -#[tokio::test] -async fn get_retries_resolution_on_each_call_when_registry_absent() { - // GtsPluginSelector does not cache errors, so each call re-attempts resolution. - // Use a failing registry (not an empty hub) so list() is actually invoked and - // we can assert the call count proves no caching. - let hub = Arc::new(ClientHub::default()); - let registry = - Arc::new(MockTypesRegistryClient::new().with_list_error(testing::internal("unavailable"))); - hub.register::(registry.clone() as Arc); - let svc = Service::new(hub, "constructorfabric".into()); - let key = SecretRef::new("my-key").unwrap(); - assert!(svc.get(&test_ctx(), &key).await.is_err()); - assert!(svc.get(&test_ctx(), &key).await.is_err()); - assert_eq!(registry.list_instance_calls(), 2); -} - -// ── resolve_plugin ─────────────────────────────────────────────────────── - -#[tokio::test] -async fn resolve_plugin_returns_plugin_not_found_when_no_instances() { - let hub = Arc::new(ClientHub::default()); - let registry: Arc = Arc::new(MockTypesRegistryClient::new()); - hub.register::(registry); - - let svc = Service::new(hub, "constructorfabric".into()); - let err = svc.resolve_plugin().await.unwrap_err(); - assert!( - matches!(err, DomainError::PluginNotFound { .. }), - "expected PluginNotFound, got: {err:?}" - ); -} - -#[tokio::test] -async fn resolve_plugin_returns_plugin_not_found_when_vendor_mismatch() { - let instance_id = test_instance_id(); - let hub = Arc::new(ClientHub::default()); - let instance = make_test_instance(&instance_id, plugin_content(&instance_id, "other-vendor")); - let registry: Arc = - Arc::new(MockTypesRegistryClient::new().with_instances([instance])); - hub.register::(registry); - - let svc = Service::new(hub, "constructorfabric".into()); - let err = svc.resolve_plugin().await.unwrap_err(); - assert!( - matches!(err, DomainError::PluginNotFound { .. }), - "expected PluginNotFound, got: {err:?}" - ); -} - -#[tokio::test] -async fn resolve_plugin_returns_invalid_when_content_malformed() { - let instance_id = test_instance_id(); - let hub = Arc::new(ClientHub::default()); - let instance = make_test_instance( - &instance_id, - serde_json::json!({ "not": "valid-plugin-content" }), - ); - let registry: Arc = - Arc::new(MockTypesRegistryClient::new().with_instances([instance])); - hub.register::(registry); - - let svc = Service::new(hub, "constructorfabric".into()); - let err = svc.resolve_plugin().await.unwrap_err(); - assert!( - matches!(err, DomainError::InvalidPluginInstance { .. }), - "expected InvalidPluginInstance, got: {err:?}" - ); -} - -#[tokio::test] -async fn resolve_plugin_returns_internal_when_registry_list_fails() { - let hub = Arc::new(ClientHub::default()); - let registry: Arc = - Arc::new(MockTypesRegistryClient::new().with_list_error(testing::internal("db down"))); - hub.register::(registry); - - let svc = Service::new(hub, "constructorfabric".into()); - let err = svc.resolve_plugin().await.unwrap_err(); - assert!( - matches!(err, DomainError::Internal(ref msg) if msg.contains("db down")), - "expected Internal containing 'db down', got: {err:?}" - ); -} - -#[tokio::test] -async fn resolve_plugin_succeeds_with_matching_vendor() { - let instance_id = test_instance_id(); - let hub = - hub_with_registry_and_plugin(&instance_id, "constructorfabric", MockPlugin::returns(None)); - - let svc = Service::new(hub, "constructorfabric".into()); - let resolved = svc.resolve_plugin().await.unwrap(); - assert_eq!(resolved, instance_id); -} - -// ── get_plugin ─────────────────────────────────────────────────────────── - -#[tokio::test] -async fn get_plugin_returns_unavailable_when_not_in_hub() { - // Registry resolves successfully, but the scoped client is absent. - let instance_id = test_instance_id(); - let hub = Arc::new(ClientHub::default()); - let instance = make_test_instance( - &instance_id, - plugin_content(&instance_id, "constructorfabric"), - ); - let registry: Arc = - Arc::new(MockTypesRegistryClient::new().with_instances([instance])); - hub.register::(registry); - - let svc = Service::new(hub, "constructorfabric".into()); - let err = svc.get_plugin().await.err().expect("expected Err"); - assert!( - matches!(err, DomainError::PluginUnavailable { .. }), - "expected PluginUnavailable, got: {err:?}" - ); -} - -#[tokio::test] -async fn get_plugin_caches_resolved_instance() { - let instance_id = test_instance_id(); - let (hub, registry) = hub_with_counting_registry_and_plugin( - &instance_id, - "constructorfabric", - MockPlugin::returns(None), - ); - - let svc = Service::new(hub, "constructorfabric".into()); - let p1 = svc.get_plugin().await.unwrap(); - let p2 = svc.get_plugin().await.unwrap(); - - assert_eq!( - registry.list_instance_calls(), - 1, - "resolve_plugin should be called exactly once; second call must use cached value" - ); - assert!( - Arc::ptr_eq(&p1, &p2), - "both calls should return the same plugin Arc (same mock instance)" - ); -} - -// ── get ────────────────────────────────────────────────────────────────── - -#[tokio::test] -async fn get_returns_some_response_on_success() { - let instance_id = test_instance_id(); - let meta = SecretMetadata { - value: SecretValue::from("s3cr3t"), - owner_id: OwnerId::nil(), - sharing: SharingMode::Tenant, - owner_tenant_id: TenantId::nil(), - }; - let hub = hub_with_registry_and_plugin( - &instance_id, - "constructorfabric", - MockPlugin::returns(Some(&meta)), - ); - - let svc = Service::new(hub, "constructorfabric".into()); - let key = SecretRef::new("my-key").unwrap(); - let resp = svc.get(&test_ctx(), &key).await.unwrap(); - - let resp = resp.expect("expected Some response"); - assert_eq!(resp.value.as_bytes(), b"s3cr3t"); - assert_eq!(resp.sharing, SharingMode::Tenant); - assert!(!resp.is_inherited, "is_inherited must always be false here"); - assert_eq!(resp.owner_tenant_id, TenantId::nil()); -} - -#[tokio::test] -async fn get_returns_none_when_plugin_returns_none() { - let instance_id = test_instance_id(); - let hub = - hub_with_registry_and_plugin(&instance_id, "constructorfabric", MockPlugin::returns(None)); - - let svc = Service::new(hub, "constructorfabric".into()); - let key = SecretRef::new("missing-key").unwrap(); - let result = svc.get(&test_ctx(), &key).await.unwrap(); - assert!(result.is_none(), "expected None for missing secret"); -} - -#[tokio::test] -async fn get_propagates_plugin_error() { - let instance_id = test_instance_id(); - let hub = hub_with_registry_and_plugin( - &instance_id, - "constructorfabric", - MockPlugin::errors_internal("backend failure"), - ); - - let svc = Service::new(hub, "constructorfabric".into()); - let key = SecretRef::new("any-key").unwrap(); - let err = svc.get(&test_ctx(), &key).await.unwrap_err(); - assert!( - matches!(err, DomainError::Internal(_)), - "expected Internal, got: {err:?}" - ); -} diff --git a/gears/credstore/credstore/src/domain/test_support.rs b/gears/credstore/credstore/src/domain/test_support.rs deleted file mode 100644 index 7bf27defa..000000000 --- a/gears/credstore/credstore/src/domain/test_support.rs +++ /dev/null @@ -1,85 +0,0 @@ -//! Shared test infrastructure for domain-layer unit tests. -//! -//! For the GTS registry mock, use `MockTypesRegistryClient` and -//! `make_test_instance` from `types_registry_sdk::testing` directly. - -use std::sync::Arc; - -use async_trait::async_trait; -use credstore_sdk::{ - CredStoreError, CredStorePluginClientV1, OwnerId, SecretMetadata, SecretValue, SharingMode, - TenantId, -}; -use toolkit_security::SecurityContext; -use uuid::Uuid; - -use credstore_sdk::SecretRef; - -// ── SecurityContext ─────────────────────────────────────────────────────────── - -/// Build a minimal [`SecurityContext`] suitable for unit tests. -/// -/// # Panics -/// -/// Panics if the builder fails, which cannot happen with `Uuid::nil()` inputs. -#[must_use] -pub fn test_ctx() -> SecurityContext { - SecurityContext::builder() - .subject_id(Uuid::nil()) - .subject_tenant_id(Uuid::nil()) - .build() - .unwrap() -} - -// ── MockPlugin ──────────────────────────────────────────────────────────────── - -type PluginFn = Arc Result, CredStoreError> + Send + Sync>; - -pub struct MockPlugin { - handler: PluginFn, -} - -impl MockPlugin { - #[must_use] - pub fn returns(meta: Option<&SecretMetadata>) -> Arc { - let bytes = meta.map(|m| m.value.as_bytes().to_vec()); - let owner_id = meta.map_or(OwnerId::nil(), |m| m.owner_id); - let sharing = meta.map_or(SharingMode::Tenant, |m| m.sharing); - let owner_tenant_id = meta.map_or(TenantId::nil(), |m| m.owner_tenant_id); - Arc::new(Self { - handler: Arc::new(move || { - Ok(bytes.as_ref().map(|b| SecretMetadata { - value: SecretValue::new(b.clone()), - owner_id, - sharing, - owner_tenant_id, - })) - }), - }) - } - - #[must_use] - pub fn errors_not_found() -> Arc { - Arc::new(Self { - handler: Arc::new(|| Err(CredStoreError::NotFound)), - }) - } - - #[must_use] - pub fn errors_internal(msg: &'static str) -> Arc { - Arc::new(Self { - handler: Arc::new(move || Err(CredStoreError::Internal(msg.into()))), - }) - } -} - -#[async_trait] -impl CredStorePluginClientV1 for MockPlugin { - async fn get( - &self, - _ctx: &SecurityContext, - _key: &SecretRef, - ) -> Result, CredStoreError> { - (self.handler)() - } -} diff --git a/gears/credstore/credstore/src/gear.rs b/gears/credstore/credstore/src/gear.rs index 7b0984415..56aa77ec6 100644 --- a/gears/credstore/credstore/src/gear.rs +++ b/gears/credstore/credstore/src/gear.rs @@ -1,30 +1,39 @@ -//! `CredStore` gear. - use std::sync::{Arc, OnceLock}; +use std::time::Duration; use async_trait::async_trait; -use credstore_sdk::CredStoreClientV1; -use toolkit::contracts::SystemCapability; -use toolkit::{Gear, GearCtx}; +use authz_resolver_sdk::{AuthZResolverClient, PolicyEnforcer, models::Capability}; +use tenant_resolver_sdk::TenantResolverClient; +use tokio_util::sync::CancellationToken; +use toolkit::api::OpenApiRegistry; +use toolkit::contracts::{DatabaseCapability, SystemCapability}; +use toolkit::lifecycle::ReadySignal; +use toolkit::{Gear, GearCtx, RestApiCapability}; +use toolkit_db::DBProvider; use tracing::info; -use crate::config::CredStoreConfig; -use crate::domain::{CredStoreLocalClient, Service}; - -/// `CredStore` gateway gear. -/// -/// This gear: -/// 1. Discovers plugin instances via types-registry (lazy, first-use) -/// 2. Routes secret operations through the selected plugin -/// 3. Registers `Arc` in `ClientHub` for consumers -/// -/// The `CredStorePluginSpecV1` schema itself reaches `types-registry` -/// automatically via the `toolkit-gts` link-time inventory — no per-init -/// registration is needed. +use types_registry_sdk::TypesRegistryClient; + +use crate::client::CredStoreLocalClient; +use crate::config::{CredStoreConfig, HierarchyCfg}; +use crate::domain::ports::metrics::CredStoreMetricsPort; +use crate::domain::secret::service::{ReaperSettings, Service}; +use crate::infra::metrics::CredStoreMetricsMeter; +use crate::infra::plugin_select::GtsCredStorePluginSelector; +use crate::infra::storage::repo_impl::SecretRepoImpl; +use crate::infra::tenant_resolver::TenantResolverDir; +use crate::infra::types_registry::GtsSecretTypeResolver; + +// `system` capability is required in this platform: consumers like `oagw` are +// system modules and resolve `CredStoreClientV1` from the ClientHub during their +// `init`. System modules initialize before non-system ones +// (`modules_by_system_priority`), so credstore must also be a system module to +// register its client before those consumers init. #[toolkit::gear( name = "credstore", - deps = ["types-registry"], - capabilities = [system] + deps = ["authz-resolver", "tenant-resolver", "types-registry"], + capabilities = [system, db, rest, stateful], + lifecycle(entry = "serve", stop_timeout = "30s", await_ready) )] pub struct CredStoreGear { service: OnceLock>, @@ -38,28 +47,189 @@ impl Default for CredStoreGear { } } +/// PDP capabilities to advertise, given the hierarchy config. When the shared +/// tenant-closure is co-located in this module's database we advertise +/// `TenantHierarchy` (structured subtree predicate, resolved by a closure +/// subquery); otherwise we omit it and the PDP pre-expands the subtree into a +/// flat membership list this module enforces without any local closure. +fn pep_capabilities(hierarchy: &HierarchyCfg) -> Vec { + if hierarchy.tenant_closure_colocated { + vec![Capability::TenantHierarchy] + } else { + Vec::new() + } +} + +impl CredStoreGear { + #[allow( + clippy::redundant_pub_crate, + reason = "module-private serve entry-point invoked by the toolkit runtime" + )] + pub(crate) async fn serve( + self: Arc, + cancel: CancellationToken, + ready: ReadySignal, + ) -> anyhow::Result<()> { + let Some(svc) = self.service.get().cloned() else { + anyhow::bail!("credstore: serve invoked before init"); + }; + + let tick = Duration::from_secs(svc.reaper_tick_secs()); + let mut interval = tokio::time::interval(tick); + interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay); + + ready.notify(); + info!( + target: "credstore.lifecycle", + reaper_tick_secs = tick.as_secs(), + "credstore reaper tick started" + ); + + loop { + tokio::select! { + biased; + () = cancel.cancelled() => break, + _ = interval.tick() => { + svc.reap_and_refresh().await; + } + } + } + + info!(target: "credstore.lifecycle", "credstore reaper tick cancelled"); + Ok(()) + } +} + #[async_trait] impl Gear for CredStoreGear { - #[tracing::instrument(skip_all, fields(vendor))] + #[tracing::instrument(skip_all, fields(module = "credstore"))] async fn init(&self, ctx: &GearCtx) -> anyhow::Result<()> { let cfg: CredStoreConfig = ctx.config_or_default()?; - tracing::Span::current().record("vendor", cfg.vendor.as_str()); - info!(vendor = %cfg.vendor); + cfg.validate() + .map_err(|err| anyhow::anyhow!("credstore config invalid: {err}"))?; + info!(vendor = %cfg.vendor, "initializing credstore module"); + + let db_raw = ctx.db_required()?; + let db: Arc> = + Arc::new(DBProvider::new(db_raw.db())); + + let repo = Arc::new(SecretRepoImpl::new(Arc::clone(&db))); + + let authz_client = ctx + .client_hub() + .get::() + .map_err(|e| anyhow::anyhow!("failed to get AuthZResolverClient: {e}"))?; + let enforcer = + PolicyEnforcer::new(authz_client).with_capabilities(pep_capabilities(&cfg.hierarchy)); + info!("authz-resolver client resolved from client hub; PolicyEnforcer wired"); + + let tr_client = ctx + .client_hub() + .get::() + .map_err(|e| anyhow::anyhow!("failed to get TenantResolverClient: {e}"))?; + + let metrics: Arc = Arc::new(CredStoreMetricsMeter::from_global()); + + let dir = Arc::new(TenantResolverDir::new( + tr_client, + Arc::clone(&metrics), + cfg.hierarchy.ancestor_cache_ttl_secs, + )); + + let plugins = Arc::new(GtsCredStorePluginSelector::new( + ctx.client_hub(), + cfg.vendor.clone(), + )); + + // Fail-closed: without the types-registry client no secret type can + // be resolved, so credstore must not come up (types-registry is a + // hard `deps` and initializes first). + let registry = ctx + .client_hub() + .get::() + .map_err(|e| anyhow::anyhow!("failed to get TypesRegistryClient: {e}"))?; + let types = Arc::new(GtsSecretTypeResolver::new(registry, Arc::clone(&metrics))); + info!("types-registry client resolved from client hub; secret-type resolver wired"); + + let svc = Arc::new(Service::new( + repo, + dir, + enforcer, + plugins, + types, + metrics, + ReaperSettings { + tick_secs: cfg.reaper.tick_secs, + provisioning_timeout_secs: cfg.reaper.provisioning_timeout_secs, + deprovisioning_timeout_secs: cfg.reaper.deprovisioning_timeout_secs, + }, + )); - // Create domain service - let hub = ctx.client_hub(); - let svc = Arc::new(Service::new(hub, cfg.vendor)); self.service - .set(svc.clone()) - .map_err(|_| anyhow::anyhow!("{} gear already initialized", Self::MODULE_NAME))?; + .set(Arc::clone(&svc)) + .map_err(|_| anyhow::anyhow!("{} module already initialized", Self::MODULE_NAME))?; - // Register local client in ClientHub - let api: Arc = Arc::new(CredStoreLocalClient::new(svc)); - ctx.client_hub().register::(api); + let client: Arc = + Arc::new(CredStoreLocalClient::new(svc)); + ctx.client_hub() + .register::(client); + info!("credstore module initialized"); Ok(()) } } -#[async_trait] +// Empty system capability: credstore needs no pre_init/post_init work, only the +// system-priority init ordering (see the module attribute above). impl SystemCapability for CredStoreGear {} + +impl DatabaseCapability for CredStoreGear { + fn migrations(&self) -> Vec> { + use sea_orm_migration::MigratorTrait; + info!("providing credstore database migrations"); + crate::infra::storage::migrations::Migrator::migrations() + } +} + +impl RestApiCapability for CredStoreGear { + fn register_rest( + &self, + _ctx: &GearCtx, + router: axum::Router, + openapi: &dyn OpenApiRegistry, + ) -> anyhow::Result { + info!("registering credstore REST routes"); + let svc = self + .service + .get() + .cloned() + .ok_or_else(|| anyhow::anyhow!("credstore Service not initialized"))?; + let router = crate::api::rest::register_routes(router, openapi, svc); + info!("credstore REST routes registered"); + Ok(router) + } +} + +#[cfg(test)] +mod tests { + use super::{Capability, pep_capabilities}; + use crate::config::HierarchyCfg; + + #[test] + fn advertises_tenant_hierarchy_when_closure_colocated() { + let cfg = HierarchyCfg { + tenant_closure_colocated: true, + ..Default::default() + }; + assert_eq!(pep_capabilities(&cfg), vec![Capability::TenantHierarchy]); + } + + #[test] + fn advertises_no_capabilities_when_closure_not_colocated() { + let cfg = HierarchyCfg { + tenant_closure_colocated: false, + ..Default::default() + }; + assert!(pep_capabilities(&cfg).is_empty()); + } +} diff --git a/gears/credstore/credstore/src/infra.rs b/gears/credstore/credstore/src/infra.rs new file mode 100644 index 000000000..0bb475599 --- /dev/null +++ b/gears/credstore/credstore/src/infra.rs @@ -0,0 +1,9 @@ +//! Infrastructure layer — DB, storage, error classifiers. +pub mod canonical_mapping; +pub mod error_conv; +pub mod metrics; +pub mod plugin_select; +pub mod sdk_error_mapping; +pub mod storage; +pub mod tenant_resolver; +pub mod types_registry; diff --git a/gears/credstore/credstore/src/infra/canonical_mapping.rs b/gears/credstore/credstore/src/infra/canonical_mapping.rs new file mode 100644 index 000000000..39d28cc9f --- /dev/null +++ b/gears/credstore/credstore/src/infra/canonical_mapping.rs @@ -0,0 +1,103 @@ +//! `DbErr` → [`DomainError`] classification ladder. + +use sea_orm::DbErr; +use toolkit_db::DbError; +use tracing::warn; + +use crate::domain::error::DomainError; +use crate::infra::error_conv::{is_check_violation, is_serialization_failure, is_unique_violation}; + +/// Classify a raw [`DbErr`] into a typed [`DomainError`]. +#[allow(clippy::needless_pass_by_value)] +pub(crate) fn classify_db_err_to_domain(db_err: DbErr) -> DomainError { + if is_unique_violation(&db_err) { + return DomainError::Conflict; + } + if is_serialization_failure(&db_err) { + warn!( + target: "credstore.db", + error = %db_err, + "serialization conflict (retry-exhausted)" + ); + return DomainError::ServiceUnavailable { + detail: "serialization conflict; retry budget exhausted".to_owned(), + retry_after: None, + cause: None, + }; + } + // Every CHECK in the schema (reference length, sharing/status domains, the + // fingerprint fence pairing) guards data the code validates or produces + // before the write — `reference` is rejected at both the SDK and REST + // boundaries, the rest never comes from user input. A firing CHECK is + // therefore a broken server-side invariant, not a bad secret reference: + // it maps to `Internal` like any other unexpected DB failure. + let diagnostic = if is_check_violation(&db_err) { + "database CHECK constraint violated" + } else { + "database error" + }; + warn!(target: "credstore.db", error = %db_err, "{diagnostic}"); + DomainError::Internal { + diagnostic: diagnostic.to_owned(), + cause: Some(Box::new(db_err)), + } +} + +impl From for DomainError { + fn from(err: DbError) -> Self { + match err { + DbError::Sea(db) => classify_db_err_to_domain(db), + other => Self::Internal { + diagnostic: "database error".to_owned(), + cause: Some(Box::new(other)), + }, + } + } +} + +#[cfg(test)] +mod tests { + use sea_orm::DbErr; + use toolkit_db::DbError; + + use super::classify_db_err_to_domain; + use crate::domain::error::DomainError; + + fn custom(msg: &str) -> DbErr { + DbErr::Custom(msg.to_owned()) + } + + #[test] + fn classify_maps_check_violation_to_internal() { + assert!(matches!( + classify_db_err_to_domain(custom("CHECK constraint failed")), + DomainError::Internal { cause: Some(_), .. } + )); + } + + #[test] + fn classify_maps_unclassified_to_internal() { + assert!(matches!( + classify_db_err_to_domain(custom("connection refused")), + DomainError::Internal { .. } + )); + } + + #[test] + fn from_dberr_sea_delegates_to_classifier() { + let mapped: DomainError = DbError::Sea(custom("CHECK constraint failed")).into(); + assert!(matches!( + mapped, + DomainError::Internal { cause: Some(_), .. } + )); + } + + #[test] + fn from_dberr_non_sea_is_internal_with_cause() { + let mapped: DomainError = DbError::InvalidConfig("bad dsn".to_owned()).into(); + assert!(matches!( + mapped, + DomainError::Internal { cause: Some(_), .. } + )); + } +} diff --git a/gears/credstore/credstore/src/infra/error_conv.rs b/gears/credstore/credstore/src/infra/error_conv.rs new file mode 100644 index 000000000..30720f647 --- /dev/null +++ b/gears/credstore/credstore/src/infra/error_conv.rs @@ -0,0 +1,95 @@ +//! DB error classification helpers for the boundary mapping in +//! [`crate::infra::canonical_mapping`]. + +use sea_orm::{DbBackend, DbErr}; +use toolkit_db::contention::is_retryable_contention; + +/// Backend-agnostic adapter. Probes `PostgreSQL` and `SQLite` (the two +/// supported backends); `MySQL` is unsupported. +pub(crate) fn is_serialization_failure(err: &DbErr) -> bool { + is_retryable_contention(DbBackend::Postgres, err) + || is_retryable_contention(DbBackend::Sqlite, err) +} + +/// Returns `true` iff `err` represents a `CHECK` constraint violation on +/// either supported backend. +pub(crate) fn is_check_violation(err: &DbErr) -> bool { + let msg = err.to_string().to_lowercase(); + msg.contains("check constraint") + || msg.contains("check_violation") + || msg.contains("sqlite_constraint_check") + || contains_anchored_pg_check_sqlstate(&msg) + || (msg.contains("sqlite") && contains_anchored_sqlite_check_code(&msg)) +} + +/// Returns `true` iff `err` is a unique-constraint violation. +pub(crate) fn is_unique_violation(err: &DbErr) -> bool { + matches!( + err.sql_err(), + Some(sea_orm::SqlErr::UniqueConstraintViolation(_)) + ) +} + +fn contains_anchored_pg_check_sqlstate(msg: &str) -> bool { + msg.contains("sqlstate 23514") + || msg.contains("sqlstate: 23514") + || msg.contains("sqlstate=23514") + || msg.contains("code 23514") + || msg.contains("code: 23514") + || msg.contains("(23514)") + || msg.contains("(23514:") + || msg.starts_with("23514:") + || msg.contains(" 23514:") +} + +fn contains_anchored_sqlite_check_code(msg: &str) -> bool { + msg.contains("code 275") + || msg.contains("code: 275") + || msg.contains("(275)") + || msg.contains("(275:") + || msg.starts_with("275:") + || msg.contains(" 275:") +} + +#[cfg(test)] +mod tests { + use sea_orm::DbErr; + + use super::{is_check_violation, is_serialization_failure, is_unique_violation}; + + fn custom(msg: &str) -> DbErr { + DbErr::Custom(msg.to_owned()) + } + + #[test] + fn check_violation_detected_by_message_and_codes() { + assert!(is_check_violation(&custom( + "CHECK constraint failed: secrets" + ))); + assert!(is_check_violation(&custom( + "error: check_violation on table" + ))); + assert!(is_check_violation(&custom( + "SQLITE_CONSTRAINT_CHECK: bad row" + ))); + // Anchored PostgreSQL SQLSTATE 23514. + assert!(is_check_violation(&custom("db error (SQLSTATE 23514)"))); + // Anchored SQLite extended result code 275, gated on "sqlite". + assert!(is_check_violation(&custom("sqlite failure (275)"))); + } + + #[test] + fn check_violation_ignores_unrelated_errors() { + assert!(!is_check_violation(&custom("connection reset by peer"))); + // Bare 275 without the sqlite marker must not match. + assert!(!is_check_violation(&custom("affected 275 rows"))); + } + + #[test] + fn unique_and_serialization_predicates_reject_opaque_errors() { + // A free-form Custom error carries no SqlErr / backend contention code. + let err = custom("opaque"); + assert!(!is_unique_violation(&err)); + assert!(!is_serialization_failure(&err)); + } +} diff --git a/gears/credstore/credstore/src/infra/metrics.rs b/gears/credstore/credstore/src/infra/metrics.rs new file mode 100644 index 000000000..98ca9c8e2 --- /dev/null +++ b/gears/credstore/credstore/src/infra/metrics.rs @@ -0,0 +1,332 @@ +//! OpenTelemetry adapter implementing [`CredStoreMetricsPort`]. +//! +//! Instruments are pulled from the process-global meter provider installed by +//! the host; a no-op until an exporter is wired. Instrument names are full +//! literal Prometheus names: counters end in `_total`, duration histograms in +//! `_seconds`, with suffixes baked in (no `.with_unit()`). Matches the +//! platform's `add_metric_suffixes: false` collector posture. + +use opentelemetry::KeyValue; +use opentelemetry::metrics::{Counter, Gauge, Histogram, Meter}; + +use crate::domain::ports::metrics::{ + CredStoreMetricsPort, Dep, DepOp, FenceVerify, Outcome, ReadOutcome, SecretCounts, +}; + +/// Meter / instrumentation scope name. +pub(crate) const METER_NAME: &str = "credstore"; + +// ─── Metric names (literal Prometheus form; `add_metric_suffixes: false`) ───── +const CREDSTORE_SECRETS: &str = "credstore_secrets"; +const CREDSTORE_SECRETS_PROVISIONING: &str = "credstore_secrets_provisioning"; +const CREDSTORE_SECRETS_DEPROVISIONING: &str = "credstore_secrets_deprovisioning"; +const CREDSTORE_TENANTS_WITH_SECRETS: &str = "credstore_tenants_with_secrets"; +const CREDSTORE_READ_OUTCOME: &str = "credstore_read_outcome_total"; +const CREDSTORE_WALKUP_DEPTH: &str = "credstore_walkup_depth"; +const CREDSTORE_DEPENDENCY_QUERY_DURATION: &str = "credstore_dependency_query_duration_seconds"; +const CREDSTORE_DEPENDENCY_HEALTH: &str = "credstore_dependency_health_total"; +const CREDSTORE_PROVISIONING_REAPED: &str = "credstore_provisioning_reaped_total"; +const CREDSTORE_DEPROVISIONING_REAPED: &str = "credstore_deprovisioning_reaped_total"; +const CREDSTORE_PROVISIONING_ROLLBACK: &str = "credstore_provisioning_rollback_total"; +const CREDSTORE_CROSS_TENANT_DENIED: &str = "credstore_cross_tenant_denied_total"; +const CREDSTORE_FENCE_VERIFY: &str = "credstore_fence_verify_total"; +const CREDSTORE_FENCE_BACKFILL: &str = "credstore_fence_backfill_total"; + +/// OpenTelemetry-backed metrics handle for the credstore module. +pub struct CredStoreMetricsMeter { + secrets: Gauge, + secrets_provisioning: Gauge, + secrets_deprovisioning: Gauge, + tenants_with_secrets: Gauge, + read_outcome: Counter, + walkup_depth: Histogram, + dependency_query_duration: Histogram, + dependency_health: Counter, + provisioning_reaped: Counter, + deprovisioning_reaped: Counter, + provisioning_rollback: Counter, + cross_tenant_denied: Counter, + fence_verify: Counter, + fence_backfill: Counter, +} + +impl std::fmt::Debug for CredStoreMetricsMeter { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("CredStoreMetricsMeter") + .finish_non_exhaustive() + } +} + +impl CredStoreMetricsMeter { + /// Build the instrument set from the supplied meter. + #[must_use] + pub fn new(meter: &Meter) -> Self { + Self { + secrets: meter + .i64_gauge(CREDSTORE_SECRETS) + .with_description("Live count of secrets by sharing scope") + .build(), + secrets_provisioning: meter + .i64_gauge(CREDSTORE_SECRETS_PROVISIONING) + .with_description("Live count of secrets in provisioning state") + .build(), + secrets_deprovisioning: meter + .i64_gauge(CREDSTORE_SECRETS_DEPROVISIONING) + .with_description("Live count of secrets in deprovisioning state") + .build(), + tenants_with_secrets: meter + .i64_gauge(CREDSTORE_TENANTS_WITH_SECRETS) + .with_description("Live count of tenants that own at least one secret") + .build(), + read_outcome: meter + .u64_counter(CREDSTORE_READ_OUTCOME) + .with_description("Secret read results by outcome") + .build(), + walkup_depth: meter + .u64_histogram(CREDSTORE_WALKUP_DEPTH) + .with_description("Tenant walk-up depth when resolving inherited secrets") + .build(), + dependency_query_duration: meter + .f64_histogram(CREDSTORE_DEPENDENCY_QUERY_DURATION) + .with_description("Upstream dependency query latency, by dependency + operation") + .with_boundaries(vec![0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25]) + .build(), + dependency_health: meter + .u64_counter(CREDSTORE_DEPENDENCY_HEALTH) + .with_description( + "Upstream dependency call outcomes, by dependency + operation + outcome", + ) + .build(), + provisioning_reaped: meter + .u64_counter(CREDSTORE_PROVISIONING_REAPED) + .with_description("Provisioning secrets reaped by the background sweeper") + .build(), + deprovisioning_reaped: meter + .u64_counter(CREDSTORE_DEPROVISIONING_REAPED) + .with_description( + "Stuck deprovisioning secrets completed by the background sweeper", + ) + .build(), + provisioning_rollback: meter + .u64_counter(CREDSTORE_PROVISIONING_ROLLBACK) + .with_description( + "Create-saga provisioning-row rollbacks after a backend write failure, by outcome", + ) + .build(), + cross_tenant_denied: meter + .u64_counter(CREDSTORE_CROSS_TENANT_DENIED) + .with_description("Cross-tenant secret access attempts that were denied") + .build(), + fence_verify: meter + .u64_counter(CREDSTORE_FENCE_VERIFY) + .with_description( + "Value-fingerprint fence verdicts on reads, by outcome \ + (mismatch = fail-closed 404, the alertable signal)", + ) + .build(), + fence_backfill: meter + .u64_counter(CREDSTORE_FENCE_BACKFILL) + .with_description( + "Lazy fingerprint backfills of out-of-band seeded rows, by outcome", + ) + .build(), + } + } + + /// Build a handle bound to the process-global meter provider. + #[must_use] + pub fn from_global() -> Self { + Self::new(&opentelemetry::global::meter(METER_NAME)) + } +} + +impl CredStoreMetricsPort for CredStoreMetricsMeter { + fn record_inventory(&self, counts: SecretCounts) { + self.secrets + .record(counts.private, &[KeyValue::new("sharing", "private")]); + self.secrets + .record(counts.tenant, &[KeyValue::new("sharing", "tenant")]); + self.secrets + .record(counts.shared, &[KeyValue::new("sharing", "shared")]); + self.secrets_provisioning.record(counts.provisioning, &[]); + self.secrets_deprovisioning + .record(counts.deprovisioning, &[]); + self.tenants_with_secrets.record(counts.tenants, &[]); + } + + fn read_outcome(&self, outcome: ReadOutcome) { + self.read_outcome + .add(1, &[KeyValue::new("outcome", outcome.as_str())]); + } + + fn walkup_depth(&self, depth: u64) { + self.walkup_depth.record(depth, &[]); + } + + fn dependency(&self, dep: Dep, op: DepOp, outcome: Outcome, secs: f64) { + self.dependency_query_duration.record( + secs, + &[ + KeyValue::new("dependency", dep.as_str()), + KeyValue::new("operation", op.as_str()), + ], + ); + self.dependency_health.add( + 1, + &[ + KeyValue::new("dependency", dep.as_str()), + KeyValue::new("operation", op.as_str()), + KeyValue::new("outcome", outcome.as_str()), + ], + ); + } + + fn provisioning_reaped(&self, n: u64) { + self.provisioning_reaped.add(n, &[]); + } + + fn deprovisioning_reaped(&self, n: u64) { + self.deprovisioning_reaped.add(n, &[]); + } + + fn provisioning_rollback(&self, outcome: Outcome) { + self.provisioning_rollback + .add(1, &[KeyValue::new("outcome", outcome.as_str())]); + } + + fn cross_tenant_denied(&self) { + self.cross_tenant_denied.add(1, &[]); + } + + fn fence_verify(&self, outcome: FenceVerify) { + self.fence_verify + .add(1, &[KeyValue::new("outcome", outcome.as_str())]); + } + + fn fence_backfill(&self, outcome: Outcome) { + self.fence_backfill + .add(1, &[KeyValue::new("outcome", outcome.as_str())]); + } +} + +#[cfg(feature = "test-support")] +pub mod test_harness { + //! In-memory OpenTelemetry harness for asserting emitted credstore metrics. + #![allow(clippy::expect_used, clippy::missing_panics_doc, dead_code)] + + use opentelemetry::metrics::{Meter, MeterProvider}; + use opentelemetry_sdk::metrics::data::{AggregatedMetrics, MetricData}; + use opentelemetry_sdk::metrics::{InMemoryMetricExporter, PeriodicReader, SdkMeterProvider}; + + use super::{CredStoreMetricsMeter, METER_NAME}; + + /// In-memory meter provider + exporter for unit and integration tests. + pub struct MetricsHarness { + provider: SdkMeterProvider, + exporter: InMemoryMetricExporter, + } + + impl MetricsHarness { + #[must_use] + pub fn new() -> Self { + let exporter = InMemoryMetricExporter::default(); + let provider = SdkMeterProvider::builder() + .with_reader(PeriodicReader::builder(exporter.clone()).build()) + .build(); + Self { provider, exporter } + } + + #[must_use] + pub fn meter(&self) -> Meter { + self.provider.meter(METER_NAME) + } + + /// A metrics handle bound to this harness's provider. + #[must_use] + pub fn metrics(&self) -> CredStoreMetricsMeter { + CredStoreMetricsMeter::new(&self.meter()) + } + + /// Flush aggregated data into the in-memory exporter. + pub fn force_flush(&self) { + self.provider + .force_flush() + .expect("test meter provider should flush"); + } + + /// Sum all matching `u64` counter data points. + #[must_use] + pub fn counter_value(&self, name: &str, expected_attrs: &[(&str, &str)]) -> u64 { + let metrics = self + .exporter + .get_finished_metrics() + .expect("in-memory exporter should be readable"); + let mut total = 0u64; + for rm in &metrics { + for sm in rm.scope_metrics() { + for metric in sm.metrics() { + if metric.name() == name + && let AggregatedMetrics::U64(MetricData::Sum(sum)) = metric.data() + { + for dp in sum.data_points() { + if attributes_match(dp.attributes(), expected_attrs) { + total += dp.value(); + } + } + } + } + } + } + total + } + + /// Sum matching histogram sample counts. + #[must_use] + pub fn histogram_count(&self, name: &str, expected_attrs: &[(&str, &str)]) -> u64 { + let metrics = self + .exporter + .get_finished_metrics() + .expect("in-memory exporter should be readable"); + let mut total = 0u64; + for rm in &metrics { + for sm in rm.scope_metrics() { + for metric in sm.metrics() { + if metric.name() == name + && let AggregatedMetrics::F64(MetricData::Histogram(hist)) = + metric.data() + { + for dp in hist.data_points() { + if attributes_match(dp.attributes(), expected_attrs) { + total += dp.count(); + } + } + } + } + } + } + total + } + } + + impl Default for MetricsHarness { + fn default() -> Self { + Self::new() + } + } + + fn attributes_match<'a>( + actual_attrs: impl Iterator, + expected: &[(&str, &str)], + ) -> bool { + let actual = actual_attrs.collect::>(); + expected.iter().all(|(k, v)| { + actual + .iter() + .any(|kv| kv.key.as_str() == *k && kv.value.as_str() == *v) + }) && actual.len() == expected.len() + } +} + +#[cfg(test)] +#[path = "metrics_tests.rs"] +mod tests; diff --git a/gears/credstore/credstore/src/infra/metrics_tests.rs b/gears/credstore/credstore/src/infra/metrics_tests.rs new file mode 100644 index 000000000..aab9ed2cf --- /dev/null +++ b/gears/credstore/credstore/src/infra/metrics_tests.rs @@ -0,0 +1,156 @@ +//! Unit tests for the OpenTelemetry-backed [`CredStoreMetricsMeter`]. + +#[cfg(feature = "test-support")] +use super::test_harness::MetricsHarness; +#[cfg(feature = "test-support")] +use crate::domain::ports::metrics::{ + CredStoreMetricsPort, Dep, DepOp, FenceVerify, Outcome, ReadOutcome, +}; + +/// Smoke test that exercises instrument construction and every recording +/// path against the global (no-op) meter — no SDK exporter required, so it +/// runs under the default feature set (unlike the `test-support` tests). +#[test] +fn global_meter_records_all_instruments() { + use super::CredStoreMetricsMeter; + use crate::domain::ports::metrics::{ + CredStoreMetricsPort, Dep, DepOp, Outcome, ReadOutcome, SecretCounts, + }; + + use crate::domain::ports::metrics::FenceVerify; + + let m = CredStoreMetricsMeter::from_global(); + assert!(!format!("{m:?}").is_empty()); + + m.record_inventory(SecretCounts { + private: 1, + tenant: 2, + shared: 3, + provisioning: 4, + deprovisioning: 1, + tenants: 2, + }); + for outcome in [ + ReadOutcome::HitOwn, + ReadOutcome::HitInherited, + ReadOutcome::Miss, + ] { + m.read_outcome(outcome); + } + m.walkup_depth(2); + m.dependency(Dep::Plugin, DepOp::PluginGet, Outcome::Success, 0.01); + m.dependency(Dep::Pdp, DepOp::Evaluate, Outcome::Error, 0.02); + m.provisioning_reaped(5); + m.provisioning_rollback(Outcome::Success); + m.provisioning_rollback(Outcome::Error); + m.cross_tenant_denied(); + m.fence_verify(FenceVerify::Ok); + m.fence_verify(FenceVerify::Legacy); + m.fence_verify(FenceVerify::Mismatch); + m.fence_backfill(Outcome::Success); +} + +#[test] +#[cfg(feature = "test-support")] +fn read_outcome_emits_with_label() { + let h = MetricsHarness::new(); + let m = h.metrics(); + m.read_outcome(ReadOutcome::HitInherited); + m.read_outcome(ReadOutcome::Miss); + h.force_flush(); + assert_eq!( + h.counter_value( + "credstore_read_outcome_total", + &[("outcome", "hit_inherited")] + ), + 1 + ); + assert_eq!( + h.counter_value("credstore_read_outcome_total", &[("outcome", "miss")]), + 1 + ); +} + +#[test] +#[cfg(feature = "test-support")] +fn dependency_emits_duration_and_health() { + let h = MetricsHarness::new(); + let m = h.metrics(); + m.dependency(Dep::Plugin, DepOp::PluginGet, Outcome::Success, 0.005); + h.force_flush(); + assert_eq!( + h.histogram_count( + "credstore_dependency_query_duration_seconds", + &[("dependency", "plugin"), ("operation", "plugin_get")] + ), + 1 + ); + assert_eq!( + h.counter_value( + "credstore_dependency_health_total", + &[ + ("dependency", "plugin"), + ("operation", "plugin_get"), + ("outcome", "success") + ] + ), + 1 + ); +} + +#[test] +#[cfg(feature = "test-support")] +fn fence_counters_emit_with_outcome_labels() { + let h = MetricsHarness::new(); + let m = h.metrics(); + m.fence_verify(FenceVerify::Ok); + m.fence_verify(FenceVerify::Mismatch); + m.fence_verify(FenceVerify::Mismatch); + m.fence_backfill(Outcome::Success); + m.fence_backfill(Outcome::NotFound); + h.force_flush(); + assert_eq!( + h.counter_value("credstore_fence_verify_total", &[("outcome", "ok")]), + 1 + ); + assert_eq!( + h.counter_value("credstore_fence_verify_total", &[("outcome", "mismatch")]), + 2 + ); + assert_eq!( + h.counter_value("credstore_fence_backfill_total", &[("outcome", "success")]), + 1 + ); + assert_eq!( + h.counter_value( + "credstore_fence_backfill_total", + &[("outcome", "not_found")] + ), + 1 + ); +} + +#[test] +#[cfg(feature = "test-support")] +fn provisioning_rollback_emits_with_outcome_label() { + let h = MetricsHarness::new(); + let m = h.metrics(); + m.provisioning_rollback(Outcome::Success); + m.provisioning_rollback(Outcome::Error); + m.provisioning_rollback(Outcome::Error); + h.force_flush(); + assert_eq!( + h.counter_value( + "credstore_provisioning_rollback_total", + &[("outcome", "success")] + ), + 1 + ); + assert_eq!( + h.counter_value( + "credstore_provisioning_rollback_total", + &[("outcome", "error")] + ), + 2 + ); +} diff --git a/gears/credstore/credstore/src/infra/plugin_select.rs b/gears/credstore/credstore/src/infra/plugin_select.rs new file mode 100644 index 000000000..d1550710d --- /dev/null +++ b/gears/credstore/credstore/src/infra/plugin_select.rs @@ -0,0 +1,97 @@ +use std::sync::Arc; + +use async_trait::async_trait; +use credstore_sdk::{CredStorePluginClientV1, CredStorePluginSpecV1}; +use toolkit::client_hub::{ClientHub, ClientScope}; +use toolkit::plugins::{GtsPluginSelector, choose_plugin_instance}; +use types_registry_sdk::{InstanceQuery, TypesRegistryClient}; + +use crate::domain::error::DomainError; +use crate::domain::ports::plugin::PluginSelector; + +/// Resolves the active `CredStore` backend plugin via the GTS types-registry. +pub struct GtsCredStorePluginSelector { + hub: Arc, + vendor: String, + selector: GtsPluginSelector, +} + +impl GtsCredStorePluginSelector { + /// Creates a new selector with lazy plugin resolution. + #[must_use] + pub fn new(hub: Arc, vendor: String) -> Self { + Self { + hub, + vendor, + selector: GtsPluginSelector::new(), + } + } + + async fn resolve_instance(&self) -> Result { + // Wire-visible details stay curated (`with_detail` contract); raw + // dependency errors live only in the cause chain / logs. + let registry = self.hub.get::().map_err(|e| { + tracing::warn!(err = %e, "types-registry client unavailable"); + DomainError::ServiceUnavailable { + detail: "types registry unavailable".to_owned(), + retry_after: None, + cause: Some(Box::new(e)), + } + })?; + + let type_id = CredStorePluginSpecV1::gts_type_id(); + let instances = registry + .list_instances(InstanceQuery::new().with_pattern(format!("{type_id}*"))) + .await + .map_err(|e| { + tracing::warn!(err = %e, "types-registry list_instances failed"); + DomainError::ServiceUnavailable { + detail: "types registry unavailable".to_owned(), + retry_after: None, + cause: Some(Box::new(e)), + } + })?; + + let gts_id = choose_plugin_instance::( + &self.vendor, + instances.iter().map(|e| (e.id.as_ref(), &e.object)), + ) + .map_err(|e| match e { + toolkit::plugins::ChoosePluginError::PluginNotFound { vendor, .. } => { + DomainError::ServiceUnavailable { + detail: format!("no credstore plugin found for vendor '{vendor}'"), + retry_after: None, + cause: None, + } + } + toolkit::plugins::ChoosePluginError::InvalidPluginInstance { gts_id, reason } => { + DomainError::Internal { + diagnostic: format!("invalid credstore plugin instance '{gts_id}': {reason}"), + cause: None, + } + } + })?; + + Ok(gts_id) + } +} + +#[async_trait] +impl PluginSelector for GtsCredStorePluginSelector { + async fn resolve(&self) -> Result, DomainError> { + let instance_id = self + .selector + .get_or_init(|| self.resolve_instance()) + .await?; + + let scope = ClientScope::gts_id(instance_id.as_ref()); + + self.hub + .try_get_scoped::(&scope) + .ok_or_else(|| DomainError::ServiceUnavailable { + detail: format!("credstore plugin client not registered yet for '{instance_id}'"), + retry_after: None, + cause: None, + }) + } +} diff --git a/gears/credstore/credstore/src/infra/sdk_error_mapping.rs b/gears/credstore/credstore/src/infra/sdk_error_mapping.rs new file mode 100644 index 000000000..8444fef04 --- /dev/null +++ b/gears/credstore/credstore/src/infra/sdk_error_mapping.rs @@ -0,0 +1,150 @@ +//! `DomainError` → [`CanonicalError`] boundary mapping for the credstore REST layer. + +use toolkit_canonical_errors::{CanonicalError, resource_error}; + +use crate::domain::error::DomainError; + +// --------------------------------------------------------------------------- +// Resource marker +// --------------------------------------------------------------------------- + +#[resource_error(gts_id!("cf.core.credstore.secret.v1~"))] +pub(crate) struct SecretResource; + +// --------------------------------------------------------------------------- +// DomainError → CanonicalError +// --------------------------------------------------------------------------- + +impl From for CanonicalError { + fn from(err: DomainError) -> Self { + match err { + DomainError::InvalidSecretRef { detail } => SecretResource::invalid_argument() + .with_field_violation("reference", detail, "INVALID_SECRET_REF") + .create(), + DomainError::NotFound => SecretResource::not_found("secret not found") + .with_resource("secret") + .create(), + DomainError::Conflict => SecretResource::already_exists("secret already exists") + .with_resource("secret") + .create(), + // No 412 in the canonical model; optimistic-lock conflicts are Aborted (409). + DomainError::VersionConflict => { + SecretResource::aborted("secret version precondition failed") + .with_reason("OPTIMISTIC_LOCK_FAILURE") + .create() + } + DomainError::InvalidPrecondition { detail } => SecretResource::invalid_argument() + .with_field_violation("If-Match", detail, "INVALID_IF_MATCH") + .create(), + DomainError::TypeViolation { + field, + reason, + detail, + } => SecretResource::invalid_argument() + .with_field_violation(field, detail, reason) + .create(), + DomainError::UnsupportedTransition { detail } => SecretResource::failed_precondition() + .with_precondition_violation("sharing", detail, "UNSUPPORTED_TRANSITION") + .create(), + DomainError::AccessDenied { .. } => SecretResource::permission_denied() + .with_reason("ACCESS_DENIED") + .create(), + DomainError::ServiceUnavailable { + detail, + retry_after, + .. + } => { + let mut builder = CanonicalError::service_unavailable().with_detail(detail); + if let Some(duration) = retry_after { + builder = builder.with_retry_after_seconds(duration.as_secs()); + } + builder.create() + } + DomainError::Internal { diagnostic, .. } => { + CanonicalError::internal(diagnostic).create() + } + #[allow(unreachable_patterns)] + other => { + CanonicalError::internal(format!("unmapped DomainError variant: {other}")).create() + } + } + } +} + +#[cfg(test)] +mod tests { + use std::time::Duration; + + use toolkit_canonical_errors::CanonicalError; + + use crate::domain::error::DomainError; + + fn status_of(err: DomainError) -> u16 { + CanonicalError::from(err).status_code() + } + + #[test] + fn every_variant_maps_to_a_client_or_server_error() { + assert_eq!(status_of(DomainError::NotFound), 404); + assert_eq!(status_of(DomainError::Conflict), 409); + assert_eq!( + status_of(DomainError::InvalidSecretRef { + detail: "bad".to_owned() + }), + 400 + ); + // Exact pins (not `>= 400`): the handler docs promise these specific + // codes, so a regression that mapped them to 500 must fail the suite + // (review finding #14). + assert_eq!( + status_of(DomainError::UnsupportedTransition { + detail: "no".to_owned() + }), + 400 + ); + assert_eq!(status_of(DomainError::AccessDenied { cause: None }), 403); + assert_eq!(status_of(DomainError::internal("boom")), 500); + // No 412 in the canonical model: optimistic-lock conflicts are Aborted (409). + assert_eq!(status_of(DomainError::VersionConflict), 409); + assert_eq!( + status_of(DomainError::InvalidPrecondition { + detail: "bad".to_owned() + }), + 400 + ); + } + + #[test] + fn service_unavailable_carries_retry_after() { + assert_eq!( + status_of(DomainError::ServiceUnavailable { + detail: "later".to_owned(), + retry_after: Some(Duration::from_secs(30)), + cause: None, + }), + 503 + ); + // Without retry_after the other branch is taken. + assert_eq!( + status_of(DomainError::ServiceUnavailable { + detail: "later".to_owned(), + retry_after: None, + cause: None, + }), + 503 + ); + } + + #[test] + fn resource_error_string_matches_sdk_constant() { + // The `#[resource_error(...)]` literal on `SecretResource` must equal the + // SDK's single source of truth (`credstore_sdk::SECRET_RESOURCE_TYPE`); + // a divergence trips here at test time, not in production. NotFound goes + // through the `SecretResource` marker, so the built error carries the type. + let err = CanonicalError::from(DomainError::NotFound); + assert_eq!( + err.resource_type(), + Some(credstore_sdk::SECRET_RESOURCE_TYPE) + ); + } +} diff --git a/gears/credstore/credstore/src/infra/storage.rs b/gears/credstore/credstore/src/infra/storage.rs new file mode 100644 index 000000000..d335fefc1 --- /dev/null +++ b/gears/credstore/credstore/src/infra/storage.rs @@ -0,0 +1,4 @@ +//! `SeaORM` entities, migrations, and repository implementation. +pub mod entity; +pub mod migrations; +pub mod repo_impl; diff --git a/gears/credstore/credstore/src/infra/storage/entity.rs b/gears/credstore/credstore/src/infra/storage/entity.rs new file mode 100644 index 000000000..e61281d68 --- /dev/null +++ b/gears/credstore/credstore/src/infra/storage/entity.rs @@ -0,0 +1,3 @@ +//! `SeaORM` entities for `credstore`. +pub mod secrets; +pub mod tenant_closure; diff --git a/gears/credstore/credstore/src/infra/storage/entity/secrets.rs b/gears/credstore/credstore/src/infra/storage/entity/secrets.rs new file mode 100644 index 000000000..472703784 --- /dev/null +++ b/gears/credstore/credstore/src/infra/storage/entity/secrets.rs @@ -0,0 +1,48 @@ +//! `SeaORM` entity for the `credstore_secrets` table. +//! +//! Tenant-scoped (`tenant_col = "tenant_id"`, `resource_col = "id"`). +//! Sharing and status columns are stored as `SMALLINT` at the DB level +//! and mapped to typed enums in the repository layer. + +use sea_orm::entity::prelude::*; +use time::OffsetDateTime; +use toolkit_db::secure::Scopable; +use uuid::Uuid; + +#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel, Scopable)] +#[sea_orm(table_name = "credstore_secrets")] +#[secure(tenant_col = "tenant_id", resource_col = "id", no_owner, no_type)] +pub struct Model { + #[sea_orm(primary_key, auto_increment = false)] + pub id: Uuid, + pub tenant_id: Uuid, + pub reference: String, + /// Sharing mode: 1=Private, 2=Tenant, 3=Shared. + pub sharing: i16, + pub owner_id: Uuid, + /// Status: 1=Provisioning, 2=Active, 3=Deprovisioning. + pub status: i16, + pub created_at: OffsetDateTime, + pub updated_at: OffsetDateTime, + /// Monotonic version for optimistic locking; seeded at 1 on insert, + /// bumped by `touch`. + pub version: i64, + /// Deterministic v5 UUID of the secret's GTS type id (resolved to the + /// type id + traits via the types-registry in the domain layer). + pub secret_type_uuid: Uuid, + /// Expiry instant for expirable types. + pub expires_at: Option, + /// Value-fingerprint fence: `HMAC-SHA256(fence_key, value)` of the value + /// this row's metadata was written for. NULL only on out-of-band seeded + /// rows (backfilled on first read / reaper sweep). Never leaves the + /// gear. + pub value_fp: Option>, + /// Id of the fence key `value_fp` was computed under (keyring + /// groundwork; NULL exactly when `value_fp` is NULL). + pub fp_key_id: Option, +} + +#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)] +pub enum Relation {} + +impl ActiveModelBehavior for ActiveModel {} diff --git a/gears/credstore/credstore/src/infra/storage/entity/tenant_closure.rs b/gears/credstore/credstore/src/infra/storage/entity/tenant_closure.rs new file mode 100644 index 000000000..f637fd4fb --- /dev/null +++ b/gears/credstore/credstore/src/infra/storage/entity/tenant_closure.rs @@ -0,0 +1,36 @@ +//! Read-only `SeaORM` entity for the `tenant_closure` system table. +//! +//! `#[secure(unrestricted)]` — this is a platform-managed hierarchy table, +//! not a tenant-scoped resource. Queried only to check descendant membership. +//! +//! **Ownership / creation**: credstore does **not** create this table — it is +//! owned and migrated by Account Management (AM's `m0001_initial_schema`). +//! credstore only reads it, and only in the **co-located** deployment where the +//! table lives in the same database as `credstore_secrets` (config +//! `hierarchy.tenant_closure_colocated`, default `false` — see DESIGN §4.4). +//! When co-located, the gear advertises the `TenantHierarchy` PDP capability and +//! the PDP emits `InTenantSubtree` predicates resolved by a closure subquery; +//! when not, the PDP emits a flat `In` list of tenant ids and this table is +//! never queried (degraded-but-correct mode), so a standalone credstore database +//! without the table is a supported configuration. Tests provision the table via +//! a test-only `CreateTenantClosure` migration. + +use sea_orm::entity::prelude::*; +use toolkit_db::secure::Scopable; +use uuid::Uuid; + +#[derive(Clone, Debug, PartialEq, Eq, DeriveEntityModel, Scopable)] +#[sea_orm(table_name = "tenant_closure")] +#[secure(unrestricted)] +pub struct Model { + #[sea_orm(primary_key, auto_increment = false)] + pub ancestor_id: Uuid, + #[sea_orm(primary_key, auto_increment = false)] + pub descendant_id: Uuid, + pub barrier: i16, +} + +#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)] +pub enum Relation {} + +impl ActiveModelBehavior for ActiveModel {} diff --git a/gears/credstore/credstore/src/infra/storage/migrations.rs b/gears/credstore/credstore/src/infra/storage/migrations.rs new file mode 100644 index 000000000..9753760e5 --- /dev/null +++ b/gears/credstore/credstore/src/infra/storage/migrations.rs @@ -0,0 +1,20 @@ +//! `SeaORM` migrations for the `credstore` module. +//! +//! * `m0001_initial_schema` — the full `credstore_secrets` table: lifecycle +//! statuses (1=provisioning, 2=active, 3=deprovisioning), the monotonic +//! `version` column, GTS secret typing (`secret_type`, `expires_at`), and +//! all indexes. The stateful gear, the deprovisioning saga, and secret +//! types shipped together, so the gear starts from one consolidated schema. + +use sea_orm_migration::prelude::*; + +pub mod m0001_initial_schema; + +pub struct Migrator; + +#[async_trait::async_trait] +impl MigratorTrait for Migrator { + fn migrations() -> Vec> { + vec![Box::new(m0001_initial_schema::Migration)] + } +} diff --git a/gears/credstore/credstore/src/infra/storage/migrations/m0001_initial_schema.rs b/gears/credstore/credstore/src/infra/storage/migrations/m0001_initial_schema.rs new file mode 100644 index 000000000..e48405e1a --- /dev/null +++ b/gears/credstore/credstore/src/infra/storage/migrations/m0001_initial_schema.rs @@ -0,0 +1,117 @@ +//! Initial credstore schema — the `credstore_secrets` table with the full +//! lifecycle status set (`provisioning`/`active`/`deprovisioning`), the +//! monotonic `version` column, GTS secret typing (`secret_type_uuid`, +//! `expires_at`), the value-fingerprint fence columns (`value_fp`, +//! `fp_key_id` — see `docs/features/001-value-fingerprint-fence.md`), and +//! all indexes. Both fence columns are NULL together (out-of-band seeded +//! rows) or set together (API-written rows), enforced by CHECK. +//! +//! The secret type is stored as the deterministic v5 UUID of its GTS type +//! id (like `tenants.tenant_type_uuid` in AM) — compact, and resolvable to +//! the type id + traits via the types-registry at operation time. +//! +//! Per-backend raw `SQL` is used (not `SeaORM`'s schema-builder) so that +//! `CHECK` constraints and partial unique indexes are preserved verbatim. +//! `MySQL` is not supported; the migration fails fast with a typed error. + +use credstore_sdk::types::GENERIC_TYPE_UUID_STR; +use sea_orm_migration::prelude::*; +use sea_orm_migration::sea_orm::ConnectionTrait; + +const MYSQL_NOT_SUPPORTED: &str = "credstore migrations: MySQL is not supported \ + (this migration set targets PostgreSQL/SQLite)"; + +#[derive(DeriveMigrationName)] +pub struct Migration; + +#[async_trait::async_trait] +impl MigrationTrait for Migration { + async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> { + let backend = manager.get_database_backend(); + let conn = manager.get_connection(); + + // Default type for rows written before typed clients: the v5 UUID of + // the `generic` type (exactly the untyped semantics). SQLite stores + // UUIDs as 16-byte blobs, so its DEFAULT is the hex-blob literal of + // the same pinned value. + let generic_uuid_hex = GENERIC_TYPE_UUID_STR.replace('-', ""); + + let table_sql = match backend { + sea_orm::DatabaseBackend::Postgres => format!( + r" +CREATE TABLE IF NOT EXISTS credstore_secrets ( + id UUID PRIMARY KEY, + tenant_id UUID NOT NULL, + reference TEXT NOT NULL CHECK (length(reference) BETWEEN 1 AND 255), + sharing SMALLINT NOT NULL CHECK (sharing IN (1, 2, 3)), + owner_id UUID NOT NULL, + status SMALLINT NOT NULL CHECK (status IN (1, 2, 3)), + created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP, + version BIGINT NOT NULL DEFAULT 1, + secret_type_uuid UUID NOT NULL DEFAULT '{GENERIC_TYPE_UUID_STR}', + expires_at TIMESTAMPTZ NULL, + value_fp BYTEA NULL, + fp_key_id SMALLINT NULL, + CHECK ((value_fp IS NULL) = (fp_key_id IS NULL)) +); + " + ), + sea_orm::DatabaseBackend::Sqlite => format!( + r" +CREATE TABLE IF NOT EXISTS credstore_secrets ( + id BLOB PRIMARY KEY NOT NULL, + tenant_id BLOB NOT NULL, + reference TEXT NOT NULL CHECK (length(reference) BETWEEN 1 AND 255), + sharing SMALLINT NOT NULL CHECK (sharing IN (1, 2, 3)), + owner_id BLOB NOT NULL, + status SMALLINT NOT NULL CHECK (status IN (1, 2, 3)), + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + version BIGINT NOT NULL DEFAULT 1, + secret_type_uuid BLOB NOT NULL DEFAULT (x'{generic_uuid_hex}'), + expires_at TEXT NULL, + value_fp BLOB NULL, + fp_key_id SMALLINT NULL, + CHECK ((value_fp IS NULL) = (fp_key_id IS NULL)) +); + " + ), + sea_orm::DatabaseBackend::MySql => { + return Err(DbErr::Custom(MYSQL_NOT_SUPPORTED.to_owned())); + } + }; + + let statements = [ + table_sql.as_str(), + // Coexistence of a private and a tenant/shared secret under one reference: + "CREATE UNIQUE INDEX IF NOT EXISTS uq_credstore_nonprivate ON credstore_secrets (tenant_id, reference) WHERE sharing <> 1;", + "CREATE UNIQUE INDEX IF NOT EXISTS uq_credstore_private ON credstore_secrets (tenant_id, reference, owner_id) WHERE sharing = 1;", + // Walk-up resolution: + "CREATE INDEX IF NOT EXISTS idx_credstore_lookup ON credstore_secrets (reference, tenant_id, status);", + // Reaper sweep over all non-active (saga) rows: + "CREATE INDEX IF NOT EXISTS idx_credstore_pending ON credstore_secrets (updated_at) WHERE status <> 2;", + // Reaper expiry sweep over active rows carrying an expiry: + "CREATE INDEX IF NOT EXISTS idx_credstore_expiry ON credstore_secrets (expires_at) WHERE expires_at IS NOT NULL AND status = 2;", + ]; + + for sql in statements { + conn.execute_unprepared(sql).await?; + } + Ok(()) + } + + async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> { + if matches!( + manager.get_database_backend(), + sea_orm::DatabaseBackend::MySql + ) { + return Err(DbErr::Custom(MYSQL_NOT_SUPPORTED.to_owned())); + } + manager + .get_connection() + .execute_unprepared("DROP TABLE IF EXISTS credstore_secrets;") + .await?; + Ok(()) + } +} diff --git a/gears/credstore/credstore/src/infra/storage/repo_impl.rs b/gears/credstore/credstore/src/infra/storage/repo_impl.rs new file mode 100644 index 000000000..fdb276310 --- /dev/null +++ b/gears/credstore/credstore/src/infra/storage/repo_impl.rs @@ -0,0 +1,219 @@ +//! `SeaORM`-backed implementation of [`SecretRepo`]. + +pub mod helpers; +mod reads; +mod writes; + +#[cfg(test)] +mod repo_tests; + +use std::sync::Arc; + +use async_trait::async_trait; +use credstore_sdk::{OwnerId, SecretRef, SharingMode, TenantId}; +use time::OffsetDateTime; +use toolkit_db::DBProvider; +use toolkit_security::AccessScope; +use uuid::Uuid; + +use crate::domain::error::DomainError; +use crate::domain::ports::metrics::SecretCounts; +use crate::domain::secret::model::{SecretRow, SecretStatus}; +use crate::domain::secret::repo::SecretRepo; +use crate::infra::storage::entity; + +pub type CredstoreDbProvider = DBProvider; + +/// `SeaORM` repository adapter for [`SecretRepo`]. +pub struct SecretRepoImpl { + pub(crate) db: Arc, +} + +impl SecretRepoImpl { + #[must_use] + pub fn new(db: Arc) -> Self { + Self { db } + } +} + +#[async_trait] +impl SecretRepo for SecretRepoImpl { + async fn resolve_for_get( + &self, + req_tenant: TenantId, + subject: OwnerId, + key: &SecretRef, + chain: &[Uuid], + ) -> Result, DomainError> { + reads::resolve_for_get(self, req_tenant, subject, key, chain).await + } + + async fn insert_provisioning( + &self, + scope: &AccessScope, + new: &crate::domain::secret::model::NewSecret, + ) -> Result<(), DomainError> { + writes::insert_provisioning(self, scope, new).await + } + + async fn mark_active(&self, scope: &AccessScope, id: Uuid) -> Result<(), DomainError> { + writes::mark_active(self, scope, id).await + } + + async fn touch( + &self, + scope: &AccessScope, + id: Uuid, + sharing: SharingMode, + expected_version: Option, + expires_at: Option, + value_fp: Vec, + ) -> Result, DomainError> { + writes::touch( + self, + scope, + id, + sharing, + expected_version, + expires_at, + value_fp, + ) + .await + } + + async fn backfill_fp( + &self, + id: Uuid, + value_fp: Vec, + fp_key_id: i16, + ) -> Result { + writes::backfill_fp(self, id, value_fp, fp_key_id).await + } + + async fn list_unfenced(&self, limit: u64) -> Result, DomainError> { + writes::list_unfenced(self, limit).await + } + + async fn find_own( + &self, + scope: &AccessScope, + tenant: TenantId, + subject: OwnerId, + key: &SecretRef, + ) -> Result, DomainError> { + reads::find_own(self, scope, tenant, subject, key).await + } + + async fn find_for_write( + &self, + scope: &AccessScope, + tenant: TenantId, + subject: OwnerId, + key: &SecretRef, + sharing: SharingMode, + ) -> Result, DomainError> { + reads::find_for_write(self, scope, tenant, subject, key, sharing).await + } + + async fn delete_by_id( + &self, + scope: &AccessScope, + id: Uuid, + expected_version: Option, + ) -> Result<(), DomainError> { + writes::delete_by_id(self, scope, id, expected_version).await + } + + async fn mark_deprovisioning( + &self, + scope: &AccessScope, + id: Uuid, + expected_version: Option, + ) -> Result { + writes::mark_deprovisioning(self, scope, id, expected_version).await + } + + async fn list_stale_pending( + &self, + provisioning_older_than_secs: u64, + deprovisioning_older_than_secs: u64, + limit: u64, + ) -> Result, DomainError> { + writes::list_stale_pending( + self, + provisioning_older_than_secs, + deprovisioning_older_than_secs, + limit, + ) + .await + } + + async fn reap_by_id(&self, id: Uuid, expected: SecretStatus) -> Result { + writes::reap_by_id(self, id, expected).await + } + + async fn mark_expired_deprovisioning(&self) -> Result { + writes::mark_expired_deprovisioning(self).await + } + + async fn inventory(&self) -> Result { + reads::inventory(self).await + } + + async fn scope_includes_tenant( + &self, + scope: &AccessScope, + tenant: Uuid, + ) -> Result { + reads::scope_includes_tenant(self, scope, tenant).await + } +} + +/// Map an entity row to the domain [`SecretRow`]. +pub(crate) fn entity_to_model(m: entity::secrets::Model) -> Result { + let sharing = sharing_from_i16(m.sharing).ok_or_else(|| DomainError::Internal { + diagnostic: format!( + "credstore_secrets.sharing out-of-domain value: {}", + m.sharing + ), + cause: None, + })?; + let status = SecretStatus::from_smallint(m.status).ok_or_else(|| DomainError::Internal { + diagnostic: format!("credstore_secrets.status out-of-domain value: {}", m.status), + cause: None, + })?; + Ok(SecretRow { + id: m.id, + tenant_id: TenantId(m.tenant_id), + reference: m.reference, + sharing, + owner_id: OwnerId(m.owner_id), + status, + version: m.version, + // Opaque here: the domain layer resolves the UUID to the type id + + // traits via the types-registry, so non-catalog types round-trip. + secret_type_uuid: m.secret_type_uuid, + expires_at: m.expires_at, + value_fp: m.value_fp, + fp_key_id: m.fp_key_id, + }) +} + +/// Map [`SharingMode`] to its `SMALLINT` storage value. +pub(crate) fn sharing_to_i16(s: SharingMode) -> i16 { + match s { + SharingMode::Private => 1, + SharingMode::Tenant => 2, + SharingMode::Shared => 3, + } +} + +/// Map a `SMALLINT` storage value to [`SharingMode`]. +pub(crate) fn sharing_from_i16(v: i16) -> Option { + match v { + 1 => Some(SharingMode::Private), + 2 => Some(SharingMode::Tenant), + 3 => Some(SharingMode::Shared), + _ => None, + } +} diff --git a/gears/credstore/credstore/src/infra/storage/repo_impl/helpers.rs b/gears/credstore/credstore/src/infra/storage/repo_impl/helpers.rs new file mode 100644 index 000000000..1c6d53fb1 --- /dev/null +++ b/gears/credstore/credstore/src/infra/storage/repo_impl/helpers.rs @@ -0,0 +1,58 @@ +//! Helpers shared across the `repo_impl` split. + +use toolkit_db::secure::ScopeError; + +use crate::domain::error::DomainError; +use crate::infra::canonical_mapping::classify_db_err_to_domain; + +/// Map a [`ScopeError`] to a [`DomainError`] outside a retry boundary. +pub(super) fn map_scope_err(err: ScopeError) -> DomainError { + match err { + ScopeError::Db(db) => classify_db_err_to_domain(db), + ScopeError::Invalid(msg) => DomainError::Internal { + diagnostic: format!("scope invalid: {msg}"), + cause: None, + }, + ScopeError::TenantNotInScope { .. } => DomainError::AccessDenied { cause: None }, + ScopeError::Denied(msg) => DomainError::Internal { + diagnostic: format!("unexpected access denied in credstore repo: {msg}"), + cause: None, + }, + } +} + +#[cfg(test)] +mod tests { + use sea_orm::DbErr; + use toolkit_db::secure::ScopeError; + use uuid::Uuid; + + use super::map_scope_err; + use crate::domain::error::DomainError; + + #[test] + fn maps_each_scope_error_variant() { + assert!(matches!( + map_scope_err(ScopeError::Invalid("bad scope")), + DomainError::Internal { .. } + )); + assert!(matches!( + map_scope_err(ScopeError::TenantNotInScope { + tenant_id: Uuid::new_v4() + }), + DomainError::AccessDenied { .. } + )); + assert!(matches!( + map_scope_err(ScopeError::Denied("not accessible")), + DomainError::Internal { .. } + )); + // Db errors delegate to the classification ladder (CHECK violations + // are server-side invariants → Internal). + assert!(matches!( + map_scope_err(ScopeError::Db(DbErr::Custom( + "CHECK constraint failed".to_owned() + ))), + DomainError::Internal { .. } + )); + } +} diff --git a/gears/credstore/credstore/src/infra/storage/repo_impl/reads.rs b/gears/credstore/credstore/src/infra/storage/repo_impl/reads.rs new file mode 100644 index 000000000..d73e43e1c --- /dev/null +++ b/gears/credstore/credstore/src/infra/storage/repo_impl/reads.rs @@ -0,0 +1,338 @@ +//! Read-only repo methods: `resolve_for_get`, `find_own`, `inventory`, +//! `scope_includes_tenant`. + +use credstore_sdk::{OwnerId, SecretRef, SharingMode, TenantId}; +use sea_orm::sea_query::Expr; +use sea_orm::{ColumnTrait, Condition, EntityTrait, FromQueryResult, QuerySelect}; +use toolkit_db::secure::{ScopeError, SecureEntityExt}; +use toolkit_security::access_scope::ScopeFilter; +use toolkit_security::{AccessScope, pep_properties}; +use uuid::Uuid; + +use crate::domain::error::DomainError; +use crate::domain::ports::metrics::SecretCounts; +use crate::domain::secret::model::{SecretRow, SecretStatus}; +use crate::infra::canonical_mapping::classify_db_err_to_domain; +use crate::infra::storage::entity; +use crate::infra::storage::repo_impl::helpers::map_scope_err; +use crate::infra::storage::repo_impl::{ + SecretRepoImpl, entity_to_model, sharing_from_i16, sharing_to_i16, +}; + +/// Minimal projection for inventory aggregate rows. +#[derive(Debug, FromQueryResult)] +struct InventoryRow { + sharing: i16, + status: i16, + c: i64, +} + +/// Minimal projection for distinct-tenant count. +#[derive(Debug, FromQueryResult)] +struct TenantCount { + t: i64, +} + +fn scope_err_to_domain(e: ScopeError) -> DomainError { + match e { + ScopeError::Db(db) => classify_db_err_to_domain(db), + other => map_scope_err(other), + } +} + +pub(super) async fn resolve_for_get( + repo: &SecretRepoImpl, + req_tenant: TenantId, + subject: OwnerId, + key: &SecretRef, + chain: &[Uuid], +) -> Result, DomainError> { + let conn = repo.db.conn()?; + let req = req_tenant.0; + // resolve_for_get applies its own chain + sharing predicates; + // PDP authorization runs upstream. allow_all skips the scope WHERE clamp. + let rows = entity::secrets::Entity::find() + .secure() + .scope_with(&AccessScope::allow_all()) + .filter(Condition::all().add(entity::secrets::Column::Reference.eq(key.as_ref()))) + .filter( + Condition::all() + .add(entity::secrets::Column::Status.eq(SecretStatus::Active.as_smallint())), + ) + // Expired secrets resolve as not-found (write paths still see the + // row: overwrite refreshes it, delete revokes it, the reaper sweeps it). + .filter( + Condition::any() + .add(entity::secrets::Column::ExpiresAt.is_null()) + .add(entity::secrets::Column::ExpiresAt.gt(time::OffsetDateTime::now_utc())), + ) + .filter(Condition::all().add(entity::secrets::Column::TenantId.is_in(chain.to_vec()))) + // Visibility by sharing class, within the ancestor `chain`: + // * Private — own tenant + owner only (`tenant_id == req AND + // owner_id == subject`). Private is owner-scoped per DESIGN §4.3 and + // never inherited, so we pin it to `req` here rather than lean on + // the authn-layer invariant that a `subject_id` belongs to a single + // tenant. That keeps disclosure closed even if a `subject_id` is + // ever reused across tenants or a cross-tenant principal is + // introduced (matches `find_own`/`find_for_write`, which already + // pin the tenant). `Shared` remains the sole inheritance vector. + // * Shared — inherited down the chain (any tenant in `chain`). + // * Tenant — own tenant only (`tenant_id == req`), never inherited. + .filter( + Condition::any() + .add( + Condition::all() + .add( + entity::secrets::Column::Sharing + .eq(sharing_to_i16(SharingMode::Private)), + ) + .add(entity::secrets::Column::TenantId.eq(req)) + .add(entity::secrets::Column::OwnerId.eq(subject.0)), + ) + .add(entity::secrets::Column::Sharing.eq(sharing_to_i16(SharingMode::Shared))) + .add( + Condition::all() + .add( + entity::secrets::Column::Sharing + .eq(sharing_to_i16(SharingMode::Tenant)), + ) + .add(entity::secrets::Column::TenantId.eq(req)), + ), + ) + .all(&conn) + .await + .map_err(scope_err_to_domain)?; + + // Winner: closest tenant in chain; private beats non-private at same level. + let pos = |t: Uuid| chain.iter().position(|c| *c == t).unwrap_or(usize::MAX); + let best = rows.into_iter().min_by(|a, b| { + pos(a.tenant_id).cmp(&pos(b.tenant_id)).then( + (a.sharing != sharing_to_i16(SharingMode::Private)) + .cmp(&(b.sharing != sharing_to_i16(SharingMode::Private))), + ) + }); + best.map(entity_to_model).transpose() +} + +pub(super) async fn find_own( + repo: &SecretRepoImpl, + scope: &AccessScope, + tenant: TenantId, + subject: OwnerId, + key: &SecretRef, +) -> Result, DomainError> { + let conn = repo.db.conn()?; + // Active rows plus deprovisioning ones — a DELETE retry must be able to + // resume a stuck delete saga. Provisioning rows stay invisible. + let rows = entity::secrets::Entity::find() + .secure() + .scope_with(scope) + .filter( + Condition::all() + .add(entity::secrets::Column::Reference.eq(key.as_ref())) + .add(entity::secrets::Column::TenantId.eq(tenant.0)) + .add(entity::secrets::Column::Status.is_in([ + SecretStatus::Active.as_smallint(), + SecretStatus::Deprovisioning.as_smallint(), + ])) + .add( + Condition::any() + .add( + Condition::all() + .add( + entity::secrets::Column::Sharing + .eq(sharing_to_i16(SharingMode::Private)), + ) + .add(entity::secrets::Column::OwnerId.eq(subject.0)), + ) + .add(entity::secrets::Column::Sharing.is_in([ + sharing_to_i16(SharingMode::Tenant), + sharing_to_i16(SharingMode::Shared), + ])), + ), + ) + .all(&conn) + .await + .map_err(scope_err_to_domain)?; + + // Prefer the private row if both exist. + let best = rows + .into_iter() + .min_by_key(|r| i32::from(r.sharing != sharing_to_i16(SharingMode::Private))); + best.map(entity_to_model).transpose() +} + +pub(super) async fn find_for_write( + repo: &SecretRepoImpl, + scope: &AccessScope, + tenant: TenantId, + subject: OwnerId, + key: &SecretRef, + sharing: SharingMode, +) -> Result, DomainError> { + let conn = repo.db.conn()?; + // Address the row by the target sharing class only — the same identity the + // partial unique indexes enforce. A private write must NOT match a coexisting + // tenant/shared row (and vice-versa); they are distinct secrets per design. + let class = if sharing == SharingMode::Private { + Condition::all() + .add(entity::secrets::Column::Sharing.eq(sharing_to_i16(SharingMode::Private))) + .add(entity::secrets::Column::OwnerId.eq(subject.0)) + } else { + Condition::all().add(entity::secrets::Column::Sharing.is_in([ + sharing_to_i16(SharingMode::Tenant), + sharing_to_i16(SharingMode::Shared), + ])) + }; + let row = entity::secrets::Entity::find() + .secure() + .scope_with(scope) + .filter( + Condition::all() + .add(entity::secrets::Column::Reference.eq(key.as_ref())) + .add(entity::secrets::Column::TenantId.eq(tenant.0)) + .add(entity::secrets::Column::Status.eq(SecretStatus::Active.as_smallint())) + .add(class), + ) + .one(&conn) + .await + .map_err(scope_err_to_domain)?; + row.map(entity_to_model).transpose() +} + +#[allow(clippy::cognitive_complexity)] +pub(super) async fn inventory(repo: &SecretRepoImpl) -> Result { + let conn = repo.db.conn()?; + + // Aggregate counts grouped by sharing + status. + let rows: Vec = entity::secrets::Entity::find() + .secure() + .scope_with(&AccessScope::allow_all()) + .project_all(&conn, |q| { + q.select_only() + .column(entity::secrets::Column::Sharing) + .column(entity::secrets::Column::Status) + .column_as(entity::secrets::Column::Id.count(), "c") + .group_by(entity::secrets::Column::Sharing) + .group_by(entity::secrets::Column::Status) + .into_model::() + }) + .await + .map_err(scope_err_to_domain)?; + + let mut counts = SecretCounts::default(); + for r in rows { + // Decode through the typed enums (not magic numbers) so a future encoding + // change can't silently miscount; an unknown encoding is logged, not dropped. + match SecretStatus::from_smallint(r.status) { + Some(SecretStatus::Provisioning) => counts.provisioning += r.c, + Some(SecretStatus::Deprovisioning) => counts.deprovisioning += r.c, + Some(SecretStatus::Active) => match sharing_from_i16(r.sharing) { + Some(SharingMode::Private) => counts.private += r.c, + Some(SharingMode::Tenant) => counts.tenant += r.c, + Some(SharingMode::Shared) => counts.shared += r.c, + None => tracing::warn!( + sharing = r.sharing, + "inventory: unknown sharing encoding, row not counted" + ), + }, + None => tracing::warn!( + status = r.status, + "inventory: unknown status encoding, row not counted" + ), + } + } + + // Distinct-tenant count for active rows. + let tenant_rows: Vec = entity::secrets::Entity::find() + .secure() + .scope_with(&AccessScope::allow_all()) + .filter( + Condition::all() + .add(entity::secrets::Column::Status.eq(SecretStatus::Active.as_smallint())), + ) + .project_all(&conn, |q| { + q.select_only() + .column_as( + Expr::col(entity::secrets::Column::TenantId).count_distinct(), + "t", + ) + .into_model::() + }) + .await + .map_err(scope_err_to_domain)?; + + if let Some(row) = tenant_rows.into_iter().next() { + counts.tenants = row.t; + } + Ok(counts) +} + +pub(super) async fn scope_includes_tenant( + repo: &SecretRepoImpl, + scope: &AccessScope, + tenant: Uuid, +) -> Result { + if scope.is_unconstrained() { + return Ok(true); + } + if scope.is_deny_all() { + return Ok(false); + } + // Fail-closed tenant-membership check. A scope's constraints are OR-ed + // (alternative grants) and the filters within a constraint are AND-ed, so + // a constraint admits `tenant` only when *every* one of its filters is a + // tenant-level predicate on `OWNER_TENANT_ID` satisfied by `tenant`. Any + // sibling filter that narrows below tenant granularity (`owner_id`, + // `resource_id`, group membership, …) or any filter this gate cannot + // evaluate makes the whole constraint non-admitting. That way a scope + // stricter than tenant granularity fails closed (403) instead of being + // silently widened to the whole tenant on a lone `OWNER_TENANT_ID` match. + // The `InTenantSubtree` closure lookup hits `tenant_closure` directly, so + // an empty `credstore_secrets` table never produces a false-negative. + let conn = repo.db.conn()?; + 'constraints: for constraint in scope.constraints() { + // An empty constraint matches everything (mirrors SecureORM's + // `build_constraint_condition`, which compiles it to `WHERE true`). + for filter in constraint.filters() { + // Only `OWNER_TENANT_ID` predicates can affirm tenant-level access. + if filter.property() != pep_properties::OWNER_TENANT_ID { + continue 'constraints; + } + let admits = match filter { + ScopeFilter::Eq(_) | ScopeFilter::In(_) => { + filter.values().iter().any(|v| v.as_uuid() == Some(tenant)) + } + ScopeFilter::InTenantSubtree(sf) => match sf.root_tenant_id().as_uuid() { + None => false, + Some(root) => { + let mut closure_cond = Condition::all() + .add(entity::tenant_closure::Column::AncestorId.eq(root)) + .add(entity::tenant_closure::Column::DescendantId.eq(tenant)); + if sf.respect_barriers() { + closure_cond = + closure_cond.add(entity::tenant_closure::Column::Barrier.eq(0_i16)); + } + entity::tenant_closure::Entity::find() + .secure() + .scope_with(&AccessScope::allow_all()) + .filter(closure_cond) + .one(&conn) + .await + .map_err(scope_err_to_domain)? + .is_some() + } + }, + // Group membership over `OWNER_TENANT_ID` is not a plain + // tenant predicate this gate resolves — fail closed. + ScopeFilter::InGroup(_) | ScopeFilter::InGroupSubtree(_) => false, + }; + if !admits { + continue 'constraints; + } + } + // Every filter affirmed `tenant` (or the constraint was empty). + return Ok(true); + } + Ok(false) +} diff --git a/gears/credstore/credstore/src/infra/storage/repo_impl/repo_tests.rs b/gears/credstore/credstore/src/infra/storage/repo_impl/repo_tests.rs new file mode 100644 index 000000000..fd02e6305 --- /dev/null +++ b/gears/credstore/credstore/src/infra/storage/repo_impl/repo_tests.rs @@ -0,0 +1,1298 @@ +//! SQLite-backed integration tests for [`SecretRepoImpl`]. +//! +//! These exercise the real `SeaORM`/`SecureORM` read + write paths against an +//! in-memory SQLite database built from the module's own migration plus a +//! test-only `tenant_closure` table. No raw SQL: schema comes from migration +//! definitions, fixtures are seeded through the repository's own write methods. +#![allow( + clippy::expect_used, + clippy::unwrap_used, + clippy::panic, + clippy::doc_markdown +)] + +use std::sync::Arc; + +use credstore_sdk::{OwnerId, SecretRef, SecretType, SharingMode, TenantId}; +use sea_orm::{EntityTrait, Set}; +use sea_orm_migration::{MigratorTrait, prelude as mig}; +use toolkit_db::migration_runner::run_migrations_for_testing; +use toolkit_db::secure::SecureInsertExt; +use toolkit_db::{ConnectOpts, DBProvider, connect_db}; +use toolkit_security::{AccessScope, ScopeConstraint, ScopeFilter, pep_properties}; +use uuid::Uuid; + +use crate::domain::error::DomainError; +use crate::domain::secret::model::{NewSecret, SecretStatus}; +use crate::domain::secret::repo::SecretRepo; +use crate::infra::storage::entity; +use crate::infra::storage::migrations::Migrator; +use crate::infra::storage::repo_impl::SecretRepoImpl; + +/// Test-only migration creating the platform-owned `tenant_closure` table. +/// In production this table is owned by another platform module; for repo tests +/// we recreate its minimal shape so the `InTenantSubtree` path is exercisable. +struct CreateTenantClosure; + +impl mig::MigrationName for CreateTenantClosure { + fn name(&self) -> &'static str { + "m_test_create_tenant_closure" + } +} + +#[async_trait::async_trait] +impl mig::MigrationTrait for CreateTenantClosure { + async fn up(&self, manager: &mig::SchemaManager) -> Result<(), mig::DbErr> { + manager + .create_table( + mig::Table::create() + .table(mig::Alias::new("tenant_closure")) + .if_not_exists() + .col( + mig::ColumnDef::new(mig::Alias::new("ancestor_id")) + .uuid() + .not_null(), + ) + .col( + mig::ColumnDef::new(mig::Alias::new("descendant_id")) + .uuid() + .not_null(), + ) + .col( + mig::ColumnDef::new(mig::Alias::new("barrier")) + .small_integer() + .not_null() + .default(0), + ) + .primary_key( + mig::Index::create() + .col(mig::Alias::new("ancestor_id")) + .col(mig::Alias::new("descendant_id")), + ) + .to_owned(), + ) + .await + } + + async fn down(&self, manager: &mig::SchemaManager) -> Result<(), mig::DbErr> { + manager + .drop_table( + mig::Table::drop() + .table(mig::Alias::new("tenant_closure")) + .if_exists() + .to_owned(), + ) + .await + } +} + +/// Build a repo backed by a fresh, isolated in-memory SQLite database. +async fn setup() -> SecretRepoImpl { + let id = Uuid::new_v4(); + let dsn = format!("sqlite:file:credstore_repo_{id}?mode=memory&cache=shared"); + let db = connect_db( + &dsn, + ConnectOpts { + max_conns: Some(1), + min_conns: Some(1), + ..Default::default() + }, + ) + .await + .expect("connect sqlite"); + + let mut migrations = Migrator::migrations(); + migrations.push(Box::new(CreateTenantClosure)); + run_migrations_for_testing(&db, migrations) + .await + .expect("run migrations"); + + SecretRepoImpl::new(Arc::new(DBProvider::::new(db))) +} + +fn sref(s: &str) -> SecretRef { + SecretRef::new(s).expect("valid secret ref") +} + +/// Insert a Provisioning row, leaving it un-promoted. +async fn seed_provisioning( + repo: &SecretRepoImpl, + tenant: Uuid, + owner: Uuid, + key: &str, + sharing: SharingMode, +) -> Uuid { + let id = Uuid::new_v4(); + repo.insert_provisioning( + &AccessScope::for_tenant(tenant), + &NewSecret { + id, + tenant_id: TenantId(tenant), + reference: sref(key), + sharing, + owner_id: OwnerId(owner), + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: vec![7u8; 32], + fp_key_id: 1, + }, + ) + .await + .expect("insert_provisioning"); + id +} + +/// Insert a row and promote it to Active. +async fn seed_active( + repo: &SecretRepoImpl, + tenant: Uuid, + owner: Uuid, + key: &str, + sharing: SharingMode, +) -> Uuid { + let id = seed_provisioning(repo, tenant, owner, key, sharing).await; + repo.mark_active(&AccessScope::for_tenant(tenant), id) + .await + .expect("mark_active"); + id +} + +/// Seed a `tenant_closure` edge (`ancestor` -> `descendant`). +async fn seed_closure(repo: &SecretRepoImpl, ancestor: Uuid, descendant: Uuid, barrier: i16) { + let conn = repo.db.conn().expect("conn"); + entity::tenant_closure::Entity::insert(entity::tenant_closure::ActiveModel { + ancestor_id: Set(ancestor), + descendant_id: Set(descendant), + barrier: Set(barrier), + }) + .secure() + .scope_unchecked(&AccessScope::allow_all()) + .expect("scope") + .exec(&conn) + .await + .expect("seed closure"); +} + +fn subtree_scope(root: Uuid) -> AccessScope { + AccessScope::from_constraints(vec![ScopeConstraint::new(vec![ + ScopeFilter::in_tenant_subtree(pep_properties::OWNER_TENANT_ID, root, true, Vec::new()), + ])]) +} + +// ── write path ────────────────────────────────────────────────────────────── + +#[tokio::test] +async fn insert_then_mark_active_promotes_row() { + let repo = setup().await; + let tenant = Uuid::new_v4(); + let owner = Uuid::new_v4(); + let id = seed_provisioning(&repo, tenant, owner, "db-pw", SharingMode::Private).await; + + // Not yet active: a get over the tenant chain finds nothing. + let found = repo + .resolve_for_get(TenantId(tenant), OwnerId(owner), &sref("db-pw"), &[tenant]) + .await + .expect("resolve"); + assert!(found.is_none(), "provisioning rows are invisible to get"); + + repo.mark_active(&AccessScope::for_tenant(tenant), id) + .await + .expect("mark_active"); + + let found = repo + .resolve_for_get(TenantId(tenant), OwnerId(owner), &sref("db-pw"), &[tenant]) + .await + .expect("resolve") + .expect("active row visible"); + assert_eq!(found.id, id); +} + +#[tokio::test] +async fn mark_active_twice_conflicts() { + let repo = setup().await; + let tenant = Uuid::new_v4(); + let id = seed_active(&repo, tenant, Uuid::new_v4(), "k", SharingMode::Tenant).await; + // Second promotion finds no Provisioning row -> Conflict. + let err = repo + .mark_active(&AccessScope::for_tenant(tenant), id) + .await + .expect_err("second mark_active conflicts"); + assert!(matches!(err, DomainError::Conflict)); +} + +#[tokio::test] +async fn touch_bumps_version_and_changes_sharing() { + let repo = setup().await; + let tenant = Uuid::new_v4(); + let owner = Uuid::new_v4(); + let id = seed_active(&repo, tenant, owner, "share-me", SharingMode::Tenant).await; + + let row = repo + .touch( + &AccessScope::for_tenant(tenant), + id, + SharingMode::Shared, + None, + None, + vec![9u8; 32], + ) + .await + .expect("touch") + .expect("row updated"); + assert_eq!(row.sharing, SharingMode::Shared); + assert_eq!(row.version, 2, "insert seeds 1; touch bumps to 2"); + + // A second touch bumps again (sharing unchanged this time). + let row = repo + .touch( + &AccessScope::for_tenant(tenant), + id, + SharingMode::Shared, + None, + None, + vec![9u8; 32], + ) + .await + .expect("touch") + .expect("row updated"); + assert_eq!(row.version, 3); +} + +#[tokio::test] +async fn touch_missing_row_returns_none() { + let repo = setup().await; + let tenant = Uuid::new_v4(); + let res = repo + .touch( + &AccessScope::for_tenant(tenant), + Uuid::new_v4(), + SharingMode::Shared, + None, + None, + vec![9u8; 32], + ) + .await + .expect("touch"); + assert!(res.is_none()); +} + +#[tokio::test] +async fn touch_provisioning_row_returns_none() { + let repo = setup().await; + let tenant = Uuid::new_v4(); + let owner = Uuid::new_v4(); + // A provisioning (un-promoted) row must not be touchable — the Status==Active + // filter guards against bumping a stuck or never-promoted row. + let id = seed_provisioning(&repo, tenant, owner, "prov", SharingMode::Tenant).await; + let res = repo + .touch( + &AccessScope::for_tenant(tenant), + id, + SharingMode::Shared, + None, + None, + vec![9u8; 32], + ) + .await + .expect("touch"); + assert!(res.is_none(), "touch must not bump a provisioning row"); +} + +#[tokio::test] +async fn touch_with_matching_expected_version_bumps() { + let repo = setup().await; + let tenant = Uuid::new_v4(); + let id = seed_active(&repo, tenant, Uuid::new_v4(), "ver", SharingMode::Tenant).await; + // seed_active inserts version 1; the matching precondition bumps to 2. + let row = repo + .touch( + &AccessScope::for_tenant(tenant), + id, + SharingMode::Tenant, + Some(1), + None, + vec![9u8; 32], + ) + .await + .expect("touch") + .expect("row updated"); + assert_eq!(row.version, 2); +} + +#[tokio::test] +async fn touch_with_stale_expected_version_returns_none() { + let repo = setup().await; + let tenant = Uuid::new_v4(); + let id = seed_active(&repo, tenant, Uuid::new_v4(), "ver", SharingMode::Tenant).await; + // Version is 1; a stale precondition (99) gates the UPDATE to 0 rows. + let res = repo + .touch( + &AccessScope::for_tenant(tenant), + id, + SharingMode::Tenant, + Some(99), + None, + vec![9u8; 32], + ) + .await + .expect("touch"); + assert!(res.is_none(), "stale expected_version must match 0 rows"); +} + +#[tokio::test] +async fn delete_by_id_removes_then_not_found() { + let repo = setup().await; + let tenant = Uuid::new_v4(); + let id = seed_active(&repo, tenant, Uuid::new_v4(), "gone", SharingMode::Tenant).await; + let scope = AccessScope::for_tenant(tenant); + + repo.delete_by_id(&scope, id, None).await.expect("delete"); + let err = repo + .delete_by_id(&scope, id, None) + .await + .expect_err("second delete is NotFound"); + assert!(matches!(err, DomainError::NotFound)); +} + +#[tokio::test] +async fn delete_by_id_with_stale_expected_version_is_not_found() { + let repo = setup().await; + let tenant = Uuid::new_v4(); + let id = seed_active( + &repo, + tenant, + Uuid::new_v4(), + "ver-del", + SharingMode::Tenant, + ) + .await; + let scope = AccessScope::for_tenant(tenant); + + // Version is 1; a stale precondition (99) gates the DELETE to 0 rows. + let err = repo + .delete_by_id(&scope, id, Some(99)) + .await + .expect_err("stale expected_version must match 0 rows"); + assert!(matches!(err, DomainError::NotFound)); + + // The matching version deletes the row. + repo.delete_by_id(&scope, id, Some(1)) + .await + .expect("matching expected_version deletes"); +} + +#[tokio::test] +async fn list_stale_pending_matches_by_status_and_age() { + let repo = setup().await; + let tenant = Uuid::new_v4(); + let prov_id = + seed_provisioning(&repo, tenant, Uuid::new_v4(), "stale", SharingMode::Tenant).await; + let deprov_id = seed_active(&repo, tenant, Uuid::new_v4(), "gone", SharingMode::Tenant).await; + assert!( + repo.mark_deprovisioning(&AccessScope::for_tenant(tenant), deprov_id, None) + .await + .expect("mark deprovisioning") + ); + seed_active(&repo, tenant, Uuid::new_v4(), "live", SharingMode::Tenant).await; + + // Fresh rows survive a stale-only sweep (cutoff far in the past). + let none = repo + .list_stale_pending(100_000, 100_000, 100) + .await + .expect("list none"); + assert!(none.is_empty(), "fresh rows are not stale"); + + // older_than 0 -> cutoff = now: both non-active rows match, active never. + let stale = repo + .list_stale_pending(0, 0, 100) + .await + .expect("list stale"); + let ids: Vec = stale.iter().map(|r| r.id).collect(); + assert_eq!(stale.len(), 2); + assert!(ids.contains(&prov_id) && ids.contains(&deprov_id)); + + // Per-status cutoffs are independent. + let only_deprov = repo + .list_stale_pending(100_000, 0, 100) + .await + .expect("list deprov only"); + assert_eq!(only_deprov.len(), 1); + assert_eq!(only_deprov[0].id, deprov_id); + + // The limit bounds the batch. + let limited = repo.list_stale_pending(0, 0, 1).await.expect("limit"); + assert_eq!(limited.len(), 1); +} + +#[tokio::test] +async fn reap_by_id_is_status_gated_and_idempotent() { + let repo = setup().await; + let tenant = Uuid::new_v4(); + let id = seed_provisioning(&repo, tenant, Uuid::new_v4(), "stale", SharingMode::Tenant).await; + + // A status mismatch never removes the row (fences a concurrent transition). + assert!( + !repo + .reap_by_id(id, SecretStatus::Deprovisioning) + .await + .expect("reap wrong-status"), + "must not reap a row whose status differs from `expected`" + ); + + // Matching status removes it and reports the removal. + assert!( + repo.reap_by_id(id, SecretStatus::Provisioning) + .await + .expect("reap") + ); + // Idempotent: a second delete of a gone row is success, reporting no removal. + assert!( + !repo + .reap_by_id(id, SecretStatus::Provisioning) + .await + .expect("reap again") + ); + + let stale = repo.list_stale_pending(0, 0, 100).await.expect("list"); + assert!(stale.is_empty()); +} + +#[tokio::test] +async fn mark_deprovisioning_gates_on_version_and_hides_from_resolution() { + let repo = setup().await; + let tenant = Uuid::new_v4(); + let owner = Uuid::new_v4(); + let scope = AccessScope::for_tenant(tenant); + let id = seed_active(&repo, tenant, owner, "revoke-me", SharingMode::Tenant).await; + + // Stale expected_version flips nothing. + assert!( + !repo + .mark_deprovisioning(&scope, id, Some(99)) + .await + .expect("gated mark") + ); + + // Matching version flips the row; resolution no longer sees it. + assert!( + repo.mark_deprovisioning(&scope, id, Some(1)) + .await + .expect("mark") + ); + let resolved = repo + .resolve_for_get( + TenantId(tenant), + OwnerId(owner), + &sref("revoke-me"), + &[tenant], + ) + .await + .expect("resolve"); + assert!( + resolved.is_none(), + "deprovisioning row must be invisible to resolution" + ); + + // find_own still returns it (saga resume), version untouched. + let own = repo + .find_own(&scope, TenantId(tenant), OwnerId(owner), &sref("revoke-me")) + .await + .expect("find_own") + .expect("row visible for delete resume"); + assert_eq!(own.id, id); + assert_eq!(own.version, 1, "mark_deprovisioning must not bump version"); + assert_eq!( + own.status, + crate::domain::secret::model::SecretStatus::Deprovisioning + ); + + // A second mark is a no-op (row no longer active). + assert!( + !repo + .mark_deprovisioning(&scope, id, None) + .await + .expect("re-mark") + ); +} + +#[tokio::test] +async fn deprovisioning_row_still_holds_unique_index() { + let repo = setup().await; + let tenant = Uuid::new_v4(); + let owner = Uuid::new_v4(); + let scope = AccessScope::for_tenant(tenant); + let id = seed_active(&repo, tenant, owner, "held", SharingMode::Tenant).await; + assert!( + repo.mark_deprovisioning(&scope, id, None) + .await + .expect("mark") + ); + + // Re-creating the same non-private reference conflicts until cleanup. + let err = repo + .insert_provisioning( + &scope, + &NewSecret { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: sref("held"), + sharing: SharingMode::Tenant, + owner_id: OwnerId(owner), + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: vec![7u8; 32], + fp_key_id: 1, + }, + ) + .await + .expect_err("unique index still held"); + assert!(matches!(err, DomainError::Conflict)); + + // After the reaper removes the row, the reference is free again. + assert!( + repo.reap_by_id(id, SecretStatus::Deprovisioning) + .await + .expect("reap") + ); + seed_provisioning(&repo, tenant, owner, "held", SharingMode::Tenant).await; +} + +#[tokio::test] +async fn duplicate_nonprivate_insert_conflicts() { + let repo = setup().await; + let tenant = Uuid::new_v4(); + let owner = Uuid::new_v4(); + seed_provisioning(&repo, tenant, owner, "dup", SharingMode::Tenant).await; + + // uq_credstore_nonprivate: one (tenant, reference) row for sharing <> 1. + // The DB unique violation is classified back to a domain Conflict. + let err = repo + .insert_provisioning( + &AccessScope::for_tenant(tenant), + &NewSecret { + id: Uuid::new_v4(), + tenant_id: TenantId(tenant), + reference: sref("dup"), + sharing: SharingMode::Tenant, + owner_id: OwnerId(owner), + secret_type_uuid: SecretType::generic().uuid(), + expires_at: None, + value_fp: vec![7u8; 32], + fp_key_id: 1, + }, + ) + .await + .expect_err("duplicate non-private insert violates unique index"); + assert!(matches!(err, DomainError::Conflict)); +} + +// ── read path ─────────────────────────────────────────────────────────────── + +#[tokio::test] +async fn find_own_matches_private_owner_and_tenant_shared() { + let repo = setup().await; + let tenant = Uuid::new_v4(); + let owner = Uuid::new_v4(); + let scope = AccessScope::for_tenant(tenant); + + // Private row owned by `owner`. + seed_active(&repo, tenant, owner, "priv", SharingMode::Private).await; + let own = repo + .find_own(&scope, TenantId(tenant), OwnerId(owner), &sref("priv")) + .await + .expect("find_own") + .expect("private row found by its owner"); + assert_eq!(own.sharing, SharingMode::Private); + + // A different subject does not see the private row. + let other = repo + .find_own( + &scope, + TenantId(tenant), + OwnerId(Uuid::new_v4()), + &sref("priv"), + ) + .await + .expect("find_own"); + assert!(other.is_none()); + + // Tenant-shared row is visible to any subject in the tenant. + seed_active(&repo, tenant, owner, "team", SharingMode::Tenant).await; + let team = repo + .find_own( + &scope, + TenantId(tenant), + OwnerId(Uuid::new_v4()), + &sref("team"), + ) + .await + .expect("find_own") + .expect("tenant-shared row visible"); + assert_eq!(team.sharing, SharingMode::Tenant); +} + +#[tokio::test] +async fn find_for_write_addresses_sharing_class_for_coexistence() { + let repo = setup().await; + let tenant = Uuid::new_v4(); + let owner = Uuid::new_v4(); + let scope = AccessScope::for_tenant(tenant); + + // A tenant/shared and a private secret coexist under the SAME reference — the + // partial unique indexes (`uq_credstore_nonprivate` / `uq_credstore_private`) + // permit both; these two seeds succeeding is itself the coexistence proof. + seed_active(&repo, tenant, owner, "dup", SharingMode::Tenant).await; + seed_active(&repo, tenant, owner, "dup", SharingMode::Private).await; + + // A private write addresses the private row, never the tenant one. + let private = repo + .find_for_write( + &scope, + TenantId(tenant), + OwnerId(owner), + &sref("dup"), + SharingMode::Private, + ) + .await + .expect("find_for_write") + .expect("private row addressed"); + assert_eq!(private.sharing, SharingMode::Private); + + // A non-private write addresses the tenant/shared row, never the private one. + for write_sharing in [SharingMode::Tenant, SharingMode::Shared] { + let nonprivate = repo + .find_for_write( + &scope, + TenantId(tenant), + OwnerId(owner), + &sref("dup"), + write_sharing, + ) + .await + .expect("find_for_write") + .expect("non-private row addressed"); + assert_eq!(nonprivate.sharing, SharingMode::Tenant); + } + + // A private write by a different owner finds nothing (would create its own). + let other = repo + .find_for_write( + &scope, + TenantId(tenant), + OwnerId(Uuid::new_v4()), + &sref("dup"), + SharingMode::Private, + ) + .await + .expect("find_for_write"); + assert!(other.is_none()); +} + +#[tokio::test] +async fn private_and_nonprivate_versions_are_independent() { + let repo = setup().await; + let tenant = Uuid::new_v4(); + let owner = Uuid::new_v4(); + let scope = AccessScope::for_tenant(tenant); + + // Coexisting tenant + private rows under the same reference. + seed_active(&repo, tenant, owner, "dup", SharingMode::Tenant).await; + seed_active(&repo, tenant, owner, "dup", SharingMode::Private).await; + + // Bump only the non-private row twice. + let nonprivate = repo + .find_for_write( + &scope, + TenantId(tenant), + OwnerId(owner), + &sref("dup"), + SharingMode::Tenant, + ) + .await + .expect("find") + .expect("non-private row"); + repo.touch( + &scope, + nonprivate.id, + SharingMode::Tenant, + None, + None, + vec![9u8; 32], + ) + .await + .expect("touch") + .expect("row"); + repo.touch( + &scope, + nonprivate.id, + SharingMode::Tenant, + None, + None, + vec![9u8; 32], + ) + .await + .expect("touch") + .expect("row"); + + // Non-private is now at 3; private is untouched at 1. + let nonprivate = repo + .find_for_write( + &scope, + TenantId(tenant), + OwnerId(owner), + &sref("dup"), + SharingMode::Tenant, + ) + .await + .expect("find") + .expect("non-private row"); + assert_eq!(nonprivate.version, 3); + + let private = repo + .find_for_write( + &scope, + TenantId(tenant), + OwnerId(owner), + &sref("dup"), + SharingMode::Private, + ) + .await + .expect("find") + .expect("private row"); + assert_eq!( + private.version, 1, + "private version is independent of the non-private bumps" + ); +} + +#[tokio::test] +async fn resolve_for_get_prefers_closest_tenant_then_private() { + let repo = setup().await; + let parent = Uuid::new_v4(); + let child = Uuid::new_v4(); + let owner = Uuid::new_v4(); + + // Same reference shared at the parent and held privately at the child. + seed_active(&repo, parent, owner, "cfg", SharingMode::Shared).await; + seed_active(&repo, child, owner, "cfg", SharingMode::Private).await; + + // Walk-up chain: child first, then parent. Closest (child) wins. + let row = repo + .resolve_for_get( + TenantId(child), + OwnerId(owner), + &sref("cfg"), + &[child, parent], + ) + .await + .expect("resolve") + .expect("row resolved"); + assert_eq!(row.tenant_id, TenantId(child)); + assert_eq!(row.sharing, SharingMode::Private); + + // Missing key resolves to None. + let none = repo + .resolve_for_get( + TenantId(child), + OwnerId(owner), + &sref("absent"), + &[child, parent], + ) + .await + .expect("resolve"); + assert!(none.is_none()); +} + +// Real-SQL coverage for the core inheritance contract (previously exercised only +// by the FakeSecretRepo): a parent Tenant-mode row is EXCLUDED from a child's +// walk-up, while a parent Shared-mode row IS inherited. Guards the +// `Condition::any()` branch against both a tenant-leak and a false-miss. +#[tokio::test] +async fn resolve_for_get_excludes_parent_tenant_mode_but_inherits_shared() { + let repo = setup().await; + let parent = Uuid::new_v4(); + let child = Uuid::new_v4(); + let owner = Uuid::new_v4(); + + // (Tenant and Shared are both non-private and cannot coexist under one ref, + // so they are seeded under distinct references.) + seed_active(&repo, parent, owner, "tenant-only", SharingMode::Tenant).await; + seed_active(&repo, parent, owner, "shared-cfg", SharingMode::Shared).await; + + // Tenant-mode parent secret must NOT leak into the child's walk-up. + let tenant_only = repo + .resolve_for_get( + TenantId(child), + OwnerId(owner), + &sref("tenant-only"), + &[child, parent], + ) + .await + .expect("resolve"); + assert!( + tenant_only.is_none(), + "parent Tenant-mode secret must not be inherited by a child" + ); + + // Shared parent secret IS inherited; the resolved row belongs to the parent. + let shared = repo + .resolve_for_get( + TenantId(child), + OwnerId(owner), + &sref("shared-cfg"), + &[child, parent], + ) + .await + .expect("resolve") + .expect("shared row must be inherited"); + assert_eq!(shared.sharing, SharingMode::Shared); + // `is_inherited` is derived at the service layer as `row.tenant_id != req`; + // here the resolved tenant is the parent, not the requesting child. + assert_eq!(shared.tenant_id, TenantId(parent)); + assert_ne!(shared.tenant_id, TenantId(child)); +} + +/// Regression: a Private secret is owner-scoped *and* pinned to its own tenant. +/// Even if the same `subject` (owner) id appears in an ancestor tenant, that +/// ancestor's private secret must never resolve for a descendant walk-up — the +/// query pins `tenant_id == req` rather than trusting the authn invariant that +/// a subject id belongs to a single tenant. +#[tokio::test] +async fn resolve_for_get_never_inherits_ancestor_private_even_for_same_owner() { + let repo = setup().await; + let parent = Uuid::new_v4(); + let child = Uuid::new_v4(); + // Same owner id present in the ancestor tenant (the invariant we no longer + // depend on: a reused/cross-tenant subject id). + let owner = Uuid::new_v4(); + + seed_active(&repo, parent, owner, "priv-key", SharingMode::Private).await; + + let resolved = repo + .resolve_for_get( + TenantId(child), + OwnerId(owner), + &sref("priv-key"), + &[child, parent], + ) + .await + .expect("resolve"); + assert!( + resolved.is_none(), + "an ancestor's private secret must not resolve for a descendant, even with a matching owner id" + ); +} + +#[tokio::test] +async fn inventory_aggregates_by_sharing_status_and_tenants() { + let repo = setup().await; + let t1 = Uuid::new_v4(); + let t2 = Uuid::new_v4(); + let owner = Uuid::new_v4(); + + seed_active(&repo, t1, owner, "a", SharingMode::Private).await; + seed_active(&repo, t1, owner, "b", SharingMode::Tenant).await; + seed_active(&repo, t2, owner, "c", SharingMode::Shared).await; + seed_provisioning(&repo, t2, owner, "d", SharingMode::Tenant).await; + + let counts = repo.inventory().await.expect("inventory"); + assert_eq!(counts.private, 1); + assert_eq!(counts.tenant, 1); + assert_eq!(counts.shared, 1); + assert_eq!(counts.provisioning, 1); + assert_eq!(counts.tenants, 2, "distinct tenants among active rows"); +} + +#[tokio::test] +async fn new_secret_defaults_to_version_one() { + let repo = setup().await; + let tenant = Uuid::new_v4(); + let owner = Uuid::new_v4(); + seed_active(&repo, tenant, owner, "v1", SharingMode::Tenant).await; + + let row = repo + .resolve_for_get(TenantId(tenant), OwnerId(owner), &sref("v1"), &[tenant]) + .await + .expect("resolve") + .expect("active row"); + assert_eq!( + row.version, 1, + "a freshly inserted secret starts at version 1" + ); +} + +// ── scope_includes_tenant ──────────────────────────────────────────────────── + +#[tokio::test] +async fn scope_includes_tenant_unconstrained_and_deny() { + let repo = setup().await; + let t = Uuid::new_v4(); + assert!( + repo.scope_includes_tenant(&AccessScope::allow_all(), t) + .await + .expect("allow_all") + ); + assert!( + !repo + .scope_includes_tenant(&AccessScope::deny_all(), t) + .await + .expect("deny_all") + ); +} + +#[tokio::test] +async fn scope_includes_tenant_direct_uuid_match() { + let repo = setup().await; + let t = Uuid::new_v4(); + assert!( + repo.scope_includes_tenant(&AccessScope::for_tenant(t), t) + .await + .expect("direct match") + ); + assert!( + !repo + .scope_includes_tenant(&AccessScope::for_tenant(t), Uuid::new_v4()) + .await + .expect("direct miss") + ); +} + +#[tokio::test] +async fn scope_includes_tenant_subtree_via_closure() { + let repo = setup().await; + let root = Uuid::new_v4(); + let child = Uuid::new_v4(); + let stranger = Uuid::new_v4(); + seed_closure(&repo, root, child, 0).await; + + let scope = subtree_scope(root); + assert!( + repo.scope_includes_tenant(&scope, child) + .await + .expect("descendant in subtree") + ); + assert!( + !repo + .scope_includes_tenant(&scope, stranger) + .await + .expect("non-descendant excluded") + ); +} + +#[tokio::test] +async fn scope_includes_tenant_sibling_owner_filter_fails_closed() { + // A constraint narrower than tenant granularity + // (`OWNER_TENANT_ID = T AND owner_id = X`) grants access only to X's + // secrets, not the whole tenant. The gate must NOT admit the tenant on + // the lone `OWNER_TENANT_ID` match — it fails closed on the sibling. + let repo = setup().await; + let t = Uuid::new_v4(); + let owner = Uuid::new_v4(); + let scope = AccessScope::from_constraints(vec![ScopeConstraint::new(vec![ + ScopeFilter::eq(pep_properties::OWNER_TENANT_ID, t), + ScopeFilter::eq(pep_properties::OWNER_ID, owner), + ])]); + assert!( + !repo + .scope_includes_tenant(&scope, t) + .await + .expect("sibling owner filter must fail closed"), + "a sub-tenant scope must not be widened to the whole tenant" + ); +} + +#[tokio::test] +async fn scope_includes_tenant_unknown_property_fails_closed() { + // A constraint whose only filter is on a property this gate does not + // treat as tenant-level (`resource_id`) must not admit the tenant. + let repo = setup().await; + let t = Uuid::new_v4(); + let scope = AccessScope::from_constraints(vec![ScopeConstraint::new(vec![ScopeFilter::eq( + pep_properties::RESOURCE_ID, + Uuid::new_v4(), + )])]); + assert!( + !repo + .scope_includes_tenant(&scope, t) + .await + .expect("non-tenant property must fail closed") + ); +} + +#[tokio::test] +async fn scope_includes_tenant_or_of_constraints_admits_on_broad_alternative() { + // Constraints are OR-ed: a narrow (non-admitting) alternative must not + // veto a sibling constraint that does grant whole-tenant access. + let repo = setup().await; + let t = Uuid::new_v4(); + let scope = AccessScope::from_constraints(vec![ + ScopeConstraint::new(vec![ + ScopeFilter::eq(pep_properties::OWNER_TENANT_ID, t), + ScopeFilter::eq(pep_properties::OWNER_ID, Uuid::new_v4()), + ]), + ScopeConstraint::new(vec![ScopeFilter::eq(pep_properties::OWNER_TENANT_ID, t)]), + ]); + assert!( + repo.scope_includes_tenant(&scope, t) + .await + .expect("broad OR alternative admits"), + "a whole-tenant alternative must still grant despite a narrower sibling" + ); +} + +#[tokio::test] +async fn expired_rows_hidden_from_resolution_and_flipped_by_expiry_sweep() { + use time::{Duration as TimeDuration, OffsetDateTime}; + let repo = setup().await; + let tenant = Uuid::new_v4(); + let owner = Uuid::new_v4(); + let scope = AccessScope::for_tenant(tenant); + + // Seed an active expirable row whose expiry is already in the past + // (repo layer does not enforce traits; that's the domain's job). + let id = Uuid::new_v4(); + repo.insert_provisioning( + &scope, + &NewSecret { + id, + tenant_id: TenantId(tenant), + reference: sref("expired"), + sharing: SharingMode::Tenant, + owner_id: OwnerId(owner), + secret_type_uuid: SecretType::from_name("bearer-token").expect("known").uuid(), + expires_at: Some(OffsetDateTime::now_utc() - TimeDuration::seconds(5)), + value_fp: vec![7u8; 32], + fp_key_id: 1, + }, + ) + .await + .expect("insert"); + repo.mark_active(&scope, id).await.expect("mark active"); + + // Expired -> not resolvable... + let resolved = repo + .resolve_for_get( + TenantId(tenant), + OwnerId(owner), + &sref("expired"), + &[tenant], + ) + .await + .expect("resolve"); + assert!(resolved.is_none(), "expired row must not resolve"); + + // ...but still addressable for writes/deletes. + assert!( + repo.find_own(&scope, TenantId(tenant), OwnerId(owner), &sref("expired")) + .await + .expect("find_own") + .is_some() + ); + + // The expiry sweep flips it into the deprovisioning saga. + let flipped = repo + .mark_expired_deprovisioning() + .await + .expect("expiry sweep"); + assert_eq!(flipped, 1); + let stale = repo.list_stale_pending(0, 0, 100).await.expect("stale"); + assert_eq!(stale.len(), 1); + assert_eq!( + stale[0].status, + crate::domain::secret::model::SecretStatus::Deprovisioning + ); + + // A live (unexpired) row is untouched by the sweep. + let live = Uuid::new_v4(); + repo.insert_provisioning( + &scope, + &NewSecret { + id: live, + tenant_id: TenantId(tenant), + reference: sref("living"), + sharing: SharingMode::Tenant, + owner_id: OwnerId(owner), + secret_type_uuid: SecretType::from_name("bearer-token").expect("known").uuid(), + expires_at: Some(OffsetDateTime::now_utc() + TimeDuration::hours(1)), + value_fp: vec![7u8; 32], + fp_key_id: 1, + }, + ) + .await + .expect("insert live"); + repo.mark_active(&scope, live).await.expect("mark active"); + assert_eq!( + repo.mark_expired_deprovisioning().await.expect("sweep"), + 0, + "future expiry must not be swept" + ); +} + +#[tokio::test] +async fn secret_type_round_trips_through_storage() { + let repo = setup().await; + let tenant = Uuid::new_v4(); + let owner = Uuid::new_v4(); + let scope = AccessScope::for_tenant(tenant); + let id = Uuid::new_v4(); + repo.insert_provisioning( + &scope, + &NewSecret { + id, + tenant_id: TenantId(tenant), + reference: sref("typed"), + sharing: SharingMode::Private, + owner_id: OwnerId(owner), + secret_type_uuid: SecretType::from_name("personal-token") + .expect("known") + .uuid(), + expires_at: None, + value_fp: vec![7u8; 32], + fp_key_id: 1, + }, + ) + .await + .expect("insert"); + repo.mark_active(&scope, id).await.expect("mark active"); + + let row = repo + .resolve_for_get(TenantId(tenant), OwnerId(owner), &sref("typed"), &[tenant]) + .await + .expect("resolve") + .expect("row"); + assert_eq!( + row.secret_type_uuid, + SecretType::from_name("personal-token") + .expect("known") + .uuid() + ); +} + +// ── value-fingerprint fence columns ────────────────────────────────────────── + +/// Simulate an out-of-band seeded row: direct INSERT with NULL fence columns +/// (the seeder contract from docs/features/001-value-fingerprint-fence.md). +async fn seed_unfenced(repo: &SecretRepoImpl, tenant: Uuid, owner: Uuid, key: &str) -> Uuid { + use sea_orm::ActiveValue; + let id = Uuid::new_v4(); + let conn = repo.db.conn().expect("conn"); + entity::secrets::Entity::insert(entity::secrets::ActiveModel { + id: ActiveValue::Set(id), + tenant_id: ActiveValue::Set(tenant), + reference: ActiveValue::Set(key.to_owned()), + sharing: ActiveValue::Set(2), // Tenant + owner_id: ActiveValue::Set(owner), + status: ActiveValue::Set(2), // Active + created_at: ActiveValue::NotSet, + updated_at: ActiveValue::NotSet, + version: ActiveValue::NotSet, + secret_type_uuid: ActiveValue::Set(SecretType::generic().uuid()), + expires_at: ActiveValue::Set(None), + value_fp: ActiveValue::Set(None), + fp_key_id: ActiveValue::Set(None), + }) + .secure() + .scope_unchecked(&AccessScope::allow_all()) + .expect("scope") + .exec(&conn) + .await + .expect("seed unfenced row"); + id +} + +#[tokio::test] +async fn backfill_fp_stamps_null_rows_once_and_preserves_version() { + let repo = setup().await; + let tenant = Uuid::new_v4(); + let owner = Uuid::new_v4(); + let id = seed_unfenced(&repo, tenant, owner, "seeded").await; + + // Listed as unfenced before the stamp. + let unfenced = repo.list_unfenced(16).await.expect("list"); + assert!(unfenced.iter().any(|r| r.id == id)); + + // First backfill stamps (CAS on NULL matches). + let stamped = repo + .backfill_fp(id, vec![1u8; 32], 1) + .await + .expect("backfill"); + assert!(stamped, "NULL fp row must be stamped"); + + let row = repo + .resolve_for_get(TenantId(tenant), OwnerId(owner), &sref("seeded"), &[tenant]) + .await + .expect("resolve") + .expect("row"); + assert_eq!(row.value_fp.as_deref(), Some([1u8; 32].as_slice())); + assert_eq!(row.fp_key_id, Some(1)); + assert_eq!( + row.version, 1, + "backfill must not bump the version (the caller's ETag stays stable)" + ); + + // Second backfill is a CAS no-op: a concurrent stamp must never clobber. + let again = repo + .backfill_fp(id, vec![2u8; 32], 1) + .await + .expect("backfill"); + assert!(!again, "an already-stamped row must not be re-stamped"); + let row = repo + .resolve_for_get(TenantId(tenant), OwnerId(owner), &sref("seeded"), &[tenant]) + .await + .expect("resolve") + .expect("row"); + assert_eq!( + row.value_fp.as_deref(), + Some([1u8; 32].as_slice()), + "the first stamp wins" + ); + + // No longer listed as unfenced. + let unfenced = repo.list_unfenced(16).await.expect("list"); + assert!(unfenced.iter().all(|r| r.id != id)); +} + +#[tokio::test] +async fn list_unfenced_skips_stamped_and_non_active_rows() { + let repo = setup().await; + let tenant = Uuid::new_v4(); + let owner = Uuid::new_v4(); + // Stamped active row (API-created) — excluded. + seed_active(&repo, tenant, owner, "stamped", SharingMode::Tenant).await; + // Unfenced but provisioning-status rows do not exist in practice (API + // creates always stamp); the sweep must still only pick ACTIVE rows. + let unfenced_active = seed_unfenced(&repo, tenant, owner, "plain").await; + + let listed = repo.list_unfenced(16).await.expect("list"); + assert_eq!(listed.len(), 1); + assert_eq!(listed[0].id, unfenced_active); +} + +#[tokio::test] +async fn touch_restamps_the_fence_fingerprint() { + let repo = setup().await; + let tenant = Uuid::new_v4(); + let owner = Uuid::new_v4(); + // seed_active stamps vec![7u8; 32] via NewSecret. + let id = seed_active(&repo, tenant, owner, "restamp", SharingMode::Tenant).await; + + let row = repo + .touch( + &AccessScope::for_tenant(tenant), + id, + SharingMode::Tenant, + None, + None, + vec![8u8; 32], + ) + .await + .expect("touch") + .expect("row updated"); + assert_eq!( + row.value_fp.as_deref(), + Some([8u8; 32].as_slice()), + "touch must persist the new fingerprint atomically with the metadata" + ); + assert_eq!(row.fp_key_id, Some(1)); + assert_eq!(row.version, 2); +} diff --git a/gears/credstore/credstore/src/infra/storage/repo_impl/writes.rs b/gears/credstore/credstore/src/infra/storage/repo_impl/writes.rs new file mode 100644 index 000000000..4bc2a56aa --- /dev/null +++ b/gears/credstore/credstore/src/infra/storage/repo_impl/writes.rs @@ -0,0 +1,384 @@ +//! Write-path repo methods: `insert_provisioning`, `mark_active`, +//! `mark_deprovisioning`, `touch`, `backfill_fp`, `delete_by_id`, +//! `list_stale_pending`, `list_unfenced`, `reap_by_id`. + +use credstore_sdk::SharingMode; +use sea_orm::sea_query::Expr; +use sea_orm::{ColumnTrait, Condition, EntityTrait, Order, QueryFilter, QueryOrder, QuerySelect}; +use time::OffsetDateTime; +use toolkit_db::secure::{SecureDeleteExt, SecureEntityExt, SecureInsertExt, SecureUpdateExt}; +use toolkit_security::AccessScope; +use uuid::Uuid; + +use crate::domain::error::DomainError; +use crate::domain::secret::model::{NewSecret, SecretRow, SecretStatus}; +use crate::infra::storage::entity; +use crate::infra::storage::repo_impl::helpers::map_scope_err; +use crate::infra::storage::repo_impl::{SecretRepoImpl, entity_to_model, sharing_to_i16}; + +pub(super) async fn insert_provisioning( + repo: &SecretRepoImpl, + scope: &AccessScope, + new: &NewSecret, +) -> Result<(), DomainError> { + use sea_orm::ActiveValue; + let conn = repo.db.conn()?; + let now = OffsetDateTime::now_utc(); + let am = entity::secrets::ActiveModel { + id: ActiveValue::Set(new.id), + tenant_id: ActiveValue::Set(new.tenant_id.0), + reference: ActiveValue::Set(new.reference.as_ref().to_owned()), + sharing: ActiveValue::Set(sharing_to_i16(new.sharing)), + owner_id: ActiveValue::Set(new.owner_id.0), + status: ActiveValue::Set(SecretStatus::Provisioning.as_smallint()), + created_at: ActiveValue::Set(now), + updated_at: ActiveValue::Set(now), + version: ActiveValue::NotSet, + secret_type_uuid: ActiveValue::Set(new.secret_type_uuid), + expires_at: ActiveValue::Set(new.expires_at), + value_fp: ActiveValue::Set(Some(new.value_fp.clone())), + fp_key_id: ActiveValue::Set(Some(new.fp_key_id)), + }; + // scope_unchecked: INSERT cannot subtree-clamp on a row that doesn't exist yet. + entity::secrets::Entity::insert(am) + .secure() + .scope_unchecked(scope) + .map_err(map_scope_err)? + .exec(&conn) + .await + .map_err(map_scope_err)?; + Ok(()) +} + +pub(super) async fn mark_active( + repo: &SecretRepoImpl, + scope: &AccessScope, + id: Uuid, +) -> Result<(), DomainError> { + let conn = repo.db.conn()?; + let now = OffsetDateTime::now_utc(); + let rows_affected = entity::secrets::Entity::update_many() + .col_expr( + entity::secrets::Column::Status, + Expr::value(SecretStatus::Active.as_smallint()), + ) + .col_expr(entity::secrets::Column::UpdatedAt, Expr::value(now)) + .filter( + Condition::all() + .add(entity::secrets::Column::Id.eq(id)) + .add(entity::secrets::Column::Status.eq(SecretStatus::Provisioning.as_smallint())), + ) + .secure() + .scope_with(scope) + .exec(&conn) + .await + .map_err(map_scope_err)? + .rows_affected; + if rows_affected == 0 { + return Err(DomainError::Conflict); + } + Ok(()) +} + +pub(super) async fn mark_deprovisioning( + repo: &SecretRepoImpl, + scope: &AccessScope, + id: Uuid, + expected_version: Option, +) -> Result { + let conn = repo.db.conn()?; + let now = OffsetDateTime::now_utc(); + // Stamp updated_at: the deprovisioning-timeout clock the reaper keys off. + // The version is deliberately left alone so an If-Match retry of the same + // delete still matches the version the client saw. + let mut filter = Condition::all() + .add(entity::secrets::Column::Id.eq(id)) + .add(entity::secrets::Column::Status.eq(SecretStatus::Active.as_smallint())); + if let Some(v) = expected_version { + filter = filter.add(entity::secrets::Column::Version.eq(v)); + } + let rows_affected = entity::secrets::Entity::update_many() + .col_expr( + entity::secrets::Column::Status, + Expr::value(SecretStatus::Deprovisioning.as_smallint()), + ) + .col_expr(entity::secrets::Column::UpdatedAt, Expr::value(now)) + .filter(filter) + .secure() + .scope_with(scope) + .exec(&conn) + .await + .map_err(map_scope_err)? + .rows_affected; + Ok(rows_affected > 0) +} + +pub(super) async fn touch( + repo: &SecretRepoImpl, + scope: &AccessScope, + id: Uuid, + sharing: SharingMode, + expected_version: Option, + expires_at: Option, + value_fp: Vec, +) -> Result, DomainError> { + let conn = repo.db.conn()?; + let now = OffsetDateTime::now_utc(); + // Atomic, id-keyed: version = version + 1, set sharing, stamp updated_at, + // and re-stamp the value fingerprint — the fp travels in the SAME UPDATE + // as the sharing label, so a fingerprint match on read transitively + // proves value and metadata came from one writer (the fence invariant). + // Keyed by id (the row found by find_for_write) so it needs no sharing-class + // filter and works for both private (sharing unchanged) and non-private + // (tenant<->shared) rows. When expected_version is set, the bump is gated on + // version = expected so a stale optimistic-lock write commits 0 rows. + let mut filter = Condition::all() + .add(entity::secrets::Column::Id.eq(id)) + .add(entity::secrets::Column::Status.eq(SecretStatus::Active.as_smallint())); + if let Some(v) = expected_version { + filter = filter.add(entity::secrets::Column::Version.eq(v)); + } + let rows_affected = entity::secrets::Entity::update_many() + .col_expr( + entity::secrets::Column::Sharing, + Expr::value(sharing_to_i16(sharing)), + ) + .col_expr(entity::secrets::Column::UpdatedAt, Expr::value(now)) + .col_expr( + entity::secrets::Column::Version, + Expr::col(entity::secrets::Column::Version).add(1_i64), + ) + // Whole-value replace: the new expiry (or its absence) wins. + .col_expr(entity::secrets::Column::ExpiresAt, Expr::value(expires_at)) + .col_expr( + entity::secrets::Column::ValueFp, + Expr::value(Some(value_fp)), + ) + .col_expr( + entity::secrets::Column::FpKeyId, + Expr::value(Some(crate::domain::secret::fence::CURRENT_FENCE_KEY_ID)), + ) + .filter(filter) + .secure() + .scope_with(scope) + .exec(&conn) + .await + .map_err(map_scope_err)? + .rows_affected; + if rows_affected == 0 { + return Ok(None); + } + // Re-read the updated row by id. + let row = entity::secrets::Entity::find() + .secure() + .scope_with(scope) + .filter(Condition::all().add(entity::secrets::Column::Id.eq(id))) + .one(&conn) + .await + .map_err(map_scope_err)?; + row.map(entity_to_model).transpose() +} + +/// Stamp the fence fingerprint onto a row that has none (out-of-band seeded): +/// CAS on `value_fp IS NULL`, so a concurrent PUT that already stamped wins +/// (0 rows → `false`, a no-op for the caller). Deliberately does NOT bump +/// `version` or `updated_at` — nothing client-visible changed, the caller's +/// `ETag` must stay stable. Unscoped (system-side heal, like the reaper). +pub(super) async fn backfill_fp( + repo: &SecretRepoImpl, + id: Uuid, + value_fp: Vec, + fp_key_id: i16, +) -> Result { + let conn = repo.db.conn()?; + let rows_affected = entity::secrets::Entity::update_many() + .col_expr( + entity::secrets::Column::ValueFp, + Expr::value(Some(value_fp)), + ) + .col_expr( + entity::secrets::Column::FpKeyId, + Expr::value(Some(fp_key_id)), + ) + .filter( + Condition::all() + .add(entity::secrets::Column::Id.eq(id)) + .add(entity::secrets::Column::ValueFp.is_null()), + ) + .secure() + .scope_with(&AccessScope::allow_all()) + .exec(&conn) + .await + .map_err(map_scope_err)? + .rows_affected; + Ok(rows_affected > 0) +} + +/// Active rows still missing a fence fingerprint (out-of-band seeded and not +/// yet read), bounded batch for the reaper's backfill sweep. Unscoped. +pub(super) async fn list_unfenced( + repo: &SecretRepoImpl, + limit: u64, +) -> Result, DomainError> { + let conn = repo.db.conn()?; + let rows = entity::secrets::Entity::find() + .filter( + Condition::all() + .add(entity::secrets::Column::Status.eq(SecretStatus::Active.as_smallint())) + .add(entity::secrets::Column::ValueFp.is_null()), + ) + // Oldest-first for a fair, deterministic sweep across ticks (same + // reasoning as list_stale_pending). + .order_by(entity::secrets::Column::UpdatedAt, Order::Asc) + .limit(limit) + .secure() + .scope_with(&AccessScope::allow_all()) + .all(&conn) + .await + .map_err(map_scope_err)?; + rows.into_iter().map(entity_to_model).collect() +} + +pub(super) async fn delete_by_id( + repo: &SecretRepoImpl, + scope: &AccessScope, + id: Uuid, + expected_version: Option, +) -> Result<(), DomainError> { + let conn = repo.db.conn()?; + let mut filter = Condition::all().add(entity::secrets::Column::Id.eq(id)); + if let Some(v) = expected_version { + filter = filter.add(entity::secrets::Column::Version.eq(v)); + } + let rows_affected = entity::secrets::Entity::delete_many() + .filter(filter) + .secure() + .scope_with(scope) + .exec(&conn) + .await + .map_err(map_scope_err)? + .rows_affected; + if rows_affected == 0 { + return Err(DomainError::NotFound); + } + Ok(()) +} + +/// Cutoff instant `older_than_secs` ago, saturating to "never reap" for absurd +/// configs. A u64 beyond `i64::MAX` would wrap to a negative duration and match +/// rows from the future; equally, subtracting a duration large enough to leave +/// the representable `OffsetDateTime` range panics. We therefore clamp the +/// offset so the subtraction always lands on a valid instant: any cutoff at or +/// before the min representable date means nothing is ever old enough to reap, +/// which is the safe bound. +fn cutoff(older_than_secs: u64) -> OffsetDateTime { + let now = OffsetDateTime::now_utc(); + let secs = i64::try_from(older_than_secs).unwrap_or(i64::MAX); + now.checked_sub(time::Duration::seconds(secs)) + .unwrap_or(OffsetDateTime::UNIX_EPOCH) +} + +pub(super) async fn list_stale_pending( + repo: &SecretRepoImpl, + provisioning_older_than_secs: u64, + deprovisioning_older_than_secs: u64, + limit: u64, +) -> Result, DomainError> { + let conn = repo.db.conn()?; + // Both arms key off updated_at (idx_credstore_pending): provisioning rows + // are never updated after insert, so updated_at equals created_at there. + let rows = entity::secrets::Entity::find() + .filter( + Condition::any() + .add( + Condition::all() + .add( + entity::secrets::Column::Status + .eq(SecretStatus::Provisioning.as_smallint()), + ) + .add( + entity::secrets::Column::UpdatedAt + .lt(cutoff(provisioning_older_than_secs)), + ), + ) + .add( + Condition::all() + .add( + entity::secrets::Column::Status + .eq(SecretStatus::Deprovisioning.as_smallint()), + ) + .add( + entity::secrets::Column::UpdatedAt + .lt(cutoff(deprovisioning_older_than_secs)), + ), + ), + ) + // Oldest-first, so a batch of persistently-failing deprovisioning rows + // (kept each tick until their backend delete finally succeeds) cannot + // starve newer stale rows: without a deterministic order the DB may + // return the same physical-order rows every tick, and once ≥ `limit` + // of them wedge, no other stale row is ever reaped. + .order_by(entity::secrets::Column::UpdatedAt, Order::Asc) + .limit(limit) + .secure() + .scope_with(&AccessScope::allow_all()) + .all(&conn) + .await + .map_err(map_scope_err)?; + rows.into_iter().map(entity_to_model).collect() +} + +pub(super) async fn mark_expired_deprovisioning(repo: &SecretRepoImpl) -> Result { + let conn = repo.db.conn()?; + let now = OffsetDateTime::now_utc(); + // Expired active rows enter the ordinary deprovisioning saga: invisible + // to resolution immediately, name held until backend cleanup completes + // via the pending sweep. Uses idx_credstore_expiry. + let rows_affected = entity::secrets::Entity::update_many() + .col_expr( + entity::secrets::Column::Status, + Expr::value(SecretStatus::Deprovisioning.as_smallint()), + ) + .col_expr(entity::secrets::Column::UpdatedAt, Expr::value(now)) + .filter( + Condition::all() + .add(entity::secrets::Column::Status.eq(SecretStatus::Active.as_smallint())) + .add(entity::secrets::Column::ExpiresAt.is_not_null()) + .add(entity::secrets::Column::ExpiresAt.lte(now)), + ) + .secure() + .scope_with(&AccessScope::allow_all()) + .exec(&conn) + .await + .map_err(map_scope_err)? + .rows_affected; + Ok(rows_affected) +} + +pub(super) async fn reap_by_id( + repo: &SecretRepoImpl, + id: Uuid, + expected: SecretStatus, +) -> Result { + let conn = repo.db.conn()?; + // Status-gated: the reaper observed this row in `expected` status, but a + // concurrent saga may have moved it on since (most importantly a slow + // create's `mark_active` flipping `Provisioning → Active`). Guarding the + // delete on the observed status makes it mutually exclusive with that + // transition — 0 rows means the row is no longer ours to reap, so the live + // secret (and its backend value) is left intact. `false` (already gone or + // moved on) is benign; the caller reports it as "not reaped". + let rows_affected = entity::secrets::Entity::delete_many() + .filter( + Condition::all() + .add(entity::secrets::Column::Id.eq(id)) + .add(entity::secrets::Column::Status.eq(expected.as_smallint())), + ) + .secure() + .scope_with(&AccessScope::allow_all()) + .exec(&conn) + .await + .map_err(map_scope_err)? + .rows_affected; + Ok(rows_affected > 0) +} diff --git a/gears/credstore/credstore/src/infra/tenant_resolver.rs b/gears/credstore/credstore/src/infra/tenant_resolver.rs new file mode 100644 index 000000000..943099a0e --- /dev/null +++ b/gears/credstore/credstore/src/infra/tenant_resolver.rs @@ -0,0 +1,141 @@ +//! Infra adapter: `TenantDirectory` backed by `TenantResolverClient` with a TTL cache. + +use std::collections::HashMap; +use std::sync::{Arc, Mutex}; +use std::time::{Duration, Instant}; + +use async_trait::async_trait; +use tenant_resolver_sdk::{BarrierMode, GetAncestorsOptions, TenantResolverClient}; +use toolkit_security::SecurityContext; +use uuid::Uuid; + +use crate::domain::error::DomainError; +use crate::domain::ports::metrics::{CredStoreMetricsPort, Dep, DepOp, Outcome}; +use crate::domain::resolver::TenantDirectory; +use credstore_sdk::TenantId; + +/// Maximum cache entries before a full eviction. +const CACHE_CAP: usize = 4096; + +/// Entry: `(chain, inserted_at)`. +type CacheEntry = (Vec, Instant); + +/// Infra implementation of [`TenantDirectory`] backed by [`TenantResolverClient`]. +pub struct TenantResolverDir { + client: Arc, + metrics: Arc, + /// Ancestor chains keyed by tenant id. Safe to share across security + /// contexts: the chain is a pure function of the tenant topology and the + /// fixed barrier-respecting mode — it carries no caller-specific data — and + /// all authorization is enforced downstream (`scope_includes_tenant` + + /// `resolve_for_get`). So a chain populated under one caller's `ctx` is + /// valid for any caller resolving the same tenant. + cache: Mutex>, + ttl: Duration, +} + +impl TenantResolverDir { + /// Construct a new adapter. + #[must_use] + pub fn new( + client: Arc, + metrics: Arc, + ttl_secs: u64, + ) -> Self { + Self { + client, + metrics, + cache: Mutex::new(HashMap::new()), + ttl: Duration::from_secs(ttl_secs), + } + } +} + +#[async_trait] +impl TenantDirectory for TenantResolverDir { + async fn ancestor_chain( + &self, + ctx: &SecurityContext, + req: TenantId, + ) -> Result, DomainError> { + // Cache lookup — no metric on hit. + { + let guard = self + .cache + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if let Some((chain, inserted)) = guard.get(&req.0) + && inserted.elapsed() < self.ttl + { + return Ok(chain.clone()); + } + } + + let t0 = Instant::now(); + // Respect self-managed tenant boundaries: a secret must not be inherited + // across an isolation barrier, so the walk-up stops at the boundary. + let opts = GetAncestorsOptions { + barrier_mode: BarrierMode::Respect, + }; + let resp = match self.client.get_ancestors(ctx, req, &opts).await { + Ok(r) => { + self.metrics.dependency( + Dep::TenantResolver, + DepOp::GetAncestors, + Outcome::Success, + t0.elapsed().as_secs_f64(), + ); + r + } + Err(e) => { + self.metrics.dependency( + Dep::TenantResolver, + DepOp::GetAncestors, + Outcome::Error, + t0.elapsed().as_secs_f64(), + ); + // Wire-visible detail stays curated (`with_detail` contract); + // the raw dependency error goes to the log + cause chain only. + tracing::warn!(err = %e, "tenant_resolver get_ancestors failed"); + return Err(DomainError::ServiceUnavailable { + detail: "tenant resolver unavailable".to_owned(), + retry_after: None, + cause: Some(Box::new(e)), + }); + } + }; + + let mut chain = Vec::with_capacity(1 + resp.ancestors.len()); + chain.push(req.0); + chain.extend(resp.ancestors.iter().map(|a| a.id.0)); + + { + let mut guard = self + .cache + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if guard.len() >= CACHE_CAP { + // Drop expired entries first; only if still at cap evict the + // single oldest. A blunt clear() would periodically blow away + // every still-fresh chain and cause a thundering herd of misses. + let ttl = self.ttl; + guard.retain(|_, (_, inserted)| inserted.elapsed() < ttl); + if guard.len() >= CACHE_CAP + && let Some(oldest) = guard + .iter() + .min_by_key(|(_, (_, inserted))| *inserted) + .map(|(k, _)| *k) + { + guard.remove(&oldest); + } + } + guard.insert(req.0, (chain.clone(), Instant::now())); + } + + Ok(chain) + } +} + +#[cfg(test)] +#[path = "tenant_resolver_tests.rs"] +mod tests; diff --git a/gears/credstore/credstore/src/infra/tenant_resolver_tests.rs b/gears/credstore/credstore/src/infra/tenant_resolver_tests.rs new file mode 100644 index 000000000..e2b2ce0f8 --- /dev/null +++ b/gears/credstore/credstore/src/infra/tenant_resolver_tests.rs @@ -0,0 +1,202 @@ +//! Unit tests for [`TenantResolverDir`]. + +use std::sync::Arc; +use std::sync::Mutex; +use std::sync::atomic::{AtomicU32, Ordering}; + +use async_trait::async_trait; +use credstore_sdk::TenantId as DomainTenantId; +use tenant_resolver_sdk::TenantResolverClient; +use tenant_resolver_sdk::error::TenantResolverError; +use tenant_resolver_sdk::models::{ + BarrierMode, GetAncestorsOptions, GetAncestorsResponse, GetDescendantsOptions, + GetDescendantsResponse, GetTenantsOptions, IsAncestorOptions, TenantId, TenantRef, + TenantStatus, +}; +use toolkit_security::SecurityContext; +use uuid::Uuid; + +use crate::domain::ports::metrics::NoopMetrics; +use crate::domain::resolver::TenantDirectory; +use crate::infra::tenant_resolver::TenantResolverDir; + +fn make_ctx() -> SecurityContext { + SecurityContext::builder() + .subject_id(Uuid::new_v4()) + .subject_tenant_id(Uuid::new_v4()) + .build() + .expect("test ctx") +} + +fn tenant_ref(id: Uuid) -> TenantRef { + TenantRef { + id: TenantId(id), + status: TenantStatus::Active, + tenant_type: None, + parent_id: None, + self_managed: false, + } +} + +// ── Fake client ─────────────────────────────────────────────────────────────── + +struct FakeTenantResolverClient { + child: Uuid, + parent: Uuid, + root: Uuid, + calls: AtomicU32, + last_barrier: Mutex>, +} + +impl FakeTenantResolverClient { + fn new(child: Uuid, parent: Uuid, root: Uuid) -> Self { + Self { + child, + parent, + root, + calls: AtomicU32::new(0), + last_barrier: Mutex::new(None), + } + } + + fn call_count(&self) -> u32 { + self.calls.load(Ordering::SeqCst) + } + + fn last_barrier_mode(&self) -> Option { + *self.last_barrier.lock().expect("lock") + } +} + +#[async_trait] +impl TenantResolverClient for FakeTenantResolverClient { + async fn get_tenant( + &self, + _ctx: &SecurityContext, + _id: TenantId, + ) -> Result { + unimplemented!() + } + + async fn get_root_tenant( + &self, + _ctx: &SecurityContext, + ) -> Result { + unimplemented!() + } + + async fn get_tenants( + &self, + _ctx: &SecurityContext, + _ids: &[TenantId], + _options: &GetTenantsOptions, + ) -> Result, TenantResolverError> { + unimplemented!() + } + + async fn get_ancestors( + &self, + _ctx: &SecurityContext, + id: TenantId, + options: &GetAncestorsOptions, + ) -> Result { + *self.last_barrier.lock().expect("lock") = Some(options.barrier_mode); + self.calls.fetch_add(1, Ordering::SeqCst); + if id.0 == self.child { + Ok(GetAncestorsResponse { + tenant: tenant_ref(self.child), + ancestors: vec![tenant_ref(self.parent), tenant_ref(self.root)], + }) + } else { + Err(TenantResolverError::TenantNotFound { tenant_id: id }) + } + } + + async fn get_descendants( + &self, + _ctx: &SecurityContext, + _id: TenantId, + _options: &GetDescendantsOptions, + ) -> Result { + unimplemented!() + } + + async fn is_ancestor( + &self, + _ctx: &SecurityContext, + _ancestor_id: TenantId, + _descendant_id: TenantId, + _options: &IsAncestorOptions, + ) -> Result { + unimplemented!() + } +} + +// ── Tests ───────────────────────────────────────────────────────────────────── + +#[tokio::test] +async fn chain_includes_self_then_ancestors() { + let child = Uuid::new_v4(); + let parent = Uuid::new_v4(); + let root = Uuid::new_v4(); + + let client: Arc = + Arc::new(FakeTenantResolverClient::new(child, parent, root)); + let dir = TenantResolverDir::new(client, Arc::new(NoopMetrics), 60); + let ctx = make_ctx(); + + let chain = dir + .ancestor_chain(&ctx, DomainTenantId(child)) + .await + .expect("ancestor_chain"); + + assert_eq!(chain, vec![child, parent, root]); +} + +#[tokio::test] +async fn cache_hit_avoids_second_call() { + let child = Uuid::new_v4(); + let parent = Uuid::new_v4(); + let root = Uuid::new_v4(); + + let fake = Arc::new(FakeTenantResolverClient::new(child, parent, root)); + let client: Arc = Arc::clone(&fake) as _; + let dir = TenantResolverDir::new(client, Arc::new(NoopMetrics), 60); + let ctx = make_ctx(); + + dir.ancestor_chain(&ctx, DomainTenantId(child)) + .await + .expect("first call"); + dir.ancestor_chain(&ctx, DomainTenantId(child)) + .await + .expect("second call"); + + assert_eq!( + fake.call_count(), + 1, + "client should be called only once within TTL" + ); +} + +// Secrets must not be inherited across a self-managed tenant boundary, so the +// ancestor chain must be requested in a barrier-respecting mode. +#[tokio::test] +async fn ancestor_chain_requests_barrier_respecting_mode() { + let child = Uuid::new_v4(); + let parent = Uuid::new_v4(); + let root = Uuid::new_v4(); + + let fake = Arc::new(FakeTenantResolverClient::new(child, parent, root)); + let client: Arc = Arc::clone(&fake) as _; + let dir = TenantResolverDir::new(client, Arc::new(NoopMetrics), 60); + + dir.ancestor_chain(&make_ctx(), DomainTenantId(child)) + .await + .expect("ancestor_chain"); + + assert_eq!( + fake.last_barrier_mode(), + Some(BarrierMode::Respect), + "walk-up must request a barrier-respecting ancestor chain" + ); +} diff --git a/gears/credstore/credstore/src/infra/types_registry.rs b/gears/credstore/credstore/src/infra/types_registry.rs new file mode 100644 index 000000000..d12ad2065 --- /dev/null +++ b/gears/credstore/credstore/src/infra/types_registry.rs @@ -0,0 +1,187 @@ +//! `GtsSecretTypeResolver` — the production [`SecretTypeResolver`] wired +//! against `types_registry_sdk::TypesRegistryClient` resolved from +//! `ClientHub`. +//! +//! Mirrors AM's `GtsTenantTypeChecker` shape: +//! +//! 1. Resolve the [`GtsTypeSchema`] for the stored/declared type UUID via +//! [`TypesRegistryClient::get_type_schema_by_uuid`] (the SDK's local +//! client keeps a short TTL cache, so this is one cached lookup per +//! operation — no cache here). +//! 2. Reject schemas whose chain does not descend from the credstore +//! secret base type (`gts.cf.core.credstore.secret.v1~`) — anything +//! else cannot legitimately carry `SecretTypeTraits`. +//! 3. Read the effective traits via [`GtsTypeSchema::effective_traits`] +//! (chain merge: leaf-declared values win, the base fills the rest) +//! and deserialize them into [`SecretTypeTraits`]. +//! 4. Map failures fail-closed: not-registered / non-secret types onto +//! [`DomainError::TypeViolation`] (reason `UNKNOWN_SECRET_TYPE`, 400); +//! registry transport failures, timeouts, and trait-resolution +//! failures onto [`DomainError::ServiceUnavailable`] (503). +//! +//! Every call is recorded on the `types_registry` dependency-health +//! metrics so a registry outage shows up on the unified dashboard. + +use std::sync::Arc; +use std::time::{Duration, Instant}; + +use async_trait::async_trait; +use credstore_sdk::{SECRET_RESOURCE_TYPE, SecretTypeTraits}; +use toolkit_canonical_errors::CanonicalError; +use types_registry_sdk::{GtsTypeSchema, TypesRegistryClient}; +use uuid::Uuid; + +use crate::domain::error::DomainError; +use crate::domain::ports::metrics::{CredStoreMetricsPort, Dep, DepOp, Outcome}; +use crate::domain::secret::type_resolver::{ResolvedSecretType, SecretTypeResolver}; +use crate::domain::secret::typing::reasons; + +/// Default registry probe timeout (ms). Keeps a hung registry from +/// stalling an operation past the 503 fail-closed boundary; a healthy +/// registry round-trip (usually a local cache hit) is well under this. +/// Mirrors AM's `GtsTenantTypeChecker` default for operational +/// consistency. +const DEFAULT_PROBE_TIMEOUT_MS: u64 = 2_000; + +/// Production [`SecretTypeResolver`] backed by the GTS types-registry. +pub struct GtsSecretTypeResolver { + registry: Arc, + metrics: Arc, + probe_timeout: Duration, +} + +impl GtsSecretTypeResolver { + /// Construct a resolver around a registry client resolved from + /// `ClientHub`, using the default probe timeout. + #[must_use] + pub fn new( + registry: Arc, + metrics: Arc, + ) -> Self { + Self::with_timeout(registry, metrics, DEFAULT_PROBE_TIMEOUT_MS) + } + + /// Construct a resolver with an explicit probe timeout. + #[must_use] + pub fn with_timeout( + registry: Arc, + metrics: Arc, + probe_timeout_ms: u64, + ) -> Self { + Self { + registry, + metrics, + probe_timeout: Duration::from_millis(probe_timeout_ms.max(1)), + } + } + + /// Whether `schema` descends from the credstore secret base type. + /// Walks `ancestors()` (self → parent → ...), so the base type itself + /// also counts — it carries the generic trait defaults. + fn descends_from_secret_envelope(schema: &GtsTypeSchema) -> bool { + schema + .ancestors() + .any(|s| s.type_id.as_ref() == SECRET_RESOURCE_TYPE) + } +} + +/// `UNKNOWN_SECRET_TYPE` violation (canonical 400): the UUID does not +/// name a registered credstore secret type. +fn unknown_type(detail: String) -> DomainError { + DomainError::TypeViolation { + field: "type", + reason: reasons::UNKNOWN_SECRET_TYPE, + detail, + } +} + +/// Registry outage / trait-resolution failure (canonical 503). The +/// wire-visible detail stays curated; the raw registry error travels in +/// the cause chain only. +fn unavailable(detail: impl Into, cause: Option) -> DomainError { + DomainError::ServiceUnavailable { + detail: detail.into(), + retry_after: None, + cause: cause.map(|e| Box::new(e) as _), + } +} + +#[async_trait] +impl SecretTypeResolver for GtsSecretTypeResolver { + async fn resolve(&self, type_uuid: Uuid) -> Result { + let t0 = Instant::now(); + let result = tokio::time::timeout( + self.probe_timeout, + self.registry.get_type_schema_by_uuid(type_uuid), + ) + .await; + + // Transport-level outcome for the dependency-health dashboard: + // a per-key NotFound is a domain condition (unknown type), not a + // health signal. + let outcome = match &result { + Ok(Ok(_)) => Outcome::Success, + Ok(Err(CanonicalError::NotFound { .. })) => Outcome::NotFound, + Ok(Err(_)) | Err(_) => Outcome::Error, + }; + self.metrics.dependency( + Dep::TypesRegistry, + DepOp::GetTypeSchemaByUuid, + outcome, + t0.elapsed().as_secs_f64(), + ); + + let schema = match result { + Err(_elapsed) => { + return Err(unavailable("types-registry: timeout exceeded", None)); + } + Ok(Err(CanonicalError::NotFound { .. })) => { + return Err(unknown_type(format!( + "secret type {type_uuid} is not registered" + ))); + } + Ok(Err(err)) => { + tracing::warn!(uuid = %type_uuid, err = %err, "types-registry resolve failed"); + return Err(unavailable("types-registry unavailable", Some(err))); + } + Ok(Ok(schema)) => schema, + }; + + if !Self::descends_from_secret_envelope(&schema) { + return Err(unknown_type(format!( + "type {} ({type_uuid}) is not a credstore secret type (does not descend from {SECRET_RESOURCE_TYPE})", + schema.type_id.as_ref(), + ))); + } + + // The base type declares `x-gts-traits` values for every trait, so + // a properly registered descendant always resolves a complete map; + // a deserialization failure means the registered chain is malformed + // — fail closed as a (registration-fixable) 503, not a caller error. + let traits: SecretTypeTraits = + serde_json::from_value(schema.effective_traits()).map_err(|e| { + tracing::warn!( + uuid = %type_uuid, + type_id = %schema.type_id.as_ref(), + err = %e, + "secret type effective traits failed to resolve" + ); + unavailable( + format!( + "types-registry: secret type {} has malformed effective traits", + schema.type_id.as_ref(), + ), + None, + ) + })?; + + Ok(ResolvedSecretType { + gts_id: schema.type_id.as_ref().to_owned(), + traits, + }) + } +} + +#[cfg(test)] +#[path = "types_registry_tests.rs"] +mod tests; diff --git a/gears/credstore/credstore/src/infra/types_registry_tests.rs b/gears/credstore/credstore/src/infra/types_registry_tests.rs new file mode 100644 index 000000000..06b476159 --- /dev/null +++ b/gears/credstore/credstore/src/infra/types_registry_tests.rs @@ -0,0 +1,328 @@ +//! Unit tests for [`GtsSecretTypeResolver`]. + +use std::collections::HashMap; +use std::sync::{Arc, Mutex}; + +use async_trait::async_trait; +use serde_json::json; +use toolkit_gts::gts_id; +use types_registry_sdk::testing::MockTypesRegistryClient; +use types_registry_sdk::{GtsInstance, GtsTypeId, InstanceQuery, RegisterResult, TypeSchemaQuery}; + +use super::*; +use crate::domain::ports::metrics::NoopMetrics; +use credstore_sdk::SharingMode; + +/// Deterministic v5 UUID of a GTS type id, via the SDK helper. +fn uuid_of(gts_id: &str) -> Uuid { + credstore_sdk::types::type_uuid(gts_id).expect("valid gts id") +} + +/// The credstore secret base type as registered by the SDK: carries the +/// traits schema and the generic trait *values* (`validate_entity_traits` +/// requires the base of a traits-schema chain to carry values), so every +/// descendant inherits complete generic defaults. +fn secret_envelope() -> Arc { + let raw = json!({ + "$id": format!("gts://{SECRET_RESOURCE_TYPE}"), + "type": "object", + "x-gts-traits": { + "allow_sharing": ["private", "tenant", "shared"], + "expirable": false, + "utf8_only": false, + }, + }); + Arc::new( + GtsTypeSchema::try_new(GtsTypeId::new(SECRET_RESOURCE_TYPE), raw, None, None) + .expect("envelope schema"), + ) +} + +/// A derived secret-type schema declaring its own `x-gts-traits`. +fn derived(type_id: &str, traits: &serde_json::Value) -> GtsTypeSchema { + let raw = json!({ + "$id": format!("gts://{type_id}"), + "type": "object", + "allOf": [{"$ref": format!("gts://{SECRET_RESOURCE_TYPE}")}], + "x-gts-traits": traits, + }); + GtsTypeSchema::try_new(GtsTypeId::new(type_id), raw, None, Some(secret_envelope())) + .expect("derived schema") +} + +fn resolver(registry: Arc) -> GtsSecretTypeResolver { + GtsSecretTypeResolver::new(registry, Arc::new(NoopMetrics)) +} + +const CUSTOM_TYPE_ID: &str = + gts_id!("cf.core.credstore.secret.v1~acme.connectors.creds.db_password.v1~"); + +#[tokio::test] +async fn resolves_gts_id_and_effective_traits() { + let schema = derived( + CUSTOM_TYPE_ID, + &json!({ + "allow_sharing": ["private"], + "expirable": true, + "max_size_bytes": 8192, + "value_schema": {"type": "object", "required": ["password"]}, + }), + ); + let registry = Arc::new(MockTypesRegistryClient::new().with_type_schemas([schema])); + let resolved = resolver(registry) + .resolve(uuid_of(CUSTOM_TYPE_ID)) + .await + .expect("registered custom type resolves"); + + assert_eq!(resolved.gts_id, CUSTOM_TYPE_ID); + assert!(resolved.traits.allows_sharing(SharingMode::Private)); + assert!(!resolved.traits.allows_sharing(SharingMode::Tenant)); + assert!(resolved.traits.expirable); + assert_eq!(resolved.traits.max_size_bytes, Some(8192)); + // Undeclared traits inherit the base type's generic values. + assert!(!resolved.traits.utf8_only); + assert_eq!( + resolved.traits.value_schema, + Some(json!({"type": "object", "required": ["password"]})) + ); +} + +#[tokio::test] +async fn resolves_the_envelope_itself_with_generic_traits() { + let envelope = (*secret_envelope()).clone(); + let registry = Arc::new(MockTypesRegistryClient::new().with_type_schemas([envelope])); + let resolved = resolver(registry) + .resolve(uuid_of(SECRET_RESOURCE_TYPE)) + .await + .expect("base type resolves"); + + assert_eq!(resolved.gts_id, SECRET_RESOURCE_TYPE); + for mode in [ + SharingMode::Private, + SharingMode::Tenant, + SharingMode::Shared, + ] { + assert!(resolved.traits.allows_sharing(mode)); + } + assert!(!resolved.traits.expirable); + assert_eq!(resolved.traits.max_size_bytes, None); + assert_eq!(resolved.traits.value_schema, None); +} + +#[tokio::test] +async fn unknown_uuid_is_an_unknown_secret_type_violation() { + let registry = Arc::new(MockTypesRegistryClient::new()); + let err = resolver(registry) + .resolve(Uuid::from_u128(0xDEAD)) + .await + .expect_err("unregistered uuid must reject"); + assert!( + matches!( + err, + DomainError::TypeViolation { + reason: reasons::UNKNOWN_SECRET_TYPE, + .. + } + ), + "got: {err:?}" + ); +} + +#[tokio::test] +async fn non_secret_type_is_an_unknown_secret_type_violation() { + // Registered and resolvable, but rooted outside the credstore secret + // envelope — must not be admitted as a secret type even though its + // traits could deserialize by luck. + let alien_root = Arc::new( + GtsTypeSchema::try_new( + GtsTypeId::new(gts_id!("acme.core.events.type.v1~")), + json!({"type": "object"}), + None, + None, + ) + .expect("alien root"), + ); + let alien_id = gts_id!("acme.core.events.type.v1~acme.commerce.orders.order.v1~"); + let alien = GtsTypeSchema::try_new( + GtsTypeId::new(alien_id), + json!({"type": "object"}), + None, + Some(alien_root), + ) + .expect("alien leaf"); + let registry = Arc::new(MockTypesRegistryClient::new().with_type_schemas([alien])); + let err = resolver(registry) + .resolve(uuid_of(alien_id)) + .await + .expect_err("non-secret type must reject"); + assert!( + matches!( + err, + DomainError::TypeViolation { + reason: reasons::UNKNOWN_SECRET_TYPE, + .. + } + ), + "got: {err:?}" + ); + assert!(err.to_string().contains("does not descend"), "got: {err}"); +} + +#[tokio::test] +async fn malformed_effective_traits_fail_closed_as_service_unavailable() { + // `allow_sharing` declared with a value outside the SharingMode enum: + // the registry would reject this at registration, so hitting it at + // resolve time means the registered chain is broken — 503, not 400. + let schema = derived(CUSTOM_TYPE_ID, &json!({"allow_sharing": ["everyone"]})); + let registry = Arc::new(MockTypesRegistryClient::new().with_type_schemas([schema])); + let err = resolver(registry) + .resolve(uuid_of(CUSTOM_TYPE_ID)) + .await + .expect_err("malformed traits must fail closed"); + assert!( + matches!(err, DomainError::ServiceUnavailable { .. }), + "got: {err:?}" + ); + assert!(err.to_string().contains("malformed"), "got: {err}"); +} + +// -------- transport / infra -------- + +/// Minimal fake for the transport-failure and timeout paths the stateful +/// SDK mock cannot express. Only `get_type_schema_by_uuid` is reachable. +struct FailingRegistry { + error: CanonicalError, + delay: Option, + calls: Mutex, +} + +impl FailingRegistry { + fn new(error: CanonicalError) -> Self { + Self { + error, + delay: None, + calls: Mutex::new(0), + } + } + + fn with_delay(mut self, delay: Duration) -> Self { + self.delay = Some(delay); + self + } +} + +#[async_trait] +impl TypesRegistryClient for FailingRegistry { + async fn register( + &self, + _entities: Vec, + ) -> Result, CanonicalError> { + unreachable!() + } + async fn register_type_schemas( + &self, + _type_schemas: Vec, + ) -> Result, CanonicalError> { + unreachable!() + } + async fn get_type_schema(&self, _type_id: &str) -> Result { + unreachable!() + } + async fn get_type_schema_by_uuid( + &self, + _type_uuid: Uuid, + ) -> Result { + *self.calls.lock().expect("lock") += 1; + if let Some(d) = self.delay { + tokio::time::sleep(d).await; + } + Err(self.error.clone()) + } + async fn get_type_schemas( + &self, + _type_ids: Vec, + ) -> HashMap> { + unreachable!() + } + async fn get_type_schemas_by_uuid( + &self, + _type_uuids: Vec, + ) -> HashMap> { + unreachable!("resolver uses the single-key variant") + } + async fn list_type_schemas( + &self, + _query: TypeSchemaQuery, + ) -> Result, CanonicalError> { + unreachable!() + } + async fn register_instances( + &self, + _instances: Vec, + ) -> Result, CanonicalError> { + unreachable!() + } + async fn get_instance(&self, _id: &str) -> Result { + unreachable!() + } + async fn get_instance_by_uuid(&self, _uuid: Uuid) -> Result { + unreachable!() + } + async fn get_instances( + &self, + _ids: Vec, + ) -> HashMap> { + unreachable!() + } + async fn get_instances_by_uuid( + &self, + _uuids: Vec, + ) -> HashMap> { + unreachable!() + } + async fn list_instances( + &self, + _query: InstanceQuery, + ) -> Result, CanonicalError> { + unreachable!() + } +} + +#[tokio::test] +async fn registry_transport_error_maps_to_service_unavailable() { + let registry = Arc::new(FailingRegistry::new(types_registry_sdk::testing::internal( + "registry exploded", + ))); + let err = resolver(registry) + .resolve(Uuid::from_u128(0x1)) + .await + .expect_err("transport error must propagate as 503"); + assert!( + matches!(err, DomainError::ServiceUnavailable { .. }), + "got: {err:?}" + ); + // Curated wire detail; the raw registry error stays in the cause chain. + assert_eq!( + err.to_string(), + "service unavailable: types-registry unavailable" + ); +} + +#[tokio::test(start_paused = true)] +async fn slow_registry_times_out_as_service_unavailable() { + let registry = Arc::new( + FailingRegistry::new(types_registry_sdk::testing::internal("never reached")) + .with_delay(Duration::from_millis(50)), + ); + let resolver = GtsSecretTypeResolver::with_timeout(registry.clone(), Arc::new(NoopMetrics), 10); + let err = resolver + .resolve(Uuid::from_u128(0x1)) + .await + .expect_err("slow registry must time out"); + assert!( + matches!(err, DomainError::ServiceUnavailable { .. }), + "got: {err:?}" + ); + assert!(err.to_string().contains("timeout exceeded"), "got: {err}"); + assert_eq!(*registry.calls.lock().expect("lock"), 1); +} diff --git a/gears/credstore/credstore/src/lib.rs b/gears/credstore/credstore/src/lib.rs index d51d91de4..004eec116 100644 --- a/gears/credstore/credstore/src/lib.rs +++ b/gears/credstore/credstore/src/lib.rs @@ -1,12 +1,9 @@ -//! `CredStore` Gateway Gear -//! -//! Implements the `CredStore` gateway gear that: -//! 1. Registers the `CredStorePluginSpecV1` schema in types-registry -//! 2. Discovers plugin instances via types-registry (lazy, first-use) -//! 3. Routes `get`/`put`/`delete` calls through the selected plugin -//! 4. Registers `Arc` in `ClientHub` for consumers -#![cfg_attr(coverage_nightly, feature(coverage_attribute))] - +//! credstore — `CredStore` module: tenant-scoped secrets over pluggable backends. +pub mod api; +pub mod client; pub mod config; pub mod domain; pub mod gear; +pub mod infra; + +pub use gear::CredStoreGear; diff --git a/gears/credstore/docs/ADR/0001-cpt-cf-credstore-adr-stateful-gear.md b/gears/credstore/docs/ADR/0001-cpt-cf-credstore-adr-stateful-gear.md new file mode 100644 index 000000000..d6328c4ae --- /dev/null +++ b/gears/credstore/docs/ADR/0001-cpt-cf-credstore-adr-stateful-gear.md @@ -0,0 +1,115 @@ +--- +status: accepted +date: 2026-07-04 +--- +# ADR-0001: Stateful Gear with Gear-Owned Secret Metadata + + + + +- [Context and Problem Statement](#context-and-problem-statement) +- [Decision Drivers](#decision-drivers) +- [Considered Options](#considered-options) +- [Decision Outcome](#decision-outcome) + - [Consequences](#consequences) + - [Confirmation](#confirmation) +- [Pros and Cons of the Options](#pros-and-cons-of-the-options) + - [Stateless gear, metadata in the backend](#stateless-gear-metadata-in-the-backend) + - [Stateful gear, value-only backend](#stateful-gear-value-only-backend) +- [More Information](#more-information) +- [Traceability](#traceability) + + + +**ID**: `cpt-cf-credstore-adr-stateful-gear` + +## Context and Problem Statement + +The original CredStore design was built on a **stateless gear**: all +per-secret metadata (`sharing`, `owner_id`, `owner_tenant_id`) lived in the +external backend and was returned on every `get`; secret identity was a +deterministic ExternalID encoding `(tenant_id, key, owner_id) → +base64url(...)`. Hierarchical resolution walked the tenant ancestor chain +with up to N×2 backend round-trips (private probe + tenant/shared probe per +level), authorization was a coarse permission-string check, and a +private↔non-private sharing change had no atomic implementation ("no implicit +migration guarantee"). Launching required the backend to first grow a +three-value `sharing` enum and an `owner_id` column. + +Where should secret metadata live, and which component owns policy, +uniqueness, and identity? + +## Decision Drivers + +* Hierarchical resolution latency must not scale with tenant-hierarchy depth on the metadata side +* Tenant isolation must be enforceable at the data layer (platform PDP / SecureORM posture, consistent with RBAC/RG) +* No backend schema prerequisite — any dumb per-tenant key-value store should qualify as a backend plugin +* Encoded-ID collision surfaces must be designed out, not mitigated +* Writes spanning two stores need a recoverable failure story (crash-safety) +* Uniqueness rules (one non-private secret per `(tenant, ref)`; per-owner private secrets; coexistence of both classes under one reference) must be enforced somewhere authoritative + +## Considered Options + +* Stateless gear, metadata in the backend (original design) +* Stateful gear owning a metadata table, value-only backend + +## Decision Outcome + +Chosen option: "Stateful gear, value-only backend". The gear owns the +`credstore_secrets` table (identity, sharing, ownership, lifecycle status, +version); the backend plugin stores only the value, keyed by +`(tenant_id, key, private-per-owner | tenant class)`. Almost every other +design property follows from this single decision: + +* uniqueness/coexistence rules become **partial unique indexes**; +* hierarchical resolution becomes **one indexed SQL query** over the ancestor chain plus at most **one** backend read for the winning row; +* authorization becomes a PDP `AccessScope` **enforced in SQL** through SecureORM clamps on the metadata table (fail-closed, anti-enumeration 404 preserved); +* secret identity is a DB row — the ExternalID encoding and its collision analysis disappear; +* writes become explicit **sagas** over the metadata row and backend value, with a lifecycle `status` column and a reaper (see ADR-0002); +* optimistic concurrency (`version` / `ETag` / `If-Match`) becomes possible at the metadata layer; +* the private↔non-private "migration" hazard is designed out: the two classes coexist under one reference and the transition is simply rejected as unsupported. + +### Consequences + +* Good, because walk-up latency no longer scales with hierarchy depth; the backend is touched at most once per resolution +* Good, because tenant isolation (including `InTenantSubtree` scopes and isolation barriers) is enforced in SQL, not in application-level string checks +* Good, because backends need no metadata schema — the in-memory static plugin and any future vault plugin implement the same three-method value-store contract +* Good, because versioning, lifecycle statuses, and metadata-only future features (list, types) become cheap +* Bad, because the gear becomes a stateful gear: it needs a database, migrations, and the `stateful` capability +* Bad, because metadata and backend value can diverge transiently on partial failure — this cost is contained by the saga + reaper design (ADR-0002) + +### Confirmation + +* `credstore_secrets` schema with partial unique indexes ships in `m0001_initial_schema`; repo tests exercise uniqueness, coexistence, and scope clamps against SQLite +* Resolution is a single `resolve_for_get` query; the walk-up depth metric confirms no per-level backend calls +* E2E suite (`testing/e2e/gears/credstore/`) verifies hierarchical inheritance, shadowing, isolation, and optimistic concurrency through the REST API + +## Pros and Cons of the Options + +### Stateless gear, metadata in the backend + +* Good, because the gear needs no database (no migrations, no stateful capability) +* Good, because secret data and metadata live in exactly one store (no divergence class of failures) +* Bad, because every resolution costs up to N×2 backend round-trips for an N-deep hierarchy +* Bad, because the backend must implement the metadata schema (`sharing`, `owner_id`) before the gear can launch, and every future metadata feature requires a backend change +* Bad, because identity-by-encoding carries a collision surface that must be validated and mitigated forever +* Bad, because authorization stays an application-level permission-string check — tenant-subtree scope cannot be expressed, and uniqueness rules are only as strong as each backend's guarantees +* Bad, because half-written state is invisible: with no gear state there is no notion of a recoverable in-flight write + +### Stateful gear, value-only backend + +* Good / Bad — see Decision Outcome and Consequences above + +## More Information + +This ADR condenses the historical `DESIGN-ADDENDUM.md`, which recorded the +stateless→stateful delta section-by-section against the original (stateless) +revision of `DESIGN.md`. The full comparison, including the original section +references, is preserved in git history (`gears/credstore/docs/DESIGN-ADDENDUM.md` +prior to its removal). + +## Traceability + +* **Design**: [DESIGN.md](../DESIGN.md) §1, §3.1, §4.7 +* **Requirements**: `cpt-cf-credstore-fr-hierarchical-resolve`, `cpt-cf-credstore-fr-authz-pdp`, `cpt-cf-credstore-nfr-tenant-isolation` +* **Related**: [ADR-0002](./0002-cpt-cf-credstore-adr-deprovisioning-saga.md) diff --git a/gears/credstore/docs/ADR/0002-cpt-cf-credstore-adr-deprovisioning-saga.md b/gears/credstore/docs/ADR/0002-cpt-cf-credstore-adr-deprovisioning-saga.md new file mode 100644 index 000000000..5001a2202 --- /dev/null +++ b/gears/credstore/docs/ADR/0002-cpt-cf-credstore-adr-deprovisioning-saga.md @@ -0,0 +1,110 @@ +--- +status: accepted +date: 2026-07-04 +--- +# ADR-0002: Status-Driven Deprovisioning Saga with Name Retention + + + + +- [Context and Problem Statement](#context-and-problem-statement) +- [Decision Drivers](#decision-drivers) +- [Considered Options](#considered-options) +- [Decision Outcome](#decision-outcome) + - [Consequences](#consequences) + - [Confirmation](#confirmation) +- [Pros and Cons of the Options](#pros-and-cons-of-the-options) + - [Backend-first delete (no lifecycle state)](#backend-first-delete-no-lifecycle-state) + - [Deprovisioning status releasing the name immediately](#deprovisioning-status-releasing-the-name-immediately) + - [Deprovisioning status holding the name until cleanup](#deprovisioning-status-holding-the-name-until-cleanup) +- [Traceability](#traceability) + + + +**ID**: `cpt-cf-credstore-adr-deprovisioning-saga` + +## Context and Problem Statement + +With a stateful gear (ADR-0001) a delete spans two stores: the metadata +row and the backend value. The initial implementation deleted backend-first, +then removed the row. Partial failures had no self-healing owner: a failed +row delete left a value-less row (misleading 404s until a re-put), a crash +between the two steps was repaired only by accident, and — unlike the create +path — there was no lifecycle state a reaper could sweep. The create saga +also carried a documented debt: a `mark_active` failure after a successful +backend write orphaned the value in the backend forever. + +How should deletion be structured so that revocation is atomic for readers, +partial failures self-heal, and no failure mode can either resurrect a +deleted secret or destroy a successor secret's value? + +## Decision Drivers + +* Revocation must be atomic from the reader's perspective — no window where a "deleted" secret still resolves +* Every partial-failure state must have an owner: client retry and/or background reaper +* The backend key for a reference is deterministic — a lagging cleanup step must never be able to delete a *successor* secret's value written under the same key +* Symmetry with the existing provisioning saga (one mental model, one reaper) +* Close the create-saga orphaned-backend-value debt + +## Considered Options + +* Keep the backend-first delete (no lifecycle state) +* `deprovisioning` status; release the unique index (the name) immediately at mark time +* `deprovisioning` status; the row keeps holding the unique index until backend cleanup completes + +## Decision Outcome + +Chosen option: "Deprovisioning status holding the name until cleanup" +(the status set ships in the consolidated `m0001_initial_schema`). +Delete is a saga symmetric to provisioning: flip the row to +`deprovisioning` (version-gated when `If-Match` was given; the version is +*not* bumped so a precondition retry still matches) → backend delete +(`NotFound` = success) → row delete. From the flip onward the secret is +invisible to resolution (single status filter — no read/delete race), while +the row still holds the partial unique index, so re-creating the reference +fails with a retryable conflict until cleanup finishes. + +A stuck saga is resumed by either path: a `DELETE` retry (`find_own` returns +`deprovisioning` rows and re-runs the idempotent steps) or the periodic +reaper. The reaper also issues a best-effort backend delete for every stale +row it sweeps — including stuck `provisioning` rows — which closes the +orphaned-value debt of the create saga. + +### Consequences + +* Good, because revocation is atomic for readers and crash-safe: every intermediate state is non-readable and owned by retry or reaper +* Good, because name retention makes the "lagging cleanup deletes the successor's value" hazard structurally impossible (the successor cannot exist until the backend key is clean) +* Good, because one reaper loop and one status column serve both sagas; observability is symmetric (`provisioning_reaped` / `deprovisioning_reaped`, per-status inventory gauges) +* Bad, because a reference stays unavailable for re-creation for up to `deprovisioning_timeout_secs` + one reaper tick after a crashed delete (bounded, configurable) +* Bad, because a caller whose backend delete fails receives a retryable error for a secret that already no longer resolves — the 5xx reports cleanup state, not visibility + +### Confirmation + +* Service tests cover: backend failure leaves a non-resolving `deprovisioning` row; `DELETE` retry resumes; reaper completes stuck sagas and keeps rows while the backend still fails; provisioning-orphan reconciliation +* Repo tests confirm the `deprovisioning` row is invisible to `resolve_for_get`, visible to `find_own`, and still holds the unique index until reaped +* E2E `test_reference_reusable_after_delete` confirms the happy path releases the name immediately + +## Pros and Cons of the Options + +### Backend-first delete (no lifecycle state) + +* Good, because fewest moving parts — two operations, no new status +* Bad, because partial failures have no owner: a value-less row yields misleading 404s until a client happens to re-put +* Bad, because nothing sweeps a crashed delete; asymmetric with the create saga +* Bad, because the caller error surface conflates "not deleted" with "half deleted" + +### Deprovisioning status releasing the name immediately + +* Good, because the reference becomes reusable the instant the delete is accepted (best availability) +* Bad, because it is unsafe: the old saga's lagging backend delete and the new secret's value share the same deterministic backend key — cleanup could erase the successor's value +* Bad, because preventing that requires per-write backend key versioning (a backend contract change) or cross-saga coordination + +### Deprovisioning status holding the name until cleanup + +* Good / Bad — see Decision Outcome and Consequences above + +## Traceability + +* **Design**: [DESIGN.md](../DESIGN.md) §6.1, §6.3, §6.4 +* **Requirements**: `cpt-cf-credstore-fr-deprovisioning`, `cpt-cf-credstore-fr-write-lifecycle` +* **Related**: [ADR-0001](./0001-cpt-cf-credstore-adr-stateful-gear.md) diff --git a/gears/credstore/docs/ADR/0003-cpt-cf-credstore-adr-value-fingerprint-fence.md b/gears/credstore/docs/ADR/0003-cpt-cf-credstore-adr-value-fingerprint-fence.md new file mode 100644 index 000000000..7f19302f3 --- /dev/null +++ b/gears/credstore/docs/ADR/0003-cpt-cf-credstore-adr-value-fingerprint-fence.md @@ -0,0 +1,153 @@ +--- +status: accepted +date: 2026-07-08 +--- +# ADR-0003: Value-Fingerprint Fence for the Metadata/Value Dual Write + + + +- [Context and Problem Statement](#context-and-problem-statement) +- [Decision Drivers](#decision-drivers) +- [Considered Options](#considered-options) +- [Decision Outcome](#decision-outcome) + - [Consequences](#consequences) + - [Confirmation](#confirmation) +- [Pros and Cons of the Options](#pros-and-cons-of-the-options) + - [Serialize writers with a distributed lock](#serialize-writers-with-a-distributed-lock) + - [`SERIALIZABLE` isolation on the metadata write](#serializable-isolation-on-the-metadata-write) + - [Stamp the version into the backend value / key](#stamp-the-version-into-the-backend-value--key) + - [Value fingerprint in the gear row (CHOSEN)](#value-fingerprint-in-the-gear-row-chosen) +- [Traceability](#traceability) + + + +**ID**: `cpt-cf-credstore-adr-value-fingerprint-fence` + +## Context and Problem Statement + +With a stateful gear (ADR-0001) a write spans two stores with no shared +transaction: the value goes to the external value-store backend +(`plugin.put`), the metadata (`sharing`, `version`, `expires_at`) to +`credstore_secrets` (`touch`). On the last-writer-wins path (no `If-Match`) +both a backend write and a metadata write happen, so two concurrent PUTs to +one reference can interleave crosswise: + +``` +Alice: PUT value=A, sharing=shared Bob: PUT value=B, sharing=tenant + plugin.put(A) plugin.put(B) → backend = B + touch(sharing=shared) touch(sharing=tenant) + # commit order: put(A), put(B), touch(Bob=tenant), touch(Alice=shared) + # end state: backend = B row.sharing = shared +``` + +A descendant tenant then resolves Bob's value under Alice's `shared` label — +a durable cross-tenant disclosure of a secret its writer marked tenant-only +(review finding #2). Independently, the `ETag` was the bare `version` +counter, which restarts at `1` for every recreated row, so a stale client's +`If-Match: "1"` matched a *different generation* and overwrote it — the ABA +lost update the strong validator exists to prevent (finding #1). + +Optimistic concurrency (the `If-Match`/version precondition) does not by itself +close the crosswise disclosure. It is opt-in — the last-writer-wins path above +carries no `If-Match` at all — and even when a client supplies one it guards +only the *metadata row's* version, within a single store. It cannot bind the +backend value to the metadata across two transaction-less stores, so it never +detects the failure here: the surviving row's `sharing` label was set by a +different writer than the one whose value survived in the backend. Serializing +the two writes would need a fence the platform's primitives do not provide: + +The platform's coordination primitives are explicitly not built to fence +this: the cluster distributed lock has **no fencing tokens** and forbids +remote I/O inside a lock's critical section (cluster ADR-002), and the +`coord` DB-lease documents the same non-guarantee for external side effects. +How do we make the metadata and the value provably consistent on read +without holding a lock across the backend write, and without a bare-version +validator? + +## Decision Drivers + +* No value under a sharing label a different writer set — fail closed, never disclose +* Correct across replicas (the gear is multi-instance) without new infrastructure +* Do not hold any lock across the external `plugin.put` (cluster ADR-002) +* Nothing derived from a secret value may leave the gear (no fingerprint on the wire, in headers, or in logs) +* The backend stays an opaque byte store — the same mechanism works for the in-memory static plugin and any future vault-backed plugin, with no plugin-contract change +* Close the ABA lost update in the same change +* Fail-closed and self-healing under every partial-failure and key-loss mode + +## Considered Options + +* Serialize writers with a distributed lock (cluster / `coord`) +* `SERIALIZABLE` isolation on the metadata write +* Stamp the version into the backend value (inline framing) or backend key +* Value fingerprint (`HMAC-SHA256(fence_key, value)`) in the gear row, verified on read + +## Decision Outcome + +Chosen option: **"Value fingerprint in the gear row"**. Each row stores +`value_fp = HMAC-SHA256(fence_key, value)`, stamped in the same atomic +`touch`/insert as `sharing`; reads recompute it from the value the backend +returned and serve only on a match, else fail closed as an anti-enumeration +404. The atomic co-write of `value_fp` and `sharing` is what proves a matched +value and its metadata came from one writer. The fence key is auto-generated +and stored in the backend under a reserved, API-unreachable entry (split +knowledge: fingerprints in the DB, key with the values). The `ETag` becomes +the generation-bound pair `"."`, closing the ABA. This is +precisely the application-level fencing cluster ADR-002 prescribes for an +"external storage layer with fencing support" — here the gear supplies +that support on the backend's behalf. See §4.10 of DESIGN. + +### Consequences + +* A crosswise LWW interleave, a half-completed overwrite, or a lost/replaced fence key can only ever produce a fingerprint mismatch → 404, healed by any subsequent successful PUT. No state serves a value under mismatched metadata. +* LWW is no longer strict last-writer-wins under a same-value/different-sharing race: the loser's intent can be lost. This is not a disclosure — a served value's sharing label was set by the writer of *that* value (the atomic co-write), so no value is ever exposed under a sharing nobody assigned it. +* Backend value bytes and the plugin contract are unchanged; a future vault-backed plugin needs no fencing support of its own. +* A value-derived artifact (the keyed HMAC) now lives at rest in the gear DB. It never leaves the gear, and the HMAC key (not in the DB) blocks offline dictionary attack on a DB-only compromise. +* A new dependency surface: the fence key must be reachable at write/read time. Its outage coincides with a backend outage (same store), which already fails the operation; a lost/rotated key fails closed (loud `fence_verify{outcome="mismatch"}`) and re-PUT heals. +* Out-of-band DB seeding must set `value_fp = NULL` (and reset it on re-seed); such rows are served on trust and backfilled on first read / reaper sweep. +* Key rotation is out of scope; the `fp_key_id` column and reserved-reference naming are the groundwork (a keyring verified by id, lazy re-stamp) for a later change. + +### Confirmation + +* Unit test: the poisoned crosswise end-state (backend value ≠ row fingerprint) reads as a 404, and a subsequent PUT heals it. +* Unit + e2e: a recreated secret rejects the previous generation's `"."` validator (409) even though the version counters coincide. +* Unit: an out-of-band `value_fp = NULL` row serves on trust, backfills exactly once via a CAS that does not bump the version, then verifies `ok`. +* Unit: fence-key bootstrap persists the key under the reserved nil-tenant entry; a tenant PUT to the same reference never clobbers it and cannot resolve it. +* Unit: a stale cached key self-heals via the one-shot refresh on mismatch. +* Metrics: `fence_verify{ok|legacy|mismatch}` and `fence_backfill{...}` are emitted with the documented labels. + +## Pros and Cons of the Options + +### Serialize writers with a distributed lock + +Hold a cluster / `coord` lock per `(tenant, ref)` across `plugin.put` + `touch`. + +* Bad, because both primitives forbid exactly this (no remote I/O in a lock critical section; no fencing tokens — cluster ADR-002), so a TTL-preempted holder still races. +* Bad, because it needs shared infrastructure (a linearizable cache / DB-lease table) and a per-write acquire on the hot path. +* Bad, because even a perfect lock cannot make a two-store write atomic; a pause between the two writes still desyncs. + +### `SERIALIZABLE` isolation on the metadata write + +* Bad, because isolation only serializes operations *within* the DB; the external `plugin.put` is outside any transaction, so the value/metadata desync is untouched. +* Bad, because holding a transaction open across the backend network call is a long-transaction anti-pattern and still does not order the external write. + +### Stamp the version into the backend value / key + +Inline-frame `magic‖version‖value`, or route each version to a distinct backend key. + +* Good, because it is airtight and keeps no value-derived data in the DB. +* Bad, because it mutates the stored secret (a direct backend reader sees framed bytes) or changes the key scheme and accumulates old-version values needing GC. +* Bad, because "is there a stamp?" becomes byte-sniffing a magic prefix (collision-prone) instead of a clean typed check; and for a structured backend (e.g. OpenBao KV) it either mangles the value field or needs a plugin-contract change. + +### Value fingerprint in the gear row (CHOSEN) + +* Good, because the backend and the plugin contract are untouched — one mechanism for every backend. +* Good, because verification is a clean recompute-and-compare, and "unstamped" is a NULL column, not a guessed prefix. +* Good, because it is the exact pattern cluster ADR-002 points external-resource fencing to (application-level), and it closes the ABA in the same `ETag` change. +* Neutral, because it stores a keyed HMAC of the value at rest — mitigated by keying and by never exposing it. +* Bad (accepted), because it does not preserve strict LWW under a same-value/different-sharing race (a lost update, never a disclosure). + +## Traceability + +- Requirements: `cpt-cf-credstore-fr-put-secret`, `cpt-cf-credstore-fr-get-secret`, `cpt-cf-credstore-fr-sharing-modes`, `cpt-cf-credstore-fr-optimistic-concurrency`, `cpt-cf-credstore-nfr-tenant-isolation`, `cpt-cf-credstore-nfr-confidentiality` +- Supersedes the bare-version `ETag` described in earlier revisions of DESIGN §4.3; builds on the dual-write model of ADR-0001 / ADR-0002. +- Related: cluster ADR-002 (`gears/system/cluster/docs/ADR/002-async-boundary-no-remote-in-critical-section.md`). diff --git a/gears/credstore/docs/DESIGN.md b/gears/credstore/docs/DESIGN.md index cd6ab4d2f..72fabe07d 100644 --- a/gears/credstore/docs/DESIGN.md +++ b/gears/credstore/docs/DESIGN.md @@ -23,22 +23,29 @@ - [4.7 Database schemas & tables](#47-database-schemas--tables) - [4.8 Deployment Topology](#48-deployment-topology) - [4.9 Technology Stack](#49-technology-stack) -- [5. Risks / Trade-offs](#5-risks--trade-offs) - - [5.1 Architectural Trade-offs](#51-architectural-trade-offs) - - [5.2 Security and Performance Risks](#52-security-and-performance-risks) -- [6. Migration Plan](#6-migration-plan) - - [6.1 Schema Migration: Two-Mode to Three-Mode Sharing](#61-schema-migration-two-mode-to-three-mode-sharing) - - [6.2 Backward Compatibility](#62-backward-compatibility) - - [6.3 Rollback Plan](#63-rollback-plan) - - [6.4 Success Criteria](#64-success-criteria) -- [7. Open Questions](#7-open-questions) - - [7.1 From PRD (Cross-Reference)](#71-from-prd-cross-reference) - - [7.2 Design-Specific Questions](#72-design-specific-questions) -- [8. Additional context](#8-additional-context) + - [4.10 Value-Fingerprint Fence](#410-value-fingerprint-fence) +- [5. Secret Types (GTS-Based, Registry-Driven)](#5-secret-types-gts-based-registry-driven) + - [5.1 Concept](#51-concept) + - [5.2 Type Traits](#52-type-traits) + - [5.3 Built-in Type Catalog (Registry Seeds)](#53-built-in-type-catalog-registry-seeds) + - [5.4 Enforcement Points](#54-enforcement-points) + - [5.5 Storage & API Changes](#55-storage--api-changes) +- [6. Secret Lifecycle & Sagas](#6-secret-lifecycle--sagas) + - [6.1 Status Model](#61-status-model) + - [6.2 Provisioning Saga](#62-provisioning-saga) + - [6.3 Deprovisioning Saga](#63-deprovisioning-saga) + - [6.4 Reaper](#64-reaper) +- [7. Risks / Trade-offs](#7-risks--trade-offs) + - [7.1 Architectural Trade-offs](#71-architectural-trade-offs) + - [7.2 Security and Performance Risks](#72-security-and-performance-risks) +- [8. Migration Plan](#8-migration-plan) +- [9. Open Questions](#9-open-questions) +- [10. Additional context](#10-additional-context) - [Plugin Registration](#plugin-registration) - [Configuration](#configuration) - [Error Mapping](#error-mapping) -- [9. Traceability](#9-traceability) + - [Observability](#observability) +- [11. Traceability](#11-traceability) @@ -66,21 +73,11 @@ NOT IN THIS DOCUMENT (see other templates): ✗ Detailed rationale for decisions → ADR/ ✗ Step-by-step implementation flows → features/ -STANDARDS ALIGNMENT: - - IEEE 1016-2009 (Software Design Description) - - IEEE 42010 (Architecture Description — viewpoints, views, concerns) - - ISO/IEC 15288 / 12207 (Architecture & Design Definition processes) - -ARCHITECTURE VIEWS (per IEEE 42010): - - Context view: system boundaries and external actors - - Functional view: components and their responsibilities - - Information view: data models and flows - - Deployment view: infrastructure topology - DESIGN LANGUAGE: - Be specific and clear; no fluff, bloat, or emoji - Reference PRD requirements using `cpt-cf-credstore-fr-{slug}` IDs - - Reference ADR documents using `cpt-cf-credstore-adr-{slug}` IDs + - Sections marked **Planned** describe target design not yet implemented; + everything else describes the shipped implementation. ============================================================================= --> @@ -88,13 +85,30 @@ DESIGN LANGUAGE: ### 1.1 Architectural Vision -CredStore follows the ToolKit Gateway + Plugins pattern (same architecture as `tenant_resolver`). A gateway gear (`credstore`) exposes a simple public API to platform consumers, enforces authorization policy, and implements hierarchical secret resolution. Backend-specific storage is implemented as plugins that register via the GTS type system and are selected at runtime by configuration. - -The SDK crate (`credstore-sdk`) defines two trait boundaries: `CredStoreClientV1` for consumers and `CredStorePluginClientV1` for backend implementations. Consumers depend only on the gateway trait and never interact with plugins directly. This decoupling allows runtime backend selection without changing consumer code. - -The architecture provides simple CRUD operations (get, put, delete) for tenant-scoped secrets. The tenant ID is always derived from SecurityCtx for self-service operations. Authorization is enforced exclusively in the gateway layer. For simple backend plugins (VendorA Credstore, OS keychain), **hierarchical secret resolution** (the walk-up algorithm that searches for secrets across tenant ancestors) is implemented in the Gateway using `tenant_resolver` to query the tenant hierarchy. These plugins are storage adapters providing per-tenant key-value operations with no policy or hierarchical logic. - -The `credentials_storage` plugin is an exception to this pattern. It is a standalone Rust microservice that implements credential merge/propagation resolution internally (own → inherited → default), along with encrypted credential storage, schema validation, field-level masking, and pluggable tenant key management via a `KeyProvider` abstraction. When this plugin is active, the Gateway delegates merge resolution to the plugin rather than performing the walk-up algorithm itself. The `KeyProvider` supports two modes: local database storage (for development/simple deployments) and external key management service integration (HashiCorp Vault, AWS KMS) for production environments requiring key–data separation. The detailed plugin architecture will be documented in `plugins/credentials-storage/DESIGN.md`. +CredStore follows the ToolKit Gear + Plugins pattern: a **stateful +gear** (`credstore`) owns all secret *metadata* (identity, sharing, ownership, +lifecycle status, version) in its own database table, enforces authorization +and hierarchical resolution, and exposes the public API; backend **plugins** +are pure per-tenant *value stores* selected at runtime by GTS vendor +configuration. The backend stores the secret value only — it carries no +metadata schema, no sharing semantics, and no policy. + +The SDK crate (`credstore-sdk`) defines two trait boundaries: +`CredStoreClientV1` for consumers and `CredStorePluginClientV1` for backend +implementations. Consumers depend only on the gear trait and never interact +with plugins directly, which allows runtime backend selection without changing +consumer code. + +Because metadata is local, hierarchical resolution (the walk-up that searches +for secrets across tenant ancestors) is a **single indexed SQL query** over the +metadata table followed by at most **one** backend read for the winning row. +Writes are **compensating sagas** over the metadata row and the backend value, +made crash-safe by an explicit lifecycle status and a periodic reaper. + +Authorization is delegated to the platform PDP (`authz-resolver`) via +`PolicyEnforcer`: each operation evaluates an `AccessScope` that is enforced +**in SQL** through SecureORM clamps on the metadata table. Tenant isolation is +therefore enforced at the data layer, consistent with the rest of the platform. ### 1.2 Architecture Drivers @@ -102,895 +116,1057 @@ The `credentials_storage` plugin is an exception to this pattern. It is a standa | Requirement | Design Response | |-------------|-----------------| -| `cpt-cf-credstore-fr-put-secret` | Plugin `put` with tenant_id, key, value, sharing → backend storage | -| `cpt-cf-credstore-fr-get-secret` | Plugin `get` with tenant_id, key, optional owner_id → backend lookup (two-phase: private then tenant/shared) | -| `cpt-cf-credstore-fr-delete-secret` | Plugin `delete` with tenant_id, key, optional owner_id → backend removal | -| `cpt-cf-credstore-fr-tenant-scoping` | Gateway extracts tenant_id from SecurityCtx before delegating to plugin | -| `cpt-cf-credstore-fr-sharing-modes` | `sharing` field in Credstore backend (VendorA); passed through from Gateway API | -| `cpt-cf-credstore-fr-authz-gateway` | Gateway checks SecurityCtx permissions before any plugin call | -| `cpt-cf-credstore-fr-rw-separation` | VendorA plugin configures separate RO/RW OAuth2 client credentials | -| `cpt-cf-credstore-fr-external-key-mgmt` | Credentials Storage plugin supports pluggable `KeyProvider`: local DB storage for dev, external KMS (Vault, AWS KMS) for production key–data separation | +| `cpt-cf-credstore-fr-put-secret` | Write saga: insert `provisioning` metadata row → plugin `put` (value only) → mark `active`; overwrite path is backend-first, then version bump | +| `cpt-cf-credstore-fr-get-secret` | Single SQL resolution over the ancestor chain, then one plugin `get` for the winning row | +| `cpt-cf-credstore-fr-delete-secret` | Deprovisioning saga: mark `deprovisioning` → backend delete → row delete (§6.3) | +| `cpt-cf-credstore-fr-tenant-scoping` | Gear derives tenant from `SecurityContext.subject_tenant_id()`; own-tenant gate + SecureORM scope clamp | +| `cpt-cf-credstore-fr-sharing-modes` | `sharing` column in the gear metadata table; partial unique indexes let private and tenant/shared coexist under one reference | +| `cpt-cf-credstore-fr-authz-pdp` | PDP `AccessScope` per operation (`read`/`write`/`delete` on the secret GTS resource type), enforced in SQL; fail-closed | +| `cpt-cf-credstore-fr-optimistic-concurrency` | Monotonic `version` column; `GET` returns a strong generation-bound `ETag` (`"."`, §4.10); `PUT`/`DELETE` honour `If-Match` | +| `cpt-cf-credstore-fr-secret-types` | GTS-based secret types with enforceable traits (§5) | +| `cpt-cf-credstore-fr-deprovisioning` | `deprovisioning` status + compensating delete saga swept by the reaper (§6.3) | #### NFR Allocation | NFR ID | NFR Summary | Allocated To | Design Response | Verification Approach | |--------|-------------|--------------|-----------------|----------------------| -| `cpt-cf-credstore-nfr-confidentiality` | Secret values never in logs | Gateway + plugins | `SecretValue` wrapper type with custom `Debug`/`Display` that redacts content; log scrubbing at transport layer | Automated log scan in integration tests | +| `cpt-cf-credstore-nfr-confidentiality` | Secret values never in logs or caches | SDK + gear + plugins | `SecretValue` wrapper with redacting `Debug`/`Display` and zeroize-on-drop; hand-written redacted `Debug` on REST DTOs; `Cache-Control: no-store` on `GET`; no lossy UTF-8 decode | Unit tests on redaction; code review | +| `cpt-cf-credstore-nfr-tenant-isolation` | No cross-tenant access outside PDP scope | Gear + repo | Scope clamps in SQL; own-tenant gate with `cross_tenant_denied` metric; barrier-respecting ancestor chain | Repo/service tests incl. barrier and scope cases | +| `cpt-cf-credstore-nfr-observability` | Operational visibility | Gear | OpenTelemetry metrics: walk-up depth, read outcome, dependency timings, saga rollback/reap counters, inventory gauge | Metrics unit tests | + +#### Key ADRs + +| ADR ID | Decision Summary | +|--------|------------------| +| `cpt-cf-credstore-adr-stateful-gear` | Stateful gear, value-only backend: the gear owns the `credstore_secrets` metadata table (identity, sharing, ownership, lifecycle status, version); the backend plugin stores only the value ([ADR-0001](./ADR/0001-cpt-cf-credstore-adr-stateful-gear.md)) | +| `cpt-cf-credstore-adr-deprovisioning-saga` | Delete is a saga symmetric to provisioning: a `deprovisioning` status holds the unique name until backend cleanup completes; stuck rows are swept by the reaper ([ADR-0002](./ADR/0002-cpt-cf-credstore-adr-deprovisioning-saga.md)) | +| `cpt-cf-credstore-adr-value-fingerprint-fence` | Value-fingerprint fence + generation-bound ETag: the gear stamps `value_fp = HMAC(fence_key, value)` in the same write as `sharing` and verifies it on read (fail-closed 404 on mismatch), and binds the strong ETag to `.`; closes the crosswise-PUT cross-tenant disclosure and the recreate ABA lost-update ([ADR-0003](./ADR/0003-cpt-cf-credstore-adr-value-fingerprint-fence.md)) | ### 1.3 Architecture Layers ``` -┌─────────────────────────────────────────────────────────────┐ -│ Consumers (OAGW, gears) │ -├─────────────────────────────────────────────────────────────┤ -│ credstore-sdk │ Public API traits, models, errors │ -├─────────────────────────────────────────────────────────────┤ -│ credstore. │ Authorization, plugin selection, REST │ -├─────────────────────────────────────────────────────────────┤ -│ Plugins │ Backend-specific storage adapters │ -│ ┌──────────────────────┐ ┌──────────────────────────────┐ │ -│ │ credstore_vendor_a │ │ os_protected_storage (P2) │ │ -│ │ (Credstore REST) │ │ (macOS Keychain / Win DPAPI) │ │ -│ └──────────────────────┘ └──────────────────────────────┘ │ -│ ┌──────────────────────────────────────────────────────────┐│ -│ │ credentials_storage (Rust microservice) ││ -│ │ AES-256-GCM encryption, KeyProvider, schema validation ││ -│ └──────────────────────────────────────────────────────────┘│ -├─────────────────────────────────────────────────────────────┤ -│ External │ VendorA Credstore, OS keychain │ -└─────────────────────────────────────────────────────────────┘ +┌───────────────────────────────────────────────────────────────┐ +│ Consumers (OAGW, mini-chat, gears) │ +├───────────────────────────────────────────────────────────────┤ +│ credstore-sdk │ Public API traits, models, errors, GTS │ +├───────────────────────────────────────────────────────────────┤ +│ credstore │ PDP authz, resolution, write sagas, REST, │ +│ (stateful) │ reaper, metrics — owns credstore_secrets │ +├───────────────────────────────────────────────────────────────┤ +│ Plugins │ Pure per-tenant value stores │ +│ ┌────────────────────────────┐ ┌──────────────────────────┐ │ +│ │ static-credstore-plugin │ │ production vaults │ │ +│ │ (in-memory, dev/test) │ │ (future: external store, │ │ +│ │ │ │ OS keychain, KMS-backed)│ │ +│ └────────────────────────────┘ └──────────────────────────┘ │ +├───────────────────────────────────────────────────────────────┤ +│ Platform deps │ authz-resolver (PDP), tenant-resolver, │ +│ │ types-registry (GTS), toolkit-db │ +└───────────────────────────────────────────────────────────────┘ ``` | Layer | Responsibility | Technology | |-------|---------------|------------| -| SDK | Public and plugin trait definitions, models, errors | Rust crate (`credstore-sdk`) | -| Gateway | Authorization enforcement, hierarchical resolution, sharing mode enforcement, plugin resolution, REST API | Rust crate (`credstore`), Axum, tenant_resolver | -| Plugins | Backend-specific secret storage operations. Simple plugins (VendorA, OS keychain) provide per-tenant CRUD only. The `credentials_storage` plugin handles merge resolution, encryption, and schema validation internally. | Rust crates, HTTP client / OS APIs | -| External | Secret persistence and encryption (no hierarchical logic) | VendorA Credstore (Go), OS keychain, External Key Service (Vault/KMS) | +| SDK | Public and plugin trait definitions, models, errors, GTS type declarations | Rust crate (`credstore-sdk`) | +| Gear | PDP authorization, hierarchical resolution, sharing enforcement, write sagas, lifecycle reaper, plugin selection, REST API | Rust crate (`credstore`), Axum, SeaORM/SecureORM | +| Plugins | Backend-specific secret **value** storage (per-tenant key-value CRUD; no policy, no hierarchy, no metadata) | Rust crates | +| Platform | Policy decisions (PDP), tenant hierarchy, plugin discovery | authz-resolver, tenant-resolver, types-registry | ## 2. Goals / Non-Goals ### 2.1 Goals -The CredStore design aims to achieve the following objectives: - - Provide secure, hierarchical secret storage for platform gears and tenant administrators - Enable flexible sharing modes: `private` (owner-only), `tenant` (tenant-wide, default), `shared` (hierarchical) - Support service-to-service secret retrieval (e.g., OAGW retrieving secrets on behalf of customer tenants) -- Implement authorization and hierarchical resolution in Gateway gear (centralized policy enforcement) -- Support multiple backend storage options via plugin architecture (VendorA Credstore, OS keychain, Credentials Storage microservice) -- Support pluggable tenant encryption key management via `KeyProvider` abstraction in the Credentials Storage plugin (local DB for dev, external KMS for production key–data separation) -- Ensure secret values never appear in logs, error messages, or debug traces -- Provide simple CRUD operations with clear REST semantics +- Enforce authorization via the platform PDP with SQL-level scope clamps (real tenant isolation at the data layer) +- Make writes crash-safe and self-healing (saga + reaper), and reads race-free against half-written secrets +- Support optimistic concurrency (version / `ETag` / `If-Match`) for lost-update detection +- Honour platform tenant-isolation barriers during hierarchical resolution +- Support multiple backend value stores via plugin architecture with GTS-based runtime selection +- Ensure secret values never appear in logs, error messages, debug traces, or intermediary caches - Enable secret shadowing: child tenants can override parent credentials without breaking existing references +- Classify secrets by GTS-based *secret types* with enforceable traits (§5) +- Symmetric, crash-safe deletion via a `deprovisioning` saga (§6.3) ### 2.2 Non-Goals -The following capabilities are explicitly out of scope for v1: +The following capabilities are explicitly out of scope: -- **Granular ACL beyond hierarchical**: Fine-grained access control (role-based, attribute-based, per-secret ACLs) is out of scope. The three-tier sharing model covers primary use cases. Future enhancement documented in PRD Open Questions. -- **Secret versioning / history**: Versioned secrets, version rollback, or secret history tracking is out of scope. -- **Secret rotation automation**: Automatic secret rotation, expiration, or lifecycle management is out of scope. -- **Direct end-user access**: Unauthenticated or untrusted client access (e.g., browser-based secret retrieval without platform authentication) is out of scope. -- **Secret templates or composition**: Dynamic secret generation, composition from templates, or secret derivation is out of scope. -- **Hierarchical resolution in simple backends**: Simple backends (VendorA Credstore, OS keychain) provide per-tenant key-value storage only — all hierarchical walk-up logic, sharing mode enforcement, and policy decisions are in Gateway. The `credentials_storage` plugin is an exception: it implements credential merge resolution internally. -- **Secret discovery / search**: Listing all secrets, searching by tags, or full-text search across secret values is out of scope for v1. +- **Granular ACL beyond hierarchical**: fine-grained per-secret ACLs (role- or attribute-based) are out of scope. The three-tier sharing model plus PDP scope covers primary use cases. +- **Secret value history / rollback**: the `version` column supports optimistic locking only; previous values are not retained. +- **Secret rotation automation**: automatic rotation is out of scope. (Secret types may carry *advisory* rotation traits, §5.2 — enforcement/automation is future work.) +- **Direct end-user access**: unauthenticated or untrusted client access is out of scope. +- **Secret templates or composition**: dynamic secret generation or derivation is out of scope. +- **Hierarchical resolution in backends**: plugins are pure per-tenant value stores — all hierarchy, sharing, and policy logic lives in the gear. +- **Secret discovery / search**: listing or searching secrets is out of scope for v1. +- **MySQL support**: migrations target PostgreSQL and SQLite; MySQL fails fast with a typed error. ## 3. Principles & Constraints ### 3.1 Design Principles -#### Authorization in Gateway +#### Stateful Gear, Value-Only Backend -- [ ] `p1` - **ID**: `cpt-cf-credstore-principle-authz-gateway` +- [ ] `p1` - **ID**: `cpt-cf-credstore-principle-stateful-gear` -Authorization (permission checks for `Secrets:Read` and `Secrets:Write`) is enforced exclusively in the gateway layer. Simple plugins (VendorA, OS keychain) are "storage adapters" that delegate to backends and MUST NOT implement authorization or policy decisions. This prevents inconsistent behavior across backends. +The gear owns all secret metadata in its own `credstore_secrets` table; the +backend plugin stores the value only, keyed by `(tenant_id, key, key-class)` +where the key class is `private-per-owner` (`owner_id = Some`) or `tenant` +(`owner_id = None`). This removes any backend metadata-schema prerequisite, +eliminates encoded-external-ID collision risk, and makes resolution and +authorization a single transactional query. -**Note**: For simple plugins, the Gateway also implements sharing mode enforcement and hierarchical resolution. The `credentials_storage` plugin is a full microservice that handles its own merge resolution, authorization (JWT + Permission Service), and sharing logic internally — in this case the Gateway delegates these responsibilities to the plugin. +#### Authorization via PDP, Enforced in SQL -#### Stateless Key Mapping +- [ ] `p1` - **ID**: `cpt-cf-credstore-principle-authz-pdp` -- [ ] `p1` - **ID**: `cpt-cf-credstore-principle-stateless-mapping` +Every operation evaluates a PDP `AccessScope` for its action (`read`, `write`, +`delete`) on the secret resource type (`gts.cf.core.credstore.secret.v1~`) and +enforces that scope in SQL through SecureORM clamps on the metadata table. +Both read and write paths additionally gate on an explicit own-tenant +invariant (`scope_includes_tenant`) and emit a `cross_tenant_denied` metric. +Out-of-scope access is fail-closed and surfaces as the canonical 404 +(anti-enumeration) or 403. Plugins MUST NOT implement authorization. -The VendorA Credstore plugin uses a deterministic, stateless mapping from `(tenant_id, key, optional owner_id)` to Credstore ExternalID. Private secrets include `owner_id` in the mapping to support per-owner namespacing. No local mapping database is required. This simplifies operations and eliminates a failure mode. - -#### Tenant from SecurityCtx +#### Tenant from SecurityContext - [ ] `p1` - **ID**: `cpt-cf-credstore-principle-tenant-from-ctx` -For self-service operations, the tenant is always derived from `SecurityCtx.tenant_id()`. This reduces API surface, prevents misuse, and aligns with existing platform patterns (consistent with `tenant_resolver`). +The operating tenant is always derived from +`SecurityContext.subject_tenant_id()`, and the owner from +`SecurityContext.subject_id()`. This reduces API surface, prevents misuse, and +aligns with platform patterns. Service-to-service consumers (OAGW) construct a +`SecurityContext` for the target tenant rather than passing tenant parameters. -### 3.2 Constraints +#### Crash-Safe Writes (Saga + Reaper) -#### OAuth2 for Credstore +- [ ] `p1` - **ID**: `cpt-cf-credstore-principle-write-saga` -- [ ] `p1` - **ID**: `cpt-cf-credstore-constraint-oauth2` +A write that spans the metadata table and the backend is a compensating saga +with an explicit lifecycle status (§6). Every failure mode either rolls back, +self-heals on retry, or is swept by the periodic reaper — a crash can never +permanently wedge a reference or leak a readable half-written secret. -All Credstore REST calls require OAuth2 client credentials authentication. Token acquisition and caching are handled by a shared `oauth_token_provider` component. +### 3.2 Constraints #### No Secret Logging - [ ] `p1` - **ID**: `cpt-cf-credstore-constraint-no-secret-logging` -Secret values MUST NOT appear in any log output, error messages, or debug traces. The `SecretValue` type implements `Debug` and `Display` with redacted output. +Secret values MUST NOT appear in any log output, error messages, or debug +traces. `SecretValue` implements redacting `Debug`/`Display` and zeroizes on +drop; request/response DTOs carry hand-written redacted `Debug`; the `GET` +response sets `Cache-Control: no-store`; non-UTF-8 values are rejected with a +typed error rather than lossily decoded. + +#### Canonical Error Model + +- [ ] `p1` - **ID**: `cpt-cf-credstore-constraint-canonical-errors` + +All trait-boundary and REST errors follow the platform canonical error model +([ADR 0005](../../../docs/arch/errors/ADR/0005-cpt-cf-adr-sdk-canonical-projection.md)): +domain errors map to canonical categories with stable `reason` +codes (e.g. `OPTIMISTIC_LOCK_FAILURE`), and the wire strips internal +diagnostics. ## 4. Technical Architecture ### 4.1 Domain Model -**Technology**: Rust structs +**Technology**: Rust structs (`#[domain_model]`) **Core Entities**: | Entity | Description | |--------|-------------| -| `SecretRef` | Human-readable key identifying a secret (e.g., `partner-openai-key`). **Format**: `[a-zA-Z0-9_-]+`, max 255 chars. Colons prohibited to prevent ExternalID collisions. | -| `SecretValue` | Opaque byte wrapper for decrypted secret data. Custom `Debug`/`Display` that redacts content. | -| `SharingMode` | Enum: `Private`, `Tenant` (default), `Shared` — controls access scope within tenant hierarchy | -| `OwnerId` | UUID identifying the creator (from `SecurityContext.subject_id()`) — used for owner-only access control in `Private` mode | -| `SecretMetadata` | Struct containing secret value and access control metadata: `{ value: SecretValue, owner_id: OwnerId, sharing: SharingMode, owner_tenant_id: TenantId }` | - -**Relationships**: -- A Secret belongs to exactly one Tenant (via `tenant_id`) -- A Secret has exactly one Owner (via `owner_id`) — the actor that created it -- **Uniqueness**: For `tenant` and `shared` modes, `(tenant_id, reference)` is unique — one non-private secret per key per tenant. For `private` mode, `(tenant_id, reference, owner_id)` is unique — each owner can have their own private secret with the same reference. A tenant can simultaneously hold one tenant/shared secret and multiple private secrets (one per owner) under the same reference. -- `ResolveResult` references the owning tenant (which may differ from the requesting tenant) - -**Sharing Mode Access Control**: -- **`Private`**: Secret metadata includes `owner_id` (populated from `SecurityContext.subject_id()`). Access checks verify `owner_id == current_subject_id` AND tenant match. -- **`Tenant`**: `owner_id` is stored for audit trail but not enforced during access checks. Any user/service in the owning tenant can access. -- **`Shared`**: `owner_id` is stored for audit trail. Hierarchical tenant resolution applies (current behavior). +| `SecretRef` | Validated secret reference key (e.g., `partner-openai-key`). **Format**: `[a-zA-Z0-9_-]+`, 1–255 chars; validated on construction and re-validated by a DB `CHECK`. | +| `SecretValue` | Opaque byte wrapper (`Vec`) for secret data. Redacting `Debug`/`Display`, zeroize-on-drop, deliberately not `Serialize`/`Deserialize`. | +| `SharingMode` | Enum: `Private`, `Tenant` (default), `Shared` — controls access scope within the tenant hierarchy. | +| `OwnerId` | UUID identifying the creator (`SecurityContext.subject_id()`) — access control key for `Private` mode. | +| `SecretStatus` | Lifecycle status of the metadata row: `Provisioning` (1), `Active` (2), `Deprovisioning` (3) (§6). Only `Active` rows are visible to resolution. | +| `SecretRow` | Metadata row: `{ id, tenant_id, reference, sharing, owner_id, status, version, value_fp, fp_key_id }`. `value_fp` is the internal value-fingerprint fence (§4.10), never serialized to the wire. | +| `NewSecret` | Insert shape for the create saga (always carries the fence fingerprint of the value being written). | +| `WritePrecondition` | Parsed `If-Match`: `Exists` (`*`) or `Version { id, version }` (quoted `"."`, generation-bound). | +| `GetSecretResponse` | SDK read result: `{ value, id, owner_tenant_id, sharing, is_inherited, version }`; `(id, version)` is the strong-validator pair. | +| `SecretType` | Catalog-resolved secret type binding the enforceable traits (§5); immutable per secret. | + +**Relationships & uniqueness**: + +- A secret belongs to exactly one tenant (`tenant_id`) and has exactly one owner (`owner_id`). +- For `tenant`/`shared` modes, `(tenant_id, reference)` is unique; for `private` mode, `(tenant_id, reference, owner_id)` is unique. Both are enforced as **partial unique indexes** (§4.7), which lets one private secret per owner and one tenant/shared secret **coexist** under the same reference. +- The uniqueness indexes ignore `status`, so an in-flight (`provisioning`) row holds the reference; failed sagas are rolled back or reaped to un-wedge it. + +**Sharing-mode access control** (evaluated during SQL resolution): + +| Mode | Visible to | Inherited by descendants? | +|------|-----------|---------------------------| +| `private` | Only where `owner_id` equals the caller's subject id; wins over non-private at the same tenant level | No | +| `tenant` (default) | Only the owning tenant | No | +| `shared` | The owning tenant **and** its descendants, bounded by isolation barriers (§4.6) | Yes (barrier-bounded) | + +`sharing` is a **visibility** mode (who may read the secret), not a quota/limit +that composes as `min(parent, child)`; resolution picks the closest accessible +secret up the ancestor chain, which is how a child tenant *shadows* a parent's +`shared` secret under the same reference. ### 4.2 Component Model ```mermaid graph TB - Consumer[Consumers
OAGW, Platform Gears] - SDK[credstore-sdk
traits + models] - GW[credstore
authz + routing] - AP[credstore_vendor_a_plugin
Credstore REST] - OSP[os_protected_storage
OS Keychain/DPAPI] - CSP[credentials_storage
Rust microservice] - CS[VendorA Credstore
Go service] - OS[OS Keychain] - PG[PostgreSQL] - KMS[External Key Service
Vault / KMS] - TR[tenant_resolver] - TReg[types_registry] + Consumer[Consumers
OAGW, mini-chat, gears] + SDK[credstore-sdk
traits + models + GTS] + GW[credstore gear
service / saga / reaper] + REPO[(credstore_secrets
SecureORM, Scopable)] + SP[static-credstore-plugin
in-memory value store] + PDP[authz-resolver
PolicyEnforcer / PDP] + TR[tenant-resolver] + TReg[types-registry] Consumer -->|ClientHub| SDK SDK --> GW - GW -->|plugin resolution| TReg - GW -->|scoped ClientHub| AP - GW -->|scoped ClientHub| OSP - GW -->|scoped ClientHub| CSP - AP -->|REST + OAuth2| CS - OSP -->|native API| OS - CSP -->|SQL| PG - CSP -->|mTLS| KMS - GW -.->|hierarchy info| TR + GW -->|AccessScope, SQL clamps| REPO + GW -->|GTS instance query| TReg + GW -->|scoped ClientHub| SP + GW -->|ancestor chain, cached| TR + GW -->|scope evaluation| PDP ``` **Components**: - [ ] `p1` - **ID**: `cpt-cf-credstore-component-sdk` -`credstore-sdk` — Trait definitions, models, error types. Interfaces: `CredStoreClientV1`, `CredStorePluginClientV1`. - -- [ ] `p1` - **ID**: `cpt-cf-credstore-component-gateway` - -`credstore` — Authorization, hierarchical resolution (walk-up algorithm), sharing mode enforcement, plugin selection, REST endpoints. Interfaces: Axum routes, ClientHub registration, tenant_resolver queries. +`credstore-sdk` — trait definitions (`CredStoreClientV1`, +`CredStorePluginClientV1`), models, canonical `CredStoreError`, and the GTS +declarations: the plugin spec type +(`gts.cf.toolkit.plugins.plugin.v1~cf.core.credstore.plugin.v1~`) and the +secret resource type (`gts.cf.core.credstore.secret.v1~`, exported as +`SECRET_RESOURCE_TYPE` — the single source of truth pinned by unit tests). -- [ ] `p1` - **ID**: `cpt-cf-credstore-component-vendor-a-plugin` +- [ ] `p1` - **ID**: `cpt-cf-credstore-component-gear` -`credstore_vendor_a_plugin` — VendorA Credstore REST integration (simple per-tenant CRUD). Interfaces: HTTP client, ExternalID mapping. +`credstore` — the stateful gear. Layers: `api/rest` (Axum routes, DTOs +with redacted `Debug`, `If-Match` parsing), `domain` (service with get/put/ +delete sagas, authz scope evaluation, resolver port, metrics port, plugin +selector port), `infra` (SecureORM repo, migrations, tenant-resolver adapter +with TTL+LRU ancestor cache, GTS plugin selector, OTel metrics, canonical +error mapping). Declares `deps = [authz-resolver, tenant-resolver, +types-registry]` and capabilities `system, db, rest, stateful`; its lifecycle +entry runs the reaper loop (§6.4). -- [ ] `p2` - **ID**: `cpt-cf-credstore-component-os-protected-storage` +- [ ] `p1` - **ID**: `cpt-cf-credstore-component-static-plugin` -`os_protected_storage` — OS keychain integration (P2) — simple per-tenant CRUD. Interfaces: Platform-native secure storage APIs. +`static-credstore-plugin` — in-memory per-tenant value store for development +and testing, optionally seeded from YAML config, writable at runtime. +Registers its GTS instance and scoped `CredStorePluginClientV1` in ClientHub. -- [ ] `p1` - **ID**: `cpt-cf-credstore-component-credentials-storage` +- [ ] `p2` - **ID**: `cpt-cf-credstore-component-production-backend` -`credentials_storage` — Standalone Rust microservice providing encrypted credential storage with schema validation, credential definitions, field-level masking, and hierarchical credential propagation (merge resolution). Encryption is handled internally via AES-256-GCM with per-tenant keys managed by a pluggable `KeyProvider` port. Two `KeyProvider` implementations: `DatabaseKeyProvider` (keys in local PostgreSQL, for dev/test) and `ExternalKeyProvider` (keys in external KMS such as HashiCorp Vault or AWS KMS, for production key–data separation). Interfaces: REST API (`/api/credentials-storage/v1/`), JWT authentication, Permission Service integration. Detailed architecture will be documented in `plugins/credentials-storage/DESIGN.md`. +Production value-store backends (external secret vault, OS keychain, +KMS-backed store) — future plugins implementing the same +`CredStorePluginClientV1` contract. Not part of the current codebase. **Interactions**: -- Consumer → Gateway: via `CredStoreClientV1` trait through ClientHub -- Gateway → Plugin: via `CredStorePluginClientV1` trait through scoped ClientHub (GTS instance ID) -- Gateway → tenant_resolver: queries tenant ancestry chain for hierarchical secret resolution walk-up (simple plugins only; `credentials_storage` handles resolution internally) -- VendorA Plugin → Credstore: HTTP REST with OAuth2 bearer token (simple per-tenant CRUD operations) -- VendorA Plugin → OAuth provider: token acquisition and caching -- Credentials Storage Plugin → PostgreSQL: encrypted credential persistence (database-agnostic in future) -- Credentials Storage Plugin → External Key Service: tenant key management via `KeyProvider` (mTLS, when `ExternalKeyProvider` is active) + +- Consumer → Gear: `CredStoreClientV1` via ClientHub (in-process) or REST. +- Gear → Repo: all metadata reads/writes clamped by the PDP `AccessScope` (SecureORM `Scopable`). +- Gear → Plugin: `CredStorePluginClientV1` via scoped ClientHub; the plugin is resolved lazily by GTS instance query filtered by the configured `vendor`. +- Gear → tenant-resolver: ancestor chain (`BarrierMode::Respect`), cached in-process with TTL + LRU eviction. +- Gear → PDP: `PolicyEnforcer.access_scope_with` per operation; capability advertisement depends on `hierarchy.tenant_closure_colocated` (§4.4). ### 4.3 API Contracts -- [ ] `p1` - **ID**: `cpt-cf-credstore-interface-vendor-a-rest` +- [ ] `p1` - **ID**: `cpt-cf-credstore-interface-clienthub` -**Technology**: REST/OpenAPI + Rust traits (ClientHub) +**Technology**: Rust traits (ClientHub) + REST/OpenAPI #### ClientHub API (in-process) -`CredStoreClientV1` trait (public API for consumers): +`CredStoreClientV1` (public consumer API): | Method | Signature | Description | |--------|-----------|-------------| -| `get` | `(ctx: &SecurityCtx, key: &SecretRef) → Result>` | Retrieve secret with metadata (value, owner_tenant_id, sharing, is_inherited) | -| `put` | `(ctx: &SecurityCtx, key: &SecretRef, value: SecretValue, sharing: SharingMode) → Result<()>` | Create or update secret with sharing mode | -| `delete` | `(ctx: &SecurityCtx, key: &SecretRef) → Result<()>` | Delete own secret | +| `get` | `(ctx: &SecurityContext, key: &SecretRef) → Result, CredStoreError>` | Hierarchical read. `Ok(None)` covers both "does not exist" and "inaccessible" (single 404 surface, anti-enumeration). | +| `put` | `(ctx, key, value: SecretValue, sharing: SharingMode) → Result<(), CredStoreError>` | Upsert within the target sharing class. | +| `create` | `(ctx, key, value, sharing) → Result<(), CredStoreError>` | Create-only; `Conflict` if a secret of the same sharing class exists (the 409 path behind REST `POST`). | +| `delete` | `(ctx, key) → Result<(), CredStoreError>` | Delete the caller's own-tenant secret. | -`CredStorePluginClientV1` trait (backend adapter interface): +`CredStorePluginClientV1` (backend SPI — pure value store): | Method | Signature | Description | |--------|-----------|-------------| -| `get` | `(ctx: &SecurityCtx, tenant_id: &TenantId, key: &SecretRef, owner_id: Option<&OwnerId>) → Result>` | Get secret from backend. If `owner_id` is `Some`, looks up the private secret for that owner; if `None`, looks up the tenant/shared secret. | -| `put` | `(ctx: &SecurityCtx, tenant_id: &TenantId, key: &SecretRef, value: SecretValue, sharing: SharingMode, owner_id: OwnerId) → Result<()>` | Store secret in backend. ExternalID is derived from sharing mode and owner_id (see ExternalID Mapping). | -| `delete` | `(ctx: &SecurityCtx, tenant_id: &TenantId, key: &SecretRef, owner_id: Option<&OwnerId>) → Result<()>` | Delete secret from backend. If `owner_id` is `Some`, deletes the private secret for that owner; if `None`, deletes the tenant/shared secret. | - -**SecretMetadata structure**: -```rust -struct SecretMetadata { - value: SecretValue, - owner_id: OwnerId, - sharing: SharingMode, - owner_tenant_id: TenantId, // Tenant that owns this secret -} -``` - -**Design Rationale**: The Plugin must return metadata (owner_id, sharing, owner_tenant_id) so the Gateway can: -1. Enforce sharing mode rules during hierarchical resolution (tenant vs shared access checks) -2. Populate response metadata (`is_inherited`, `owner_tenant_id`) for clients - -For `private` secrets, owner match is guaranteed by ExternalID construction (owner_id is baked into the ExternalID), so no additional access check is needed. For `tenant`/`shared` secrets, the Gateway uses the returned metadata to enforce access control. - -#### 4.3.1 REST API (Gateway) - -| Method | Path | Description | Stability | -|--------|------|-------------|-----------| -| `POST` | `/credstore/v1/secrets` | Create secret with sharing mode | stable | -| `PUT` | `/credstore/v1/secrets/{ref}` | Update secret value and/or sharing mode | stable | -| `GET` | `/credstore/v1/secrets/{ref}` | Get own secret value | stable | -| `DELETE` | `/credstore/v1/secrets/{ref}` | Delete own secret | stable | - -**Create Secret Request:** +| `get` | `(ctx, tenant_id, key, owner_id: Option<&OwnerId>) → Result, CredStoreError>` | `owner_id = Some` selects the owner's private key class; `None` the tenant key class. Returns the value only. | +| `put` | `(ctx, tenant_id, key, value, owner_id: Option<&OwnerId>) → Result<(), CredStoreError>` | Store the value for the addressed key class. | +| `delete` | `(ctx, tenant_id, key, owner_id: Option<&OwnerId>) → Result<(), CredStoreError>` | Delete the value; `NotFound` is treated as success by the gear (idempotent). | + +**Design rationale**: the plugin returns **no metadata** — sharing, ownership, +inheritance, and version all come from the gear's metadata row resolved +*before* the backend is touched. This keeps every policy decision in one +place and backends trivially simple. + +#### 4.3.1 REST API (Gear) + +Routes are registered under `/credstore/v1` (served behind the platform API +gear under its public prefix, e.g. `/cf/credstore/v1/...`). All routes are +authenticated; errors use the canonical `Problem` envelope. + +The machine-readable API is generated from the handlers: the platform-wide +OpenAPI document lives at [`docs/api/api.json`](../../../docs/api/api.json) +(regenerate with `make openapi`), and a credstore-scoped rendering is at +[`openapi.yaml`](./api/openapi.yaml). + +| Method | Path | Success | Description | +|--------|------|---------|-------------| +| `POST` | `/credstore/v1/secrets` | `201` + `Location` | Create-only; `409` if the same sharing class already holds the reference | +| `PUT` | `/credstore/v1/secrets/{ref}` | `204` | Upsert; honours `If-Match` | +| `GET` | `/credstore/v1/secrets/{ref}` | `200` + `ETag`, `Cache-Control: no-store` | Hierarchical read | +| `DELETE` | `/credstore/v1/secrets/{ref}` | `204` | Delete own secret; honours `If-Match` | + +**Create Secret Request (`POST`)**: ```json { "reference": "partner-openai-key", "value": "demo-secret-value-123", - "sharing": "tenant" + "sharing": "tenant", + "type": "gts.cf.core.credstore.secret.v1~cf.core.credstore.api_key.v1~" } ``` -**Sharing mode values**: `"private"` (owner-only), `"tenant"` (tenant-wide, default), `"shared"` (hierarchical) +**Update Secret Request (`PUT`)** — same body without `reference`. +`sharing` values: `"private"`, `"tenant"` (default), `"shared"`. +`type` (optional): the secret type's **full GTS type id** (§5.5), built-in or +custom; defaults to the generic type. `expires_at` (optional, RFC 3339) for +expirable types. -**Get Secret Response (200 OK):** +**Field notes**: + +- `reference` is a **caller-chosen opaque label**, not a GTS id — `^[A-Za-z0-9_-]+$`, 1–255 chars. It is deliberately free-form because it is user-facing (a human-picked name such as `partner-openai-key`); requiring a GTS identifier would be too restrictive. Uniqueness is per sharing class within one tenant (§4.7). +- `value` is carried over REST as a **UTF-8 JSON string**, but is stored as raw **bytes** end-to-end, so binary secrets (e.g. a DER certificate or a raw key) are supported — for secret types whose `utf8_only` trait is `false` (currently `generic`) and via the **SDK/ClientHub**, which carries bytes. Over REST a binary value must be text-encoded by the caller (e.g. PEM, base64). A PEM certificate is UTF-8 text and works on any type; see the `utf8_only` trait (§5.3). + +**Get Secret Response (200)**: ```json { "value": "demo-secret-value-456", "metadata": { - "owner_tenant_id": "partner-acme", + "owner_tenant_id": "22222222-2222-2222-2222-222222222222", "sharing": "shared", - "is_inherited": true + "is_inherited": true, + "version": 3, + "type": "gts.cf.core.credstore.secret.v1~cf.core.credstore.api_key.v1~" } } ``` -**Response Metadata Fields**: -- `owner_tenant_id`: The tenant that owns this secret (may differ from requesting tenant if inherited) -- `sharing`: The sharing mode (`private`, `tenant`, `shared`) -- `is_inherited`: `true` if secret was retrieved from an ancestor via hierarchical resolution, `false` if owned by requesting tenant - -**Use case**: Child tenants can see that a secret is inherited and can be shadowed by creating their own secret with the same reference. - -**Update Secret Request:** -```json -{ - "value": "updated-demo-value-789", - "sharing": "shared" -} -``` - -**Error Responses:** - -| Status | Error Type | Scenario | -|--------|-----------|----------| -| 401 | Unauthorized | Invalid or missing token | -| 403 | AccessDenied | Insufficient permissions (`Secrets:Read` or `Secrets:Write` missing) | -| 404 | NotFound | Secret not found OR inaccessible (owner mismatch, private/tenant scope mismatch, not in hierarchy). **Security**: Always return 404 for inaccessible secrets to prevent enumeration attacks. | -| 409 | Conflict | Secret with this reference already exists within the same scope (POST create-only endpoint). Private secrets are scoped per-owner, so different owners never conflict. | -| 500 | InternalError | Backend or encryption errors | +- `owner_tenant_id`: tenant that owns the resolved secret (differs from the requesting tenant when inherited) +- `is_inherited`: `true` when resolved from an ancestor +- `version`: monotonic per-generation version; the strong `ETag` is the + generation-bound pair `"."` (the row UUID is minted fresh + for every recreated secret, so validators never repeat across + delete+recreate — no ABA lost update) +- `type`: the secret type's full GTS type id (plus `expires_at` when set) + +**Optimistic concurrency**: `PUT` and `DELETE` accept `If-Match: *` (target +must exist) or `If-Match: "."` (both the generation id and the +version must match the current row). The precondition is checked before the +backend write *and* re-enforced as a `version = ?` SQL filter on the metadata +commit (the id is already the UPDATE key). A mismatch — including a validator +minted for an earlier generation of a recreated secret — surfaces as +canonical `Aborted` / **409** with reason `OPTIMISTIC_LOCK_FAILURE` (the +canonical model has no 412; 409 is the deliberate platform-correct status). +A malformed `If-Match` (including a bare quoted version) is a 400; a +precondition on a non-existent target is a 409. + +**Error Responses**: + +| Status | Canonical category | Scenario | +|--------|--------------------|----------| +| 400 | `InvalidArgument` | Invalid `reference` format, malformed `If-Match`, unsupported sharing transition (private ↔ tenant/shared) | +| 401 | `Unauthenticated` | Invalid or missing token | +| 403 | `AccessDenied` | PDP denies the action, or the caller's scope excludes their own tenant | +| 404 | `NotFound` | Secret not found **or** inaccessible (anti-enumeration: always 404, never 403, for per-secret access) | +| 409 | `AlreadyExists` / `Aborted` | `POST` conflict; `If-Match` version conflict (`OPTIMISTIC_LOCK_FAILURE`) | +| 503 | `ServiceUnavailable` | PDP evaluation failure, types-registry outage / stored secret type unresolvable (§5.4), no storage plugin registered (with stable detail, no `retry_after`), backend outage (with `Retry-After` when hinted) | +| 500 | `Internal` | Invariant violations, non-UTF-8 stored value on `GET` (binary written via the SDK cannot cross the JSON transport); diagnostic stripped from the wire | ### 4.4 External Interfaces & Protocols -#### VendorA Credstore REST API - -- [ ] `p1` - **ID**: `cpt-cf-credstore-design-interface-vendor_a-rest` - -**Type**: External System - -**Direction**: outbound - -**Data Format**: JSON over HTTP/REST - -**Authentication**: OAuth2 client credentials (token acquired via shared `oauth_token_provider`) - -**Endpoints used:** - -| Operation | Credstore Endpoint | Notes | -|-----------|-------------------|-------| -| Read | `GET /credentials/{external_id}?tenant_id={tid}&include_secret=true` | Returns secret value. 404 → `None` | -| Write (update) | `PUT /credentials/{external_id}/identity_and_secret?tenant_id={tid}` | Updates secret value. If 404, fall through to create | -| Write (create) | `POST /credentials?tenant_id={tid}` | Body includes `id=external_id`, `secret=`, `sharing` field, `owner_id` field | -| Delete | `DELETE /credentials/{external_id}?tenant_id={tid}` | Removes credential | - -**Sharing Mode Field**: The `sharing` field is stored in Credstore backend as metadata. Values: `private` (owner-only), `tenant` (tenant-wide), `shared` (hierarchical). The Gateway reads this field and enforces access control during hierarchical resolution. The Backend provides simple per-tenant storage without hierarchical logic. - -**Owner ID Field**: The `owner_id` field (UUID) is stored in Credstore backend and identifies the creator (from `SecurityContext.subject_id()`). For `private` mode, access checks must verify `owner_id` matches the requesting subject. For `tenant` and `shared` modes, `owner_id` is stored for audit trail but not enforced in access checks. - -**Compatibility Note**: VendorA Credstore backend schema must support: -1. Three-value `sharing` enum: `private`, `tenant`, `shared` -2. `owner_id` UUID field for creator identification -The VendorA Credstore backend must support these fields before deployment. Backend schema/feature implementation is a prerequisite for launching the credstore gear. - -**ExternalID Mapping:** -``` -# Tenant/Shared secrets (one per tenant+key): -raw = "{tenant_id}:{key}" -external_id = base64url_no_pad(raw) + "@secret" - -# Private secrets (one per tenant+key+owner): -raw = "{tenant_id}:{key}:p:{owner_id}" -external_id = base64url_no_pad(raw) + "@secret" -``` - -The plugin derives the ExternalID variant from the `sharing` mode (on `put`) or from the `owner_id` parameter (on `get`/`delete`): `Some(owner_id)` → private variant, `None` → tenant/shared variant. - -This deterministic, stateless mapping avoids maintaining a local mapping database and achieves idempotent operations. - -**Collision Prevention**: SecretRef format is constrained to `[a-zA-Z0-9_-]+` (no colons) to prevent collisions. Tenant_id is a UUID (no colons), owner_id is a UUID (no colons), and colons serve as delimiters. The `:p:` segment distinguishes private ExternalIDs from tenant/shared ones, ensuring each `(tenant_id, key, scope)` tuple maps to a unique ExternalID. - -**Compatibility**: Plugin adapts to Credstore API version. ExternalID format must remain stable across versions to avoid breaking lookups. - -#### External Key Management Service - -- [ ] `p1` - **ID**: `cpt-cf-credstore-design-interface-external-kms` - -**Type**: External System - -**Direction**: outbound (from Credentials Storage plugin) - -**Purpose**: Tenant encryption key storage and lifecycle when `ExternalKeyProvider` is active in the Credentials Storage plugin. Provides key–data separation for production security posture — encryption keys are stored in a separate security domain from encrypted credentials. - -**Protocol**: HTTPS/mTLS (Vault HTTP API, AWS KMS API, or custom REST/gRPC) - -**Authentication**: Service-specific — Vault token, Kubernetes ServiceAccount, IAM role. Credentials for the key service are injected via Kubernetes Secret and never stored in the application database. - -**Error Handling**: Key service unavailability blocks all encrypt/decrypt operations in the Credentials Storage plugin. Readiness probe reflects KMS connectivity. Circuit breaker pattern for key service calls. - -**Deployment note**: Required only when the `credentials_storage` plugin is active with `ExternalKeyProvider`. Not applicable to VendorA or OS keychain plugins. - -#### Sharing Mode Transitions (Constraints & Plugin Capabilities) - -The API exposes `sharing` as a field that can be set on `put` / `update`, but not all "sharing transitions" are equivalent at the storage layer. - -Because secret identity differs between private and non-private secrets (via ExternalID mapping), transitions fall into two classes: - -- **`tenant` ↔ `shared` (non-private)**: Same ExternalID (`{tenant_id}:{key}`) and can be implemented as an in-place metadata update *if* the backend supports updating `sharing` on an existing record. -- **`private` ↔ (`tenant`/`shared`)**: Different ExternalIDs (`{tenant_id}:{key}:p:{owner_id}` vs `{tenant_id}:{key}`), so a "conversion" cannot be a single in-place update. A portable implementation requires creating a new record in the target scope and (optionally) deleting the old one; this is not atomic. - -This has two practical implications: - -- **Plugin/backend capability**: Whether a given backend can support a specific transition (especially private ↔ non-private) is a plugin/backing-store capability and may be restricted in v1. -- **No implicit migration guarantee**: If a client updates `sharing` across the private/non-private boundary, the system MUST NOT assume an atomic "move". Implementations should document whether they perform a best-effort two-step migrate (create + delete) or instead reject such transitions. +#### PDP (authz-resolver) + +- [ ] `p1` - **ID**: `cpt-cf-credstore-design-interface-pdp` + +**Type**: platform service (in-process client via ClientHub) + +Every operation calls `PolicyEnforcer.access_scope_with(ctx, resource, +action, …)` **once**, with the `owner_tenant_id` PEP property and +`action ∈ {read, write, delete}`. The `resource` is always the secret's +**full concrete type** — including `generic` +(`…secret.v1~cf.core.credstore.generic.v1~`) — so policies can target any +type without a separate base-type gate (§5.4). The type is known before the +evaluation via a prefetch (post-resolution on read, post-lookup on +overwrite/delete, from the requested/default type on create) followed by a +types-registry resolution of the stored `secret_type_uuid` to its GTS type +id (§5.4); the returned `AccessScope` is enforced in SQL. Enforcement is +fail-closed: `Denied`/`CompileFailed` → 403 (404 on read, +anti-enumeration), `EvaluationFailed` → 503. + +**Capability advertisement** (config `hierarchy.tenant_closure_colocated`): +when the shared tenant-closure table is co-located with the credstore +database, the gear advertises `Capability::TenantHierarchy` and the PDP emits +a structured `InTenantSubtree` predicate resolved by a closure subquery. +Otherwise (default) the PDP pre-expands the subtree into a flat `In` list the +gear enforces with no local closure access (degraded but correct mode). + +#### Tenant hierarchy (tenant-resolver) + +- [ ] `p1` - **ID**: `cpt-cf-credstore-design-interface-tenant-resolver` + +The gear fetches the requesting tenant's ancestor chain (self first, root +last) with `BarrierMode::Respect`, so a `shared` secret is **not** inherited +across a `self_managed` isolation barrier. The chain carries no +caller-specific data and is cached in-process with a TTL +(`hierarchy.ancestor_cache_ttl_secs`, default 300 s) and LRU eviction. + +#### Secret-type resolution & plugin discovery (types-registry) + +- [ ] `p1` - **ID**: `cpt-cf-credstore-design-interface-gts` + +The types-registry is a hard `deps` of the gear (fail-closed `init` when +`TypesRegistryClient` is absent from ClientHub) and serves two roles: + +**Secret-type resolution** (§5): every operation resolves the secret's +stored type UUID via `get_type_schema_by_uuid` — one lookup against the +registry client's built-in TTL cache; credstore adds no cache of its own, +so type re-registrations take effect within the client TTL. The resolver +(`GtsSecretTypeResolver`, mirroring AM's `GtsTenantTypeChecker`) verifies +the schema descends from the secret base type +(`gts.cf.core.credstore.secret.v1~`), merges the chain's effective traits +(`x-gts-traits`, leaf wins, base fills defaults), and deserializes them +into `SecretTypeTraits`. Failure mapping is fail-closed: an +unregistered/non-secret type is `UNKNOWN_SECRET_TYPE` (400) when the caller +named it, 503 when it came from a stored row (deregistration is an +operational inconsistency, not a caller error); registry outage / timeout +(2 s probe) / malformed traits → 503. Calls are recorded on the +`types_registry` dependency-health metrics. + +**Plugin discovery**: plugins register GTS instances derived from the +plugin spec type (`…~cf.core.credstore.plugin.v1~`). The gear lazily +queries instances by type-id prefix, filters by the configured `vendor`, +picks the highest-priority active instance, and resolves its scoped +`CredStorePluginClientV1` from ClientHub. No plugin available surfaces as a +non-retryable 503 ("no storage plugin registered"). + +#### Sharing-mode transitions + +`tenant ↔ shared` is an in-place metadata update (same unique-index class). +`private ↔ (tenant|shared)` crosses key classes; since private and non-private +secrets coexist by design, such a "transition" has no atomic meaning and is +**rejected** as `UnsupportedTransition` (400) rather than performed +non-atomically. Writes always address the row of their own sharing class, so +a write of one class never affects the other. ### 4.5 Service-to-Service Pattern -#### Architecture Pattern - -The CredStore Gateway supports two distinct integration patterns: - -1. **Self-Service Pattern**: Platform gears and tenant admins retrieve secrets for their own tenant. The tenant_id is derived from SecurityCtx. -2. **Service-to-Service Pattern**: Authorized service accounts (e.g., OAGW) retrieve secrets on behalf of arbitrary tenants by constructing an explicit SecurityCtx with the target tenant_id. - -#### Service-to-Service Flow (OAGW Example) - -**Actor**: `cpt-cf-credstore-actor-oagw` (Outbound API Gateway) - -**Use Case**: OAGW needs to retrieve a partner's shared API key when making an upstream call on behalf of a customer tenant. +Two integration patterns share one API: -**Flow**: -1. **SecurityCtx Construction**: OAGW constructs a SecurityCtx with: - - `tenant_id`: Target tenant (e.g., `customer-123`) - - `subject_id`: OAGW service account ID - - `permissions`: `Secrets:Read` permission for the service account -2. **Gateway Invocation**: OAGW calls Gateway's standard `get(ctx, key)` operation (same API as self-service) -3. **Authorization**: Gateway verifies: - - Service account has `Secrets:Read` permission - - Service account is authorized to construct SecurityCtx with arbitrary tenant_id (service-level authorization) -4. **Hierarchical Resolution**: Gateway extracts `tenant_id` from SecurityCtx and performs hierarchical walk-up: - - Queries `tenant_resolver` for ancestor chain - - Walks up hierarchy calling Plugin `get` at each level - - Checks sharing mode and owner_id for access control - - Returns first accessible secret -5. **Response**: Gateway returns secret value and metadata to OAGW +1. **Self-Service**: gears and tenant admins operate on their own tenant; tenant and owner derive from their `SecurityContext`. +2. **Service-to-Service**: authorized service accounts (OAGW) act on behalf of arbitrary tenants by constructing a `SecurityContext` for the target tenant (S2S client-credentials exchange) and calling the same `get`. -**Key Differences from Self-Service**: -- **Tenant Derivation**: Explicit tenant_id in SecurityCtx (not derived from authenticated user) -- **Authorization**: Service account must be authorized for cross-tenant access -- **Use Case**: Service acts on behalf of another tenant (delegation pattern) -- **Auditing**: Audit trail must record both service account ID and target tenant_id +**Flow (OAGW)**: OAGW builds a `SecurityContext` with the target tenant, calls +`get(ctx, key)`; the PDP decides whether that subject may `read` secrets in +that tenant's scope; resolution and metadata behave identically to +self-service. Audit/metrics record the caller subject and target tenant. +OAGW consumes credstore only through the SDK client — there is no separate +integration path. -**Design Rationale**: -- **Unified API**: Both patterns use the same Gateway `get` operation (no separate service-to-service endpoint) -- **Security**: Service authorization is enforced at Gateway layer before hierarchical resolution -- **Transparency**: Hierarchical resolution is identical for both patterns (implemented in Gateway) -- **Flexibility**: Service accounts can retrieve secrets for any tenant they're authorized to access - -**Implementation Note**: OAGW is a ToolKit gear that uses the standard CredStore SDK client. There is no separate integration path or direct backend access. All operations flow through Gateway→Plugin→Backend. +**Provisioning note**: with the stateful gear a provider's backend secret +may not exist at startup (secrets are created at runtime through the +credstore API; there is no startup seed). Consumers that provision +infrastructure from secrets at boot (e.g. mini-chat registering OAGW +upstreams) treat a failed secret lookup as non-fatal: the affected provider +is skipped and remains unavailable until its secret is provisioned. ### 4.6 Interactions & Sequences -#### Self-Service CRUD (put example) +#### Hierarchical read -- [ ] `p1` - **ID**: `cpt-cf-credstore-seq-self-service-crud` +- [ ] `p1` - **ID**: `cpt-cf-credstore-seq-hierarchical-read` ```mermaid sequenceDiagram - participant T as Tenant / Gear + participant C as Consumer participant GW as credstore + participant TR as tenant-resolver + participant DB as credstore_secrets + participant GTS as types-registry + participant PDP as authz-resolver participant P as Plugin - participant B as Backend - - T->>GW: put(ctx, "my-key", value, shared) - GW->>GW: Check Secrets:Write permission - GW->>GW: Extract tenant_id from SecurityCtx - GW->>P: put(tenant_id, "my-key", value, shared) - P->>B: Store secret - B-->>P: OK - P-->>GW: OK - GW-->>T: OK -``` - -#### Write Flow (VendorA Credstore — upsert) - -- [ ] `p1` - **ID**: `cpt-cf-credstore-seq-vendor-a-write` -```mermaid -sequenceDiagram - participant P as credstore_vendor_a_plugin - participant CS as VendorA Credstore - - P->>CS: PUT /credentials/{ext_id}/identity_and_secret?tenant_id={tid}
Body: {secret, sharing} - alt Secret exists - CS-->>P: 200 OK (updated) - else Secret not found - CS-->>P: 404 Not Found - P->>CS: POST /credentials?tenant_id={tid}
{id: ext_id, secret: base64(value), sharing} - CS-->>P: 201 Created - end + C->>GW: get(ctx, key) + GW->>TR: ancestor_chain(tenant) [cached, barrier-respecting] + TR-->>GW: [self, parent, ..., root] + GW->>DB: resolve_for_get(reference, chain, subject) — one query + DB-->>GW: winning row (or none → 404, no PDP call) + GW->>GTS: get_type_schema_by_uuid(secret_type_uuid) [client TTL cache] + GTS-->>GW: type id + effective traits + GW->>PDP: access_scope(read, concrete type) + PDP-->>GW: AccessScope + GW->>DB: scope_includes_tenant(caller tenant)? + GW->>P: get(owner_tenant, key, owner?) — value only + P-->>GW: SecretValue + GW-->>C: value + {owner_tenant_id, sharing, is_inherited, version, type} ``` -**Key Flows**: Reference use cases from PRD: -- `cpt-cf-credstore-usecase-create-shared` → Self-Service CRUD (with `sharing: shared`) -- `cpt-cf-credstore-usecase-crud` → Self-Service CRUD -- `cpt-cf-credstore-usecase-hierarchical-resolve` → Hierarchical resolution implemented in Gateway gear - -**Hierarchical Resolution Implementation**: The Gateway gear implements a two-phase walk-up algorithm: -1. Extract `tenant_id` and `subject_id` from SecurityCtx -2. Query `tenant_resolver` to get ancestor chain (child → parent → ... → root) -3. For each tenant in the chain (starting from requesting tenant), perform two-phase lookup: - - **Phase 1 — Private**: Call Plugin `get(ctx, tenant_id, key, Some(subject_id))` to look up a private secret for this owner - - If found and `sharing == private`: owner match is guaranteed by ExternalID construction → return secret value - - **Phase 2 — Tenant/Shared**: Call Plugin `get(ctx, tenant_id, key, None)` to look up the tenant/shared secret - - If found, check `sharing` mode for access control: - - **`tenant` mode**: Check `metadata.owner_tenant_id == ctx.tenant_id()` - - **`shared` mode**: Allow if requester is descendant or same tenant as owner_tenant_id - - If accessible, return secret value + response metadata (owner_tenant_id, sharing, is_inherited) - - If neither phase yields an accessible secret, continue to next ancestor -4. If no accessible secret found in entire chain, return NotFound - -**Two-Phase Rationale**: Private secrets are stored under a separate ExternalID (`{tenant_id}:{key}:p:{owner_id}`), so a single `get` call cannot return both private and tenant/shared secrets. The Gateway always tries the caller's private secret first (phase 1), then falls back to the tenant/shared secret (phase 2). This costs at most 2 plugin calls per tenant in the chain. - -**Shadowing Semantics**: When the requesting tenant has secrets with the same reference: -- **Private secret exists for this owner**: Return it immediately (phase 1 hit) — ancestor never checked -- **No private secret, but tenant/shared secret exists and is accessible**: Return it (phase 2 hit) — ancestor never checked -- **Neither phase yields an accessible secret**: Continue walk-up to ancestors - -**Example**: User B in tenant X requests `key1`: -- Tenant X has `key1` with `sharing: private, owner_id: UserA` (stored under UserA's ExternalID) -- User B has no private `key1` in tenant X -- Tenant X has no tenant/shared `key1` -- Parent Y has `key1` with `sharing: shared` (accessible to descendants) -- Result: Phase 1 for X → miss (no private secret for User B). Phase 2 for X → miss (no tenant/shared). Move to parent Y → phase 2 hit → return parent's shared secret. - -This allows User B to access the parent's shared credential. Meanwhile, User A in tenant X would get their own private `key1` (phase 1 hit). +**Resolution query semantics** (single indexed query over +`idx_credstore_lookup`): filter by `reference`, tenant ∈ ancestor chain, +`status = active`, and sharing-class visibility (`private` rows only for the +caller's `owner_id`; `tenant` rows only for the requesting tenant; `shared` +rows for any chain member). The winner is picked in-process: **closest tenant +wins; `private` beats non-private at the same level**. The backend is read +once, for the winning row only. Walk-up depth and read outcome +(own/inherited/miss) are recorded as metrics. -### 4.7 Database schemas & tables +**Shadowing**: a child's accessible secret always wins over an ancestor's; +an *inaccessible* child secret (e.g. someone else's private) does not block +fallback to an ancestor's shared secret. -The gateway gear has no local database. Secrets are persisted in the external backend (VendorA Credstore or OS keychain). The gateway and the VendorA/OS plugins are stateless. +#### Write (create saga) — see §6.2 -The `credentials_storage` plugin maintains its own database with tables for schemas, credential definitions, credentials (encrypted), and tenant keys (when `DatabaseKeyProvider` is active). The initial implementation uses PostgreSQL; the storage layer is designed to become database-agnostic in future iterations. The full database schema is specified in the plugin design document (`plugins/credentials-storage/DESIGN.md`, planned). +- [ ] `p1` - **ID**: `cpt-cf-credstore-seq-write-saga` -### 4.8 Deployment Topology +The full step-by-step sequence, failure handling, and the overwrite path are +described in §6.2 Provisioning Saga. -**Kubernetes Environment:** +#### Delete (deprovisioning saga) — see §6.3 -```mermaid -graph LR - Platform["Platform
(OAGW, gears)"] --> GW["credstore +
vendor_a plugin"] - GW --> CS["VendorA Credstore
(Go svc)"] - GW --> OAuth["OAuth/OIDC
Provider"] +### 4.7 Database schemas & tables + +The gear owns one table, `credstore_secrets` (migration +`m0001_initial_schema`; raw per-backend SQL to preserve `CHECK` and +partial-index semantics; PostgreSQL and SQLite; MySQL fails fast): + +```sql +CREATE TABLE credstore_secrets ( + id UUID PRIMARY KEY, + tenant_id UUID NOT NULL, + reference TEXT NOT NULL CHECK (length(reference) BETWEEN 1 AND 255), + sharing SMALLINT NOT NULL CHECK (sharing IN (1,2,3)), -- private/tenant/shared + owner_id UUID NOT NULL, + status SMALLINT NOT NULL CHECK (status IN (1,2,3)), -- provisioning/active/deprovisioning + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + version BIGINT NOT NULL DEFAULT 1, + secret_type_uuid UUID NOT NULL DEFAULT '', -- deterministic v5 of the GTS type id + expires_at TIMESTAMPTZ NULL, -- expirable types only + value_fp BYTEA NULL, -- value-fingerprint fence: HMAC-SHA256(fence_key, value); internal-only + fp_key_id SMALLINT NULL, -- fence-key id the fp was computed under (keyring groundwork) + CHECK ((value_fp IS NULL) = (fp_key_id IS NULL)) -- NULL only on out-of-band seeded rows +); +-- coexistence of a private and a tenant/shared secret under one reference: +CREATE UNIQUE INDEX uq_credstore_nonprivate ON credstore_secrets (tenant_id, reference) WHERE sharing <> 1; +CREATE UNIQUE INDEX uq_credstore_private ON credstore_secrets (tenant_id, reference, owner_id) WHERE sharing = 1; +-- walk-up resolution, reaper sweep (all non-active rows), expiry sweep: +CREATE INDEX idx_credstore_lookup ON credstore_secrets (reference, tenant_id, status); +CREATE INDEX idx_credstore_pending ON credstore_secrets (updated_at) WHERE status <> 2; +CREATE INDEX idx_credstore_expiry ON credstore_secrets (expires_at) WHERE expires_at IS NOT NULL AND status = 2; ``` -**Credentials Storage Plugin (standalone microservice):** +Created by the single `m0001_initial_schema` migration (§8). The table +is a `Scopable` SecureORM entity — PDP scope clamps are applied to every +query, which is what makes authorization enforceable in SQL. -```mermaid -graph LR - Platform["Platform
(OAGW, gears)"] --> GW["credstore +
credentials_storage plugin"] - GW --> CSP["Credentials Storage
(Rust svc)"] - CSP --> PG["PostgreSQL
(credentials)"] - CSP -->|"mTLS"| KMS["External Key Service
(Vault / KMS)"] -``` +### 4.8 Deployment Topology -**Desktop/VM Environment (P2):** +The gear is a gear inside the platform process; it requires a database +(PostgreSQL in production, SQLite in dev/e2e) via the platform DB provider, +and exactly one registered value-store plugin. ```mermaid graph LR - Platform["Platform
(gears)"] --> GW["credstore +
os_storage plugin"] - GW --> OS["OS Keychain
/ DPAPI"] + Platform["Platform
(OAGW, mini-chat, gears)"] --> GW["credstore gear
+ value-store plugin"] + GW --> DB["PostgreSQL / SQLite
(credstore_secrets)"] + GW --> PDP["authz-resolver"] + GW --> TR["tenant-resolver"] ``` +Development/testing runs the in-memory `static-credstore-plugin`; production +deployments substitute a real vault-backed plugin (future work) with no +gear changes. + ### 4.9 Technology Stack | Layer | Technology | Rationale | |-------|------------|-----------| -| Gateway | Axum (REST), ToolKit gear macro | Platform standard for HTTP services | -| VendorA Plugin | `toolkit-http` (`HttpClient`) | Platform-standard HTTP client with OAuth2 support | -| OAuth2 | Shared `oauth_token_provider` component | Centralized token acquisition and caching | -| OS Plugin (P2) | `keyring` crate or platform-native FFI | Cross-platform OS keychain access | +| Gear | Axum (REST), `#[toolkit::gear]` with `system, db, rest, stateful` capabilities | Platform standard | +| Persistence | SeaORM + SecureORM (`Scopable`), raw-SQL migrations | Scope clamps in SQL; partial unique indexes | +| AuthZ | `authz-resolver-sdk` `PolicyEnforcer` (PDP) | Platform policy plane | +| Hierarchy | `tenant-resolver-sdk` (`BarrierMode::Respect`) + TTL/LRU cache | Barrier-aware ancestor chains | +| Discovery & typing | `toolkit-gts` / `types-registry-sdk` | Vendor-based plugin selection; runtime secret-type resolution (uuid → type id + traits) | +| Observability | OpenTelemetry metrics (typed port) | Operational visibility | | Serialization | `serde` | Platform standard | -| Errors | `thiserror` | Platform standard | - -## 5. Risks / Trade-offs - -### 5.1 Architectural Trade-offs - -#### Hierarchical Resolution in Gateway (for simple plugins) - -**Decision**: Implement hierarchical walk-up algorithm in Gateway gear for simple plugins (VendorA Credstore, OS keychain). The `credentials_storage` plugin handles merge resolution internally. - -**Trade-offs**: -- ✅ **Pro**: Centralized policy enforcement for simple plugins — all sharing mode logic and access control is in one place -- ✅ **Pro**: Simple backends remain simple — just per-tenant key-value storage, easier to maintain and test -- ✅ **Pro**: Plugin portability — OS keychain plugin doesn't need to understand hierarchy -- ✅ **Pro**: Easier to change hierarchy logic without backend changes -- ❌ **Con**: Multiple backend calls during walk-up (N calls for N-deep hierarchy) — not applicable to `credentials_storage` which resolves internally -- ❌ **Con**: Gateway must query tenant_resolver for hierarchy information (simple plugins only) - -**Mitigation**: Cache tenant hierarchy queries in Gateway; implement early termination on first accessible secret. The `credentials_storage` plugin avoids this overhead by resolving the merge chain in a single service call. - -#### Three-Tier Sharing Model (not RBAC/ABAC) +| Errors | `thiserror` + toolkit canonical errors | Platform standard ([ADR 0005](../../../docs/arch/errors/ADR/0005-cpt-cf-adr-sdk-canonical-projection.md)) | + +### 4.10 Value-Fingerprint Fence + +> Decision: [ADR-0003](./ADR/0003-cpt-cf-credstore-adr-value-fingerprint-fence.md) +> (`cpt-cf-credstore-adr-value-fingerprint-fence`). Closes the cross-tenant +> disclosure of two crosswise concurrent unconditional PUTs and the ABA +> lost-update of a recreated secret. + +A secret is a **dual write**: the value goes to the external value-store +backend (`plugin.put`), the metadata (`sharing`, `version`, `expires_at`) to +`credstore_secrets`. No transaction spans both stores, so two concurrent +last-writer-wins PUTs to the same reference can interleave crosswise and +commit one caller's value under the other caller's sharing label — a durable +cross-tenant disclosure when the surviving label is `shared`. The platform's +coordination primitives deliberately exclude fencing of external effects +([cluster ADR-002](../../system/cluster/docs/ADR/002-async-boundary-no-remote-in-critical-section.md): +no fencing tokens, no remote calls in a lock's critical +section; `coord` leases likewise), so the fence lives at the application +layer, as that ADR prescribes. + +**Mechanism.** Each row stores `value_fp = HMAC-SHA256(fence_key, value)` of +the value its metadata was written for. Every write stamps it in the same +atomic `touch`/insert as `sharing`; every read recomputes it from the value +the backend returned and serves the value only when the row agrees, else +fails closed as an anti-enumeration miss (404). Because the fingerprint and +the sharing label are written by one writer in one UPDATE, a fingerprint +match transitively proves the value and the metadata are from the same PUT — +a value can never be served under a sharing label a different writer set. +Metadata is not hashed into the fingerprint (the atomic write is what binds +them). + +**Fence key.** Deployment state, not configuration: auto-generated (OS RNG, +32 bytes) and stored in the value-store backend under the reserved entry +`(tenant = nil, reference = "cfs-internal-fence-key", owner = None)`. It has +no metadata row, so no API path resolves, overwrites, or deletes it +(resolution requires a row; external callers always carry a real tenant). All +replicas read the one shared entry; on a virgin deployment the first writer +generates it (put-then-reread convergence). The service caches it in-process +and, on a fingerprint mismatch, performs a one-shot re-read (a replica whose +cached key went stale after a key re-creation self-heals before failing +closed). Split knowledge: fingerprints live in the gear DB, the key with +the values — a DB-only compromise yields HMACs under an unknown key (no +offline dictionary attack), and backend compromise yields the plaintexts +regardless, so the fence adds no attack surface. + +**Failure semantics** are always fail-closed and self-healing: a +half-completed overwrite (backend write committed, metadata `touch` failed, +or vice-versa) leaves the row fingerprinting a different value than the +backend holds, so reads 404 until a retried PUT lands both; a lost/replaced +key makes fenced reads 404 (observable via `fence_verify{outcome="mismatch"}`) +and re-PUT re-stamps each secret. No failure mode serves a value under +mismatched metadata. + +**Generation-bound validator.** The strong `ETag` is `"."`. +The row UUID is minted fresh per created secret, so a validator from a +deleted-and-recreated secret's earlier generation never matches the new row +even when the restarted version counters coincide (closing the ABA +lost-update). `version` stays the per-generation optimistic-lock counter and +the row-level CAS still gates on it (the id is already the UPDATE key). + +**Out-of-band seeding.** A row provisioned directly in the DB with +`value_fp = NULL` (value placed directly in the backend) is served on trust +and its fingerprint backfilled — lazily on first read, or by the reaper's +bounded sweep for rows nobody reads (both via a `value_fp IS NULL` CAS that +never bumps the version). A re-seed of an existing row must reset `value_fp` +to NULL in the same operation. This is the only path that produces a NULL +fingerprint; API writes always stamp. + +## 5. Secret Types (GTS-Based, Registry-Driven) + +> Requirement: `cpt-cf-credstore-fr-secret-types`. Implemented: registry +> seeds in the SDK + runtime resolution (`SecretTypeResolver`), gear +> trait enforcement on the resolved traits (incl. the per-type PDP gate), +> UUID type column in `m0001_initial_schema`. + +### 5.1 Concept + +Today every secret is an opaque byte string with identical semantics. The +platform, however, stores materially different kinds of secrets (LLM provider +API keys consumed by OAGW, OAuth2 client credentials, personal tokens, +certificates), and their handling rules differ — most importantly *whether a +secret may be shared down the tenant hierarchy at all*. + +A **secret type** is a GTS type derived from the credstore secret base type +(GTS segments are `vendor.package.namespace.type.vN`, so the derived segment +carries the type name directly), e.g. for the built-in types: -**Decision**: Use simple three-tier sharing modes (private/tenant/shared) instead of granular ACL. - -**Trade-offs**: -- ✅ **Pro**: Simple to understand and reason about -- ✅ **Pro**: Covers 90% of use cases (owner-only keys, team credentials, hierarchical platform creds) -- ✅ **Pro**: Easy to implement and test -- ❌ **Con**: Cannot express complex policies (e.g., "only billing team in tenant can access") -- ❌ **Con**: No role-based access control within tenant - -**Mitigation**: Future enhancement for granular ACL is documented in PRD Open Questions. - -#### 404 for Inaccessible Secrets (not 403) - -**Decision**: Return 404 NotFound for all inaccessible secrets (owner-mismatch, private/tenant scope mismatch, not in hierarchy). - -**Trade-offs**: -- ✅ **Pro**: Prevents enumeration attacks — attackers cannot probe for secret existence -- ✅ **Pro**: Simpler client logic — 404 means "not available" -- ❌ **Con**: Harder to debug — cannot distinguish "doesn't exist" from "no access" -- ❌ **Con**: Less RESTful (REST semantics prefer 403 for permission denied) - -**Mitigation**: Audit logs record access attempts for debugging; error messages to admin users can be more detailed. - -#### PUT-as-Upsert (not POST create + PUT update) - -**Decision**: Use PUT /secrets/{ref} for create-or-update (idempotent upsert). - -**Trade-offs**: -- ✅ **Pro**: Idempotent — clients can retry safely -- ✅ **Pro**: Simpler client usage — no need to check if secret exists first -- ✅ **Pro**: Reduces API surface (one endpoint vs two) -- ❌ **Con**: Less RESTful (POST typically for create, PUT for update) -- ❌ **Con**: Cannot detect accidental overwrites (no "create only" option) - -**Mitigation**: POST /secrets endpoint available for explicit create-with-conflict-detection if needed. - -### 5.2 Security and Performance Risks - -#### Risk: Secret Values Leaked Through Logs - -**Impact**: Critical security incident, compliance violations, potential credential theft - -**Mitigation**: -- `SecretValue` type with redacted `Debug`/`Display` implementation -- Code review enforcement -- Log scrubbing at transport layer -- Automated log scanning in integration tests -- NFR requirement (`cpt-cf-credstore-nfr-confidentiality`) +``` +gts.cf.core.credstore.secret.v1~cf.core.credstore..v1~ +``` -**Likelihood**: Medium | **Impact**: Critical | **Priority**: P1 +Each type declares a set of **traits** — machine-readable behavioral +properties the gear enforces uniformly. The **types-registry is the +runtime source of truth** (mirroring tenant types in Account Management): +the base type `SecretV1` carries the trait vocabulary as its +`x-gts-traits-schema` (generated from `SecretTypeTraits`, closed to unknown +keys), every registered type derived from it declares its `x-gts-traits` +values against that shape, and the gear resolves a type's effective +traits from the registry per operation (§5.4). The compiled-in SDK catalog +(`credstore_sdk::SECRET_TYPE_CATALOG`) only **seeds** the built-in type +schemas through the link-time inventory; unit tests pin the seeds to the +catalog descriptors so the two views cannot drift. + +**Adding a type requires no credstore release**: registering a GTS schema +that descends from `gts.cf.core.credstore.secret.v1~` (with its +`x-gts-traits`) makes the type immediately writable, trait-enforced, and +addressable as a PDP resource type — enabling per-type RBAC (e.g., a role +that may read `api-key` secrets but not `certificate` secrets) without new +authorization machinery. + +The type of a secret is chosen at creation (REST field `type`: the secret +type's full GTS type id), defaults to `generic`, and is **immutable** for the +lifetime of the secret (rejected with `TYPE_IMMUTABLE`, mirroring the +private ↔ non-private rule). + +### 5.2 Type Traits + +| Trait | Type | Semantics (gear-enforced unless noted) | +|-------|------|-------------------------------------------| +| `allow_sharing` | list of `SharingMode` | Sharing modes permitted for secrets of this type. A `put`/`create` with a mode outside the list is rejected (400, `SHARING_NOT_ALLOWED_FOR_TYPE`) — including a disallowed mode change on update. The traits schema constrains entries to the `SharingMode` enum, so a typo fails at registration. | +| `value_schema` | embedded JSON Schema (optional) | Structural validation of the (JSON) value on write (400, `VALUE_SCHEMA_VIOLATION`); violation details never echo the value. Absent ⇒ opaque value. Carried in `x-gts-traits` like every other trait; the validator is compiled per write (schemas are dynamic; the registry client caches the resolution). A registered schema that fails to compile is a broken registration → 503, not 400. | +| `max_size_bytes` | integer (optional) | Upper bound on value size (400, `VALUE_TOO_LARGE`); absent ⇒ platform default only. | +| `expirable` | bool | Whether secrets of this type may carry `expires_at` (else 400, `EXPIRY_NOT_SUPPORTED_FOR_TYPE`; a past expiry is `EXPIRY_IN_THE_PAST`); expired secrets resolve as 404 (read-time SQL filter) and are moved into the deprovisioning saga by the reaper (§6.4). | +| `rotation_period_secs` | integer (optional, advisory) | Recommended rotation cadence; metadata-only — rotation automation stays a non-goal. | +| `utf8_only` | bool | Whether the value must be valid UTF-8 (400, `VALUE_NOT_UTF8`; only `generic` currently allows binary, reachable via the SDK). | + +Trait values resolve through the GTS chain merge (`effective_traits`): +leaf-declared values win, ancestors fill the rest — the base type declares +generic values for every trait, so a derived type only states what it +restricts. Traits are enforced in the gear domain layer at well-defined +points (§5.4); plugins remain trait-agnostic value stores. + +### 5.3 Built-in Type Catalog (Registry Seeds) + +Built-in types seeded into the registry by the SDK (REST name → derived GTS +segment uses `_`, e.g. `api-key` → `…~cf.core.credstore.api_key.v1~`): + +| Type | `allow_sharing` | `value_schema` | Notes | +|------|-----------------|----------------|-------| +| `generic` | private, tenant, shared | — | Default; fully backward-compatible with existing secrets (binary allowed). | +| `api-key` | private, tenant, shared | — | Third-party provider keys (OpenAI etc.) — the core hierarchical-sharing use case (OAGW). | +| `personal-token` | **private only** | — ; `expirable` | Personal access tokens; the flagship `allow_sharing` restriction — can never be shared tenant-wide or inherited. | +| `oauth2-client` | tenant, shared | `{client_id, client_secret, [token_url, scopes]}` | Structured; consumed by OAGW OAuth2 client-credentials auth. | +| `basic-auth` | private, tenant, shared | `{username, password}` | Structured HTTP basic credentials. | +| `bearer-token` | private, tenant | — ; `expirable` | Short-lived tokens; expiry enforced on read. | +| `certificate` | tenant, shared | — ; `expirable`, `rotation_period_secs` = 90 d advisory | TLS material; `not_after` maps to `expires_at`. Format (PEM) checks deferred. | +| `ssh-key` | private, tenant | — | Deploy/automation keys. Format checks deferred. | +| `webhook-hmac` | tenant, shared | — | Webhook signing secrets shared with descendant integrations. | +| `connection-string` | tenant | — ; `max_size_bytes` = 4 KiB | DSNs; tenant-local by policy (contain embedded endpoints/credentials). | + +Adding a **built-in** type (shipped with the platform, with a short REST +name) = adding a catalog entry + its seed registration in the SDK (one +release). Adding a **custom** type = registering a GTS schema derived from +the secret base type — no release; the type is addressed by its full GTS id +on the API. The enforcement code changes only when a new *trait* +is introduced. + +### 5.4 Enforcement Points + +1. **Type resolution** (every operation): the type UUID — from the stored row (read/overwrite/delete prefetch) or from the request (create; default `generic`) — is resolved through `SecretTypeResolver` against the types-registry: envelope check (must descend from `gts.cf.core.credstore.secret.v1~`) + effective-traits merge (§4.4). Unknown/non-secret type: `UNKNOWN_SECRET_TYPE` (400) on create, 503 for a stored row; registry outage/timeout/malformed traits: 503. No credstore-side cache — the registry client's TTL cache bounds both latency and staleness. +2. **Create / put**: validate `sharing ∈ allow_sharing`, value against the `value_schema` trait (compiled per write), size against `max_size_bytes`, UTF-8 against `utf8_only`, and the expiry gate — all on the **resolved traits**, before any side effect. Violations → 400 (`InvalidArgument`) with the stable per-trait reason (§5.2). +3. **Update**: type immutable (`TYPE_IMMUTABLE` when an explicit differing `type` is sent — compared by UUID; absent `type` inherits the row's); the new value/sharing/expiry re-validated against the resolved traits. A PUT is a whole-value replace: omitting `expires_at` clears a stored expiry. +4. **Read**: rows with `expires_at <= now` are filtered out in the resolution SQL (404); `type` (and `expires_at`, when set) are returned in response metadata. +5. **Authorization**: a **single** PDP evaluation per operation targets the secret's **full concrete GTS type** — including `generic` — as returned by the type resolution (step 1). Its `AccessScope` is enforced in SQL and its gate must include the **caller's** tenant (hierarchical visibility of inherited/shared secrets is decided by the resolver, not the PDP). Denial surfaces as the anti-enumeration 404 on read and 403 on write/delete; a PDP outage is 503. Every type (incl. `generic` and custom types) reaches the PDP, so a per-type policy can be added with no credstore change. On read the PDP is consulted only for a secret that resolves (a missing secret is a 404 without a PDP or registry call). +6. **Reaper**: each tick first flips expired `active` rows into the ordinary deprovisioning saga (`mark_expired_deprovisioning`), which then cleans the backend value and releases the reference via the pending sweep (§6.4). The reaper never resolves types — sweeping is type-agnostic. + +### 5.5 Storage & API Changes + +- `credstore_secrets` carries `secret_type_uuid UUID NOT NULL DEFAULT ''` — the deterministic v5 UUID of the type's GTS id (`GtsID::to_uuid`, pinned as `credstore_sdk::GENERIC_TYPE_UUID_STR`), like AM's `tenants.tenant_type_uuid` — and `expires_at TIMESTAMPTZ NULL`, plus the partial expiry-sweep index (§4.7, §8). The stored UUID is opaque to the storage layer; only the type resolution interprets it. +- REST: optional `type` (the secret type's full GTS type id) and `expires_at` (RFC 3339) on `POST`/`PUT`; `GET` metadata returns `type` (the resolved full GTS type id) and `expires_at`. +- SDK: `CredStoreClientV1` gains `put_opts`/`create_opts` taking `WriteOptions { secret_type: Option, expires_at, precondition }` — the gear resolves the `GtsId` to the type's deterministic UUID; `put`/`create` are provided methods delegating with defaults (source-compatible). `GetSecretResponse.secret_type` is the resolved full GTS type id. +- Plugin SPI: **unchanged** — types are a metadata/policy concern. + +## 6. Secret Lifecycle & Sagas + +### 6.1 Status Model -#### Risk: Hierarchy Walk-up Performance at Deep Nesting +``` + create saga delete saga + ┌──────────────┐ plugin.put OK ┌────────┐ mark deprovisioning ┌────────────────┐ + │ provisioning ├────────────────►│ active ├──────────────────────►│ deprovisioning │ + └──────┬───────┘ └────────┘ └───────┬────────┘ + │ backend failure → rollback (row deleted) │ plugin.delete OK + │ stuck → reaped │ → row deleted + ▼ ▼ stuck → reaped + (gone) (gone) +``` -**Impact**: Increased latency for secret resolution in deeply nested tenant hierarchies (10+ levels) +| Status | smallint | Visible to resolution | Holds unique index | Swept by reaper | +|--------|----------|----------------------|--------------------|-----------------| +| `provisioning` | 1 | no | yes | yes (after `provisioning_timeout_secs`) | +| `active` | 2 | **yes** | yes | no | +| `deprovisioning` | 3 | no | yes | yes (after `deprovisioning_timeout_secs`) | -**Mitigation**: -- Early termination: stop walk-up on first accessible secret -- Cache tenant hierarchy queries from tenant_resolver -- Monitor resolution depth and latency metrics -- Consider future optimization: backend-side hierarchy resolution if performance becomes critical +Only `active` rows are returned by `resolve_for_get`; a secret therefore +becomes visible atomically at saga commit and invisible atomically at delete +start. -**Likelihood**: Low | **Impact**: Medium | **Priority**: P2 +### 6.2 Provisioning Saga -#### Risk: Credstore API Changes Break Plugin +Sequence ID: `cpt-cf-credstore-seq-write-saga` (declared in §4.6). -**Impact**: Plugin stops working, secrets become inaccessible, platform outage +**Create path** (no row of the target sharing class exists): -**Mitigation**: -- Pin VendorA Credstore API version in plugin -- Integration tests against Credstore (run in CI) -- Version compatibility matrix documented -- Plugin adapts to API changes with feature flags or version detection +1. Fail fast if no plugin is available (before any metadata side effect). +2. `INSERT` row with `status = provisioning` (scope-clamped). +3. `plugin.put(tenant, key, value, owner-class)`. +4. `UPDATE status = active` (`mark_active`). -**Likelihood**: Medium | **Impact**: High | **Priority**: P1 +**Failure handling**: -#### Risk: ExternalID Encoding Collision +- Step 3 fails → **compensate**: best-effort delete of the provisioning row (metric `provisioning_rollback`); a failed cleanup defers to the reaper. The reference is never permanently wedged. +- Step 4 fails → the backend value exists but the row stays `provisioning`: never readable; a client retry overwrites it; otherwise reaped — and the reaper issues a best-effort `plugin.delete` for every row it reaps (§6.4), so the orphaned backend value is reconciled rather than leaked. +- **Create race** (unique-index conflict on step 2): a create-only `POST` surfaces 409; a `PUT` retries `find_for_write` up to 3 times with 25 ms backoff (the winner marks active within ms) and degrades to a retry-safe 409 if the winner appears stuck. -**Impact**: Wrong secret returned, data corruption, security incident +**Overwrite path** (row of the target class exists): the ordering of the +backend write vs the version-gated `touch` (version bump + sharing update + +fence stamp) depends on the precondition. **Without `If-Match`** +(last-writer-wins): backend `plugin.put` first, then `touch` — a plugin +failure leaves metadata untouched; a `touch` failure after a committed +backend write fails closed (fence mismatch → 404) until the next put +retries; a row that vanished concurrently (delete won) is reported as a +retryable 409 so the caller re-runs the put. **With `If-Match`** +(optimistic concurrency): the version-gated `touch` claims the CAS +**before** `plugin.put`, so a losing writer (0-row `touch` → 409) never +reaches the backend and cannot clobber the winner's value; a backend +failure after the claim fails closed the same way until the caller retries +against the new version. -**Mitigation**: -- Deterministic base64url encoding with no-padding -- SecretRef format validation: `[a-zA-Z0-9_-]+` (no colons) -- Tenant ID is UUID (no colons) -- Comprehensive test coverage for edge cases -- Documented encoding algorithm +### 6.3 Deprovisioning Saga -**Likelihood**: Very Low | **Impact**: Critical | **Priority**: P1 +> Requirement: `cpt-cf-credstore-fr-deprovisioning`. Status-driven saga +> symmetric to provisioning (statuses ship in `m0001_initial_schema`); +> replaced the earlier backend-first delete design. -#### Risk: External Key Service Unavailability +**Delete path**: -**Impact**: All encrypt/decrypt operations blocked in Credentials Storage plugin; credential reads and writes fail +1. `find_own` + version pre-check (`If-Match`), as today. +2. `UPDATE status = deprovisioning` gated by `version = ?` when a precondition was given. From this instant the secret is invisible to resolution (single-status filter — no read/delete race), while the row keeps holding the partial unique index. +3. `plugin.delete(tenant, key, owner-class)` — `NotFound` is success (idempotent). +4. `DELETE` the metadata row. -**Mitigation**: -- High-availability key service deployment -- Readiness probe reflects KMS connectivity -- Key caching with short TTL for read-path resilience -- Circuit breaker for key service calls +**Failure handling**: -**Likelihood**: Low | **Impact**: Critical | **Priority**: P1 +- Step 3/4 fails → the row stays `deprovisioning`; the caller gets the mapped error (503 for an unavailable backend, 500 for an internal plugin fault) and can retry. A **retry of `DELETE` resumes the saga**: `find_own` sees the `deprovisioning` row and re-runs steps 3–4 (both idempotent). Absent a retry, the reaper completes it. +- Crash between 2 and 4 → same recovery: reaper or client retry finishes cleanup. No state leaves a readable secret or a permanently held reference. -#### Risk: Keys Co-located with Encrypted Data (DatabaseKeyProvider) +**Concurrent create during deprovisioning**: the row keeps the unique index +until step 4, so a `POST`/`PUT` for the same reference conflicts (409, +retryable) until cleanup completes. This is deliberate: releasing the index +earlier would let a new secret's backend value be deleted by the old row's +lagging step 3 (the backend key is identical). The window is bounded by the +reaper cadence. -**Impact**: Single breach exposes both ciphertext and keys when using `DatabaseKeyProvider` +**Interface impact**: REST/SDK signatures unchanged (`DELETE` stays 204 on +success); `SecretStatus` has `Deprovisioning = 3` (status `CHECK` in §4.7); +config has `reaper.deprovisioning_timeout_secs` (default 300 s); metric +`deprovisioning_reaped` mirrors `provisioning_reaped`. -**Mitigation**: -- Use `ExternalKeyProvider` in production multi-tenant deployments -- `DatabaseKeyProvider` restricted to development/test environments by deployment policy -- Document key–data separation requirement in operational runbooks +### 6.4 Reaper -**Likelihood**: Medium | **Impact**: Critical | **Priority**: P1 +The gear's lifecycle entry (`serve`) runs a cancellable loop on +`reaper.tick_secs` (default 60 s; delayed missed-tick behavior). Each tick: -#### Risk: Owner ID Migration for Existing Secrets +1. **Expiry sweep**: flip expired `active` rows (`expires_at <= now`) into the deprovisioning saga (they already stopped resolving at read time; this cleans the backend value and releases the reference). +2. **List stale non-active rows** (bounded batch) older than `provisioning_timeout_secs` / `deprovisioning_timeout_secs` (both default 300 s), via the pending partial index. +3. **Backend reconciliation**: issue a best-effort `plugin.delete` for every stale row (closing the orphaned-value debt of §6.2); `NotFound` counts as success. +4. **Remove rows**: `provisioning` rows unconditionally (the reference must not stay wedged); `deprovisioning` rows only after a successful backend delete — otherwise the row (and the name it holds) waits for the next tick. Counts → `provisioning_reaped` / `deprovisioning_reaped`. +5. **Refresh inventory gauges** (row counts per status). -**Impact**: Existing secrets without owner_id cannot be accessed in `private` mode +Errors are logged, never propagated — the reaper must survive transient DB or +plugin outages. -**Mitigation**: -- Migration script to backfill owner_id for existing secrets (default to tenant admin) -- Backward compatibility: if owner_id is null, treat as `tenant` mode -- Phased rollout: new secrets get owner_id, existing secrets migrated gradually +## 7. Risks / Trade-offs -**Likelihood**: High | **Impact**: Medium | **Priority**: P1 +### 7.1 Architectural Trade-offs -## 6. Migration Plan +#### Stateful gear (metadata table) vs stateless pass-through -### 6.1 Schema Migration: Two-Mode to Three-Mode Sharing +**Decision**: the gear owns metadata; backends store values only. -**Background**: The current design introduces a three-tier sharing model (`private`/`tenant`/`shared`) replacing the previous two-mode system. The new `private` mode (owner-only) is more restrictive than the old `private` mode (tenant-wide). +- ✅ Hierarchical resolution and authorization in one transactional, indexed SQL query — latency independent of hierarchy depth on the metadata side +- ✅ No backend metadata-schema prerequisite; any dumb value store qualifies as a backend +- ✅ Sharing/uniqueness rules enforced by partial unique indexes rather than by backend-specific behavior +- ✅ No encoded-external-ID collision surface +- ❌ The gear needs a database and migrations (`stateful` capability) +- ❌ Metadata and backend value can diverge transiently — mitigated by the saga + reaper design (§6) and idempotent retries -**Migration Strategy**: +#### Authorization via PDP scope in SQL (not permission strings) -#### Phase 1: Backend Schema Update (VendorA Credstore) +**Decision**: PDP `AccessScope` + SecureORM clamps instead of coarse +`Secrets:Read`/`Secrets:Write` permission checks. -1. **Add `owner_id` field** to Credstore schema: - - Type: UUID - - Nullable: YES (for backward compatibility with existing secrets) - - Default: NULL - - Index: Add index on (tenant_id, owner_id) for owner-based queries +- ✅ Real tenant-subtree isolation enforced at the data layer, consistent with platform RBAC/PDP +- ✅ Degraded flat-`In` mode keeps correctness when the tenant closure is not co-located +- ❌ Every operation pays a PDP evaluation (timed as a dependency metric; 503 on PDP outage — fail-closed) -2. **Extend `sharing` enum** from 2 values to 3: - - Old values: `private`, `shared` - - New values: `private` (owner-only), `tenant` (tenant-wide), `shared` (hierarchical) - - Migration: Map old `private` → new `tenant`, old `shared` → new `shared` +#### Three-tier sharing model (not RBAC/ABAC per secret) -3. **Backfill `owner_id` for existing secrets**: - - Strategy: Set owner_id to tenant admin or service account for existing secrets - - Alternative: Leave owner_id as NULL and treat NULL as `tenant` mode (accessible to all in tenant) +Unchanged from the original design: simple to reason about, covers the primary +use cases; per-secret ACLs remain out of scope. Secret types add a +*type-level* restriction axis (`allow_sharing`, §5) without introducing +per-secret ACLs. -#### Phase 2: Gateway and Plugin Update +#### 404 for inaccessible secrets (not 403) -1. **Update `credstore_vendor_a_plugin`**: - - Update PUT/POST calls to include `owner_id` field - - Update GET response parsing to extract `owner_id` from backend - - Handle NULL `owner_id` (treat as `tenant` mode) +Unchanged: prevents enumeration; per-secret access failures are +indistinguishable from absence. 403 is reserved for PDP-level denial of the +*operation* (e.g., subject may not `read` at all) and the own-tenant gate. -2. **Update `credstore` Gateway**: - - Update hierarchical resolution algorithm to check `owner_id` for `private` mode - - Update PUT/POST operations to capture `owner_id` from SecurityContext.subject_id() - - Update error handling to return 404 for owner-mismatch +#### PUT-as-upsert + POST-as-create-only -3. **Update `credstore-sdk`**: - - Update API contracts to include `owner_id` in SecretMetadata - - Update sharing mode enum to three values +Unchanged semantics, now with optimistic concurrency: `PUT` is an idempotent +upsert honouring `If-Match`; `POST` is create-only with 409 on same-class +conflict. -#### Phase 3: Client Migration +### 7.2 Security and Performance Risks -1. **Inform consumers** about new sharing mode semantics: - - Old `private` mode behavior is now `tenant` mode - - New `private` mode is owner-only (more restrictive) - - Existing secrets with old `private` will be migrated to new `tenant` +#### Risk: Secret values leaked through logs or caches -2. **Provide migration guide** for consumers to update sharing modes if needed +**Mitigation**: `SecretValue` redaction + zeroize; hand-written redacted +`Debug` on DTOs; `Cache-Control: no-store`; no lossy UTF-8 decode; code +review. **Likelihood**: Medium | **Impact**: Critical | **Priority**: P1 -3. **Backward compatibility**: - - Gateway accepts both old and new sharing mode values - - API defaults to `tenant` mode if sharing not specified +#### Risk: Metadata/backend divergence (saga partial failure) -#### Phase 4: Rollout +**Impact**: value-less rows (404 until re-put), orphaned backend values +(storage leak, never readable), wedged references behind the unique index. -1. **Deploy backend schema changes** (VendorA Credstore update) -2. **Deploy Gateway and Plugin updates** (with backward compatibility) -3. **Run migration script** to backfill owner_id and update sharing values -4. **Monitor** for errors and performance issues -5. **Gradually deprecate** old two-mode sharing values in API (future release) +**Mitigation**: compensating rollback on create; backend-first ordering on +overwrite; the deprovisioning saga + reaper backend reconciliation (§6.3, +§6.4); configurable timeouts; rollback/reap metrics for operational +visibility. -### 6.2 Backward Compatibility +**Likelihood**: Medium | **Impact**: Medium | **Priority**: P1 -- **API**: Gateway accepts old sharing mode values and maps them to new values -- **Defaults**: Secrets created without `owner_id` are treated as `tenant` mode -- **Clients**: No breaking changes to existing client code (old behavior preserved under new `tenant` mode) +#### Risk: PDP or tenant-resolver outage -### 6.3 Rollback Plan +**Impact**: all operations fail (503) — fail-closed by design. -If migration fails: -1. **Revert Gateway and Plugin** to previous version -2. **Keep backend schema changes** (owner_id and new sharing enum are additive) -3. **Use feature flag** to disable three-tier sharing mode enforcement +**Mitigation**: ancestor-chain TTL cache absorbs short tenant-resolver blips; +dependency latency/outcome metrics; PDP evaluation is per-request with no +local policy cache (deliberate: policy freshness over availability). -### 6.4 Success Criteria +**Likelihood**: Low | **Impact**: High | **Priority**: P1 -- ✅ All existing secrets remain accessible after migration -- ✅ New secrets can be created with all three sharing modes -- ✅ Owner-only access control works for `private` mode -- ✅ No performance degradation in hierarchical resolution -- ✅ Zero downtime during rollout +#### Risk: Ancestor-chain cache staleness -## 7. Open Questions +**Impact**: up to `ancestor_cache_ttl_secs` (300 s) of stale hierarchy — a +re-parented tenant may briefly resolve secrets along the old chain. -This section cross-references open questions from [PRD.md Section 13](./PRD.md#13-open-questions) and adds design-specific questions. +**Mitigation**: short TTL + LRU; the chain carries no caller-specific data +(safe to share across security contexts); PDP scope still clamps every query, +so a stale chain can widen *candidate rows* but never *authorized rows*. -### 7.1 From PRD (Cross-Reference) +**Likelihood**: Low | **Impact**: Medium | **Priority**: P2 -All open questions below are documented in detail in PRD.md Section 13 (lines 605-609). +#### Risk: Reference wedged by stuck saga rows -1. **Batch Retrieval**: Should `resolve` support batch retrieval (multiple references in one call) for OAGW efficiency? - - **Design Impact**: Would require new API endpoint `POST /secrets/batch` with array of SecretRef inputs +**Impact**: 409 on create for up to the reaper timeout after a crashed write +(or delete, once deprovisioning ships). -2. **P2/Future - Human vs Service Access**: Should human users be restricted from retrieving raw secret values for inherited shared secrets, while service accounts can? - - **Design Impact**: Requires distinguishing human vs service authentication in SecurityCtx; separate authorization rules; metadata-only response for humans +**Mitigation**: bounded create-race retries; rollback-before-reaper on the +common failure path; configurable `provisioning_timeout_secs` / +`deprovisioning_timeout_secs`; `provisioning_reaped` alerting. -3. **P2/Future - Audit Trails**: All credential operations should leave audit trails (timestamps, actor, tenant, outcome) with tamper-evident storage - - **Design Impact**: New audit gear; event emission from Gateway; secure log storage; never log plaintext secret values +**Likelihood**: Low | **Impact**: Low | **Priority**: P2 -4. **P2/Future - Schema Validation**: Should secrets support JSON schema validation (using GTS)? - - **Design Impact**: Schema registry; validation hooks in Gateway put operation; structured secret storage +## 8. Migration Plan -5. **P2/Future - Compare-and-Swap (CAS)**: Should secret updates support atomic CAS validation? - - **Design Impact**: Optional `expected_value` parameter in PUT operation; use VendorA Credstore CAS endpoint +Schema is managed by SeaORM migrations (raw per-backend SQL, PostgreSQL + +SQLite, MySQL fails fast). The stateful gear, the deprovisioning saga, and +secret types shipped together — before any deployment of this gear — so the +gear starts from one consolidated migration: -### 7.2 Design-Specific Questions +- **`m0001_initial_schema`**: the full `credstore_secrets` table of §4.7 — lifecycle statuses `(1,2,3)`, the monotonic `version`, `secret_type_uuid` (default = the generic type's v5 UUID) + `expires_at`, partial unique indexes, and the lookup / pending-sweep / expiry-sweep indexes. -6. **Plugin Failure Handling**: If a plugin call fails during hierarchical walk-up (e.g., network timeout), should Gateway: - - Option A: Fail entire request (fail-fast) - - Option B: Continue to next ancestor (best-effort) - - Option C: Cache last-known value (eventual consistency) - - **Recommendation**: Option A (fail-fast) for consistency; revisit if reliability issues emerge +Future schema changes are additive migrations on top. **Backward +compatibility** for clients: untyped writes behave exactly as before +(`generic` type, all sharing modes, no expiry). Rollback = revert the gear +and run the migration `down`. -7. **Tenant Hierarchy Caching**: Should Gateway cache tenant_resolver hierarchy queries? - - **Design Impact**: In-memory cache with TTL; invalidation strategy; memory pressure considerations - - **Recommendation**: Yes, with 5-minute TTL and LRU eviction +## 9. Open Questions -8. **Secret Metadata in List Operation**: Should `GET /secrets` (list all secrets for tenant) include metadata fields (owner_tenant_id, sharing, is_inherited)? - - **Design Impact**: Additional plugin calls during list; performance implications - - **Recommendation**: P2, list is not in v1 scope +1. **Batch retrieval** (from PRD): should `get` support batch retrieval (multiple references in one call) for OAGW efficiency? Design impact: `POST /secrets/batch`; single resolution query already amortizes well. +2. **Human vs service access** (P2, from PRD): restrict human users to metadata-only for inherited shared secrets while service accounts read values? +3. **Audit trail** (P2, from PRD): emit structured audit events (actor, tenant, outcome — never values) to a platform audit sink. +4. **List/metadata endpoint** (P2): a metadata-only list (no values) becomes cheap with local metadata; interacts with the anti-enumeration stance and per-type authorization — needs a dedicated design pass. -9. **Owner ID for Service Accounts**: For service-to-service operations (e.g., OAGW creating secrets on behalf of tenants), should owner_id be: - - Option A: Service account subject_id (OAGW's ID) - - Option B: Target tenant admin (impersonation) - - **Recommendation**: Option A (service account ID) for audit trail clarity +(The former "dynamic type descriptors" question is resolved: types are +registry-driven per §5, with the fail-closed semantics of §5.4.) -## 8. Additional context +## 10. Additional context ### Plugin Registration -Following the ToolKit plugin pattern (as documented in `docs/TOOLKIT_PLUGINS.md` and exemplified by `tenant_resolver`): +Following the ToolKit plugin pattern: -1. `credstore` registers the plugin GTS schema during init -2. Each plugin registers its GTS instance and scoped `CredStorePluginClientV1` in ClientHub -3. Gateway resolves the active plugin via GTS instance query and vendor configuration +1. The SDK's `CredStorePluginSpecV1` schema reaches types-registry automatically via the `toolkit-gts` link-time inventory (no per-init registration). +2. Each plugin registers its GTS instance and its scoped `CredStorePluginClientV1` in ClientHub. +3. The gear lazily resolves the active plugin: instance query by type-id prefix → filter by configured `vendor` → highest-priority active instance. -**Exactly one storage plugin is active per deployment** (selected by configuration `vendor` field match). For simple plugins (VendorA, OS keychain), the Gateway handles all cross-cutting concerns (authorization, hierarchical resolution, sharing mode enforcement), while the plugins provide simple per-tenant CRUD operations. The `credentials_storage` plugin is a full microservice that handles merge resolution, authorization, and encryption internally — when active, the Gateway delegates these responsibilities to the plugin. +**Exactly one storage plugin is active per deployment** (vendor match). The +gear handles all cross-cutting concerns; plugins are per-tenant value +stores only. **GTS Types:** -- Schema: `gts.cf.toolkit.plugins.plugin.v1~cf.core.credstore.plugin.v1~` -- VendorA instance: `gts.cf.toolkit.plugins.plugin.v1~cf.core.credstore.plugin.v1~cf.core.vendor_a.app._.plugin.v1` -- OS storage instance: `gts.cf.toolkit.plugins.plugin.v1~cf.core.credstore.plugin.v1~cf.core.os_protected.app._.plugin.v1` -- Credentials Storage instance: `gts.cf.toolkit.plugins.plugin.v1~cf.core.credstore.plugin.v1~cf.core.credentials_storage.app._.plugin.v1` +- Plugin spec: `gts.cf.toolkit.plugins.plugin.v1~cf.core.credstore.plugin.v1~` +- Secret resource type: `gts.cf.core.credstore.secret.v1~` (= `SECRET_RESOURCE_TYPE`, the PDP resource type; registered with an empty property set — authorization needs only the type id) +- Secret types: derived from the secret base type — built-ins `gts.cf.core.credstore.secret.v1~cf.core.credstore..v1~` (one seed per catalog entry, traits as `x-gts-traits`), plus any custom registered descendant; §5 ### Configuration -**Gateway:** ```yaml gears: credstore: - vendor: "vendor_a" # Selects plugin by matching vendor -``` - -**VendorA Plugin:** -```yaml -gears: - credstore_vendor_a_plugin: - vendor: "vendor_a" - priority: 100 - base_url: "https://credstore.internal.example.com" - client_id: "credstore-client" - client_secret: "${CREDSTORE_CLIENT_SECRET}" - # Optional: separate RO/RW credentials - # ro_client_id: "credstore-ro" - # ro_client_secret: "${CREDSTORE_RO_SECRET}" - scopes: ["credstore"] - timeout_ms: 5000 - retry_count: 3 + database: + server: "postgres_main" # platform DB provider reference + config: + vendor: "virtuozzo" # selects the value-store plugin by GTS vendor (default: "virtuozzo") + hierarchy: + ancestor_cache_ttl_secs: 300 # ancestor-chain cache TTL (> 0) + tenant_closure_colocated: false # advertise TenantHierarchy PDP capability (opt-in) + reaper: + tick_secs: 60 # reaper cadence (> 0) + provisioning_timeout_secs: 300 # stuck-provisioning sweep threshold (> 0) + deprovisioning_timeout_secs: 300 # stuck-deprovisioning sweep threshold (> 0) ``` -**Credentials Storage Plugin:** -```yaml -gears: - credentials_storage: - vendor: "credentials_storage" - priority: 100 - base_url: "http://credentials-storage.internal:8080" - database_url: "${CREDENTIALS_STORAGE_DB_URL}" - key_provider: "database" # "database" or "external" - # External key provider settings (when key_provider = "external"): - # kms_url: "https://vault.internal:8200" - # kms_auth: "${KMS_AUTH_TOKEN}" -``` +Config is validated at init (`deny_unknown_fields`; non-empty vendor; all +periods > 0); an invalid config fails gear startup. ### Error Mapping -| Backend Response | Plugin Error | Gateway/Consumer Error | -|-----------------|-------------|----------------------| -| Credstore 404 | `None` | `NotFound` | -| Credstore 401/403 | `PermissionError` | `Internal` (credentials misconfigured) | -| Credstore 5xx | `BackendError` (retryable) | `Internal` | -| Credstore timeout | `BackendError` (retryable) | `Internal` | -| OS keychain item not found | `None` | `NotFound` | -| OS keychain access denied | `PermissionError` | `Internal` | +Domain → canonical (wire) mapping (`sdk_error_mapping`, pinned by tests): + +| DomainError | Canonical category | HTTP | +|-------------|--------------------|------| +| `InvalidSecretRef`, `InvalidPrecondition`, `UnsupportedTransition` | `InvalidArgument` | 400 | +| `NotFound` | `NotFound` | 404 | +| `Conflict` | `AlreadyExists` | 409 | +| `VersionConflict` | `Aborted` (`OPTIMISTIC_LOCK_FAILURE`) | 409 | +| `AccessDenied` | `AccessDenied` | 403 | +| `ServiceUnavailable` (incl. "no storage plugin registered") | `ServiceUnavailable` (+ `Retry-After` when hinted) | 503 | +| `Internal` | `Internal` (diagnostic stripped from the wire) | 500 | + +Plugin-layer `CredStoreError`s are normalized by `map_plugin_err`; a plugin +returning `UnsupportedTransition` or `InvalidSecretRef` is a contract +violation surfaced as `Internal`. + +### Observability + +Typed OpenTelemetry metrics via `CredStoreMetricsPort`: + +- `walkup_depth` — ancestor distance of the winning row +- `read_outcome` — own / inherited / miss +- `dependency` — latency + outcome per dependency (PDP evaluate, tenant-resolver chain, types-registry type resolution, plugin get/put/delete) +- `cross_tenant_denied` — own-tenant gate rejections +- `provisioning_rollback`, `provisioning_reaped`, `deprovisioning_reaped` — saga health +- inventory gauge — row counts per status, refreshed by the reaper -## 9. Traceability +## 11. Traceability - **PRD**: [PRD.md](./PRD.md) -- **ADRs**: [ADR/](./ADR/) -- **Features**: [features/](./features/) -- **Credentials Storage Plugin Design**: `plugins/credentials-storage/DESIGN.md` (planned) +- **ADRs**: [ADR/](./ADR/) — [ADR-0001 stateful gear](./ADR/0001-cpt-cf-credstore-adr-stateful-gear.md), [ADR-0002 deprovisioning saga](./ADR/0002-cpt-cf-credstore-adr-deprovisioning-saga.md) +- **Features**: features/ (planned) diff --git a/gears/credstore/docs/PRD.md b/gears/credstore/docs/PRD.md index 366fbf650..8c0987382 100644 --- a/gears/credstore/docs/PRD.md +++ b/gears/credstore/docs/PRD.md @@ -20,8 +20,10 @@ - [5.1 P1 — Core Operations](#51-p1--core-operations) - [5.2 P1 — Hierarchical Sharing](#52-p1--hierarchical-sharing) - [5.3 P1 — Authorization](#53-p1--authorization) - - [5.4 P1 — Encryption & Key Management](#54-p1--encryption--key-management) - - [5.5 P2 — Planned](#55-p2--planned) + - [5.4 P1 — Reliability & Concurrency](#54-p1--reliability--concurrency) + - [5.5 P1 — Secret Types](#55-p1--secret-types) + - [5.6 P1 — Deprovisioning Lifecycle](#56-p1--deprovisioning-lifecycle) + - [5.7 P2 — Planned](#57-p2--planned) - [6. Non-Functional Requirements](#6-non-functional-requirements) - [6.1 Gear-Specific NFRs](#61-gear-specific-nfrs) - [7. Public Library Interfaces](#7-public-library-interfaces) @@ -53,20 +55,18 @@ SCOPE: ✓ Assumptions, dependencies, risks NOT IN THIS DOCUMENT (see other templates): - ✗ Stakeholder needs (managed at project/task level by steering committee) ✗ Technical architecture, design decisions → DESIGN.md ✗ Why a specific technical approach was chosen → ADR/ ✗ Detailed implementation flows, algorithms → features/ -STANDARDS ALIGNMENT: - - IEEE 830 / ISO/IEC/IEEE 29148:2018 (requirements specification) - - IEEE 1233 (system requirements) - - ISO/IEC 15288 / 12207 (requirements definition) - REQUIREMENT LANGUAGE: - Use "MUST" or "SHALL" for mandatory requirements (implicit default) - Do not use "SHOULD" or "MAY" — use priority p2/p3 instead + - Requirements marked **Planned** are specified but not yet implemented; + everything else is implemented. - Be specific and clear; no fluff, bloat, duplication, or emoji + - Keep transport/mechanism detail (endpoints, status codes, headers) out of + this doc — it lives in DESIGN.md; the PRD states capabilities and outcomes. ============================================================================= --> @@ -74,34 +74,40 @@ REQUIREMENT LANGUAGE: ### 1.1 Purpose -CredStore provides per-tenant secret storage and retrieval for the platform. It abstracts backend differences behind a unified API, enabling platform gears to store and access credentials without coupling to a specific storage technology. +CredStore provides per-tenant secret storage and retrieval for the platform. It owns all secret metadata (identity, sharing, ownership, lifecycle status, version) and enforces policy; pluggable backends store only the secret values. This abstracts backend differences behind a unified API, enabling platform gears to store and access credentials without coupling to a specific storage technology. ### 1.2 Background / Problem Statement Platform gears — most notably the Outbound API Gateway (OAGW) — need access to secrets (API keys, tokens, credentials) for making upstream API calls on behalf of tenants. These secrets must be stored securely, scoped per tenant, and accessible only to authorized consumers. -Standard credential stores provide per-tenant isolation but do not support hierarchical multi-tenant sharing. In the platform's business model, parent tenants (partners) share API credentials with child tenants (customers). For example, a partner with an OpenAI API key and quota allows their customers to make requests through OAGW using the partner's key — without the customer ever seeing the actual secret value. This requires a hierarchical resolution model: when a customer requests a secret, the system walks up the tenant tree to find a shared secret from an ancestor. +Standard credential stores provide per-tenant isolation but do not support hierarchical multi-tenant sharing. In the platform's business model, parent tenants (partners) share API credentials with child tenants (customers). For example, a partner with an OpenAI API key and quota allows their customers to make requests through OAGW using the partner's key — without the customer ever seeing the actual secret value. This requires a hierarchical resolution model: when a customer requests a secret, the system walks up the tenant tree to find a shared secret from an ancestor — while honouring the platform's tenant-isolation barriers. -Additionally, the platform runs in multiple environments: Kubernetes (where an external credential store like VendorA Credstore is available) and desktop/VM (where OS-level protected storage is appropriate). A plugin-based architecture allows runtime selection of the appropriate backend without changing consumer code. +Keeping secret metadata in the gear's own database (rather than in the backend) makes hierarchical resolution and authorization a single transactional query, removes any backend schema prerequisite, and allows any simple key-value store to serve as a backend plugin. ### 1.3 Goals (Business Outcomes) - Enable OAGW to retrieve tenant credentials for upstream API calls without exposing secret values to end users - Support hierarchical credential sharing so partners can share API access with customers - Decouple platform gears from specific credential storage backends -- Enforce least-privilege access: read vs write authorization, service-to-service vs tenant self-service +- Enforce least-privilege access through the platform policy plane (PDP), with tenant isolation guaranteed at the data layer +- Make secret writes and deletes crash-safe: no partial failure may leak a readable half-written secret or permanently block a secret name ### 1.4 Glossary | Term | Definition | |------|------------| | Secret | A key-value pair where the value is sensitive (API key, token, password) | -| Secret reference | A human-readable key identifying a secret within a tenant's namespace (e.g., `partner-openai-key`). **Format**: alphanumeric characters, hyphens, and underscores only (`[a-zA-Z0-9_-]+`). Max length: 255 characters. Colons are prohibited to prevent ExternalID collisions. | -| Sharing mode | Controls secret access scope: `private` (owner only), `tenant` (all users in tenant), or `shared` (tenant + descendants) | +| Secret reference | A human-readable key identifying a secret within a tenant's namespace (e.g., `partner-openai-key`). **Format**: `[a-zA-Z0-9_-]+`, 1–255 characters. | +| Sharing mode | Controls secret access scope: `private` (owner only), `tenant` (all users in tenant, default), or `shared` (tenant + descendants) | | Owner | The specific actor (identified by `subject_id` from SecurityContext) that created the secret | -| Hierarchical resolution | Lookup algorithm that walks from child to parent to root tenant, returning the first matching accessible secret | +| Hierarchical resolution | Lookup that resolves a reference against the requesting tenant and its ancestors, returning the closest accessible secret | | Secret shadowing | When a child tenant creates a secret with the same reference as a parent's shared secret, the child's own secret takes precedence | -| SecurityCtx | Request security context containing the authenticated tenant ID, subject ID, and permissions | +| Isolation barrier | A tenant-hierarchy boundary (`self_managed`) across which `shared` secrets are not inherited | +| Secret status | Lifecycle state of a secret: `provisioning` (write in flight), `active` (readable), `deprovisioning` (delete in flight) | +| Secret type | A GTS-registered classification of a secret (e.g., `api-key`, `personal-token`) carrying enforceable traits such as `allow_sharing`; `generic` by default, immutable per secret | +| Version | Monotonic per-secret counter used for optimistic concurrency (lost-update detection) | +| SecurityContext | Request security context carrying the authenticated tenant ID, subject ID, and claims | +| PDP | The platform policy decision point (`authz-resolver`) that evaluates access scopes | ## 2. Actors @@ -123,7 +129,7 @@ Additionally, the platform runs in multiple environments: Kubernetes (where an e **ID**: `cpt-cf-credstore-actor-oagw` -**Role**: Service that proxies outbound API calls to external services. Retrieves secrets on behalf of tenants using service-to-service authentication with explicit tenant_id. Primary consumer of hierarchical secret resolution. +**Role**: Service that proxies outbound API calls to external services. Retrieves secrets on behalf of tenants by constructing a SecurityContext for the target tenant. Primary consumer of hierarchical secret resolution. #### Platform Gear @@ -131,15 +137,23 @@ Additionally, the platform runs in multiple environments: Kubernetes (where an e **ID**: `cpt-cf-credstore-actor-platform-gear` -**Role**: Any internal gear consuming secrets via ClientHub in-process API. Reads or writes secrets using the calling tenant's SecurityCtx. +**Role**: Any internal gear consuming secrets via the ClientHub in-process API. Reads or writes secrets using the calling tenant's SecurityContext. -#### CredStore Backend +#### Value-Store Backend (Plugin) **ID**: `cpt-cf-credstore-actor-backend` -**Role**: External storage system that persists encrypted secrets. Examples: VendorA Credstore (Go service with REST API), OS-protected storage (macOS Keychain, Windows DPAPI). Accessed exclusively through plugins. +**Role**: Pluggable per-tenant key-value store that persists secret **values only** (no metadata, no policy). Current implementation: `static-credstore-plugin` (in-memory, for development/testing). Production vault-backed plugins are planned. Accessed exclusively through the gear. + + +#### Platform Policy & Directory Services + +**ID**: `cpt-cf-credstore-actor-platform-services` + + +**Role**: `authz-resolver` (PDP) evaluates per-operation access scopes; `tenant-resolver` supplies barrier-aware tenant ancestor chains; `types-registry` provides GTS-based plugin discovery and receives the secret-type registrations. ## 3. Operational Concept & Environment @@ -148,36 +162,38 @@ Additionally, the platform runs in multiple environments: Kubernetes (where an e ### 3.1 Gear-Specific Environment Constraints -- VendorA Credstore plugin requires network access to the Credstore Go service and valid OAuth2 client credentials -- OS-protected storage plugin requires platform-specific keychain/credential APIs (macOS Keychain, Windows DPAPI) -- Only one storage plugin is active per deployment (selected by configuration) +- The gear is a **stateful** gear: it requires a database (PostgreSQL or SQLite; MySQL is rejected at migration time) +- Exactly one value-store plugin is active per deployment (selected by GTS `vendor` configuration) +- The gear depends on `authz-resolver`, `tenant-resolver`, and `types-registry`, and initializes at system priority (its consumers, e.g. OAGW, resolve the client during their own init) +- A background reaper task runs for the lifetime of the gear (lifecycle entry), sweeping stuck lifecycle rows and refreshing inventory metrics ## 4. Scope ### 4.1 In Scope -- Store, retrieve, and delete per-tenant secrets -- Sharing modes: private (owner-only), tenant (tenant-wide, default), and shared (hierarchical) -- Owner-based access control for private secrets (using `subject_id` from SecurityContext) -- Hierarchical secret resolution with walk-up through tenant ancestry +- Store, retrieve, and delete per-tenant secrets (ClientHub + REST) +- Sharing modes: private (owner-only), tenant (tenant-wide, default), shared (hierarchical) +- Owner-based access control for private secrets (`subject_id` from SecurityContext) +- Hierarchical secret resolution across tenant ancestry, honouring isolation barriers - Secret shadowing (child overrides parent) -- Service-to-service retrieval with explicit tenant_id (for OAGW) -- Gateway + Plugin architecture with runtime backend selection -- VendorA Credstore REST plugin (P1) -- Credentials Storage microservice plugin with encrypted storage and schema validation (P1) -- Pluggable tenant key management via `KeyProvider` abstraction — local DB default, optional external KMS (P1) -- OS-protected storage plugin (P2) -- Gear-level authorization enforcement (read vs write) +- Service-to-service retrieval on behalf of arbitrary tenants (OAGW pattern) +- PDP-based authorization with tenant-scope enforcement at the data layer +- Crash-safe write and delete lifecycles (provisioning/deprovisioning sagas + reaper) +- Optimistic concurrency: per-secret version with conditional update/delete preconditions +- Gear + plugin architecture with runtime backend selection; in-memory static plugin for development/testing +- GTS-based secret types with enforceable traits (`allow_sharing`, value schemas, size/format limits, expiry) +- Operational metrics (resolution depth/outcome, dependency health, saga health, inventory) ### 4.2 Out of Scope -- Full Credstore RAML API parity (only subset needed) -- Automatic secret rotation or expiration -- Secret versioning or history +- Secret value history or rollback (the version counter serves optimistic locking only) +- Automatic secret rotation (type-level rotation traits are advisory only) - Cross-tenant secret transfer (secrets cannot change ownership) -- Unauthenticated or untrusted client access (all access requires platform authentication via SecurityCtx) +- Unauthenticated or untrusted client access (all access requires platform authentication via SecurityContext) - Secret listing or search operations (only retrieval by known reference) -- Granular access control lists (ACLs) for sharing with specific tenant subsets. Current design supports only hierarchical sharing modes (`private`, `tenant`, `shared`). Custom ACLs (e.g., "share with tenants A, B, C only" or "share outside hierarchy") are not supported in this version. +- Granular per-secret ACLs beyond the sharing modes (e.g., "share with tenants A, B, C only" or sharing outside the hierarchy) +- Hierarchical or policy logic in backend plugins (plugins are pure value stores) +- MySQL as a metadata database ## 5. Functional Requirements @@ -188,21 +204,21 @@ Additionally, the platform runs in multiple environments: Kubernetes (where an e - [ ] `p1` - **ID**: `cpt-cf-credstore-fr-put-secret` -The system **MUST** allow a tenant to store a secret with a reference (key), a value, and a sharing mode. For `tenant` and `shared` modes: if a secret with the same reference already exists for that tenant, the value and sharing mode are updated. For `private` mode: each owner (identified by `subject_id`) can store their own secret under the same reference — multiple private secrets with the same reference can coexist within one tenant (one per owner). A private secret and a tenant/shared secret with the same reference can also coexist. +The system **MUST** allow a tenant to store a secret with a reference (key), a value, and a sharing mode. Two write operations exist: an idempotent upsert and a create-only operation that fails with a conflict when a secret of the same sharing class already exists. For `tenant` and `shared` modes a write updates the single non-private secret for `(tenant, reference)`; for `private` mode each owner has an independent secret under `(tenant, reference, owner)`. A private secret and a tenant/shared secret with the same reference coexist; a write of one sharing class **MUST NOT** affect the other. Changing a secret between `private` and `tenant`/`shared` is rejected as an unsupported transition. -**Rationale**: Core capability — tenants need to manage their own API credentials. +**Rationale**: Core capability — tenants manage their own credentials; the coexistence rule makes private and team secrets independent under common names. **Actors**: `cpt-cf-credstore-actor-tenant-admin`, `cpt-cf-credstore-actor-platform-gear` -#### Retrieve Own Secret +#### Retrieve Secret - [ ] `p1` - **ID**: `cpt-cf-credstore-fr-get-secret` -The system **MUST** allow a tenant to retrieve the decrypted value of their own secret by reference. Returns the secret value or not-found if no secret with that reference exists for the tenant. +The system **MUST** allow a caller to retrieve the decrypted value of an accessible secret by reference, together with access metadata: owning tenant, sharing mode, whether the secret was inherited from an ancestor, and its version. Only fully provisioned (`active`) secrets are visible. Not-found and inaccessible are indistinguishable in the response (a single not-found surface). -**Rationale**: Tenants need to verify or use their own stored credentials. -**Actors**: `cpt-cf-credstore-actor-tenant-admin`, `cpt-cf-credstore-actor-platform-gear` +**Rationale**: Consumers need the value plus enough metadata to understand inheritance and support concurrency control. +**Actors**: `cpt-cf-credstore-actor-tenant-admin`, `cpt-cf-credstore-actor-platform-gear`, `cpt-cf-credstore-actor-oagw` #### Delete Secret @@ -210,9 +226,9 @@ The system **MUST** allow a tenant to retrieve the decrypted value of their own - [ ] `p1` - **ID**: `cpt-cf-credstore-fr-delete-secret` -The system **MUST** allow a tenant to delete their own secret by reference. Descendants using a shared secret lose access immediately upon deletion. +The system **MUST** allow a tenant to delete their own secret by reference (own-tenant only; the private class targets the caller's own private secret). Descendants using a shared secret lose access immediately upon deletion. Deleting a missing backend value is not an error (idempotent delete). -**Rationale**: Tenants must be able to revoke credentials. +**Rationale**: Tenants must be able to revoke credentials reliably. **Actors**: `cpt-cf-credstore-actor-tenant-admin`, `cpt-cf-credstore-actor-platform-gear` @@ -221,9 +237,9 @@ The system **MUST** allow a tenant to delete their own secret by reference. Desc - [ ] `p1` - **ID**: `cpt-cf-credstore-fr-tenant-scoping` -The system **MUST** derive the owning tenant from the request SecurityCtx for all CRUD operations. Tenants MUST NOT create, update, or delete secrets belonging to other tenants. +The system **MUST** derive the operating tenant from the request SecurityContext (`subject_tenant_id`) and the owner from `subject_id` for all operations. Tenants **MUST NOT** create, update, or delete secrets belonging to other tenants. If the caller's authorized scope does not include their own tenant, the operation is denied before any side effect and the denial is recorded (cross-tenant metric). -**Rationale**: Prevents cross-tenant data manipulation. +**Rationale**: Prevents cross-tenant data manipulation; fail-closed before side effects. **Actors**: `cpt-cf-credstore-actor-tenant-admin`, `cpt-cf-credstore-actor-platform-gear` @@ -232,9 +248,9 @@ The system **MUST** derive the owning tenant from the request SecurityCtx for al - [ ] `p1` - **ID**: `cpt-cf-credstore-fr-secretref-validation` -The system **MUST** validate SecretRef format: alphanumeric characters, hyphens, and underscores only (`[a-zA-Z0-9_-]+`). Max length: 255 characters. Colons and other special characters are prohibited. Invalid references are rejected with a validation error. +The system **MUST** validate the secret reference format: `[a-zA-Z0-9_-]+`, 1–255 characters. Invalid references are rejected with a validation error at the API boundary, and the same constraint is enforced by a database `CHECK`. -**Rationale**: Prevents ExternalID collisions in the deterministic mapping (e.g., `base64url("{tenant_id}:{key}")` for tenant/shared, `base64url("{tenant_id}:{key}:p:{owner_id}")` for private). Colons in SecretRef could cause different tenant/key pairs to map to the same ExternalID. +**Rationale**: A restricted, portable key alphabet keeps references safe for every backend key namespace and URL path segment. **Actors**: `cpt-cf-credstore-actor-tenant-admin`, `cpt-cf-credstore-actor-platform-gear` @@ -246,11 +262,11 @@ The system **MUST** validate SecretRef format: alphanumeric characters, hyphens, Each secret **MUST** have a sharing mode: `private`, `tenant` (default), or `shared`. -- `private`: accessible only to the owner (the specific actor identified by `subject_id` from SecurityContext that created the secret) +- `private`: accessible only to the owner (the actor identified by `subject_id` that created the secret) - `tenant`: accessible to all users and services within the owning tenant -- `shared`: accessible to all users in the owning tenant and all descendant tenants in the hierarchy +- `shared`: accessible to all users in the owning tenant and all descendant tenants in the hierarchy (subject to isolation barriers) -**Rationale**: Partners need flexible credential sharing. Personal API keys and sensitive credentials should be owner-only (`private`). Team credentials should be tenant-wide (`tenant`). Platform-level credentials for customer access should be hierarchical (`shared`). +**Rationale**: Partners need flexible credential sharing. Personal API keys should be owner-only (`private`), team credentials tenant-wide (`tenant`), platform-level credentials for customer access hierarchical (`shared`). **Actors**: `cpt-cf-credstore-actor-tenant-admin` @@ -259,13 +275,13 @@ Each secret **MUST** have a sharing mode: `private`, `tenant` (default), or `sha - [ ] `p1` - **ID**: `cpt-cf-credstore-fr-hierarchical-resolve` -The system **MUST** support hierarchical secret resolution: given a secret reference and a tenant_id, the system walks from the specified tenant up through its ancestors (parent, grandparent, ... root), returning the first secret that matches the reference and is accessible (owned by the tenant, or shared by an ancestor). If no accessible secret is found, the system returns not-found. +The system **MUST** resolve a secret reference against the requesting tenant and its ancestor chain (parent, grandparent, … root), returning the closest accessible secret; at the same tenant level the caller's private secret takes precedence over a tenant/shared one. If no accessible secret exists, the system returns not-found. -**Hierarchical Direction**: Resolution is **upward-only** (child → parent → root). A tenant can access ancestor secrets marked as `shared`, but parent tenants **cannot** access child tenant secrets (even if marked as `shared`). This enforces least privilege and enables shadowing. +**Hierarchical direction**: resolution is **upward-only** (child → parent → root). A tenant can access ancestor secrets marked `shared`, but parents **cannot** access child secrets. -**Implementation Note**: For simple plugins (VendorA Credstore, OS keychain), this walk-up algorithm and sharing mode enforcement are implemented in the Gateway gear (credstore). The Gateway queries the tenant hierarchy via `tenant_resolver`, then at each level performs a two-phase lookup: first the Plugin's `get` with the caller's `owner_id` (private secret), then `get` without `owner_id` (tenant/shared secret). These plugins provide simple per-tenant key-value storage without hierarchical logic. The `credentials_storage` plugin implements credential merge resolution internally — when active, the Gateway delegates resolution to the plugin. +**Isolation barriers**: a `shared` secret **MUST NOT** be inherited across a `self_managed` isolation barrier in the tenant hierarchy. -**Rationale**: Enables the core business use case — OAGW retrieves a partner's shared API key when making calls on behalf of a customer. +**Rationale**: Enables the core business use case — OAGW retrieves a partner's shared API key when acting for a customer — without violating platform tenant-isolation guarantees. **Actors**: `cpt-cf-credstore-actor-oagw` @@ -274,11 +290,9 @@ The system **MUST** support hierarchical secret resolution: given a secret refer - [ ] `p1` - **ID**: `cpt-cf-credstore-fr-secret-shadowing` -When a tenant owns a secret with the same reference as an ancestor's shared secret, and that secret is **accessible** to the requester, the tenant's own secret **MUST** take precedence during hierarchical resolution. The ancestor's secret is never checked in this case. - -If the tenant owns a secret with the same reference but it is **inaccessible** to the requester (e.g., `private` mode with owner mismatch, or insufficient permissions), the hierarchical resolution **MUST** continue to check ancestors according to the walk-up algorithm. +When a tenant owns a secret with the same reference as an ancestor's shared secret, and that secret is **accessible** to the requester, the tenant's own secret **MUST** take precedence during hierarchical resolution. If the tenant's same-reference secret is **inaccessible** to the requester (e.g., another owner's `private` secret), resolution **MUST** continue to ancestors. -**Rationale**: Allows customers to override partner defaults with their own credentials while maintaining hierarchical fallback when the tenant's secret is not accessible to the requester. +**Rationale**: Customers can override partner defaults with their own credentials while keeping hierarchical fallback when the local secret is not theirs. **Actors**: `cpt-cf-credstore-actor-oagw` @@ -287,83 +301,103 @@ If the tenant owns a secret with the same reference but it is **inaccessible** t - [ ] `p1` - **ID**: `cpt-cf-credstore-fr-service-retrieve` -The system **MUST** provide a retrieval operation that accepts an explicit tenant_id parameter (not derived from SecurityCtx). This operation is restricted to authorized service accounts (e.g., OAGW). The response **MUST** include the decrypted secret value. +The system **MUST** support retrieval on behalf of an arbitrary tenant by an authorized service account: the service constructs a SecurityContext for the target tenant and calls the standard `get` operation; the PDP decides whether that subject may read in that tenant's scope. The response includes the decrypted value. There is no separate service-to-service operation. -**Implementation Note**: OAGW is a ToolKit gear that uses the standard CredStore SDK client (not a separate integration path). For service-to-service retrieval with explicit tenant_id, OAGW constructs a SecurityCtx with the target tenant_id and calls the Gateway's standard get operation. Hierarchical resolution is implemented in the Gateway gear: the Gateway uses `tenant_resolver` to walk up the tenant hierarchy, performing a two-phase Plugin lookup at each level (private for caller, then tenant/shared) until a matching accessible secret is found. - -**Rationale**: OAGW operates as a service account and needs to retrieve secrets on behalf of arbitrary tenants with hierarchical resolution. +**Rationale**: OAGW operates as a service account and needs hierarchical retrieval for arbitrary tenants through the same audited, policy-checked path. **Actors**: `cpt-cf-credstore-actor-oagw` ### 5.3 P1 — Authorization -#### Read Authorization +#### PDP-Based Authorization -- [ ] `p1` - **ID**: `cpt-cf-credstore-fr-authz-read` +- [ ] `p1` - **ID**: `cpt-cf-credstore-fr-authz-pdp` -The system **MUST** require `Secrets:Read` permission for get and resolve operations. +Every operation **MUST** be authorized through the platform PDP: the gear evaluates an access scope for the operation's action (`read` for get, `write` for put/create, `delete` for delete) against the secret's resolved concrete GTS type (including `generic`), and **MUST** enforce the returned scope on every metadata query at the data layer, enabling per-type policies (e.g., a role that reads `api-key` but not `certificate` secrets). Enforcement is fail-closed: a PDP denial denies the operation; a PDP evaluation failure surfaces as unavailable; out-of-scope or type-denied secrets are indistinguishable from non-existent ones on read. -**Rationale**: Least-privilege access control. +**Rationale**: Real tenant isolation enforced in SQL, consistent with the platform policy plane; least privilege per action. **Actors**: `cpt-cf-credstore-actor-tenant-admin`, `cpt-cf-credstore-actor-oagw`, `cpt-cf-credstore-actor-platform-gear` -#### Write Authorization +#### Gear-Level Enforcement -- [ ] `p1` - **ID**: `cpt-cf-credstore-fr-authz-write` +- [ ] `p1` - **ID**: `cpt-cf-credstore-fr-authz-gear` -The system **MUST** require `Secrets:Write` permission for put and delete operations. +Authorization, sharing-mode enforcement, and hierarchy logic **MUST** live exclusively in the gear. Plugins are pure value stores and **MUST NOT** implement authorization or policy decisions. -**Rationale**: Least-privilege access control. -**Actors**: `cpt-cf-credstore-actor-tenant-admin`, `cpt-cf-credstore-actor-platform-gear` +**Rationale**: Prevents inconsistent authorization behavior across backends; keeps backends trivially simple. +**Actors**: `cpt-cf-credstore-actor-platform-gear`, `cpt-cf-credstore-actor-backend` -#### Gateway-Level Enforcement +### 5.4 P1 — Reliability & Concurrency -- [ ] `p1` - **ID**: `cpt-cf-credstore-fr-authz-gateway` +#### Crash-Safe Write Lifecycle + +- [ ] `p1` - **ID**: `cpt-cf-credstore-fr-write-lifecycle` -Authorization **MUST** be enforced in the gateway layer, not in plugins. Plugins are storage adapters and MUST NOT implement authorization logic. +A secret write that spans metadata and backend **MUST** be crash-safe: a new secret becomes readable only after its value is durably stored in the backend (`provisioning` → `active`); a failed backend write rolls the metadata back; a crash mid-write leaves a non-readable in-flight record that is swept by a periodic reaper within a configurable timeout. No failure mode may leave a readable secret without a value or permanently block the reference. -**Rationale**: Prevents inconsistent authorization behavior across different backends. +**Rationale**: Readers must never observe half-written secrets; writers must never permanently wedge a secret name. **Actors**: `cpt-cf-credstore-actor-platform-gear` -### 5.4 P1 — Encryption & Key Management +#### Optimistic Concurrency -#### External Key Management +- [ ] `p1` - **ID**: `cpt-cf-credstore-fr-optimistic-concurrency` -- [ ] `p1` - **ID**: `cpt-cf-credstore-fr-external-key-mgmt` + +Each secret **MUST** carry a monotonic version, exposed on retrieval. Update and delete **MUST** honour an optional caller-supplied version precondition ("must exist", or "the specified generation must still be current"), enforced atomically with the metadata commit. A failed precondition surfaces as a conflict (lost-update detection); a malformed precondition is a validation error. +**Rationale**: Lost-update detection for concurrent secret management. +**Actors**: `cpt-cf-credstore-actor-tenant-admin`, `cpt-cf-credstore-actor-platform-gear` -The Credentials Storage plugin **SHOULD** support pluggable key management via a `KeyProvider` abstraction. The default implementation stores tenant encryption keys in the same database as encrypted credentials. An optional alternative implementation **MAY** delegate to an external key management service (e.g., HashiCorp Vault, AWS KMS) for production environments where key–data separation is desired. When the external `KeyProvider` is active, encryption keys **MUST** be stored separately from the database containing encrypted credentials, so that a single database compromise does not expose both ciphertext and decryption keys. Plugins that do not implement `KeyProvider` (e.g., VendorA Credstore, OS keychain) are not affected by this requirement. -**Rationale**: A pluggable `KeyProvider` enables operational flexibility — simple single-database deployments by default, with the option to enforce defense-in-depth key–data separation for environments with regulatory or enterprise security requirements. This capability is specific to the Credentials Storage plugin and does not impose changes on existing plugins. -**Actors**: `cpt-cf-credstore-actor-backend` +### 5.5 P1 — Secret Types + +#### GTS-Based Secret Types + +- [ ] `p1` - **ID**: `cpt-cf-credstore-fr-secret-types` + + +Each secret **MUST** have a *secret type* chosen at creation (default: `generic`) and immutable thereafter. Secret types are GTS types derived from the credstore secret base type and registered in the types-registry. Each type declares machine-readable **traits** that the gear enforces uniformly; at minimum: + +- `allow_sharing`: the set of sharing modes permitted for the type. A write requesting a disallowed mode **MUST** be rejected (e.g., `personal-token` secrets are `private`-only and can never be shared). +- `value_schema` (optional): structural validation of the value on write. +- `expirable` (+ optional expiry): expired secrets resolve as not-found. +- `max_size_bytes`, `utf8_only`: value-format constraints. + +The initial type catalog covers `generic`, `api-key`, `personal-token`, `oauth2-client`, `basic-auth`, `bearer-token`, `certificate`, `ssh-key`, `webhook-hmac`, and `connection-string` (see DESIGN §5.3). Untyped existing secrets behave as `generic` with unchanged semantics. Expired secrets of expirable types resolve as not-found and are cleaned up by the reaper through the deprovisioning lifecycle. + +**Rationale**: Different kinds of secrets have different safe-handling rules; encoding them as GTS type traits gives one enforcement point in the gear, platform-native discoverability/versioning, and per-type policy targeting (PDP) without per-secret ACLs. +**Actors**: `cpt-cf-credstore-actor-tenant-admin`, `cpt-cf-credstore-actor-platform-gear` -### 5.5 P2 — Planned +### 5.6 P1 — Deprovisioning Lifecycle -#### OS Protected Storage Plugin +#### Crash-Safe Delete (Deprovisioning Saga) -- [ ] `p2` - **ID**: `cpt-cf-credstore-fr-os-storage` +- [ ] `p1` - **ID**: `cpt-cf-credstore-fr-deprovisioning` -The system **MUST** provide an OS-protected storage plugin for desktop/VM environments using platform-native secure storage (macOS Keychain, Windows DPAPI). This plugin supports basic per-tenant get/put/delete operations. In desktop/single-tenant environments, hierarchical resolution is not applicable (no multi-tenant hierarchy exists), so the Gateway returns only the requesting tenant's own secrets. +Secret deletion **MUST** be a crash-safe lifecycle symmetric to provisioning: the secret first enters a `deprovisioning` status — at which instant it atomically stops resolving — then the backend value is deleted, then the metadata record is removed. A failure or crash at any step leaves a non-readable `deprovisioning` record that (a) a client retry of the delete resumes idempotently, and (b) the reaper completes within a configurable timeout. While a reference is deprovisioning, re-creating it **MUST** fail with a retryable conflict (the name is released only after backend cleanup completes). -**Rationale**: Desktop/VM environments lack access to VendorA Credstore. -**Actors**: `cpt-cf-credstore-actor-platform-gear` +**Rationale**: A plain backend-first delete leaves metadata/backend divergence on partial failure with no self-healing owner; the status-driven saga plus reaper makes revocation reliable and observable, and closes the orphaned-backend-value debt of the write saga (the reaper reconciles backend values for all reaped records). +**Actors**: `cpt-cf-credstore-actor-tenant-admin`, `cpt-cf-credstore-actor-platform-gear` -#### Read-Only / Read-Write Credential Separation +### 5.7 P2 — Planned -- [ ] `p2` - **ID**: `cpt-cf-credstore-fr-rw-separation` +#### Production Value-Store Backend + +- [ ] `p2` - **ID**: `cpt-cf-credstore-fr-production-backend` -The VendorA Credstore plugin **MUST** support optional separate OAuth2 client credentials for read-only and read-write operations, enabling least-privilege deployment configurations. +The system **MUST** provide at least one production-grade value-store plugin (external secret vault, KMS-backed store, or OS-protected storage for desktop/VM environments) implementing the same plugin contract as the development in-memory plugin. Backend selection remains a deployment-time configuration with no consumer-visible change. -**Rationale**: Production environments benefit from separate credentials: RO for get/resolve, RW for put/delete. +**Rationale**: The in-memory static plugin is suitable for development and testing only (values do not survive process restart). **Actors**: `cpt-cf-credstore-actor-backend` @@ -376,11 +410,34 @@ The VendorA Credstore plugin **MUST** support optional separate OAuth2 client cr - [ ] `p1` - **ID**: `cpt-cf-credstore-nfr-confidentiality` -Secret values **MUST NOT** appear in logs, error messages, or debug output at any level (gateway, plugin, transport). Secret values **MUST** be encrypted at rest in the backend storage. +Secret values **MUST NOT** appear in logs, error messages, or debug output at any level (gear, plugin, transport), **MUST NOT** be cacheable by intermediaries, and **MUST NOT** be silently corrupted (non-UTF-8 values are rejected on the string transport rather than lossily decoded). Secret memory is zeroized on drop. **Threshold**: Zero plaintext secret values in any log output -**Rationale**: Secrets are the most sensitive data in the platform. Leaking them through logs or error messages would be a critical security incident. -**Architecture Allocation**: See DESIGN.md for implementation approach +**Rationale**: Secrets are the most sensitive data in the platform. +**Architecture Allocation**: See DESIGN.md §3.2 for the implementation approach + + +#### Tenant Isolation + +- [ ] `p1` - **ID**: `cpt-cf-credstore-nfr-tenant-isolation` + + +No operation may read or modify secret metadata outside the caller's PDP-authorized tenant scope; enforcement happens at the data layer on every query. Inaccessible secrets are indistinguishable from non-existent ones (anti-enumeration). + +**Threshold**: Zero cross-tenant reads/writes outside the authorized scope +**Rationale**: Multi-tenant platform guarantee. +**Architecture Allocation**: PDP scope + data-layer clamps; see DESIGN.md §3.1 + + +#### Observability + +- [ ] `p1` - **ID**: `cpt-cf-credstore-nfr-observability` + + +The gear **MUST** emit operational metrics sufficient to detect resolution anomalies and lifecycle divergence: walk-up depth, read outcome (own/inherited/miss), per-dependency latency and outcome (PDP, tenant-resolver, plugin), cross-tenant denials, saga rollback/reap counters, and per-status inventory gauges. Metric labels **MUST NOT** contain secret references or values. + +**Rationale**: Sagas and hierarchical resolution fail in partial, quiet ways; operators need signals, not log archaeology. +**Architecture Allocation**: See DESIGN.md §10 Observability ## 7. Public Library Interfaces @@ -394,7 +451,7 @@ Secret values **MUST NOT** appear in logs, error messages, or debug output at an **Type**: Rust trait (async) **Stability**: stable -**Description**: Public API for platform gears to store, retrieve, and delete secrets. Registered in ClientHub without scope. Operations: `get`, `put`, `delete`. Hierarchical secret resolution is implemented in the Gateway gear and accessed through standard `get` operations. +**Description**: Public API for platform gears. Registered in ClientHub without scope. Operations: `get` (hierarchical read returning value + metadata: owning tenant, sharing, inherited flag, version, secret type, expiry), `put`/`create` (upsert / create-only) plus typed-options variants accepting write options (secret type, expiry), `delete`. Hierarchical resolution is internal to the gear. **Breaking Change Policy**: Major version bump required @@ -405,7 +462,7 @@ Secret values **MUST NOT** appear in logs, error messages, or debug output at an **Type**: Rust trait (async) **Stability**: unstable -**Description**: Plugin SPI for backend implementations. Registered in ClientHub with GTS instance scope. Operations: `get` (takes optional `owner_id` to distinguish private vs tenant/shared lookup; returns SecretMetadata), `put` (stores value with metadata; ExternalID derived from sharing mode), `delete` (takes optional `owner_id`). The Plugin must return metadata so the Gateway can enforce sharing mode access control rules. +**Description**: Plugin SPI for backend value stores. Registered in ClientHub with GTS instance scope. Operations: `get`/`put`/`delete` keyed by `(tenant_id, key, owner_id: Option)` where `Some(owner)` addresses the owner's private key class and `None` the tenant key class. Returns the value only — no metadata, no policy. **Breaking Change Policy**: Minor version bump (unstable API) @@ -416,19 +473,19 @@ Secret values **MUST NOT** appear in logs, error messages, or debug output at an - [ ] `p1` - **ID**: `cpt-cf-credstore-contract-rest-api` -**Direction**: provided by library -**Protocol/Format**: HTTP/REST, JSON -**Compatibility**: Versioned URL path (`/credstore/v1/...`), backward-compatible within major version +**Direction**: provided +**Protocol/Format**: HTTP/REST, JSON, canonical `Problem` error envelope, under a versioned path served beneath the platform API prefix. Exposes create-only and idempotent-upsert writes, retrieval, and delete over the secret reference, with optional secret-type and expiry inputs on writes, version-precondition support on update/delete, and value-confidentiality response controls. See DESIGN.md for the concrete endpoints, methods, status codes, and headers. +**Compatibility**: Backward-compatible within major version -#### VendorA Credstore REST +#### GTS Registration -- [ ] `p1` - **ID**: `cpt-cf-credstore-contract-vendor_a-rest` +- [ ] `p1` - **ID**: `cpt-cf-credstore-contract-gts` -**Direction**: required from client (outbound to VendorA Credstore) -**Protocol/Format**: HTTP/REST, JSON. OAuth2 client credentials for authentication. Credstore RAML API subset. -**Compatibility**: Plugin adapts to Credstore API version. Breaking Credstore API changes require plugin update. +**Direction**: provided to types-registry +**Protocol/Format**: GTS link-time inventory. Registered types: the plugin spec, the secret resource type used by the PDP (carrying the secret-type traits schema), and the derived secret-type family (traits mirrored as `x-gts-traits`). See DESIGN §5 for the concrete type ids. +**Compatibility**: Type ids are stable identifiers; new versions are new ids ## 8. Use Cases @@ -441,19 +498,21 @@ Secret values **MUST NOT** appear in logs, error messages, or debug output at an **Actor**: `cpt-cf-credstore-actor-tenant-admin` **Preconditions**: -- Tenant is authenticated with `Secrets:Write` permission +- Tenant is authenticated; PDP authorizes `write` on secrets in the tenant's scope **Main Flow**: -1. Partner tenant calls put with reference `partner-openai-key`, value `PARTNER_DEMO_KEY_XYZ`, sharing `shared` -2. Gateway verifies `Secrets:Write` authorization -3. Gateway delegates to plugin -4. Plugin stores secret in backend +1. Partner tenant stores `partner-openai-key` with a value and sharing `shared` +2. Gear evaluates the PDP write scope and the own-tenant gate +3. Gear runs the write saga: provisioning row → backend value write → active +4. Secret is immediately resolvable by the partner and descendant tenants not separated by an isolation barrier **Postconditions**: -- Secret is stored and accessible to partner and all descendant tenants +- Secret is stored and accessible to partner and descendants **Alternative Flows**: -- **Secret already exists**: Value and sharing mode are updated +- **Secret already exists (same class)**: value/sharing updated, version bumped +- **Create-only write**: fails with a conflict if the reference is taken in that sharing class +- **Backend write fails**: provisioning row rolled back; reference not wedged; caller retries #### UC-002: OAGW Retrieves Secret for Customer (Hierarchical Resolution) @@ -464,28 +523,25 @@ Secret values **MUST NOT** appear in logs, error messages, or debug output at an **Actor**: `cpt-cf-credstore-actor-oagw` **Preconditions**: -- OAGW has valid service token with `Secrets:Read` permission -- Partner tenant has created a shared secret with reference `partner-openai-key` -- Customer is a descendant of partner in tenant hierarchy +- OAGW holds a service identity authorized to read secrets in the customer's scope +- Partner has a `shared` secret `partner-openai-key`; customer is a descendant of partner **Main Flow**: -1. OAGW constructs SecurityCtx for `customer-123` tenant and calls Gateway's `get` with reference `partner-openai-key` -2. Gateway verifies service authorization (`Secrets:Read` permission) -3. Gateway extracts tenant_id from SecurityCtx (`customer-123`) -4. Gateway queries `tenant_resolver` to get ancestor chain: `[customer-123, partner-acme, root]` -5. Gateway walks up hierarchy: calls Plugin `get(customer-123, "partner-openai-key")` → not found -6. Gateway tries next ancestor: calls Plugin `get(partner-acme, "partner-openai-key")` → found with `sharing: shared` -7. Gateway checks sharing mode: `shared` allows access by descendants → returns secret value to OAGW -8. OAGW uses the secret value for upstream API call +1. OAGW constructs a SecurityContext for `customer-123` and calls `get("partner-openai-key")` +2. Gear evaluates the PDP read scope for that context +3. Gear obtains the customer's barrier-respecting ancestor chain (cached) +4. Gear resolves the reference against the whole chain in one metadata query → partner's `shared` row wins (customer has none) +5. Gear reads the value from the plugin for the winning row only +6. OAGW receives the value plus metadata (owning tenant = partner, inherited = true, version) **Postconditions**: -- OAGW has the decrypted secret. Customer never sees the actual value. -- Hierarchical resolution is transparent to OAGW (handled by Gateway) +- OAGW has the decrypted secret; the customer never sees the value +- Resolution depth and inherited-read outcome are recorded as metrics **Alternative Flows**: -- **Customer has own secret**: Customer's secret returned (shadowing), parent not checked -- **Secret is private**: Walk-up continues to next ancestor -- **No secret in hierarchy**: Not-found error returned +- **Customer has own accessible secret**: it wins (shadowing); the parent row is not considered +- **Secret is above an isolation barrier**: not inherited → not-found +- **No accessible secret in the chain**: not-found #### UC-003: Customer Overrides Parent Secret (Shadowing) @@ -496,65 +552,58 @@ Secret values **MUST NOT** appear in logs, error messages, or debug output at an **Actor**: `cpt-cf-credstore-actor-tenant-admin` **Preconditions**: -- Partner has shared secret with reference `partner-openai-key` -- Customer is a descendant of partner +- Partner has shared secret `partner-openai-key`; customer is a descendant **Main Flow**: -1. Customer creates own secret with same reference `partner-openai-key`, value `CUSTOMER_DEMO_KEY_ABC`, sharing `tenant` -2. OAGW calls resolve for `partner-openai-key`, tenant_id `customer-123` -3. System finds customer's own secret first — returns `CUSTOMER_DEMO_KEY_ABC` -4. Partner's secret is never checked +1. Customer creates own secret with the same reference (sharing `tenant`) +2. OAGW resolves `partner-openai-key` for the customer +3. The customer's row is closer in the chain → customer's value returned +4. Partner's secret remains available to other descendants **Postconditions**: -- Customer uses own key. Partner's shared secret remains available to other descendants. +- Customer uses its own key; partner's shared secret is unaffected **Alternative Flows**: -- **Customer uses private mode**: If customer creates secret with `sharing: private`, it's accessible only to the customer's user/service that created it (owner-only) +- **Customer uses `private` mode**: the override applies only to the creating owner; other subjects in the customer tenant still resolve the partner's shared secret -#### UC-004: Private Secret Access Denied vs Shadowing with Fallback +#### UC-004: Private Secret Access & Fallback - [ ] `p1` - **ID**: `cpt-cf-credstore-usecase-private-denied` **Actor**: `cpt-cf-credstore-actor-oagw` -**Scenario A: Parent's private secret (no shadowing)** +**Scenario A: Parent's private secret (no leak)** **Preconditions**: -- Partner has secret with reference `internal-admin-key`, sharing `private`, owned by PartnerAdmin -- Customer is a descendant of partner, has no own secret with this reference +- Partner has `internal-admin-key` with sharing `private` (owned by PartnerAdmin); customer has no secret with this reference **Main Flow**: -1. OAGW calls resolve for `internal-admin-key`, tenant_id `customer-123` -2. Gateway walks up: Customer — phase 1 (private for OAGW): miss. Phase 2 (tenant/shared): miss. -3. Partner — phase 1 (private for OAGW): miss (OAGW is not the owner). Phase 2 (tenant/shared): miss (no tenant/shared secret). -4. Walk-up exhausted — system returns not-found error (404) +1. OAGW resolves `internal-admin-key` for the customer +2. The resolution query only matches private rows owned by the requesting subject; PartnerAdmin's row is invisible to OAGW +3. No row matches → not-found **Postconditions**: -- Customer cannot access partner's private secret. OAGW never sees it because the ExternalID doesn't match. +- A parent's private secret is never disclosed to descendants or other subjects **Scenario B: Another user's private secret with fallback to parent's shared** **Preconditions**: -- Customer has secret `api-key`, sharing `private`, owner_id: User A -- Partner (parent) has secret `api-key`, sharing `shared` -- User B in customer tenant requests `api-key` +- Customer has `api-key` (sharing `private`, owner User A); partner has `api-key` (sharing `shared`); User B in the customer tenant requests `api-key` **Main Flow**: -1. User B calls get for `api-key` in customer tenant -2. Gateway phase 1 for customer: looks up private `api-key` for User B → miss (no private secret for User B) -3. Gateway phase 2 for customer: looks up tenant/shared `api-key` → miss (no tenant/shared secret at customer level) -4. Gateway walks up to parent (partner) — phase 2: finds `api-key` with sharing `shared` -5. Shared secret is accessible to descendants — return partner's secret value +1. User B calls `get("api-key")` +2. User A's private row is invisible to User B; the customer tenant has no tenant/shared row +3. The partner's `shared` row is the closest accessible match → returned **Postconditions**: -- User B accesses the partner's shared secret as fallback. User A's private secret is invisible to User B (stored under a different ExternalID). +- User B falls back to the partner's shared secret; User A's private secret stays invisible -**Rationale**: Private secrets are namespaced per-owner via ExternalID. The two-phase lookup (private for caller → tenant/shared) ensures that inaccessible private secrets don't block fallback to ancestor shared secrets. +**Rationale**: Private secrets are per-owner; inaccessible private rows never block fallback to ancestor shared secrets. -#### UC-005: Tenant CRUD Own Secrets +#### UC-005: Tenant CRUD Own Secrets (with Concurrency Control) - [ ] `p1` - **ID**: `cpt-cf-credstore-usecase-crud` @@ -562,21 +611,22 @@ Secret values **MUST NOT** appear in logs, error messages, or debug output at an **Actor**: `cpt-cf-credstore-actor-tenant-admin` **Preconditions**: -- Tenant is authenticated with appropriate permissions +- Tenant is authenticated with PDP-authorized read/write/delete scope **Main Flow**: -1. Tenant creates secret: put(reference, value, sharing) — `sharing` can be `private`, `tenant` (default), or `shared` -2. Tenant reads secret: get(reference) — returns value (subject to sharing mode access control) -3. Tenant updates secret: put(reference, new_value, new_sharing) — overwrites value and/or sharing mode -4. Tenant deletes secret: delete(reference) — removed +1. Create (create-only) or upsert a secret by reference +2. Read the secret → value + metadata + current version +3. Guarded update with a version precondition → success, or conflict on a stale version +4. Guarded delete with an optional version precondition → success **Postconditions**: -- Secret lifecycle managed. Descendants of shared secrets lose access on delete. +- Secret lifecycle managed; descendants of shared secrets lose access on delete **Alternative Flows**: -- **Get non-existent secret**: Not-found error -- **Delete non-existent secret**: Not-found error (Gateway returns 404 regardless of plugin behavior) -- **Get private secret created by different owner**: Not-found error (owner mismatch returns 404 to prevent enumeration) +- **Get/delete non-existent secret**: not-found +- **Get another owner's private secret**: not-found (anti-enumeration) +- **Stale version precondition**: conflict, no changes applied +- **Malformed version precondition**: validation error #### UC-006: Owner-Only Private Secret Access Control @@ -587,84 +637,124 @@ Secret values **MUST NOT** appear in logs, error messages, or debug output at an **Actor**: `cpt-cf-credstore-actor-tenant-admin` **Preconditions**: -- User A and User B are both authenticated users in the same tenant -- Both have `Secrets:Write` permission +- User A and User B are authenticated users in the same tenant with write access + +**Main Flow**: +1. User A stores `my-personal-api-key` with sharing `private` → row keyed `(tenant, ref, ownerA)` +2. User B stores the same reference with sharing `private` → independent row `(tenant, ref, ownerB)`, no conflict +3. Each user's `get` resolves their own private secret + +**Postconditions**: +- Independent per-owner private secrets under one reference; no cross-owner visibility + +**Alternative Flows**: +- **User C (no private secret) reads the reference**: falls back to the tenant/shared secret or not-found +- **User B attempts to delete User A's private secret**: deletes address only the caller's own class → User A's secret is untouched (User B gets not-found if they have none) + + +#### UC-007: Type-Restricted Sharing + +- [ ] `p1` - **ID**: `cpt-cf-credstore-usecase-type-restricted-sharing` + + +**Actor**: `cpt-cf-credstore-actor-tenant-admin` + +**Preconditions**: +- The `personal-token` secret type is registered with trait `allow_sharing = [private]` **Main Flow**: -1. User A calls put with reference `my-personal-api-key`, value `user-a-demo-key`, sharing `private` -2. Gateway stores secret with `owner_id = UserA.subject_id` (private ExternalID includes owner_id) -3. User B calls put with same reference `my-personal-api-key`, value `user-b-demo-key`, sharing `private` -4. Gateway stores a separate secret with `owner_id = UserB.subject_id` (different ExternalID — no conflict) -5. User A calls get for `my-personal-api-key` — Gateway looks up private secret for UserA → returns `user-a-demo-key` -6. User B calls get for `my-personal-api-key` — Gateway looks up private secret for UserB → returns `user-b-demo-key` +1. User stores a secret with type `personal-token` and sharing `private` → accepted +2. User (or a later update) attempts sharing `tenant` or `shared` for the same type → rejected (`SHARING_NOT_ALLOWED_FOR_TYPE`) +3. Retrieval returns `type: personal-token` in metadata **Postconditions**: -- Each user has their own independent private secret under the same reference -- Users cannot see or access each other's private secrets +- Personal tokens can never be widened beyond their owner, regardless of caller permissions **Alternative Flows**: -- **Owner updates secret**: User A can update value or change sharing mode to `tenant` or `shared` -- **Non-owner attempts delete**: User B cannot delete User A's private secret (owner-only authorization) -- **No private secret for caller**: If User C (who never created a private secret) calls get, Gateway falls back to tenant/shared secret or returns not-found (404) +- **Type omitted**: defaults to `generic` (all sharing modes allowed — current behavior) +- **Attempt to change the type of an existing secret**: rejected as unsupported transition + + +#### UC-008: Reliable Revocation via Deprovisioning + +- [ ] `p1` - **ID**: `cpt-cf-credstore-usecase-deprovisioning` + + +**Actor**: `cpt-cf-credstore-actor-tenant-admin` + +**Preconditions**: +- Tenant owns an `active` secret consumed by descendants + +**Main Flow**: +1. Tenant deletes the secret by reference +2. The secret enters `deprovisioning` — it instantly stops resolving for every consumer +3. The backend value is deleted; the metadata record is removed + +**Postconditions**: +- Secret fully revoked; the reference becomes reusable -**Rationale**: Personal API keys and sensitive user-specific credentials should not be shared tenant-wide. The `private` sharing mode enables users to store secrets under common names (e.g., `github-token`) without namespace conflicts between owners. +**Alternative Flows**: +- **Backend delete fails**: caller gets a retryable failure; the secret already does not resolve; a delete retry or the reaper completes cleanup +- **Create during deprovisioning**: retryable conflict until the backend value is cleaned up (bounded by the reaper cadence) +- **Crash mid-delete**: the reaper finishes the saga within the configured timeout ## 9. Acceptance Criteria - [ ] Tenant can store, retrieve, and delete secrets via both ClientHub and REST API -- [ ] Private secrets are accessible only to the owner (subject_id match required) -- [ ] Multiple users in the same tenant can each store a private secret under the same reference without conflict -- [ ] Tenant secrets are accessible to all users/services within the owning tenant -- [ ] Shared secrets are accessible to the owning tenant and descendant tenants via hierarchical resolution -- [ ] Secret shadowing works: child's own secret takes precedence over parent's -- [ ] OAGW can retrieve secrets on behalf of any tenant using service-to-service authentication -- [ ] Authorization is enforced at the gateway level for all operations -- [ ] Secret values never appear in log output -- [ ] Owner ID is captured from SecurityContext.subject_id() for all secret creation operations +- [ ] Create-only writes conflict on a same-class duplicate; upserts are idempotent +- [ ] Private secrets are accessible only to the owner; multiple owners can hold private secrets under one reference; a private and a tenant/shared secret coexist under one reference +- [ ] Tenant secrets are accessible to all subjects within the owning tenant and never inherited; shared secrets are inherited by descendants, bounded by isolation barriers +- [ ] Shadowing: the closest accessible secret wins; inaccessible private rows do not block fallback +- [ ] OAGW can retrieve secrets on behalf of any tenant it is authorized for, through the standard API +- [ ] Every operation is PDP-authorized and scope-clamped at the data layer; inaccessible reads are not-found; operation-level denial is refused; a PDP outage fails closed +- [ ] Half-written secrets are never readable; failed writes roll back; stuck lifecycle rows are reaped within the configured timeout +- [ ] Retrieval exposes the current version and value-confidentiality controls; update/delete honour the version precondition with a conflict on stale versions +- [ ] Secret values never appear in log output or metric labels; non-UTF-8 values are rejected on the REST transport, not corrupted +- [ ] Secret types: a write violating the type's `allow_sharing`, `value_schema`, size/format, or expiry traits is rejected with a stable reason; the type is immutable, defaults to `generic`, and is returned in metadata; expired secrets resolve as not-found and are reaped +- [ ] Deprovisioning: a deleted secret stops resolving atomically at delete start; partial delete failures self-heal via retry or reaper; the reference conflicts (retryably) until cleanup completes ## 10. Dependencies | Dependency | Description | Criticality | |------------|-------------|-------------| -| VendorA Credstore | External Go service for secret persistence (Kubernetes environments) | `p1` | -| OAGW | Primary consumer of hierarchical secret retrieval (uses CredStore SDK client) | `p1` | -| OAuth/token provider | Shared component for Credstore REST authentication tokens | `p1` | -| `tenant_resolver` | Provides tenant hierarchy information (used by Gateway gear for hierarchical resolution walk-up) | `p1` | -| `types_registry` | GTS-based plugin registration and discovery | `p1` | -| External Key Service | External key management service (Vault, KMS) for tenant key storage when `ExternalKeyProvider` is active. Required for production deployments with key–data separation. | `p1` | +| `authz-resolver` | PDP: per-operation access-scope evaluation (fail-closed) | `p1` | +| `tenant-resolver` | Barrier-aware tenant ancestor chains for hierarchical resolution | `p1` | +| `types-registry` | GTS plugin discovery; secret resource type + secret-type registrations | `p1` | +| Database (PostgreSQL / SQLite) | Gear-owned secret metadata (`credstore_secrets`) | `p1` | +| Value-store plugin | Per-tenant secret value persistence (`static-credstore-plugin` for dev/test; production vault plugin planned) | `p1` | +| OAGW | Primary consumer of hierarchical secret retrieval (uses the SDK client) | `p1` | ## 11. Assumptions -- For simple plugins (VendorA, OS keychain), hierarchical secret resolution (walk-up algorithm and sharing mode enforcement) is implemented in the Gateway gear (credstore). The `credentials_storage` plugin handles merge resolution internally. -- Simple plugins and their backends provide per-tenant key-value storage without hierarchical logic. The `credentials_storage` plugin is a full microservice with its own resolution, encryption, and authorization. -- OAGW is a ToolKit gear that uses the standard CredStore SDK client (all access flows through Gateway→Plugin→Backend) -- Gateway provides tenant-scoped CRUD operations, hierarchical resolution, and routes requests to the active storage plugin -- Tenant hierarchy is managed externally and accessible via `tenant_resolver` (used by Gateway for hierarchical walk-up) -- `sharing` field is stored in VendorA Credstore schema as metadata (used by Gateway for access control decisions) -- One storage plugin is active per deployment -- Tenant encryption keys are managed by a pluggable `KeyProvider` (Credentials Storage plugin); default is local database storage, with optional external key management service for key–data separation +- The gear owns all secret metadata; backends store values only and provide per-tenant key-value CRUD without hierarchical or policy logic +- Exactly one value-store plugin is active per deployment (GTS vendor match) +- Tenant hierarchy (including isolation barriers) is managed externally and served by `tenant-resolver`; short-TTL caching of ancestor chains is acceptable +- The PDP is the sole authorization authority; there is no local policy cache (policy freshness over availability) +- Consumers provisioning infrastructure from secrets at startup (e.g., mini-chat → OAGW upstreams) tolerate missing secrets by degrading per-provider rather than failing boot +- OAGW is a ToolKit gear that uses the standard CredStore SDK client (all access flows through Gear → Plugin) ## 12. Risks | Risk | Impact | Mitigation | |------|--------|------------| -| Credstore API changes break plugin | Plugin stops working, secrets inaccessible | Pin Credstore API version, integration tests against Credstore | -| Secret values leaked through logs | Critical security incident | NFR enforcement, code review, log scrubbing | -| Hierarchy walk-up performance at deep nesting | Increased latency for resolve operations | Gateway implements efficient walk-up with early termination; cache tenant hierarchy queries; monitor resolution depth | -| ExternalID encoding collision | Wrong secret returned | Deterministic encoding with base64url; comprehensive test coverage | -| External key service unavailability | All encrypt/decrypt operations blocked; credential reads and writes fail | High-availability key service deployment; readiness probe reflects KMS connectivity; key caching with short TTL for read-path resilience; circuit breaker for key service calls | -| Keys co-located with encrypted data (DatabaseKeyProvider) | Single breach exposes both ciphertext and keys | Use `ExternalKeyProvider` in production; `DatabaseKeyProvider` restricted to development/test environments by deployment policy | +| Secret values leaked through logs/caches | Critical security incident | NFR enforcement (redaction, zeroize, no-store responses), code review | +| Metadata/backend divergence on partial saga failure | Orphaned backend values, temporarily wedged references | Compensating rollback; deprovisioning saga; reaper backend reconciliation with configurable timeouts; saga metrics | +| PDP or tenant-resolver outage | Operations fail closed (unavailable) | Ancestor-chain cache absorbs blips; dependency metrics for fast diagnosis | +| Ancestor-chain cache staleness | Briefly stale hierarchy after re-parenting | Short TTL + LRU; PDP scope still clamps every query | +| In-memory static plugin in non-dev use | Secret values lost on restart | Production vault plugin (`cpt-cf-credstore-fr-production-backend`); deployment policy | +| Type-trait misconfiguration | Overly permissive or broken writes for a type | Compiled-in catalog pinned to registered GTS schemas by unit tests; catalog changes are code-reviewed SDK releases; `generic` keeps legacy behavior | ## 13. Open Questions -- **P2/Future - Human vs Service Access**: Should human users (tenant admins via UI) be restricted from retrieving raw secret values for inherited shared secrets, while service accounts (OAGW) can retrieve them? Constructor pattern: tenant admins can see metadata (reference, sharing mode, owner) but cannot get the decrypted value for shared secrets. This would require distinguishing human vs service authentication in SecurityCtx and different authorization rules. -- **P2/Future - Audit Trails**: All credential operations (create, read, update, delete, access) should leave audit trails with timestamps, actor, tenant, and outcome. Audit entries must never contain plaintext secret values. Audit logs should be tamper-evident and stored securely. -- **P2/Future - Schema Validation**: Should Constructor-managed secrets support JSON schema validation (using GTS)? On create/update, validate secret structure against registered schema. Useful for complex credentials (SMTP config, OAuth client credentials) where multiple fields must be present and correctly formatted. Schema can also specify raw string values for simple secrets. -- **P2/Future**: Should secret updates support atomic compare-and-swap (CAS) validation? VendorA Credstore has a newer endpoint that validates the current secret value before allowing updates, preventing race conditions. This could be added as an optional parameter to the `put` operation without breaking existing clients. +- **Batch retrieval**: should `get` support multiple references per call for OAGW efficiency? (Single-query resolution makes this cheap on the metadata side.) +- **P2/Future — Human vs service access**: should human users be restricted to metadata-only for inherited shared secrets while service accounts can read values? +- **P2/Future — Audit trails**: structured audit events (actor, tenant, outcome — never values) to a tamper-evident platform sink. +- **P2/Future — Metadata list endpoint**: a values-free list becomes cheap with gear-owned metadata; must be reconciled with the anti-enumeration stance and per-type authorization. +- **Dynamic type descriptors** (see DESIGN §9): resolve traits from the types-registry at runtime so vendors can add types without an SDK release. ## 14. Traceability - **Design**: [DESIGN.md](./DESIGN.md) -- **ADRs**: [ADR/](./ADR/) -- **Features**: [features/](./features/) \ No newline at end of file +- **ADRs**: [ADR/](./ADR/) — [ADR-0001 stateful gear](./ADR/0001-cpt-cf-credstore-adr-stateful-gear.md), [ADR-0002 deprovisioning saga](./ADR/0002-cpt-cf-credstore-adr-deprovisioning-saga.md), [ADR-0003 value-fingerprint fence](./ADR/0003-cpt-cf-credstore-adr-value-fingerprint-fence.md) +- **Features**: features/ (planned) diff --git a/gears/credstore/docs/api/openapi.yaml b/gears/credstore/docs/api/openapi.yaml new file mode 100644 index 000000000..26e64fd89 --- /dev/null +++ b/gears/credstore/docs/api/openapi.yaml @@ -0,0 +1,432 @@ +# Generated from docs/api/api.json (make openapi) — do not edit by hand. +openapi: 3.1.0 +info: + title: CredStore API + version: 0.1.0 + description: Stateful, tenant-scoped credential-storage gear. Extracted (credstore paths + referenced + components) from the platform OpenAPI document `docs/api/api.json`; regenerate via `make openapi`. +paths: + /credstore/v1/secrets: + post: + description: Create a new secret for the authenticated tenant. + operationId: credstore.create_secret + requestBody: + content: + application/json: + schema: + $ref: '#/components/schemas/CreateSecretRequestDto' + description: Secret reference, value, and sharing mode + required: true + responses: + '201': + description: Secret created (see Location header) + '400': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Bad Request + '401': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Unauthorized + '403': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Forbidden + '409': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Conflict + '500': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Internal Server Error + '503': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Service Unavailable + security: + - bearerAuth: [] + summary: Create a secret + tags: + - Credential Store + /credstore/v1/secrets/{ref}: + delete: + description: Delete a secret owned by the authenticated tenant. + operationId: credstore.delete_secret + parameters: + - description: Secret reference (`[a-zA-Z0-9_-]+`, maximum length 255 characters) + in: path + name: ref + required: true + schema: + type: string + - description: Optimistic-concurrency precondition (RFC 7232). `*` requires the secret to exist; + a quoted `"."` ETag requires the current version to match, otherwise the request + fails with `409 OPTIMISTIC_LOCK_FAILURE`. + in: header + name: If-Match + required: false + schema: + type: string + responses: + '204': + description: Secret deleted + '400': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Bad Request + '401': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Unauthorized + '403': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Forbidden + '404': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Not Found + '409': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Conflict + '500': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Internal Server Error + '503': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Service Unavailable + security: + - bearerAuth: [] + summary: Delete a secret by reference + tags: + - Credential Store + get: + description: Retrieve a secret for the authenticated tenant, with walk-up resolution. + operationId: credstore.get_secret + parameters: + - description: Secret reference (`[a-zA-Z0-9_-]+`, maximum length 255 characters) + in: path + name: ref + required: true + schema: + type: string + responses: + '200': + content: + application/json: + schema: + $ref: '#/components/schemas/GetSecretResponseDto' + description: Resolved secret value and metadata + '400': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Bad Request + '401': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Unauthorized + '403': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Forbidden + '404': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Not Found + '500': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Internal Server Error + '503': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Service Unavailable + security: + - bearerAuth: [] + summary: Get a secret by reference + tags: + - Credential Store + put: + description: Store a secret for the authenticated tenant. + operationId: credstore.put_secret + parameters: + - description: Secret reference (`[a-zA-Z0-9_-]+`, maximum length 255 characters) + in: path + name: ref + required: true + schema: + type: string + - description: Optimistic-concurrency precondition (RFC 7232). `*` requires the secret to exist; + a quoted `"."` ETag requires the current version to match, otherwise the request + fails with `409 OPTIMISTIC_LOCK_FAILURE`. + in: header + name: If-Match + required: false + schema: + type: string + requestBody: + content: + application/json: + schema: + $ref: '#/components/schemas/UpdateSecretRequestDto' + description: Secret value and sharing mode + required: true + responses: + '204': + description: Secret stored + '400': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Bad Request + '401': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Unauthorized + '403': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Forbidden + '409': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Conflict + '500': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Internal Server Error + '503': + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + description: Service Unavailable + security: + - bearerAuth: [] + summary: Create or update a secret by reference + tags: + - Credential Store +components: + schemas: + CreateSecretRequestDto: + additionalProperties: false + description: 'Request body for `POST /credstore/v1/secrets`. + + + `Debug` is hand-written to redact `value` — a derived `Debug` would expose + + the plaintext secret if this DTO is ever `{:?}`-logged by a future layer.' + properties: + expires_at: + description: Expiry instant (RFC 3339); only for expirable types. + format: date-time + type: + - string + - 'null' + reference: + description: Secret reference key — `[a-zA-Z0-9_-]+`, max 255 characters. + maxLength: 255 + minLength: 1 + pattern: ^[A-Za-z0-9_-]+$ + type: string + sharing: + $ref: '#/components/schemas/SharingModeDto' + description: Sharing mode for the secret. + type: + description: 'Secret type as a full GTS type id (built-in or custom); defaults to + + the generic type. Resolved and existence-checked against the + + types-registry.' + type: + - string + - 'null' + value: + description: Secret value as a UTF-8 string. + type: string + required: + - reference + - value + type: object + GetSecretResponseDto: + description: 'Response body for `GET /credstore/v1/secrets/{ref}`. + + + `Debug` is hand-written to redact `value` (see [`CreateSecretRequestDto`]).' + properties: + metadata: + $ref: '#/components/schemas/SecretMetadataDto' + description: Access metadata for the resolved secret. + value: + description: Secret value as a UTF-8 string. + type: string + required: + - value + - metadata + type: object + Problem: + description: RFC 9457 problem+json. `context` varies by error category. + properties: + context: + type: object + detail: + type: string + instance: + type: string + status: + format: int32 + type: integer + title: + type: string + trace_id: + type: string + type: + type: string + required: + - type + - title + - status + - detail + - context + type: object + SecretMetadataDto: + description: Access metadata returned alongside the secret value. + properties: + expires_at: + description: Expiry instant (RFC 3339), when set. + format: date-time + type: + - string + - 'null' + is_inherited: + description: Whether the secret came from an ancestor tenant. + type: boolean + owner_tenant_id: + description: The tenant that owns this secret. + format: uuid + type: string + sharing: + $ref: '#/components/schemas/SharingModeDto' + description: The sharing mode that governed the lookup result. + type: + description: Secret type as its full GTS type id. + type: string + version: + description: Monotonic version of the resolved secret (also returned as `ETag`). + format: int64 + type: integer + required: + - owner_tenant_id + - sharing + - is_inherited + - version + - type + type: object + SharingModeDto: + description: Sharing mode for the REST transport layer. + enum: + - private + - tenant + - shared + type: string + UpdateSecretRequestDto: + additionalProperties: false + description: 'Request body for `PUT /credstore/v1/secrets/{ref}`. + + + `Debug` is hand-written to redact `value` (see [`CreateSecretRequestDto`]).' + properties: + expires_at: + description: 'Expiry instant (RFC 3339); only for expirable types. A PUT is a + + whole-value replace: omitting `expires_at` clears a stored expiry.' + format: date-time + type: + - string + - 'null' + sharing: + oneOf: + - type: 'null' + - $ref: '#/components/schemas/SharingModeDto' + description: 'Sharing mode for the secret. **Optional: when omitted on an overwrite + + the secret''s current sharing is preserved**, so rotating a `shared` + + secret''s value with `{"value": "..."}` alone no longer silently narrows + + it back to `tenant` (review finding #8). On create-via-upsert an omitted + + `sharing` defaults to `tenant`. Send an explicit value to change the + + sharing mode.' + type: + description: 'Secret type as a full GTS type id. Optional; when present must match + + the existing secret''s type (the type is immutable). On create via + + upsert it selects the new secret''s type (default: the generic type).' + type: + - string + - 'null' + value: + description: Secret value as a UTF-8 string. + type: string + required: + - value + type: object + securitySchemes: + bearerAuth: + bearerFormat: JWT + scheme: bearer + type: http diff --git a/gears/credstore/plugins/static-credstore-plugin/README.md b/gears/credstore/plugins/static-credstore-plugin/README.md index dd66ef5b1..dbd61b638 100644 --- a/gears/credstore/plugins/static-credstore-plugin/README.md +++ b/gears/credstore/plugins/static-credstore-plugin/README.md @@ -1,22 +1,36 @@ # Static CredStore Plugin -CredStore storage-backend plugin that serves pre-configured secrets from YAML configuration. Designed for development, testing, and fixed-credential deployments where a full secrets vault is unnecessary. +CredStore **value-store** backend for development and testing: an in-memory +per-tenant secret store, optionally seeded from YAML configuration. Implements +the `CredStorePluginClientV1` contract (`get`/`put`/`delete`) so the stateful +`credstore` gear can use it as a backend without a full secrets vault. ## Overview -The `cf-gears-static-credstore-plugin` gear provides: +The `cf-gears-static-credstore-plugin` module provides: -- **Static secret mapping** — secrets defined in YAML config, loaded and validated at init -- **Four sharing scopes** — Private, Tenant, Shared (tenant-scoped), and Global -- **O(1) lookup** — secrets are pre-indexed into separate `HashMap`s per scope -- **Deterministic precedence** — lookup order: Private → Tenant → Shared → Global -- **Strict config validation** — invalid keys, duplicate entries, and contradictory field combinations are rejected at startup +- **Per-tenant value store** — `get`/`put`/`delete` keyed by `tenant_id` + `key` + + optional `owner_id` (`Some` = private key class, `None` = tenant key class). + No sharing/hierarchy/policy here — that lives in the gear. +- **Writable at runtime** — the gear's write saga (`put`/`delete`) mutates the + in-memory store, so it works as a development backend, not just a read fixture. +- **Config seeding** — secrets defined in YAML are loaded and validated at init. +- **Read fallbacks** — config-seeded `shared`/global entries serve `owner_id = + None` reads when no tenant-class entry exists. +- **Strict config validation** — invalid keys, duplicate entries, and + contradictory field combinations are rejected at startup. -The plugin registers itself via the types registry as a `CredStorePluginClientV1` implementation and is discovered by the `credstore` gateway gear. +> **Note (stateful gear):** the gear resolves metadata from its own +> database first and only then reads the value here. A secret seeded **only** in +> this plugin's config (with no corresponding gear metadata row) is therefore +> *not* reachable through the gear — write it via the credstore API +> (`POST/PUT /credstore/v1/secrets`) so a metadata row exists. + +The plugin registers itself via the types registry as a `CredStorePluginClientV1` implementation and is discovered by the `credstore` gear module. ## Configuration -Add the plugin section under your gear configuration: +Add the plugin section under your module configuration: ```yaml static-credstore-plugin: @@ -35,7 +49,7 @@ static-credstore-plugin: key: "team-api-key" value: "sk-team-456" - # Shared secret — tenant-scoped, visible to descendant tenants via gateway walk-up + # Shared secret — tenant-scoped, visible to descendant tenants via gear walk-up - tenant_id: "11111111-1111-1111-1111-111111111111" key: "org-api-key" value: "sk-org-789" @@ -80,38 +94,38 @@ The plugin rejects invalid configurations at startup with a descriptive error: - **Global with non-Shared mode** — `tenant_id: None` only allows `shared` (or inferred `shared`) - **Duplicate keys** — within the same scope (same tenant + sharing mode), keys must be unique -## Lookup precedence - -When a secret is requested, the plugin checks maps in this order: +## Read resolution -1. **Private** — keyed by `(tenant_id, owner_id, key)`, matched against `SecurityContext` -2. **Tenant** — keyed by `(tenant_id, key)`, any subject in the tenant -3. **Shared** — keyed by `(tenant_id, key)`, tenant-scoped but visible to descendants -4. **Global** — keyed by `key` only, fallback for any caller +The gear calls `get(tenant_id, key, owner_id)`. The plugin resolves against +its in-memory key classes: -The first match wins. This means a Private secret shadows a Tenant secret with the same key for the matching user, while other users in the same tenant still see the Tenant-level value. +- **`owner_id = Some`** → the **private** class only: `(tenant_id, owner_id, key)`. +- **`owner_id = None`** → the **tenant** class `(tenant_id, key)`, falling back to + config-seeded **shared** `(tenant_id, key)` then **global** `key`. -### `SecretMetadata::owner_id` resolution - -For **Private** secrets, `owner_id` comes from the config. For **Tenant**, **Shared**, and **Global** secrets, `owner_id` is not stored — the plugin fills it from `SecurityContext::subject_id()` of the caller at lookup time. +`put`/`delete` target the private class when `owner_id = Some`, otherwise the +tenant class (with `delete` also sweeping the `shared`/global fallbacks). The +config-seeded `shared`/global maps exist only to keep development configs +resolving; they are never written by `put`. The plugin returns the raw +`SecretValue` — all sharing/owner metadata is owned by the gear. ## Architecture -``` -gear.rs ToolKit gear — init, config loading, GTS registration +```text +module.rs ModKit module — init, config loading, GTS registration config.rs YAML config model + resolve_sharing() + validation docs domain/ - service.rs Service — from_config() builder + get() lookup - client.rs CredStorePluginClientV1 impl (maps SecretEntry → SecretMetadata) + service.rs Service — from_config() seeder + get_value/put_value/delete_value (RwLock store) + client.rs CredStorePluginClientV1 impl (get/put/delete -> SecretValue) mod.rs Re-exports ``` ### Init sequence -1. Load `StaticCredStorePluginConfig` from gear config +1. Load `StaticCredStorePluginConfig` from module config 2. `Service::from_config()` — validate all entries, build lookup maps 3. Register GTS plugin instance in types-registry -4. Store `Arc` in gear state +4. Store `Arc` in module state 5. Register `CredStorePluginClientV1` scoped client in `ClientHub` ## Testing @@ -122,12 +136,11 @@ cargo test -p cf-gears-static-credstore-plugin The test suite covers: -- Lookup per scope (private, tenant, shared, global) -- Precedence across all four scopes -- Owner/tenant isolation +- Read per key class (private vs tenant) and `shared`/global fallbacks +- `put`/`delete` round-trips and owner/tenant isolation - Config validation (all rejection rules) - Sharing mode inference and explicit overrides -- `SecretMetadata` owner resolution from `SecurityContext` +- The `CredStorePluginClientV1` trait impl (`get`/`put`/`delete`) ## License diff --git a/gears/credstore/plugins/static-credstore-plugin/src/config.rs b/gears/credstore/plugins/static-credstore-plugin/src/config.rs index 2e543dec6..38780e2f7 100644 --- a/gears/credstore/plugins/static-credstore-plugin/src/config.rs +++ b/gears/credstore/plugins/static-credstore-plugin/src/config.rs @@ -39,7 +39,7 @@ pub struct SecretConfig { /// `SharingMode::Shared` on the wire but stored in a separate /// global map in the static plugin). /// - `Some` with `SharingMode::Shared` → **shared** secret scoped to - /// this tenant, visible to descendants via gateway hierarchy walk-up. + /// this tenant, visible to descendants via gear hierarchy walk-up. /// - `Some` with `SharingMode::Tenant` → **tenant** secret, visible /// only within this tenant. /// diff --git a/gears/credstore/plugins/static-credstore-plugin/src/domain/client.rs b/gears/credstore/plugins/static-credstore-plugin/src/domain/client.rs index 94828e8b1..dd37c9021 100644 --- a/gears/credstore/plugins/static-credstore-plugin/src/domain/client.rs +++ b/gears/credstore/plugins/static-credstore-plugin/src/domain/client.rs @@ -1,43 +1,48 @@ -// Updated: 2026-04-07 by Constructor Tech +// Updated: 2026-06-06 — implements the per-tenant value-store `CredStorePluginClientV1`. use async_trait::async_trait; use credstore_sdk::{ - CredStoreError, CredStorePluginClientV1, OwnerId, SecretMetadata, SecretRef, SecretValue, - TenantId, + CredStoreError, CredStorePluginClientV1, OwnerId, SecretRef, SecretValue, TenantId, }; use toolkit_security::SecurityContext; use super::service::Service; +/// The static plugin is a pure per-tenant value store: it ignores the +/// security context (the gear has already authorized the request and +/// resolved tenant/owner) and keys purely on `(tenant_id, key, owner_id)`. #[async_trait] impl CredStorePluginClientV1 for Service { async fn get( &self, - ctx: &SecurityContext, + _ctx: &SecurityContext, + tenant_id: &TenantId, key: &SecretRef, - ) -> Result, CredStoreError> { - let Some(entry) = self.get(ctx, key) else { - return Ok(None); - }; + owner_id: Option<&OwnerId>, + ) -> Result, CredStoreError> { + Ok(self.get_value(tenant_id, key, owner_id)) + } - // For Shared/Tenant entries the stored owner_id/owner_tenant_id are nil - // placeholders — resolve them from the caller's security context. - let owner_id = if entry.owner_id.is_nil() { - OwnerId(ctx.subject_id()) - } else { - entry.owner_id - }; - let owner_tenant_id = if entry.owner_tenant_id.is_nil() { - TenantId(ctx.subject_tenant_id()) - } else { - entry.owner_tenant_id - }; + async fn put( + &self, + _ctx: &SecurityContext, + tenant_id: &TenantId, + key: &SecretRef, + value: SecretValue, + owner_id: Option<&OwnerId>, + ) -> Result<(), CredStoreError> { + self.put_value(tenant_id, key, value, owner_id); + Ok(()) + } - Ok(Some(SecretMetadata { - value: SecretValue::new(entry.value.as_bytes().to_vec()), - owner_id, - sharing: entry.sharing, - owner_tenant_id, - })) + async fn delete( + &self, + _ctx: &SecurityContext, + tenant_id: &TenantId, + key: &SecretRef, + owner_id: Option<&OwnerId>, + ) -> Result<(), CredStoreError> { + self.delete_value(tenant_id, key, owner_id); + Ok(()) } } diff --git a/gears/credstore/plugins/static-credstore-plugin/src/domain/client_tests.rs b/gears/credstore/plugins/static-credstore-plugin/src/domain/client_tests.rs index 178d379be..4e4ab8860 100644 --- a/gears/credstore/plugins/static-credstore-plugin/src/domain/client_tests.rs +++ b/gears/credstore/plugins/static-credstore-plugin/src/domain/client_tests.rs @@ -1,220 +1,106 @@ -// Created: 2026-04-07 by Constructor Tech -use super::*; -use crate::config::{SecretConfig, StaticCredStorePluginConfig}; +// Created: 2026-06-06 — tests for the `CredStorePluginClientV1` trait impl. +use credstore_sdk::{CredStorePluginClientV1, OwnerId, SecretRef, SecretValue, TenantId}; +use toolkit_security::SecurityContext; use uuid::Uuid; -fn tenant_a() -> Uuid { - Uuid::parse_str("11111111-1111-1111-1111-111111111111").unwrap() -} - -fn tenant_b() -> Uuid { - Uuid::parse_str("22222222-2222-2222-2222-222222222222").unwrap() -} - -fn owner_a() -> Uuid { - Uuid::parse_str("33333333-3333-3333-3333-333333333333").unwrap() -} - -fn owner_b() -> Uuid { - Uuid::parse_str("44444444-4444-4444-4444-444444444444").unwrap() -} +use crate::config::{SecretConfig, StaticCredStorePluginConfig}; +use crate::domain::service::Service; -fn ctx(tenant_id: Uuid, subject_id: Uuid) -> SecurityContext { +fn ctx() -> SecurityContext { SecurityContext::builder() - .subject_id(subject_id) - .subject_tenant_id(tenant_id) + .subject_tenant_id(Uuid::new_v4()) + .subject_id(Uuid::new_v4()) .build() - .unwrap() + .expect("test security context") +} + +fn empty_service() -> Service { + Service::from_config(&StaticCredStorePluginConfig::default()).expect("config builds") } -/// Private secret: `tenant_a` + `owner_a`. -fn service_with_single_secret() -> Service { +fn seeded_service() -> Service { let cfg = StaticCredStorePluginConfig { secrets: vec![SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: Some(owner_a()), - key: "openai_api_key".to_owned(), - value: "sk-test-123".to_owned(), + tenant_id: Some(Uuid::parse_str("00000000-0000-0000-0000-000000000001").unwrap()), + owner_id: None, + key: "openai-key".to_owned(), + value: "seeded".to_owned(), sharing: None, }], - ..StaticCredStorePluginConfig::default() + ..Default::default() }; + Service::from_config(&cfg).expect("config builds") +} - Service::from_config(&cfg).unwrap() +fn tid() -> TenantId { + TenantId(Uuid::parse_str("00000000-0000-0000-0000-000000000001").unwrap()) } #[tokio::test] -async fn get_returns_metadata_for_matching_tenant_and_owner() { - let service = service_with_single_secret(); - let plugin: &dyn CredStorePluginClientV1 = &service; - let key = SecretRef::new("openai_api_key").unwrap(); - - let metadata = plugin - .get(&ctx(tenant_a(), owner_a()), &key) +async fn get_seeded_tenant_secret() { + let svc = seeded_service(); + let got = svc + .get(&ctx(), &tid(), &SecretRef::new("openai-key").unwrap(), None) .await .unwrap() - .unwrap(); - assert_eq!(metadata.value.as_bytes(), b"sk-test-123"); - assert_eq!(metadata.owner_id, OwnerId(owner_a())); - assert_eq!(metadata.owner_tenant_id, TenantId(tenant_a())); + .expect("secret present"); + assert_eq!(got.as_bytes(), b"seeded"); } #[tokio::test] -async fn get_returns_none_for_other_tenant() { - let service = service_with_single_secret(); - let plugin: &dyn CredStorePluginClientV1 = &service; - let key = SecretRef::new("openai_api_key").unwrap(); - - let result = plugin.get(&ctx(tenant_b(), owner_a()), &key).await.unwrap(); - assert!(result.is_none()); -} - -#[tokio::test] -async fn get_returns_none_for_other_owner() { - let service = service_with_single_secret(); - let plugin: &dyn CredStorePluginClientV1 = &service; - let key = SecretRef::new("openai_api_key").unwrap(); - - let result = plugin.get(&ctx(tenant_a(), owner_b()), &key).await.unwrap(); - assert!(result.is_none()); -} - -#[tokio::test] -async fn get_returns_none_for_missing_key() { - let service = service_with_single_secret(); - let plugin: &dyn CredStorePluginClientV1 = &service; - let key = SecretRef::new("missing").unwrap(); - - let result = plugin.get(&ctx(tenant_a(), owner_a()), &key).await.unwrap(); - assert!(result.is_none()); -} - -#[tokio::test] -async fn get_returns_none_when_no_secrets_configured() { - let service = Service::from_config(&StaticCredStorePluginConfig::default()).unwrap(); - let plugin: &dyn CredStorePluginClientV1 = &service; - let key = SecretRef::new("openai_api_key").unwrap(); - - let result = plugin.get(&ctx(tenant_a(), owner_a()), &key).await.unwrap(); - assert!(result.is_none()); -} - -// --- Shared secret fills owner from SecurityContext --- - -#[tokio::test] -async fn shared_secret_resolves_owner_from_context() { - let cfg = StaticCredStorePluginConfig { - secrets: vec![SecretConfig { - tenant_id: None, - owner_id: None, - key: "global_key".to_owned(), - value: "global-val".to_owned(), - sharing: None, - }], - ..StaticCredStorePluginConfig::default() - }; - let service = Service::from_config(&cfg).unwrap(); - let plugin: &dyn CredStorePluginClientV1 = &service; - let key = SecretRef::new("global_key").unwrap(); - - let metadata = plugin - .get(&ctx(tenant_a(), owner_b()), &key) +async fn get_missing_returns_none() { + let svc = empty_service(); + let got = svc + .get(&ctx(), &tid(), &SecretRef::new("absent").unwrap(), None) .await - .unwrap() .unwrap(); - - assert_eq!(metadata.value.as_bytes(), b"global-val"); - assert_eq!(metadata.owner_id, OwnerId(owner_b())); - assert_eq!(metadata.owner_tenant_id, TenantId(tenant_a())); + assert!(got.is_none()); } -// --- Tenant secret fills owner from SecurityContext --- - #[tokio::test] -async fn tenant_secret_resolves_owner_from_context() { - let cfg = StaticCredStorePluginConfig { - secrets: vec![SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: None, - key: "scoped_key".to_owned(), - value: "scoped-val".to_owned(), - sharing: None, - }], - ..StaticCredStorePluginConfig::default() - }; - let service = Service::from_config(&cfg).unwrap(); - let plugin: &dyn CredStorePluginClientV1 = &service; - let key = SecretRef::new("scoped_key").unwrap(); +async fn put_then_get_roundtrip() { + let svc = empty_service(); + let key = SecretRef::new("written").unwrap(); - let metadata = plugin - .get(&ctx(tenant_a(), owner_b()), &key) + svc.put(&ctx(), &tid(), &key, SecretValue::from("v1"), None) .await - .unwrap() .unwrap(); - assert_eq!(metadata.owner_id, OwnerId(owner_b())); - assert_eq!(metadata.owner_tenant_id, TenantId(tenant_a())); + let got = svc.get(&ctx(), &tid(), &key, None).await.unwrap(); + assert_eq!(got.unwrap().as_bytes(), b"v1"); } -// --- Lookup precedence via plugin --- - #[tokio::test] -async fn private_takes_precedence_over_tenant_and_shared_via_plugin() { - let cfg = StaticCredStorePluginConfig { - secrets: vec![ - SecretConfig { - tenant_id: None, - owner_id: None, - key: "k".to_owned(), - value: "shared-val".to_owned(), - sharing: None, - }, - SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: None, - key: "k".to_owned(), - value: "tenant-val".to_owned(), - sharing: None, - }, - SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: Some(owner_a()), - key: "k".to_owned(), - value: "private-val".to_owned(), - sharing: None, - }, - ], - ..StaticCredStorePluginConfig::default() - }; - let service = Service::from_config(&cfg).unwrap(); - let plugin: &dyn CredStorePluginClientV1 = &service; - let key = SecretRef::new("k").unwrap(); - - // owner_a in tenant_a → Private - let meta = plugin - .get(&ctx(tenant_a(), owner_a()), &key) - .await - .unwrap() - .unwrap(); - assert_eq!(meta.value.as_bytes(), b"private-val"); - assert_eq!(meta.owner_id, OwnerId(owner_a())); +async fn private_put_is_owner_scoped() { + let svc = empty_service(); + let key = SecretRef::new("owned").unwrap(); + let owner = OwnerId(Uuid::new_v4()); + + svc.put( + &ctx(), + &tid(), + &key, + SecretValue::from("secret"), + Some(&owner), + ) + .await + .unwrap(); + + // Visible to the owner, not to the tenant class. + assert!( + svc.get(&ctx(), &tid(), &key, Some(&owner)) + .await + .unwrap() + .is_some() + ); + assert!(svc.get(&ctx(), &tid(), &key, None).await.unwrap().is_none()); +} - // owner_b in tenant_a → Tenant (owner resolved from ctx) - let meta = plugin - .get(&ctx(tenant_a(), owner_b()), &key) - .await - .unwrap() - .unwrap(); - assert_eq!(meta.value.as_bytes(), b"tenant-val"); - assert_eq!(meta.owner_id, OwnerId(owner_b())); +#[tokio::test] +async fn delete_removes_value() { + let svc = seeded_service(); + let key = SecretRef::new("openai-key").unwrap(); - // tenant_b → Shared (owner resolved from ctx) - let meta = plugin - .get(&ctx(tenant_b(), owner_b()), &key) - .await - .unwrap() - .unwrap(); - assert_eq!(meta.value.as_bytes(), b"shared-val"); - assert_eq!(meta.owner_id, OwnerId(owner_b())); - assert_eq!(meta.owner_tenant_id, TenantId(tenant_b())); + svc.delete(&ctx(), &tid(), &key, None).await.unwrap(); + assert!(svc.get(&ctx(), &tid(), &key, None).await.unwrap().is_none()); } diff --git a/gears/credstore/plugins/static-credstore-plugin/src/domain/service.rs b/gears/credstore/plugins/static-credstore-plugin/src/domain/service.rs index 1e80e2027..d3d69d2ba 100644 --- a/gears/credstore/plugins/static-credstore-plugin/src/domain/service.rs +++ b/gears/credstore/plugins/static-credstore-plugin/src/domain/service.rs @@ -1,68 +1,66 @@ -// Updated: 2026-04-07 by Constructor Tech +// Updated: 2026-06-06 — adapted to the per-tenant value-store `CredStorePluginClientV1`. use std::collections::HashMap; +use std::sync::RwLock; use credstore_sdk::{OwnerId, SecretRef, SecretValue, SharingMode, TenantId}; use toolkit_macros::domain_model; -use toolkit_security::SecurityContext; use uuid::Uuid; use crate::config::StaticCredStorePluginConfig; -/// Pre-built secret entry for O(1) lookup. +/// In-memory key classes backing the static plugin. +/// +/// The `private`/`tenant` classes mirror the two classes the gear selects +/// via `owner_id` (`Some`/`None`). The `shared`/`global` maps hold +/// config-seeded entries that have no equivalent at write time; they are kept +/// as read-only fallbacks for `owner_id = None` lookups so existing +/// development configs keep resolving. #[domain_model] -pub struct SecretEntry { - pub value: SecretValue, - pub sharing: SharingMode, - pub owner_id: OwnerId, - pub owner_tenant_id: TenantId, +#[derive(Debug, Default)] +struct Store { + /// Private key class — keyed by `(tenant, owner, key)`. + private: HashMap<(TenantId, OwnerId, SecretRef), SecretValue>, + /// Tenant key class — keyed by `(tenant, key)`. + tenant: HashMap<(TenantId, SecretRef), SecretValue>, + /// Config-seeded `shared` secrets — read fallback for the tenant key class. + shared: HashMap<(TenantId, SecretRef), SecretValue>, + /// Config-seeded global secrets — final read fallback for the tenant class. + global: HashMap, } -/// Static credstore service. -/// -/// Secrets are stored in four maps based on their resolved `SharingMode` -/// and whether a `tenant_id` is present: +/// Static credstore backend. /// -/// - **`Private`**: keyed by `(TenantId, OwnerId, SecretRef)` — accessible only -/// when both tenant and subject match. -/// - **`Tenant`**: keyed by `(TenantId, SecretRef)` — accessible by any subject -/// within the matching tenant. -/// - **`Shared`**: keyed by `(TenantId, SecretRef)` — tenant-scoped but -/// accessible by descendant tenants via hierarchical resolution in the -/// gateway. The plugin stores them per-tenant; walk-up is the gateway's job. -/// - **Global**: keyed by `SecretRef` only — no `tenant_id`; returned as -/// fallback for any caller. Not a `SharingMode` variant; it is an -/// operational shortcut specific to the static plugin. +/// A pure per-tenant value store implementing the `CredStorePluginClientV1` +/// contract: `owner_id = Some` selects the private key class, `None` the +/// tenant key class. Sharing, hierarchy and policy live in the gear, not +/// here. /// -/// Lookup order: **Private → Tenant → Shared → Global** (most specific first). +/// The store is seeded from configuration and stays mutable at runtime, so the +/// stateful gear's write saga (`put`/`delete`) can use it as a development +/// backend. Config-seeded `shared`/global entries remain read-only fallbacks +/// for `owner_id = None` lookups. #[domain_model] -#[allow(clippy::struct_field_names)] +#[derive(Debug, Default)] pub struct Service { - private_secrets: HashMap<(TenantId, OwnerId, SecretRef), SecretEntry>, - tenant_secrets: HashMap<(TenantId, SecretRef), SecretEntry>, - shared_secrets: HashMap<(TenantId, SecretRef), SecretEntry>, - global_secrets: HashMap, + inner: RwLock, } impl Service { /// Create a service from plugin configuration. /// - /// Validates each secret key via `SecretRef::new` and builds the lookup maps. + /// Validates each configured key via `SecretRef::new` and seeds the + /// in-memory key classes from the resolved sharing mode. /// /// # Errors /// /// Returns an error if: /// - any configured key fails `SecretRef` validation /// - duplicate keys within the same sharing scope - /// - a global secret has an explicit sharing mode other than `Shared` /// - a secret without `owner_id` has an explicit `SharingMode::Private` /// - `tenant_id` or `owner_id` is an explicit nil UUID /// - `owner_id` is set without `tenant_id` pub fn from_config(cfg: &StaticCredStorePluginConfig) -> anyhow::Result { - let mut private_secrets: HashMap<(TenantId, OwnerId, SecretRef), SecretEntry> = - HashMap::new(); - let mut tenant_secrets: HashMap<(TenantId, SecretRef), SecretEntry> = HashMap::new(); - let mut shared_secrets: HashMap<(TenantId, SecretRef), SecretEntry> = HashMap::new(); - let mut global_secrets: HashMap = HashMap::new(); + let mut store = Store::default(); for entry in &cfg.secrets { if entry.tenant_id == Some(Uuid::nil()) { @@ -71,7 +69,6 @@ impl Service { if entry.owner_id == Some(Uuid::nil()) { anyhow::bail!("secret '{}': owner_id must not be nil UUID", entry.key); } - if entry.tenant_id.is_none() && entry.owner_id.is_some() { anyhow::bail!( "secret '{}': owner_id cannot be set without tenant_id", @@ -88,7 +85,6 @@ impl Service { entry.key ); } - if entry.owner_id.is_none() && sharing == SharingMode::Private { anyhow::bail!( "secret '{}' with sharing mode 'private' requires an explicit owner_id", @@ -97,40 +93,26 @@ impl Service { } let key = SecretRef::new(&entry.key)?; + let value = SecretValue::from(entry.value.as_str()); match (sharing, entry.tenant_id) { (SharingMode::Shared, None) => { - // Global secret: no tenant_id, accessible by any caller. - let secret_entry = SecretEntry { - value: SecretValue::from(entry.value.as_str()), - sharing, - owner_id: OwnerId::nil(), - owner_tenant_id: TenantId::nil(), - }; - if global_secrets.contains_key(&key) { + if store.global.contains_key(&key) { anyhow::bail!("duplicate global secret key '{}'", entry.key); } - global_secrets.insert(key, secret_entry); + store.global.insert(key, value); } (SharingMode::Shared, Some(raw_tenant_id)) => { - // Shared secret: tenant-scoped, visible to descendants - // via gateway hierarchical resolution. let tenant_id = TenantId(raw_tenant_id); - let secret_entry = SecretEntry { - value: SecretValue::from(entry.value.as_str()), - sharing, - owner_id: OwnerId::nil(), - owner_tenant_id: tenant_id, - }; let map_key = (tenant_id, key); - if shared_secrets.contains_key(&map_key) { + if store.shared.contains_key(&map_key) { anyhow::bail!( "duplicate shared secret key '{}' for tenant {}", entry.key, tenant_id ); } - shared_secrets.insert(map_key, secret_entry); + store.shared.insert(map_key, value); } (SharingMode::Tenant, _) => { let tenant_id = TenantId(entry.tenant_id.ok_or_else(|| { @@ -139,21 +121,15 @@ impl Service { entry.key ) })?); - let secret_entry = SecretEntry { - value: SecretValue::from(entry.value.as_str()), - sharing, - owner_id: OwnerId::nil(), - owner_tenant_id: tenant_id, - }; let map_key = (tenant_id, key); - if tenant_secrets.contains_key(&map_key) { + if store.tenant.contains_key(&map_key) { anyhow::bail!( "duplicate tenant secret key '{}' for tenant {}", entry.key, tenant_id ); } - tenant_secrets.insert(map_key, secret_entry); + store.tenant.insert(map_key, value); } (SharingMode::Private, _) => { let tenant_id = TenantId(entry.tenant_id.ok_or_else(|| { @@ -162,21 +138,14 @@ impl Service { entry.key ) })?); - // owner_id is guaranteed Some by the validation above. let owner_id = OwnerId(entry.owner_id.ok_or_else(|| { anyhow::anyhow!( "secret '{}': private sharing mode requires owner_id", entry.key ) })?); - let secret_entry = SecretEntry { - value: SecretValue::from(entry.value.as_str()), - sharing, - owner_id, - owner_tenant_id: tenant_id, - }; let map_key = (tenant_id, owner_id, key); - if private_secrets.contains_key(&map_key) { + if store.private.contains_key(&map_key) { anyhow::bail!( "duplicate private secret key '{}' for tenant {} owner {}", entry.key, @@ -184,32 +153,89 @@ impl Service { owner_id ); } - private_secrets.insert(map_key, secret_entry); + store.private.insert(map_key, value); } } } Ok(Self { - private_secrets, - tenant_secrets, - shared_secrets, - global_secrets, + inner: RwLock::new(store), }) } - /// Look up a secret using the caller's security context. + /// Read a value for the selected key class. /// - /// Lookup order: **Private → Tenant → Shared → Global** (most specific first). + /// `owner_id = Some` reads the private class; `None` reads the tenant class + /// and falls back to config-seeded `shared` then global entries. #[must_use] - pub fn get(&self, ctx: &SecurityContext, key: &SecretRef) -> Option<&SecretEntry> { - let tenant_id = TenantId(ctx.subject_tenant_id()); - let subject_id = OwnerId(ctx.subject_id()); + pub fn get_value( + &self, + tenant_id: &TenantId, + key: &SecretRef, + owner_id: Option<&OwnerId>, + ) -> Option { + let store = self + .inner + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let bytes = match owner_id { + Some(owner) => store + .private + .get(&(*tenant_id, *owner, key.clone())) + .map(SecretValue::as_bytes), + None => store + .tenant + .get(&(*tenant_id, key.clone())) + .or_else(|| store.shared.get(&(*tenant_id, key.clone()))) + .or_else(|| store.global.get(key)) + .map(SecretValue::as_bytes), + }; + // `SecretValue` is not `Clone` (it zeroizes on drop), so reconstruct. + bytes.map(|b| SecretValue::new(b.to_vec())) + } + + /// Insert or overwrite a value in the selected key class. + pub fn put_value( + &self, + tenant_id: &TenantId, + key: &SecretRef, + value: SecretValue, + owner_id: Option<&OwnerId>, + ) { + let mut store = self + .inner + .write() + .unwrap_or_else(std::sync::PoisonError::into_inner); + match owner_id { + Some(owner) => { + store + .private + .insert((*tenant_id, *owner, key.clone()), value); + } + None => { + store.tenant.insert((*tenant_id, key.clone()), value); + } + } + } - self.private_secrets - .get(&(tenant_id, subject_id, key.clone())) - .or_else(|| self.tenant_secrets.get(&(tenant_id, key.clone()))) - .or_else(|| self.shared_secrets.get(&(tenant_id, key.clone()))) - .or_else(|| self.global_secrets.get(key)) + /// Remove a value from the selected key class. + /// + /// Deletes address the **runtime** maps only. The config-seeded `shared` + /// and global fallbacks are cross-tenant reference data: a tenant-scoped + /// delete (including gear saga retries and reaper reconciliation + /// issued on behalf of one tenant) must never destroy an entry that + /// serves other tenants. A miss is a no-op — the gear treats a + /// missing backend value as success. + pub fn delete_value(&self, tenant_id: &TenantId, key: &SecretRef, owner_id: Option<&OwnerId>) { + let mut store = self + .inner + .write() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if let Some(owner) = owner_id { + store.private.remove(&(*tenant_id, *owner, key.clone())); + } else { + store.tenant.remove(&(*tenant_id, key.clone())); + } } } diff --git a/gears/credstore/plugins/static-credstore-plugin/src/domain/service_tests.rs b/gears/credstore/plugins/static-credstore-plugin/src/domain/service_tests.rs index 92372bacb..547daf078 100644 --- a/gears/credstore/plugins/static-credstore-plugin/src/domain/service_tests.rs +++ b/gears/credstore/plugins/static-credstore-plugin/src/domain/service_tests.rs @@ -1,777 +1,231 @@ -// Created: 2026-04-07 by Constructor Tech -use super::*; -use crate::config::SecretConfig; +// Created: 2026-06-06 — tests for the static credstore in-memory value store. use uuid::Uuid; -fn tenant_a() -> Uuid { - Uuid::parse_str("11111111-1111-1111-1111-111111111111").unwrap() -} - -fn tenant_b() -> Uuid { - Uuid::parse_str("22222222-2222-2222-2222-222222222222").unwrap() -} - -fn owner_a() -> Uuid { - Uuid::parse_str("33333333-3333-3333-3333-333333333333").unwrap() -} - -fn owner_b() -> Uuid { - Uuid::parse_str("44444444-4444-4444-4444-444444444444").unwrap() -} - -fn ctx(tenant_id: Uuid, subject_id: Uuid) -> SecurityContext { - SecurityContext::builder() - .subject_id(subject_id) - .subject_tenant_id(tenant_id) - .build() - .unwrap() -} - -/// Default config: `tenant_a` + `owner_a` → Private secret. -fn cfg_with_single_secret() -> StaticCredStorePluginConfig { - StaticCredStorePluginConfig { - secrets: vec![SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: Some(owner_a()), - key: "openai_api_key".to_owned(), - value: "sk-test-123".to_owned(), - sharing: None, - }], - ..StaticCredStorePluginConfig::default() - } -} - -#[test] -fn from_config_rejects_invalid_secret_ref() { - let cfg = StaticCredStorePluginConfig { - secrets: vec![SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: Some(owner_a()), - key: "invalid:key".to_owned(), - value: "value".to_owned(), - sharing: None, - }], - ..StaticCredStorePluginConfig::default() - }; +use credstore_sdk::{OwnerId, SecretRef, SecretValue, SharingMode, TenantId}; - let result = Service::from_config(&cfg); - assert!(result.is_err()); -} +use crate::config::{SecretConfig, StaticCredStorePluginConfig}; -// --- Private secret lookup --- +use super::Service; -#[test] -fn private_secret_returned_for_matching_tenant_and_owner() { - let service = Service::from_config(&cfg_with_single_secret()).unwrap(); - let key = SecretRef::new("openai_api_key").unwrap(); +const T1: &str = "00000000-0000-0000-0000-000000000001"; +const T2: &str = "00000000-0000-0000-0000-000000000002"; +const O1: &str = "11111111-0000-0000-0000-000000000001"; +const O2: &str = "22222222-0000-0000-0000-000000000002"; - let entry = service.get(&ctx(tenant_a(), owner_a()), &key).unwrap(); - assert_eq!(entry.value.as_bytes(), b"sk-test-123"); - assert_eq!(entry.owner_id, OwnerId(owner_a())); - assert_eq!(entry.owner_tenant_id, TenantId(tenant_a())); - assert_eq!(entry.sharing, SharingMode::Private); +fn tid(s: &str) -> TenantId { + TenantId(Uuid::parse_str(s).unwrap()) } - -#[test] -fn private_secret_not_returned_for_different_owner() { - let service = Service::from_config(&cfg_with_single_secret()).unwrap(); - let key = SecretRef::new("openai_api_key").unwrap(); - - assert!(service.get(&ctx(tenant_a(), owner_b()), &key).is_none()); +fn oid(s: &str) -> OwnerId { + OwnerId(Uuid::parse_str(s).unwrap()) } - -#[test] -fn private_secret_not_returned_for_different_tenant() { - let service = Service::from_config(&cfg_with_single_secret()).unwrap(); - let key = SecretRef::new("openai_api_key").unwrap(); - - assert!(service.get(&ctx(tenant_b(), owner_a()), &key).is_none()); +fn sref(s: &str) -> SecretRef { + SecretRef::new(s).unwrap() } -#[test] -fn get_returns_none_for_missing_key() { - let service = Service::from_config(&cfg_with_single_secret()).unwrap(); - let key = SecretRef::new("missing").unwrap(); - - assert!(service.get(&ctx(tenant_a(), owner_a()), &key).is_none()); -} - -#[test] -fn from_config_with_empty_secrets_returns_none() { - let cfg = StaticCredStorePluginConfig::default(); - let service = Service::from_config(&cfg).unwrap(); - let key = SecretRef::new("any-key").unwrap(); - assert!(service.get(&ctx(tenant_a(), owner_a()), &key).is_none()); -} - -// --- Tenant secret lookup --- - -#[test] -fn tenant_secret_returned_for_any_subject_in_same_tenant() { - let cfg = StaticCredStorePluginConfig { - secrets: vec![SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: None, - key: "team_key".to_owned(), - value: "team-val".to_owned(), - sharing: None, - }], - ..StaticCredStorePluginConfig::default() - }; - let service = Service::from_config(&cfg).unwrap(); - let key = SecretRef::new("team_key").unwrap(); - - let e1 = service.get(&ctx(tenant_a(), owner_a()), &key).unwrap(); - assert_eq!(e1.value.as_bytes(), b"team-val"); - assert_eq!(e1.sharing, SharingMode::Tenant); - - let e2 = service.get(&ctx(tenant_a(), owner_b()), &key).unwrap(); - assert_eq!(e2.value.as_bytes(), b"team-val"); - - assert!(service.get(&ctx(tenant_b(), owner_a()), &key).is_none()); -} - -// --- Global secret lookup --- - -#[test] -fn global_secret_returned_for_any_tenant_and_subject() { - let cfg = StaticCredStorePluginConfig { - secrets: vec![SecretConfig { - tenant_id: None, - owner_id: None, - key: "global_key".to_owned(), - value: "global-val".to_owned(), - sharing: None, - }], - ..StaticCredStorePluginConfig::default() - }; - let service = Service::from_config(&cfg).unwrap(); - let key = SecretRef::new("global_key").unwrap(); - - let e1 = service.get(&ctx(tenant_a(), owner_a()), &key).unwrap(); - assert_eq!(e1.value.as_bytes(), b"global-val"); - assert_eq!(e1.sharing, SharingMode::Shared); - - let e2 = service.get(&ctx(tenant_b(), owner_b()), &key).unwrap(); - assert_eq!(e2.value.as_bytes(), b"global-val"); -} - -// --- Shared (tenant-scoped) secret lookup --- - -#[test] -fn shared_secret_returned_only_for_owning_tenant() { - let cfg = StaticCredStorePluginConfig { - secrets: vec![SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: None, - key: "shared_key".to_owned(), - value: "shared-val".to_owned(), - sharing: Some(SharingMode::Shared), - }], - ..StaticCredStorePluginConfig::default() - }; - let service = Service::from_config(&cfg).unwrap(); - let key = SecretRef::new("shared_key").unwrap(); - - // Same tenant — accessible - let e = service.get(&ctx(tenant_a(), owner_a()), &key).unwrap(); - assert_eq!(e.value.as_bytes(), b"shared-val"); - assert_eq!(e.sharing, SharingMode::Shared); - assert_eq!(e.owner_tenant_id, TenantId(tenant_a())); - - // Different tenant — not accessible at plugin level - // (gateway walk-up would call the plugin with parent tenant_id) - assert!(service.get(&ctx(tenant_b(), owner_a()), &key).is_none()); -} - -// --- Lookup precedence: Private > Tenant > Shared > Global --- - -#[test] -fn private_takes_precedence_over_tenant_shared_and_global() { - let cfg = StaticCredStorePluginConfig { - secrets: vec![ - SecretConfig { - tenant_id: None, - owner_id: None, - key: "k".to_owned(), - value: "global-val".to_owned(), - sharing: None, - }, - SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: None, - key: "k".to_owned(), - value: "shared-val".to_owned(), - sharing: Some(SharingMode::Shared), - }, - SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: None, - key: "k".to_owned(), - value: "tenant-val".to_owned(), - sharing: None, - }, - SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: Some(owner_a()), - key: "k".to_owned(), - value: "private-val".to_owned(), - sharing: None, - }, - ], - ..StaticCredStorePluginConfig::default() - }; - let service = Service::from_config(&cfg).unwrap(); - let key = SecretRef::new("k").unwrap(); - - // owner_a in tenant_a → Private - let e = service.get(&ctx(tenant_a(), owner_a()), &key).unwrap(); - assert_eq!(e.value.as_bytes(), b"private-val"); - assert_eq!(e.sharing, SharingMode::Private); - - // owner_b in tenant_a → Tenant (no private match) - let e = service.get(&ctx(tenant_a(), owner_b()), &key).unwrap(); - assert_eq!(e.value.as_bytes(), b"tenant-val"); - assert_eq!(e.sharing, SharingMode::Tenant); - - // tenant_b → Global (no private, tenant, or shared match) - let e = service.get(&ctx(tenant_b(), owner_a()), &key).unwrap(); - assert_eq!(e.value.as_bytes(), b"global-val"); - assert_eq!(e.sharing, SharingMode::Shared); +fn secret(tenant: Option<&str>, owner: Option<&str>, key: &str, value: &str) -> SecretConfig { + SecretConfig { + tenant_id: tenant.map(|t| Uuid::parse_str(t).unwrap()), + owner_id: owner.map(|o| Uuid::parse_str(o).unwrap()), + key: key.to_owned(), + value: value.to_owned(), + sharing: None, + } } -#[test] -fn tenant_takes_precedence_over_shared_and_global() { - let cfg = StaticCredStorePluginConfig { - secrets: vec![ - SecretConfig { - tenant_id: None, - owner_id: None, - key: "k".to_owned(), - value: "global-val".to_owned(), - sharing: None, - }, - SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: None, - key: "k".to_owned(), - value: "shared-val".to_owned(), - sharing: Some(SharingMode::Shared), - }, - SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: None, - key: "k".to_owned(), - value: "tenant-val".to_owned(), - sharing: None, - }, - ], - ..StaticCredStorePluginConfig::default() - }; - let service = Service::from_config(&cfg).unwrap(); - let key = SecretRef::new("k").unwrap(); - - let e = service.get(&ctx(tenant_a(), owner_a()), &key).unwrap(); - assert_eq!(e.value.as_bytes(), b"tenant-val"); - - let e = service.get(&ctx(tenant_b(), owner_a()), &key).unwrap(); - assert_eq!(e.value.as_bytes(), b"global-val"); +fn cfg(secrets: Vec) -> StaticCredStorePluginConfig { + StaticCredStorePluginConfig { + secrets, + ..Default::default() + } } -#[test] -fn shared_takes_precedence_over_global() { - let cfg = StaticCredStorePluginConfig { - secrets: vec![ - SecretConfig { - tenant_id: None, - owner_id: None, - key: "k".to_owned(), - value: "global-val".to_owned(), - sharing: None, - }, - SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: None, - key: "k".to_owned(), - value: "shared-val".to_owned(), - sharing: Some(SharingMode::Shared), - }, - ], - ..StaticCredStorePluginConfig::default() - }; - let service = Service::from_config(&cfg).unwrap(); - let key = SecretRef::new("k").unwrap(); - - // tenant_a has a shared secret → takes precedence over global - let e = service.get(&ctx(tenant_a(), owner_a()), &key).unwrap(); - assert_eq!(e.value.as_bytes(), b"shared-val"); - assert_eq!(e.sharing, SharingMode::Shared); - - // tenant_b has no shared secret → falls through to global - let e = service.get(&ctx(tenant_b(), owner_a()), &key).unwrap(); - assert_eq!(e.value.as_bytes(), b"global-val"); +fn svc(secrets: Vec) -> Service { + Service::from_config(&cfg(secrets)).expect("config builds") } -// --- Duplicate key validation --- - -#[test] -fn from_config_rejects_duplicate_private_key() { - let secret = SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: Some(owner_a()), - key: "dup".to_owned(), - value: "v1".to_owned(), - sharing: None, - }; - let cfg = StaticCredStorePluginConfig { - secrets: vec![ - secret.clone(), - SecretConfig { - value: "v2".to_owned(), - ..secret - }, - ], - ..StaticCredStorePluginConfig::default() - }; - - match Service::from_config(&cfg) { - Ok(_) => panic!("expected error for duplicate private key"), - Err(e) => { - let err = e.to_string(); - assert!(err.contains("duplicate"), "expected 'duplicate' in: {err}"); - assert!(err.contains("dup"), "expected key name in: {err}"); - } - } +#[track_caller] +fn assert_value(v: Option, expected: &str) { + assert_eq!( + v.expect("value present").as_bytes(), + expected.as_bytes(), + "secret value mismatch" + ); } #[test] -fn from_config_rejects_duplicate_tenant_key() { - let cfg = StaticCredStorePluginConfig { - secrets: vec![ - SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: None, - key: "dup".to_owned(), - value: "v1".to_owned(), - sharing: None, - }, - SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: None, - key: "dup".to_owned(), - value: "v2".to_owned(), - sharing: None, - }, - ], - ..StaticCredStorePluginConfig::default() - }; +fn tenant_class_read() { + let s = svc(vec![secret(Some(T1), None, "openai-key", "tenant-val")]); - match Service::from_config(&cfg) { - Ok(_) => panic!("expected error for duplicate tenant key"), - Err(e) => { - let err = e.to_string(); - assert!(err.contains("duplicate"), "expected 'duplicate' in: {err}"); - } - } + assert_value( + s.get_value(&tid(T1), &sref("openai-key"), None), + "tenant-val", + ); + // Other tenant cannot see it. + assert!(s.get_value(&tid(T2), &sref("openai-key"), None).is_none()); + // Tenant secret is not exposed to the private key class. + assert!( + s.get_value(&tid(T1), &sref("openai-key"), Some(&oid(O1))) + .is_none() + ); } #[test] -fn from_config_rejects_duplicate_global_key() { - let cfg = StaticCredStorePluginConfig { - secrets: vec![ - SecretConfig { - tenant_id: None, - owner_id: None, - key: "dup".to_owned(), - value: "v1".to_owned(), - sharing: None, - }, - SecretConfig { - tenant_id: None, - owner_id: None, - key: "dup".to_owned(), - value: "v2".to_owned(), - sharing: None, - }, - ], - ..StaticCredStorePluginConfig::default() - }; +fn private_class_read_is_owner_scoped() { + let s = svc(vec![secret( + Some(T1), + Some(O1), + "openai-key", + "private-val", + )]); - match Service::from_config(&cfg) { - Ok(_) => panic!("expected error for duplicate global key"), - Err(e) => { - let err = e.to_string(); - assert!(err.contains("duplicate"), "expected 'duplicate' in: {err}"); - } - } + assert_value( + s.get_value(&tid(T1), &sref("openai-key"), Some(&oid(O1))), + "private-val", + ); + // Wrong owner -> miss. + assert!( + s.get_value(&tid(T1), &sref("openai-key"), Some(&oid(O2))) + .is_none() + ); + // Tenant-class read does not see a private secret. + assert!(s.get_value(&tid(T1), &sref("openai-key"), None).is_none()); } #[test] -fn from_config_rejects_duplicate_shared_key() { - let cfg = StaticCredStorePluginConfig { - secrets: vec![ - SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: None, - key: "dup".to_owned(), - value: "v1".to_owned(), - sharing: Some(SharingMode::Shared), - }, - SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: None, - key: "dup".to_owned(), - value: "v2".to_owned(), - sharing: Some(SharingMode::Shared), - }, - ], - ..StaticCredStorePluginConfig::default() - }; +fn global_secret_is_a_tenant_class_fallback() { + // No tenant_id -> global (resolved sharing == Shared). + let s = svc(vec![secret(None, None, "azure-key", "global-val")]); - match Service::from_config(&cfg) { - Ok(_) => panic!("expected error for duplicate shared key"), - Err(e) => { - let err = e.to_string(); - assert!(err.contains("duplicate"), "expected 'duplicate' in: {err}"); - } - } + assert_value( + s.get_value(&tid(T1), &sref("azure-key"), None), + "global-val", + ); + assert_value( + s.get_value(&tid(T2), &sref("azure-key"), None), + "global-val", + ); + // Not visible to the private key class. + assert!( + s.get_value(&tid(T1), &sref("azure-key"), Some(&oid(O1))) + .is_none() + ); } -// --- Config validation --- - #[test] -fn from_config_rejects_non_shared_global_secret() { - for mode in [SharingMode::Private, SharingMode::Tenant] { - let cfg = StaticCredStorePluginConfig { - secrets: vec![SecretConfig { - tenant_id: None, - owner_id: None, - key: "global_key".to_owned(), - value: "val".to_owned(), - sharing: Some(mode), - }], - ..StaticCredStorePluginConfig::default() - }; +fn shared_secret_is_a_tenant_class_fallback() { + let mut entry = secret(Some(T1), None, "shared-key", "shared-val"); + entry.sharing = Some(SharingMode::Shared); + let s = svc(vec![entry]); - assert!( - Service::from_config(&cfg).is_err(), - "expected error for global secret with {mode:?} sharing" - ); - } + assert_value( + s.get_value(&tid(T1), &sref("shared-key"), None), + "shared-val", + ); + // Scoped to the owning tenant (gear handles hierarchical walk-up). + assert!(s.get_value(&tid(T2), &sref("shared-key"), None).is_none()); } #[test] -fn from_config_rejects_private_without_owner_id() { - let cfg = StaticCredStorePluginConfig { - secrets: vec![SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: None, - key: "private_key".to_owned(), - value: "val".to_owned(), - sharing: Some(SharingMode::Private), - }], - ..StaticCredStorePluginConfig::default() - }; - - match Service::from_config(&cfg) { - Ok(_) => panic!("expected error for private without owner_id"), - Err(e) => { - let err = e.to_string(); - assert!(err.contains("requires an explicit owner_id"), "got: {err}"); - } - } +fn put_then_get_tenant_class() { + let s = svc(vec![]); + s.put_value(&tid(T1), &sref("k"), SecretValue::from("written"), None); + assert_value(s.get_value(&tid(T1), &sref("k"), None), "written"); } #[test] -fn from_config_rejects_owner_id_without_tenant_id() { - let cfg = StaticCredStorePluginConfig { - secrets: vec![SecretConfig { - tenant_id: None, - owner_id: Some(owner_a()), - key: "bad_key".to_owned(), - value: "val".to_owned(), - sharing: None, - }], - ..StaticCredStorePluginConfig::default() - }; - - match Service::from_config(&cfg) { - Ok(_) => panic!("expected error for owner_id without tenant_id"), - Err(e) => { - let err = e.to_string(); - assert!( - err.contains("owner_id cannot be set without tenant_id"), - "got: {err}" - ); - } - } +fn put_then_get_private_class() { + let s = svc(vec![]); + s.put_value( + &tid(T1), + &sref("k"), + SecretValue::from("owned"), + Some(&oid(O1)), + ); + assert_value(s.get_value(&tid(T1), &sref("k"), Some(&oid(O1))), "owned"); + // Private write is invisible to the tenant class and other owners. + assert!(s.get_value(&tid(T1), &sref("k"), None).is_none()); + assert!(s.get_value(&tid(T1), &sref("k"), Some(&oid(O2))).is_none()); } #[test] -fn from_config_rejects_owner_id_for_non_private() { - for mode in [SharingMode::Tenant, SharingMode::Shared] { - let cfg = StaticCredStorePluginConfig { - secrets: vec![SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: Some(owner_a()), - key: "bad_key".to_owned(), - value: "val".to_owned(), - sharing: Some(mode), - }], - ..StaticCredStorePluginConfig::default() - }; - - match Service::from_config(&cfg) { - Ok(_) => panic!("expected error for owner_id with {mode:?} sharing"), - Err(e) => { - let err = e.to_string(); - assert!( - err.contains("owner_id is only valid for private sharing mode"), - "got: {err}" - ); - } - } - } +fn put_overwrites_existing_value() { + let s = svc(vec![secret(Some(T1), None, "k", "old")]); + s.put_value(&tid(T1), &sref("k"), SecretValue::from("new"), None); + assert_value(s.get_value(&tid(T1), &sref("k"), None), "new"); } #[test] -fn from_config_accepts_shared_with_tenant_id() { - let cfg = StaticCredStorePluginConfig { - secrets: vec![SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: None, - key: "k".to_owned(), - value: "v".to_owned(), - sharing: Some(SharingMode::Shared), - }], - ..StaticCredStorePluginConfig::default() - }; - - let service = Service::from_config(&cfg).unwrap(); - let key = SecretRef::new("k").unwrap(); - let e = service.get(&ctx(tenant_a(), owner_a()), &key).unwrap(); - assert_eq!(e.sharing, SharingMode::Shared); - assert_eq!(e.owner_tenant_id, TenantId(tenant_a())); +fn delete_removes_tenant_value() { + let s = svc(vec![secret(Some(T1), None, "k", "v")]); + s.delete_value(&tid(T1), &sref("k"), None); + assert!(s.get_value(&tid(T1), &sref("k"), None).is_none()); } #[test] -fn from_config_rejects_nil_tenant_id() { - let cfg = StaticCredStorePluginConfig { - secrets: vec![SecretConfig { - tenant_id: Some(Uuid::nil()), - owner_id: Some(owner_a()), - key: "k".to_owned(), - value: "v".to_owned(), - sharing: None, - }], - ..StaticCredStorePluginConfig::default() - }; - - match Service::from_config(&cfg) { - Ok(_) => panic!("expected error for nil tenant_id"), - Err(e) => { - let err = e.to_string(); - assert!(err.contains("tenant_id must not be nil UUID"), "got: {err}"); - } - } +fn delete_removes_private_value_without_touching_tenant_class() { + let s = svc(vec![ + secret(Some(T1), None, "k", "tenant-v"), + secret(Some(T1), Some(O1), "k", "private-v"), + ]); + s.delete_value(&tid(T1), &sref("k"), Some(&oid(O1))); + assert!(s.get_value(&tid(T1), &sref("k"), Some(&oid(O1))).is_none()); + // Tenant-class value under the same key is untouched. + assert_value(s.get_value(&tid(T1), &sref("k"), None), "tenant-v"); } #[test] -fn from_config_rejects_nil_owner_id() { - let cfg = StaticCredStorePluginConfig { - secrets: vec![SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: Some(Uuid::nil()), - key: "k".to_owned(), - value: "v".to_owned(), - sharing: None, - }], - ..StaticCredStorePluginConfig::default() - }; +fn tenant_delete_never_destroys_config_seeded_fallbacks() { + // Config-seeded shared/global entries serve *other* tenants too; a + // tenant-scoped delete (or a gear reaper retry) must not sweep them. + let mut shared_entry = secret(Some(T1), None, "shared-key", "shared-val"); + shared_entry.sharing = Some(SharingMode::Shared); + let s = svc(vec![ + secret(None, None, "global-key", "global-val"), + shared_entry, + ]); - match Service::from_config(&cfg) { - Ok(_) => panic!("expected error for nil owner_id"), - Err(e) => { - let err = e.to_string(); - assert!(err.contains("owner_id must not be nil UUID"), "got: {err}"); - } - } -} - -// --- Sharing mode defaults --- + s.delete_value(&tid(T1), &sref("global-key"), None); + s.delete_value(&tid(T1), &sref("shared-key"), None); -#[test] -fn default_sharing_is_shared_for_global() { - let cfg = StaticCredStorePluginConfig { - secrets: vec![SecretConfig { - tenant_id: None, - owner_id: None, - key: "g".to_owned(), - value: "v".to_owned(), - sharing: None, - }], - ..StaticCredStorePluginConfig::default() - }; - let service = Service::from_config(&cfg).unwrap(); - let key = SecretRef::new("g").unwrap(); - assert_eq!( - service - .get(&ctx(tenant_a(), owner_a()), &key) - .unwrap() - .sharing, - SharingMode::Shared + assert_value( + s.get_value(&tid(T2), &sref("global-key"), None), + "global-val", ); -} - -#[test] -fn default_sharing_is_tenant_for_scoped_without_owner() { - let cfg = StaticCredStorePluginConfig { - secrets: vec![SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: None, - key: "t".to_owned(), - value: "v".to_owned(), - sharing: None, - }], - ..StaticCredStorePluginConfig::default() - }; - let service = Service::from_config(&cfg).unwrap(); - let key = SecretRef::new("t").unwrap(); - assert_eq!( - service - .get(&ctx(tenant_a(), owner_a()), &key) - .unwrap() - .sharing, - SharingMode::Tenant + assert_value( + s.get_value(&tid(T1), &sref("shared-key"), None), + "shared-val", ); } #[test] -fn default_sharing_is_private_for_scoped_with_owner() { - let cfg = StaticCredStorePluginConfig { - secrets: vec![SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: Some(owner_a()), - key: "p".to_owned(), - value: "v".to_owned(), - sharing: None, - }], - ..StaticCredStorePluginConfig::default() - }; - let service = Service::from_config(&cfg).unwrap(); - let key = SecretRef::new("p").unwrap(); - assert_eq!( - service - .get(&ctx(tenant_a(), owner_a()), &key) - .unwrap() - .sharing, - SharingMode::Private - ); +fn delete_missing_is_noop() { + let s = svc(vec![]); + s.delete_value(&tid(T1), &sref("absent"), None); + s.delete_value(&tid(T1), &sref("absent"), Some(&oid(O1))); + assert!(s.get_value(&tid(T1), &sref("absent"), None).is_none()); } #[test] -fn explicit_sharing_overrides_default() { - // tenant_id + no owner_id defaults to Tenant; override to Shared. - let cfg = StaticCredStorePluginConfig { - secrets: vec![SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: None, - key: "k".to_owned(), - value: "v".to_owned(), - sharing: Some(SharingMode::Shared), - }], - ..StaticCredStorePluginConfig::default() - }; - let service = Service::from_config(&cfg).unwrap(); - let key = SecretRef::new("k").unwrap(); - assert_eq!( - service - .get(&ctx(tenant_a(), owner_a()), &key) - .unwrap() - .sharing, - SharingMode::Shared - ); +fn from_config_rejects_nil_tenant() { + let nil = "00000000-0000-0000-0000-000000000000"; + let err = Service::from_config(&cfg(vec![secret(Some(nil), None, "k", "v")])) + .expect_err("nil tenant rejected"); + assert!(err.to_string().contains("nil UUID"), "{err}"); } -// --- Same key in different scopes --- - #[test] -fn allows_same_key_in_different_tenants() { - let cfg = StaticCredStorePluginConfig { - secrets: vec![ - SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: None, - key: "api_key".to_owned(), - value: "val-a".to_owned(), - sharing: None, - }, - SecretConfig { - tenant_id: Some(tenant_b()), - owner_id: None, - key: "api_key".to_owned(), - value: "val-b".to_owned(), - sharing: None, - }, - ], - ..StaticCredStorePluginConfig::default() - }; - let service = Service::from_config(&cfg).unwrap(); - let key = SecretRef::new("api_key").unwrap(); - - assert_eq!( - service - .get(&ctx(tenant_a(), owner_a()), &key) - .unwrap() - .value - .as_bytes(), - b"val-a" - ); - assert_eq!( - service - .get(&ctx(tenant_b(), owner_a()), &key) - .unwrap() - .value - .as_bytes(), - b"val-b" - ); +fn from_config_rejects_duplicate_tenant_key() { + let err = Service::from_config(&cfg(vec![ + secret(Some(T1), None, "dup", "a"), + secret(Some(T1), None, "dup", "b"), + ])) + .expect_err("duplicate rejected"); + assert!(err.to_string().contains("duplicate"), "{err}"); } #[test] -fn same_key_across_all_four_scopes() { - let cfg = StaticCredStorePluginConfig { - secrets: vec![ - SecretConfig { - tenant_id: None, - owner_id: None, - key: "k".to_owned(), - value: "global".to_owned(), - sharing: None, - }, - SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: None, - key: "k".to_owned(), - value: "shared".to_owned(), - sharing: Some(SharingMode::Shared), - }, - SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: None, - key: "k".to_owned(), - value: "tenant".to_owned(), - sharing: None, - }, - SecretConfig { - tenant_id: Some(tenant_a()), - owner_id: Some(owner_a()), - key: "k".to_owned(), - value: "private".to_owned(), - sharing: None, - }, - ], - ..StaticCredStorePluginConfig::default() - }; - - assert!(Service::from_config(&cfg).is_ok()); +fn from_config_rejects_invalid_secret_ref() { + let err = Service::from_config(&cfg(vec![secret(Some(T1), None, "bad key!", "v")])) + .expect_err("invalid ref rejected"); + assert!(!err.to_string().is_empty()); } diff --git a/gears/mini-chat/mini-chat/src/config.rs b/gears/mini-chat/mini-chat/src/config.rs index 81809f12c..85371fd1c 100644 --- a/gears/mini-chat/mini-chat/src/config.rs +++ b/gears/mini-chat/mini-chat/src/config.rs @@ -208,6 +208,16 @@ pub struct ProviderTenantOverride { pub auth_config: Option>, } +impl ProviderTenantOverride { + /// Whether this override can yield a distinct per-tenant OAGW upstream. + /// Only a `host` or an `upstream_alias` derives a distinct upstream; an + /// override with neither is inert (or auth-only) and cannot be registered. + #[must_use] + pub fn has_distinct_upstream(&self) -> bool { + self.host.is_some() || self.upstream_alias.is_some() + } +} + impl ProviderEntry { /// Effective host for a given tenant. Returns the tenant override host /// if configured, otherwise the root host. @@ -257,16 +267,18 @@ impl ProviderEntry { )); } - let overrides_auth = - tenant_override.auth_plugin_type.is_some() || tenant_override.auth_config.is_some(); - let has_distinct_upstream = - tenant_override.host.is_some() || tenant_override.upstream_alias.is_some(); - - if overrides_auth && !has_distinct_upstream { + // A tenant override must carry a host or an upstream_alias — that is + // the only way to derive a distinct OAGW upstream for the tenant. + // Without either, the override is inert (empty) or contradictory + // (e.g. auth-only), and registration cannot produce a per-tenant + // upstream. Reject it deterministically at boot (fail-fast) rather + // than leaving the provider silently unavailable at registration + // time (the `Misconfigured` path in `oagw_provisioning` is now only a + // defensive fallback for this). + if !tenant_override.has_distinct_upstream() { return Err(format!( - "provider '{provider_id}': tenant override '{tid}' overrides auth \ - without host or upstream_alias - \ - set one to create a distinct upstream" + "provider '{provider_id}': tenant override '{tid}' has neither host nor \ + upstream_alias - set one to create a distinct upstream for the tenant" )); } } @@ -1055,6 +1067,64 @@ fn default_vendor() -> String { mod tests { use super::*; + #[test] + fn provider_tenant_override_must_have_host_or_alias() { + fn base() -> ProviderEntry { + ProviderEntry { + kind: ProviderKind::OpenAiResponses, + upstream_alias: None, + host: "api.example.com".to_owned(), + port: None, + use_http: false, + api_path: default_api_path(), + auth_plugin_type: None, + auth_config: None, + storage_backend: None, + supports_file_search_filters: true, + storage_kind: StorageKind::OpenAi, + api_version: None, + rag_provider: None, + tenant_overrides: HashMap::new(), + } + } + fn ovr( + host: Option<&str>, + alias: Option<&str>, + auth: Option<&str>, + ) -> ProviderTenantOverride { + ProviderTenantOverride { + host: host.map(str::to_owned), + upstream_alias: alias.map(str::to_owned), + auth_plugin_type: auth.map(str::to_owned), + auth_config: None, + } + } + let with_override = |o: ProviderTenantOverride| { + let mut e = base(); + e.tenant_overrides.insert("t".to_owned(), o); + e + }; + + // No overrides, and host- or alias-carrying overrides are valid. + base().validate("p").expect("base valid"); + with_override(ovr(Some("t.example.com"), None, None)) + .validate("p") + .expect("host override valid"); + with_override(ovr(None, Some("t-alias"), None)) + .validate("p") + .expect("alias override valid"); + + // Neither host nor alias → fail-fast at boot, regardless of auth: the + // override can never yield a distinct per-tenant upstream. + let err = with_override(ovr(None, None, None)) + .validate("p") + .expect_err("empty override rejected"); + assert!(err.contains("neither host nor"), "got: {err}"); + with_override(ovr(None, None, Some("apikey"))) + .validate("p") + .expect_err("auth-only override rejected"); + } + #[test] fn default_config_is_valid() { StreamingConfig::default().validate().unwrap(); @@ -1538,7 +1608,7 @@ mod tests { }; let err = entry.validate("azure_openai").unwrap_err(); assert!(err.contains("tenant-a")); - assert!(err.contains("overrides auth")); + assert!(err.contains("neither host nor")); } #[test] @@ -1573,7 +1643,7 @@ mod tests { }; let err = entry.validate("azure_openai").unwrap_err(); assert!(err.contains("tenant-b")); - assert!(err.contains("overrides auth")); + assert!(err.contains("neither host nor")); } #[test] diff --git a/gears/mini-chat/mini-chat/src/gear.rs b/gears/mini-chat/mini-chat/src/gear.rs index 1eb977e75..9badbdb62 100644 --- a/gears/mini-chat/mini-chat/src/gear.rs +++ b/gears/mini-chat/mini-chat/src/gear.rs @@ -505,12 +505,40 @@ impl RunnableCapability for MiniChatGear { let ctx = exchange_client_credentials(&deferred.authn, &deferred.client_credentials).await?; let mut providers = deferred.providers.clone(); - crate::infra::oagw_provisioning::register_oagw_upstreams( + let deferred_ids = crate::infra::oagw_provisioning::register_oagw_upstreams( &deferred.gateway, &ctx, &mut providers, ) .await?; + + // Providers whose backend secret was not accessible at boot are + // registered lazily. With the stateful credstore, provider secrets + // are created at runtime via the credstore API, so the upstream + // cannot be registered until the secret exists. Retry in the + // background rather than blocking startup: start() must return for + // the server to report healthy, which is itself a precondition for + // the secret to be provisioned. + if !deferred_ids.is_empty() { + tracing::warn!( + deferred = ?deferred_ids, + "OAGW upstreams deferred at boot (secret not yet accessible); \ + retrying registration in the background" + ); + let gateway = Arc::clone(&deferred.gateway); + let authn = Arc::clone(&deferred.authn); + let creds = deferred.client_credentials.clone(); + let providers = providers.clone(); + let cancel = cancel.clone(); + tokio::spawn(reconcile_deferred_upstreams_with_retry( + gateway, + authn, + creds, + providers, + deferred_ids, + cancel, + )); + } } // Start the outbox pipeline now that OAGW upstreams are registered. @@ -739,6 +767,118 @@ impl MiniChatGear { } } +/// Background retry loop for providers whose OAGW upstream could not be +/// registered at boot because their credstore secret was not yet accessible. +/// +/// Re-exchanges an S2S context and re-attempts registration on a backing-off +/// cadence until every deferred provider is registered or the gear is +/// cancelled. There is no attempt budget: with the stateful credstore a +/// provider's secret can be provisioned at any time after boot, and giving up +/// would leave that provider unavailable until an operator restart. The +/// interval grows from `RETRY_INTERVAL_MIN` to `RETRY_INTERVAL_MAX` so late +/// provisioning is still picked up without hammering OAGW indefinitely. +/// Registration is idempotent, so any provider registered on a previous +/// attempt is reused rather than duplicated. +// The retry loop's `select!`/tracing macros inflate the measured cognitive +// complexity; the control flow itself is a simple backing-off loop. +#[allow(clippy::cognitive_complexity)] +async fn reconcile_deferred_upstreams_with_retry( + gateway: Arc, + authn: Arc, + creds: crate::config::ClientCredentialsConfig, + providers: std::collections::HashMap, + mut deferred: Vec, + cancel: CancellationToken, +) { + const RETRY_INTERVAL_MIN: Duration = Duration::from_secs(2); + const RETRY_INTERVAL_MAX: Duration = Duration::from_mins(1); + // Warn once (not on every slow tick) after this much wall-clock time still + // deferred, so an operator notices without log spam. Time-based, not + // attempt-based: the interval backs off, so a fixed attempt count would + // drift far from the intended window. + const WARN_AFTER: Duration = Duration::from_mins(2); + + let started = tokio::time::Instant::now(); + let mut warned = false; + let mut backoff = RETRY_INTERVAL_MIN; + let mut attempt: u32 = 0; + loop { + tokio::select! { + () = cancel.cancelled() => { + info!(remaining = deferred.len(), "OAGW upstream reconcile cancelled"); + return; + } + () = tokio::time::sleep(backoff) => {} + } + + attempt = attempt.saturating_add(1); + deferred = + reconcile_deferred_once(&gateway, &authn, &creds, &providers, deferred, attempt).await; + if deferred.is_empty() { + info!("OAGW reconcile: all deferred provider upstreams registered"); + return; + } + + if !warned && started.elapsed() >= WARN_AFTER { + warned = true; + tracing::warn!( + remaining = ?deferred, + elapsed_secs = started.elapsed().as_secs(), + "OAGW reconcile: provider(s) still UNAVAILABLE; will keep retrying at a slower \ + cadence until their secret is provisioned. Check for a missing secret or a \ + misconfigured secret_ref/host for these providers" + ); + } + backoff = (backoff * 2).min(RETRY_INTERVAL_MAX); + } +} + +/// One reconcile attempt: exchange a fresh S2S context and re-register the +/// still-deferred providers. Returns the ids that remain deferred (the input +/// unchanged on a transient failure, so the caller keeps retrying). +// Two match arms plus tracing macros push the measured complexity over the +// threshold; the logic is linear. +#[allow(clippy::cognitive_complexity)] +async fn reconcile_deferred_once( + gateway: &Arc, + authn: &Arc, + creds: &crate::config::ClientCredentialsConfig, + providers: &std::collections::HashMap, + deferred: Vec, + attempt: u32, +) -> Vec { + let ctx = match exchange_client_credentials(authn, creds).await { + Ok(ctx) => ctx, + Err(e) => { + tracing::warn!(error = %e, attempt, "OAGW reconcile: credential exchange failed; will retry"); + return deferred; + } + }; + + match crate::infra::oagw_provisioning::reconcile_deferred_upstreams( + gateway, &ctx, providers, &deferred, + ) + .await + { + Ok(still_deferred) => { + let registered = deferred.len().saturating_sub(still_deferred.len()); + if registered > 0 { + info!( + registered, + remaining = still_deferred.len(), + attempt, + "OAGW reconcile: registered deferred provider upstream(s)" + ); + } + still_deferred + } + Err(e) => { + tracing::warn!(error = %e, attempt, "OAGW reconcile attempt failed; will retry"); + deferred + } + } +} + /// Exchange `OAuth2` client credentials via the `AuthN` resolver to obtain /// a `SecurityContext` for OAGW upstream provisioning. async fn exchange_client_credentials( diff --git a/gears/mini-chat/mini-chat/src/infra/oagw_provisioning.rs b/gears/mini-chat/mini-chat/src/infra/oagw_provisioning.rs index 1cea0dddb..d708e8047 100644 --- a/gears/mini-chat/mini-chat/src/infra/oagw_provisioning.rs +++ b/gears/mini-chat/mini-chat/src/infra/oagw_provisioning.rs @@ -13,6 +13,7 @@ use std::sync::Arc; use oagw_sdk::HTTP_PROTOCOL_ID; use oagw_sdk::ServiceGatewayClientV1; +use toolkit_canonical_errors::CanonicalError; use tracing::{info, warn}; use crate::config::ProviderEntry; @@ -23,55 +24,213 @@ use crate::config::ProviderEntry; /// [`ProviderEntry::upstream_alias`] (root) and /// [`ProviderTenantOverride::upstream_alias`] (per-tenant). /// +/// Returns the ids of providers whose upstream could **not** be registered +/// because their backend secret is not yet accessible (see +/// [`create_or_reuse_upstream`]). With the stateful credstore, provider +/// secrets are created at runtime via the credstore API and may not exist at +/// boot; such providers are deferred so the caller can retry once the secret +/// is provisioned (see [`reconcile_deferred_upstreams`]) rather than crashing +/// the gear or leaving the provider permanently unavailable. +/// /// The caller is responsible for obtaining a valid `SecurityContext` /// (typically via S2S client credentials exchange). pub async fn register_oagw_upstreams( gateway: &Arc, ctx: &toolkit_security::SecurityContext, providers: &mut HashMap, -) -> anyhow::Result<()> { +) -> anyhow::Result> { + // Each provider is registered independently: a failure for one provider + // (secret not ready, transient OAGW error, or misconfiguration) never + // aborts registration of the others. A single persistently-failing + // provider must not starve its healthy peers of registration. + let mut deferred = Vec::new(); for (provider_id, entry) in providers.iter_mut() { - // Register root upstream + route. Fail hard — without upstreams the - // gear cannot proxy LLM requests. - let upstream = create_upstream(gateway, ctx, provider_id, entry) - .await - .ok_or_else(|| { - anyhow::anyhow!("OAGW upstream registration failed for provider '{provider_id}'") - })?; - entry.upstream_alias = Some(upstream.alias.clone()); - register_route(gateway, ctx, provider_id, entry, &upstream) - .await - .map_err(|e| { - anyhow::anyhow!("OAGW route registration failed for provider '{provider_id}': {e}") - })?; - - // Register tenant-specific upstreams (share the same route/api_path). - let tenant_ids: Vec = entry.tenant_overrides.keys().cloned().collect(); - for tenant_id in &tenant_ids { - let tenant_override = &entry.tenant_overrides[tenant_id]; - if tenant_override.host.is_none() && tenant_override.upstream_alias.is_none() { - anyhow::bail!( - "provider '{provider_id}': tenant override '{tenant_id}' \ - has no host and no upstream_alias - \ - cannot create distinct upstream" - ); - } + match register_provider(gateway, ctx, provider_id, entry).await { + ProviderRegistration::Deferred => deferred.push(provider_id.clone()), + // Registered: nothing to do. Misconfigured: already error-logged, and + // retrying cannot fix a deterministic config error, so it is not + // added to the retry set. + ProviderRegistration::Registered | ProviderRegistration::Misconfigured => {} + } + } - let label = format!("{provider_id}[tenant={tenant_id}]"); - let alias = create_tenant_upstream(gateway, ctx, &label, entry, tenant_id) - .await - .ok_or_else(|| { - anyhow::anyhow!( - "OAGW tenant upstream registration failed for provider '{provider_id}', tenant '{tenant_id}'" - ) - })?; - if let Some(tenant_override) = entry.tenant_overrides.get_mut(tenant_id) { - tenant_override.upstream_alias = Some(alias); + Ok(deferred) +} + +/// Outcome of registering a single provider's OAGW upstream(s) + route. +enum ProviderRegistration { + /// Root upstream, route, and every tenant-override upstream are registered. + Registered, + /// Could not complete — the backend secret is not accessible yet (with the + /// stateful credstore it may be provisioned after boot) or a transient OAGW + /// error occurred. The caller should retry (see [`reconcile_deferred_upstreams`]). + Deferred, + /// A deterministic misconfiguration that cannot self-heal (logged at error). + /// The provider is left unavailable; retrying would not help. + Misconfigured, +} + +/// Register one provider's root upstream, route, and tenant-override upstreams +/// as an isolated unit. Never panics or propagates — the outcome is reported +/// so the batch loop can keep going for other providers. +/// +/// `AlreadyExists` is treated as success throughout (the upstream/route survives +/// from a previous registration — restart / idempotent re-run / retry), so this +/// is safe to call repeatedly for the same provider. +// The per-step branches + tracing macros inflate the measured cognitive +// complexity; the control flow is a linear sequence of register-or-defer steps. +#[allow(clippy::cognitive_complexity)] +async fn register_provider( + gateway: &Arc, + ctx: &toolkit_security::SecurityContext, + provider_id: &str, + entry: &mut ProviderEntry, +) -> ProviderRegistration { + let Some(upstream) = create_or_reuse_upstream(gateway, ctx, provider_id, entry).await else { + return ProviderRegistration::Deferred; + }; + entry.upstream_alias = Some(upstream.alias.clone()); + + if let Err(e) = register_route(gateway, ctx, provider_id, entry, &upstream).await { + warn!(provider_id, error = %e, "OAGW route registration failed; deferring provider for retry"); + return ProviderRegistration::Deferred; + } + + // Tenant-specific upstreams (share the same route/api_path as the root). + let tenant_ids: Vec = entry.tenant_overrides.keys().cloned().collect(); + for tenant_id in &tenant_ids { + let tenant_override = &entry.tenant_overrides[tenant_id]; + if !tenant_override.has_distinct_upstream() { + // Defensive fallback: this deterministic misconfiguration is already + // rejected at boot by `ProviderEntry::validate` (fail-fast), so it + // should be unreachable here. Kept as defense-in-depth — surface it + // and leave the provider unavailable rather than crashing the gear. + tracing::error!( + provider_id, + tenant_id, + "tenant override has no host and no upstream_alias; \ + skipping provider (misconfiguration)" + ); + return ProviderRegistration::Misconfigured; + } + + match create_or_reuse_tenant_upstream(gateway, ctx, provider_id, entry, tenant_id).await { + Ok(alias) => { + if let Some(tenant_override) = entry.tenant_overrides.get_mut(tenant_id) { + tenant_override.upstream_alias = Some(alias); + } + } + Err(e) => { + warn!(provider_id, tenant_id, error = %e, "OAGW tenant upstream registration failed; deferring provider for retry"); + return ProviderRegistration::Deferred; } } } - Ok(()) + ProviderRegistration::Registered +} + +/// Retry registration for the providers [`register_oagw_upstreams`] deferred. +/// +/// Re-attempts registration for exactly `deferred` and returns the ids that +/// are **still** deferred (their secret is still not accessible). Registration +/// is idempotent — providers that succeeded on a previous attempt are reused +/// via `AlreadyExists` — so this is safe to call repeatedly. +pub async fn reconcile_deferred_upstreams( + gateway: &Arc, + ctx: &toolkit_security::SecurityContext, + providers: &HashMap, + deferred: &[String], +) -> anyhow::Result> { + let mut subset: HashMap = deferred + .iter() + .filter_map(|id| providers.get(id).map(|e| (id.clone(), e.clone()))) + .collect(); + register_oagw_upstreams(gateway, ctx, &mut subset).await +} + +/// Create the root upstream, or recover the existing one on `AlreadyExists`. +/// +/// Returns `None` (with a warning log) when the provider must be skipped: +/// registration failed for a reason other than a survived registration +/// (typically its credstore secret is not provisioned yet). +async fn create_or_reuse_upstream( + gateway: &Arc, + ctx: &toolkit_security::SecurityContext, + provider_id: &str, + entry: &ProviderEntry, +) -> Option { + match create_upstream(gateway, ctx, provider_id, entry).await { + Ok(u) => Some(u), + Err(CanonicalError::AlreadyExists { resource_name, .. }) => { + reuse_existing_upstream(gateway, ctx, provider_id, resource_name.as_deref()).await + } + Err(e) => { + warn!( + provider_id, + error = %e, + "skipping OAGW provisioning for provider: upstream registration failed \ + (its credstore secret may not be accessible yet); the provider will be \ + unavailable until its secret is provisioned" + ); + None + } + } +} + +/// Recover the upstream object behind an `AlreadyExists` conflict. +async fn reuse_existing_upstream( + gateway: &Arc, + ctx: &toolkit_security::SecurityContext, + provider_id: &str, + alias: Option<&str>, +) -> Option { + let Some(u) = find_upstream_by_alias(gateway, ctx, alias).await else { + warn!( + provider_id, + "OAGW upstream already exists but could not be looked up by alias; \ + skipping provider" + ); + return None; + }; + info!( + provider_id, + alias = %u.alias, + upstream_id = %u.id, + "OAGW upstream already registered; reusing existing" + ); + Some(u) +} + +/// Create a tenant-override upstream, or reuse the taken alias on +/// `AlreadyExists` (a survived registration from a previous run). +async fn create_or_reuse_tenant_upstream( + gateway: &Arc, + ctx: &toolkit_security::SecurityContext, + provider_id: &str, + entry: &ProviderEntry, + tenant_id: &str, +) -> anyhow::Result { + let label = format!("{provider_id}[tenant={tenant_id}]"); + match create_tenant_upstream(gateway, ctx, &label, entry, tenant_id).await { + Ok(alias) => Ok(alias), + // The conflict carries the taken alias — exactly what we need to reuse. + Err(CanonicalError::AlreadyExists { + resource_name: Some(alias), + .. + }) => { + info!( + label, + alias = %alias, + "OAGW tenant upstream already registered; reusing existing alias" + ); + Ok(alias) + } + Err(e) => anyhow::bail!( + "OAGW tenant upstream registration failed for provider \ + '{provider_id}', tenant '{tenant_id}': {e}" + ), + } } /// Create an OAGW upstream for a single provider entry. @@ -79,8 +238,6 @@ pub async fn register_oagw_upstreams( /// Only passes `upstream_alias` to OAGW when explicitly configured /// (required for IP-based hosts). For hostname-based hosts OAGW /// auto-derives the alias. -/// -/// Returns `None` (with a warning log) if registration fails. fn endpoint_for(entry: &ProviderEntry) -> oagw_sdk::Endpoint { use oagw_sdk::{Endpoint, Scheme}; let scheme = if entry.use_http { @@ -96,12 +253,42 @@ fn endpoint_for(entry: &ProviderEntry) -> oagw_sdk::Endpoint { } } +/// Look up an existing upstream by alias (paging through `list_upstreams`). +/// +/// Used to recover the upstream object when `create_upstream` reports +/// `AlreadyExists` — `alias` is the taken alias the conflict names. +async fn find_upstream_by_alias( + gateway: &Arc, + ctx: &toolkit_security::SecurityContext, + alias: Option<&str>, +) -> Option { + let alias = alias?; + let mut query = oagw_sdk::ListQuery::default(); + loop { + let page = match gateway.list_upstreams(ctx.clone(), &query).await { + Ok(page) => page, + Err(e) => { + warn!(alias, error = %e, "OAGW upstream lookup by alias failed"); + return None; + } + }; + let page_len = page.len(); + if let Some(u) = page.into_iter().find(|u| u.alias == alias) { + return Some(u); + } + if page_len < query.top as usize { + return None; + } + query.skip += query.top; + } +} + async fn create_upstream( gateway: &Arc, ctx: &toolkit_security::SecurityContext, provider_id: &str, entry: &ProviderEntry, -) -> Option { +) -> Result { use oagw_sdk::{AuthConfig, CreateUpstreamRequest, Server}; let server = Server { @@ -127,25 +314,16 @@ async fn create_upstream( builder = builder.headers(headers); } - match gateway.create_upstream(ctx.clone(), builder.build()).await { - Ok(u) => { - info!( - provider_id, - alias = %u.alias, - upstream_id = %u.id, - "OAGW upstream registered" - ); - Some(u) - } - Err(e) => { - warn!( - provider_id, - error = %e, - "OAGW upstream registration failed (may already exist)" - ); - None - } - } + let u = gateway + .create_upstream(ctx.clone(), builder.build()) + .await?; + info!( + provider_id, + alias = %u.alias, + upstream_id = %u.id, + "OAGW upstream registered" + ); + Ok(u) } /// Create an OAGW upstream for a tenant-specific override. @@ -155,14 +333,14 @@ async fn create_upstream( /// sets one (required for IP-based hosts). For hostname-based hosts OAGW /// auto-derives the alias. /// -/// Returns the OAGW-assigned alias on success, `None` on failure. +/// Returns the OAGW-assigned alias on success. async fn create_tenant_upstream( gateway: &Arc, ctx: &toolkit_security::SecurityContext, label: &str, entry: &ProviderEntry, tenant_id: &str, -) -> Option { +) -> Result { use oagw_sdk::{AuthConfig, CreateUpstreamRequest, Server}; let host = entry.effective_host_for_tenant(tenant_id); @@ -201,25 +379,16 @@ async fn create_tenant_upstream( builder = builder.headers(headers); } - match gateway.create_upstream(ctx.clone(), builder.build()).await { - Ok(u) => { - info!( - label, - alias = %u.alias, - upstream_id = %u.id, - "OAGW tenant upstream registered" - ); - Some(u.alias) - } - Err(e) => { - warn!( - label, - error = %e, - "OAGW tenant upstream registration failed (may already exist)" - ); - None - } - } + let u = gateway + .create_upstream(ctx.clone(), builder.build()) + .await?; + info!( + label, + alias = %u.alias, + upstream_id = %u.id, + "OAGW tenant upstream registered" + ); + Ok(u.alias) } /// Derive route match rules from `api_path` and register the OAGW route. @@ -249,21 +418,33 @@ async fn register_route( grpc: None, }; - let route = gateway + match gateway .create_route( ctx.clone(), CreateRouteRequest::builder(upstream.id, match_rules) .enabled(true) .build(), ) - .await?; - - info!( - provider_id, - route_id = %route.id, - route_path = %route_prefix, - "OAGW route registered" - ); + .await + { + Ok(route) => { + info!( + provider_id, + route_id = %route.id, + route_path = %route_prefix, + "OAGW route registered" + ); + } + // Idempotent re-run against a reused upstream: the route is already there. + Err(CanonicalError::AlreadyExists { .. }) => { + info!( + provider_id, + route_path = %route_prefix, + "OAGW route already registered; reusing existing" + ); + } + Err(e) => return Err(e.into()), + } // Register RAG-related routes (Files API, Vector Stores API) on the same upstream. register_rag_routes(gateway, ctx, provider_id, entry, upstream).await?; @@ -476,4 +657,501 @@ mod tests { let params = extract_query_allowlist("/path?key=&other=val"); assert_eq!(params, vec!["key", "other"]); } + + // ── Provisioning logic: registration / deferral / reuse / reconcile ────── + + use std::sync::Mutex; + use std::sync::atomic::{AtomicBool, AtomicU32, Ordering}; + + use toolkit_canonical_errors::{CanonicalError, resource_error}; + use toolkit_gts::gts_id; + use toolkit_security::SecurityContext; + + #[resource_error(gts_id!("cf.core.oagw.upstream.v1~"))] + struct TestUpstreamScope; + + /// Scripted `create_upstream` behavior for the mock gateway. + #[derive(Clone)] + enum CreateOutcome { + /// Registration succeeds (fresh upstream). + Ok, + /// Backend secret not accessible / transient error → caller should defer. + Defer, + /// The upstream already exists under the given alias (restart / retry). + AlreadyExists(String), + } + + /// Configurable in-memory `ServiceGatewayClientV1` for provisioning tests. + struct MockGw { + outcome: Mutex, + existing: Mutex>, + create_calls: AtomicU32, + list_calls: AtomicU32, + route_calls: AtomicU32, + fail_route: AtomicBool, + } + + impl MockGw { + fn new(outcome: CreateOutcome, existing: Vec) -> Arc { + Arc::new(Self { + outcome: Mutex::new(outcome), + existing: Mutex::new(existing), + create_calls: AtomicU32::new(0), + list_calls: AtomicU32::new(0), + route_calls: AtomicU32::new(0), + fail_route: AtomicBool::new(false), + }) + } + + fn set_outcome(&self, outcome: CreateOutcome) { + *self.outcome.lock().unwrap() = outcome; + } + + /// Make every subsequent `create_route` fail (transient OAGW error). + fn fail_routes(&self) { + self.fail_route.store(true, Ordering::SeqCst); + } + } + + fn upstream_with_alias(alias: &str) -> oagw_sdk::Upstream { + oagw_sdk::Upstream { + id: uuid::Uuid::new_v4(), + tenant_id: uuid::Uuid::nil(), + alias: alias.to_owned(), + server: oagw_sdk::Server { + endpoints: vec![oagw_sdk::Endpoint { + scheme: oagw_sdk::Scheme::Https, + host: "example.com".to_owned(), + port: 443, + }], + }, + protocol: gts_id!("cf.core.oagw.protocol.v1~cf.core.oagw.http.v1").to_owned(), + enabled: true, + auth: None, + headers: None, + plugins: None, + rate_limit: None, + cors: None, + tags: vec![], + } + } + + fn dummy_route() -> oagw_sdk::Route { + oagw_sdk::Route { + id: uuid::Uuid::new_v4(), + tenant_id: uuid::Uuid::nil(), + upstream_id: uuid::Uuid::nil(), + match_rules: oagw_sdk::MatchRules { + http: None, + grpc: None, + }, + plugins: None, + rate_limit: None, + cors: None, + tags: vec![], + priority: 0, + enabled: true, + } + } + + #[async_trait::async_trait] + impl ServiceGatewayClientV1 for MockGw { + async fn create_upstream( + &self, + _: SecurityContext, + _: oagw_sdk::CreateUpstreamRequest, + ) -> Result { + self.create_calls.fetch_add(1, Ordering::SeqCst); + match self.outcome.lock().unwrap().clone() { + CreateOutcome::Ok => Ok(upstream_with_alias("mock-upstream")), + // Any non-AlreadyExists error routes to the defer path. + CreateOutcome::Defer => Err(CanonicalError::service_unavailable().create()), + CreateOutcome::AlreadyExists(alias) => { + Err(TestUpstreamScope::already_exists("upstream already exists") + .with_resource(alias) + .create()) + } + } + } + async fn list_upstreams( + &self, + _: SecurityContext, + query: &oagw_sdk::ListQuery, + ) -> Result, CanonicalError> { + self.list_calls.fetch_add(1, Ordering::SeqCst); + let existing = self.existing.lock().unwrap(); + let page = existing + .iter() + .skip(query.skip as usize) + .take(query.top as usize) + .cloned() + .collect(); + Ok(page) + } + async fn create_route( + &self, + _: SecurityContext, + _: oagw_sdk::CreateRouteRequest, + ) -> Result { + self.route_calls.fetch_add(1, Ordering::SeqCst); + if self.fail_route.load(Ordering::SeqCst) { + return Err(CanonicalError::service_unavailable().create()); + } + Ok(dummy_route()) + } + async fn get_upstream( + &self, + _: SecurityContext, + _: uuid::Uuid, + ) -> Result { + unimplemented!() + } + async fn update_upstream( + &self, + _: SecurityContext, + _: uuid::Uuid, + _: oagw_sdk::UpdateUpstreamRequest, + ) -> Result { + unimplemented!() + } + async fn delete_upstream( + &self, + _: SecurityContext, + _: uuid::Uuid, + ) -> Result<(), CanonicalError> { + unimplemented!() + } + async fn get_route( + &self, + _: SecurityContext, + _: uuid::Uuid, + ) -> Result { + unimplemented!() + } + async fn list_routes( + &self, + _: SecurityContext, + _: Option, + _: &oagw_sdk::ListQuery, + ) -> Result, CanonicalError> { + unimplemented!() + } + async fn update_route( + &self, + _: SecurityContext, + _: uuid::Uuid, + _: oagw_sdk::UpdateRouteRequest, + ) -> Result { + unimplemented!() + } + async fn delete_route( + &self, + _: SecurityContext, + _: uuid::Uuid, + ) -> Result<(), CanonicalError> { + unimplemented!() + } + async fn resolve_proxy_target( + &self, + _: SecurityContext, + _: &str, + _: &str, + _: &str, + ) -> Result<(oagw_sdk::Upstream, oagw_sdk::Route), CanonicalError> { + unimplemented!() + } + async fn proxy_request( + &self, + _: SecurityContext, + _: http::Request, + ) -> Result, CanonicalError> { + unimplemented!() + } + } + + fn ctx() -> SecurityContext { + SecurityContext::anonymous() + } + + fn provider(host: &str) -> ProviderEntry { + ProviderEntry { + kind: crate::infra::llm::providers::ProviderKind::OpenAiResponses, + upstream_alias: None, + host: host.to_owned(), + port: None, + use_http: false, + api_path: "/v1/responses".to_owned(), + auth_plugin_type: None, + auth_config: None, + storage_backend: None, + supports_file_search_filters: true, + storage_kind: crate::config::StorageKind::OpenAi, + api_version: None, + rag_provider: None, + tenant_overrides: HashMap::new(), + } + } + + fn bad_tenant_override() -> crate::config::ProviderTenantOverride { + // No host and no upstream_alias → nothing to derive a distinct upstream + // from → deterministic misconfiguration. + crate::config::ProviderTenantOverride { + host: None, + upstream_alias: None, + auth_plugin_type: None, + auth_config: None, + } + } + + fn good_tenant_override(host: &str) -> crate::config::ProviderTenantOverride { + // A distinct host → a valid tenant-specific upstream can be derived. + crate::config::ProviderTenantOverride { + host: Some(host.to_owned()), + upstream_alias: None, + auth_plugin_type: None, + auth_config: None, + } + } + + #[tokio::test] + async fn register_provider_registers_on_success() { + let gw = MockGw::new(CreateOutcome::Ok, vec![]); + let dyn_gw: Arc = gw.clone(); + let mut entry = provider("api.openai.com"); + let outcome = register_provider(&dyn_gw, &ctx(), "openai", &mut entry).await; + assert!(matches!(outcome, ProviderRegistration::Registered)); + assert!(entry.upstream_alias.is_some()); + // The primary route plus the RAG routes are registered on success. + assert!(gw.route_calls.load(Ordering::SeqCst) >= 1); + } + + #[tokio::test] + async fn register_provider_defers_when_secret_not_accessible() { + let gw = MockGw::new(CreateOutcome::Defer, vec![]); + let dyn_gw: Arc = gw.clone(); + let mut entry = provider("api.openai.com"); + let outcome = register_provider(&dyn_gw, &ctx(), "openai", &mut entry).await; + assert!(matches!(outcome, ProviderRegistration::Deferred)); + // No route registered when the upstream itself was deferred. + assert_eq!(gw.route_calls.load(Ordering::SeqCst), 0); + } + + #[tokio::test] + async fn register_provider_reuses_existing_on_already_exists() { + let gw = MockGw::new( + CreateOutcome::AlreadyExists("api.openai.com".to_owned()), + vec![upstream_with_alias("api.openai.com")], + ); + let dyn_gw: Arc = gw.clone(); + let mut entry = provider("api.openai.com"); + let outcome = register_provider(&dyn_gw, &ctx(), "openai", &mut entry).await; + assert!(matches!(outcome, ProviderRegistration::Registered)); + assert_eq!(entry.upstream_alias.as_deref(), Some("api.openai.com")); + } + + #[tokio::test] + async fn register_provider_defers_when_already_exists_lookup_fails() { + // Conflict reported, but the alias is not found on lookup → cannot reuse. + let gw = MockGw::new( + CreateOutcome::AlreadyExists("api.openai.com".to_owned()), + vec![], + ); + let dyn_gw: Arc = gw.clone(); + let mut entry = provider("api.openai.com"); + let outcome = register_provider(&dyn_gw, &ctx(), "openai", &mut entry).await; + assert!(matches!(outcome, ProviderRegistration::Deferred)); + } + + #[tokio::test] + async fn register_provider_misconfigured_tenant_without_host_or_alias() { + let gw = MockGw::new(CreateOutcome::Ok, vec![]); + let dyn_gw: Arc = gw.clone(); + let mut entry = provider("api.openai.com"); + entry + .tenant_overrides + .insert("tenant-a".to_owned(), bad_tenant_override()); + let outcome = register_provider(&dyn_gw, &ctx(), "openai", &mut entry).await; + assert!(matches!(outcome, ProviderRegistration::Misconfigured)); + } + + #[tokio::test] + async fn register_batch_isolates_a_misconfigured_provider_from_healthy_peers() { + // One clean provider + one with a misconfigured tenant override. The bad + // one must not abort the batch: the clean one still registers regardless + // of (nondeterministic) HashMap iteration order. + let gw = MockGw::new(CreateOutcome::Ok, vec![]); + let dyn_gw: Arc = gw.clone(); + let mut providers = HashMap::new(); + providers.insert("good".to_owned(), provider("good.example.com")); + let mut bad = provider("bad.example.com"); + bad.tenant_overrides + .insert("tenant-a".to_owned(), bad_tenant_override()); + providers.insert("bad".to_owned(), bad); + + let deferred = register_oagw_upstreams(&dyn_gw, &ctx(), &mut providers) + .await + .unwrap(); + + // The misconfigured provider is not deferred for retry (won't self-heal), + // and the healthy provider was registered despite it. + assert!( + deferred.is_empty(), + "misconfig must not be queued for retry" + ); + assert!( + providers["good"].upstream_alias.is_some(), + "healthy provider must register even when a peer is misconfigured" + ); + } + + #[tokio::test] + async fn register_batch_defers_all_unavailable_providers_without_stopping() { + let gw = MockGw::new(CreateOutcome::Defer, vec![]); + let dyn_gw: Arc = gw.clone(); + let mut providers = HashMap::new(); + providers.insert("a".to_owned(), provider("a.example.com")); + providers.insert("b".to_owned(), provider("b.example.com")); + + let mut deferred = register_oagw_upstreams(&dyn_gw, &ctx(), &mut providers) + .await + .unwrap(); + deferred.sort(); + // Both deferred → the loop did not bail after the first failure. + assert_eq!(deferred, vec!["a".to_owned(), "b".to_owned()]); + } + + #[tokio::test] + async fn reconcile_converges_once_secret_becomes_available() { + let gw = MockGw::new(CreateOutcome::Defer, vec![]); + let dyn_gw: Arc = gw.clone(); + let mut providers = HashMap::new(); + providers.insert("openai".to_owned(), provider("api.openai.com")); + + let deferred = register_oagw_upstreams(&dyn_gw, &ctx(), &mut providers) + .await + .unwrap(); + assert_eq!(deferred, vec!["openai".to_owned()]); + + // Secret gets provisioned at runtime → the next reconcile registers it. + gw.set_outcome(CreateOutcome::Ok); + let still = reconcile_deferred_upstreams(&dyn_gw, &ctx(), &providers, &deferred) + .await + .unwrap(); + assert!( + still.is_empty(), + "provider should register once secret exists" + ); + } + + #[tokio::test] + async fn find_upstream_by_alias_pages_across_multiple_pages() { + // 50 non-matching (a full first page) + the target on the second page. + let mut ups: Vec<_> = (0..50) + .map(|i| upstream_with_alias(&format!("other-{i}"))) + .collect(); + ups.push(upstream_with_alias("target")); + let gw = MockGw::new(CreateOutcome::Ok, ups); + let dyn_gw: Arc = gw.clone(); + + let found = find_upstream_by_alias(&dyn_gw, &ctx(), Some("target")).await; + assert_eq!(found.map(|u| u.alias), Some("target".to_owned())); + assert!( + gw.list_calls.load(Ordering::SeqCst) >= 2, + "must have paged past the first full page" + ); + } + + #[tokio::test] + async fn find_upstream_by_alias_terminates_when_absent() { + let ups: Vec<_> = (0..10) + .map(|i| upstream_with_alias(&format!("other-{i}"))) + .collect(); + let gw = MockGw::new(CreateOutcome::Ok, ups); + let dyn_gw: Arc = gw.clone(); + + let found = find_upstream_by_alias(&dyn_gw, &ctx(), Some("absent")).await; + assert!(found.is_none()); + // A short-than-full page ends paging (no infinite loop). + assert_eq!(gw.list_calls.load(Ordering::SeqCst), 1); + } + + #[tokio::test] + async fn find_upstream_by_alias_none_when_no_alias() { + let gw = MockGw::new(CreateOutcome::Ok, vec![]); + let dyn_gw: Arc = gw.clone(); + assert!( + find_upstream_by_alias(&dyn_gw, &ctx(), None) + .await + .is_none() + ); + assert_eq!(gw.list_calls.load(Ordering::SeqCst), 0); + } + + #[tokio::test] + async fn register_provider_registers_tenant_override_upstream() { + // A provider with a valid tenant override (distinct host) registers the + // root upstream *and* a tenant-specific one, stamping the OAGW-assigned + // alias onto the override. + let gw = MockGw::new(CreateOutcome::Ok, vec![]); + let dyn_gw: Arc = gw.clone(); + let mut entry = provider("api.openai.com"); + entry.tenant_overrides.insert( + "tenant-a".to_owned(), + good_tenant_override("tenant-a.openai.com"), + ); + + let outcome = register_provider(&dyn_gw, &ctx(), "openai", &mut entry).await; + + assert!(matches!(outcome, ProviderRegistration::Registered)); + assert_eq!( + entry.tenant_overrides["tenant-a"].upstream_alias.as_deref(), + Some("mock-upstream"), + "the OAGW-assigned alias must be stamped onto the tenant override" + ); + // Two upstreams created: the root and the tenant-specific one. + assert_eq!(gw.create_calls.load(Ordering::SeqCst), 2); + } + + #[tokio::test] + async fn register_provider_reuses_existing_tenant_upstream_on_already_exists() { + // On restart the tenant upstream already exists: the `AlreadyExists` + // conflict carries the taken alias, reused directly without a lookup. + let gw = MockGw::new( + CreateOutcome::AlreadyExists("api.openai.com".to_owned()), + vec![upstream_with_alias("api.openai.com")], + ); + let dyn_gw: Arc = gw.clone(); + let mut entry = provider("api.openai.com"); + entry.tenant_overrides.insert( + "tenant-a".to_owned(), + good_tenant_override("tenant-a.openai.com"), + ); + + let outcome = register_provider(&dyn_gw, &ctx(), "openai", &mut entry).await; + + assert!(matches!(outcome, ProviderRegistration::Registered)); + assert_eq!( + entry.tenant_overrides["tenant-a"].upstream_alias.as_deref(), + Some("api.openai.com"), + "tenant override reuses the alias named by the AlreadyExists conflict" + ); + } + + #[tokio::test] + async fn register_provider_defers_when_route_registration_fails() { + // Upstream registers, but the route call hits a transient OAGW error → + // the provider is deferred for retry (not crashed, not misconfigured). + let gw = MockGw::new(CreateOutcome::Ok, vec![]); + gw.fail_routes(); + let dyn_gw: Arc = gw.clone(); + let mut entry = provider("api.openai.com"); + + let outcome = register_provider(&dyn_gw, &ctx(), "openai", &mut entry).await; + + assert!(matches!(outcome, ProviderRegistration::Deferred)); + // The root upstream was created; the route attempt failed and aborted + // the provider before any tenant work. + assert_eq!(gw.create_calls.load(Ordering::SeqCst), 1); + assert!(gw.route_calls.load(Ordering::SeqCst) >= 1); + } } diff --git a/gears/system/oagw/oagw/Cargo.toml b/gears/system/oagw/oagw/Cargo.toml index ea6e7ee76..79c108deb 100644 --- a/gears/system/oagw/oagw/Cargo.toml +++ b/gears/system/oagw/oagw/Cargo.toml @@ -21,6 +21,7 @@ path = "src/lib.rs" fips = ["toolkit-http/fips"] test-utils = [ "axum/ws", + "credstore-sdk/test-util", "toolkit/bootstrap", "dep:async-stream", "dep:tower", @@ -90,6 +91,7 @@ rcgen = { workspace = true, optional = true } [dev-dependencies] cf-gears-oagw = { path = ".", features = ["test-utils"] } +credstore-sdk = { workspace = true, features = ["test-util"] } toolkit = { workspace = true, features = ["bootstrap"] } opentelemetry_sdk = { workspace = true, features = ["testing"] } types-registry-sdk = { workspace = true, features = ["test-util"] } diff --git a/gears/system/oagw/oagw/src/domain/test_support.rs b/gears/system/oagw/oagw/src/domain/test_support.rs index 4e295d612..a4e3ba87b 100644 --- a/gears/system/oagw/oagw/src/domain/test_support.rs +++ b/gears/system/oagw/oagw/src/domain/test_support.rs @@ -17,10 +17,7 @@ use authz_resolver_sdk::{ AuthZResolverClient, AuthZResolverError, EvaluationRequest, EvaluationResponse, EvaluationResponseContext, PolicyEnforcer, }; -use credstore_sdk::{ - CredStoreClientV1, CredStoreError, GetSecretResponse, SecretRef, SecretValue, SharingMode, - TenantId as CredstoreTenantId, -}; +use credstore_sdk::CredStoreClientV1; use oagw_sdk::api::ServiceGatewayClientV1; use tenant_resolver_sdk::{ GetAncestorsOptions, GetAncestorsResponse, GetDescendantsOptions, GetDescendantsResponse, @@ -161,68 +158,13 @@ impl AuthZResolverClient for CapturingAuthZResolverClient { } } -/// Mock `CredStoreClientV1` for tests. Stores secrets in memory keyed by -/// the bare secret name (without `cred://` prefix). -pub struct MockCredStoreClient { - store: HashMap>, -} - -impl MockCredStoreClient { - /// Create a mock pre-loaded with secrets. - /// - /// Keys may optionally include the `cred://` prefix — it is stripped - /// automatically so lookups work regardless of the prefix convention. - pub fn with_secrets(creds: Vec<(String, String)>) -> Self { - let store = creds - .into_iter() - .map(|(k, v)| { - let key = k.strip_prefix("cred://").unwrap_or(k.as_str()).to_string(); - (key, v.into_bytes()) - }) - .collect(); - Self { store } - } - - /// Create an empty mock (all lookups return `Ok(None)`). - pub fn empty() -> Self { - Self { - store: HashMap::new(), - } - } -} - -#[async_trait] -impl CredStoreClientV1 for MockCredStoreClient { - async fn get( - &self, - _ctx: &SecurityContext, - key: &SecretRef, - ) -> Result, CredStoreError> { - Ok(self.store.get(key.as_ref()).map(|v| GetSecretResponse { - value: SecretValue::new(v.clone()), - owner_tenant_id: CredstoreTenantId::nil(), - sharing: SharingMode::default(), - is_inherited: false, - })) - } -} - -/// Mock `CredStoreClientV1` that always returns `CredStoreError::Internal`. -/// Useful for testing error-handling paths. -#[cfg(test)] -pub struct FailingCredStoreClient; - -#[cfg(test)] -#[async_trait] -impl CredStoreClientV1 for FailingCredStoreClient { - async fn get( - &self, - _ctx: &SecurityContext, - _key: &SecretRef, - ) -> Result, CredStoreError> { - Err(CredStoreError::Internal("backend failure".into())) - } -} +// The `CredStoreClientV1` test double is centralized in the SDK +// (`credstore_sdk::test_util`, behind its `test-util` feature) so every gear +// shares one configurable mock instead of hand-rolling its own. Modes: +// `empty()` (all `get` → None), `with_secrets(..)` (keyed store), +// `returning_raw_value(..)` (any ref → fixed bytes, e.g. non-UTF-8), +// `always_failing()` (every op → `Internal`). +pub use credstore_sdk::test_util::MockCredStoreClient; /// Re-export for tests that need a `CredStoreClientV1` mock. pub use MockCredStoreClient as TestCredStoreClient; diff --git a/gears/system/oagw/oagw/src/infra/plugin/apikey_auth.rs b/gears/system/oagw/oagw/src/infra/plugin/apikey_auth.rs index f85478af9..0323b0dad 100644 --- a/gears/system/oagw/oagw/src/infra/plugin/apikey_auth.rs +++ b/gears/system/oagw/oagw/src/infra/plugin/apikey_auth.rs @@ -69,15 +69,11 @@ mod tests { use std::collections::HashMap; use std::sync::Arc; - use credstore_sdk::{ - CredStoreClientV1, CredStoreError, GetSecretResponse, SecretRef, SecretValue, SharingMode, - TenantId as CredstoreTenantId, - }; use toolkit_security::SecurityContext; use uuid::Uuid; use crate::domain::plugin::{AuthContext, AuthPlugin, PluginError}; - use crate::domain::test_support::{FailingCredStoreClient, MockCredStoreClient}; + use crate::domain::test_support::MockCredStoreClient; use super::*; @@ -177,7 +173,7 @@ mod tests { #[tokio::test] async fn credstore_error_maps_to_internal() { - let plugin = ApiKeyAuthPlugin::new(Arc::new(FailingCredStoreClient)); + let plugin = ApiKeyAuthPlugin::new(Arc::new(MockCredStoreClient::always_failing())); let mut ctx = make_auth_ctx(make_config("authorization", "Bearer ", "cred://some-key")); let err = plugin.authenticate(&mut ctx).await.unwrap_err(); @@ -186,25 +182,11 @@ mod tests { #[tokio::test] async fn invalid_utf8_in_secret_returns_internal_error() { - struct Utf8ErrorCredStore; - - #[async_trait::async_trait] - impl CredStoreClientV1 for Utf8ErrorCredStore { - async fn get( - &self, - _ctx: &SecurityContext, - _key: &SecretRef, - ) -> Result, CredStoreError> { - Ok(Some(GetSecretResponse { - value: SecretValue::new(vec![0xFF, 0xFE]), - owner_tenant_id: CredstoreTenantId::nil(), - sharing: SharingMode::default(), - is_inherited: false, - })) - } - } - - let plugin = ApiKeyAuthPlugin::new(Arc::new(Utf8ErrorCredStore)); + // Any reference resolves to non-UTF-8 bytes. + let plugin = + ApiKeyAuthPlugin::new(Arc::new(MockCredStoreClient::returning_raw_value(vec![ + 0xFF, 0xFE, + ]))); let mut ctx = make_auth_ctx(make_config("authorization", "", "cred://bad-utf8")); let err = plugin.authenticate(&mut ctx).await.unwrap_err(); diff --git a/gears/system/oagw/oagw/src/infra/plugin/oauth2_client_cred_auth.rs b/gears/system/oagw/oagw/src/infra/plugin/oauth2_client_cred_auth.rs index 0b229dbdf..da3c8b848 100644 --- a/gears/system/oagw/oagw/src/infra/plugin/oauth2_client_cred_auth.rs +++ b/gears/system/oagw/oagw/src/infra/plugin/oauth2_client_cred_auth.rs @@ -260,13 +260,10 @@ mod tests { use toolkit_security::SecurityContext; use uuid::Uuid; - use credstore_sdk::{ - CredStoreClientV1, CredStoreError, GetSecretResponse, SecretRef, SecretValue, SharingMode, - TenantId as CredstoreTenantId, - }; + use credstore_sdk::CredStoreClientV1; use crate::domain::plugin::{AuthContext, AuthPlugin, PluginError}; - use crate::domain::test_support::{FailingCredStoreClient, MockCredStoreClient}; + use crate::domain::test_support::MockCredStoreClient; use super::*; @@ -463,7 +460,7 @@ mod tests { #[tokio::test] async fn credstore_error_maps_to_internal() { let server = MockServer::start(); - let plugin = make_plugin(Arc::new(FailingCredStoreClient)); + let plugin = make_plugin(Arc::new(MockCredStoreClient::always_failing())); let mut ctx = make_auth_ctx(make_config(&server)); let err = plugin.authenticate(&mut ctx).await.unwrap_err(); @@ -472,26 +469,11 @@ mod tests { #[tokio::test] async fn invalid_utf8_secret_returns_internal_error() { - struct Utf8ErrorCredStore; - - #[async_trait::async_trait] - impl CredStoreClientV1 for Utf8ErrorCredStore { - async fn get( - &self, - _ctx: &toolkit_security::SecurityContext, - _key: &SecretRef, - ) -> Result, CredStoreError> { - Ok(Some(GetSecretResponse { - value: SecretValue::new(vec![0xFF, 0xFE]), - owner_tenant_id: CredstoreTenantId::nil(), - sharing: SharingMode::default(), - is_inherited: false, - })) - } - } - let server = MockServer::start(); - let plugin = make_plugin(Arc::new(Utf8ErrorCredStore)); + // Any reference resolves to non-UTF-8 bytes. + let plugin = make_plugin(Arc::new(MockCredStoreClient::returning_raw_value(vec![ + 0xFF, 0xFE, + ]))); let mut ctx = make_auth_ctx(make_config(&server)); let err = plugin.authenticate(&mut ctx).await.unwrap_err(); diff --git a/gears/system/oagw/oagw/src/infra/proxy/service.rs b/gears/system/oagw/oagw/src/infra/proxy/service.rs index dadaef6cf..18b3a3b12 100644 --- a/gears/system/oagw/oagw/src/infra/proxy/service.rs +++ b/gears/system/oagw/oagw/src/infra/proxy/service.rs @@ -1892,7 +1892,7 @@ mod tests { AuthZResolverClient, AuthZResolverError, EvaluationRequest, EvaluationResponse, EvaluationResponseContext, PolicyEnforcer, }; - use credstore_sdk::{CredStoreClientV1, CredStoreError, GetSecretResponse, SecretRef}; + use credstore_sdk::CredStoreClientV1; use toolkit_security::SecurityContext; struct AllowAllAuthZ; @@ -1912,19 +1912,8 @@ mod tests { } } - struct NoopCredStore; - #[async_trait] - impl CredStoreClientV1 for NoopCredStore { - async fn get( - &self, - _ctx: &SecurityContext, - _key: &SecretRef, - ) -> Result, CredStoreError> { - Ok(None) - } - } - - let credstore: Arc = Arc::new(NoopCredStore); + let credstore: Arc = + Arc::new(credstore_sdk::test_util::MockCredStoreClient::empty()); let policy_enforcer = PolicyEnforcer::new(Arc::new(AllowAllAuthZ)); // Minimal CP — never called by select_endpoint(). diff --git a/testing/e2e/gears/credstore/__init__.py b/testing/e2e/gears/credstore/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/testing/e2e/gears/credstore/conftest.py b/testing/e2e/gears/credstore/conftest.py new file mode 100644 index 000000000..efe5c3810 --- /dev/null +++ b/testing/e2e/gears/credstore/conftest.py @@ -0,0 +1,118 @@ +"""Pytest fixtures for CredStore E2E tests. + +The suite runs against the standard e2e server (``config/e2e-local.yaml``). +Tokens map to static identities (static-authn-plugin) inside the static +tenant tree (static-tr-plugin):: + + e2e-root (00000000-df51-...953) <- e2e-token-tenant-a + hierarchy-root (...0001) <- e2e-token-hierarchy-root + hierarchy-l1a (...0002) <- e2e-token-hierarchy-l1a + hierarchy-l1b (...0005) <- e2e-token-hierarchy-l1b + +Every test creates its own uniquely-named secrets (``unique_ref``) and +registers them for teardown (``cleanup``), so tests are order-independent and +re-runnable against a shared long-lived server. +""" +from __future__ import annotations + +import os +import uuid + +import httpx +import pytest + +TENANT_A = "00000000-df51-5b42-9538-d2b56b7ee953" +HIERARCHY_ROOT = "00000000-0000-0000-0000-000000000001" +HIERARCHY_L1A = "00000000-0000-0000-0000-000000000002" +HIERARCHY_L1B = "00000000-0000-0000-0000-000000000005" + + +@pytest.fixture +def base_url(): + """API Gateway base URL.""" + return os.getenv("E2E_BASE_URL", "http://localhost:8086") + + +def _bearer(token: str) -> dict: + return {"Authorization": f"Bearer {token}"} + + +@pytest.fixture +def tenant_a_headers(): + """Headers for the e2e-root tenant (root of the whole tree).""" + return _bearer(os.getenv("E2E_AUTH_TOKEN", "e2e-token-tenant-a")) + + +@pytest.fixture +def root_headers(): + """Headers for hierarchy-root (...0001) — parent of l1a and l1b.""" + return _bearer("e2e-token-hierarchy-root") + + +@pytest.fixture +def l1a_headers(): + """Headers for hierarchy-l1a (...0002), child of hierarchy-root.""" + return _bearer("e2e-token-hierarchy-l1a") + + +@pytest.fixture +def l1b_headers(): + """Headers for hierarchy-l1b (...0005), sibling of l1a.""" + return _bearer("e2e-token-hierarchy-l1b") + + +@pytest.fixture +def unique_ref(): + """Factory for unique secret references (safe on a shared server).""" + + def make(prefix: str = "e2e-cs") -> str: + return f"{prefix}-{uuid.uuid4().hex[:12]}" + + return make + + +@pytest.fixture +def secrets_url(base_url): + """CredStore secrets collection URL.""" + return f"{base_url}/credstore/v1/secrets" + + +@pytest.fixture +def cleanup(secrets_url): + """Register (headers, ref) pairs; teardown deletes them best-effort. + + A private and a tenant/shared secret coexist under one reference and + DELETE removes the caller's private one first, so deletion is retried + until 404 (bounded). + """ + registered: list[tuple[dict, str]] = [] + + def register(headers: dict, ref: str) -> str: + registered.append((headers, ref)) + return ref + + yield register + + with httpx.Client(timeout=10.0) as client: + for headers, ref in registered: + for _ in range(3): + try: + resp = client.delete(f"{secrets_url}/{ref}", headers=headers) + except httpx.RequestError: + break + if resp.status_code != 204: + break + + +@pytest.fixture(scope="session", autouse=True) +def _check_credstore_reachable(): + """Skip the whole suite when the e2e server is not running.""" + url = os.getenv("E2E_BASE_URL", "http://localhost:8086") + try: + # Any HTTP response (401/404 included) means the gateway is up. + httpx.get(f"{url}/credstore/v1/secrets/e2e-reachability-probe", timeout=5.0) + except httpx.ConnectError: + pytest.skip(f"e2e server not running at {url}", allow_module_level=True) + except Exception: + # Timeout or transient error — still try to run the tests. + pass diff --git a/testing/e2e/gears/credstore/test_concurrency.py b/testing/e2e/gears/credstore/test_concurrency.py new file mode 100644 index 000000000..88cc55b4c --- /dev/null +++ b/testing/e2e/gears/credstore/test_concurrency.py @@ -0,0 +1,342 @@ +"""CredStore E2E: optimistic concurrency (generation-bound ETag / If-Match). + +GET returns a strong, generation-bound ETag of the form ``"."`` +(``id`` = the metadata row's UUID, minted fresh for every recreated secret; +``version`` = the per-generation monotonic counter). PUT and DELETE honour +``If-Match: *`` (target must exist) and ``If-Match: "."``. +A failed precondition is the canonical 409 (OPTIMISTIC_LOCK_FAILURE) — the +canonical error model has no 412. A validator minted for an earlier +generation of a deleted-and-recreated secret never matches, even when the +version counters coincide (no ABA lost update). +""" +import asyncio +import re +import uuid + +import httpx +import pytest + +# ``"."`` — the strong validator's wire shape. +ETAG_RE = re.compile(r'^"([0-9a-f-]{36})\.(\d+)"$') + + +def parse_etag(etag: str) -> tuple[str, int]: + """Split a composite ETag into its (generation id, version) halves.""" + m = ETAG_RE.match(etag) + assert m, f"ETag {etag!r} is not a composite . validator" + return m.group(1), int(m.group(2)) + + +class TestIfMatchPut: + """Guarded updates.""" + + @pytest.mark.smoke + @pytest.mark.asyncio + async def test_version_bumps_and_guarded_put_succeeds( + self, secrets_url, tenant_a_headers, unique_ref, cleanup + ): + """ETag tracks the version within a generation; a matching If-Match PUT commits.""" + ref = cleanup(tenant_a_headers, unique_ref("e2e-oc")) + + async with httpx.AsyncClient(timeout=10.0) as client: + await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": "v1", "sharing": "tenant"}, + ) + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + etag = resp.headers["etag"] + gen, version = parse_etag(etag) + assert version == 1 + + resp = await client.put( + f"{secrets_url}/{ref}", + headers={**tenant_a_headers, "If-Match": etag}, + json={"value": "v2", "sharing": "tenant"}, + ) + assert resp.status_code == 204, resp.text + + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + assert resp.json()["value"] == "v2" + assert resp.headers["etag"] == f'"{gen}.2"', ( + "an in-place overwrite bumps the version within the same generation" + ) + + @pytest.mark.asyncio + async def test_stale_if_match_put_conflicts_without_side_effect( + self, secrets_url, tenant_a_headers, unique_ref, cleanup + ): + """A stale If-Match PUT is a 409 and changes nothing.""" + ref = cleanup(tenant_a_headers, unique_ref("e2e-oc")) + + async with httpx.AsyncClient(timeout=10.0) as client: + await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": "v1", "sharing": "tenant"}, + ) + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + etag = resp.headers["etag"] + gen, _ = parse_etag(etag) + + resp = await client.put( + f"{secrets_url}/{ref}", + headers={**tenant_a_headers, "If-Match": f'"{gen}.99"'}, + json={"value": "lost-update", "sharing": "tenant"}, + ) + assert resp.status_code == 409, resp.text + + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + assert resp.json()["value"] == "v1" + assert resp.headers["etag"] == etag + + @pytest.mark.asyncio + async def test_recreated_secret_rejects_previous_generation_validator( + self, secrets_url, tenant_a_headers, unique_ref, cleanup + ): + """No ABA: delete + recreate restarts the version counter, but the old + generation's validator must never match the new row.""" + ref = cleanup(tenant_a_headers, unique_ref("e2e-oc-aba")) + + async with httpx.AsyncClient(timeout=10.0) as client: + await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": "gen1", "sharing": "tenant"}, + ) + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + gen1_etag = resp.headers["etag"] + _, gen1_version = parse_etag(gen1_etag) + + # Delete and recreate: a fresh generation whose counter restarts. + resp = await client.delete( + f"{secrets_url}/{ref}", headers=tenant_a_headers + ) + assert resp.status_code == 204, resp.text + await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": "gen2", "sharing": "tenant"}, + ) + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + gen2_etag = resp.headers["etag"] + _, gen2_version = parse_etag(gen2_etag) + assert gen2_version == gen1_version, "the ABA setup: versions coincide" + assert gen2_etag != gen1_etag, "a recreated secret is a new generation" + + # The stale generation's validator must not overwrite gen2. + resp = await client.put( + f"{secrets_url}/{ref}", + headers={**tenant_a_headers, "If-Match": gen1_etag}, + json={"value": "stale-write", "sharing": "tenant"}, + ) + assert resp.status_code == 409, resp.text + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + assert resp.json()["value"] == "gen2", "no lost update across generations" + + @pytest.mark.asyncio + async def test_if_match_on_missing_target_conflicts( + self, secrets_url, tenant_a_headers, unique_ref + ): + """If-Match requires an existing target: PUT on a fresh ref is 409.""" + ref = unique_ref("e2e-oc-missing") + + async with httpx.AsyncClient(timeout=10.0) as client: + for if_match in ("*", f'"{uuid.uuid4()}.1"'): + resp = await client.put( + f"{secrets_url}/{ref}", + headers={**tenant_a_headers, "If-Match": if_match}, + json={"value": "v", "sharing": "tenant"}, + ) + assert resp.status_code == 409, (if_match, resp.text) + + # No side effect: the reference was never created. + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + assert resp.status_code == 404 + + @pytest.mark.asyncio + async def test_malformed_if_match_is_400( + self, secrets_url, tenant_a_headers, unique_ref, cleanup + ): + """If-Match must be * or a quoted . pair.""" + ref = cleanup(tenant_a_headers, unique_ref("e2e-oc")) + + async with httpx.AsyncClient(timeout=10.0) as client: + await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": "v1", "sharing": "tenant"}, + ) + # A bare quoted version (the pre-composite format) is malformed too. + for bad in ("not-a-version", '"1"', '"not-a-uuid.1"'): + resp = await client.put( + f"{secrets_url}/{ref}", + headers={**tenant_a_headers, "If-Match": bad}, + json={"value": "v2", "sharing": "tenant"}, + ) + assert resp.status_code == 400, (bad, resp.text) + + +class TestIfMatchDelete: + """Guarded deletes.""" + + @pytest.mark.asyncio + async def test_stale_if_match_delete_conflicts_then_matching_deletes( + self, secrets_url, tenant_a_headers, unique_ref, cleanup + ): + """A stale If-Match DELETE is a 409; the matching one commits.""" + ref = cleanup(tenant_a_headers, unique_ref("e2e-oc-del")) + + async with httpx.AsyncClient(timeout=10.0) as client: + await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": "v1", "sharing": "tenant"}, + ) + # Bump to version 2 so the guard is meaningful. + await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": "v2", "sharing": "tenant"}, + ) + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + etag = resp.headers["etag"] + gen, version = parse_etag(etag) + assert version == 2 + + resp = await client.delete( + f"{secrets_url}/{ref}", + headers={**tenant_a_headers, "If-Match": f'"{gen}.1"'}, + ) + assert resp.status_code == 409, resp.text + # Still readable — the stale delete had no effect. + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + assert resp.status_code == 200 + assert resp.headers["etag"] == etag + + resp = await client.delete( + f"{secrets_url}/{ref}", + headers={**tenant_a_headers, "If-Match": etag}, + ) + assert resp.status_code == 204, resp.text + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + assert resp.status_code == 404 + + @pytest.mark.asyncio + async def test_if_match_star_delete_requires_existence( + self, secrets_url, tenant_a_headers, unique_ref, cleanup + ): + """If-Match: * deletes an existing secret; a missing one is 404.""" + ref = cleanup(tenant_a_headers, unique_ref("e2e-oc-star")) + + async with httpx.AsyncClient(timeout=10.0) as client: + resp = await client.delete( + f"{secrets_url}/{ref}", + headers={**tenant_a_headers, "If-Match": "*"}, + ) + assert resp.status_code == 404, resp.text + + await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": "v", "sharing": "tenant"}, + ) + resp = await client.delete( + f"{secrets_url}/{ref}", + headers={**tenant_a_headers, "If-Match": "*"}, + ) + assert resp.status_code == 204, resp.text + + +class TestConcurrentRacingRequests: + """Genuinely parallel requests (``asyncio.gather``), not the sequential + ``await``s the rest of this module uses. These exercise the server-side + optimistic-lock CAS and the crosswise dual-write fence under real overlap. + """ + + @pytest.mark.asyncio + async def test_racing_guarded_puts_exactly_one_wins( + self, secrets_url, tenant_a_headers, unique_ref, cleanup + ): + """N concurrent If-Match PUTs sharing one validator: exactly one commits, + the rest are 409, and the version advances by exactly one — no lost + update, no double-apply. This is the CAS the file name promises but the + sequential tests never actually race.""" + ref = cleanup(tenant_a_headers, unique_ref("e2e-race")) + n = 8 + + async with httpx.AsyncClient(timeout=10.0) as client: + await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": "v0", "sharing": "tenant"}, + ) + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + etag = resp.headers["etag"] + gen, base_version = parse_etag(etag) + + async def guarded_put(i: int) -> int: + r = await client.put( + f"{secrets_url}/{ref}", + headers={**tenant_a_headers, "If-Match": etag}, + json={"value": f"racer-{i}", "sharing": "tenant"}, + ) + return r.status_code + + statuses = await asyncio.gather(*(guarded_put(i) for i in range(n))) + + wins = [s for s in statuses if s == 204] + conflicts = [s for s in statuses if s == 409] + assert len(wins) == 1, f"exactly one guarded PUT must win, got {statuses}" + assert len(conflicts) == n - 1, f"losers must all be 409, got {statuses}" + + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + new_gen, new_version = parse_etag(resp.headers["etag"]) + assert new_gen == gen, "in-place overwrite stays in the same generation" + assert new_version == base_version + 1, ( + "version advances by exactly one commit, not once per racer" + ) + assert resp.json()["value"].startswith("racer-") + + @pytest.mark.asyncio + async def test_racing_unconditional_puts_never_leak_and_heal_on_rewrite( + self, secrets_url, tenant_a_headers, unique_ref, cleanup + ): + """Concurrent unconditional PUTs are the crosswise dual-write case. The + fence may legitimately fail closed (404) on a poisoned interleave, so a + read afterwards is either the winner's value or 404 — never a value no + one wrote. A settling rewrite always heals the ref back to a known-good + value (self-heal on rewrite).""" + ref = cleanup(tenant_a_headers, unique_ref("e2e-race")) + n = 8 + written = {f"w-{i}" for i in range(n)} + + async with httpx.AsyncClient(timeout=10.0) as client: + + async def put(i: int) -> int: + r = await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": f"w-{i}", "sharing": "tenant"}, + ) + return r.status_code + + statuses = await asyncio.gather(*(put(i) for i in range(n))) + assert all(s in (201, 204) for s in statuses), statuses + + # By-design outcomes only: the winner's value (200) or fail-closed + # (404) on a crosswise interleave. Never a foreign/garbled value. + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + assert resp.status_code in (200, 404), resp.text + if resp.status_code == 200: + assert resp.json()["value"] in written, "final value must be one racer's" + + # A settling rewrite heals the ref regardless of the race outcome. + await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": "settled", "sharing": "tenant"}, + ) + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + assert resp.status_code == 200, resp.text + assert resp.json()["value"] == "settled" diff --git a/testing/e2e/gears/credstore/test_crud.py b/testing/e2e/gears/credstore/test_crud.py new file mode 100644 index 000000000..587838bd9 --- /dev/null +++ b/testing/e2e/gears/credstore/test_crud.py @@ -0,0 +1,204 @@ +"""CredStore E2E: CRUD semantics of the REST API. + +Covers the core lifecycle (PUT upsert / POST create-only / GET / DELETE), +response headers on the value-bearing GET, reference validation, the single +404 surface, and authentication. +""" +import httpx +import pytest + + +class TestCrudLifecycle: + """Create / read / update / delete through the gateway.""" + + @pytest.mark.smoke + @pytest.mark.asyncio + async def test_put_get_delete_roundtrip( + self, secrets_url, tenant_a_headers, unique_ref, cleanup + ): + """PUT creates, GET returns value + metadata, DELETE removes.""" + ref = cleanup(tenant_a_headers, unique_ref()) + + async with httpx.AsyncClient(timeout=10.0) as client: + resp = await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": "e2e-demo-value-1", "sharing": "tenant"}, + ) + assert resp.status_code == 204, resp.text + + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + assert resp.status_code == 200, resp.text + body = resp.json() + assert body["value"] == "e2e-demo-value-1" + meta = body["metadata"] + assert meta["sharing"] == "tenant" + assert meta["is_inherited"] is False + assert meta["version"] == 1 + + resp = await client.delete(f"{secrets_url}/{ref}", headers=tenant_a_headers) + assert resp.status_code == 204, resp.text + + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + assert resp.status_code == 404 + + @pytest.mark.asyncio + async def test_put_upsert_overwrites_value( + self, secrets_url, tenant_a_headers, unique_ref, cleanup + ): + """A second PUT replaces the value (idempotent upsert).""" + ref = cleanup(tenant_a_headers, unique_ref()) + + async with httpx.AsyncClient(timeout=10.0) as client: + for value in ("first", "second"): + resp = await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": value, "sharing": "tenant"}, + ) + assert resp.status_code == 204, resp.text + + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + assert resp.status_code == 200 + assert resp.json()["value"] == "second" + + @pytest.mark.asyncio + async def test_post_is_create_only_with_location_and_conflict( + self, secrets_url, tenant_a_headers, unique_ref, cleanup + ): + """POST returns 201 + Location; a duplicate POST returns 409.""" + ref = cleanup(tenant_a_headers, unique_ref()) + + async with httpx.AsyncClient(timeout=10.0) as client: + resp = await client.post( + secrets_url, + headers=tenant_a_headers, + json={"reference": ref, "value": "v1", "sharing": "tenant"}, + ) + assert resp.status_code == 201, resp.text + assert resp.headers.get("location", "").endswith(f"/secrets/{ref}") + + resp = await client.post( + secrets_url, + headers=tenant_a_headers, + json={"reference": ref, "value": "v2", "sharing": "tenant"}, + ) + assert resp.status_code == 409, resp.text + + # The conflicting POST must not have overwritten the value. + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + assert resp.json()["value"] == "v1" + + @pytest.mark.asyncio + async def test_reference_reusable_after_delete( + self, secrets_url, tenant_a_headers, unique_ref, cleanup + ): + """A completed delete releases the reference for re-creation.""" + ref = cleanup(tenant_a_headers, unique_ref()) + + async with httpx.AsyncClient(timeout=10.0) as client: + resp = await client.post( + secrets_url, + headers=tenant_a_headers, + json={"reference": ref, "value": "old", "sharing": "tenant"}, + ) + assert resp.status_code == 201, resp.text + resp = await client.delete(f"{secrets_url}/{ref}", headers=tenant_a_headers) + assert resp.status_code == 204, resp.text + + # Happy-path delete completes its saga fully: POST succeeds again. + resp = await client.post( + secrets_url, + headers=tenant_a_headers, + json={"reference": ref, "value": "new", "sharing": "tenant"}, + ) + assert resp.status_code == 201, resp.text + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + assert resp.json()["value"] == "new" + # A fresh secret starts a fresh version counter. + assert resp.json()["metadata"]["version"] == 1 + + @pytest.mark.asyncio + async def test_delete_is_not_idempotent_at_http_level( + self, secrets_url, tenant_a_headers, unique_ref, cleanup + ): + """The second DELETE of the same reference is a 404.""" + ref = cleanup(tenant_a_headers, unique_ref()) + + async with httpx.AsyncClient(timeout=10.0) as client: + await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": "v", "sharing": "tenant"}, + ) + resp = await client.delete(f"{secrets_url}/{ref}", headers=tenant_a_headers) + assert resp.status_code == 204 + resp = await client.delete(f"{secrets_url}/{ref}", headers=tenant_a_headers) + assert resp.status_code == 404 + + +class TestResponseHeaders: + """Confidentiality / concurrency headers on the value-bearing GET.""" + + @pytest.mark.asyncio + async def test_get_returns_etag_and_no_store( + self, secrets_url, tenant_a_headers, unique_ref, cleanup + ): + """GET carries a strong generation-bound ETag and Cache-Control: no-store.""" + import re + + ref = cleanup(tenant_a_headers, unique_ref()) + + async with httpx.AsyncClient(timeout=10.0) as client: + await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": "v", "sharing": "tenant"}, + ) + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + assert resp.status_code == 200 + # Strong validator: ".", version 1 on create. + assert re.fullmatch( + r'"[0-9a-f-]{36}\.1"', resp.headers.get("etag", "") + ), resp.headers.get("etag") + assert resp.headers.get("cache-control") == "no-store" + + +class TestValidationAndErrors: + """Reference validation, 404 surface, authentication.""" + + @pytest.mark.asyncio + async def test_invalid_reference_rejected(self, secrets_url, tenant_a_headers): + """A reference outside [a-zA-Z0-9_-]+ is a 400, not a 404.""" + async with httpx.AsyncClient(timeout=10.0) as client: + # POST validates the body-supplied reference. + resp = await client.post( + secrets_url, + headers=tenant_a_headers, + json={"reference": "has:colon", "value": "v", "sharing": "tenant"}, + ) + assert resp.status_code == 400, resp.text + + # Path-supplied reference on GET (percent-encoded colon). + resp = await client.get( + f"{secrets_url}/has%3Acolon", headers=tenant_a_headers + ) + assert resp.status_code == 400, resp.text + + @pytest.mark.asyncio + async def test_get_missing_secret_is_404( + self, secrets_url, tenant_a_headers, unique_ref + ): + """A never-created reference yields the canonical 404.""" + async with httpx.AsyncClient(timeout=10.0) as client: + resp = await client.get( + f"{secrets_url}/{unique_ref()}", headers=tenant_a_headers + ) + assert resp.status_code == 404 + + @pytest.mark.asyncio + async def test_missing_token_is_401(self, secrets_url, unique_ref): + """All routes are authenticated.""" + async with httpx.AsyncClient(timeout=10.0) as client: + resp = await client.get(f"{secrets_url}/{unique_ref()}") + assert resp.status_code == 401 diff --git a/testing/e2e/gears/credstore/test_sharing.py b/testing/e2e/gears/credstore/test_sharing.py new file mode 100644 index 000000000..05bfbee21 --- /dev/null +++ b/testing/e2e/gears/credstore/test_sharing.py @@ -0,0 +1,250 @@ +"""CredStore E2E: sharing modes and hierarchical resolution. + +Uses the static tenant tree: hierarchy-root (...0001) is the parent of +hierarchy-l1a (...0002) and hierarchy-l1b (...0005). Each token carries a +distinct subject, so a parent's ``private`` secret is never visible to the +child tenants' subjects. +""" +import httpx +import pytest + +from .conftest import HIERARCHY_ROOT + + +class TestHierarchicalResolution: + """shared secrets are inherited downward; tenant/private are not.""" + + @pytest.mark.smoke + @pytest.mark.asyncio + async def test_shared_secret_inherited_by_child( + self, secrets_url, root_headers, l1a_headers, unique_ref, cleanup + ): + """A parent's shared secret resolves from the child with metadata.""" + ref = cleanup(root_headers, unique_ref("e2e-shared")) + + async with httpx.AsyncClient(timeout=10.0) as client: + resp = await client.put( + f"{secrets_url}/{ref}", + headers=root_headers, + json={"value": "parent-shared-value", "sharing": "shared"}, + ) + assert resp.status_code == 204, resp.text + + resp = await client.get(f"{secrets_url}/{ref}", headers=l1a_headers) + assert resp.status_code == 200, resp.text + body = resp.json() + assert body["value"] == "parent-shared-value" + meta = body["metadata"] + assert meta["is_inherited"] is True + assert meta["owner_tenant_id"] == HIERARCHY_ROOT + assert meta["sharing"] == "shared" + + # The owner itself resolves it as its own (not inherited). + resp = await client.get(f"{secrets_url}/{ref}", headers=root_headers) + assert resp.json()["metadata"]["is_inherited"] is False + + @pytest.mark.asyncio + async def test_tenant_secret_not_inherited_by_child( + self, secrets_url, root_headers, l1a_headers, unique_ref, cleanup + ): + """tenant sharing stays within the owning tenant.""" + ref = cleanup(root_headers, unique_ref("e2e-tenant")) + + async with httpx.AsyncClient(timeout=10.0) as client: + resp = await client.put( + f"{secrets_url}/{ref}", + headers=root_headers, + json={"value": "parent-tenant-value", "sharing": "tenant"}, + ) + assert resp.status_code == 204, resp.text + + resp = await client.get(f"{secrets_url}/{ref}", headers=l1a_headers) + assert resp.status_code == 404, resp.text + + @pytest.mark.asyncio + async def test_private_secret_not_inherited_by_child( + self, secrets_url, root_headers, l1a_headers, unique_ref, cleanup + ): + """A parent's private secret is invisible to a child subject.""" + ref = cleanup(root_headers, unique_ref("e2e-priv")) + + async with httpx.AsyncClient(timeout=10.0) as client: + resp = await client.put( + f"{secrets_url}/{ref}", + headers=root_headers, + json={"value": "parent-private-value", "sharing": "private"}, + ) + assert resp.status_code == 204, resp.text + + resp = await client.get(f"{secrets_url}/{ref}", headers=l1a_headers) + assert resp.status_code == 404, resp.text + + # The owner still reads it. + resp = await client.get(f"{secrets_url}/{ref}", headers=root_headers) + assert resp.status_code == 200 + assert resp.json()["metadata"]["sharing"] == "private" + + @pytest.mark.asyncio + async def test_resolution_is_upward_only( + self, secrets_url, root_headers, l1a_headers, unique_ref, cleanup + ): + """A child's shared secret is not visible to its parent.""" + ref = cleanup(l1a_headers, unique_ref("e2e-up")) + + async with httpx.AsyncClient(timeout=10.0) as client: + resp = await client.put( + f"{secrets_url}/{ref}", + headers=l1a_headers, + json={"value": "child-shared-value", "sharing": "shared"}, + ) + assert resp.status_code == 204, resp.text + + resp = await client.get(f"{secrets_url}/{ref}", headers=root_headers) + assert resp.status_code == 404, resp.text + + @pytest.mark.asyncio + async def test_sibling_does_not_leak( + self, secrets_url, l1a_headers, l1b_headers, unique_ref, cleanup + ): + """A shared secret in one subtree is invisible to a sibling subtree.""" + ref = cleanup(l1a_headers, unique_ref("e2e-sib")) + + async with httpx.AsyncClient(timeout=10.0) as client: + resp = await client.put( + f"{secrets_url}/{ref}", + headers=l1a_headers, + json={"value": "l1a-shared-value", "sharing": "shared"}, + ) + assert resp.status_code == 204, resp.text + + resp = await client.get(f"{secrets_url}/{ref}", headers=l1b_headers) + assert resp.status_code == 404, resp.text + + +class TestShadowing: + """The closest accessible secret wins; deletion restores fallback.""" + + @pytest.mark.asyncio + async def test_child_shadows_parent_and_fallback_after_delete( + self, secrets_url, root_headers, l1a_headers, unique_ref, cleanup + ): + """Child's own secret wins over the parent's shared one.""" + ref = unique_ref("e2e-shadow") + cleanup(root_headers, ref) + cleanup(l1a_headers, ref) + + async with httpx.AsyncClient(timeout=10.0) as client: + resp = await client.put( + f"{secrets_url}/{ref}", + headers=root_headers, + json={"value": "parent-value", "sharing": "shared"}, + ) + assert resp.status_code == 204, resp.text + resp = await client.put( + f"{secrets_url}/{ref}", + headers=l1a_headers, + json={"value": "child-override", "sharing": "tenant"}, + ) + assert resp.status_code == 204, resp.text + + # The child resolves its own secret (shadowing). + resp = await client.get(f"{secrets_url}/{ref}", headers=l1a_headers) + body = resp.json() + assert body["value"] == "child-override" + assert body["metadata"]["is_inherited"] is False + + # Deleting the override restores fallback to the parent's secret. + resp = await client.delete(f"{secrets_url}/{ref}", headers=l1a_headers) + assert resp.status_code == 204, resp.text + resp = await client.get(f"{secrets_url}/{ref}", headers=l1a_headers) + body = resp.json() + assert body["value"] == "parent-value" + assert body["metadata"]["is_inherited"] is True + + @pytest.mark.asyncio + async def test_deleting_shared_secret_revokes_child_access( + self, secrets_url, root_headers, l1a_headers, unique_ref, cleanup + ): + """Descendants lose access immediately when the owner deletes.""" + ref = cleanup(root_headers, unique_ref("e2e-revoke")) + + async with httpx.AsyncClient(timeout=10.0) as client: + await client.put( + f"{secrets_url}/{ref}", + headers=root_headers, + json={"value": "to-be-revoked", "sharing": "shared"}, + ) + resp = await client.get(f"{secrets_url}/{ref}", headers=l1a_headers) + assert resp.status_code == 200 + + resp = await client.delete(f"{secrets_url}/{ref}", headers=root_headers) + assert resp.status_code == 204, resp.text + + resp = await client.get(f"{secrets_url}/{ref}", headers=l1a_headers) + assert resp.status_code == 404 + + +class TestSharingClasses: + """private and tenant/shared secrets coexist under one reference.""" + + @pytest.mark.asyncio + async def test_private_and_tenant_coexist_private_wins_for_owner( + self, secrets_url, tenant_a_headers, unique_ref, cleanup + ): + """One reference holds both classes; the owner resolves private.""" + ref = cleanup(tenant_a_headers, unique_ref("e2e-coex")) + + async with httpx.AsyncClient(timeout=10.0) as client: + resp = await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": "tenant-wide", "sharing": "tenant"}, + ) + assert resp.status_code == 204, resp.text + resp = await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": "owner-only", "sharing": "private"}, + ) + assert resp.status_code == 204, resp.text + + # Private beats non-private for its owner. + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + body = resp.json() + assert body["value"] == "owner-only" + assert body["metadata"]["sharing"] == "private" + + # DELETE addresses the private class first, unmasking the tenant one. + resp = await client.delete( + f"{secrets_url}/{ref}", headers=tenant_a_headers + ) + assert resp.status_code == 204, resp.text + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + body = resp.json() + assert body["value"] == "tenant-wide" + assert body["metadata"]["sharing"] == "tenant" + + @pytest.mark.asyncio + async def test_tenant_to_shared_transition_in_place( + self, secrets_url, tenant_a_headers, unique_ref, cleanup + ): + """tenant -> shared is an in-place update of the same secret.""" + ref = cleanup(tenant_a_headers, unique_ref("e2e-trans")) + + async with httpx.AsyncClient(timeout=10.0) as client: + await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": "v", "sharing": "tenant"}, + ) + resp = await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": "v", "sharing": "shared"}, + ) + assert resp.status_code == 204, resp.text + + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + meta = resp.json()["metadata"] + assert meta["sharing"] == "shared" + assert meta["version"] == 2, "in-place transition bumps the version" diff --git a/testing/e2e/gears/credstore/test_types.py b/testing/e2e/gears/credstore/test_types.py new file mode 100644 index 000000000..ad25a2199 --- /dev/null +++ b/testing/e2e/gears/credstore/test_types.py @@ -0,0 +1,292 @@ +"""CredStore E2E: GTS secret types and their enforceable traits. + +Types are chosen at creation via the optional ``type`` field — a **full GTS +type id** (default ``generic``; PR #4204 review C9: the catalog short name and +the raw type UUID are no longer accepted). Types are immutable and drive trait +enforcement: permitted sharing modes (``allow_sharing``), embedded value +schemas, and expiry. +""" +import httpx +import pytest + + +def _type_id(leaf: str) -> str: + """Full GTS type id for a built-in secret type (leaf uses underscores).""" + return f"gts.cf.core.credstore.secret.v1~cf.core.credstore.{leaf}.v1~" + + +GENERIC = _type_id("generic") +API_KEY = _type_id("api_key") +PERSONAL_TOKEN = _type_id("personal_token") +OAUTH2_CLIENT = _type_id("oauth2_client") +CONNECTION_STRING = _type_id("connection_string") +BEARER_TOKEN = _type_id("bearer_token") + + +class TestTypeSelection: + """Type defaults, metadata echo, and validation.""" + + @pytest.mark.smoke + @pytest.mark.asyncio + async def test_default_type_is_generic_and_echoed_in_metadata( + self, secrets_url, tenant_a_headers, unique_ref, cleanup + ): + """An untyped secret is generic; a typed one reports its type.""" + ref = cleanup(tenant_a_headers, unique_ref("e2e-type")) + typed_ref = cleanup(tenant_a_headers, unique_ref("e2e-type")) + + async with httpx.AsyncClient(timeout=10.0) as client: + resp = await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": "v", "sharing": "tenant"}, + ) + assert resp.status_code == 204, resp.text + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + assert resp.json()["metadata"]["type"] == GENERIC + + resp = await client.put( + f"{secrets_url}/{typed_ref}", + headers=tenant_a_headers, + json={"value": "sk-123", "sharing": "tenant", "type": API_KEY}, + ) + assert resp.status_code == 204, resp.text + resp = await client.get( + f"{secrets_url}/{typed_ref}", headers=tenant_a_headers + ) + assert resp.json()["metadata"]["type"] == API_KEY + + @pytest.mark.asyncio + async def test_unknown_type_rejected( + self, secrets_url, tenant_a_headers, unique_ref + ): + """A well-formed but unregistered GTS type id is a 400.""" + async with httpx.AsyncClient(timeout=10.0) as client: + resp = await client.post( + secrets_url, + headers=tenant_a_headers, + json={ + "reference": unique_ref("e2e-type"), + "value": "v", + "sharing": "tenant", + "type": _type_id("no_such_type"), + }, + ) + assert resp.status_code == 400, resp.text + assert "UNKNOWN_SECRET_TYPE" in resp.text + + @pytest.mark.asyncio + async def test_non_gts_type_spelling_rejected( + self, secrets_url, tenant_a_headers, unique_ref + ): + """A catalog short name / raw UUID is not a GTS type id → 400.""" + async with httpx.AsyncClient(timeout=10.0) as client: + for bad in ("api-key", "00000000-0000-0000-0000-000000000000"): + resp = await client.post( + secrets_url, + headers=tenant_a_headers, + json={ + "reference": unique_ref("e2e-type"), + "value": "v", + "sharing": "tenant", + "type": bad, + }, + ) + assert resp.status_code == 400, (bad, resp.text) + + @pytest.mark.asyncio + async def test_type_is_immutable( + self, secrets_url, tenant_a_headers, unique_ref, cleanup + ): + """Changing an existing secret's type is rejected; omitting keeps it.""" + ref = cleanup(tenant_a_headers, unique_ref("e2e-type")) + + async with httpx.AsyncClient(timeout=10.0) as client: + await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": "sk-1", "sharing": "tenant", "type": API_KEY}, + ) + resp = await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": "sk-2", "sharing": "tenant", "type": GENERIC}, + ) + assert resp.status_code == 400, resp.text + assert "TYPE_IMMUTABLE" in resp.text + + # Untyped overwrite inherits the stored type. + resp = await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": "sk-3", "sharing": "tenant"}, + ) + assert resp.status_code == 204, resp.text + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + assert resp.json()["metadata"]["type"] == API_KEY + assert resp.json()["value"] == "sk-3" + + +class TestAllowSharingTrait: + """The flagship trait: per-type sharing-mode restrictions.""" + + @pytest.mark.smoke + @pytest.mark.asyncio + async def test_personal_token_is_private_only( + self, secrets_url, tenant_a_headers, unique_ref, cleanup + ): + """personal-token can never be tenant-wide or shared.""" + ref = cleanup(tenant_a_headers, unique_ref("e2e-ptok")) + + async with httpx.AsyncClient(timeout=10.0) as client: + for sharing in ("tenant", "shared"): + resp = await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={ + "value": "tok", + "sharing": sharing, + "type": PERSONAL_TOKEN, + }, + ) + assert resp.status_code == 400, (sharing, resp.text) + assert "SHARING_NOT_ALLOWED_FOR_TYPE" in resp.text + + resp = await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={"value": "tok", "sharing": "private", "type": PERSONAL_TOKEN}, + ) + assert resp.status_code == 204, resp.text + + @pytest.mark.asyncio + async def test_connection_string_is_tenant_only( + self, secrets_url, tenant_a_headers, unique_ref + ): + """connection-string cannot be shared down the hierarchy.""" + async with httpx.AsyncClient(timeout=10.0) as client: + resp = await client.put( + f"{secrets_url}/{unique_ref('e2e-dsn')}", + headers=tenant_a_headers, + json={ + "value": "postgres://u:p@host/db", + "sharing": "shared", + "type": CONNECTION_STRING, + }, + ) + assert resp.status_code == 400, resp.text + assert "SHARING_NOT_ALLOWED_FOR_TYPE" in resp.text + + +class TestValueSchemaTrait: + """Structured types validate their value against an embedded schema.""" + + @pytest.mark.asyncio + async def test_oauth2_client_schema_enforced( + self, secrets_url, tenant_a_headers, unique_ref, cleanup + ): + """oauth2-client requires client_id + client_secret JSON.""" + ref = cleanup(tenant_a_headers, unique_ref("e2e-oauth2")) + + async with httpx.AsyncClient(timeout=10.0) as client: + # Valid payload passes. + resp = await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={ + "value": '{"client_id": "cid", "client_secret": "s3cr3t"}', + "sharing": "tenant", + "type": OAUTH2_CLIENT, + }, + ) + assert resp.status_code == 204, resp.text + + # Missing required field and non-JSON payloads are rejected — + # and the error must not echo the secret material. + for bad in ('{"client_id": "cid-only"}', "not-json-at-all"): + resp = await client.put( + f"{secrets_url}/{unique_ref('e2e-oauth2')}", + headers=tenant_a_headers, + json={"value": bad, "sharing": "tenant", "type": OAUTH2_CLIENT}, + ) + assert resp.status_code == 400, resp.text + assert "VALUE_SCHEMA_VIOLATION" in resp.text + assert "cid-only" not in resp.text, "must not echo the value" + + +class TestExpiryTrait: + """expirable types accept expires_at; expired secrets stop resolving.""" + + @pytest.mark.asyncio + async def test_expiry_rejected_for_non_expirable_type( + self, secrets_url, tenant_a_headers, unique_ref + ): + """generic (and api-key) secrets cannot carry expires_at.""" + async with httpx.AsyncClient(timeout=10.0) as client: + resp = await client.put( + f"{secrets_url}/{unique_ref('e2e-exp')}", + headers=tenant_a_headers, + json={ + "value": "v", + "sharing": "tenant", + "expires_at": "2099-01-01T00:00:00Z", + }, + ) + assert resp.status_code == 400, resp.text + assert "EXPIRY_NOT_SUPPORTED_FOR_TYPE" in resp.text + + @pytest.mark.asyncio + async def test_expirable_type_roundtrip_and_past_expiry_rejected( + self, secrets_url, tenant_a_headers, unique_ref, cleanup + ): + """bearer-token accepts a future expiry and echoes it; past is 400.""" + ref = cleanup(tenant_a_headers, unique_ref("e2e-exp")) + + async with httpx.AsyncClient(timeout=10.0) as client: + resp = await client.put( + f"{secrets_url}/{ref}", + headers=tenant_a_headers, + json={ + "value": "tok", + "sharing": "tenant", + "type": BEARER_TOKEN, + "expires_at": "2099-01-01T00:00:00Z", + }, + ) + assert resp.status_code == 204, resp.text + resp = await client.get(f"{secrets_url}/{ref}", headers=tenant_a_headers) + meta = resp.json()["metadata"] + assert meta["type"] == BEARER_TOKEN + assert meta["expires_at"].startswith("2099-01-01T00:00:00") + + resp = await client.put( + f"{secrets_url}/{unique_ref('e2e-exp')}", + headers=tenant_a_headers, + json={ + "value": "tok", + "sharing": "tenant", + "type": BEARER_TOKEN, + "expires_at": "2000-01-01T00:00:00Z", + }, + ) + assert resp.status_code == 400, resp.text + assert "EXPIRY_IN_THE_PAST" in resp.text + + @pytest.mark.asyncio + async def test_malformed_expires_at_is_400( + self, secrets_url, tenant_a_headers, unique_ref + ): + """expires_at must be RFC 3339.""" + async with httpx.AsyncClient(timeout=10.0) as client: + resp = await client.put( + f"{secrets_url}/{unique_ref('e2e-exp')}", + headers=tenant_a_headers, + json={ + "value": "tok", + "sharing": "tenant", + "type": BEARER_TOKEN, + "expires_at": "tomorrow", + }, + ) + assert resp.status_code == 400, resp.text + assert "INVALID_EXPIRES_AT" in resp.text diff --git a/testing/e2e/gears/mini_chat/config/base.yaml b/testing/e2e/gears/mini_chat/config/base.yaml index 75770816c..629f3c032 100644 --- a/testing/e2e/gears/mini_chat/config/base.yaml +++ b/testing/e2e/gears/mini_chat/config/base.yaml @@ -427,6 +427,15 @@ gears: config: enabled: true + credstore: + database: + server: "sqlite_users" + file: "credstore.db" + config: + # Gateway selects its value-store backend plugin by GTS vendor; + # must match static-credstore-plugin (default "constructorfabric"). + vendor: "constructorfabric" + static-credstore-plugin: config: secrets: diff --git a/testing/e2e/gears/mini_chat/conftest.py b/testing/e2e/gears/mini_chat/conftest.py index 946960513..89296b9e8 100644 --- a/testing/e2e/gears/mini_chat/conftest.py +++ b/testing/e2e/gears/mini_chat/conftest.py @@ -8,6 +8,7 @@ import re import sqlite3 import tempfile +import time from dataclasses import dataclass, field from pathlib import Path @@ -455,6 +456,139 @@ def server(test_env): _db_summary_text = _print_db_summary() +@pytest.fixture(scope="session", autouse=True) +def _provision_credstore_secrets(request, server): + """Provision the LLM provider secrets through the credstore gateway. + + mini-chat routes LLM calls through OAGW, whose auth plugin resolves the + provider ``secret_ref`` (``openai-key`` / ``azure-openai-key``) via the + now-stateful credstore gateway. The gateway resolves a secret's metadata + from its own DB before reading the value, so the secrets must be created + through its API — pre-seeding the static plugin config alone is no longer + reachable. + + The mini-chat rig is single-tenant (``single-tenant-tr-plugin``) with + ``accept_all`` authn, so a ``tenant``-scoped secret resolves for the S2S + context OAGW uses when proxying. Offline mode (default) uses mock values + (the mock provider ignores them); online mode uses the real env keys, the + same ones ``_patch_mini_chat_config`` would otherwise inject. + + PUT is an idempotent upsert, so reruns are safe. + """ + if request.config.getoption("mode") == "online": + secrets = { + "openai-key": os.environ.get("OPENAI_API_KEY", ""), + "azure-openai-key": os.environ.get("AZURE_OPENAI_API_KEY", ""), + } + else: + secrets = { + "openai-key": "mock-key-openai", + "azure-openai-key": "mock-key-azure", + } + headers = { + "Authorization": "Bearer mini-chat-e2e", + "content-type": "application/json", + } + # Iterate over a plain tuple of reference names (not `secrets.items()`) so + # nothing derived from the secret values flows into log/error messages, + # and never echo the response body — it could reflect the secret. + provisioned = 0 + reachable = True + for ref in ("openai-key", "azure-openai-key"): + value = secrets[ref] + if not value: + continue # online mode without that provider's key configured + try: + # The mini-chat rig serves all routes under the api-gateway + # `prefix_path: "/cf"` (same prefix as API_PREFIX above). + resp = httpx.put( + f"{server}/cf/credstore/v1/secrets/{ref}", + headers=headers, + json={"value": value, "sharing": "tenant"}, + timeout=5.0, + ) + except httpx.RequestError: + # Any transport-level error (connect/timeout/read-timeout) — keep + # provisioning non-fatal and fall through to the `yield`. `break`, + # not `return`: a bare return before the `yield` in this generator + # fixture raises "did not yield a value" and ERRORs the whole + # session (review finding #5). Broadened from `ConnectError` so a + # read-timeout against a still-warming server is caught too. + reachable = False + break + if resp.status_code not in (200, 201, 204): + msg = f"could not provision credstore secret {ref!r}: HTTP {resp.status_code}" + if request.config.getoption("mode") == "offline": + # Offline mode is deterministic — a provisioning failure is a real + # problem, so fail fast instead of letting tests fail opaquely. + raise RuntimeError(f"[e2e] {msg}") + print(f"[e2e] WARN: {msg}") + else: + provisioned += 1 + + # mini-chat registers its OAGW provider upstreams at boot, but the stateful + # credstore has no secret then, so registration is deferred and retried in + # the background once the secret exists (which is exactly what we just did). + # Wait for that reconcile to converge so provider-dependent tests don't race + # it and see a transient "Upstream not found". Skip the wait when the rig + # was unreachable — _await_oagw_upstreams would only time out. + if reachable: + _await_oagw_upstreams(server, headers, provisioned) + yield + + +def _await_oagw_upstreams(server, headers, expected, timeout=60.0): + """Best-effort wait until at least ``expected`` OAGW upstreams are live. + + The mini-chat gear retries deferred provider registration on a short cadence + after its secret is provisioned; poll ``GET /oagw/v1/upstreams`` (a JSON + array) until the provider upstreams appear, then let routes settle. This + only *waits* — it never fails the session; the tests' own assertions remain + the source of truth. + + ``expected`` is the number of secrets provisioned, used as a proxy for the + number of provider upstreams. That holds for this rig (each provider has one + distinct ``secret_ref`` and no ``tenant_overrides``); if the config gains + tenant overrides or shares a ``secret_ref`` across providers, revisit this + count (upstream count would no longer equal secret count). + + Robustness: a persistently non-observable endpoint (e.g. an authz change + returning non-200) is detected within a short probe window and we proceed + rather than spinning the full timeout; a non-JSON/invalid body is treated + as "not ready yet" rather than raising. + """ + if expected == 0: + return + start = time.monotonic() + deadline = start + timeout + probe_window = min(10.0, timeout) # time allowed to observe *any* 200 + saw_endpoint = False + while time.monotonic() < deadline: + count = None + try: + resp = httpx.get(f"{server}/cf/oagw/v1/upstreams", headers=headers, timeout=5.0) + if resp.status_code == 200: + saw_endpoint = True + body = resp.json() + if isinstance(body, list): + count = len(body) + except (httpx.RequestError, ValueError): + pass # transport error or non-JSON body — treat as "not ready yet" + + if count is not None and count >= expected: + # Upstreams are up; give per-provider route registration (which runs + # just after each create_upstream) a brief moment to catch up. + time.sleep(1.0) + return + # If the endpoint never became observable within the probe window, it is + # likely gated (authz/route); stop waiting and let the tests decide. + if not saw_endpoint and time.monotonic() - start >= probe_window: + print("[e2e] WARN: OAGW upstreams endpoint not observable; proceeding") + return + time.sleep(0.5) + print(f"[e2e] WARN: OAGW upstreams did not reach >= {expected} within {timeout:.0f}s") + + @pytest.fixture(scope="session") def mock_provider(test_env): """Access to the mock provider sidecar (for set_next_scenario).""" diff --git a/testing/e2e/gears/oagw/conftest.py b/testing/e2e/gears/oagw/conftest.py index 4d917606f..9cfb4592e 100644 --- a/testing/e2e/gears/oagw/conftest.py +++ b/testing/e2e/gears/oagw/conftest.py @@ -122,3 +122,70 @@ def _check_oagw_reachable(): except Exception: # Timeout or other transient error — still try to run tests. pass + + +# --------------------------------------------------------------------------- +# Provision the secrets the auth-injection tests resolve via `cred://` +# --------------------------------------------------------------------------- + +# Secrets the apikey / oauth2 auth plugins resolve via `cred://`. +# Values mirror the historical static-credstore-plugin seed in +# config/e2e-local.yaml. +_CREDSTORE_SECRETS = { + "openai-key": "sk-test-e2e-fake-key", + "test-oauth2-client-id": "test-client-id", + "test-oauth2-client-secret": "test-client-secret", +} + + +@pytest.fixture(scope="session", autouse=True) +def _provision_credstore_secrets(_check_oagw_reachable): + """Write the auth-injection secrets through the credstore gateway API. + + The credstore gateway is now stateful: a GET resolves the secret's + metadata from the gateway's own database first, then reads the value from + the backend plugin. A secret merely pre-seeded in the static plugin config + has no gateway metadata row and is therefore unreachable. So we create the + secrets via the gateway API (which writes the metadata row *and* the + backend value). + + We PUT (upsert -> idempotent) with the same token the proxied requests + carry (``E2E_AUTH_TOKEN`` -> tenant ``00000000-df51-...``). Sharing is + ``tenant`` so any subject in that tenant resolves the value — this matches + the proxied-request context regardless of subject_id (the historical seed + used owner-bound ``private``; the gateway lookup does not depend on the + OAGW auth-config ``sharing`` label). + """ + base_url = os.getenv("E2E_OAGW_BASE_URL", "http://localhost:8086") + token = os.getenv("E2E_AUTH_TOKEN", "e2e-token-tenant-a") + headers = { + "Authorization": f"Bearer {token}", + "content-type": "application/json", + } + # Iterate over a literal tuple of reference names (not the value mapping) + # so nothing derived from the secret values flows into log messages. + for ref in ("openai-key", "test-oauth2-client-id", "test-oauth2-client-secret"): + try: + resp = httpx.put( + f"{base_url}/credstore/v1/secrets/{ref}", + headers=headers, + json={"value": _CREDSTORE_SECRETS[ref], "sharing": "tenant"}, + timeout=5.0, + ) + except httpx.RequestError: + # Any transport-level error (connect/timeout/transport) — keep + # provisioning non-fatal; the per-test reachability check skips tests. + # `break`, not `return`: this is a generator fixture, so a bare + # return before the `yield` below raises "did not yield a value" + # and ERRORs the whole session instead of letting the per-test skip + # take over (review finding #5). + break + if resp.status_code not in (200, 201, 204): + # Don't fail the session; the auth tests skip if the secret is + # absent. Log only the ref name and status — never the response + # body, which could reflect the secret value. + print( + f"[e2e] WARN: could not provision credstore secret {ref!r}: " + f"HTTP {resp.status_code}" + ) + yield