feat(prisma): object storage (resources + bindings), locked Postgres state backend, Deployment.redeployOn - #1061
Conversation
…, Postgres state backend
- Prisma.Bucket / Prisma.BucketKey covering the previously deferred
/v1/buckets Management API routes. BucketKey secrets are reveal-once:
persisted state is authoritative, keys are created under a
deterministic instanceId-derived physical name and orphans from lost
create responses are revoked before recreating.
- providers({ dev }) lets an embedder supply its own local provider
layer for alchemy dev; liveProviders() exports the live layer for
frameworks composing their own selection. Defaults unchanged.
- State/PostgresState: postgresState({ client | dsn }) state backend
with a per-(stack,stage) session advisory lock, cross-connection
lease verification, and a race-safe schema migration. Deep-import
only, keeping pg off the State barrel for worker bundles.
- Resource.ts: type Aliases as readonly string[] | undefined so
ResourceClass assigns to ResourceClassLike under
exactOptionalPropertyTypes.
Signed-off-by: willbot <w.a.madden+machine@gmail.com>
|
Can you share sample of DX in PR description |
|
@sam-goodwin I rewrote the PR description - does the example there suffice? |
| /** | ||
| * Replacement provider layer used in `alchemy dev` mode instead of the | ||
| * built-in dev stubs. This lets an embedding framework supply its own | ||
| * emulator implementations (e.g. a local Postgres or object-storage | ||
| * emulator) while keeping the live Management API providers untouched. | ||
| * The layer must register a provider for every Prisma resource. Live | ||
| * (non-dev) deployments are unaffected. | ||
| */ | ||
| dev?: PrismaLocalProviders; |
There was a problem hiding this comment.
This isn't something we typically do. Take a look at Provider.dual that we set up in #963
The user shouldn't provide them. If they want to override them, they can construct a new layer.
There was a problem hiding this comment.
Done in 5c2a64b.
All resources now register via ProviderLayer.dual with their local stubs moved into the module. Shared helpers are in Internal/DevStub.ts.
Removed providers({dev}), PrismaLocalProviders, and liveProviders(). providers() is back to just the collection.
Responding to review: replace the providers({ dev }) override option with
the ProviderLayer.dual registration introduced in alchemy-run#963, matching the
Cloudflare providers.
Each Prisma resource module now exports its Provider() factory as a dual
layer: live = the Management API implementation, local = the dev stub
(Prisma.Database keeps its @prisma/dev-backed local implementation, Compute
keeps ComputeDevProvider as its local variant). The engine resolves the
variant per run and per resource, so alchemy dev picks local automatically
and Alchemy.remote() opts individual resources back into live.
- remove providers({ dev }), PrismaLocalProviders, and liveProviders();
embedders replacing implementations construct their own layer from the
exported per-resource Provider() factories, the Providers collection, and
managementApi()
- move the dev stub helpers to Prisma/Internal/DevStub.ts and each stub
into its resource module as the dual's local thunk
- providers() now always carries the (lazy) stack management-api layer; it
registers auth without resolving credentials, so dev needs no token
- tests: provide AlchemyContext where provider layers are built directly,
assert dual mode metadata, drop the dev-override test
Signed-off-by: willbot <w.a.madden+machine@gmail.com>
Signed-off-by: Will Madden <madden@prisma.io>
…lared inputs change Prisma snapshots environment variables into a deployment at create, so an env-value-only change updates the platform's variable record but never reaches the running app. Prisma.Compute solves this by folding env values into its own deployment fingerprint; the low-level Deployment resource owns no env, so it now takes the values to track as an explicit prop. redeployOn is hashed with a domain-separating salt (Redacted members unwrapped first) and persisted as a Redacted redeployHash attribute, a sibling of artifactHash so existing rows keep their artifact semantics. The diff plans a replacement when the recorded fingerprint differs, and runs before the resolved-props early return so a deferred diff cannot silently downgrade the change to an update. Only diff can plan a replacement, so a redeployOn that is still unresolved at plan time counts as changed, but only once a fingerprint has been recorded — adding the prop to an existing deployment records it through a plain update. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
|
Hey @sam-goodwin, I added one more fix for an issue already present: changing an environment variable's value never reached a running deployment. Prisma Cloud snapshots env vars into the deployment at create, and I fixed this by adding an optional The resolved values are salted-hashed and stored |
| Effect.gen(function* () { | ||
| const conn = yield* attempt(() => client.connect()); | ||
| const releaseConn = Effect.sync(() => { | ||
| try { | ||
| conn.release(); | ||
| } catch { | ||
| // The pool may already be closed; nothing left to release. | ||
| } | ||
| }); |
There was a problem hiding this comment.
Please use effect SQL instead of the async api
| class AmbiguousPrismaBucketKeyError extends Error { | ||
| readonly _tag = "AmbiguousPrismaBucketKeyError"; |
| * }); | ||
| * ``` | ||
| */ | ||
| export const BucketKey = Resource<BucketKey>("Prisma.BucketKey"); |
There was a problem hiding this comment.
Interesting. We'll also need to add bindings like we have in src/AWS/S3
Each API gets a binding and will automatically create the BucketKey resource and bind that to the environment.
For reference:
- look in src/AWS/S3 and see how each operation (e.g. GetObject.ts) has a Binding.Service and a corresponding layer
GetObjectHttp.ts. - look in src/Cloudflare/R2/ReadBucketHttp.ts - see how it creates an Account Api Token and binds granular policies to it.
Right now, your PR only considers the non-effectful DX. Every resource needs effectful bindings for interacting with it.
There was a problem hiding this comment.
I've added the bindings: ReadBucket / WriteBucket / ReadWriteBucket, following the R2 shape — binding a bucket auto-creates the BucketKey (read-scoped for reads; the API has no write-only role, so write bindings carry read_write), env round-trip follows Connect's naming contract, and the runtime client reuses the S3 machinery with an endpoint override. Get/put presign included.
Overall ~1.8k lines including tests. I debated extracting object storage to a separate PR — the bindings are the latest two commits, but I have a clean split prepared if you'd prefer it: #1061 would then deliver the state backend, the Aliases fix, dual registration and redeployOn, with buckets + bindings as the follow-up.
Also merged main — the pg optional-peer change (#1069) is honored: PostgresState now lazy-loads the driver with the same install-hint the other drivers use.
Transport-independent object types (BucketObject, BucketObjectBody, the get/put/list option shapes) plus the runtime plumbing the bucket bindings build on: a distilled S3 client pointed at a bucket key's endpoint via the Endpoint service, signed with that key's own credentials, and SigV4 query-string presigning against the same endpoint. The custom endpoint makes the client address buckets path-style, which is what Prisma Object Store serves. No new dependencies: this reuses the distilled AWS S3 client the AWS layers already call. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…uckets Three capability-level bindings — Prisma.ReadBucket, Prisma.WriteBucket, Prisma.ReadWriteBucket — matching R2's access-level split rather than one binding per S3 operation. Binding a bucket creates the Prisma.BucketKey for it, so the caller never handles a credential. The key's logical ID is derived from the bucket and the access level, which keeps it stable across reconciles and lets the deployed bundle derive the same identity with no host resolved. The key's endpoint, provider-side bucket name, access key ID, and secret access key are carried into the host under names derived the way connectEnvKeys derives Connection's: env vars for Prisma Compute and AWS Lambda, text bindings for Cloudflare Workers, and a die naming the three for any other host. The provisioning half stays separable from the runtime half: an embedder can call makeBucketBinding with its own makeClient and reuse the key provisioning and env naming without Alchemy's Effect client. Prisma bucket keys have only read and read_write roles, so a Read binding is genuinely scoped but a Write binding still carries a read_write credential. The Write/ReadWrite split is enforced client-side for now and documented as such on every binding. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…ser-needs # Conflicts: # packages/alchemy/src/State/index.ts
Replaces the hand-rolled `pg` Pool/Client surface with an `@effect/sql` SqlClient: tagged-template statements, `withTransaction` for the schema migration, and `SqlClient.reserve` for the connection that owns the session-scoped advisory lock. Lock statements are pinned to the reserved connection through the client's own transaction service, the way `withTransaction` pins them, so they keep the usual spans and row handling. Store semantics are unchanged — one lock per (stack, stage), TTL-amortized lease checks, and stage-less deleteStack still locking every stage first — with two hardenings the pooled model needs: - The lease check reports the backend it ran on. A single-connection client routes the check back to the lock holder, which cannot vouch for itself; that is now refused instead of silently trusted, and the `client` option documents that it must be pool-backed. - Taking the lock and registering its unlock finalizer is uninterruptible, and the reserved connection lives in a scope forked from the store's. An interrupt in between used to hand a connection holding a session lock back to the pool, stranding the lock until the process exited. Internal queries run through `withoutTransforms()`, so a client configured with column-name transforms cannot rename the columns this store reads. `@effect/sql-pg` brings its own `pg` driver, so the lazy `pg` import shim is gone; the package itself stays an optional peer, loaded lazily so a caller-supplied client works without it installed. The hermetic tests now stub the `@effect/sql` layer: a real SqlClient over the real `@effect/sql-pg` compiler with in-memory connections that answer `pg_backend_pid()` with their own id, so the pooled and single-connection cases are both covered. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
Renames `dsn` to `url` and gives it the same shape `SQL/Postgres.ts` uses:
a `Redacted<string>`, or an Effect yielding one, generic in that Effect's
error and requirements. `Config.redacted("STATE_DATABASE_URL")` is itself
an Effect, so it can be passed straight through and handed to
`@effect/sql-pg` untouched. The plain-string arm is gone deliberately —
that is the arm that invites `process.env.DATABASE_URL!` back.
Every JSDoc example reads its configuration through effect/Config instead
of process.env.
Signed-off-by: willbot <w.a.madden+machine@gmail.com>
Signed-off-by: Will Madden <madden@prisma.io>
…Error The key-recovery path raised a bare `Error` subclass with a hand-written `_tag`. It now uses `Data.TaggedError` like every other Prisma error, carries the bucket, name, and match count as fields, and is exported the way `BucketError` and `PrismaApiError` are. Adds the missing test for the ambiguous-name failure. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
There was a problem hiding this comment.
Don't stub. Deploy a real stack (in fixtures/) that deploys an actual app and then have the tests execute request with HttpClient
There was a problem hiding this comment.
see how we use beforeAll and afterAll in other tests
Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…t suite Deploy a Cloudflare Worker alongside the three Compute apps so the text-binding branch of makeBucketBinding is driven end to end, restore presignGet/presignPut coverage through the deployed hosts, and pass contentType/metadata and list delimiter/limit/cursor through the routes. The read-write Compute app binds the bucket twice so a second bucket key under the same logical id would fail the deploy, which is how key reuse is now covered. Accepted loss: the old stub test asserting that an unsupported host dies has no end-to-end equivalent, because covering it would mean deploying a host the binding refuses to support. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…ads directly Drop the Cloudflare Worker fixture and Cloudflare.state() so the suite needs no credentials for any cloud but Prisma; state is local, like the in-memory scratch state the Compute live smoke uses. The Worker host branch of the binding goes back to having no live coverage — offered to the reviewer as a follow-up instead of shipped unasked. Reads assert directly instead of retrying until the value matches: the bucket is backed by Tigris, which is strongly consistent, so a retry-until-match read only masks failures. The one retained retry (untilOk) rides out the app endpoint's deploy propagation. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…ime read Hosts deliver env values as plain strings — Prisma Compute env rows are written with Redacted.value and Cloudflare secret_text carries the unwrapped text — so the secret reaches the runtime as a string, and the distilled signer's Redacted.value call on it dies with 'Unable to get redacted value' (a 500 on every bucket operation in a deployed app). Rebuild the wrapper at the read, the same shape as Connect's runtime accessors. Found by the live binding suite; the deleted stub suite's runtimeDouble stored outputs without ever crossing the env-string boundary, so it could not catch this. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
Bucket creation 500s on the Management API when the owning project's name is 51 or more characters (bisected live, 2026-08-05; the engine's generated physical name is 64). Until the platform validates the derived provider bucket name, an explicit short name keeps the suite deployable. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
…re comment Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
|
you were right about the stubs. When I rewrote the tests against the real endpoints I found some errors that would only have surfaced when users ran it in production I also realized I haven't been following the instructions in the repo or established conventions. I educated myself, gave my agent a kick in the pants and corrected my implementation and tests. there should be no novel decisions in my PR any more BucketBinding.test.ts now follows the R2 Binding.test.ts shape: a fixtures/ stack deploying three Compute apps binding one shared bucket, beforeAll(deploy) / afterAll(destroy), every client method incl. presigned URLs driven over HttpClient. live runs are opt-in like Compute.live.test.ts; a cold create/exercise/destroy cycle runs green against real Prisma Cloud and the description now shows the two-file export default shape from Compute's JSDoc |
…ggedError Same treatment the review asked for on the bucket-key failure: the two remaining bare Errors in Bucket's typed failure channel (convergence and delete refusing a bucket that belongs to another project) become one tagged error carrying the bucket and both project ids. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io>
This PR completes three gaps in the Prisma provider that show up the moment a real framework embeds it. After it, this works:
The three ship together because they're the set a production embedder needs at once — we (Prisma Composer, the framework layer over Prisma Cloud) are deleting our own resource implementations in favor of this provider, and these were the things we couldn't do with it. Say the word and we'll split any of them out.
1. Object storage: resources + bindings
Covers the 7 previously deferred
/v1/bucketsroutes (deferredRoutesis now empty in the contract fixture; coverage 71 → 78), with both halves of the equation:Resources.
Bucket(read-then-create reconcile, replace on identity change, identity-verified delete, reallistfor nuke) andBucketKey. The key secret is reveal-once — the API never returns it after create, so persisted state is authoritative (theConnectionpattern). Keys are created under a deterministicinstanceId-derived physical name and looked up before create, so a crash between the create call and the state write cannot mint a second, unenumerable credential — the orphan from the lost response is found by name and revoked before a fresh key is created.Bindings.
ReadBucket/WriteBucket/ReadWriteBucket, following the R2 capability shape: binding a bucket auto-creates aBucketKeyfor that (bucket, capability) — read-scoped forReadBucket; the API has no write-only role, so write bindings carryread_write(documented) — and carries endpoint/bucket/credentials into the host environment onConnect's env-naming contract. Host dispatch matchesConnectBinding: env for Prisma Compute and Lambda, worker bindings for Cloudflare Workers. The runtime client reuses the distilled S3 machinery with an endpoint override (Prisma object storage is S3-compatible SigV4); get/head/put/delete/list plus presigned get/put. The credential-provisioning half is a separable function boundary (à lamakeHttpBucketBinding), so embedders can reuse provisioning + env naming without the Effect runtime client.Docs:
prisma/data/bucketspage + sidebar entry; the three bindings carry@bindingreference docs.2. Dual-registered dev mode
Per review:
providers()takes no options; every resource registers its live and dev implementations viaProviderLayer.dual(#963), with the dev stubs moved in-module (Internal/DevStub.ts). The engine picks by run mode,Alchemy.remote()works per-resource, and an embedder that wants different behavior constructs its own layer from the per-resource exports. Defaults unchanged.3. A locked state backend:
State/PostgresStateCompute's recovery docs tell users to "use a durable, locked state backend" — and no in-tree backend has locking. This adds one on the dependency the repo already carries (pg):(stack, stage)session advisory lock (pg_try_advisory_lock(hashtextextended(key, 0))) held on a reserved connection, with the holder re-verified againstpg_locksfrom a different pool connection — so a silently dropped lock connection is detected, not trusted;deleteStacklocks each stage before touching it;create table if not existsgenuinely fails on Postgres (duplicatepg_typeerrors — reproduced on PG 15).Deliberately not re-exported from the
Statebarrel (the barrel is bundled for workers;pgmust stay off that graph — a comment says so). Deep import:alchemy/State/PostgresState. Tests are hermetic stubs per the existing backend convention; the two real-Postgres behaviors verified live during development are documented in comments.4.
Deployment.redeployOnPrisma Cloud snapshots env vars into a deployment at create, and the low-level
Deploymentreuses the running deployment unless the artifact moves — so an env-value-only change (a rotated secret) updates the platform row but never reaches the running app.redeployOntakes any serializable inputs; their resolved values are salted-hashed into a newredeployHashattribute storedRedacted(no plaintext in state), and a changed hash plans a replace — the low-level counterpart of whatComputealready does with its env fingerprint. Old rows lack the field and never replace on upgrade.5. One-line core fix:
AliasestypingResourceClass.Aliasesisreadonly string[] | undefined, butResourceClassLike.Aliases?: readonly string[]— under a consumer tsconfig withexactOptionalPropertyTypes: true, everyProvider.effect(cls, …)call fails to typecheck (we carry a pnpm patch for this today). The fix widens the optional to| undefined.Verification
bun run format:checkclean;bun tsc -b(monorepo) clean.fixtures/holds three Compute apps binding one shared bucket (read / write / read-write), the tests drive every client operation overHttpClientthrough the deployed apps — including presigned PUT/GET fetched by the test with no credentials of its own — withbeforeAll(deploy)/afterAll(destroy). A cold full cycle (create → exercise → destroy, 8 resources) runs green against Prisma Cloud. Live runs are guarded byALCHEMY_RUN_LIVE_PRISMA_TESTS+ credentials, matchingCompute.live.test.ts; without credentials the suite skips cleanly and the pure derivation tests still run.Redactedwrapper does not survive host env delivery (env rows carry plain strings), so the signer failed on every bucket operation in a deployed app. Fixed by rebuilding the wrapper at the runtime read, the same shape asConnect's accessors.bun alchemy-test --fast test/Prisma test/State: 0 failed; core engine suites (exercising theResource.tschange) green.generate-api-reference: Prisma gains Bucket/BucketKey and the three binding pages; no new category;docs:checkbuilds clean.Alternatives considered
Connection's; the bindings provision through the resources).postgresStatefrom theStatebarrel. Poisons worker bundles withpg; deep import + comment instead.create table if not exists). Fails under concurrency on real Postgres; see above.