Key organizations by GUID consistently across Admin, Lobby and Core

[einari]

Organization identity is inconsistent across the services, and it is why only one organization shows a trial in the Admin listing.

What is wrong today

Admin holds 7 OrganizationTrialGranted events but organizationTrials has a single row. Six of them are keyed by organization GUID; one is keyed by tenant name. OrganizationTrial is keyed by tenant name, so only the last one materializes. TrialReportingReactor was changed to key by tenant name at some point, which silently orphaned everything granted before it.

Six organizations (Cratis, Elvion, Hageskal, HiveConsulting, Identigo, Vybraze) have no OrganizationTrialStarted at all - they predate the trial flow, and are exactly the ones carrying manual SeatsGranted events as a workaround.

The old events cannot be salvaged by replaying: a projection keyed by tenant name cannot recover the tenant from a GUID, and projections cannot join to organizations to look it up.

The model we want

The organization's GUID is its event source id everywhere, and never appears as a payload property. Keying on the GUID rather than the name is what would let an organization be renamed later without orphaning its history - which is precisely the failure above, in a different guise.

Changes

Core has to learn the organization's identity. It currently has no notion of the GUID at all: Admin mints it and publishes OrganizationOnboardingAccepted(OrganizationId, TenantName), but only Lobby consumes it. Core should consume it from the Admin inbox and record the organization in its tenant namespace, so it can key on the GUID afterwards.

Contract events drop the tenant from their payload, carrying it as the event source id instead:

• Core.Contracts TrialGrantedForOrganization(Tenant, Sku, Expiration) -> (Sku, Expiration)
• Admin.Contracts SeatsGrantedToTenant(Tenant, Sku, Quantity, Expiration) -> (Sku, Quantity, Expiration)

Organization-level facts key on the organization, not the user. OrganizationTrialStarted is appended against the user id today; it is an organization fact and belongs on the organization's stream.

Admin's reactors and read models key on OrganizationId. TrialReportingReactor then needs no lookup at all - the event source is already the organization.

Existing events get migrated. Few enough to rewrite in place, and we are in beta.

Integration specs

End to end, one flow: invite from Admin -> acceptance in Lobby -> user registered and seat assigned in Core for the target tenant -> organization registered in Admin with the correct seat assignment. The existing onboarding spec covers part of this path already and is the natural place to extend.

Note

Cratis stays at 3 seats - a SingleUser trial grants one, and its manual grant is what covers the other two.

[einari]

The code side is done and on main (c07d1ada). All gates green: 2614 Core specs, 186 Admin specs, both frontends typecheck, lint and build clean.

What landed

• Admin tells Core who the organization is, once, when it registers it: OrganizationRegisteredForTenant, keyed by the identity rather than carrying it as a value, published from both onboarding paths. Core records it in the organization's own namespace and names the organization by it from then on.
• OrganizationId moved to Admin.Contracts and became an EventSourceId<Guid> rather than a plain concept - Admin mints it but all three services speak about the same organization, and it is what events are keyed by rather than a value they carry.
• The trial round trip is identity-keyed end to end: TrialGrantedForOrganization dropped Tenant from its payload, Core reports against the organization, Admin's reactor uses the event source, and the read model, query, seat join and frontend caller all key on it.
• SeatsGrantedToTenant is now keyed by the organization too; the tenant name stays only because it names the namespace Core writes into.
• The onboarding flow spec asserts the seat count on the organization in Admin. It already covered the invite through to Core assigning the trial seat, but stopped before the half that was broken - and the organization row is there either way, so only the count says the report arrived and was matched.

Two things worth recording because they were nearly wrong:

• TrialOutbox first took the Organization read model as a handler parameter. Chronicle materializes a reactor's read models from the triggering event's source id, which for a trial is the user the seat was assigned to - so it would have resolved nothing, silently. It reads through a lookup service instead, since reactors must not touch storage directly (CHR0032).
• The spec documenting the listing outage described trials "keyed by organization id, which left no organization on them". That was the same mismatch from the other side. Both sides now key on identity, so that particular inversion cannot recur.

Not done: the production data

None of this is live. Publish only triggers on workflow_dispatch or a labelled PR merge, so commits pushed straight to main are never released - production is still on 0.65.7.

The data pass therefore has to wait for a release, and this one needs more than a normal deploy:

1. TrialGrantedForOrganization lost a property, so the event-type registry likely needs the same treatment as the last schema change - stop Chronicle, drop the type's doc from Studio+es.event-types, start.
2. OrganizationTrial's key changed from name to identity, so organizationTrials needs reprojecting rather than accumulating.
3. Core wants scale-to-zero on release, or two Cores race on schema registration.

Then: a trial for every organization starting today (Cratis stays at 3 seats, since a SingleUser trial grants one and its manual grant covers the other two), and the stale announcement row 46f81105 cleared from every tenant.

[einari]

Current state — supersedes the "not live" note in my previous comment

The code is deployed. Production runs 0.66.1 (registry.cratis.studio/core:0.66.1, admin:0.66.1), every pod healthy, Core 2/2, Admin 1/1, site serving. Everything in the previous comment under "What landed" is live. Only the data is outstanding.

The event-type registry is already handled — do not redo it

TrialGrantedForOrganization dropped a property, so Chronicle refused the new schema with:

EventTypeSchemaChanged: Event type 'TrialGrantedForOrganization' at generation 1 has a
schema that differs from the already registered schema. Schema changes are not allowed
without creating a new generation.

Cleared from both Studio+es and StudioAdmin+es (db["event-types"].deleteOne({_id:"TrialGrantedForOrganization"})), Chronicle restarted, then Admin and Core restarted. Admin was crash-looping on this and is now healthy.

The ordering lesson, because it cost a crash loop: clear the registry after the new code is deployed, not before. Clearing it first, then having old-version pods start for any reason, re-registers the old schema and the new code then conflicts with it. That is exactly what happened here - a failed first release left old pods running, which re-registered the old shape over the cleared entry.

What the data looks like right now

StudioAdmin.organizationTrials holds one row, _id: "Powerworks" - the old tenant-name key. Nothing has been migrated.

Admin's StudioAdmin+es+Default event log holds 7 OrganizationTrialGranted:

eventSourceId	organization	keyed by
23c5a8dd-…	OrgWithGoogle	identity
894dd3ae-…	SOmethingCool	identity
f33600fb-…	Acme	identity
168a325c-…	Ollie	identity
694241e8-…	nebulit	identity
a11e3ead-…	JaasJazz	identity
Powerworks	Powerworks	tenant name

With the new identity-keyed read model, the six identity-keyed events become correct and the Powerworks one becomes the odd one out - the inverse of before. Its event source needs rewriting to Powerworks' organization id, 0b58801e-32a8-41b0-b734-16f2832dc5b7.

Six organizations have no OrganizationTrialStarted in their tenant log at all - Cratis, Elvion, Hageskal, HiveConsulting, Identigo, Vybraze - and are exactly the six carrying manual SeatsGranted events as the workaround.

Per-tenant counts (Studio+{tenant}): every organization is 1 user / 1 seat except Cratis, which is 3 users / 3 seats.

Remaining work

1. Reproject organizationTrials. The read model changed key (OrganizationName -> OrganizationId), so existing rows are the wrong shape and it needs rebuilding rather than accumulating. I did not establish the mechanism - resetting the observer's position versus dropping the collection - and that is the first thing to settle.
2. Rewrite the Powerworks event's source from Powerworks to 0b58801e-32a8-41b0-b734-16f2832dc5b7.
3. Trial for every organization, starting today. There is no supported path: a trial is only appended during onboarding (AcceptingInvitation / AcceptingRegistration), so this is either raw event insertion or a small new command. Cratis stays at 3 seats - a SingleUser trial grants one, and its manual grant covers the other two, so do not remove that grant.
4. Clear the stale announcement. 46f81105-8a4b-414f-a427-36b13801c0ab sits in every tenant's announcements despite being withdrawn in Admin, left by the ordering bug fixed in Announcements not quite working #151. Not visible today (its activeUntil was 2026-08-13, so the client filters it) but wrong data.

Access

• Mongo primary is studio-mongo-rs0-1 (check with db.hello().isWritablePrimary; it moves).
• Credentials: kubectl get secret studio-mongo-secrets -n studio-production -o jsonpath='{.data.MONGODB_DATABASE_ADMIN_PASSWORD}' | base64 -d, user databaseAdmin, authenticationDatabase admin. The clusterAdmin user can list databases but cannot read collections.
• Databases: Studio+es+{tenant} (tenant event log), Studio+{tenant} (tenant read models), StudioAdmin+es+Default (Admin event log), StudioAdmin (Admin read models).
• Event documents key the type on a lowercase type field holding the type name directly - not Type.Id.

Event-log edits are sanctioned for this: few events, beta, and backups exist.

[einari]

The data pass is done — and it flushed out two silent code bugs

Picking up from the session that was stopped mid-flight, with production unstable: Core was crash-looping between versions, Chronicle would not start, and the Admin organization listing showed nothing at all.

Stabilization

• Core was stuck between versions. The serving ReplicaSet had been rolled back to 0.65.7; those pods re-registered the old TrialGrantedForOrganization schema whenever Chronicle restarted, so the 0.66.1 pod the deployment wanted crash-looped on EventTypeSchemaChanged — the ordering trap from the previous comment, now in a loop. Resolved with the documented sequence, this time with Core at scale-to-zero throughout: stop Core → stop Chronicle → delete the stale registry doc from Studio+es → start Chronicle → start Core on 0.66.1. The registry now holds the new shape (sku, expiration) and both Core pods are clean.
• Chronicle itself was crash-looping on MongoWaitQueueFullException. Its startup activates an event-sequence grain per namespace across every event store, and ensuring their indexes concurrently exhausts the driver's default 100-connection pool — the server was nowhere near its own limits; the queue that fills is the client's. Fixed with maxPoolSize=500, patched onto the live statefulset and codified in ChronicleDeployment.cs.

Why Admin showed no organizations

The leftover organizationTrials row keyed "Powerworks" could not deserialize into the now identity-keyed OrganizationTrial (OrganizationId is Guid-backed), so the listing's Enrich threw on every emission and the observable completed with zero events — 200 OK, Content-Length: 0. The stale data from item 1 was actively blanking the UI, not merely sitting there.

The data pass (items 1–4)

1. organizationTrials reprojected via Chronicle's API: POST /api/event-store/StudioAdmin/observers/Default/replay/Admin.Organizations.Trials.OrganizationTrial. Replay rebuilt the container from scratch. That settles the open mechanism question — observer replay through the API, not dropping collections.
2. Powerworks' event source rewritten to 0b58801e-…, and the seven old-contract TrialGrantedForOrganization copies still sitting in inbox-Studio rewritten from their random source ids to organization ids, so a future reactor replay cannot mint wrongly-keyed trials from them.
3. Every organization has a trial. All appends went through Chronicle's REST API, so sequencing, encryption and observers were kernel-handled throughout:
   • OrganizationRegisteredForTenant for all 14 organizations — none had ever been told its identity. The full cascade ran in production: outbox → inbox → OrganizationIdentityInbox → OrganizationEstablished and an identity read model in every tenant namespace.
   • OrganizationTrialStarted (SingleUser, expiring 2026-09-13) for the six that had none: Cratis, Elvion, Hageskal, HiveConsulting, Identigo, Vybraze. Cratis keeps its manual grant — 3 seats in use, untouched.
   • Claessens turned out to be a seventh casualty: its trial (2026-08-05) reached the Default outbox as event 25, but the Studio->StudioAdmin subscription was established later and skipped it permanently. Repaired with a direct outbox append carrying its true expiration.
   • organizationTrials now holds 14 identity-keyed rows, and the listing shows live activity and seats for every organization.
4. The stale announcement 46f81105-… was already gone. Every tenant log carries a later publish/amend/withdraw triple and no tenant holds the row. Verified, nothing to do.

Bug 1: every trial report was being dropped, silently

The backfill exposed it. The six OrganizationTrialStarted appends were handled by TrialOutbox — observer advanced, no failed partitions — and nothing reached the outbox.

Two causes stacked:

• OrganizationIdentity read IMongoCollection<Organization> from constructor injection, and a reactor is constructed tenant-less, so the collection was bound to the Default namespace where there is no organization to find.
• Deeper: onboarding produces the trial and the identity by different routes — the trial reaches Core straight from the Lobby, the identity travels Lobby → Admin → Core — so the trial nearly always arrives first. Reporting only when the trial arrives means reporting nothing, and [OnceOnly] then consumes the event for good.

Since c07d1ada this had made trial reporting inoperative for every newly onboarded organization — which is exactly why all six had to be hand-carried above. Fixed both ways: the identity is resolved through the event store the triggering event's context names, and whichever of the two facts arrives last now does the reporting.

Bug 2: sign-up addresses rendered as ciphertext

Organization.SignupEmail showed Q0VOVg… for every organization. The keys are all present in Vault and the kernel decrypts correctly — verified by calling Compliance.Release against the production kernel by hand.

The fault was on the read path: the listing read the sign-ups straight out of their MongoDB collection and asked for a release afterwards. That release resolves one subject for the whole row — the row's own key — but an address is sealed under the identity the person signed in with (Lobby SignedInIdentity), which is not the same value. Nothing resolved, so the release never ran and the ciphertext passed straight through. Ciphertext is a value rather than an error, so nothing reported it. Carrying the subject on the read model does not help either: a [Subject]-marked property is never materialized by the projection engine.

Fixed by reading the sign-ups through IReadModels — the kernel releases a materialized read model on the way out, which is where the owning subject is known. Verified against live production data on the deployed version: all 14 addresses come back readable, zero ciphertext.

Reported upstream as Cratis/Chronicle#3708, and already fixed there in 16.32.0: Chronicle now infers per-property PII ownership from projection provenance, stores it alongside the row, and releases each property under its owner — so this shape is supported transparently and the declaration attributes are gone.

Also fixed

The onboarding integration spec asserted the seat count and the readable address but could never observe either: the harness never told Admin where Core is, so activity and seats were always absent. Now wired.

Shipped and verified

Cratis/Studio#1165 merged with all twelve checks green — including the onboarding integration specs, which now assert both halves and pass. Released as 0.66.2 and deployed.

One gotcha on the way out: the deploy restarts Chronicle (the pool-size change above), and Admin came up during that window and stuck in a connection-timeout loop — every observable query returned an empty stream, so Organizations and Invitations both rendered blank. Restarting Admin after Chronicle was healthy cleared it. Worth knowing for any release that touches Chronicle's statefulset: bring the app services up after Chronicle is ready, not alongside it.

Verified against production after the deploy:

• Organizations listing: 15 rows, 14 readable sign-up addresses, zero ciphertext, and 14 rows carrying their trial seat.
• Invitations listing: renders, no ciphertext.

Where this leaves things

• Production is stable and serving on 0.66.2; both listings work; all trials are identity-keyed.
• OrganizationTrialStarted is still appended against the user at onboarding rather than the organization. Nothing reads that key — TrialOutbox resolves the identity separately — but the code and this issue's model still disagree on that point if anyone wants to close it out.