Seats are not showing in Admin

[einari]

In Core we produce the event OrganizationTrialStarted which then sets up the organization with the trial seats for a specified period after onboarding a user to create an organization.

There doesn't seem to be anything like this on the Admin service, so a new organization ends up with no seats, which is not true. We need to reflect this correctly here as well.

This is what we see:

This also needs to be corrected in production. We need to see the same number of seats in Admin as the logic in Core sees.

Purchasing Seats

When a purchase is done for seats, the Admin service should be notified by a public event from Core that Admin consumes and then produce whatever correct internal Admin event. It also means that we need to also be notified when cancellations are done.

Billing

When we get billing events from Mollie into Core, we also need to produce events that goes out publicly from Core and the Admin consumes to update. Admin should reflect billing and be able to see the billing history of an organization in the organization view.

[einari]

Implemented in Cratis/Studio#1149, left as a draft because the deploy is not safe on its own — see the warning at the top of that PR.

The premise was wrong twice, and the real bug is worse

The trial pipeline already existed and works. OrganizationTrialStarted → outbox → TrialGrantedForOrganization → Admin's TrialReportingReactor. Production agrees: 6 of 13 organizations have a trial in Admin, and they are exactly the 6 created after that shipped.

But Admin has been silently losing data, and I could measure it. Read-only against the production Mongo primary:

• Admin's inbox-Lobby holds 6 onboarding LegalTermsAccepted events. Admin's event log holds 1 OrganizationAcceptedLegalTerms — and that one is the checkout acceptance for Cratis. All six onboarding acceptances were dropped.
• For OrgWithGoogle the acceptance arrived at 17:19:10.868; Admin appended OrganizationRegistered at 17:19:11.012. The report beat the registration by 144 ms.
• The trial survives only by luck. OrgWithGoogle: OrganizationRegistered 17:19:11.012, OrganizationTrialGranted 17:19:11.088 — a 76 ms margin, against a materialized read model that lags its own observer. Other organizations had 2–42 s.

The cause is structural rather than a missing null check: OrganizationLookup.ByTenant searches the eventually-consistent Organization read model, and a search that finds nothing cannot distinguish "not registered yet" from "not ours" — so it drops. Admin's identity is a minted Guid while the cross-service identity is a name, so joining them is a search, and searches are never strongly consistent.

Two further findings:

• Self-service registration never granted a trial at all. OrganizationRegistrationReactor provisioned the Maintainer role and no seat — so a self-registered owner had no seat, no license in Core, and nothing for Admin to show. Not yet hit in production (all 13 organizations came through invitations) but /register is live. Fixed.
• Claessens has a trial in Core (outbox seq 25, 2026-08-05) that never reached Admin's inbox — it predates the deployment that added TrialGrantedForOrganization to the subscription. A deployment-timing artifact, not a code bug.

The fix

Trials and legal acceptances are recorded against the tenant name — no lookup, no drop, arrival order irrelevant. Regression specs assert both "recorded even though no organization exists" and "consulted no read model". Purchases deliberately keep the organization-id path, since a purchase can only follow a long-established organization (3/3 resolve in production).

Purchases, cancellations and Mollie billing history are all included; see the PR.

Production correction — planned, not executed

1. Drop StudioAdmin.organizationTrials and StudioAdmin.organizationLegalAcceptances. ⚠️ Deploy-blocking: existing _id values are BSON UUIDs and the new models key on a string, so a query over the mixed collection fails to deserialize.
2. Replay the two read-model observers to rebuild them from Admin's event log.
3. Replay the three inbox reactors. Admin's inboxes still hold every source event, so the new code re-emits them correctly keyed — this recovers all six lost acceptances with no data entry.
4. For Claessens only, replay the cross-store forwarding observer Studio→StudioAdmin from 0. Safe under the new keying: every Admin-side record is last-write-wins per key, so re-forwarding the whole outbox is idempotent.
5. Verify 7 rows in each collection, keyed by name.

Worth knowing: this design leans on the organization name being unique and never reassigned while the organization lives. That holds today (UniqueOrganizationName, released only on deletion, no rename command). If a rename is ever added, these records need re-keying.

Unrelated but spotted: AnnouncementWithdrawalOutbox and AnnouncementPublishingOutbox are in runningState=4 in production and look stuck.

[einari]

The fix shipped in 0.64.0 (Cratis/Studio#1149) and is live. Everything from here on is recorded correctly. The historical recovery is partly done and partly blocked — details below, because the blocker is not something I can work around from Studio.

What production actually looked like

Two separate defects, not one:

1. The onboarding acceptances were never written. The Lobby inbox holds 6 LegalTermsAccepted, and the reactor counted all 6 as handled — but only 1 OrganizationAcceptedLegalTerms exists in Admin's log, and that one came from checkout, not onboarding. So all 6 onboarding acceptances were dropped by the lookup that lost the race. That is the bug this issue is about, and it is fixed.

2. The read models had never materialized at all. Separately from the above, organizationTrials and organizationLegalAcceptances did not exist as collections — zero documents ever written — even though their observers reported the events as handled and reported no failed partitions. Every other Admin read model matched its event count 1:1; only these two were empty. Trials were not affected by defect 1: all 6 TrialGrantedForOrganization reached the log correctly. They were invisible purely because of this second problem.

What is fixed now

Replaying the two projections materialized both read models: organizationTrials now holds 6 documents and organizationLegalAcceptances 1, where both were previously empty.

What is still outstanding

Those recovered documents are keyed by the organization id, because that is how the pre-fix reactors keyed the events they appended. The queries look them up by organization name, so the rows exist but the Admin panel still will not show them. Re-keying the history means re-appending those events through the corrected reactors — i.e. replaying the reactors.

Replaying a reactor is a no-op on this Chronicle version. The endpoint returns isSuccess: true, but no replay job is created, the observer's position never moves and nothing is re-handled — while the identical call against a projection works immediately. The first attempt also parked the reactor in Disconnected until the application was restarted. Filed upstream as Cratis/Chronicle#3701.

So the remaining work is: recover the 6 onboarding acceptances and re-key the 6 trials plus 1 checkout acceptance. Both need the reactors to reprocess their inboxes, which needs the Chronicle fix — or a deliberate one-off backfill that appends to the production event log, which I would not do without sign-off.

Keeping this open until the history is recovered.

[einari]

Correcting my previous comment. It said both read models had been materialized by replaying their projections. That was true when I wrote it and is not true now: an hour later those same rows took the Admin organization listing down, and I deleted them. organizationTrials is empty again. Nothing here is fixed, and the comment should not have been left standing as if it were.

What the replayed rows did

Replaying the projection materialized 6 trials keyed by organization id, which meant they carried no organization value at all. The listing's seat enrichment keys every trial with (string)trial.Organization, and OrganizationName's implicit conversion is name.Value — so each of those rows threw a NullReferenceException inside the enrichment every emission passes through. The page reported "No organizations registered" over fourteen organizations whose data was entirely intact.

So the replay did not just fail to help; the rows it produced were actively harmful, and the read model being empty is what is currently keeping the listing up. Cratis/Studio#1155 fixes the listing so an unkeyable row can never blank it again, which is worth doing regardless of this issue.

Exactly what is missing, measured

Six trials reached Admin from Core and were recorded against the organization id rather than the tenant name:

JaasJazz · nebulit · Ollie · Acme · SOmethingCool · OrgWithGoogle

Those are precisely the six organizations the listing currently shows with 0 seats allocated and 1 in use — the trial seat that should supply the allocation is the one that never materialized. Every other organization's seats come from Core's entitlements and show correctly (Cratis 12/3, Identigo 6/1, Hageskal 4/1, and so on), which is why this reads as "seats work" until you look at the six that have only a trial.

Separately, the 6 onboarding legal acceptances were never appended at all — that is the original defect, fixed going forward in #1149.

Why it is still not recovered

Both halves need the reactors to reprocess their inboxes, keyed the corrected way. Replaying a reactor is a no-op on the Chronicle version production runs — the call reports success, no job is created, the observer never moves (Cratis/Chronicle#3701). The source events are all still there in inbox-Studio and inbox-Lobby, so nothing is lost; there is simply no supported mechanism to re-run them on 16.22.2.

Chronicle 16.28.0 is already merged and waiting to deploy. The next step is to retry the reactor replay on it, which is the supported path if it works there. If it does not, the remaining option is a one-off backfill appending the corrected events — a deliberate, signed-off write to a production event log, not something to slip in.