feat(core): prepare v0.4.0 with new routes for awards and orgs - #976
Conversation
* feat(core): add boolean and integer comparison filter models Add BooleanComparisonFilter (eq/neq against a boolean) and IntegerComparisonFilter (comparison + equivalence operators against an integer), exported via filters/index.tsp at protocol v0.3.0. Refs #895. * docs: correct changeset — comparison filters emit to JSON Schema only Filter models aren't route-referenced, so the openapi3 emitter omits them; only the json-schema emitter outputs them. The changeset claimed both. * fix(core): version new comparison filters at v0.4 v0.3 is already published, so BooleanComparisonFilter and IntegerComparisonFilter first ship in v0.4. Add v0_4 to the Versions enum and retag both models; correct the changeset to match. * chore: remove accidentally committed .venv and gitignore it The lib/python-sdk/.venv virtualenv (1,039 files) was swept in by an overly broad git add. Untrack it and add .venv/ to the python-sdk .gitignore so it can't recur.
|
🚀 Website Preview Deployed! Preview your changes at: https://cg-pr-976.billy-daly.workers.dev This preview will be automatically deleted when the PR is closed. |
* chore(website): upgrade to Astro 7 (Starlight 0.41, @astrojs/react 6) * feat(website): add ID registry catalog (lib, components, pages) * refactor(website): group catalogs in sidebar and header nav * docs(website): Update ADR-0023 with new registry code Switches to `<schema>:<scope>:<prop>` format, like `org:us:ein` to match the ID registry catalog * feat(website): add code to registry search fields The RegistryCard search tags only included the registry.code segments, which meant that searches for the full code didn't return a match. We fix this by replacing the segments with the full code (org:us:ein), which still matches searches that contain only the segments (ein).
* feat(core): add Identifier and SystemId models Add the Identifier and SystemId field models, the IdentifierT template they instantiate, and the Identifiers collection. Tighten the employerTaxId, samUEI, and duns scalars to their registry-specific formats. * refactor!(core): replace org IDs with collection Remove the ein, uei, and duns fields from OrganizationBase and add an identifiers collection (OrgIds) built from the OrgIdEin, OrgIdUei, and OrgIdDuns registry-specific models. BREAKING CHANGE: OrganizationBase.ein, uei, and duns are removed in favor of the identifiers collection. * fix(website): apply removed fields in versioning The per-version schema reconstruction now strips fields whose @removed version is at or before the target version, not just fields added after it. Without this, v0.4 OrganizationBase kept the removed ein/uei/duns fields. * docs(governance): trim Identifier.registry in ADR Rename IdentifierCollection to Identifiers and OrgIdentifierCollection to OrgIds in ADR-0023 to match the implemented models. * feat(website): document org identifiers Add the Identifiers field page (Identifier, SystemId, Identifiers) and per-identifier sections (OrgIds, OrgIdEin, OrgIdUei, OrgIdDuns) on the Organization model page. * refactor(website): map questions/forms to org IDs Repoint the question-bank and SF-424 form mappings from the removed org ein/uei/duns fields to the identifiers collection (organizations.primary.identifiers.org:us:ein.id, etc.). * chore(website): regenerate schemas for v0.4 IDs Regenerate the published JSON schemas for the identifier models and the updated OrganizationBase. * refactor(core): simplify Identifier model Removes a few fields like `verifiedAt`, `uri`, `registry.schema` etc. to prevent us from prematurely adding props that we wind up removing later. Also makes all of the props in this model optional. * docs(website): fix duplicate wording in ADR-0023 Refers to `IdentifierCollection` as a "field" instead of a "collection" to avoid repetition. * fix(changelog): dedupe added/removed versions for template instantiations A model or property defined via `is`/`extends` on a versioned base inherits the base's `@added`/`@removed` decorators in addition to its own, so `getAddedOnVersions`/`getRemovedOnVersions` can report the same version more than once. This produced duplicate "Added X model" changelog entries for the identifier template instantiations (Identifier, SystemId, OrgIdEin, OrgIdUei, OrgIdDuns). Collapse duplicate versions before logging model/property additions and removals, since an entity can only be added or removed once per version. * docs(website): fix stale TypeSpec line ranges The `refactor(core): simplify Identifier model` change and the org-identifier additions shifted line numbers in identifier.tsp, organization.tsp, types.tsp, and filters/numeric.tsp, but the doc frontmatter ranges were not updated, so several TypeSpec tabs rendered the wrong model or truncated code. Realign every affected range to its declaration (doc comment through closing line): identifier/SystemId/IdentifierCollection, OrgSocialLinks, the string, date, and numeric scalars, and the number range/array filters.
* refactor!(core): rename Application.name to title
Also renames the application's `name` field to `title`, for
consistency with the other models and the new reference models.
Adopters reading application data at protocol version 0.4.0 should
expect `title` in place of `name`. Also renames name -> title in
the request body for `POST /common-grants/applications/start/`
* feat(core): add lightweight reference models
Adds a lightweight reference model for opportunities, applications, and
organizations (OppRef, AppRef, OrgRef, plus an OrgRefCollection for
grouping organizations). A reference model previews the key fields a
consumer needs to identify a record and look it up or join it to
another response, without repeating the full model.
Each base model now includes its own reference model, so the reference
always matches the fields on the base, and any later change to those
shared fields appears in the base model's changelog. Record ids are
also now read-only across these models, matching opportunities.
* feat(core): add the award model and endpoints
Adds AwardBase, a model for reporting on a grant award: its funder,
recipient, financial amounts, and key dates, along with references to
the opportunity and application it resulted from. Awards can name
either an organization or an individual as the recipient.
Adds three endpoints an adopter can offer for awards:
- list awards: GET /common-grants/awards/
- view an award: GET /common-grants/awards/{awdId}
- search awards: POST /common-grants/awards/search
* docs(website): document the award model
Adds an Award page under Protocol > Models covering the award model
and its supporting types. Each reference model is documented on the
page for the model it points to (opportunity, application, and
organization). Regenerates the published schemas and OpenAPI
specification so the new models appear in the docs.
* chore(website): remove unused OpenAPI tags
Adds a build step after the OpenAPI generation that removes any tag from
a version's tag list when none of that version's routes use it. TypeSpec
lists every tag in every version, so earlier versions advertised tags
like Awards or Competitions that only have routes in later versions.
Now each version's docs only list tags a consumer can actually find
routes for. The step edits only the tag list, so the rest of each
generated file stays unchanged.
* docs(website): add AwdIdFain and surface v0.4.0
Documents AwdIdFain on the award model page, which was missing, and adds
v0.4.0 as the latest option in the API docs version dropdown so its
routes (including awards) are viewable. Also regenerates the schemas
after the recipientType removal.
…with T suffix (#1007) * feat!(core): add non-templated response schemas and rename templates with T suffix Templated response models (Ok<T>, Paginated<T>, Sorted<T>, Filtered<ItemsT, FilterT>, Created<T>) are never emitted as standalone JSON schemas, which left them out of the schema changelog and forced hand-written schema blocks in the website docs. - Add concrete Ok, Paginated, Sorted, Filtered, and Created models that emit JSON schemas, versioned from v0.1.0 - Rename the templated variants with a T suffix (OkT, PaginatedT, SortedT, FilteredT, CreatedT); routes keep using these, and shared *Base<T> templates keep the concrete and templated variants in sync - Document the concrete schemas on the website responses pages with changelogs, using the emitted schema files instead of hand-written blocks Closes #960 * review: address correctness and simplicity findings - SortedT now extends Success, keeping it in sync with the concrete Sorted model (no route uses SortedT directly, so emitted output is unchanged) - Add missing @doc to CreatedT - Consolidate the templates-don't-emit rationale into one file-header comment - Fix docs frontmatter line ranges after the above edits * chore(core): migrate awards routes to T-suffixed templates and regenerate OpenAPI for the v0.4.0 base Rebasing onto HOLD-core-v0.4.0 brought in routes that referenced the old templated response names. Regenerating also syncs the CLI OpenAPI copies (including a new openapi.0.4.0.yaml) that earlier merges to the hold branch had only regenerated on the website side. * review: revert bundled CLI OpenAPI specs to base Regenerating the CLI's default specs happens in a follow-up PR once the core v0.4.0 changes are locked, with its own minor CLI changeset.
* feat(core): add ExtensibleEnum schema and templated model (#961) Formalize the extensible enum pattern (value + optional customValue + description) as a reusable Fields.ExtensibleEnum model and Fields.ExtensibleEnumT<T> template, mirroring the CommonBenefits protocol. OppStatus, AppStatus, ApplicantType, FormResponseStatus, and CompetitionStatus are now one-line instantiations of the template. Adds the custom option to FormResponseStatusOptions at v0.4.0 (its docstring already listed it) and drops a stray customValue from the CompetitionStatus example. Emitted schemas are unchanged apart from property description wording; published versions 0.1.0-0.3.0 keep their exact validation shape. * docs(website): drop stray customValue from Competition docs example The hand-written CompetitionBase example in the competition docs page had the same incoherent status example (customValue set while value is open) that was removed from the TypeSpec source; customValue only applies when value is custom. * docs(website): lead extensible-enum page with the ExtensibleEnumT template The intro presents the template as the way fields adopt the pattern, so its section now comes first; the untyped base model follows, keeping the page's changelog at the end of the final section like other field pages. * refactor(core): derive ExtensibleEnum from ExtensibleEnumT per review Define only the template and derive the untyped model as ExtensibleEnumT<unknown>, so the two can't drift. Rework the doc split: ExtensibleEnum carries the reader-facing pattern description (emitted JSON schema), ExtensibleEnumT carries maintainer-facing usage docs with the instantiation example. Re-pin the docs page line references and align its table wording with the emitted schema. Emitted output is unchanged except the ExtensibleEnum descriptions; all versioned OpenAPI and the five model schemas are byte-identical. * chore(deps): bump astro and brace-expansion Bump astro to 7.1.3 (latest 7.x) to clear the reflected XSS advisory GHSA-4g3v-8h47-v7g6, and refresh brace-expansion to 2.1.2 to clear the DoS advisory GHSA-3jxr-9vmj-r5cp that failed the high audit gate. Both are lockfile/range updates, no overrides. * chore(deps): fix remaining vulnerability for js-yaml --------- Co-authored-by: widal001 <billy.daly@agile6.com>
* feat(core): add 202 Accepted and 403 Forbidden responses Add Responses.Accepted<T>, a 202 envelope with a Location header, for requests that are accepted for later processing, such as a change submitted for review. As well as an an alias for 403 Forbidden errors. * feat(core): add org syncing models and routes Add the models and routes for syncing organization profiles across systems, following ADR-0026. Models: - OrgPatchData, a JSON Merge Patch body derived from OrganizationBase with all fields optional and read-only fields excluded. - RevisionT and its untyped form Revision, a change record with a status, patch, and snapshot, plus RevisionStatus. - OrgRevision, binding a revision to organization profiles. Routes under /common-grants/orgs, each requiring an OAuth 2.0 scope: list and read orgs, patch an org, submit a change, and list or view changes. The routes declare an OAuth2 security scheme with placeholder flow URLs each deployment overrides. All added to version 0.4.0 and marked experimental. * docs(website): document org syncing models Add an OrgPatchData section to the Organization page and a new Revision page covering Revision, RevisionStatus, and OrgRevision, with the change status lifecycle. Regenerate the schemas and OpenAPI so the new models and v0.4.0 routes appear in the docs. * docs(website): add authentication and scopes page Add a Protocol page documenting the OAuth 2.0 flows, bearer JWT tokens, the scope vocabulary and per-route scopes, the org_id binding, required JWT claims, and the inbound trust checks. Add it to the Protocol sidebar. * docs(website): align ADR 0026 examples with impl Fix the parts of the ADR that directly contradict the implementation of the routes and models for organization profile syncing. --------- Co-authored-by: Bryan Thompson <18094023+SnowboardTechie@users.noreply.github.com>
# Conflicts: # pnpm-lock.yaml # website/package.json
Catches the hold branch up to main so the dep-audit gate passes. The branch was missing the `sharp@<0.35.0` floor added in #1038, which failed the website `validate` job on GHSA-f88m-g3jw-g9cj (high). Reconciles `pnpm-workspace.yaml` overrides to main's set: takes main's split `brace-expansion` ranges, drops the `fast-uri` floor main removed, and keeps the `immutable`/`js-yaml` floors that only exist here. Regenerates the lockfile from the merged manifests.
* fix(core): use fully qualified registry codes in org docs and examples Registry codes follow `<schema>:<scope>:<prop>`, so the canonical forms are `org:us:ein`, `org:us:uei`, and `org:grants.gov:system`. The `orgs.tsp` `registry` query param doc and the ADR-0026 examples still used short forms, which contradicted the `OrgIds` and `IdentifierT` definitions. Also corrects the catalog links alongside the codes: slugs derive from the code (`org:us:ein` -> `org-us-ein`), so `/registries/us-ein` and `/registries/grants-gov-org` pointed at pages that do not exist. Regenerates the 0.4.0 OpenAPI spec to pick up the param doc change. * docs(changeset): cover the org identifier models The identifier work from #957 landed on this branch without a changeset, so the 0.4.0 release notes for `@common-grants/core` would have shipped without `Identifier`, `SystemId`, `IdentifierCollection`, or `OrgIds`, and without either of its breaking changes: the removal of `OrganizationBase.ein`/`uei`/ `duns`, and the tightened `employerTaxId`/`samUEI`/`duns` scalar formats. Minor, matching the other 0.4.0 changesets, so core still lands on 0.4.0. * docs(core): align ADR-0026 and org-syncing changeset with shipped names - .changeset/org-profile-syncing.md: the 202 envelope is AcceptedT<T> (plus non-templated Accepted) per the #1007 T-suffix convention, and the org routes also return the new Responses.Forbidden alias - ADR-0026: use OkT/PaginatedT in the contract summary, and drop the registry.scope/kind fields from the GET /orgs/{orgId} example — IdentifierT.registry has only code and url; catalog URLs are now absolute, matching the identifier.tsp examples
…ta (#1053) * fix(core): emit RFC 7396-compliant merge-patch schemas for OrgPatchData MergePatchUpdate emits two shapes that reject valid merge patches: record members get no null branch (per-key deletion fails), and models referenced through extends keep the un-patched base schema via allOf (inherited fields are not optional or clearable). Both defects live upstream in @typespec/http. Work around both by hand-composing the merge-patch sources: a non-emitted CommonGrants.Patch namespace mirrors each org profile sub-model with extends flattened via spread and record members accepting null, and OrgProfileWritable references the mirrors. The single MergePatchUpdate derivation and all emitted OrgPatch* schema names stay unchanged; the collection-level OrCreate schemas now inline as $defs of OrgPatchData instead of standalone files. * test(website): cover RFC 7396 behavior of the OrgPatchData schema Validates the published schemas verbatim with a dedicated Ajv instance (the shared loader strips top-level unevaluatedProperties): record-key deletion, inherited-field clearing, sealed-schema rejection, and a drift guard comparing OrganizationBase properties against OrgPatchData. * test(core): cover all merge-patch mirrors and guard both RFC 7396 invariants Three of the six hand-written `CommonGrants.Patch` mirrors had no dedicated test case: `addresses`, `phones`, and `orgType` appeared only as bare key names in the top-level drift guard, despite being behaviors these schemas changed. Adds the missing per-member deletion cases plus invalid cases for each. Adds two structural invariants over the emitted `OrgPatch*` schemas, replacing what the top-level drift guard could not cover — the per-mirror surface the namespace comment calls out as hand-maintained: - every record member accepts `null`, so per-key deletion works - no patch schema inherits an un-patched read schema via `allOf` Both are asserted over the emitted shape rather than a hand-listed set of fields, so they need no upkeep as the base models change. Verified to discriminate: against the pre-fix schemas they report five and two violations respectively, and the `allOf` guard also fires when `extends` is added to a leaf type such as `Fields.Address`, which no existing test caught. Also renames the drift guard to say top-level, and notes the leaf-type case in the `Patch` namespace comment. Found by the test-coverage and type-design lenses of pr-self-review. * refactor(core): move OrgProfileWritable into the Patch namespace It is a compile-time source for OrgPatchData, not an emitted schema, which is what the CommonGrants.Patch namespace holds. Keeping it in the root CommonGrants namespace made it the only model there outside index.tsp. OrgPatchData pins its own component name through the `is` declaration, so the emitted 0.4.0 OpenAPI and OrgPatch* schemas are unchanged.
Reorders the OpenAPI tags so that Opportunities and Applications appear above Organizations and Awards in the auto-generated OpenAPI docs on the website
### Summary Second cleanup pass over the v0.4.0 hold branch, covering the three items left open on #1032: - `BooleanComparisonFilter` ships in 0.4.0 but had no docs page, so every plugin's `booleanComparison` filter linked to the filters overview instead of a model. - The two status models added in 0.4.0 hand-rolled the extensible enum shape instead of instantiating `Fields.ExtensibleEnumT<T>` like every other extensible enum field. - ADR-0023's recommended examples described a richer `registry` object than the one that shipped. - Closes #1032 - Time to review: 10 minutes ### Changes proposed > What was added, updated, or removed in this PR. **Document `BooleanComparisonFilter`** — adds `protocol/filters/boolean.mdx` (table, format tabs, changelog) and a Boolean card on the filters overview, then points the plugin filter map at `/protocol/filters/boolean#booleancomparisonfilter`. The map previously sent `booleanComparison` to the overview because no boolean model existed when the custom filters section was built in #1045; the model landed in #953 on this branch. Both tests that pinned the old fallback are dropped rather than repointed: removing the `if (!anchor) continue;` guard raises the anchor-exists check from the nine types with anchors to all ten, which covers `booleanComparison` on its own, and the loader resolves every filter through one `filterDocsHref` call, so the `agency` case already covers filterType to docsHref. The py-SDK docstring saying no spec model backs the filter is removed for the same reason. **Define the 0.4.0 status models via `ExtensibleEnumT`** — `AwdStatus` and `RevisionStatus` both spelled out `value` / `customValue` / `description` by hand. `OppStatus`, `AppStatus`, `ApplicantType`, `FormResponseStatus`, and `CompetitionStatus` all instantiate the template, so these two were the only extensible enum fields with a second source of truth for the shape. `RevisionStatus` is included alongside `AwdStatus` because it has the same defect and ships in the same release. Wire shape is unchanged; the regenerated `AwdStatus.yaml`, `RevisionStatus.yaml`, and `openapi.0.4.0.yaml` diffs are property descriptions moving to the template's wording, which is what `.changeset/extensible-enum-field.md` already documents for the other five fields. That changeset's list is extended rather than adding a new one. **Align ADR-0023 with the shipped identifier** — the recommended nested-hybrid example carried `scope`, `kind`, `schema`, and a top-level `uri`, and the recommended parent-reference example carried `scope` and `kind`. `IdentifierT.registry` defines `code` and `url` only, and both are optional, so the prose describing `code`/`url` as required is corrected too. The pros, cons, and three comparison rows that rested on the dropped fields move with the example: inline scope/kind filtering and per-registry schema validation are now ❌ for the nested hybrid, payload brevity ✅, and self-contained-without-a-catalog ❌. The remaining `scope`/`kind` examples are in the rejected flat and fully-decomposed options, where the richer shape is the point, and the `US-EIN`/`US-SAM` codes are in the rejected adopt-org-id.guide-codes option. This is the follow-up to karinamzalez's question on #1040 about whether ADR-0023 should drop the fields ADR-0026's examples dropped. ### Context for reviewers > Testing instructions, background context, more in-depth details of the implementation, and anything else you'd like to call out or ask reviewers. Explain how the changes were verified. Verified with `pnpm run ci` from the root (exit 0), plus `make checks && make test` in `lib/python-sdk` (0 pyright errors, 396 passed). The website build reports all internal links valid, which covers the new overview card and the anchors on the new page. The anchor check was confirmed to discriminate: pointing `booleanComparison` back at `/protocol/filters/` fails it. ### Additional information > Screenshots, GIF demos, code examples or output to help show the changes working as expected. The shipped shape the ADR examples now match, from `lib/core/lib/core/fields/identifier.tsp`: ```typespec model IdentifierT<Id extends string = string, Code extends string = string> { registry?: { code?: Code; url?: url; }; id?: Id; allIds?: Array<{ id: Id; status: IdentifierStatus }>; } ``` The regenerated schema diff for both status models is descriptions only: ```diff value: $ref: AwdStatusOptions.yaml - description: The status of the award, from a predefined set of options + description: The selected value, from a predefined set of options customValue: type: string - description: A custom value for the status + description: A custom value, used when the selected value is the `custom` option ```
Resolve three-way conflicts from #1054 (simplify openapi doc) and #1056. - website/src/components/OpenApiDocs.tsx: accept main's deletion. The branch's only change was adding v0.4.0 to the version list, which is ported to the replacement Astro page below. - website/src/pages/protocol/api-docs.astro: take main's swagger-ui-dist page and add v0.4.0 to availableVersions as the latest, demoting v0.3.0. - pnpm-lock.yaml: regenerate from main's lockfile so the swagger-ui-react -> swagger-ui-dist swap is kept, then reinstall to reapply the branch's immutable and js-yaml overrides.
Changes `/protocol/fields/identifiers` -> `protocol/fields/identifier/` to match the other fields and model pages which are singular
ce8059c to
2263d19
Compare
This filter was descoped from the custom filter release in the SDKs, so we don't need or want to add it to v0.4.0 in the core library. Integer comparisons can be handled by NumberComparisonFilter
@common-grants/core v0.4.0 release
SnowboardTechie
left a comment
There was a problem hiding this comment.
The rename commit updated the Identifier link card title but not the hrefs, so two links still point at the old plural path and 404 on the preview. Inline suggestions for both.
Approving.
Fixes cross-linking to /protocol/fields/identifier/ after renaming the page Co-authored-by: Bryan Thompson <18094023+SnowboardTechie@users.noreply.github.com>
|
🗑️ Preview Cleaned Up The preview for this PR has been automatically deleted. |
karinamzalez
left a comment
There was a problem hiding this comment.
Looks like this is mostly auto-generated content? I do have a few clarifying questions :)
- What's the merge order relative to #1051? (I just reviewed this one and it looks like it's deleting changeset directory)
- are these breaking changes and is it intentional that they're shipping as minor?
| ``z.boolean()`` semantics of the TS SDK's ``BooleanComparisonFilterSchema``. | ||
|
|
||
| No spec model backs this filter yet: the core spec defines no boolean | ||
| filter, so both SDKs carry an SDK-level model until one lands. |
There was a problem hiding this comment.
is it intentional that ts-sdk is untouched and that only the boolean filter is touched here?
There was a problem hiding this comment.
Great question! This is just modifying the docstring that said a boolean filter didn't yet exist. The TypeScript SDK didn't have a comment like that on the equivalent schema
| * Trims each generated OpenAPI document's top-level `tags` list to the tags its | ||
| * operations actually use. | ||
| * | ||
| * TypeSpec emits every `@tagMetadata` entry into every version's `tags` list, |
There was a problem hiding this comment.
Is there an upstream TypeSpec issue for this @tagMetadata workaround? The versioning workaround in version-generator.ts links to microsoft/typespec#2051, which makes it clear when that code can go away. A similar link here would tell us when this script is safe to delete.
Summary
Collects changes we're queuing for the v0.4.0 release of the
@common-grants/corelibrary@common-grants/coreminor version with awards #956Note
Except for merge conflicts and updates from
main, each of the commits that merged into this HOLD branch have been reviewed in separate PRs, so the changes shouldn't need a full review.Changes proposed
ExtensibleEnumfield type to standardize fields likeOppStatus,AppStatus, etc.Ok,Created,Paginated, etc.IdentifierandIdentifierCollectionExtensibleEnumAwardand related sub-modelsRevisionandOrgRevisionOrgPatchData@removed()(@typespec/json-schemadoesn't do this natively)Context for reviewers
Final review checks
OrgPatchDataon OrganizationAdditional information
Open API spec
New v0.4.0 with Awards and Organizations
v0.1.0 without the extra tags
Updated schemas
Identifier
Extensible enum
Award
ID registry catalog
Changesets
The output of running
pnpm changeset version