[Issue #11658] Relax CommonGrants URL pre-validation from strict to lax pydantic HttpUrl

[SnowboardTechie]

Summary

Fixes HHS/simpler-grants-protocol#1068

Prod logs a steady stream of Dropping URL field from CG response: failed dual validation for URLs the response pipeline serves fine, e.g. NASA NSPIRES links with raw {} in the query string. The strict-mode pre-check that drops them mirrors SDK behavior that no longer exists: the SDK dropped model-level strict=True in 0.6.x (HHS/simpler-grants-protocol#555), leaving the pre-check stricter than every validator in the actual egress path. _UrlValidator now uses lax HttpUrl, matching OpportunityBase.source.

The #9904 search-500 protection is untouched. The dual-check's marshmallow leg still drops comma-separated URLs and TLD-less hosts, and HttpUrl still restricts schemes to http/https regardless of strictness (new test covers that).

The old NASA-rejection test becomes a round-trip guard: URLs surviving validate_url() must construct OpportunityBase and load through the marshmallow response schema, so a future SDK that re-tightens URL validation fails CI instead of dropping fields in prod.

Measured impact on real data

additional_info_url comes from legacy fd_link_url (transform_util.py:252), so I ran every AdditionalInformationURL in the public grants.gov daily extract (GrantsDBExtract20260727v2, 32,170 distinct values across 82,975 records) through the dual-check both ways:

	distinct	occurrences
accepted before and after	30,454	50,154
newly served (was dropped and warned)	778	1,469
still dropped	938	2,638

The 778 are mostly trailing whitespace (497 distinct) and NSPIRES braces (245). Nothing in the corpus contains a tab, newline, or carriage return, so no URL gets silently rewritten by control-character stripping. The 938 still dropped are not links at all: www.grants.gov, askcsid@fema.dhs.gov, See Invitation Letter, N/A. Remaining cg_event=url_validation_error warnings become upstream data-quality signal.

Behavior notes

• Consumers may now receive RFC-3986-invalid characters ({}, |, ^) in query strings. Both SDK response validators accept them: the Python SDK's lax HttpUrl and the TS SDK's z.string().url().
• AttachmentValue.downloadUrl shares the helper, so attachments relax the same way, restoring the pre-[Issue #9904] Dual-validate URLs in CG transformation to fix search 500 #9911 behavior that [Issue #9904] Dual-validate URLs in CG transformation to fix search 500 #9911 flagged as an unintended tightening.
• 244 of the newly-served URLs carry a literal & from HTML-escaped legacy data, so source can be populated with a link that mis-targets. Roughly 1,480 such URLs already pass strict and are served today; this adds 244 to that count.

Validation steps

make test args="tests/src/services/common_grants tests/src/api/common_grants"

All 110 pass locally with Postgres + OpenSearch up. make format-check, make lint, make lint-security, and make db-check are clean.

After deploy, confirm the NSPIRES record's source is populated and no cg_event=url_validation_error WARNING is logged for it:

curl -X POST 'https://api.simpler.grants.gov/common-grants/opportunities/search' \
  -H 'X-API-Key: <KEY>' -H 'Content-Type: application/json' \
  -d '{"search":"NSPIRES","pagination":{"page":1,"pageSize":10}}'

[karinamzalez]

A few quick questions!

• Consumers will now get characters like {}, |, and ^ in query strings. The SDKs accept them, but is there anything downstream-- e.g. API docs, the contract, consumer code-- that expects clean URLs and might trip on these? (same Q for attachments since it looks like it uses the same URL helper)
• The 938 values that don't make it through will keep logging cg_event=url_validation_error. Since those are mostly bad data, is that log volume a concern, or are we treating them as a data-quality signal going forward?

[SnowboardTechie]

Good questions. Nothing downstream should trip on them. The marshmallow URL validator is what gates these fields on the way out, and it accepts {}, |, and ^, so the response schema and the OpenAPI it generates agree with what we're serving. Attachments go through the same helper and relax identically. An external consumer applying a stricter RFC parser would still need to handle them itself.

On the logging: not a volume concern, about 3% of records hit it. Treating it as upstream data-quality signal going forward. The remaining values aren't links at all (See Invitation Letter, N/A), and a dropped field is real suppression, so I'd keep it at warning rather than INFO.