N-11 · Lock down the account-linking SNS topic (phases 1 and 2) - #106
Merged
Conversation
Found on the live AWS verification of N-11, not in the test suite. With a deliberately-wrong TopsInstallId, the ping was correctly filtered out of teemops_main and landed in the quarantine queue -- and then `aws:link-rejections` reported "No account-linking messages have been rejected." Both numbers were individually right. The cache counters are written by aws:process-sqs and only ever see messages that reached the poller; a message filtered at the SNS topic never gets that far. But the command an operator is told to run was giving a falsely reassuring answer about the exact failure the quarantine queue exists to make visible. The silent failure this feature set out to close had moved one layer out. The command now reports quarantine depth alongside the counters and only gives the all-clear when both are positively known to be empty. A queue it cannot read reports as unknown rather than as zero, because those are different things -- an install that never ran the AWS step still gets its counters instead of an error. The SQS client resolves from the container when one is bound, so both the populated and the unreachable branches are covered by tests rather than skipped the way ProcessSqsMessages' client is. Verified against the live queue that exposed it: 2 quarantined, reported. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
12 tasks
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
teemops-snsacceptedsns:Publishfrom any AWS principal on the internet — the one inbound path into a TOPS install. #103 documented it and deliberately deferred the fix; this is the fix, both phases.Closes #100 and #101. Roadmap N-11.
The topic policy is unchanged, on purpose.
sns:Publishsupports no message-content condition keys, and CloudFormation's custom-resource publish carries no message attributes at all — so the commented-outsns:MessageAttributes.*condition was dead on arrival twice over. It is deleted, with the reasoning left in its place so it is not revived. Narrowing happens on the subscription and in the consumer instead.User story
Not a feature. Security work, tracked as N-11 in
docs/roadmap.md, with the research indocs/features/sns-topic-publish-authorization.md.Phase 1 — the consumer stops trusting the message body (
ProcessSqsMessages)StackId(set by CloudFormation) cross-checked against the account inTopsRoleArn(set by the publisher)pendingrecords are linkable — a replayedCreatecan no longer repoint a live accountTOPS_ACCOUNT_LINK_WINDOW_HOURS, default 24h,0disables)SnsSignatureVerifiervia a newverifyPayload()php artisan aws:link-rejectionsPhase 2 — install-scoped filter secret
TopsInstallIdminted once by the installer, never regenerated, persisted togenerated/teemops.envMessageBody, accepting a list so the value can be rotatedNoEchoparameter threaded through both child templates and the quick-create URLStated honestly, as the doc does: this is a speed bump, not an authentication boundary. The value is shared with every onboarded account admin, travels in a URL query string, and is readable via
cloudformation:DescribeStacks. It raises the bar from anyone on the internet to anyone ever given an onboarding link. Phase 1 is what actually validates a link request.How it was tested
709 tests passing (3 skipped), 46 installer assertions,
scan:validate-rulesclean.ProcessSqsMessagesTest— 8 → 20 tests, one per rejection pathShowLinkRejectionsTest— 9 newtests/install-messaging.test.sh— mint-once/reuse/rotation, and that the id reaches the stack and both templatesaws cloudformation validate-templateTwo existing test fixtures had mismatched
StackIdandTopsRoleArnaccounts — they were encoding the vulnerability as expected behaviour, and are corrected.Verified end to end on a real AWS account (
848310106659/us-west-2), which is the part no test could cover — what was unknown was AWS's own behaviour:tops-vendor-audit-failtest) → filtered out ofteemops_main, landed in quarantine, never linkedTwo things the live run caught that tests did not
1.
aws:link-rejectionsgave a falsely reassuring answer. With a message sitting in quarantine, it reported "No account-linking messages have been rejected." Both numbers were individually correct — the cache counters only see messages that reached the poller, and a filtered message never does — but the command an operator is told to run was reassuring them about exactly the failure the quarantine queue exists to expose. The silent failure had moved one layer out rather than being closed. It now reports quarantine depth alongside the counters, gives the all-clear only when both are positively zero, and reports an unreadable queue as unknown rather than zero.2. Reuse + expiry combined into a trap.
initkeeps one pending row per organization and reuses it, so measuring the window fromcreated_atwould have made expiry permanent — an org that abandoned onboarding and came back would get a fresh-looking link backed by a stale row, and every retry would reuse that same row and be rejected. Measured fromupdated_atinstead, and issuing a link touches the record. Both halves were individually correct, which is why it would have shipped.Beyond the written acceptance criteria
Two additions, because closing the gaps exactly as specified would have left the same hole one step to the side:
StackIdcross-check applies toUpdateas well asCreate—Updaterepoints a live account's roleQuality gate
docs/practices/Practices check
unique_id+external_id, now with strictly more checksTOPS_INSTALL_IDblank in.env.example,NoEchoon both templates, no template defaults); the install id is never written to a log line; no raw SQLShowLinkRejections: a write-only counter is the silent-failure anti-patternCLAUDE.mdwarns about, so the criterion is not met without a readout. If you disagree, it is the cleanest thing to deletescan:validate-rulespasses (untouched)Deployment note for existing installs
A child stack created before this filter existed carries no
TopsInstallId, so itsDeleteping is filtered out and the stack hangs until CloudFormation's ~1hr custom-resource timeout. Either delete such stacks before deploying the new SNS stack, or use--retain-resources TopsCustomNotifier. Documented in the feature doc.generated/teemops.envmust now be backed up — the install id is not recoverable from AWS, and regenerating it invalidates every issued onboarding link and silently filters every linked account'sDeleteping.Out of scope
TopsInstallIdparameter here for parity, but itsParentAWSAccountIddefault and thetemplates/sync.shpublic-read question are untouched; it is labelledroadmap:nextaws:PrincipalOrgIDas an opt-in install parameter — complementary, not a replacement, and filed separately in the feature doc's sequencing🤖 Generated with Claude Code