rfc: Forge S3 tenant management

[alanshaw]

A short proposal for tenant management in the Forge S3 facade project.

📖 Preview

[Peeja]

question: What are the fields of this invocation? This says Ingot issues it, but /provider/add's subject is the account, which I would assume is the tenant here. Has the tenant delegated this to Ingot? (I think this is meant to be answered below, but I wasn't certain after reading it.)

[hannahhoward]

I'm confused -- why is Ingot doing any of this? Tenant keys should live in Sprue / centralized upload service, created directly in the website GUI.

I believe ingot should just maintain a mapping of access-key-id to ucan delegations

It should be built in the moment upon receiving a request with an associated access key.

If the access-key is in the table used the cached delegation. If not, dial home to Sprue with an invocation like "access/retrieve" or something. "access/retrieve" would return a 24hour UCAN delegation and a 24 hour signing key (signing keys are by nature 24hours expiration in the sigv4 spec)

Of note: don't hold secret keys inside ingot.

[alanshaw]

I'm confused -- why is Ingot doing any of this? Tenant keys should live in Sprue / centralized upload service, created directly in the website GUI.

That did occur to me shortly before I opened the PR but I didn't have time to edit/suggest that as an alternative. I think most of the proposal here would apply but I just need to work it through.

Of note: don't hold secret keys inside ingot.

Okay I get it 😆

[bajtos]

Can we prefix the secret with something like f1_ to make it clear this is an FilOne secret?

Can we use a different encoding than Base64 to avoid characters like +, / and = - these characters are not HTTP safe and must be often encoded in transit. Proposal: Base64URL with no padding.

[alanshaw]

Why do we need to prefix? Also why would anyone put the secret in a URL?

[bajtos]

Why do we need to prefix?

It's a good practice that is being adopted by companies like Slack, Stripe, GitHub and others.

Access tokens are prefixed to make them easier to identify and distinguish from other types of data, which enhances security by reducing the chances of false positives during token detection. This design choice also improves readability and usability when handling tokens.

See also:

• https://github.blog/engineering/platform-security/behind-githubs-new-authentication-token-formats/
• https://developers.cloudflare.com/fundamentals/api/get-started/token-formats/

Also why would anyone put the secret in a URL?

Right, that was not a good argument.

What I am looking for: a secret token should contain only word characters, so that double-clicking on a token selects the entire secret.

[Copilot]

Pull request overview

This PR adds an RFC proposing a tenant-management model for the Forge S3 facade (Ingot), introducing Hilt as a centralized service to manage tenants, buckets/spaces, S3 access keys, and UCAN delegations used to authorize SigV4/SigV4a requests.

Changes:

• Adds an RFC describing Hilt’s Tenant API for tenant/bucket/access-key creation and delegation storage.
• Specifies a UCAN API (/aws/request/authorize) for authorizing AWS API requests and returning derived signing keys + proof chains.
• Documents alternative approaches and references relevant AWS/SigV4a resources.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[hannahhoward]

main blocking is about the use of SigV4a

[hannahhoward]

can provider/add not be delegated from tenant key to regional provider DID?

[alanshaw]

Yes, but that raises a lot of questions for me:

• How is this delegation communicated to Ingot? It is on-demand or a one time thing at setup?
  • For on-demand we'd have to create an API for that on Hilt, have Ingot call it to retrieve the delegation and provide some params to verify the request is legit (comes from a valid access key).
  • If it's one-time at setup then we have to build an setup step that calls the delegator(?) or rely on engineers to manually issue a delegation and have the operator add it to config.
  • Also, a one-time at setup delegation would have to be allowed to provision any space, since we don't know ahead of time what the DID is. I know this is a trusted service but IMO we should limit trust where possible and I think allowing it to provision any space is too much.
• How does Hilt know when to /access/claim? I assume on the first read/write after bucket creation?
  • I think things are simpler if we implemented something like /access/delegate on Hilt and have it only accept delegations addressed to itself. Maybe that's an option?
• How does Hilt remove the delegation from Sprue when a bucket is deleted? There's currently no facility for this in Sprue.

On the whole it feels like a lot more work to go this route...

[hannahhoward]

I think there’s a misunderstanding with the role of signing keys.

Let’s talk about SigV4 and SigV4a. Basically these are both schemes from AWS to achieve exactly the function we’re doing in our forge deployment with hilt and ingot:

• The secret access key is only given to the client and stored in a single multiregion IAM service analogous to hilt.
• In both SigV4 and SigV4a, the client uses the secret access key to generate a signature on each request.
• The regional provider receives a value that can be used to VERIFY the signature on a request, but with a limited scope.
  • In the case of SigV4, this is the “signing key” which is an HMAC symmetric derivation from the secret access key containing the date and some other parameters (including the region) — because it's a symmetric derivation the provider could technically use it to sign it's own SigV4 requests for a 24 hour period, but it is only used by the regional provider to verify requests to the region from clients without calling home again for 24hours
  • In the case of SigV4A, an EDCSA public/private key pair is derived with a key derivation function from the secret access key, and only the public key is sent to the regional provider. The regional provider can only use this public key to VERIFY requests, never to sign them. It's not 100% clear from docs if the key is date scoped.

Either way, the decision to use SigV4 vs SigV4a is made on the client. We have to support both.

Moreover, we should never be sending SigV4a private keys over the wire -- there is no need for a provider to generate additional requests. Instead, if we need UCAN delegations, we can just delegate to the regional provider's public DID with a 24 hour scope.

So then key section would look like this:

type KeyKind enum {
   | SigV4
   | SigV4a
}

type VerificationKey struct {
   kind KeyKind
   verificationData bytes
}

type AuthorizeOK struct {
  spaceDID DIDBytes
  keys        { String: [VerificationKey] }
  delegations { String: [Link] }
}

Except here key is an access-key-id

[bajtos]

Instead, if we need UCAN delegations, we can just delegate to the regional provider's public DID with a 24 hour scope.

IIUC, in this model, the requests from Ingot to Sprue/Piri will be using the provider's public DID as the invoker identity.

As a result, we will lose the audit trail about who made the original request - Sprue & Piri will see all requests as invoked by Ingot, not by the S3 Access Key. Although, if the delegation chain in the UCAN ticket differs depending on which S3 Access Key was used to sign the S3 request, then we would still be able to differentiate invocations received by Sprue/Piri in relation to the S3 Access Key invoked.

I don't know whether it's a relevant requirement for Sprue and Piri to be able to tell if two requests coming from Ingot were signed by the same S3 Access Key or not, so perhaps it's not something we need to worry about.

[alanshaw]

I think there’s a misunderstanding with the role of signing keys.

I guess so - I thought the idea was to issue invocations from a key derived from the access key for audit purposes. In this proposal a delegation from the access key will be present in the delegation chain - is that good enough? I think so - having the access key in the delegation chain makes it easy to delete it, you just revoke the access key delegation and then you've cancelled all invocations that might use it.

I had overlooked the case where a second request would need to use a cached sigv4 key to verify the request signature and so, since the client is in charge of generating the sigv4 we'd have to return the same type the client used...

I'm mildly worried about delegating directly to Ingot though - that gives the ingot key a LOT of power over all buckets it is managing. I would prefer if the key used to issue invocations on Forge were at least unique per bucket. Thinking about it - that could be something that Ingot creates rather than Hilt and could be added later. I'll update the RFC to specify that the returned delegations should target the issuer of the /aws/request/authorize invocation - which will be Ingot, but could in the future be a key ingot generates per bucket or something.

[hannahhoward]

It's not 100% clear how these last 3 operations work through /aws/request/authorize

[hannahhoward]

Is UCAN just a modelling language here? I continue to wonder whether we shouldn't just use this simpler alternative.

[hannahhoward]

I wonder if this shouldn't be broken up into a lot of requests with different caching properties -- it's doing a whole lot.

specifically:
/aws/access-key/authorize
-> request contains:
-> access key id
-> sends back:
-> 24hr signing/verification keys
-> all delegations from tenant to regional provider did through the access key id good for 24 hours

/s3/bucket/info
-> request contains
-> bucket name
-> sends back
-> space did
-> delegation from space did to tenant

There are just different caching properties here that might mean less work to do?

[bajtos]

"A Fil One organization" can be potentially misleading - aren't we the Fil One organization?

Suggestion:

A Fil One customer account (organization).

[bajtos]

The decision record may not be the best resource for the tenant management API docs. Have you considered linking to README instead?

https://github.com/fil-one/fil-one/blob/main/docs/service-orchestrator-integration/README.md

[bajtos]

Should we mention the requirement that Hilt must support rotation of this pre-shared bearer key? I.e. to support more than one bearer key being active at any time, plus methods for creating and revoking keys.

[bajtos]

Which component creates a new tenant key - is it Hilt or Ingot? I assume Ingot should not have access to tenant keys, therefore a new key must be created by Hilt. If that's the case, then we need either:

• Hilt to register the new key with Ingot before Ingot can issue a /account/add invocation to Sprue,
• or let Hilt issue the /account/add invocation to Sprue.

Thoughts?

[bajtos]

Should we also prepend a multiformat varint for base64url encoding (U+0075/u)?

The example in the document a few lines later shows a secret access key string starting with the u character, so I think the multibase prefix is already prepended by the multibase.Encode() call; so perhaps I just misunderstood what the RFC is saying.

Is it worth clarifying the RFC text here?

[bajtos]

I am a bit concerned about the readability of command names like /s3/putobjectlegalhold. Have you considered using kebab-case or snake-case?

• /s3/put_object_legal_hold
• /s3/put-object-legal-hold

[bajtos]

Instead, if we need UCAN delegations, we can just delegate to the regional provider's public DID with a 24 hour scope.

IIUC, in this model, the requests from Ingot to Sprue/Piri will be using the provider's public DID as the invoker identity.

As a result, we will lose the audit trail about who made the original request - Sprue & Piri will see all requests as invoked by Ingot, not by the S3 Access Key. Although, if the delegation chain in the UCAN ticket differs depending on which S3 Access Key was used to sign the S3 request, then we would still be able to differentiate invocations received by Sprue/Piri in relation to the S3 Access Key invoked.

I don't know whether it's a relevant requirement for Sprue and Piri to be able to tell if two requests coming from Ingot were signed by the same S3 Access Key or not, so perhaps it's not something we need to worry about.