Streaming signature validation

[yaleman]

Ref #10 this is an initial run on doing validation.

It looks pretty horrible but you know the internals of your lib better than I do, and I didn't know how far I should chase optimising things... it's behind the feature "streaming" for now. I'm not precious about feedback so am happy to do whatever 😄

[dacut]

I’m traveling today, but will hopefully have a chance to look at this while I’m in the air (airplane WiFi willing).

[yaleman]

No need for a rush, I kind of went on some side-quests, apologies

[dacut]

FYI, I am working on this. Exploring how a few things might integrate with it, which is what's taking so long.

[dacut]

I love the spooled tempfile/bufferedbody work! However, I'm not sure that having separate validate/validate_streaming functions will work with the authentication as-is. In particular, we don't know whether the request requires checking X-Amz-Content-SHA256 until we've cracked open the headers.

I'm thinking of keeping the validate/validate_streaming separation as you've done it solely for how we handle the body. S3 and S3-like services will prefer the validate_streaming method (though there might be others?). We can rely on the s3 setting in SignatureOptions to handle the various methods of authenticating the body.

Just for my own reference, the cases we need to handle are:

• Single chunk
  • Signed payload: Content SHA256 is sent in the x-amz-content-sha256 header and canonical request.
  • Unsigned payload: UNSIGNED-PAYLOAD is sent in the canonical request. Check whether x-amz-content-sha256 is set to this or the SHA256 of this; I remember being surprised by this.
  • Can this be sent as a query parameter? AWS docs suggest no, but these aren't always correct.
• Multiple chunks
  • x-amz-decoded-content-length must be set.
  • x-amz-content-sha256 is one of STREAMING-UNSIGNED-PAYLOAD-TRAILER, STREAMING-AWS4-HMAC-SHA256-PAYLOAD, STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER, STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD, or STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER
  • Trailer requests need to have x-amz-trailer to be set to the headers transferred there (e.g. x-amz-checksum-crc32c).

[yaleman]

In my codebase I've got a check for streaming bodies:

        // Check if this is a streaming/chunked request
        let is_streaming = if let Some(content_sha256) = parts.headers.get(X_AMZ_CONTENT_SHA256) {
            if let Ok(sha_str) = content_sha256.to_str() {
                sha_str.starts_with("STREAMING-") || sha_str == "UNSIGNED-PAYLOAD"
            } else {
                false
            }
        } else {
            false
        };

which then goes off to handle verification for streaming/non streaming bodies

[dacut]

@yaleman Would it be useful to move this logic into scratchstack-aws-signature, or would you expect others writing similar services to want to handle it themselves? (I can make arguments in my head either way...)

[yaleman]

It seems totally fine to move it in, but still leave the options open to use the downstream calls for testing/other uses 😄 Making things easier for folks makes sense, it's a pretty messy thing to implement!

[yaleman]

For what it's worth my is_streaming_request got pulled out into its own function because it got bigger than the original:

https://github.com/yaleman/crabcakes/blob/4272d7644a62af374b6f7703de50631208c4c4a8/src/web/s3_handlers.rs#L69-L92

[dacut]

I've been traveling a lot, but have been working on validating that the error codes/messages here match AWS' behavior (similar to the existing documentation I have. My previous code for this was a bit of a mess, so I've begun reorganizing it here: https://github.com/dacut/aws-auth-flow-tester

I've discovered that AWS has changed what they report back for certain cases, so the existing diagram and code is no longer correct. It looks like AWS has taken a renewed interest in formalizing some implicit SigV4 behaviors (e.g. https://github.com/aws-samples/sigv4-signing-examples), possibly as part of publicly documenting SigV4a.

[yaleman]

That's great that they're finally documenting some of their things for better interoperability 😄

[dacut]

This ends up being a lot more involved than I had expected once you start accounting for other S3 APIs that also have PUT request bodies. Trying to account for some of the particular S3 behaviors (when to return errors, for example) starts bringing a lot of S3 behavior into this crate that ... well, probably belongs in the S3-like service itself.

In particular, some of this requires un-chunking the aws-chunked encoding and rewriting the body back to things like SpooledTempFile, which feels weird. Also, handling of S3 trailers ends up being difficult to handshake between the validation routine and the S3-like service.

I'm going to take a different approach and provide the primitives necessary for an S3-like service to perform validation. This will have two separate functions:

• One to validate after the headers have been read but before the body has been consumed.
• Another to validate each chunk from the amz-chunked format.

[yaleman]

Makes sense, it's a very messy thing to support 😄