Recently we started encountering AzureBlob::Http::Error: AzureBlob::Http::Error when trying to use compose to copy a blob which was originally created from a call to compose with 2 source blobs. Investigating this determined that the operation failed due to the source blob being AppendBlob instead of BlockBlob. (Note: I'm the original author that added copy_blob support).
Investigating the Azure api, I've determined that Put Blob From URL is a suitable alternative. With the caveat that my testing has shown that source blobs <= 2000MiB seem to complete normally and files greater than that fail with timeout errors. I have a working branch which switches to use that solution https://github.com/testdouble/azure-blob/compare/main...phac-nml:azure-blob:put-blob?expand=1
Though Put Blob From URL works, I think a better solution would be to entirely rewrite compose to use Put Block From URL, which is a similar to how compose currently works with multiple large files but this doesn't require downloading and reuploading blocks. I am going to investigate this solution further but wanted to capture the bug here if others happen to encounter it.
Recently we started encountering
AzureBlob::Http::Error: AzureBlob::Http::Errorwhen trying to usecomposeto copy a blob which was originally created from a call tocomposewith 2 source blobs. Investigating this determined that the operation failed due to the source blob beingAppendBlobinstead ofBlockBlob. (Note: I'm the original author that added copy_blob support).Investigating the Azure api, I've determined that Put Blob From URL is a suitable alternative. With the caveat that my testing has shown that source blobs <= 2000MiB seem to complete normally and files greater than that fail with timeout errors. I have a working branch which switches to use that solution https://github.com/testdouble/azure-blob/compare/main...phac-nml:azure-blob:put-blob?expand=1
Though
Put Blob From URLworks, I think a better solution would be to entirely rewritecomposeto use Put Block From URL, which is a similar to howcomposecurrently works with multiple large files but this doesn't require downloading and reuploading blocks. I am going to investigate this solution further but wanted to capture the bug here if others happen to encounter it.