Abstract
- At a high level, multipart upload allows us to upload in parallel, increasing overall throughput. However, it incurs more operations since we need to use
CreateMultipartUpload
,UploadPart
, andCompleteMultipartUpload
instead ofPutObject
in the case of S3
Size of each part is important for cost savings
if it’s too small, more API calls are needed to initiate the upload.
If it’s too large, retries become more expensive if the upload fails and may lead to higher data usage.
multipart download
GET /my-bucket/my-large-file HTTP/1.1
Host: s3.amazonaws.com
Range: bytes=0-999999
- The above retrieves the first 1,000,000 bytes of the file
- Multipart download can be achieved by issuing multiple requests, each using a separate thread with a different byte range, and merge the parts after downloading