Object storage
S3 multipart upload explained: part size, retries, checksums and cleanup
An S3 multipart upload splits one object into numbered parts that can be transferred independently and then completed as one object. That makes failed-part retries smaller and can improve throughput, but the upload is not finished until the completion request succeeds. Amazon S3 also keeps incomplete uploaded parts until the upload is completed or aborted, so cleanup is part of the design rather than optional housekeeping.
An S3 multipart upload has a lifecycle
The client creates a multipart upload and receives an upload ID, uploads numbered parts, records the part numbers and returned ETags, and finally sends the ordered part list in a completion request. Amazon S3 assembles the completed object in ascending part-number order.
Until completion, those uploaded parts are not the final object the application expects to read. If the client crashes, the upload can remain incomplete and consume storage until it is completed or aborted.
AWS: multipart upload overview | AWS API: CompleteMultipartUpload
Know the part limits before choosing a chunk size
Amazon S3 currently documents part numbers from 1 through 10,000 and part sizes from 5 MiB through 5 GiB, with no minimum size for the last part. Its current multipart limits page lists a 48.8 TiB maximum object size. Those are Amazon S3 limits, not a promise that every S3-compatible service exposes the same ceiling.
Choose a part size that keeps the expected object comfortably below the backend's part-count limit. Larger parts reduce the number of requests and bookkeeping entries; smaller parts reduce the amount retransmitted after one failed part. Verify the exact limits of the S3-compatible service and SDK you will use.
Parallel parts can improve transfer speed
Sending several parts concurrently can use available bandwidth more effectively, especially over higher-latency links. Too much parallelism can exhaust memory, file descriptors or network capacity on the client.
Tune concurrency with representative object sizes and networks. A setting that helps a datacentre server can overwhelm a small office connection, and a provider may enforce request or connection limits before the client reaches its own ceiling.
Part retries are smaller than restarting the whole object
When one part fails, a well-behaved client retransmits only that part, so a 100 GB object does not restart from byte zero. This is one reason backup and transfer tools often use multipart operations for large archives.
Retry logic should still have limits and visible errors. Infinite retries can hide a broken credential, endpoint, network path or quota condition for hours. Record the upload ID and part state in workflows that need to resume safely after a process restart.
Do not treat the final multipart ETag as a whole-file MD5
On Amazon S3, the ETag of an object created by multipart upload is not the MD5 digest of the entire object. If end-to-end integrity matters, use the checksum mechanism supported by the exact SDK and storage backend instead of comparing the final ETag to a local whole-file MD5.
Amazon S3 currently supports full-object and composite checksum modes for multipart uploads, with algorithm-specific rules. For example, part-level composite checksums require consecutive part numbers beginning with 1. S3-compatible services can differ, so exercise the checksum mode you intend to depend on instead of assuming AWS-identical behavior.
Capacity planning must include in-progress parts
Amazon S3 bills storage for uploaded multipart parts until the upload is completed or aborted. Other providers may account for incomplete parts differently, but a client near its hard capacity limit can still fail while assembling a large object even if the final object would barely fit.
Leave operational headroom and monitor usage reported by the storage platform. Treat quota failures as a capacity event and resolve the capacity problem before retrying writes.
Abort abandoned uploads, then verify cleanup
A small SDK upload can use one PUT and never exercise multipart APIs. Force a multipart test object, verify the completed object, start a second upload, abort it and confirm its unfinished parts are removed.
Amazon S3 warns that part uploads already in flight can still succeed or fail after an abort request, so complete cleanup can require another abort and a ListParts check until the parts list is empty. Amazon S3 also supports a lifecycle action for automatically aborting incomplete multipart uploads after a configured number of days; verify that an S3-compatible backend supports equivalent lifecycle behavior before relying on it.
AWS API: AbortMultipartUpload | AWS: lifecycle cleanup for incomplete multipart uploads