Large File Uploads to S3

Uploading a large file to S3 works best when you treat it as a coordinated multi-step transfer rather than one giant request. The core idea is multipart upload: split the object into parts, upload the parts independently, then ask S3 to assemble them into the final object.

Why multipart upload exists

A single long upload is fragile. If a connection drops after several gigabytes, the client may need to start from zero. Large uploads also make it harder to parallelise bandwidth and harder to recover from transient failures. Multipart upload fixes that by giving each part its own request, checksum, and retry boundary.

A common production flow looks like this:

1. The client asks your backend to start an upload.
2. The backend creates a multipart upload in S3 and returns an upload ID plus presigned URLs for each part.
3. The client slices the file into parts and uploads several in parallel.
4. The client reports the part numbers and returned ETags.
5. The backend calls CompleteMultipartUpload so S3 can assemble the object.

This design keeps AWS credentials off the client while still letting the client send bytes directly to S3.

Choosing part sizes and concurrency

Part size is a tradeoff. Very small parts create too many requests and too much overhead. Very large parts reduce retry efficiency. Many teams start somewhere between 8 MiB and 64 MiB, then tune based on browser memory, mobile networks, and expected object sizes. Parallelism improves throughput, but too much parallelism can overwhelm the client, saturate weak networks, or trigger server-side throttling.

Reliability details that matter

Resumability is often the real requirement. Store the upload ID and the list of completed parts so a browser refresh or app restart does not discard progress. Checksum validation helps detect corrupted parts early. Timeouts should apply per part, not only to the overall transfer.

You also need cleanup. Incomplete multipart uploads can leave orphaned parts that cost money. S3 lifecycle rules can automatically abort old incomplete uploads after a chosen time window.

Security and correctness

Presigned URLs should be short-lived and scoped to the exact object key and operation you expect. Validate file ownership, maximum size, and content type at your backend before issuing them. If the object key encodes tenant information, generate it on the server rather than trusting the client.

Multipart upload is not just a performance feature. It is a reliability pattern for large transfers over unreliable networks. Break the upload into restartable units, keep state about progress, and finalise only when every part has been acknowledged.

In browser and mobile clients, user feedback matters as much as transport logic. Progress bars, pause or retry controls, and clear completion states reduce support issues because users can tell whether the transfer is still healthy or has stalled on one specific part.