Uploading large files: multipart and resumable uploads

Sending a 4 GB video in a single HTTP request works until the Wi-Fi hiccups at 97%. The upload drops and you start over from zero. A server receiving that file also has to hold a connection open for a long time, with proxy timeouts and request body limits along the way.

Multipart upload fixes this at the root: the file becomes several smaller parts, each with its own request. The numbers below come from Amazon S3, which sets the pattern almost every S3-compatible storage follows; check the current limits in the official documentation.

How does a multipart upload work?#

The S3 documentation describes three steps:

1. Initiate (CreateMultipartUpload). The storage returns an UploadId, which identifies this upload in every later call. Object metadata goes in this call.
2. Upload the parts (UploadPart). Each part has a number from 1 to 10,000. The storage answers with an ETag for each one. You can send them in any order and in parallel.
3. Complete (CompleteMultipartUpload). You send the list of part numbers and their ETags. The storage joins the parts in number order and creates the object.

Uploading a part with a number already used overwrites the earlier one. That is how you resend a failed part without touching the others.

What are the multipart limits on S3?#

Two calculations follow. With at most 10,000 parts, a 100 GB file needs parts of at least 10 MB. And 5 MiB parts cap the file at about 48.8 GiB (10,000 × 5 MiB). Compute the part size as file_size / 10,000, rounded up, and never below 5 MiB.

Other S3-compatible storage may have different limits. Check your provider's documentation before fixing a part size.

How do you upload a large file in Python?#

The shortest path is boto3's upload_file, which decides by itself when to use multipart and sends parts in threads. Install it with pip install boto3.

import os

import boto3
from boto3.s3.transfer import TransferConfig

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ.get("S3_ENDPOINT"),  # omit on AWS S3
    aws_access_key_id=os.environ["S3_ACCESS_KEY"],
    aws_secret_access_key=os.environ["S3_SECRET_KEY"],
)

config = TransferConfig(
    multipart_threshold=64 * 1024 * 1024,  # default: 8 MiB
    multipart_chunksize=64 * 1024 * 1024,  # default: 8 MiB per part
    max_concurrency=10,                    # default: 10 threads
)

s3.upload_file("video.mp4", "my-bucket", "videos/video.mp4", Config=config)

The TransferConfig defaults are 8 MiB for both threshold and part size, and 10 threads. On a stable connection, larger parts mean fewer requests. On an unstable one, smaller parts lose less on each failure.

To verify, run aws s3 ls s3://my-bucket/videos/ --human-readable (add --endpoint-url for other storage) and compare the size with your local file.

How do you resume an interrupted upload?#

upload_file does not resume by itself: if the process dies, the parts already sent stay in the storage, but the next upload_file starts a new upload. To resume, use the low-level calls and write the upload ID and each part's ETag to a local file as the parts are accepted.

import json
import os

import boto3

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ.get("S3_ENDPOINT"),
    aws_access_key_id=os.environ["S3_ACCESS_KEY"],
    aws_secret_access_key=os.environ["S3_SECRET_KEY"],
)

BUCKET, KEY, PATH = "my-bucket", "videos/video.mp4", "video.mp4"
PART = 64 * 1024 * 1024  # 64 MiB: covers files up to ~625 GiB in 10,000 parts
STATE = PATH + ".upload.json"  # upload ID and ETags, saved after every part

if os.path.exists(STATE):
    with open(STATE) as f:
        state = json.load(f)
else:
    upload_id = s3.create_multipart_upload(Bucket=BUCKET, Key=KEY)["UploadId"]
    state = {"upload_id": upload_id, "parts": {}}


def save():
    with open(STATE, "w") as f:
        json.dump(state, f)


save()
done = state["parts"]  # part number (a string in JSON) -> ETag

with open(PATH, "rb") as f:
    number = 1
    while chunk := f.read(PART):
        if str(number) not in done:
            r = s3.upload_part(
                Bucket=BUCKET, Key=KEY, UploadId=state["upload_id"],
                PartNumber=number, Body=chunk,
            )
            done[str(number)] = r["ETag"]
            save()
            print("part", number, "uploaded")
        number += 1

s3.complete_multipart_upload(
    Bucket=BUCKET, Key=KEY, UploadId=state["upload_id"],
    MultipartUpload={
        "Parts": [
            {"PartNumber": int(n), "ETag": e}
            for n, e in sorted(done.items(), key=lambda item: int(item[0]))
        ]
    },
)
os.remove(STATE)
print("done")

If the script dies midway, run it again: it reads the state file, skips the parts already recorded and sends only the missing ones. The S3 documentation says to complete with your own list of part numbers and ETags; list_parts is only for checking what reached the storage. PART must be the same in both runs, or the old parts will not line up. The code reads one part at a time, so memory use is one part, not the whole file.

The example uploads the parts sequentially to keep the code readable. For parallelism, use the upload_file from the previous example or a ThreadPoolExecutor calling upload_part.

How do you upload from the browser without going through your server?#

The usual pattern is for your server to sign one presigned URL per part, and for the browser to send each part straight to the storage. On the server, after create_multipart_upload:

url = s3.generate_presigned_url(
    "upload_part",
    Params={"Bucket": BUCKET, "Key": KEY, "UploadId": upload_id, "PartNumber": 1},
    ExpiresIn=3600,
)

In the browser, slice the file with file.slice(start, end) and send each slice with fetch(url, { method: "PUT", body: slice }). The response carries the ETag in the ETag header. For JavaScript to read it, the bucket's CORS configuration must expose that header (ExposeHeaders). At the end, the browser sends the list of ETags back to your server, which calls complete_multipart_upload.

When should you use the tus protocol?#

S3 multipart resumes because the storage keeps the parts. tus is another route: an open protocol over HTTP (version 1.0.0) in which the server remembers how many bytes it received. The client asks with HEAD and continues with PATCH:

HEAD /files/24e533e02ec3bc40c387f1a0e460e216 HTTP/1.1
Tus-Resumable: 1.0.0

HTTP/1.1 200 OK
Upload-Offset: 70
Upload-Length: 100
Cache-Control: no-store
Tus-Resumable: 1.0.0

PATCH /files/24e533e02ec3bc40c387f1a0e460e216 HTTP/1.1
Tus-Resumable: 1.0.0
Upload-Offset: 70
Content-Type: application/offset+octet-stream
Content-Length: 30

[remaining 30 bytes]

If the Upload-Offset sent does not match the server's, it answers 409 Conflict. Pick tus when you already run an upload server and want to resume from any client. For sending straight to storage, multipart with presigned URLs removes the need for that server.

Which mistakes should you avoid?#

• Parts smaller than 5 MiB. On S3, every part except the last must be at least 5 MiB.
• Losing the ETag or part number. complete needs the full list. Record each pair as soon as the part is accepted.
• Changing the part size when resuming. Old parts no longer line up with the new ones.
• Leaving incomplete uploads behind. On S3, parts stay stored and billed until the upload is completed or aborted. Call abort_multipart_upload when you give up, and set a lifecycle rule with AbortIncompleteMultipartUpload (for example, DaysAfterInitiation: 7). See also object lifecycle rules.
• Expecting the ETag to be the file's MD5. On multipart objects, the ETag is not necessarily the MD5 of the data. To check integrity, use S3 checksums.
• Using multipart for small files. Each part is one more request. For small files, a plain PUT does the job in a single request.

How to do this on Square Cloud#

On Square Cloud Blob Storage, the 4 GB video from the opening fits with room to spare: each object can be up to 10 GB. Multipart is available and only required above 100 MB per PUT; below that, a simple PUT does the job. Since the API is S3-compatible, both scripts in this article run without changing their logic: set S3_ENDPOINT=https://s3-blob.squarecloud.app, AWS_DEFAULT_REGION=auto and the S3 keys you create in the "S3 access" tab of the dashboard, then replace my-bucket with public or private, the account's fixed buckets you write to.

The example's 64 MiB multipart_threshold already sits below 100 MB, so every file that requires multipart goes through it, and the resume script uses the same multipart calls Blob accepts. The part limits in the table are Amazon S3's; on Blob, what changes is the 10 GB ceiling on the final object. For browser uploads of up to 100 MB, a single presigned PUT URL is enough: it lasts up to 7 days, and Blob's CORS accepts PUT from any origin.

There is no ingress or egress fee, and each plan includes a quota from 5 GB to 250 GB. Create the keys in the dashboard and run the resume script on one of your files, stopping it halfway to watch the uploaded parts get skipped. Explore Blob Storage and check each plan's quota on Pricing.