What is object storage?

Every app ends up needing to store files: avatars, product images, PDFs, videos, exports, logs. Keeping everything on the server disk works at first, but the disk fills up, vanishes when the server is recreated and is not shared between several instances.

Object storage replaces the disk with a remote service that stores files and serves them over HTTP. It is the model behind Amazon S3 and many similar services.

What are objects, keys and buckets?#

Almost everything comes down to three concepts:

• Object: the file plus its metadata. Metadata is a set of name-value pairs, such as Content-Type, last modified date and custom fields you define.
• Key: the unique name of the object inside the bucket, for example photos/2026/cat.jpg.
• Bucket: the container for objects. This is where you set rules, such as who can access the data.

Think of object storage as a huge map: bucket + key leads to an object. There is no directory structure underneath. When a dashboard shows a photos/ folder, it is only grouping the keys that start with that prefix.

How does object storage work under the hood?#

You talk to it through an HTTP API. The basic operations are PUT (upload), GET (download), HEAD (read only the metadata), DELETE and LIST (list keys by prefix). There is no open, seek or append.

This model has practical consequences:

• Objects are replaced whole. To change an object, you write a new one in its place. The S3 documentation describes this as whole-object updates, unlike a file system, which accepts partial writes.
• It scales without capacity planning. You don't pick a volume size. The service spreads data across many servers and a bucket accepts any number of objects.
• Durability comes from replication. The service keeps copies of the data in more than one place, so a single failed disk doesn't cost you the file.
• Consistency. On S3, a read after a successful PUT returns the new data, and listings reflect the change too. Other providers may follow different rules, so check their documentation.
• Large files are uploaded in parts. Multipart upload splits the file, sends the parts in parallel and lets you retry only the part that failed. On S3 it is recommended from 100 MB. See the walkthrough in how to upload large files.

How do you upload and download an object in code?#

Most services follow the S3 API, so the same code works with many providers. The example uses boto3, the AWS SDK for Python (pip install boto3). The endpoint and credentials come from environment variables, never from the code.

import os
import boto3

s3 = boto3.client(
    "s3",
    endpoint_url=os.environ["S3_ENDPOINT"],
    aws_access_key_id=os.environ["S3_ACCESS_KEY"],
    aws_secret_access_key=os.environ["S3_SECRET_KEY"],
)

bucket = os.environ["S3_BUCKET"]

with open("cat.jpg", "rb") as f:  # the key is just text, "folders" are a prefix
    s3.put_object(
        Bucket=bucket,
        Key="photos/2026/cat.jpg",
        Body=f,
        ContentType="image/jpeg",
    )

resp = s3.list_objects_v2(Bucket=bucket, Prefix="photos/2026/")
for obj in resp.get("Contents", []):
    print(obj["Key"], obj["Size"])

s3.download_file(bucket, "photos/2026/cat.jpg", "copy.jpg")

s3.delete_object(Bucket=bucket, Key="photos/2026/cat.jpg")

Recent boto3 versions compute integrity checksums by default. If an S3-compatible provider rejects uploads because of that, set AWS_REQUEST_CHECKSUM_CALCULATION=when_required and AWS_RESPONSE_CHECKSUM_VALIDATION=when_required in the environment.

To hand a private file to a user without routing it through your server, use a presigned URL.

What is object storage used for?#

It is the natural choice for data that is written once and read many times:

• User media: avatars, photos, videos and attachments, usually served through a CDN.
• Static site assets: images, scripts and downloads.
• Backups and exports: database dumps, reports, compressed archives.
• Logs and analytics data: large files processed in batches.
• Build artifacts: installers and packages that many people download.

In practice, the app stores only the object key in the database (avatar_key) and builds the URL when needed. That keeps the database light, as the article on images in a database vs object storage explains.

When should you not use object storage?#

Object storage doesn't replace everything:

• Files that change constantly. A log that grows one appended line at a time, or a file edited in pieces, gets slow and costly because each change rewrites the whole object.
• Databases and workloads with many small writes. Access over HTTP has higher latency than a local disk.
• Programs that expect a file system. If the software needs paths, POSIX permissions and fseek, use a disk or a file volume.
• Data you query by content. To find records by filter, use a database. Object storage only finds things by key or prefix.

The full comparison with block disks and file systems is in object vs block vs file storage.

Which terms show up alongside object storage?#

• S3-compatible: a service that accepts the same API as Amazon S3. Details in S3-compatible storage.
• Egress: a fee some providers charge on data leaving the storage.
• Multipart upload: sending a large file in parts.
• Prefix: the beginning of a key, used to simulate folders.
• Metadata: name-value pairs stored with the object, such as content type and cache settings.
• Versioning: a feature of some services that keeps older versions of an object.

How to do this on Square Cloud#

On Square Cloud, an application's disk persists across restarts and deploys, but it belongs to that application alone: two applications behind the same load balancer can't see each other's files. Avatars, PDFs and videos that every instance needs to read go to Blob Storage, the platform's object storage, with an S3-compatible API (aws-cli, rclone, boto3 and the AWS SDKs), a REST API and a dashboard.

The code above runs against it with three changes: the https://s3-blob.squarecloud.app endpoint with the auto region, the S3 keys you create in the "S3 access" tab of the dashboard, and one of the account's fixed buckets, public or private (the third, legacy, is read-only). You don't create named buckets: folders are key prefixes, just as you saw above.

Each object can be up to 10 GB, and multipart is only required above 100 MB. Public files are served by a global CDN at the edge. Private files open through a presigned URL (up to 7 days), a temporary dashboard link (up to 24 hours) or a revocable share link (up to 30 days). Per-folder rules set visibility, expiration, maximum size and allowed extensions.

There is no egress or ingress fee, and the included quota ranges from 5 GB to 250 GB depending on the plan. Create the keys in the dashboard, point S3_ENDPOINT at Blob and upload your first file. Explore Blob Storage or compare quotas on Pricing.