Object lifecycle rules: expiring and cleaning up files automatically

Every app that accepts files accumulates junk: temporary thumbnails, exports a user downloads once, logs, attachments from abandoned drafts. If nobody deletes them, storage grows and so does the bill. A cron job that lists and deletes objects works, but it is one more piece of code to maintain and one more thing that can fail.

Lifecycle rules solve this inside the storage itself. You declare the policy once and it applies to every object that matches the filter, existing and new. The examples below use the Amazon S3 API, the de facto standard that many compatible storage services also accept.

What is a lifecycle rule?#

A rule has three parts: a filter that picks the objects, an action and a time period. The most common action is expiration (delete). Services with storage classes also offer transition (move to a cheaper class).

On S3 each rule has an ID (up to 255 characters), a status (Enabled or Disabled) and a filter by key prefix, by tags, by object size or a combination. An empty filter matches the whole bucket. The prefix is what you think of as a folder: tmp/, exports/, logs/.

When does the rule run?#

There is no exact time. On S3 the number of days is added to the object's creation time (which is its last modified time) and the result is rounded up to the next midnight UTC. A file created on January 15 at 10:30 UTC with a 3-day rule becomes eligible on January 19 at 00:00 UTC. The removal itself happens afterwards, asynchronously.

In practice, use expiration for cleanup, not for security. If a link must stop working in 10 minutes, use a short-lived presigned URL and treat the lifecycle rule as housekeeping only.

How do you create an expiration rule with boto3?#

Install with pip install boto3 and read credentials and endpoint from environment variables. The example expires everything under tmp/ after 7 days and aborts multipart uploads stalled for 3 days.

import os

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    region_name="us-east-1",
    endpoint_url=os.environ.get("S3_ENDPOINT"),  # omit for AWS; set for S3-compatible storage
    aws_access_key_id=os.environ["S3_ACCESS_KEY"],
    aws_secret_access_key=os.environ["S3_SECRET_KEY"],
    config=Config(signature_version="s3v4"),
)

s3.put_bucket_lifecycle_configuration(
    Bucket=os.environ["S3_BUCKET"],
    LifecycleConfiguration={
        "Rules": [
            {
                "ID": "expire-tmp-after-7-days",
                "Status": "Enabled",
                "Filter": {"Prefix": "tmp/"},
                "Expiration": {"Days": 7},
            },
            {
                "ID": "abort-stalled-multipart",
                "Status": "Enabled",
                "Filter": {"Prefix": ""},
                "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 3},
            },
        ]
    },
)

Careful: put_bucket_lifecycle_configuration replaces the whole configuration. To add a rule, read the current ones with get_bucket_lifecycle_configuration, append the new one and send the full list. If you send only the new rule, the others disappear.

The same with the AWS CLI, using a lifecycle.json file containing {"Rules": [...]}:

aws s3api put-bucket-lifecycle-configuration \
  --bucket "$S3_BUCKET" \
  --endpoint-url "$S3_ENDPOINT" \
  --lifecycle-configuration file://lifecycle.json

What about versioning and incomplete uploads?#

Two cases trip up a lot of people.

Versioned buckets. The Expiration action only touches the current version: it adds a delete marker and the old version becomes noncurrent, still taking up space. To remove it for good, add NoncurrentVersionExpiration with NoncurrentDays. That period is your recovery window for accidental deletes. You can use NewerNoncurrentVersions (1 to 100) to keep the latest N versions, but then the rule must include a Filter element.

Abandoned multipart uploads. An expiration rule does not remove parts of an upload that never finished, and those parts keep using space. AbortIncompleteMultipartUpload fixes that, as in the example above, but it cannot be used in rules with a tag filter. See more in how to upload large files.

How do you verify the rule works?#

1. Read the configuration back: s3.get_bucket_lifecycle_configuration(Bucket=...) should return the rules with Status: Enabled.
2. Before enabling in production, list the prefix with list_objects_v2(Prefix="tmp/"): that is exactly the set the rule would hit.
3. Upload a test object under tmp/, wait out the period (use 1 day on a test bucket) and confirm it is gone.
4. Confirm that a folder with no rule, such as invoices/, was not affected.

What are the most common mistakes?#

• An empty filter by accident. Filter: {"Prefix": ""} matches the whole bucket. On an expiration rule, that deletes every object older than the period.
• Forgetting the trailing slash. tmp also matches tmp-final/. Use tmp/.
• Date-based rules. An action with Date does not run just once: it keeps applying to new objects while the rule is enabled. Prefer Days.
• Replacing the configuration without reading the old one. See the warning above.
• Using expiration where you need retention. For database copies, choose the period deliberately, as in backups to object storage.

How to do this on Square Cloud#

On Square Cloud Blob Storage, the example rule (delete what sits in tmp/ after 7 days) doesn't go through the S3 lifecycle API: it becomes a "delete after N days" folder rule that you set in the dashboard. The count starts from each file's last write, and periods under 7 days are only available on Enterprise; on the other plans, the example's 7 days are already the minimum. The S3 endpoint (https://s3-blob.squarecloud.app, region auto) covers object operations: List, Head, Get, Put, Copy, Delete, multipart and presigned URLs.

The same folder takes other rules that round out the cleanup: visibility, cache, maximum size and allowed extensions. On each object, you can also set expiration and Cache-Control. The advice from the section on when the rule runs still applies: for a link that must stop working soon, use a presigned URL or the dashboard's temporary link of up to 24 hours, and keep the folder rule for cleanup only.

There is no egress or ingress fee, and the included quota runs from 5 GB to 250 GB depending on the plan; deleting what is temporary is what keeps you within it. To start, set the 7-day rule on the tmp/ folder in the dashboard. Explore Blob Storage and compare quotas on Pricing.