Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 70 ยท Storage with S3

S3 Buckets and Objects

Learn how Amazon S3 stores data: buckets, object keys, prefixes that look like folders, strong consistency, multipart uploads, presigned URLs, the CLI and boto3.

Amazon S3 (Simple Storage Service) is object storage: you hand it a blob of bytes and a name, and it keeps that blob safe and gives it back when you ask. There are no disks to size, no file systems to mount and no capacity to plan. S3 holds website assets, backups, logs, data lakes, machine learning datasets and the uploads of countless applications, and many other AWS services read from or write to it.

This lesson covers the object model that everything else in S3 builds on. The real service cannot run on this site, so all CLI commands and SDK code below are read-only. Two Python emulators let you experiment with the ideas: a tiny object store that behaves like S3 for listing and ETags, and a model of multipart uploads and presigned URLs. Both are teaching stand-ins, not the real service.

Buckets and objects

A bucket is a named container for objects. An object is the data itself plus metadata. Three facts about buckets matter early:

  • A bucket lives in one Region. The data stays there unless you replicate it.
  • A bucket name must be globally unique across all AWS accounts in the same partition, 3 to 63 characters, lowercase letters, digits, hyphens and dots. You cannot rename a bucket later, so pick names you can live with, for example acme-prod-invoices-eu-west-1.
  • A bucket has no size limit and you pay for what you store, not for what you might store.

Each object is identified by its key, a string of up to 1,024 bytes of UTF-8. An object holds up to 5 TB. Besides the body, an object carries system metadata (content type, size, last modified, ETag), optional user metadata (x-amz-meta-* headers), and up to ten tags.

S3 is a flat namespace. There is a bucket, and inside it a set of keys. The key reports/2026/march/summary.csv is one string; there is no reports directory object that contains a 2026 directory. The console draws folders for you because you can ask S3 to group keys by a delimiter, usually /. The part of a key before the delimiter is called a prefix.

flowchart TD
    B["Bucket<br/>acme-prod-invoices"] --> K1["Key: reports/2026/march/summary.csv"]
    B --> K2["Key: reports/2026/april/summary.csv"]
    B --> K3["Key: logo.png"]
    K1 --> O1["Object: bytes + metadata"]

A bucket is a flat map from keys to objects; slashes in keys only look like folders.

This flatness has practical consequences. Renaming a "folder" means copying every object to a new key and deleting the old ones. An "empty folder" is either nothing at all or a zero-byte object whose key ends in / that the console creates for you. And listings are always sorted by key in binary order, so a well-chosen prefix scheme (tenant-id/date/file) lets you list one tenant or one day cheaply.

Working with objects

The AWS CLI has two layers for S3. aws s3 offers friendly commands, and aws s3api maps one to one onto the API operations. All of the following is read-only here, because it needs an account:

Terminal
# Create a bucket in a Region
aws s3 mb s3://acme-prod-invoices-eu-west-1 --region eu-west-1

# Upload, download, list
aws s3 cp invoice-1001.pdf s3://acme-prod-invoices-eu-west-1/2026/03/invoice-1001.pdf
aws s3 ls s3://acme-prod-invoices-eu-west-1/2026/03/
aws s3 cp s3://acme-prod-invoices-eu-west-1/2026/03/invoice-1001.pdf ./copy.pdf

# Mirror a folder, copying only what changed
aws s3 sync ./site s3://acme-prod-invoices-eu-west-1/site/

# Low-level equivalent of "ls": returns JSON, at most 1,000 keys per call
aws s3api list-objects-v2 --bucket acme-prod-invoices-eu-west-1 --prefix 2026/ --delimiter /

The same operations in Python with boto3, the AWS SDK for Python:

Python
import boto3

s3 = boto3.client("s3")
s3.put_object(
    Bucket="acme-prod-invoices-eu-west-1",
    Key="2026/03/invoice-1001.pdf",
    Body=open("invoice-1001.pdf", "rb"),
    ContentType="application/pdf",
)

obj = s3.get_object(Bucket="acme-prod-invoices-eu-west-1", Key="2026/03/invoice-1001.pdf")
data = obj["Body"].read()

# A paginator hides the 1,000-key page limit
paginator = s3.get_paginator("list_objects_v2")
for page in paginator.paginate(Bucket="acme-prod-invoices-eu-west-1", Prefix="2026/"):
    for item in page.get("Contents", []):
        print(item["Key"], item["Size"])

Behind every one of these calls is a signed HTTPS request, as you saw in Console, CLI and SDKs. Access to the objects is controlled by IAM and bucket policies, which S3 Security covers.

Consistency and the ETag

Since December 2020, S3 provides strong read-after-write consistency for all operations in all Regions, at no extra cost. After a successful PUT of a new object or an overwrite, the next GET or LIST sees the new data. A successful DELETE makes the object disappear from the next GET and LIST. Old guidance about "eventual consistency" for overwrites and listings is out of date.

Two caveats remain. Writes to the same key from different clients are last-writer-wins unless you use conditional writes (If-None-Match, If-Match) or versioning. And S3 does not support partial updates: to change one byte, you upload the whole object again.

Every object has an ETag, an identifier for a specific version of the content. For an object uploaded in a single PUT without KMS or customer-provided keys, it is the MD5 hash of the bytes. For multipart uploads it is something different, as the second emulator shows.

The first stand-in is a toy object store with S3-like semantics for put, get, head, delete and prefix and delimiter listing.

Python A toy object store with prefix and delimiter listing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that reports/ never exists as an object. The "folders" are just the distinct prefixes that appear before the next delimiter. The real ListObjectsV2 returns exactly this split as Contents and CommonPrefixes.

Multipart uploads

A single PUT can carry up to 5 GB, but for anything over about 100 MB you should use multipart upload. The client splits the file into parts (5 MiB to 5 GiB each, except the last part, which may be smaller; at most 10,000 parts), uploads them in parallel, and then asks S3 to assemble them. Benefits: failed parts can be retried alone, uploads can be resumed, and throughput improves. The CLI and SDKs do this automatically above a threshold. The API has three steps: CreateMultipartUpload, UploadPart (many times) and CompleteMultipartUpload.

One trap: if you never complete or abort an upload, the uploaded parts stay in the bucket, invisible in normal listings but billed. A lifecycle rule that aborts incomplete multipart uploads after a few days is standard hygiene (Storage Classes and Lifecycle Rules shows how).

Presigned URLs

Buckets are private by default. A presigned URL lets you grant temporary access to one object without sharing credentials: your code signs the request for GET or PUT of a given key and an expiry time, and anyone holding the URL can perform just that action until it expires. This is how apps let a browser upload a photo directly to S3 without routing the bytes through a server.

Python
url = s3.generate_presigned_url(
    "get_object",
    Params={"Bucket": "acme-prod-invoices-eu-west-1", "Key": "2026/03/invoice-1001.pdf"},
    ExpiresIn=900,          # 15 minutes
)

The URL carries the signature, so it is only as safe as the channel you send it over. It is also only as long-lived as the credentials that signed it: if you sign with temporary role credentials, the URL stops working when those credentials expire, even if ExpiresIn is longer.

The second stand-in models multipart ETags and signed, expiring links. The signature here is a plain HMAC over the request, which shows the idea of signing; the real AWS Signature Version 4 is more elaborate.

Python Multipart ETags and an expiring signed link

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The multipart ETag is the MD5 of the concatenated part digests, followed by a dash and the part count. That is why you cannot verify a large upload by comparing its ETag with the MD5 of the file, and why S3 offers additional checksum algorithms (CRC32, CRC32C, SHA-1, SHA-256) for end-to-end integrity checking.

Practical guidance

  • Design keys for the way you list and process data, not for how folders look. Put the stable, selective part first.
  • S3 scales request rates per prefix: at least 3,500 write and 5,500 read requests per second for each prefix, and you can spread load across several prefixes. The old advice to add random characters to every key is no longer needed.
  • Avoid personal data in key names. Keys appear in logs, URLs and events.
  • Enable versioning if overwrites or deletes by mistake would hurt. It is covered in the next lesson.
  • Do not use S3 like a database. It is perfect for whole objects and poor for tiny partial updates and queries.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is true about the key reports/2026/summary.csv in an S3 bucket?

    S3 has a flat namespace. Prefixes and delimiters let you group keys the way folders would, but no directory objects exist.
  2. 2. You overwrite an existing object and immediately read it. What do you get today?

    Since December 2020, S3 is strongly consistent for puts, overwrites, deletes and listings in every Region.
  3. 3. Why use multipart upload for a 20 GB file?

    A single PUT tops out at 5 GB. Multipart uploads split the file into parts for parallelism, resumability and per-part retries.
  4. 4. A presigned URL was created with temporary role credentials and ExpiresIn of 7 days. When does it stop working at the latest?

    A presigned URL is valid only while the signing credentials are valid, so short-lived role credentials shorten its life.

Summary

  • A bucket is a Region-bound, globally uniquely named container; an object is a key plus bytes plus metadata, up to 5 TB.
  • The namespace is flat: prefixes and delimiters create the illusion of folders.
  • S3 is strongly consistent for reads after writes, overwrites, deletes and listings.
  • Use multipart upload for large files and abort stale uploads; the multipart ETag is not the file's MD5.
  • Presigned URLs grant temporary, single-purpose access without sharing credentials.

Next: Storage Classes and Lifecycle Rules.