AWS S3: Object Storage Explained

S3 is the storage most people meet first on AWS, and it is easy to use badly precisely because it is so easy to use at all. You can aws s3 cp a file up in ten seconds and never think about it again - until a bucket gets left public and leaks data, or a "folder" you deleted turns out to have been thousands of objects, or a bill balloons because terabytes of logs sat in the wrong storage class for a year. This guide builds the mental model of what S3 actually is (and is not), then walks the parts you will touch daily: how objects and keys work, the security model, encryption, versioning, lifecycle rules for cost, presigned URLs, and the CLI commands you will run over and over.

What S3 is: object storage, not a filesystem

The first thing to get right is that S3 is object storage, and object storage is a genuinely different thing from the disk on your laptop. It helps to line up the three models:

• Block storage (an EBS volume, a raw disk) - a device you format with a filesystem and mount. You read and write at the block level, one machine at a time. This is what your OS and databases want.
• File storage (NFS, EFS) - a shared hierarchical filesystem with real directories, permissions, and the ability to update a file in place. Many machines can mount it at once.
• Object storage (S3) - a flat pool of objects, each addressed by a key, reached over HTTP. There is no filesystem, no mounting, and no partial updates. You PUT a whole object and GET a whole object.

That last point is the one that trips people up. You cannot append a line to an object or edit its middle - every write replaces the entire object. There is no directory tree to cd into. What you get instead is effectively infinite capacity, access from anywhere over an API, and eleven-nines durability, at the cost of the conveniences of a filesystem. S3 is built for storing artifacts - files you write once and read many times, or write and keep - not for things you mutate constantly like a database's data files. Use block storage (EBS) for a database and file storage (EFS) for a shared mounted filesystem; reach for S3 when you want durable, cheap, API-addressable blobs.

Buckets, objects, and keys - and the flat keyspace

Everything in S3 lives in a bucket, a top-level container whose name is globally unique across all of AWS (not just your account), tied to a region. Inside a bucket you store objects, and each object has a key - its full name within the bucket - plus its data and some metadata.

Here is the part everyone gets wrong at first: the keyspace is flat. There are no real folders. When you upload something to logs/2026/07/app.log, you have not created three nested directories - you have created a single object whose key is the literal string logs/2026/07/app.log. The slashes are just characters in the key. The console draws a folder tree, and the CLI lets you ls a prefix, but that is a presentation convenience built on top of prefix matching. There is no logs/ folder that exists on its own; it exists only as long as some object's key starts with logs/.

This matters in practice. "Deleting a folder" is really deleting every object under a prefix, one delete per object. Listing is prefix-based, so s3://bucket/logs/ matches by string prefix. And an "empty folder" you see in the console is usually a zero-byte placeholder object someone created, not a directory.

# Upload an object - the key is the whole path string
aws s3 cp app.log s3://my-bucket/logs/2026/07/app.log

# List "inside" a prefix (this is prefix matching, not opening a directory)
aws s3 ls s3://my-bucket/logs/2026/07/

# List everything in the bucket, recursively
aws s3 ls s3://my-bucket --recursive

The s3://bucket/key form is the address you will use everywhere. Bucket name, then the key, and the key is the whole slash-containing string.

Durability vs availability

S3 advertises eleven nines of durability (99.999999999%) and a separate, much lower availability figure (around 99.9-99.99% depending on storage class). These are two different promises and confusing them leads to bad decisions.

• Durability is the probability your object is not lost. S3 achieves it by redundantly storing every object across multiple devices in multiple Availability Zones within the region. Eleven nines means that if you store ten million objects, you would statistically expect to lose one object roughly every ten thousand years. For practical purposes, S3 does not lose data.
• Availability is the probability you can reach your object at any given moment. It is lower because it covers transient issues - a request failing, a brief service degradation - where the data is perfectly safe but a given call might error and need a retry.

The takeaway: durability being astronomical does not mean you skip backups. S3 will not lose your object to hardware failure, but it will happily store the wrong thing if you overwrite or delete it. Durability protects against disk death; it does nothing against your own aws s3 rm or a buggy deploy. That is what versioning and cross-region replication are for, covered below.

Storage classes and lifecycle policies

Not all data is accessed equally, and paying Standard prices to store logs nobody has read in two years is how S3 bills get ugly. Storage classes let you trade retrieval speed and access cost against storage cost:

• S3 Standard - the default. Highest storage cost, instant access, no retrieval fee. For data accessed frequently.
• S3 Standard-Infrequent Access (Standard-IA) - cheaper to store, but you pay a per-GB retrieval fee and there is a minimum storage duration. For data you keep but rarely read (older backups, say). Still millisecond access.
• S3 One Zone-IA - like Standard-IA but stored in a single AZ, so cheaper and less durable against an AZ loss. Fine for reproducible data you could regenerate.
• S3 Glacier Instant Retrieval - archive pricing with millisecond access, for rarely-touched data you still need immediately when you do touch it.
• S3 Glacier Flexible Retrieval and Glacier Deep Archive - the cheapest tiers, for true archives. Retrieval takes minutes to hours and costs extra. Deep Archive is the cold storage floor - compliance data you may never read.
• S3 Intelligent-Tiering - S3 watches access patterns and moves each object between tiers automatically for a small monitoring fee. Good default when access patterns are unknown or varied.

You rarely move objects by hand. Instead you attach a lifecycle policy to the bucket - rules that transition objects to cheaper classes as they age, and expire (delete) them past some threshold. This is the single biggest cost lever S3 offers.

{
  "Rules": [
    {
      "ID": "archive-and-expire-logs",
      "Filter": { "Prefix": "logs/" },
      "Status": "Enabled",
      "Transitions": [
        { "Days": 30, "StorageClass": "STANDARD_IA" },
        { "Days": 90, "StorageClass": "GLACIER" }
      ],
      "Expiration": { "Days": 365 }
    }
  ]
}

aws s3api put-bucket-lifecycle-configuration \
  --bucket my-bucket \
  --lifecycle-configuration file://lifecycle.json

That rule keeps logs in Standard for 30 days, moves them to Infrequent Access, archives to Glacier at 90 days, and deletes them entirely after a year - all automatically, with no code. The right lifecycle policy on a high-volume bucket routinely cuts its storage bill by an order of magnitude.

Security: private by default, and how not to leak a bucket

The most important sentence in this whole guide: S3 buckets and objects are private by default. A newly created bucket is reachable by nobody except its owner. Every classic "company leaks millions of records via an open S3 bucket" story is the result of someone deliberately loosening that default, usually to make something "just work," and then forgetting.

There are two layers of access control, and it is worth keeping them straight:

• IAM policies - attached to a user, group, or role, they say "this principal may do these S3 actions on these resources." This is how you grant your own services and people access. Answers "what is this identity allowed to do?"
• Bucket policies - attached to the bucket, they say "these principals (possibly anyone) may do these actions on this bucket." This is how you grant cross-account access or, dangerously, public access. Answers "who is allowed to touch this bucket?"

Because a bucket policy can grant access to "Principal": "*" (everyone on the internet), it is the usual vector for a leak. To make that mistake much harder, AWS provides Block Public Access - a set of four switches, on by default at both the account and bucket level, that override any policy or ACL trying to make things public. Leave it on. If a tool or teammate is fighting Block Public Access to expose a bucket, that is a signal to stop and ask whether the data should really be public, and whether a presigned URL or CloudFront distribution is the right answer instead.

# Confirm Block Public Access is on (all four should be true)
aws s3api get-public-access-block --bucket my-bucket

# Turn it on explicitly if it is not
aws s3api put-public-access-block --bucket my-bucket \
  --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

The rule of thumb: grant access through IAM roles for your services, use bucket policies only for deliberate cross-account or public sharing, and keep Block Public Access on everywhere unless you have a specific, reviewed reason. If you need a deeper grounding in principals, policies, and roles, see the IAM guide - S3 access control is mostly IAM applied to a specific service.

Encryption at rest and in transit

Two separate concerns, both worth having on:

• In transit - always talk to S3 over HTTPS (the CLI and SDKs do by default). You can enforce it by adding a bucket policy that denies any request where aws:SecureTransport is false, so a plain-HTTP request is refused outright.
• At rest - S3 encrypts every object at rest by default now, and you choose how the keys are managed:
  • SSE-S3 - S3 owns and manages the encryption keys entirely. Zero configuration, nothing for you to operate. Fine for most data.
  • SSE-KMS - keys live in AWS KMS, which gives you an audit trail (every decrypt is logged in CloudTrail), key rotation, and fine-grained control over who can decrypt (KMS key policy) separately from who can read the bucket. Use this when you need auditability or want decryption gated by a key you control. It costs a little more per request and can hit KMS rate limits at very high throughput.

# Upload with SSE-S3
aws s3 cp secret.dump s3://my-bucket/backups/secret.dump \
  --sse AES256

# Upload with SSE-KMS using a specific key
aws s3 cp secret.dump s3://my-bucket/backups/secret.dump \
  --sse aws:kms --sse-kms-key-id alias/my-app-key

The practical guidance: SSE-S3 is a sane default; reach for SSE-KMS when you need to prove who can decrypt sensitive data or you need the CloudTrail audit trail. And enforce HTTPS with a bucket policy so encryption in transit is not optional.

Versioning: your undo button

By default, uploading an object with a key that already exists overwrites it, and a delete is permanent. There is no undo. Versioning changes that: once enabled on a bucket, every write keeps the old version instead of discarding it, and a "delete" just adds a delete marker on top while the real data stays underneath. You can list old versions and restore any of them.

This is the direct defense against the failure mode durability does not cover: you accidentally overwriting a good object with a bad one, or deleting something you needed. With versioning on, a fat-fingered rm or a bad deploy that clobbers a file is recoverable - you delete the delete marker or copy the prior version back.

# Turn on versioning
aws s3api put-bucket-versioning --bucket my-bucket \
  --versioning-configuration Status=Enabled

# See every version and delete marker for a key
aws s3api list-object-versions --bucket my-bucket --prefix config.json

The trade-off is cost: every version is a stored object you pay for, so a bucket with heavy overwrites can grow surprisingly fast. Pair versioning with a lifecycle rule that expires noncurrent versions after some days so old versions do not accumulate forever. Versioning is close to mandatory for buckets holding anything you would hate to lose - Terraform state and release artifacts are prime examples.

Presigned URLs: temporary access without going public

Sometimes you need to let a browser or a third party download (or upload) one specific object without giving them AWS credentials and without making the bucket public. That is exactly what a presigned URL is: a normal-looking HTTPS URL that carries a signature granting time-limited access to one object for one operation. It is generated using your credentials, it works for anyone who holds the link, and it expires.

# Generate a link that lets anyone download this object for 1 hour
aws s3 presign s3://my-bucket/reports/q2.pdf --expires-in 3600

That URL will fetch the object until it expires, then return an error. This is the correct answer to "how do I let a user download this private file?" - not making the bucket public. The classic use is a "download your export" link in an app, or letting a client upload directly to S3 without routing the bytes through your servers. Keep the expiry short, and remember that anyone who gets the link during its lifetime can use it, so treat presigned URLs like short-lived secrets.

Common DevOps uses

Where S3 actually shows up in day-to-day operations:

• Build artifacts - CI pushes compiled binaries, container image layers (via ECR, which is S3-backed), and release bundles here. Versioning plus a lifecycle rule to expire old builds keeps it tidy and cheap.
• Backups and log storage - database dumps, application logs, and audit trails land in S3, with lifecycle rules transitioning them to Glacier and expiring them on a retention schedule. Cheap, durable, and out of the way.
• Static website hosting - S3 can serve a static site (HTML, JS, CSS) directly, though in practice you put CloudFront in front for TLS, caching, and a custom domain, with the bucket kept private and reached only through the CDN.
• Terraform state - the standard remote backend for Terraform is an S3 bucket (state file) plus DynamoDB (state locking). This bucket should have versioning on (so a corrupted state write is recoverable), encryption on, and Block Public Access locked down - state files contain sensitive values.

# Sync a whole build directory up (only changed files transfer)
aws s3 sync ./dist s3://my-artifacts/app/v1.4.0/

# Pull a backup back down
aws s3 sync s3://my-backups/db/2026-07-01/ ./restore/

# Copy an object between buckets
aws s3 cp s3://src-bucket/data.json s3://dst-bucket/data.json

aws s3 sync is the workhorse for artifacts and backups - it compares source and destination and transfers only what changed, which makes repeated uploads fast and cheap.

The shape of it

S3 is object storage: a flat keyspace of durable blobs addressed by key over HTTP, not a filesystem you mount and edit in place. Objects live in globally-named buckets, and the "folders" you see are just prefixes on flat keys. Durability is astronomical but is not a backup - only versioning protects you from your own overwrites and deletes. Storage classes plus lifecycle policies are your main cost lever, aging cold data down to Glacier and expiring it automatically. Security starts from private by default: grant access through IAM, use bucket policies only for deliberate sharing, and keep Block Public Access on so you never become the next leaked-bucket headline. Encrypt at rest with SSE-S3 or SSE-KMS and enforce HTTPS in transit. When you need to hand out access to one object, generate a presigned URL rather than opening the bucket. Get those pieces right and S3 is exactly what it promises: cheap, durable, boring storage you can stop thinking about.