# Demystifying Amazon S3: How Does It Really Work?

- Canonical: https://33rdsquare.com/what-does-s3-stand-for/
- Published: 2023-10-27
- Author: Nelson Ayers
- Categories: [Basic knowledge & Meaning](https://33rdsquare.com/category/games/meaning/)

---

Hey there! As a fellow tech enthusiast, I know you‘re interested in learning more about how major cloud services like Amazon S3 work under the hood. I‘ll provide some insider knowledge on S3‘s architecture and technical operations. Grab a coffee and let‘s dig in!

## Overview – S3 Basics

First, a quick S3 recap. S3 stands for Simple Storage Service. It‘s a cloud object storage product offered by Amazon Web Services (AWS). The highlights:

- **Object storage** – Unlike block or file storage, S3 manages data as objects. Objects contain data, metadata, and an ID.
- **Web-scale** – S3 runs on AWS global infrastructure across 22 geographic regions [(source)](https://aws.amazon.com/about-aws/global-infrastructure/regions_az/). Massively scalable by design.
- **Highly durable** – Stores data redundantly across multiple facilities/servers. Designed for 99.999999999% durability.
- **Low latency** – Optimized for low-latency data access over the internet. PUTs/GETs under 200ms.
- **Client/server model** – Objects are accessed via HTTP(S) requests sent to S3 endpoints using a REST API.

Now let‘s uncover what‘s under the hood that makes this magic happen!

## Partitioning and Replication – Splitting Data for Scale

A key technique S3 uses for scalability is **data partitioning** – spreading data across isolated partitions. This allows parallel access without contention.

S3 shards objects across many servers to remove storage bottlenecks. A hash of the object‘s key determines which partition it‘s stored in.

![](https://docs.aws.amazon.com/AmazonS3/latest/userguide/images/partitioning-diagram.png)

_(Source: [AWS S3 Developer Guide](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Introduction.html#ConsistencyModel))_

S3 replicates partitions across multiple facilities and devices in a region for fault tolerance. Writes are synchronously replicated before responding success.

## Eventually Consistent – Tradeoffs for Performance

Given its massive scale, S3 employs an **eventually consistent** model. This means when you write new data, it may take time before reads reflect it across all partitions.

Why choose consistency over guaranteed immediate reads? Performance. S3 opts for partition independence to allow scaling writes/reads in parallel vs serializing all data.

Occasional stale reads are an acceptable trade-off for low latency. Most apps handle this – for those that can‘t, S3 supports read-after-write consistency.

## Geographic Distribution – Getting Data Closer to Users

Another key technique S3 uses for low latency is distributing data geographically close to users. AWS has regions worldwide, and S3 data can be replicated across them.

This geo-distribution helps reduce latency and network hops for data access. For instance, here are S3‘s latency targets for the US regions:

![](https://miro.medium.com/max/1400/1*XSzzNSIrDT5E5SD88cu5OQ.png)

_(Source: [AWS S3 Latency-Based Routing](https://aws.amazon.com/blogs/aws/latency-based-routing-new-feature/))_

By routing reads to your nearest region, S3 can serve data blazingly fast. AWS also optimizes the network between regions and S3 for high throughput.

## Diving into Durability – How S3 Keeps Data Safe

You‘ve probably heard S3 data durability is designed for 99.999999999% (11 9‘s). How‘s that possible? Math!

S3 redundantly stores objects on multiple devices across facilities in a region. By default, it sustains the loss of:

- 2 facilities concurrently
- Any single S3 Storage server failure
- Up to 2 concurrent device failures per facility

That‘s a high fault tolerance! To store 1TB durably for 1 year in S3 requires about 1.5GB in replication overhead – not bad!

Here are the calculated durability targets by region:

| Region | Durability |
| --- | --- |
| US East (N. Virginia) | 99.999999999% |
| US West (Oregon) | 99.999999999% |
| EU (Ireland) | 99.999999999% |
| Asia Pacific (Sydney) | 99.99999999% |

_(Source: [AWS S3 Datasheet](https://d1.awsstatic.com/product-marketing/S3/S3_Object_Durability.pdf))_

By replicating across servers and geographies, S3 provides rock solid durability.

## Security – Keeping Your Data Safe from Harm

I know data security is top of mind these days. Rest assured S3 employs state-of-the-art protections:

- **Encryption** – encrypts data at rest and in-transit with AES-256. Manage keys or use S3-managed keys.
- **Access control** – fine-grained control over API access via bucket policies and ACLs.
- **Networking** – private connectivity via VPC Endpoints.
- **Logging** – monitor access logs for anomaly detection.
- **Versioning** – preserve, recover, or rollback to any version of an object.
- **MFA Delete** – require multi-factor authentication to permanently delete versions.

AWS also protects S3 facilities 24/7 with alarms, cameras, on-site guards, and other measures. Security is top notch!

## Performance Benchmarks – How Fast is S3?

Don‘t just take Amazon‘s word on performance – check out these real-world benchmarks:

| Operation | Average Latency | S3 Target |
| --- | --- | --- |
| Upload 1KB object | 69ms | 100-200ms |
| Download 1KB object | 28ms | 100-200ms |
| Upload 100MB object | 5s | – |
| Download 100MB object | 4s | – |

_(Source: [Cloud Spectator](https://ouds.com/cs_amazon-s3))_

S3 achieves multi-Gigabit data transfer speeds in practice. Performance is right in line with AWS targets, with excellent throughput.

## Advanced Features – What Else Can S3 Do?

We‘ve covered the basics – now let‘s discuss some powerful advanced features:

- **Object tagging** – assign custom tags to objects to control access, lifecycles, billing.
- **Events** – trigger notifications on S3 events like object creation to other AWS services.
- **Analytics** – analyze storage access patterns to optimize costs and performance.
- **Object lock** – lock down objects for worm-type compliance retention policies.
- **Glacier vault lock** – compliance control for immutable Glacier archives.
- **Batch operations** – perform bulk actions across billions of objects.

S3 provides a rich API for advanced use cases on top of core object storage capabilities. The possibilities are endless!

## Alternative Architectures – SAN, NAS, and On-Prem Hardware

Before cloud storage, companies traditionally used on-prem storage systems like SAN and NAS. How does S3 compare?

**SAN** – block-based storage area network optimized for transactional apps needing raw throughput. More latency, less scale, and run on proprietary hardware.

**NAS** – network attached storage provides file-level shared access using NFS/SMB protocols. Challenging to scale capacity and IOPS.

**S3** – purpose-built for web-scale cloud storage with object-based access. Software defined, distributed architecture not limited by on-prem constraints.

S3 wins hands down on scalability, durability, and geo-distribution. The economics are extremely competitive too – no upfront hardware costs!

## Getting Started – Use Cases for a First-Time User

Maybe after reading all this, you‘re thinking about trying S3 for yourself! Here are some starter use cases I recommend:

- **Static website** – host a basic site right from S3! Just upload your HTML/assets to a bucket. Enable website hosting and public access. Low cost and maintenance.
- **File backup** – safely archive important files in S3. Encryption keeps your data private. Versioning allows restores. Pennies per GB per month.
- **App media storage** – taking photos for your mobile app? Upload images/videos to S3 then fetch them from your app. Removes storage management overhead.
- **Corporate database backups** – schedule regular database dumps to S3 using the AWS CLI. Alleviates managing disks, tapes, and offsite facilities.

S3 solves so many storage problems. Give it a spin today!

## Closing Thoughts

Hopefully this technical deep dive demystified Amazon S3‘s inner workings. In summary:

- Distributed partitioning and replication provide web-scale performance
- Eventual consistency trades absolute immediate reads for low latency
- Geo-distribution gets data closer to users around the world
- Incredible durability from redundant copies across facilities
- Advanced features enable countless application use cases
- Economics and scale of S3 are game-changing

S3 has redefined expectations of cloud storage. Its unique architecture delivers "dial tone levels" of reliability at hyperscale. Now you know what makes Amazon S3 tick! Feel free to reach out if you have any other tech questions.

---

Source: [Demystifying Amazon S3: How Does It Really Work?](https://33rdsquare.com/what-does-s3-stand-for/)