Blob Storage: S3, GCS, and Azure Blob Patterns for Engineers

Overview

Every team eventually hits the same wall: user uploads, backups, logs, and media start piling up on app servers or in database blobs, and suddenly storage is the bottleneck — or worse, the bill. Blob (binary large object) storage is the answer the three big clouds converged on: flat namespaces of immutable objects behind an HTTP API, priced per GB-month and per request.

AWS S3, Google Cloud Storage, and Azure Blob Storage solve the same problem with nearly identical concepts but different names, limits, and pricing quirks. This guide walks through the decisions that matter — bucket layout, access control, multipart uploads, lifecycle policies, and cost optimization — with working code for all three providers.

When to Use

Reach for blob storage when you are dealing with:

• User-generated content: avatars, documents, videos, report exports. Anything a browser uploads that your database shouldn’t hold.
• Backups and archives: database dumps, log rotation output, compliance archives that must outlive the app.
• Static assets at scale: public files served directly or through a CDN, without hitting your web tier.
• Data lake inputs: raw landing zone for ETL — JSON, Parquet, CSV that downstream jobs read in bulk.
• Cross-region artifacts: objects that must survive a region outage via replication.

When NOT to Use

Blob storage is the wrong tool when:

• You need a filesystem: objects are immutable; there’s no append, no partial write, no rename. Mounting a bucket as a drive hides this badly.
• You need low-latency key-value access: Redis or a database serves that in microseconds; a bucket serves it in tens of milliseconds.
• The data is structured and queried: use a database. “Scan the bucket and parse” is not a query engine.
• Writes are tiny and constant: per-request pricing makes millions of small PUTs surprisingly expensive — batch them.

Core Concepts

The three providers share one model with different vocabulary:

Concept	AWS S3	GCS	Azure Blob
Container	Bucket	Bucket	Container
Object path	Key	Object name	Blob name
Cheap archive class	Glacier / Deep Archive	Archive	Archive tier
Hot default class	Standard	Standard	Hot
Infrequent access	Standard-IA	Nearline / Coldline	Cool / Cold
Delegated access URL	Presigned URL	Signed URL	SAS token
Max single PUT	5 GB	5 GB	256 MiB (block blob)
Multipart threshold	5 MB+ recommended	5 MB+	100 MiB+

Three facts to internalize early:

1. The namespace is flat. reports/2025/01/ is just a key prefix, not a folder — listing “directories” is a prefix query, and it’s rate-limited.
2. Objects are immutable. “Updating” a file means uploading a new version; enable versioning if overwrites need a safety net.
3. You pay for storage AND requests. A million tiny GETs can cost more than the data itself. Storage class and request patterns both matter.

Provider Comparison

	S3	GCS	Azure Blob
Durability	11 nines	11 nines	11 nines (LRS+)
Free egress	None	None	None — egress is the real bill everywhere
Auto-tiering	Intelligent-Tiering	Autoclass	Not built-in (lifecycle rules)
Object size max	5 TB	5 TB	~190 TB (block blob)
Event notifications	EventBridge / SQS / Lambda	Pub/Sub	Event Grid
Strong consistency	Yes (since 2020)	Yes	Yes

All three are strongly consistent today — a PUT is immediately readable everywhere. Egress is where the money goes: serving 1 TB/month out of a bucket typically costs $80–120 regardless of provider, while storing it costs ~$2–25 depending on class.

Step-by-Step Blob Storage Implementation

1. Design Your Bucket Structure

Buckets are globally named and roughly free — create more than one. A layout that ages well:

myapp-uploads-prod        # user uploads, private, versioning ON
myapp-reports-prod        # generated exports, lifecycle to archive
myapp-static-assets       # public behind CDN, cache-friendly names
myapp-backups-prod        # encrypted dumps, retention-locked

Rules that save pain later:

• Prefix keys with yyyy/mm/dd/ or tenant ID — prefixes determine listing performance and make lifecycle rules easy to target.
• Never put PII or secrets in a bucket that’s also used for public assets. Split them.
• Turn on Block Public Access (S3) / uniform bucket-level access (GCS) / disable blob public access (Azure) everywhere, then open specific buckets deliberately.

# S3: block all public access at the bucket level
aws s3api put-public-access-block \
  --bucket myapp-uploads-prod \
  --public-access-block-configuration \
    "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

2. Implement Secure Access

Three access patterns cover nearly everything:

• SDK with credentials — backend services using IAM roles (not keys in env vars).
• Presigned URL — browser uploads/downloads without proxying through your API.
• Bucket policy / ACL — the dangerous one; use only for genuinely public content.

Presigned upload flow — the client gets a time-limited URL from your API and PUTs directly to storage:

# boto3 — generate a presigned PUT URL valid for 15 minutes
import boto3

s3 = boto3.client("s3")
url = s3.generate_presigned_url(
    "put_object",
    Params={
        "Bucket": "myapp-uploads-prod",
        "Key": "uploads/2025/10/report.pdf",
        "ContentType": "application/pdf",
    },
    ExpiresIn=900,
)
# Client then: requests.put(url, data=file_bytes, headers={"Content-Type": "application/pdf"})

# Azure — user delegation SAS for a 1-hour read
from datetime import datetime, timedelta, timezone
from azure.storage.blob import BlobServiceClient, generate_blob_sas, BlobSasPermissions

sas = generate_blob_sas(
    account_name="mystorageacct",
    container_name="uploads",
    blob_name="reports/q3.pdf",
    user_delegation_key=service_client.get_user_delegation_key(
        datetime.now(timezone.utc),
        datetime.now(timezone.utc) + timedelta(hours=1),
    ),
    permission=BlobSasPermissions(read=True),
    expiry=datetime.now(timezone.utc) + timedelta(hours=1),
)
url = f"https://mystorageacct.blob.core.windows.net/uploads/reports/q3.pdf?{sas}"

Gotchas: restrict presigned PUTs by ContentType and Content-Length (users will upload a 10 GB video to your avatar endpoint otherwise), keep expiry short, and prefer POST policies over plain PUTs when you need size caps enforced server-side.

One more browser gotcha everyone hits once: direct uploads need CORS on the bucket. A presigned PUT from https://yourapp.com fails with a confusing opaque error unless the bucket allows it:

[
  {
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["PUT", "POST"],
    "AllowedOrigins": ["https://yourapp.com"],
    "ExposeHeaders": ["ETag"],
    "MaxAgeSeconds": 3000
  }
]

Applied with aws s3api put-bucket-cors --bucket myapp-uploads-prod --cors-configuration file://cors.json — same idea exists as CORS rules on GCS and Azure. Scope AllowedOrigins to your real domain, never * on a private-data bucket.

3. Upload Large Files with Multipart

Single PUT tops out at 5 GB on S3/GCS and 256 MiB on Azure block blobs — and any network hiccup wastes the whole upload. Multipart splits the file into chunks uploaded independently (and in parallel), then assembled server-side. AWS and Azure retry failed parts; the whole file isn’t re-sent.

# boto3 — automatic multipart above the threshold
import boto3
from boto3.s3.transfer import TransferConfig

config = TransferConfig(
    multipart_threshold=50 * 1024 * 1024,   # 50 MB
    multipart_chunksize=25 * 1024 * 1024,   # 25 MB parts
    max_concurrency=10,
)
s3 = boto3.client("s3")
s3.upload_file("backup-2025-10.tar.gz", "myapp-backups-prod",
               "dumps/backup-2025-10.tar.gz", Config=config)

# Azure — block blob upload is chunked automatically by the SDK
from azure.storage.blob import BlobClient

blob = BlobClient.from_connection_string(
    conn_str, container_name="backups", blob_name="dumps/backup.tar.gz"
)
with open("backup.tar.gz", "rb") as f:
    blob.upload_blob(f, max_concurrency=4, overwrite=True)

Operational notes: aborted multipart uploads leave invisible orphaned parts that still bill storage — always attach a lifecycle rule that expires incomplete multipart uploads after 7 days (AbortIncompleteMultipartUpload on S3). That one rule has saved teams thousands of dollars.

4. Implement Lifecycle Policies

Objects cool down predictably: write-heavy for days, read occasionally for weeks, then effectively cold. Lifecycle rules automate the march to cheaper classes:

{
  "Rules": [{
    "ID": "reports-lifecycle",
    "Status": "Enabled",
    "Filter": {"Prefix": "reports/"},
    "Transitions": [
      {"Days": 30, "StorageClass": "STANDARD_IA"},
      {"Days": 90, "StorageClass": "GLACIER"},
      {"Days": 365, "StorageClass": "DEEP_ARCHIVE"}
    ],
    "AbortIncompleteMultipartUpload": {"DaysAfterInitiation": 7},
    "Expiration": {"Days": 2555}
  }]
}

Applied with aws s3api put-bucket-lifecycle-configuration --bucket myapp-reports-prod --lifecycle-configuration file://lifecycle.json. Azure equivalent: Management → Lifecycle management rules with tierToCool / tierToArchive actions. GCS: lifecycle rules on the bucket.

Two mistakes are expensive:

• Archiving small objects — Glacier has a 90-day minimum and a per-object metadata surcharge; objects under ~128 KB often cost MORE in archive classes. Filter rules by size or prefix accordingly.
• Retrieval shock — reading an archived object means a restore request (hours) plus retrieval fees. Archive is for disaster recovery and compliance, not “old stuff we might show users.”

5. Optimize for Cost and Performance

Lever	When it pays	Watch out for
Standard-IA / Nearline / Cool	Data read < 1x/month	Per-GB retrieval fees
Intelligent-Tiering (S3) / Autoclass (GCS)	Unknown access patterns	Monitoring fee per object
Archive tiers	Backups, compliance	Hours to restore + fees
CDN in front	Public hot assets	Cache invalidation discipline
Multipart + parallel	Files > 100 MB	Orphaned parts without lifecycle rule

For unknown or spiky access, S3 Intelligent-Tiering (~$0.0025/1000 objects monitoring fee) usually wins over guessing wrong on a class. For public-facing assets, put CloudFront/Cloud CDN/Front Door in front — a cached hit avoids both the request fee and the egress, and cuts latency by an order of magnitude for far users.

A worked example makes the trade-offs concrete. Say your app stores 500 GB of user uploads, receives 2 million PUTs/month, serves 3 million GETs/month, and pushes 800 GB of egress through a CDN:

Line item	Standard	With lifecycle + CDN
500 GB storage	~$11.50	~$5.75 (half migrated to IA after 30d)
2M PUT requests	~$10.00	~$10.00
3M GET requests	~$1.20	~$0.30 (CDN absorbs most)
800 GB egress	~$96.00	~$8.00 (egress to CDN edge is priced differently)
Total	~$118/mo	~$24/mo

The storage line is a rounding error. The bill is requests and egress — which is why “optimize blob storage costs” almost never means “find a cheaper storage class” and almost always means “put a cache in front and stop hammering the API.”

6. Wire Event Notifications

Polling a bucket on a schedule is the pattern everyone starts with and everyone regrets. All three providers emit events when objects are created, deleted, or restored — wire those into your processing pipeline instead:

S3     → EventBridge / SQS / SNS / Lambda
GCS    → Pub/Sub notifications on the bucket
Azure  → Event Grid → Function / Queue / Webhook

A typical flow: user uploads uploads/raw/report.csv → event fires → a worker validates and transforms it → writes uploads/processed/report.csv → marks the upload complete in your database. The event payload gives you bucket, key, size, and etag — enough to dispatch a job without a list call.

Two practical rules: events are at-least-once, so make consumers idempotent (the idempotent-consumer pattern covers this), and events can lag by seconds — don’t read the object metadata expecting writes to be settled if you need it instantly.

7. Replicate and Recover

For anything you’d hate to lose, turn on cross-region replication (S3 CRR, GCS dual-region, Azure GRS/RA-GRS) on the critical buckets. It replicates asynchronously in the background — new objects appear in the secondary region within minutes. Pair it with versioning so deletions and overwrites don’t replicate blindly: replication keeps the deleted copy only if you configure delete-marker replication deliberately.

Then do the part everyone skips: a quarterly restore drill. Pick a random object from the archive class, restore it, time it, and write the number down next to your RTO. The drill costs almost nothing and is the only proof that the backup exists in more than name.

What Works

• Versioning on write-once buckets: accidental overwrites and ransomware both hurt less when old versions survive. Pair with lifecycle rules to expire old versions or the bill doubles.
• Server-side encryption by default: SSE-S3 / Google-managed / Azure encryption are free and always on; bring your own KMS key only when compliance asks. There is zero reason to leave any bucket unencrypted — and equally little reason to pay KMS request fees on millions of tiny objects when a bucket key exists for exactly that case.
• Object tags for cost allocation: cost-center, env, data-class tags make the bill explainable and feed lifecycle filters.
• Event notifications for pipelines: S3 → Lambda, GCS → Pub/Sub, Azure → Event Grid. Trigger processing on upload instead of polling buckets on a schedule.
• Test restores quarterly: the backup you can’t restore in your RTO is a donation to your cloud bill.
• Name files for machines, not humans: report-2025-10-28-a3f9.csv sorts correctly, deduplicates naturally, and survives URL encoding. Final Report (2) FINAL.pdf does none of those things — put the human-readable name in object metadata where it belongs.

Common Mistakes

1. Public buckets by accident: one wildcard bucket policy or ACL and your upload bucket is on the news. Keep Block Public Access on; audit with aws s3control / Access Analyzer regularly.
2. Presigned URLs with no constraints: no ContentType/size limits means your users’ browser can upload anything anywhere under that key.
3. Orphaned multipart parts: aborted uploads bill invisibly until you add AbortIncompleteMultipartUpload.
4. Lifecycle rules on tiny objects: sub-128 KB objects cost more archived than in Standard.
5. Ignoring egress math: moving 5 TB/month out of cloud storage costs more than storing 200 TB. Design for same-region reads or a CDN.
6. PII in filenames: object keys leak into logs, URLs, and CDN analytics — never put user data in the key itself.
7. Replication without versioning discipline: enabling cross-region replication on a versioned bucket also replicates every old version — that’s a second copy of every edit you’ve ever made, billed at the destination’s class and storage rate. Scope replication rules by prefix (backups/, not the whole bucket) and by tag where possible, or you’ll pay double for data nobody will ever read.
8. Trusting the SDK defaults silently: default retry counts and timeouts differ wildly between providers and SDK versions. A retry storm against a throttled bucket makes outages worse, not better — cap retries, add jittered backoff (the retry and backoff recipe covers the math), and log retry counts so you can actually see the storms happening.

Variants

Scenario	Recommended shape
High-volume uploads from browsers	Presigned POST + direct-to-bucket + CDN read path
Data lake landing zone	Raw/clean/curated buckets, Parquet, partitioning by date
Compliance archive (WORM)	Object Lock / Bucket Lock retention + archive class
Hybrid / on-prem	S3 API-compatible (MinIO, Ceph) or provider hybrid (Storage Gateway, Azure File Sync)
Multi-cloud redundancy	Cross-region replication on one provider beats dual-provider writes in almost every case

Further Reading

• Amazon S3 documentation — storage classes, lifecycle rules, presigned URLs
• Google Cloud Storage docs — Autoclass, signed URLs, transfer options
• Azure Blob Storage docs — access tiers, SAS tokens, lifecycle management
• S3 pricing — the egress and request-fee math worth reading twice

See Also

• Data Migration Guide — moving large datasets between storage systems
• Multi-Cloud Guide — when dual-provider storage does and doesn’t make sense
• Secrets Management Guide — keeping cloud credentials out of code
• Caching Strategies — CDN behavior in front of object storage
• Runnable examples — presigned URLs, multipart upload, and lifecycle config for the three providers