Skip to content
Tutorials Center
My learning

Topics

Lesson 130 of 149 · Data and the Web

Python in the Cloud – AWS, Azure and Serverless

Use Python with cloud platforms - AWS boto3, the Azure SDK, serverless functions, retries, pagination and testing cloud code without credentials using runnable fakes.

Cloud computing means renting computers, storage and services over the internet instead of owning them. You pay for what you use, you can add capacity in minutes and the provider runs the hardware. Python is one of the main languages of the cloud: every large provider has an official Python SDK, serverless platforms run Python functions directly, and most automation and infrastructure tooling is written in it.

This lesson shows the shape of cloud code in Python: the SDKs for AWS and Azure, how serverless functions look, and the habits that keep cloud code reliable (retries, pagination, credentials and testing). The SDK calls themselves need an account and credentials, so they are read-only here, and every pattern that is plain Python is runnable with a local fake.

Warning: Cloud accounts cost money when resources run. Use the free tier, set a billing alert and delete what you create. Never paste access keys into code or commit them to a repository.

What the cloud offers a Python programmer

Service type What it gives you AWS Azure
Object storage Files of any size, addressed by key S3 Blob Storage
Virtual machines A server you control EC2 Virtual Machines
Serverless functions Code that runs per event and bills per run Lambda Functions
Managed databases Hosted SQL and NoSQL RDS, DynamoDB SQL Database, Cosmos DB
Queues and messaging Decouple parts of a system SQS, SNS Service Bus, Queue Storage
App platforms Deploy an app without managing servers Elastic Beanstalk App Service
Containers Run Docker images at scale ECS, EKS Container Apps, AKS

Google Cloud has matching services (Cloud Storage, Cloud Functions, Cloud Run) with the google-cloud-* packages.

AWS with boto3

boto3 is the official AWS SDK for Python (pip install boto3). It has two layers. A client maps one-to-one to the service's API and returns plain dictionaries. A resource is a higher-level, object-style wrapper available for some services. Clients are the most complete, so the examples use them.

Credentials

boto3 finds credentials by itself, in this order of precedence: parameters in code, environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY), the shared files ~/.aws/credentials and ~/.aws/config (created by aws configure), and, when your code runs on AWS, the IAM role attached to the machine or function. Prefer roles: they hold no secret that can leak.

Working with S3

Python
import boto3
from botocore.exceptions import ClientError

s3 = boto3.client("s3")

s3.upload_file("report.pdf", "my-example-bucket", "reports/2026/report.pdf")
s3.download_file("my-example-bucket", "reports/2026/report.pdf", "copy.pdf")

response = s3.list_objects_v2(Bucket="my-example-bucket", Prefix="reports/")
for item in response.get("Contents", []):
    print(item["Key"], item["Size"])

try:
    s3.head_object(Bucket="my-example-bucket", Key="missing.txt")
except ClientError as error:
    code = error.response["Error"]["Code"]       # for example "404" or "AccessDenied"
    print("failed with", code)

url = s3.generate_presigned_url(                  # a temporary link, no credentials needed to use it
    "get_object",
    Params={"Bucket": "my-example-bucket", "Key": "reports/2026/report.pdf"},
    ExpiresIn=3600,
)

Errors from AWS arrive as botocore.exceptions.ClientError, with the details in error.response. Most calls in the SDK follow the same style: keyword arguments with capitalized names (Bucket=, Key=), and a dictionary back.

Azure with its SDK

The Azure SDK is split into one package per service. Authentication uses azure-identity, whose DefaultAzureCredential tries environment variables, a managed identity (the Azure equivalent of an IAM role), the Azure CLI login and other sources in turn, so the same code works on your laptop and in Azure.

Python
from azure.identity import DefaultAzureCredential
from azure.storage.blob import BlobServiceClient

service = BlobServiceClient(
    account_url="https://myexampleaccount.blob.core.windows.net",
    credential=DefaultAzureCredential(),
)
container = service.get_container_client("reports")

container.upload_blob("2026/report.pdf", open("report.pdf", "rb"), overwrite=True)

for blob in container.list_blobs(name_starts_with="2026/"):
    print(blob.name, blob.size)

Install the packages azure-identity and azure-storage-blob. The concepts match AWS: a client object per service, names for containers and objects, and a credential that is found automatically.

Serverless functions

In a serverless architecture you write a function and the platform runs it when an event arrives, for example an HTTP request, a file uploaded to storage or a message in a queue. You don't manage a server. The platform starts copies when there is traffic and charges per execution, and it can scale down to zero when idle. The trade-offs are that each run is short-lived and stateless (anything you need later must go to storage or a database) and that the first call after idle time is slower (a cold start).

AWS Lambda calls a function with an event (a dictionary) and a context object:

Python
import json

def lambda_handler(event, context):
    name = (event.get("queryStringParameters") or {}).get("name", "world")
    return {
        "statusCode": 200,
        "headers": {"Content-Type": "application/json"},
        "body": json.dumps({"message": f"Hello, {name}!"}),
    }

Azure Functions (Python v2 programming model) uses decorators on a FunctionApp:

Python
import azure.functions as func

app = func.FunctionApp()

@app.route(route="hello", auth_level=func.AuthLevel.ANONYMOUS)
def hello(req: func.HttpRequest) -> func.HttpResponse:
    name = req.params.get("name", "world")
    return func.HttpResponse(f"Hello, {name}!")

Google Cloud Functions work the same way with functions-framework (@functions_framework.http).

Test cloud code without a cloud

A handler is just a function, so you can call it with a hand-made event. Better still, pass the storage client into your code instead of creating it inside. Then tests can substitute an in-memory fake that has the same method names, and nothing touches the network or your bill. The listing below does that with a fake object store and a handler that saves an uploaded note.

Python A Lambda-style handler tested with a fake S3

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In a real project you would build the handler with boto3.client("s3"), and use the moto library or botocore's Stubber in tests when you want to exercise the real client code.

Retries and pagination

Cloud calls fail in ways local code doesn't: networks drop, services throttle, and one request returns only a page of results. Two habits handle these.

  • Retry with exponential backoff and jitter. Wait a little, then longer, with some randomness so many clients don't retry at the same instant. boto3 retries many errors by itself (configurable with botocore.config.Config(retries={"max_attempts": 10, "mode": "standard"})). Only retry operations that are safe to repeat.
  • Paginate. Listing calls such as list_objects_v2 return up to 1,000 items and a continuation token. With boto3, s3.get_paginator("list_objects_v2") loops over pages for you.
Python Backoff and pagination against a flaky fake service

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Deploying and scaling

Two more words you will meet. Docker packages your app and its dependencies into an image that runs the same everywhere, and Kubernetes runs many containers across machines, restarting and scaling them. Platforms-as-a-service such as AWS Elastic Beanstalk or Azure App Service sit one level higher: you upload code and they handle servers, load balancing and scaling. The lesson on deploying Python apps covers the practical steps, and Task Queues with Celery shows background work that often runs next to cloud queues.

A few habits make cloud projects safe:

  • Give every identity least privilege: only the permissions the code needs.
  • Read configuration from environment variables or a secrets manager, not from source code.
  • Add tags and budgets so you can see what costs money.
  • Write infrastructure as code (Terraform, AWS CDK or Pulumi, which can all be written in Python) so environments are repeatable.

Check your understanding

Quick check · 5 questions

of 5 answered You scored / 5 · perfect!

  1. 1. What is the main benefit of attaching an IAM role (or managed identity) to your code's environment?

    Roles and managed identities supply short-lived credentials automatically, so there is no access key to leak.
  2. 2. How do AWS errors arrive in boto3 code?

    Service errors are raised as ClientError, and the error code is in error.response["Error"]["Code"].
  3. 3. What does serverless mean?

    Servers still exist, but you don't manage them. Functions are short-lived and stateless, so persistent data belongs in storage or a database.
  4. 4. Why pass a storage client into a handler instead of creating it inside?

    Dependency injection keeps the logic testable without credentials or costs.
  5. 5. Why add random jitter to the retry delay?

    Without jitter, clients that failed together retry together and can overload the service again.

Summary

  • Python has official SDKs for every major cloud: boto3 for AWS, the azure-* packages for Azure and google-cloud-* for Google Cloud.
  • Credentials should come from roles, managed identities or environment variables, never from source code.
  • Serverless functions (Lambda, Azure Functions) are small handlers that run per event and keep no state between runs.
  • Cloud errors, throttling and paging are normal: use retries with backoff and jitter, and paginate list calls.
  • Design code so the cloud client is passed in, and test with an in-memory fake.
  • Use least privilege, environment-based configuration and cost alerts.

Next: Python for Cybersecurity, where Python becomes a tool for inspecting networks and logs.