How to store files in S3 from Python and JavaScript

Steps

1. Create a bucket and an access key

   Do it in the console of your storage service. Give the key access to this one bucket only: a leaked key then reaches one bucket, not the account.

2. Put the key in the environment

   Both SDKs read these two variables, so the key stays out of your source files. Boto3 also reads ~/.aws/credentials.

   $ export AWS_ACCESS_KEY_ID=your-key-id
   $ export AWS_SECRET_ACCESS_KEY=your-secret

3. Upload from Python

   Install boto3 in a virtual environment. upload_file takes a path; upload_fileobj takes a file opened in binary mode; put_object sends a small body in one request.

   $ python3 -m venv .venv && source .venv/bin/activate
   $ pip install boto3

   import boto3
   
   s3 = boto3.client("s3", region_name="eu-west-1")
   
   # a file from disk
   s3.upload_file("report.pdf", "my-bucket", "reports/report.pdf")
   
   # a file object, opened in binary mode
   with open("report.pdf", "rb") as f:
       s3.upload_fileobj(f, "my-bucket", "reports/copy.pdf")
   
   # a small body in one request
   s3.put_object(
       Bucket="my-bucket",
       Key="notes/hello.txt",
       Body=b"hello",
       ContentType="text/plain",
   )

4. Upload from JavaScript

   The AWS SDK for JavaScript v3 sends one command per call. Use an ES module (.mjs) so import and top-level await work.

   $ npm install @aws-sdk/client-s3

   import { readFileSync } from "node:fs";
   import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
   
   const client = new S3Client({ region: "eu-west-1" });
   
   await client.send(
     new PutObjectCommand({
       Bucket: "my-bucket",
       Key: "reports/report.pdf",
       Body: readFileSync("report.pdf"),
       ContentType: "application/pdf",
     }),
   );

5. Send big files in parts

   In Python, upload_file splits a large file into chunks and uploads them in parallel (a multipart upload). In JavaScript, a single PutObjectCommand sends everything at once. For a big file or a stream, use Upload from @aws-sdk/lib-storage. It does multipart under the hood: parts of at least 5 MB, 4 parts at a time by default, and it aborts a failed upload unless leavePartsOnError is true.

   $ npm install @aws-sdk/lib-storage

   import { createReadStream } from "node:fs";
   import { S3Client } from "@aws-sdk/client-s3";
   import { Upload } from "@aws-sdk/lib-storage";
   
   const client = new S3Client({ region: "eu-west-1" });
   
   const upload = new Upload({
     client,
     params: {
       Bucket: "my-bucket",
       Key: "backups/big.bin",
       Body: createReadStream("big.bin"),
     },
   });
   upload.on("httpUploadProgress", (p) => console.log(p.loaded, "bytes"));
   await upload.done();

6. Point the client at another S3-compatible service

   For storage that is not Amazon's, add the service's address. In Python, endpoint_url is the complete URL to use for the client, scheme included. In JavaScript, pass endpoint, and add forcePathStyle: true if the service wants path-style URLs.

   # Python
   s3 = boto3.client("s3", endpoint_url="https://s3.example.com", region_name="us-east-1")
   
   // JavaScript
   const client = new S3Client({
     region: "us-east-1",
     endpoint: "https://s3.example.com",
     forcePathStyle: true,
   });

Check that it works

List the bucket and download one file back. A key you uploaded shows up with the size you expect. We ran the Python and JavaScript upload code on this page against a local S3 test server, including a 20 MB file in parts.

for obj in s3.list_objects_v2(Bucket="my-bucket")["Contents"]:
    print(obj["Key"], obj["Size"])

s3.download_file("my-bucket", "reports/report.pdf", "back.pdf")

To hand a file to someone who has no key, make a temporary link. It stops working after ExpiresIn seconds. In JavaScript, getSignedUrl from @aws-sdk/s3-request-presigner does the same: getSignedUrl(client, command, { expiresIn: 3600 }); the default is 900 seconds, the S3 maximum is 7 days.

url = s3.generate_presigned_url(
    "get_object",
    Params={"Bucket": "my-bucket", "Key": "reports/report.pdf"},
    ExpiresIn=3600,
)

Upload to Uno with an S3 key

Make a key for one bucket: on the Storage page in the console, or with POST /api/v1/buckets/{id}/keys and a permission of read_write, read_only or write_only. The answer carries access_key_id, secret_access_key, endpoint and bucket. The secret is shown once, so put it in your environment right away. One key opens exactly one bucket, and deleting the key cuts it off at once.

export AWS_ACCESS_KEY_ID=...        # access_key_id
export AWS_SECRET_ACCESS_KEY=...    # secret_access_key
export AWS_DEFAULT_REGION=auto
aws s3 cp report.pdf s3://<bucket>/reports/ --endpoint-url https://s3.uno.place

s3 = boto3.client("s3", endpoint_url="https://s3.uno.place", region_name="auto")
s3.upload_file("report.pdf", "<bucket>", "reports/report.pdf")

In JavaScript pass endpoint: "https://s3.uno.place", region: "auto" and forcePathStyle: true. Uploads, downloads, listing, copy, delete, multipart and presigned GET and PUT work. Creating buckets, ACLs and versions do not: make buckets through the API or the console.

Upload to Uno with a presigned link

Your code makes a bucket, asks the Uno API for a link, and sends the file to the link. POST /api/v1/buckets with a name creates the bucket, and GET /api/v1/buckets lists yours with their ids. POST /api/v1/buckets/{id}/presign with a key and a method of put or get returns a url. A link lives up to 3,600 seconds. Make an API key in the console (it reaches sites, computers, storage and secrets; no billing and no new keys) and keep it in UNO_API_KEY.

import os
import requests

API = "https://console.uno.place/api/v1"
H = {"Authorization": f"Bearer {os.environ['UNO_API_KEY']}"}

# make the bucket once
bucket = requests.post(f"{API}/buckets", headers=H, json={"name": "assets"}).json()

# ask for a link, then send the file to it
link = requests.post(
    f"{API}/buckets/{bucket['id']}/presign",
    headers=H,
    json={"key": "reports/report.pdf", "method": "put"},
).json()["url"]

with open("report.pdf", "rb") as f:
    requests.put(link, data=f).raise_for_status()

import { readFileSync } from "node:fs";

const API = "https://console.uno.place/api/v1";
const H = { Authorization: `Bearer ${process.env.UNO_API_KEY}`, "Content-Type": "application/json" };

const { buckets } = await (await fetch(`${API}/buckets`, { headers: H })).json();
const bucket = buckets.find((b) => b.name === "assets");

const { url } = await (
  await fetch(`${API}/buckets/${bucket.id}/presign`, {
    method: "POST",
    headers: H,
    body: JSON.stringify({ key: "reports/report.pdf", method: "put" }),
  })
).json();

const res = await fetch(url, { method: "PUT", body: readFileSync("report.pdf") });
if (!res.ok) throw new Error(`upload failed: ${res.status}`);

The answer says how big a file one link takes (max_bytes): 4.9 GB on most buckets, 50 MB on older ones. For a bigger file, send its size with the request and Uno returns part links instead. You send part n to link n, and Uno assembles the file about 30 seconds after the last part. On an older bucket, a single PUT over about 95 MB fails with HTTP 500, so large files go in parts.

What this does not do

• An S3 key cannot create or delete buckets, and Uno has no ACLs, object versions or presigned POST forms.
• Storage counts toward your plan: 25 GB on Small, 100 GB on Plus. Going over the included storage costs $0.01 per GB a month, and nothing is deleted for being over.
• A bucket is private. For a public website, publish the folder as a site instead.

Common problems