boto3 S3-compatible storage: custom endpoint, put_object and connection tests

1. Set up a separate credential profile

Install boto3 in your Python environment. The AWS CLI can create a named profile for the storage access key and secret. Enter them at its prompts; do not put secrets in source code or a command you will save in shell history.

Use credentials issued by the S3-compatible provider, not your AWS account keys. The example defaults to the profile named dotmoose. On a server, a protected credential file or your deployment secret manager must make that profile available to the application.

python -m pip install boto3
aws configure --profile dotmoose
aws configure set s3.addressing_style path --profile dotmoose

Boto3 credential configuration

2. Set the endpoint, region and bucket

Replace the example values with the details from your service. The endpoint is the HTTPS base address; do not append the bucket name. The bucket is a separate argument in the request.

These shell commands are for macOS or Linux. On Windows, configure equivalent environment variables in PowerShell or your application settings. The Python code is the same.

# Replace these with the values supplied by your storage provider.
export S3_ENDPOINT_URL='https://storage.example.com'
export S3_REGION='provider-region'
export S3_BUCKET='your-assigned-bucket'

3. Start with this read-only Python check

Save this as s3_check.py and run python s3_check.py. It requires all three settings, uses Signature Version 4, keeps TLS certificate validation enabled and requests at most five objects from the assigned bucket.

Path-style addressing places the bucket in the request path. Use the addressing style supported by your provider; a service requiring virtual-hosted addressing also needs matching bucket-hostname DNS and certificates. This example uses path style.

import os
from urllib.parse import urlsplit

import boto3
from botocore.config import Config
from botocore.exceptions import BotoCoreError, ClientError


def list_sample(env=os.environ):
    required = ("S3_ENDPOINT_URL", "S3_REGION", "S3_BUCKET")
    missing = [name for name in required if not env.get(name, "").strip()]
    if missing:
        raise ValueError("Set " + ", ".join(missing))

    endpoint = env["S3_ENDPOINT_URL"].strip()
    url = urlsplit(endpoint)
    if (url.scheme != "https" or not url.hostname
            or url.username or url.password or url.query or url.fragment
            or url.path not in ("", "/")):
        raise ValueError("Use the provider's HTTPS endpoint, without a bucket or credentials in the URL")

    # A named profile keeps these keys separate from your default AWS account.
    session = boto3.Session(profile_name=env.get("S3_PROFILE", "dotmoose"))
    client = session.client(
        "s3",
        endpoint_url=endpoint,
        region_name=env["S3_REGION"].strip(),
        config=Config(
            signature_version="s3v4",
            s3={"addressing_style": "path"},
            connect_timeout=10,
            read_timeout=30,
            retries={"mode": "standard", "total_max_attempts": 3},
        ),
    )
    try:
        result = client.list_objects_v2(
            Bucket=env["S3_BUCKET"].strip(), MaxKeys=5
        )
        return len(result.get("Contents", []))
    finally:
        client.close()


if __name__ == "__main__":
    try:
        print(f"Listed {list_sample()} objects (at most 5); no files changed.")
    except ValueError as error:
        raise SystemExit(str(error)) from None
    except ClientError as error:
        # Do not dump credentials, signed requests or object names into logs.
        code = error.response.get("Error", {}).get("Code", "Unknown")
        raise SystemExit(f"S3 request failed: {code}") from None
    except BotoCoreError:
        raise SystemExit("Check the named profile, network, endpoint and TLS trust.") from None

Boto3 client endpoint and TLS options | Botocore addressing, timeout and retry settings

4. Optionally prove put_object with one isolated round trip

Run a write test only in a bucket and credential scope where you are authorized to create, read and delete a test object. The example refuses to start until S3_ALLOW_WRITE_TEST is exactly YES, generates a fresh UUID under the dotmoose-boto3-check/ prefix, uploads a short bytes payload with put_object, downloads the same key, compares the returned bytes and deletes only that key.

Use this after the read-only check when write compatibility matters. A successful round trip proves those operations for this endpoint, credential, SDK version and test object; it does not prove multipart uploads, versioning, Object Lock, lifecycle rules or every Amazon S3 feature.

import os
import uuid
from urllib.parse import urlsplit

import boto3
from botocore.config import Config
from botocore.exceptions import BotoCoreError, ClientError

PAYLOAD = b"DotMoose boto3 write compatibility check\n"


def write_round_trip(env=os.environ):
    required = ("S3_ENDPOINT_URL", "S3_REGION", "S3_BUCKET")
    missing = [name for name in required if not env.get(name, "").strip()]
    if missing:
        raise ValueError("Set " + ", ".join(missing))
    if env.get("S3_ALLOW_WRITE_TEST", "").strip() != "YES":
        raise ValueError(
            "Set S3_ALLOW_WRITE_TEST=YES only when you are ready to create and delete one test object"
        )

    endpoint = env["S3_ENDPOINT_URL"].strip()
    url = urlsplit(endpoint)
    if (url.scheme != "https" or not url.hostname
            or url.username or url.password or url.query or url.fragment
            or url.path not in ("", "/")):
        raise ValueError("Use the provider's HTTPS endpoint, without a bucket or credentials in the URL")

    bucket = env["S3_BUCKET"].strip()
    key = f"dotmoose-boto3-check/{uuid.uuid4().hex}.txt"
    session = boto3.Session(profile_name=env.get("S3_PROFILE", "dotmoose"))
    client = session.client(
        "s3",
        endpoint_url=endpoint,
        region_name=env["S3_REGION"].strip(),
        config=Config(
            signature_version="s3v4",
            s3={"addressing_style": "path"},
            connect_timeout=10,
            read_timeout=30,
            retries={"mode": "standard", "total_max_attempts": 3},
        ),
    )
    created = False
    body = None
    try:
        client.put_object(
            Bucket=bucket,
            Key=key,
            Body=PAYLOAD,
            ContentType="text/plain",
        )
        created = True
        result = client.get_object(Bucket=bucket, Key=key)
        body = result["Body"]
        if body.read() != PAYLOAD:
            raise RuntimeError("Downloaded test bytes did not match the upload")
        return key
    finally:
        if body is not None:
            body.close()
        if created:
            # Delete exactly the unique key this run created; never use a broad delete here.
            client.delete_object(Bucket=bucket, Key=key)
        client.close()


if __name__ == "__main__":
    try:
        write_round_trip()
        print("Write/read/delete round trip passed; the unique test object was removed.")
    except ValueError as error:
        raise SystemExit(str(error)) from None
    except ClientError as error:
        code = error.response.get("Error", {}).get("Code", "Unknown")
        raise SystemExit(f"S3 write test failed: {code}") from None
    except BotoCoreError:
        raise SystemExit("Check the named profile, network, endpoint and TLS trust.") from None

Boto3 S3 PutObject reference

5. Interpret errors without weakening the client blindly

A successful empty listing is valid for an empty bucket. AccessDenied can mean the key is valid but lacks the specific ListBucket, PutObject, GetObject or DeleteObject permission required by the operation being tested. NoSuchBucket usually means the bucket name or endpoint is wrong. For SignatureDoesNotMatch, check the key, signing region, endpoint, system clock and any proxy changing requests.

Current AWS SDK integrity defaults matter when testing S3-compatible services. The current AWS SDK reference lists CRC32 as Boto3's default S3 checksum algorithm and request_checksum_calculation defaults to when_supported. If a modern client fails on an unexpected checksum header or validation response, compare the storage backend's supported S3 operations with the installed boto3/botocore behavior before changing checksum settings. Do not disable TLS or integrity checks merely to make the test pass.

For a TLS error, fix the hostname or certificate trust. Do not make verify=False the production workaround. For a profile error, check that the application user can read the named profile.

AWS SDK S3 data-integrity defaults | Boto3 checksum configuration

6. Test the rest of the workload you actually depend on

The isolated object test covers one small PutObject, GetObject and DeleteObject path. Test larger multipart files, presigned requests, retention controls and restores separately when your application depends on them. Keep destructive tests inside dedicated prefixes and clean up only objects created by the test.

DotMoose provides the endpoint, region, bucket and access keys with an Object Storage service. Compare the plans for storage capacity, or check the exact operations your software requires before buying. No AWS-only feature is implied by S3 compatibility.

Compare Canadian Object Storage plans | Use AWS CLI with the same endpoint | How S3 presigned URLs work