Configuring S3 Storage

Use this guide to back Pkgly repositories with an Amazon S3 bucket or any S3-compatible object store (RustFS, DigitalOcean Spaces, Ceph, etc.). The storage driver lives in crates/storage/src/s3 and is loaded through the same storage-management APIs as the local backend, so the setup happens entirely through the admin interface or REST API—no code changes are required.

When to choose S3

• You need cheap, durable blob storage that multiple Pkgly nodes can share.
• You run Pkgly in Kubernetes or another stateless platform and do not want to manage block volumes.
• You already store artifacts in an S3-compatible service and want Pkgly to sit in front of that bucket.

Operational behavior:

• Moves use an ETag-guarded server-side CopyObject; objects larger than 5 GiB use multipart copy. The source is deleted only after a successful copy. If the guarded delete conflicts, both source and destination remain so no changed source data is lost.
• Generic storage appends use a read/modify/write sequence guarded by If-Match (or If-None-Match: * for creation). Docker upload sessions use local disk staging instead: chunks are streamed to a bounded temporary file and uploaded once when the digest is finalized. A conditional conflict is returned to the caller, which should retry with a fresh object version.
• Successful overwrites and appends invalidate the previous disk-cache entry; objects larger than the cache capacity are not retained.
• Metadata lookups for disk-cached objects use the in-memory size, content type, and modification time without reading the cached body or contacting S3. Cache entries are verified by size and SHA-256 when written and during startup recovery; normal hits only perform a bounded file-type and size check. Cache misses use S3 metadata requests.

Prerequisites

1. Network reachability from every Pkgly instance to the S3 endpoint (public AWS endpoint, VPC endpoint, on-prem RustFS URL, etc.).
2. Bucket created ahead of time. Pkgly never creates buckets; it assumes the bucket exists and is empty or reserved for Pkgly data.
3. Credential source Pkgly can use to talk to the bucket. Supply one of:
   • Static IAM keys (access key + secret, optional session token) scoped to the bucket with at least:
     • s3:ListBucket
     • s3:GetObject
     • s3:PutObject
     • s3:DeleteObject
     • s3:HeadObject
   • IAM role to assume (role_arn, optional role_session_name / external_id). Pkgly calls AssumeRole through the AWS SDK and uses those temporary credentials for all S3 traffic.
   • Instance/container profile – leave every credential field blank and Pkgly will fall back to the AWS SDK default chain (environment variables, shared config/credentials files, ECS/EKS credentials, IMDS, etc.).
4. Administrative login that carries the StorageManager capability (system manager or admin) so you can call /api/storage/**.

Bucket layout expectations

• Pkgly prefixes every object with the repository UUID, then appends the repository-relative path: s3://<bucket>/<repository-uuid>/<path/to/object>. This comes directly from S3StorageInner::s3_path.
• Keep the bucket dedicated to Pkgly or at least ensure no other process writes under those prefixes; otherwise Pkgly’s collision checks may fail deployments.
• Versioning and encryption are optional but recommended for mission-critical data.

Step 1 – Discover the region identifier

Pkgly exposes the list of built-in AWS regions via the API:

curl -s -H "Authorization: Bearer $PKGLY_TOKEN" \
  https://pkgly.example.com/api/storage/s3/regions | jq

Pick one of the canonical raw identifiers (for example, us-east-1). The list is only a convenience: Pkgly accepts any non-empty provider region string, including identifiers not yet listed. For S3-compatible endpoints that report a custom name (RustFS, Ceph), use a custom endpoint and optional signing-region label.

Step 2 – Decide between path-style and virtual-hosted-style URLs

path_style defaults to true, which means Pkgly will connect as https://endpoint/bucket/object. Keep the default for RustFS and most self-hosted gateways. Set path_style to false when talking to AWS’ public endpoints so requests use https://bucket.s3.amazonaws.com/object, which avoids the deprecation path for classic path-style access.

Step 3 – Create the storage (UI or API)

The Admin UI’s Storages → Create Storage → S3 / Object Storage form exposes every server-side option (region, custom endpoint, credential fields, role information, session token, path-style toggle, and the optional on-disk cache described below). For automation you can call the REST API directly; the payload mirrors StorageTypeConfig (type + settings object).

curl -X POST https://pkgly.example.com/api/storage/new/s3 \
  -H "Authorization: Bearer $PKGLY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "name": "s3-prod",
        "config": {
          "type": "S3",
          "settings": {
            "bucket_name": "pkgly-prod-artifacts",
            "region": "us-east-1",
            "credentials": {
              "access_key": "AKIA...",
              "secret_key": "••••••••",
              "session_token": "",
              "role_arn": "",
              "role_session_name": "",
              "external_id": ""
            },
            "path_style": false,
            "cache": {
              "enabled": true,
              "path": "/var/lib/pkgly/cache/s3-prod",
              "max_bytes": 2147483648,
              "max_entries": 4096
            }
          }
        }
      }'

Notes:

• name must be unique; Pkgly rejects duplicates with HTTP 409.
• region is mandatory unless you supply a custom endpoint (see below).
• Credentials are stored encrypted at rest inside the storages table. Rotate them the same way you created them: create a new storage (or future update endpoint once available) and reassign repositories to it.
• During creation Pkgly invokes S3StorageFactory::test_storage_config(). If anything fails (missing region, auth error, bucket not found) the API responds with 400 Invalid Storage Config and the precise driver error appears in the response body and server logs.

Region migration and rollback

The region field is stored as a raw string. The migration translates all legacy enum tokens (such as UsEast1) to canonical identifiers such as us-east-1; newly serialized configurations always use the raw form. Unknown provider identifiers are accepted and preserved unchanged. Rolling the migration back converts only the original 24 identifiers and deliberately leaves newer or custom values untouched. An enum-only binary cannot read those untouched values, so change them manually before downgrading to such a binary.

Custom endpoint example (RustFS, DigitalOcean Spaces, etc.)

The S3 driver flattens the optional CustomRegion struct, so provide custom_region (name label, optional) and endpoint (full URL) directly inside settings. When endpoint is present it takes precedence over region.

{
  "name": "s3-rustfs",
  "config": {
    "type": "S3",
    "settings": {
      "bucket_name": "pkgly-artifacts",
      "credentials": {
        "access_key": "rustfs",
        "secret_key": "rustfs-secret"
      },
      "path_style": true,
      "custom_region": "onprem",
      "endpoint": "https://rustfs.internal.example.com"
    }
  }
}

Optional: Local disk cache

To keep hot artifacts on the node and avoid repeated S3 downloads, enable the cache block. Pkgly keeps an LRU index keyed by the S3 path, writes files under the provided directory, and enforces both a byte cap and an entry cap. When the cache grows past max_bytes, the least recently used entries are evicted and their files deleted.

{
  "name": "s3-with-cache",
  "config": {
    "type": "S3",
    "settings": {
      "bucket_name": "pkgly-artifacts",
      "region": "us-east-1",
      "credentials": { "role_arn": "arn:aws:iam::123:role/pkgly" },
      "path_style": true,
      "cache": {
        "enabled": true,
        "path": "/var/lib/pkgly/cache/s3",
        "max_bytes": 1073741824,
        "max_entries": 2048
      }
    }
  }
}

Leave path blank to fall back to $(TMPDIR)/pkgly/s3-cache/<storage-name>. Prefer a directory dedicated to one Pkgly S3 storage. The cache writes versioned content sidecars atomically, verifies size and SHA-256 on writes and restart recovery, and rebuilds its LRU index after restart while preserving unrelated files in the configured directory.

Cache hits retain the S3 object Last-Modified timestamp. Files without a usable provider timestamp fall back to the request time; virtual directories and common prefixes are always assigned one request-time timestamp because S3 does not expose a directory creation time. Docker manifest indexes are kept in a per-storage LRU cache for 30 seconds and are refreshed after that window or after a successful mutation.

The driver uses fixed internal deadlines for control traffic: 30 seconds per attempt and 90 seconds for the complete control operation. Copy and multipart-copy requests allow five minutes per attempt and 20 minutes for the operation. Streamed GET and PUT operations intentionally have no wall-clock deadline so legitimate large transfers can finish; the SDK still stops stalled connections.

The configured cache directory should be dedicated to this storage. Pkgly only removes recognized cache sidecars, generation files, and legacy hash-layout artifacts, so unrelated files are retained, but sharing a directory between multiple Pkgly caches can still cause avoidable churn.

Upload resource limits

The S3 driver shares a 256 MiB process-wide budget for buffered bodies and streams larger or unknown-length responses. Docker upload chunks are spooled incrementally, with at most 64 staged uploads and 64 GiB of staged bytes per process. Incoming chunk files draw from the same 64 GiB budget as staged bytes, and appends reserve their capacity before writing, so concurrent push sessions cannot collectively exceed the limit; exceeding it rejects the request until capacity frees up. Proxy downloads to temporary files reserve their advertised size upfront and fail fast when the shared 64 GiB temporary-file budget is exhausted instead of blocking. Unfinished sessions are in-memory and expire when the process restarts; graceful shutdown removes their temporary files. Run one Pkgly writer for a repository when using Docker upload staging.

IAM role / default chain example

To avoid long-lived keys entirely, leave the key fields blank and provide only the role metadata. Pkgly will load the default AWS credential chain (environment variables, shared ~/.aws config, ECS/EKS task metadata, or EC2 IMDS) and then call AssumeRole before it touches S3.

{
  "name": "s3-assume-role",
  "config": {
    "type": "S3",
    "settings": {
      "bucket_name": "pkgly-artifacts",
      "region": "us-east-1",
      "credentials": {
        "role_arn": "arn:aws:iam::123456789012:role/pkgly-deploy",
        "role_session_name": "pkgly-ci",
        "external_id": "customer-123"
      },
      "path_style": false
    }
  }
}

Step 4 – Verify

1. Confirm the storage record exists:

   curl -s -H "Authorization: Bearer $PKGLY_TOKEN" \
     "https://pkgly.example.com/api/storage/list?active_only=true" | jq

   Ensure your new storage shows storage_type: "s3" and active: true.
2. Create or edit a repository so it points at the new storage, then push a small artifact. Pkgly writes to s3://<bucket>/<repository-uuid>/.... Inspect the bucket to make sure the object landed where expected.
3. Watch logs for Successfully connected to S3 bucket (emitted by S3StorageFactory::test_storage_config) or any subsequent AWS SDK error entries:

   docker compose logs -f pkgly | rg -i s3

Troubleshooting

Symptom	Likely cause	Fix
Invalid Storage Config: NoRegionSpecified	Neither region nor endpoint provided	Supply a non-empty provider region identifier or a custom endpoint
AWS SDK error (AccessDenied): InvalidAccessKeyId	Wrong credentials or user lacks permission	Regenerate the key pair or fix the role policy so it has the required bucket-scoped IAM actions
Cached file never expires	Cache size too small or directory not writable	Increase cache.max_bytes / max_entries or ensure Pkgly can write to the cache path
Bucket Does Not Exist	Typo in bucket_name or Pkgly lacks access to the bucket	Verify the bucket name and the IAM policy’s Resource list
Uploads stall or time out	The endpoint is unreachable or a control request exceeded its deadline	Check endpoint reachability and logs; control operations use 30-second attempt and 90-second total limits, while streamed transfers have stalled-read protection without a wall-clock cap

Operational tips

• Repository GET/HEAD dispatch, Docker manifest refresh/download, and S3 read/write operations use heap-allocated futures to limit nested request stack frames. This also protects local-storage Docker proxies, whose shared storage interface includes S3 operations, in debug builds with the runtime's default worker stack size.
• Keep the bucket lifecycle rules aligned with Pkgly retention policies. Pkgly never deletes repositories automatically, so lifecycle rules that expire objects will surface as 404s to clients.
• Monitor object count and size to detect runaway storages. Directory and manifest listings consume paginated S3 LIST calls; Docker manifest indexes are cached per repository for up to 30 seconds and are invalidated after successful mutations.
• Back up the storages table whenever you rotate credentials—losing it means Pkgly forgets how to talk to the bucket even though the data still exists.

Direct uploads larger than 5 GiB, distributed cache invalidation across Pkgly nodes, automatic egress-policy discovery, and cache metrics endpoints are outside this driver’s scope.