Migrate Object Storage (MinIO to S3)

Overview

This article walks the installer through migrating LILT object storage from an in-cluster MinIO deployment to an Amazon S3 bucket. It applies when moving an existing self-managed installation onto a fully managed cluster, such as the environment described in Install System (AWS EKS), where S3 replaces MinIO as the object store. Object data is read out of MinIO through its S3-compatible API into a plain directory tree on a staging host, then uploaded to the destination bucket. The procedure is read-only against the source cluster: nothing in the export stage modifies the running MinIO deployment or its volume, so the original environment remains available as a fallback until the migration is accepted. Database migration is a separate procedure and is not covered here.

Tools you will need

AWS user with permissions to read and write the destination S3 bucket, and to use its KMS key if the bucket is encrypted with SSE-KMS.
Required utilities for the staging host:
- AWS CLI
- kubectl, with access to the source cluster
A staging host with:
- Free disk space greater than the total size of the object data
- A case-sensitive filesystem (Linux ext4 or xfs)

Structure of this Article

The first section explains why the MinIO storage volume cannot be copied directly, and how to confirm which on-disk format your deployment uses. The remaining sections cover preparing the destination bucket, exporting the objects, uploading them without overwriting existing data, verifying the result, and the configuration changes required before the application can read from the new bucket.

Installer privileges

All commands are run from a staging host by a user with the permissions stated in the “Tools” section above. Commands that target the source cluster assume the LILT namespace is lilt; adjust if your installation uses a different namespace.

MinIO On-Disk Format

Why the storage volume cannot be copied directly

Current MinIO releases do not store objects as plain files, even in single-drive deployments. Objects are held in an erasure-coded layout:

Each object key is a directory containing a metadata file named xl.meta
Object data is written into a version-UUID subdirectory as part.1, part.2, and so on
Objects smaller than 128 KiB are inlined into xl.meta and have no separate data file
Drive-level state is kept in a hidden .minio.sys directory

Copying that directory tree into S3 uploads unusable metadata fragments under altered key names, and objects below the inline threshold are lost entirely because they never existed as files. The layout is misleading at a glance. Object keys containing / become nested directories under either format, so the directory tree on the volume resembles the output of aws s3 ls. Recognising the prefix structure is not evidence that a direct copy will work.

Confirm the on-disk format

Read the drive format from the MinIO pod. The data volume is mounted at /export:

kubectl -n lilt exec <minio-pod> -- cat /export/.minio.sys/format.json

The format field determines which method applies:

xl-single or xl — erasure-coded. Use the procedure in this article.
fs — a legacy plain-file layout, used only by installations created by much older MinIO releases. A direct copy of the bucket directory is valid in that case; see “Legacy filesystem layout” at the end of this article.

A second check confirms the result. Under the erasure-coded format, no file is named after an object key:

kubectl -n lilt exec <minio-pod> -- sh -c 'find /export -name xl.meta | head -3'

Prepare the Destination

Set the user environment variables

export AWS_REGION="us-east-1"
export DEST_BUCKET="example-object-storage"
export STAGE_DIR="/export/objects"

Confirm bucket access and configuration

Record the encryption and versioning settings; they determine which permissions are required and whether an accidental overwrite can be recovered:

aws s3api head-bucket --bucket $DEST_BUCKET
aws s3api get-bucket-encryption --bucket $DEST_BUCKET
aws s3api get-bucket-versioning --bucket $DEST_BUCKET

Required permissions

The uploading principal needs the following. If get-bucket-encryption reports aws:kms, the KMS permissions are also required — without them every upload fails with AccessDenied, which is easily mistaken for an S3 policy problem:

s3:PutObject, s3:AbortMultipartUpload, s3:ListBucket, s3:GetBucketLocation, s3:GetObject
kms:GenerateDataKey, kms:Encrypt, kms:DescribeKey    (only when the bucket uses SSE-KMS)

Validate write access with a single object before starting a long transfer:

echo test > /tmp/probe
aws s3 cp /tmp/probe s3://$DEST_BUCKET/migration-probe
aws s3 rm s3://$DEST_BUCKET/migration-probe
rm /tmp/probe

Export Objects from MinIO

Stop application writes

An object store copy is only consistent if nothing is writing to it. Scale down the application workloads while leaving the data services running, since MinIO is the source of the export. Record the current replica counts first, so the environment can be restored if the migration is abandoned:

kubectl -n lilt get deploy,statefulset \
  -o custom-columns='KIND:.kind,NAME:.metadata.name,REPLICAS:.spec.replicas' \
  --no-headers > /tmp/replicas-before.txt

Scale down the application, excluding the data services:

DATA_SVCS='minio|mysql|mongodb|rabbitmq|clickhouse|elasticsearch|redis|dragonfly|qdrant'

kubectl -n lilt get deploy -o name \
  | grep -vE "$DATA_SVCS" \
  | xargs -r -I{} kubectl -n lilt scale {} --replicas=0

kubectl -n lilt get rollout -o name 2>/dev/null | xargs -r -I{} \
  kubectl -n lilt patch {} --type=merge -p '{"spec":{"replicas":0}}'

Suspend scheduled work, which would otherwise continue writing after the deployments are scaled down:

kubectl -n lilt get cronjob -o name 2>/dev/null | xargs -r -I{} \
  kubectl -n lilt patch {} --type=merge -p '{"spec":{"suspend":true}}'

kubectl -n lilt get cronworkflow -o name 2>/dev/null | xargs -r -I{} \
  kubectl -n lilt patch {} --type=merge -p '{"spec":{"suspend":true}}'

Do not scale down every deployment in the namespace with a single command. Doing so stops MinIO, which the export reads from, and discards the replica counts needed to restore the environment.

Retrieve the MinIO credentials

Read them from the cluster rather than assuming the installation defaults are still in place:

kubectl -n lilt get secret minio -o jsonpath='{.data.rootUser}' | base64 -d; echo
kubectl -n lilt get secret minio -o jsonpath='{.data.rootPassword}' | base64 -d; echo

Open a connection to the MinIO API

Forward the MinIO service port to the staging host:

kubectl -n lilt port-forward svc/minio 9000:9000 &
PF_PID=$!
sleep 2
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:9000/minio/health/live

A response of 200 confirms the connection. Port forwarding tunnels through the Kubernetes API server and can drop during a long transfer. If the connection proves unstable, capture the service definition, expose it as a NodePort for the duration of the migration, connect directly to a node, and restore the original definition afterwards:

kubectl -n lilt get svc minio -o yaml > /tmp/svc-minio-before.yaml
kubectl -n lilt patch svc minio -p \
  '{"spec":{"type":"NodePort","ports":[{"name":"http","port":9000,"targetPort":9000,"nodePort":32000}]}}'

# transfer, then restore
kubectl -n lilt apply -f /tmp/svc-minio-before.yaml

Configure the AWS CLI for MinIO

MinIO does not provide per-bucket DNS, so path-style addressing is required. This profile is used only to read the source; the upload uses your normal AWS credentials:

aws configure set aws_access_key_id     "<minio-root-user>"     --profile minio
aws configure set aws_secret_access_key "<minio-root-password>" --profile minio
aws configure set region                $AWS_REGION             --profile minio
aws configure set s3.addressing_style   path                    --profile minio

mio() { aws --profile minio --endpoint-url http://127.0.0.1:9000 "$@"; }

TLS-enabled MinIO endpoints

A default installation serves the MinIO API over plain HTTP inside the cluster, which is what the endpoint above assumes. If custom certificates have been applied to MinIO outside of the service mesh, the endpoint must use https instead, and certificate verification usually has to be relaxed for the export:

mio() { aws --profile minio --endpoint-url https://127.0.0.1:9000 --no-verify-ssl "$@"; }

Verification fails even with a valid certificate when connecting through a forwarded port, because the certificate is issued for the in-cluster service name and does not match 127.0.0.1. --no-verify-ssl is acceptable here since the connection is a local tunnel to a port on the same host. The alternative is to trust the certificate authority properly and connect by a name the certificate covers, which avoids disabling verification:

export AWS_CA_BUNDLE=/path/to/ca-bundle.crt

The same applies to the health check earlier in this section: use curl -k https://127.0.0.1:9000/minio/health/live when the endpoint is served over TLS. These settings affect only the export from MinIO. Connections to the destination S3 bucket use public AWS certificates, which verify against the system trust store, so no equivalent option is needed and AWS_CA_BUNDLE must not remain set for those commands. Identify the source bucket. A default installation uses a single bucket for application object storage:

mio s3 ls
export SOURCE_BUCKET="<bucket-name-from-previous-command>"

Record the source inventory

These files are the reference for every verification step, including after the upload. Retain them:

mio s3 ls --recursive --summarize s3://$SOURCE_BUCKET/ > /tmp/src-listing.txt
tail -3 /tmp/src-listing.txt

awk 'NF>3 { $1=""; $2=""; $3=""; sub(/^ +/,""); print }' /tmp/src-listing.txt \
  | sort > /tmp/src-keys.txt
wc -l /tmp/src-keys.txt

df -h $STAGE_DIR

Free space on the staging host must exceed the reported total size.

Check the object key names

Some valid S3 keys cannot be represented as filesystem paths. Check before starting a long transfer rather than after. No output from all four commands is the expected result:

grep -c '/$' /tmp/src-keys.txt                        # directory-marker keys
grep -n '//' /tmp/src-keys.txt                        # empty path segment
grep -nE '(^|/)\.\.?(/|$)' /tmp/src-keys.txt          # . or .. path segments
tr 'A-Z' 'a-z' < /tmp/src-keys.txt | sort | uniq -d   # case-only collisions

Any result here requires a decision before proceeding. S3 keys are case-sensitive, so two keys differing only in case are distinct objects that a case-insensitive filesystem would silently merge into one file. A key ending in / cannot round-trip through a filesystem at all.

Export to the staging directory

mkdir -p $STAGE_DIR
aws configure set default.s3.max_concurrent_requests 32 --profile minio
mio s3 sync s3://$SOURCE_BUCKET/ $STAGE_DIR/ --no-progress 2>&1 | tee /tmp/export.log

Verify the export format

Confirm the export contains objects rather than MinIO’s internal layout. All four commands must return 0. A non-zero result means the storage volume was copied instead of the API contents, and the export cannot be used:

find $STAGE_DIR -name 'xl.meta'    | wc -l
find $STAGE_DIR -name 'part.*'     | wc -l
find $STAGE_DIR -name '.minio.sys' | wc -l
find $STAGE_DIR -type d -empty     | wc -l

Confirm the exported entries are regular files. Under the erasure-coded format these same paths were directories, so this is the positive signal:

find $STAGE_DIR -type f | head -5 | xargs -I{} sh -c 'printf "%s\n  " "{}"; file -b "{}"'

Verify the export is complete

The object count and byte total must match the inventory, and both comm commands must produce no output:

find $STAGE_DIR -type f | wc -l      ; grep 'Total Objects' /tmp/src-listing.txt
du -sb --apparent-size $STAGE_DIR    ; grep 'Total Size'    /tmp/src-listing.txt

( cd $STAGE_DIR && find . -type f | sed 's|^\./||' | sort ) > /tmp/exported-keys.txt
comm -23 /tmp/src-keys.txt /tmp/exported-keys.txt    # in MinIO, missing locally
comm -13 /tmp/src-keys.txt /tmp/exported-keys.txt    # local, not in MinIO

Use --apparent-size with du. Without it, du reports allocated blocks and overstates the total against the byte count S3 reports.

Verify content integrity on a sample

Matching names and sizes do not prove the contents are correct. For objects uploaded in a single part, the MinIO ETag is the MD5 of the content and can be compared directly. Objects uploaded in multiple parts have an ETag ending in -<n> and are skipped by this check:

shuf -n 25 /tmp/src-keys.txt | while IFS= read -r key; do
  etag=$(mio s3api head-object --bucket $SOURCE_BUCKET --key "$key" \
           --query ETag --output text 2>/dev/null | tr -d '"')
  case "$etag" in *-*) printf 'SKIP (multipart) %s\n' "$key"; continue ;; esac
  local_md5=$(md5sum "$STAGE_DIR/$key" | cut -d' ' -f1)
  [ "$etag" = "$local_md5" ] && printf 'OK   %s\n' "$key" \
    || printf 'MISMATCH %s (minio=%s local=%s)\n' "$key" "$etag" "$local_md5"
done

Also check the largest objects, where a truncated transfer appears first:

sort -k3 -n -r /tmp/src-listing.txt | head -10

Any MISMATCH result means the export must be repeated for those keys before continuing.

Close the connection

kill $PF_PID

Upload Objects to S3

Object keys are preserved exactly. Only the bucket name changes: $STAGE_DIR/<key> maps to s3://$DEST_BUCKET/<key>. Do not introduce an additional prefix at the destination — the application stores these keys in its database, and adding a prefix invalidates every stored reference.

Inventory the destination

The destination bucket may already contain objects that must not be overwritten. Record its contents in detail first. This listing is also the baseline that proves, after the upload, that nothing pre-existing was modified:

aws s3api list-objects-v2 --bucket $DEST_BUCKET \
  --query 'Contents[].[Key,ETag,LastModified,Size]' --output text \
  | sort > /tmp/dst-before-detail.txt
cut -f1 /tmp/dst-before-detail.txt | sort > /tmp/dst-before.txt

comm -12 /tmp/src-keys.txt /tmp/dst-before.txt > /tmp/collisions.txt
comm -23 /tmp/src-keys.txt /tmp/dst-before.txt > /tmp/to-upload.txt
wc -l /tmp/collisions.txt /tmp/to-upload.txt

Keys listed in collisions.txt are skipped by the upload, so the object already in the destination is kept and the exported copy is not migrated. Review the list before continuing: for any colliding key whose contents differ, the source version is silently left behind, and the application will serve the destination version because it references keys rather than contents.

How aws s3 sync decides what to transfer

aws s3 sync does not blindly re-upload everything. The reference documentation states the rule:

A local file will require uploading if the size of the local file is different than the size of the S3 object, the last modified time of the local file is newer than the last modified time of the S3 object, or the local file does not exist under the specified bucket and prefix.

Two consequences matter for this migration:

An object already present at the destination, with the same size and an equal or later modification time, is not transferred. sync will not re-upload identical objects, so re-running it after an interruption resumes rather than repeating work.
An object already present at the destination is replaced if its size differs from the exported copy, or if the exported copy carries a newer modification time. Presence at the destination is therefore not by itself a guarantee that the existing object is preserved.

For a migration into a bucket that already holds data, rely on the key list built in the previous step rather than on the comparison rule. Uploading only the keys that are absent from the destination makes an overwrite impossible, because no request is ever issued against an existing key. This method also behaves identically on every AWS CLI version.

Upload only the keys absent from the destination

export DEST_BUCKET STAGE_DIR

upload_one() {
  key="$1"
  if aws s3api head-object --bucket "$DEST_BUCKET" --key "$key" >/dev/null 2>&1; then
    printf 'SKIPPED-EXISTS %s\n' "$key"
  elif aws s3 cp "$STAGE_DIR/$key" "s3://$DEST_BUCKET/$key" --only-show-errors; then
    printf 'UPLOADED %s\n' "$key"
  else
    printf 'FAILED %s\n' "$key"
  fi
}
export -f upload_one

tr '\n' '\0' < /tmp/to-upload.txt \
  | xargs -0 -P 8 -I{} bash -c 'upload_one "$@"' _ {} \
  | tee /tmp/upload.log

Each key is checked immediately before it is written, which also catches anything added to the destination since the inventory was taken. Confirm the results:

grep -c UPLOADED      /tmp/upload.log    # equals the line count of to-upload.txt
grep   SKIPPED-EXISTS /tmp/upload.log    # expected empty
grep   FAILED         /tmp/upload.log    # must be empty

The command requires bash for export -f. Remove -P 8 to run the uploads serially. Re-running is safe: keys already uploaded are skipped by the check. Do not set --sse or --sse-kms-key-id; the bucket’s default encryption is applied server-side, and specifying a different key only introduces a failure.

Using aws s3 sync instead

If the destination bucket is empty — confirmed by the inventory in the previous step reporting no objects — aws s3 sync is simpler and parallelises automatically:

aws configure set default.s3.max_concurrent_requests 32
aws s3 sync $STAGE_DIR/ s3://$DEST_BUCKET/ --dryrun | wc -l
aws s3 sync $STAGE_DIR/ s3://$DEST_BUCKET/ --no-progress 2>&1 | tee /tmp/upload.log

The trailing slashes are required for keys to map correctly. Never pass --delete, which removes destination objects that are absent from the source. Recent AWS CLI versions also provide a --no-overwrite option, described in the reference as preventing overwriting at the destination so that only files not already present are transferred. It is not available in every version, so confirm it before depending on it:

aws --version
PAGER=cat aws s3 sync help | col -b | grep -c -- '--no-overwrite'

A result of 0 means the installed version does not support it, and a sync issued with the flag would fail or ignore it. Use the key-list method above in that case.

Stricter alternative for a bucket in active use

If other writers may be adding objects to the bucket while the upload runs, a conditional write removes the remaining race. The service rejects the request with PreconditionFailed when the key already exists, so an overwrite is impossible at the API level rather than by prior inspection:

aws s3api put-object --bucket $DEST_BUCKET --key "<key>" \
  --body "$STAGE_DIR/<key>" --if-none-match '*'

A single request of this form is limited to 5 GiB, so any larger object must be uploaded by one of the methods above. Identify them first:

awk 'NF>3 && $3+0 > 5368709120 { size=$3; $1="";$2="";$3=""; sub(/^ +/,"");
     print size "\t" $0 }' /tmp/src-listing.txt

Verify the Migration

Confirm no existing objects were modified

Take the detailed listing again and compare it against the baseline recorded before the upload. No output is the expected result, and confirms that no pre-existing object changed:

aws s3api list-objects-v2 --bucket $DEST_BUCKET \
  --query 'Contents[].[Key,ETag,LastModified,Size]' --output text \
  | sort > /tmp/dst-after-detail.txt

comm -23 /tmp/dst-before-detail.txt /tmp/dst-after-detail.txt

Any line in the output identifies an object that was overwritten. If bucket versioning is enabled, the previous version can be recovered:

aws s3api list-object-versions --bucket $DEST_BUCKET --prefix "<key>"

Confirm every source object arrived

This comparison must produce no output:

aws s3api list-objects-v2 --bucket $DEST_BUCKET \
  --query 'Contents[].Key' --output text | tr '\t' '\n' | sort > /tmp/dst-keys.txt

comm -23 /tmp/src-keys.txt /tmp/dst-keys.txt

Compare the object counts and total sizes as well:

aws s3 ls --recursive --summarize s3://$DEST_BUCKET/ | tail -3
tail -3 /tmp/src-listing.txt

The destination totals include any objects that were already present, so they will exceed the source totals by that amount. Compare the key sets and per-object sizes rather than the totals alone. Do not compare ETags between MinIO and S3. Server-side encryption and multipart uploads both change the ETag, so a difference between the two systems carries no meaning. ETag comparison within the destination bucket, as used in the check above, is valid because the bucket and its encryption settings are the same on both sides.

Confirm the expected prefixes are present

aws s3 ls s3://$DEST_BUCKET/

Post-Migration Configuration

Migrating the data does not by itself connect the application to the new bucket. Confirm the following before returning the environment to service.

Bucket name. The bucket name configured for the application must match the destination bucket. Check every location where the deployment sets it, including any per-service override, since a single global value can be shadowed by a stale one elsewhere.
Object storage endpoint. Any MinIO endpoint, region placeholder, or TLS verification override used for in-cluster storage must be replaced with the AWS S3 endpoint and the real region.
Credentials. If pods obtain AWS credentials from an IAM role for service accounts, any static access key left in the configuration must be cleared. A static value takes precedence over the role in the AWS SDK credential chain, so leftover MinIO development keys cause every read to fail against S3.
Certificate bundle. Any custom CA bundle configured for a private object store must be unset. AWS certificates verify against the system trust store.
Upload notifications. In-cluster MinIO publishes object-created events over AMQP. With S3, the equivalent notification is delivered through SQS, so the queue the application subscribes to must exist in the same account and be named correctly in the configuration. Without it, uploads never complete in the interface.
Notification event types. The bucket notification must include s3:ObjectCreated:CompleteMultipartUpload in addition to s3:ObjectCreated:Put. Browser uploads above the multipart threshold emit the former, so a configuration covering only Put results in small files working and large files appearing to hang:

aws s3api get-bucket-notification-configuration --bucket $DEST_BUCKET

CORS. Browsers upload directly to the bucket, so the CORS configuration must allow the application’s front-end origin:

aws s3api get-bucket-cors --bucket $DEST_BUCKET

Cleanup

Restore the application to its previous replica counts using the file recorded before the export, then remove the staged copy and temporary credentials:

rm -rf $STAGE_DIR
aws configure --profile minio set aws_access_key_id ""
aws configure --profile minio set aws_secret_access_key ""

Remove any temporary IAM access key created for the upload, and restore the MinIO service definition if it was exposed as a NodePort. Retain the inventory files and transfer logs as a record of the migration. Keep the MinIO storage volume intact until the migration has been accepted. No step in the export modifies it, so the original environment remains a complete fallback.

Legacy Filesystem Layout

This section applies only if the format check reported fs, which occurs on installations created by much older MinIO releases. In that case the bucket directory contains regular files named after the object keys, and it can be synchronised directly. Inspect the dry run before running the transfer. Every key listed must be a real object key, with no xl.meta, no part.1, and no UUID path segments. If any of those appear, the format check was misread and the procedure in this article applies instead:

aws s3 sync <minio-data-dir>/<bucket>/ s3://$DEST_BUCKET/ \
  --exclude '.minio.sys/*' --dryrun

Debugging

AccessDenied on upload with correct S3 permissions: the bucket uses SSE-KMS and the uploading principal is missing kms:GenerateDataKey or kms:Encrypt on the bucket’s key. The error names S3 rather than KMS, which makes this easy to misdiagnose. The export contains xl.meta or part.1 files: the storage volume was copied instead of reading through the API. Delete the staged copy and repeat the export using the MinIO endpoint. An object already in the destination bucket was replaced: aws s3 sync transfers an object whose size differs from the exported copy, even when the key already exists. Use the key-list method in “Upload only the keys absent from the destination” for a bucket that already holds data. If versioning is enabled on the bucket, recover the previous version with aws s3api list-object-versions --bucket $DEST_BUCKET --prefix "<key>". Small files migrate but large files do not appear in the application: the bucket notification is missing s3:ObjectCreated:CompleteMultipartUpload. The transfer stops partway with a connection error: the port-forward tunnel dropped. Re-establish it and run the export again; aws s3 sync skips objects already present in the staging directory, so the transfer resumes rather than restarting. Objects are present in the bucket but the application reports them as missing: the configured bucket name does not match the destination bucket, or an additional prefix was introduced during the upload. Compare a key from src-keys.txt against the same object in the bucket.

​Overview

​Tools you will need

​Structure of this Article

​Installer privileges

​MinIO On-Disk Format

​Why the storage volume cannot be copied directly

​Confirm the on-disk format

​Prepare the Destination

​Set the user environment variables

​Confirm bucket access and configuration

​Required permissions

​Export Objects from MinIO

​Stop application writes

​Retrieve the MinIO credentials

​Open a connection to the MinIO API

​Configure the AWS CLI for MinIO

​TLS-enabled MinIO endpoints

​Record the source inventory

​Check the object key names

​Export to the staging directory

​Verify the export format

​Verify the export is complete

​Verify content integrity on a sample

​Close the connection

​Upload Objects to S3

​Inventory the destination

​How aws s3 sync decides what to transfer

​Upload only the keys absent from the destination

​Using aws s3 sync instead

​Stricter alternative for a bucket in active use

​Verify the Migration

​Confirm no existing objects were modified

​Confirm every source object arrived

​Confirm the expected prefixes are present

​Post-Migration Configuration

​Cleanup

​Legacy Filesystem Layout

​Debugging