Back Up and Restore Elasticsearch

Overview

LILT runs Elasticsearch under the Elastic Cloud on Kubernetes (ECK) operator. Elasticsearch holds the indices behind translation-memory search, user resources and job search. The installer does not configure a snapshot repository. Until you register one, Elasticsearch has nowhere to write a backup to, so a new installation has no backup even though the upgrade procedures tell you to take one. This article covers registering a repository, taking and restoring snapshots, and rebuilding the search index from the LILT database. For the upgrade itself, see Upgrade Elasticsearch Across Major Versions.

Every command on this page assumes the LILT namespace is lilt. If you installed into a different namespace, substitute it for NAMESPACE.

Snapshot or rebuild?

Everything LILT stores in Elasticsearch is derived from the LILT database, so a lost Elasticsearch cluster is a rebuild rather than permanent data loss. One detail decides which path you take:

	Restore a snapshot	Rebuild from the database
Time to complete	Minutes	Hours on a large installation
Rebuilds the `segments` index	Yes	Yes
Rebuilds `user_resources` and `jobs`	Yes	No tool exists
Needs setup beforehand	Yes, a repository	No

Set up snapshots even though the data is derived. A rebuild covers only the segments index, so it is a partial recovery, and it is much slower. Use a rebuild for a damaged or incomplete segments index, and a snapshot for the loss of a cluster.

Connect to the cluster

Every curl command on this page reaches Elasticsearch through a port forward. Target a pod rather than a service: LILT applications reach Elasticsearch through a service of type ExternalName, which kubectl port-forward cannot use.

NAMESPACE=lilt
ESPOD=$(kubectl -n $NAMESPACE get pods -l common.k8s.elastic.co/type=elasticsearch \
  -o jsonpath='{.items[0].metadata.name}')
kubectl -n $NAMESPACE port-forward "pod/$ESPOD" 9200:9200 &

Leave this running while you work, and stop it when you finish. The shipped cluster runs with TLS disabled, because Istio carries the transport security, so these commands need no credentials. That is also a reason not to leave the port forward open.

Register a snapshot repository

Elasticsearch writes backups to a snapshot repository, which you register once. Elasticsearch includes S3 support, so both procedures that follow use the s3 repository type. A shared-filesystem repository is not an option: the Elasticsearch volumes are ReadWriteOnce and a filesystem repository must be mounted on every node in the cluster.

Add the credential settings to eck-cluster/on-prem-values.yaml or eck-cluster/eks-values.yaml. A LILT release replaces those files, so re-apply the settings after each upgrade and confirm that snapshots still succeed. The repository itself is stored in the Elasticsearch cluster state and survives an upgrade; the credentials that make it usable do not.

On AWS with an S3 bucket

Your installation already has an AWS identity for S3: install-s3-irsa.sh creates the lilt-s3-workload service account and binds it to the app-workload IAM role. Run Elasticsearch as that service account rather than creating a second role. In eck-cluster/eks-values.yaml:

serviceAccount:
  create: false
  name: lilt-s3-workload

This works under both of the mechanisms the installer supports, so you do not need to know which one your cluster uses.

Do not add an eks.amazonaws.com/role-arn annotation to the service account. That annotation is correct only when IAM_AUTH_METHOD=irsa. When your installation uses EKS Pod Identity, install-s3-irsa.sh deliberately removes the annotation, because a leftover one takes precedence over the Pod Identity binding and sends the AWS SDK to a web identity provider your account may not have. Naming the service account, as shown, is correct either way.

The app-workload role’s policy has to allow the snapshot bucket. It is scoped to the buckets LILT itself uses, so if you follow the recommendation to keep snapshots in a separate bucket, add that bucket to the role’s policy first. Reusing the service account gives Elasticsearch the identity; it does not widen what that identity may do. If no AWS identity is available, store access keys in a Kubernetes secret instead:

kubectl -n $NAMESPACE create secret generic es-s3-credentials \
  --from-literal=s3.client.default.access_key='ACCESS_KEY' \
  --from-literal=s3.client.default.secret_key='SECRET_KEY'

Then reference the secret in eck-cluster/eks-values.yaml:

secureSettings:
  - secretName: es-s3-credentials

The two key names are exact. They are Elasticsearch keystore entries, so a different name is ignored. Apply the change. A component script does not read install.env — only install-lilt.sh and install-lilt-eks.sh load that file — so load it yourself and set VALS_FILE on AWS:

set -a; . ./install.env; set +a
VALS_FILE=eks-values.yaml sh install_scripts/install-elastic-eck-cluster.sh

On an on-premise installation, omit VALS_FILE.

Do not skip install.env. With DNS_DOMAIN unset the script stops with ERROR: DNS_DOMAIN is required. If DNS_DOMAIN happens to be exported but REGISTRY_BASE is not, the values resolve to a LILT-internal registry host your cluster cannot pull from and the script completes anyway, leaving Elasticsearch unable to start its image. The check that catches a missing REGISTRY_BASE lives in the entrypoint, not in the component script.

The operator loads the credentials without restarting the cluster. Register the repository:

curl -s -X PUT 'localhost:9200/_snapshot/lilt-es-backup' \
  -H 'Content-Type: application/json' -d '{
  "type": "s3",
  "settings": { "bucket": "BUCKET_NAME", "client": "default" }
}'

On-premise with MinIO

An on-premise installation has no S3, but it does include MinIO. Point the same repository type at it, and set an endpoint. Create the credentials secret as described in the preceding section, using your MinIO access key pair, then register the repository:

curl -s -X PUT 'localhost:9200/_snapshot/lilt-es-backup' \
  -H 'Content-Type: application/json' -d '{
  "type": "s3",
  "settings": {
    "bucket": "lilt-es-snapshots",
    "endpoint": "minio.lilt.svc.cluster.local:9000",
    "protocol": "http",
    "path_style_access": true,
    "client": "default"
  }
}'

Create the bucket first with mc mb.

Use a separate bucket, not the lilt bucket. The lilt bucket holds your translation documents, so snapshots stored there share a failure domain with the data they protect. Treat a MinIO snapshot as a convenience copy, and replicate it to storage outside the cluster for disaster recovery.

Verify the repository

A repository you have never verified is not a backup. Check that every node can write to it:

curl -s -X POST 'localhost:9200/_snapshot/lilt-es-backup/_verify'

A successful response lists the cluster nodes. A 403 or a repository_verification_exception means the credentials or the bucket permissions are wrong, and a snapshot fails at the moment you need it.

Take a snapshot

Take a snapshot before every upgrade. Each version step changes the data on disk and cannot be reversed.

SNAPSHOT="pre-upgrade-$(date +%Y%m%d-%H%M%S)"
curl -s -X PUT "localhost:9200/_snapshot/lilt-es-backup/$SNAPSHOT?wait_for_completion=true" \
  -H 'Content-Type: application/json' \
  -d '{"indices": "segments,user_resources,jobs", "include_global_state": false}'

curl -s 'localhost:9200/_cat/indices/segments,user_resources,jobs?h=index,docs.count' \
  > es-backup-baseline.txt

The response reports a state. SUCCESS means the snapshot is complete. PARTIAL means some shards were unavailable and the snapshot is not a usable backup: fix the cluster health and take it again. Keep es-backup-baseline.txt. The document counts it records are how you tell a complete restore from a partial one. To take snapshots on a schedule, create a snapshot lifecycle management policy:

curl -s -X PUT 'localhost:9200/_slm/policy/nightly' \
  -H 'Content-Type: application/json' -d '{
  "schedule": "0 30 2 * * ?",
  "name": "<nightly-{now/d}>",
  "repository": "lilt-es-backup",
  "config": { "indices": "segments,user_resources,jobs", "include_global_state": false },
  "retention": { "expire_after": "30d", "min_count": 5, "max_count": 30 }
}'

Restore from a snapshot

A restore replaces the contents of the indices it targets. Run it in a maintenance window with LILT quiesced, otherwise the indexer writes into the indices while the restore is in progress.

Close the indices you are restoring. A restore cannot write to an open index. Skip this step when you restore into an empty cluster, where the indices do not yet exist:
curl -s -X POST 'localhost:9200/segments,user_resources,jobs/_close'

Restore the snapshot:

curl -s -X POST "localhost:9200/_snapshot/lilt-es-backup/$SNAPSHOT/_restore?wait_for_completion=true" \
  -H 'Content-Type: application/json' \
  -d '{"indices": "segments,user_resources,jobs", "include_global_state": false}'

Reopen the indices:

curl -s -X POST 'localhost:9200/segments,user_resources,jobs/_open'

Confirm that the cluster is healthy and the document counts match your baseline:

curl -s 'localhost:9200/_cluster/health?wait_for_status=green&timeout=120s'
diff es-backup-baseline.txt \
  <(curl -s 'localhost:9200/_cat/indices/segments,user_resources,jobs?h=index,docs.count')

Rebuild the search index from the database

Use this when the segments index is damaged or incomplete and you have no usable snapshot. Read the scope limits at the end of this section before you plan a recovery around this procedure. Run the tool from the indexer pod, which has both database and Elasticsearch access:

kubectl -n $NAMESPACE exec -it deploy/indexer-core -- sh -c \
  './core-services/scripts/lltrun com.lilt.indexer.tools.ReindexMemoryTool reindexOrganization \
   --orgId=ORG_ID --threads=4 --batchSize=1000 \
   --dburl="jdbc:mysql://$DB_HOST:3306/DB_NAME"'

The command runs through sh -c inside single quotes so that $DB_HOST expands in the pod, not in your own shell. Without the single quotes your shell substitutes it first, and it is empty there. Substitute the organization to rebuild for ORG_ID and your database name for DB_NAME. The pod’s own environment supplies the rest. The database user and password come from DB_USER and DB_PASSWORD, and Elasticsearch is reached through ELASTIC_HOST with the default port and scheme, so none of those need a flag.

Pass --dburl even though the other connection details come from the environment. The option reads the DB_URL variable, which the indexer pod does not set — it sets DB_HOST instead. With no --dburl, the tool falls back to jdbc:mysql://localhost:3306/, which fails to reach your database rather than reporting a missing setting.

To rebuild named translation memories instead of a whole organization, use reindexMemory with a comma-separated list of memory IDs:

... ReindexMemoryTool reindexMemory --memoryId=12345,12346 \
   --dburl="jdbc:mysql://$DB_HOST:3306/DB_NAME"

The tool removes each memory’s existing documents from Elasticsearch, then reads the confirmed segments back from the database. Rebuilding a memory is safe to repeat, but it is not additive: if a run is interrupted, that memory is left partly indexed, so run it again to completion.

Scope limits

One organization per run. reindexOrganization takes a single --orgId. To rebuild every organization, run the command once per organization and track which ones finished.
No progress reporting or retries. The tool logs its own progress, and nothing resumes it if it stops.
Only the segments index can be rebuilt. The user_resources and jobs indices are populated as LILT runs, and no tool rebuilds them. This is why a snapshot repository is worth setting up even though the data is derived.

If you lose Elasticsearch with no snapshot, rebuild segments with the preceding procedure and contact LILT support about the remaining indices.

Upgrade Elasticsearch Across Major Versions

​Overview

​Snapshot or rebuild?

​Connect to the cluster

​Register a snapshot repository

​On AWS with an S3 bucket

​On-premise with MinIO

​Verify the repository

​Take a snapshot

​Restore from a snapshot

​Rebuild the search index from the database

​Scope limits

​Related pages

Overview

Snapshot or rebuild?

Connect to the cluster

Register a snapshot repository

On AWS with an S3 bucket

On-premise with MinIO

Verify the repository

Take a snapshot

Restore from a snapshot

Rebuild the search index from the database

Scope limits

Related pages