> ## Documentation Index
> Fetch the complete documentation index at: https://support.lilt.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Back Up and Restore Elasticsearch

## Overview

LILT runs Elasticsearch under the Elastic Cloud on Kubernetes (ECK) operator.
Elasticsearch holds the indices behind translation-memory search, user resources
and job search.

The installer does not configure a snapshot repository. Until you register one,
Elasticsearch has nowhere to write a backup to, so a new installation has no
backup even though the upgrade procedures tell you to take one. This article
covers registering a repository, taking and restoring snapshots, and rebuilding
the search index from the LILT database.

For the upgrade itself, see [Upgrade Elasticsearch Across Major
Versions](/kb/upgrade-elasticsearch).

<Note>
  Every command on this page assumes the LILT namespace is `lilt`. If you
  installed into a different namespace, substitute it for `NAMESPACE`.
</Note>

## Snapshot or rebuild?

Everything LILT stores in Elasticsearch is derived from the LILT database, so a
lost Elasticsearch cluster is a rebuild rather than permanent data loss. One
detail decides which path you take:

|                                      | Restore a snapshot | Rebuild from the database     |
| ------------------------------------ | ------------------ | ----------------------------- |
| Time to complete                     | Minutes            | Hours on a large installation |
| Rebuilds the `segments` index        | Yes                | Yes                           |
| Rebuilds `user_resources` and `jobs` | Yes                | No tool exists                |
| Needs setup beforehand               | Yes, a repository  | No                            |

Set up snapshots even though the data is derived. A rebuild covers only the
`segments` index, so it is a partial recovery, and it is much slower. Use a
rebuild for a damaged or incomplete `segments` index, and a snapshot for the
loss of a cluster.

## Connect to the cluster

Every `curl` command on this page reaches Elasticsearch through a port forward.
Target a pod rather than a service: LILT applications reach Elasticsearch
through a service of type `ExternalName`, which `kubectl port-forward` cannot
use.

```bash theme={null}
NAMESPACE=lilt
ESPOD=$(kubectl -n $NAMESPACE get pods -l common.k8s.elastic.co/type=elasticsearch \
  -o jsonpath='{.items[0].metadata.name}')
kubectl -n $NAMESPACE port-forward "pod/$ESPOD" 9200:9200 &
```

Leave this running while you work, and stop it when you finish. The shipped
cluster runs with TLS disabled, because Istio carries the transport security, so
these commands need no credentials. That is also a reason not to leave the port
forward open.

## Register a snapshot repository

Elasticsearch writes backups to a *snapshot repository*, which you register
once.

Elasticsearch includes S3 support, so both procedures that follow use the `s3`
repository type. A shared-filesystem repository is not an option: the
Elasticsearch volumes are `ReadWriteOnce` and a filesystem repository must be
mounted on every node in the cluster.

<Warning>
  Add the credential settings to `eck-cluster/on-prem-values.yaml` or
  `eck-cluster/eks-values.yaml`. A LILT release replaces those files, so
  re-apply the settings after each upgrade and confirm that snapshots still
  succeed. The repository itself is stored in the Elasticsearch cluster state
  and survives an upgrade; the credentials that make it usable do not.
</Warning>

### On AWS with an S3 bucket

Your installation already has an AWS identity for S3: `install-s3-irsa.sh`
creates the `lilt-s3-workload` service account and binds it to the
app-workload IAM role. Run Elasticsearch as that service account rather than
creating a second role.

In `eck-cluster/eks-values.yaml`:

```yaml theme={null}
serviceAccount:
  create: false
  name: lilt-s3-workload
```

This works under both of the mechanisms the installer supports, so you do not
need to know which one your cluster uses.

<Warning>
  Do not add an `eks.amazonaws.com/role-arn` annotation to the service account.
  That annotation is correct only when `IAM_AUTH_METHOD=irsa`. When your
  installation uses EKS Pod Identity, `install-s3-irsa.sh` deliberately
  removes the annotation, because a leftover one takes precedence over the Pod
  Identity binding and sends the AWS SDK to a web identity provider your
  account may not have. Naming the service account, as shown, is correct
  either way.
</Warning>

The app-workload role's policy has to allow the snapshot bucket. It is scoped
to the buckets LILT itself uses, so if you follow the recommendation to keep
snapshots in a separate bucket, add that bucket to the role's policy first.
Reusing the service account gives Elasticsearch the identity; it does not
widen what that identity may do.

If no AWS identity is available, store access keys in a Kubernetes secret
instead:

```bash theme={null}
kubectl -n $NAMESPACE create secret generic es-s3-credentials \
  --from-literal=s3.client.default.access_key='ACCESS_KEY' \
  --from-literal=s3.client.default.secret_key='SECRET_KEY'
```

Then reference the secret in `eck-cluster/eks-values.yaml`:

```yaml theme={null}
secureSettings:
  - secretName: es-s3-credentials
```

The two key names are exact. They are Elasticsearch keystore entries, so a
different name is ignored.

Apply the change. A component script does not read `install.env` — only
`install-lilt.sh` and `install-lilt-eks.sh` load that file — so load it
yourself and set `VALS_FILE` on AWS:

```bash theme={null}
set -a; . ./install.env; set +a
VALS_FILE=eks-values.yaml sh install_scripts/install-elastic-eck-cluster.sh
```

On an on-premise installation, omit `VALS_FILE`.

<Warning>
  Do not skip `install.env`. With `DNS_DOMAIN` unset the script stops with
  `ERROR: DNS_DOMAIN is required`. If `DNS_DOMAIN` happens to be exported but
  `REGISTRY_BASE` is not, the values resolve to a LILT-internal registry host
  your cluster cannot pull from and the script completes anyway, leaving
  Elasticsearch unable to start its image. The check that catches a missing
  `REGISTRY_BASE` lives in the entrypoint, not in the component script.
</Warning>

The operator loads the credentials without restarting the cluster.

Register the repository:

```bash theme={null}
curl -s -X PUT 'localhost:9200/_snapshot/lilt-es-backup' \
  -H 'Content-Type: application/json' -d '{
  "type": "s3",
  "settings": { "bucket": "BUCKET_NAME", "client": "default" }
}'
```

### On-premise with MinIO

An on-premise installation has no S3, but it does include MinIO. Point the same
repository type at it, and set an endpoint.

Create the credentials secret as described in the preceding section, using your
MinIO access key pair, then register the repository:

```bash theme={null}
curl -s -X PUT 'localhost:9200/_snapshot/lilt-es-backup' \
  -H 'Content-Type: application/json' -d '{
  "type": "s3",
  "settings": {
    "bucket": "lilt-es-snapshots",
    "endpoint": "minio.lilt.svc.cluster.local:9000",
    "protocol": "http",
    "path_style_access": true,
    "client": "default"
  }
}'
```

Create the bucket first with `mc mb`.

<Warning>
  Use a separate bucket, not the `lilt` bucket. The `lilt` bucket holds your
  translation documents, so snapshots stored there share a failure domain with
  the data they protect. Treat a MinIO snapshot as a convenience copy, and
  replicate it to storage outside the cluster for disaster recovery.
</Warning>

### Verify the repository

A repository you have never verified is not a backup. Check that every node can
write to it:

```bash theme={null}
curl -s -X POST 'localhost:9200/_snapshot/lilt-es-backup/_verify'
```

A successful response lists the cluster nodes. A `403` or a
`repository_verification_exception` means the credentials or the bucket
permissions are wrong, and a snapshot fails at the moment you need it.

## Take a snapshot

Take a snapshot before every upgrade. Each version step changes the data on disk
and cannot be reversed.

```bash theme={null}
SNAPSHOT="pre-upgrade-$(date +%Y%m%d-%H%M%S)"
curl -s -X PUT "localhost:9200/_snapshot/lilt-es-backup/$SNAPSHOT?wait_for_completion=true" \
  -H 'Content-Type: application/json' \
  -d '{"indices": "segments,user_resources,jobs", "include_global_state": false}'

curl -s 'localhost:9200/_cat/indices/segments,user_resources,jobs?h=index,docs.count' \
  > es-backup-baseline.txt
```

The response reports a `state`. `SUCCESS` means the snapshot is complete.
`PARTIAL` means some shards were unavailable and the snapshot is not a usable
backup: fix the cluster health and take it again.

Keep `es-backup-baseline.txt`. The document counts it records are how you tell a
complete restore from a partial one.

To take snapshots on a schedule, create a snapshot lifecycle management policy:

```bash theme={null}
curl -s -X PUT 'localhost:9200/_slm/policy/nightly' \
  -H 'Content-Type: application/json' -d '{
  "schedule": "0 30 2 * * ?",
  "name": "<nightly-{now/d}>",
  "repository": "lilt-es-backup",
  "config": { "indices": "segments,user_resources,jobs", "include_global_state": false },
  "retention": { "expire_after": "30d", "min_count": 5, "max_count": 30 }
}'
```

## Restore from a snapshot

A restore replaces the contents of the indices it targets. Run it in a
maintenance window with LILT quiesced, otherwise the indexer writes into the
indices while the restore is in progress.

1. Close the indices you are restoring. A restore cannot write to an open index.
   Skip this step when you restore into an empty cluster, where the indices do
   not yet exist:

   ```bash theme={null}
   curl -s -X POST 'localhost:9200/segments,user_resources,jobs/_close'
   ```

2. Restore the snapshot:

   ```bash theme={null}
   curl -s -X POST "localhost:9200/_snapshot/lilt-es-backup/$SNAPSHOT/_restore?wait_for_completion=true" \
     -H 'Content-Type: application/json' \
     -d '{"indices": "segments,user_resources,jobs", "include_global_state": false}'
   ```

3. Reopen the indices:

   ```bash theme={null}
   curl -s -X POST 'localhost:9200/segments,user_resources,jobs/_open'
   ```

4. Confirm that the cluster is healthy and the document counts match your
   baseline:

   ```bash theme={null}
   curl -s 'localhost:9200/_cluster/health?wait_for_status=green&timeout=120s'
   diff es-backup-baseline.txt \
     <(curl -s 'localhost:9200/_cat/indices/segments,user_resources,jobs?h=index,docs.count')
   ```

## Rebuild the search index from the database

Use this when the `segments` index is damaged or incomplete and you have no
usable snapshot. Read the scope limits at the end of this section before you
plan a recovery around this procedure.

Run the tool from the indexer pod, which has both database and Elasticsearch
access:

```bash theme={null}
kubectl -n $NAMESPACE exec -it deploy/indexer-core -- sh -c \
  './core-services/scripts/lltrun com.lilt.indexer.tools.ReindexMemoryTool reindexOrganization \
   --orgId=ORG_ID --threads=4 --batchSize=1000 \
   --dburl="jdbc:mysql://$DB_HOST:3306/DB_NAME"'
```

The command runs through `sh -c` inside single quotes so that `$DB_HOST`
expands in the pod, not in your own shell. Without the single quotes your shell
substitutes it first, and it is empty there.

Substitute the organization to rebuild for `ORG_ID` and your database name for
`DB_NAME`.

The pod's own environment supplies the rest. The database user and password
come from `DB_USER` and `DB_PASSWORD`, and Elasticsearch is reached through
`ELASTIC_HOST` with the default port and scheme, so none of those need a flag.

<Warning>
  Pass `--dburl` even though the other connection details come from the
  environment. The option reads the `DB_URL` variable, which the indexer pod
  does not set — it sets `DB_HOST` instead. With no `--dburl`, the tool falls
  back to `jdbc:mysql://localhost:3306/`, which fails to reach your database
  rather than reporting a missing setting.
</Warning>

To rebuild named translation memories instead of a whole organization, use
`reindexMemory` with a comma-separated list of memory IDs:

```bash theme={null}
... ReindexMemoryTool reindexMemory --memoryId=12345,12346 \
   --dburl="jdbc:mysql://$DB_HOST:3306/DB_NAME"
```

The tool removes each memory's existing documents from Elasticsearch, then reads
the confirmed segments back from the database. Rebuilding a memory is safe to
repeat, but it is not additive: if a run is interrupted, that memory is left
partly indexed, so run it again to completion.

### Scope limits

* **One organization per run.** `reindexOrganization` takes a single `--orgId`.
  To rebuild every organization, run the command once per organization and track
  which ones finished.
* **No progress reporting or retries.** The tool logs its own progress, and
  nothing resumes it if it stops.
* **Only the `segments` index can be rebuilt.** The `user_resources` and `jobs`
  indices are populated as LILT runs, and no tool rebuilds them. This is why a
  snapshot repository is worth setting up even though the data is derived.

If you lose Elasticsearch with no snapshot, rebuild `segments` with the
preceding procedure and contact LILT support about the remaining indices.

## Related pages

* [Upgrade Elasticsearch Across Major Versions](/kb/upgrade-elasticsearch)