> ## Documentation Index
> Fetch the complete documentation index at: https://support.lilt.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Upgrade Elasticsearch Across Major Versions

## Overview

LILT runs Elasticsearch under the ECK operator. A LILT release can ship both a
newer operator and a newer Elasticsearch major version, and **the order in
which they are upgraded matters**: a newer operator refuses to manage
Elasticsearch nodes below its supported floor. If the operator is upgraded
first, it stops reconciling the existing cluster — the discovery configuration
goes stale, and any Elasticsearch pod that restarts afterwards can no longer
rejoin the cluster.

This applies to **any existing install whose running Elasticsearch major
version is behind the version shipped with the new release**, regardless of
which LILT version it was installed from.

Newer installers detect this and abort the operator upgrade with:

```
install-elastic-eck-operator: ERROR refusing to upgrade the operator over Elasticsearch <running> (target <shipped>)
```

This page is the procedure to follow when you hit that error, or whenever you
know your upgrade crosses an Elasticsearch major version.

## The Rule

Never skip an Elasticsearch major version, and never upgrade the operator past
what your running Elasticsearch supports. Per major version, the order is:

1. **With the currently installed operator**, upgrade the Elasticsearch
   resource to the **last minor of the running major** (the bridge version —
   for 8.x it is 8.19, shipped as the pinned tag `8.19.20`).
2. Wait until the cluster is `green` and every node reports the bridge
   version.
3. Run the new installer normally: it upgrades the operator, then applies the
   shipped Elasticsearch version, which is now a supported single-major hop.

If your install is **more than one major behind** the shipped version, upgrade
through the intermediate LILT release(s), or contact LILT support — the
installer you have only ships one operator version, and each major must be
bridged under an operator that supports it.

## Prerequisites

* The cluster is `green` before you start:

  ```bash theme={null}
  kubectl -n lilt get elasticsearch eck-cluster
  ```

* The **bridge-version Elasticsearch image is available to your nodes** (in
  your registry or loaded locally). Deliveries that cross an Elasticsearch
  major include the pinned bridge image (currently `8.19.20`) as an additional
  image — verify it is present in your registry before starting.

## Steps (example: 8.x to 9.x)

Run from the installer directory on the master node:

```bash theme={null}
# 1) With the CURRENT (old) operator still installed, move the
#    Elasticsearch resource to the bridge version of the running major,
#    using the pinned bridge tag shipped with your delivery (currently
#    8.19.20). Keep version and image.tag identical.
. install_scripts/utils.sh
VALS=$(resolve_placeholders "eck-cluster/${VALS_FILE:-on-prem-values.yaml}")
helm upgrade eck-cluster "$(chart_ref eck-cluster)" -n lilt \
  -f "$VALS" \
  --server-side true --force-conflicts \
  --set version=8.19.20 --set image.tag=8.19.20

# 2) Wait until health is green and every node reports the bridge version:
kubectl -n lilt get elasticsearch eck-cluster \
  -o jsonpath='{.status.health} {.status.availableNodes} {.status.version}'

# 3) Re-run the installer: it upgrades the operator and then applies the
#    shipped Elasticsearch version.
./install-lilt.sh
```

## Gotchas

1. **Order is everything.** Elasticsearch data is upgraded in place; each step
   is irreversible for the data. Take a backup before starting.
2. **The Elasticsearch StatefulSet uses `updateStrategy: OnDelete`** — a
   `kubectl rollout restart` rolls nothing. The operator handles pod cycling;
   if you must recycle by hand, delete pods one at a time and wait for ready.
3. **Do not downgrade the operator to recover.** It collides with immutable
   StatefulSet fields the newer operator wrote. If you already upgraded the
   operator first and the cluster froze, contact LILT support — the recovery
   (deleting the StatefulSet with `--cascade=orphan` so pods and data survive)
   is easy to get wrong.
4. **Long maintenance windows can outlive registry credentials.** If your
   registry uses short-lived pull tokens, refresh them before the final
   rolling upgrade, or the new-image pull fails and a pod is left waiting on
   an image.

## Verification

```bash theme={null}
kubectl -n lilt get elasticsearch eck-cluster
```

must show `green`, the full node count, and the shipped version. The
Elasticsearch-dependent applications (indexer, search, segment, core-api)
recover on their own once the cluster is reachable again — no restarts needed.