Upgrade Elasticsearch Across Major Versions

Overview

LILT runs Elasticsearch under the ECK operator. A LILT release can ship both a newer operator and a newer Elasticsearch major version, and the order in which they are upgraded matters: a newer operator refuses to manage Elasticsearch nodes below its supported floor. If the operator is upgraded first, it stops reconciling the existing cluster — the discovery configuration goes stale, and any Elasticsearch pod that restarts afterwards can no longer rejoin the cluster. This applies to any existing install whose running Elasticsearch major version is behind the version shipped with the new release, regardless of which LILT version it was installed from. Newer installers detect this and abort the operator upgrade with:

install-elastic-eck-operator: ERROR refusing to upgrade the operator over Elasticsearch <running> (target <shipped>)

This page is the procedure to follow when you hit that error, or whenever you know your upgrade crosses an Elasticsearch major version.

The Rule

Never skip an Elasticsearch major version, and never upgrade the operator past what your running Elasticsearch supports. Per major version, the order is:

With the currently installed operator, upgrade the Elasticsearch resource to the last minor of the running major (the bridge version — for 8.x it is 8.19, shipped as the pinned tag 8.19.20).
Wait until the cluster is green and every node reports the bridge version.
Run the new installer normally: it upgrades the operator, then applies the shipped Elasticsearch version, which is now a supported single-major hop.

If your install is more than one major behind the shipped version, upgrade through the intermediate LILT release(s), or contact LILT support — the installer you have only ships one operator version, and each major must be bridged under an operator that supports it.

Prerequisites

The cluster is green before you start:

kubectl -n lilt get elasticsearch eck-cluster

The bridge-version Elasticsearch image is available to your nodes (in your registry or loaded locally). Deliveries that cross an Elasticsearch major include the pinned bridge image (currently 8.19.20) as an additional image — verify it is present in your registry before starting.

Steps (example: 8.x to 9.x)

Run from the installer directory on the master node:

# 1) With the CURRENT (old) operator still installed, move the
#    Elasticsearch resource to the bridge version of the running major,
#    using the pinned bridge tag shipped with your delivery (currently
#    8.19.20). Keep version and image.tag identical.
. install_scripts/utils.sh
VALS=$(resolve_placeholders "eck-cluster/${VALS_FILE:-on-prem-values.yaml}")
helm upgrade eck-cluster "$(chart_ref eck-cluster)" -n lilt \
  -f "$VALS" \
  --server-side true --force-conflicts \
  --set version=8.19.20 --set image.tag=8.19.20

# 2) Wait until health is green and every node reports the bridge version:
kubectl -n lilt get elasticsearch eck-cluster \
  -o jsonpath='{.status.health} {.status.availableNodes} {.status.version}'

# 3) Re-run the installer: it upgrades the operator and then applies the
#    shipped Elasticsearch version.
./install-lilt.sh

Gotchas

Order is everything. Elasticsearch data is upgraded in place; each step is irreversible for the data. Take a backup before starting.
The Elasticsearch StatefulSet uses updateStrategy: OnDelete — a kubectl rollout restart rolls nothing. The operator handles pod cycling; if you must recycle by hand, delete pods one at a time and wait for ready.
Do not downgrade the operator to recover. It collides with immutable StatefulSet fields the newer operator wrote. If you already upgraded the operator first and the cluster froze, contact LILT support — the recovery (deleting the StatefulSet with --cascade=orphan so pods and data survive) is easy to get wrong.
Long maintenance windows can outlive registry credentials. If your registry uses short-lived pull tokens, refresh them before the final rolling upgrade, or the new-image pull fails and a pod is left waiting on an image.

Verification

kubectl -n lilt get elasticsearch eck-cluster

must show green, the full node count, and the shipped version. The Elasticsearch-dependent applications (indexer, search, segment, core-api) recover on their own once the cluster is reachable again — no restarts needed.

​Overview

​The Rule

​Prerequisites

​Steps (example: 8.x to 9.x)

​Gotchas

​Verification