> ## Documentation Index
> Fetch the complete documentation index at: https://support.lilt.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Upgrade Third Party Applications

## Overview

LILT depends on third-party applications for storage, messaging, networking and
scheduling. The installer upgrades all of them as part of a LILT upgrade, so you
do not normally upgrade one on its own.

Read [Which applications need care](#which-applications-need-care) before you
upgrade. Most components upgrade in place with no attention, but four do not,
and two of those stop the installer rather than risk your data.

## What LILT installs

An on-premise installation includes the following. An AWS EKS installation
replaces MySQL with Amazon RDS, MinIO with Amazon S3, and MetalLB, Flannel and
the local persistent-volume provisioner with the equivalent AWS services.

| Function | Applications |
| - | - |
| Databases and storage | MySQL, MongoDB, ClickHouse, MinIO |
| Search and vectors | Elasticsearch (via the ECK operator), Qdrant |
| Cache and messaging | Dragonfly, RabbitMQ |
| Database proxy | ProxySQL |
| Networking | Istio, Gateway API, MetalLB, Flannel |
| Scheduling and delivery | Argo CD, Argo Workflows, Argo Rollouts, Ray |
| Secrets | HashiCorp Vault, External Secrets Operator |
| Monitoring | Prometheus, metrics-server |
| GPU | NVIDIA GPU Operator or NVIDIA device plugin |

<Note>
  Earlier releases of LILT also installed Grafana, ingress-nginx and Redis.
  Istio and the Gateway API replaced ingress-nginx, and Dragonfly replaced
  Redis. If you are upgrading from a release that installed them, the upgrade
  removes them.
</Note>

## Which applications need care

Most of these applications upgrade in place, keeping their persistent volumes
bound to the new pods, and need no attention from you. Four are different.

<Warning>
  Do not assume that a failed upgrade of a data store repairs itself. Only data
  that LILT can regenerate is rebuilt automatically. MySQL, MinIO and Qdrant
  hold data that exists nowhere else.
</Warning>

**MySQL and MinIO hold all of your LILT data.** On an on-premise installation,
back both up before you upgrade. See [Back up MySQL and
MinIO](#back-up-mysql-and-minio).

**Qdrant cannot skip a minor version, and cannot be rebuilt from the
database.** If your installation is more than one minor version behind, the
installer stops. See [Upgrade Qdrant](/self-managed/v6.1/upgrade-qdrant).

**Elasticsearch cannot cross a major version in one step.** If your
installation is a major version behind, the installer stops. See [Upgrade
Elasticsearch Across Major Versions](/self-managed/v6.1/upgrade-elasticsearch). Elasticsearch
data is derived from the LILT database, so it can be rebuilt, but only in part:
see [Back Up and Restore
Elasticsearch](/self-managed/v6.1/backup-and-restore-elasticsearch), which also covers
registering the snapshot repository the installer does not create.

## Upgrade

Run the installer, which upgrades LILT and every third-party application
together:

```bash theme={null}
sh install-lilt.sh
```

On an AWS EKS installation, run `sh install-lilt-eks.sh` instead.

To upgrade a single application, run its install script. Using ClickHouse as an
example:

```bash theme={null}
sh install_scripts/install-clickhouse.sh
```

Each script runs `helm upgrade`, which adds a new revision of the release and
leaves the previous one available to `helm rollback`.

<Warning>
  Do not use `helm delete` or `helm uninstall` to recover from a failed
  upgrade. Uninstalling removes every revision of the release, and for most of
  these applications it deletes the persistent volume claim with it. Use
  `helm rollback` instead.
</Warning>

## Back up MySQL and MinIO

This procedure applies to an on-premise installation, where MySQL and MinIO run
in the cluster. An AWS EKS installation keeps this data in Amazon RDS and Amazon
S3, so back those up with RDS snapshots and S3 object versioning instead. This
procedure does not apply there.

### 1. Stop writes

Disable LILT `front` so that nothing writes to MySQL or MinIO while you copy
the data:

```bash theme={null}
vi lilt/environments/seed_custom_values/values.yaml

# Set front to disabled:
front:
  enabled: false

sh install_scripts/install-lilt-charts.sh
```

### 2. Back up MySQL

Record which database migrations have been applied. You need this list to
restore, and it is specific to your installation, so capture it now rather than
reusing one from another system:

```bash theme={null}
export db_pass="DB_PASSWORD"

kubectl exec -n lilt mysql-0 -- \
  mysql --user root -p$db_pass lilt_dev -N -e 'SELECT name FROM SequelizeMeta' \
  > sequelize_meta.txt
```

Substitute your database password for `DB_PASSWORD`.

Create the dump and copy it off the pod:

```bash theme={null}
kubectl exec -n lilt mysql-0 -- /bin/bash -c \
  "mysqldump --host mysql.lilt.svc.cluster.local --user root -p$db_pass lilt_dev > /tmp/mysql_dump.sql"

kubectl cp lilt/mysql-0:/tmp/mysql_dump.sql ./mysql_dump.sql
```

### 3. Back up MinIO

MinIO stores objects on a persistent volume, so you copy the volume's contents.
All LILT data is under the `lilt` subdirectory of the claim.

```bash theme={null}
kubectl get pvc -n lilt | grep minio
```

The output names the volume, for example
`pvc-f6c7eda9-e7bb-4975-bb40-80c808718d0d`. Connect to the worker node and copy
it:

```bash theme={null}
mkdir /backup
cp -a /liltdata/PVC_NAME /backup/PVC_NAME
```

Substitute the volume name for `PVC_NAME`.

### 4. Re-enable front

```bash theme={null}
vi lilt/environments/seed_custom_values/values.yaml

# Set front to enabled:
front:
  enabled: true

sh install_scripts/install-lilt-charts.sh
```

## Restore MySQL and MinIO

### Restore MySQL

Copy the dump back to the pod and load it:

```bash theme={null}
kubectl cp ./mysql_dump.sql lilt/mysql-0:/tmp/mysql_dump.sql

kubectl exec -n lilt mysql-0 -- /bin/bash -c \
  "mysql --user root -p$db_pass -D lilt_dev < /tmp/mysql_dump.sql"
```

Restore the migration list you captured in step 2. The dump records migration
filenames with the wrong extension, so LILT re-runs migrations that have already
been applied unless you restore the list:

```bash theme={null}
while read -r name; do
  kubectl exec -n lilt mysql-0 -- \
    mysql -u root -p$db_pass lilt_dev \
    -e "INSERT IGNORE INTO \`SequelizeMeta\` VALUES (\"$name\")"
done < sequelize_meta.txt
```

### Restore MinIO

Copy the `lilt` directory from your backup into the new volume. Do not
overwrite data that the new volume already holds:

```bash theme={null}
kubectl get pvc -n lilt | grep minio

cp -an /backup/OLD_PVC_NAME/lilt /liltdata/NEW_PVC_NAME/
```

Substitute the old and new volume names for `OLD_PVC_NAME` and `NEW_PVC_NAME`.

## Related pages

* [Upgrade Elasticsearch Across Major Versions](/self-managed/v6.1/upgrade-elasticsearch)
* [Back Up and Restore Elasticsearch](/self-managed/v6.1/backup-and-restore-elasticsearch)
* [Upgrade Qdrant](/self-managed/v6.1/upgrade-qdrant)
* [Upgrade LILT](/self-managed/v6.1/upgrade-system)