Upgrade Third Party Applications

Overview

LILT depends on third-party applications for storage, messaging, networking and scheduling. The installer upgrades all of them as part of a LILT upgrade, so you do not normally upgrade one on its own. Read Which applications need care before you upgrade. Most components upgrade in place with no attention, but four do not, and two of those stop the installer rather than risk your data.

What LILT installs

An on-premise installation includes the following. An AWS EKS installation replaces MySQL with Amazon RDS, MinIO with Amazon S3, and MetalLB, Flannel and the local persistent-volume provisioner with the equivalent AWS services.

Function	Applications
Databases and storage	MySQL, MongoDB, ClickHouse, MinIO
Search and vectors	Elasticsearch (via the ECK operator), Qdrant
Cache and messaging	Dragonfly, RabbitMQ
Database proxy	ProxySQL
Networking	Istio, Gateway API, MetalLB, Flannel
Scheduling and delivery	Argo CD, Argo Workflows, Argo Rollouts, Ray
Secrets	HashiCorp Vault, External Secrets Operator
Monitoring	Prometheus, metrics-server
GPU	NVIDIA GPU Operator or NVIDIA device plugin

Earlier releases of LILT also installed Grafana, ingress-nginx and Redis. Istio and the Gateway API replaced ingress-nginx, and Dragonfly replaced Redis. If you are upgrading from a release that installed them, the upgrade removes them.

Which applications need care

Most of these applications upgrade in place, keeping their persistent volumes bound to the new pods, and need no attention from you. Four are different.

Do not assume that a failed upgrade of a data store repairs itself. Only data that LILT can regenerate is rebuilt automatically. MySQL, MinIO and Qdrant hold data that exists nowhere else.

MySQL and MinIO hold all of your LILT data. On an on-premise installation, back both up before you upgrade. See Back up MySQL and MinIO. Qdrant cannot skip a minor version, and cannot be rebuilt from the database. If your installation is more than one minor version behind, the installer stops. See Upgrade Qdrant. Elasticsearch cannot cross a major version in one step. If your installation is a major version behind, the installer stops. See Upgrade Elasticsearch Across Major Versions. Elasticsearch data is derived from the LILT database, so it can be rebuilt, but only in part: see Back Up and Restore Elasticsearch, which also covers registering the snapshot repository the installer does not create.

Upgrade

Run the installer, which upgrades LILT and every third-party application together:

sh install-lilt.sh

On an AWS EKS installation, run sh install-lilt-eks.sh instead. To upgrade a single application, run its install script. Using ClickHouse as an example:

sh install_scripts/install-clickhouse.sh

Each script runs helm upgrade, which adds a new revision of the release and leaves the previous one available to helm rollback.

Do not use helm delete or helm uninstall to recover from a failed upgrade. Uninstalling removes every revision of the release, and for most of these applications it deletes the persistent volume claim with it. Use helm rollback instead.

Back up MySQL and MinIO

This procedure applies to an on-premise installation, where MySQL and MinIO run in the cluster. An AWS EKS installation keeps this data in Amazon RDS and Amazon S3, so back those up with RDS snapshots and S3 object versioning instead. This procedure does not apply there.

1. Stop writes

Disable LILT front so that nothing writes to MySQL or MinIO while you copy the data:

vi lilt/environments/seed_custom_values/values.yaml

# Set front to disabled:
front:
  enabled: false

sh install_scripts/install-lilt-charts.sh

2. Back up MySQL

Record which database migrations have been applied. You need this list to restore, and it is specific to your installation, so capture it now rather than reusing one from another system:

export db_pass="DB_PASSWORD"

kubectl exec -n lilt mysql-0 -- \
  mysql --user root -p$db_pass lilt_dev -N -e 'SELECT name FROM SequelizeMeta' \
  > sequelize_meta.txt

Substitute your database password for DB_PASSWORD. Create the dump and copy it off the pod:

kubectl exec -n lilt mysql-0 -- /bin/bash -c \
  "mysqldump --host mysql.lilt.svc.cluster.local --user root -p$db_pass lilt_dev > /tmp/mysql_dump.sql"

kubectl cp lilt/mysql-0:/tmp/mysql_dump.sql ./mysql_dump.sql

3. Back up MinIO

MinIO stores objects on a persistent volume, so you copy the volume’s contents. All LILT data is under the lilt subdirectory of the claim.

kubectl get pvc -n lilt | grep minio

The output names the volume, for example pvc-f6c7eda9-e7bb-4975-bb40-80c808718d0d. Connect to the worker node and copy it:

mkdir /backup
cp -a /liltdata/PVC_NAME /backup/PVC_NAME

Substitute the volume name for PVC_NAME.

4. Re-enable front

vi lilt/environments/seed_custom_values/values.yaml

# Set front to enabled:
front:
  enabled: true

sh install_scripts/install-lilt-charts.sh

Restore MySQL and MinIO

Restore MySQL

Copy the dump back to the pod and load it:

kubectl cp ./mysql_dump.sql lilt/mysql-0:/tmp/mysql_dump.sql

kubectl exec -n lilt mysql-0 -- /bin/bash -c \
  "mysql --user root -p$db_pass -D lilt_dev < /tmp/mysql_dump.sql"

Restore the migration list you captured in step 2. The dump records migration filenames with the wrong extension, so LILT re-runs migrations that have already been applied unless you restore the list:

while read -r name; do
  kubectl exec -n lilt mysql-0 -- \
    mysql -u root -p$db_pass lilt_dev \
    -e "INSERT IGNORE INTO \`SequelizeMeta\` VALUES (\"$name\")"
done < sequelize_meta.txt

Restore MinIO

Copy the lilt directory from your backup into the new volume. Do not overwrite data that the new volume already holds:

kubectl get pvc -n lilt | grep minio

cp -an /backup/OLD_PVC_NAME/lilt /liltdata/NEW_PVC_NAME/

Substitute the old and new volume names for OLD_PVC_NAME and NEW_PVC_NAME.

​Overview

​What LILT installs

​Which applications need care

​Upgrade

​Back up MySQL and MinIO

​1. Stop writes

​2. Back up MySQL

​3. Back up MinIO

​4. Re-enable front

​Restore MySQL and MinIO

​Restore MySQL

​Restore MinIO

​Related pages

Overview

What LILT installs

Which applications need care

Upgrade

Back up MySQL and MinIO

1. Stop writes

2. Back up MySQL

3. Back up MinIO

4. Re-enable front

Restore MySQL and MinIO

Restore MySQL

Restore MinIO

Related pages