> ## Documentation Index
> Fetch the complete documentation index at: https://support.lilt.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Upgrade Third Party Applications

LILT relies on third-party apps for operation, a majority of which can be upgraded with no concerns. As of LILT version 5.1, the following third-party apps are utilized:

* Clickhouse

* Elasticsearch

* Flannel

* Grafana

* Ingress-nginx

* Istio

* Localpv-provisioner

* Minio

* Mongodb

* Rabbitmq

* MySql

* Prometheus

* Redis

All apps are designed to upgrade in place with PVC (persistent storage) remaining bound to the new pods. The two apps of most concern are `MySql` and `Minio`, which retain all LILT data. The other third-party apps do have persistent data in cache but will be automatically rewritten if an upgrade fails.

## Upgrade

Upgrade apps by running the following respective helm script (using `Clickhouse` as an example):

```bash theme={null}
sh install_scripts/install-clickhouse.sh
```

This runs a `helm upgrade` command that adds a new version of the app, allowing for helm rollback if necessary. Be cautious of using `helm delete` as this removes all previous versions and deletes all persistent data for the app..

## Backup App Data

As a precaution, data can be backed up for `MySql` and `Minio` in case of data loss during upgrades

### 1. Disable Front

As best practice, disable LILT `front`access so that no data is written to `MySql` or `Minio` during backup/restore.

Disable `front`:

```bash theme={null}
vi lilt/environments/seed_custom_values/values.yaml

# set to disable
front:
  enabled: false

# run install lilt charts
sh install_scripts/install-lilt-charts.sh
```

### 2. Backup/Restore MySql

Exec to MySql pod and create a db dump file. That file can then be backed up on another local machine:

```bash theme={null}
# set db password
export db_pass="putthedbpasswordhere"

# create db dump file
kubectl exec --stdin --tty mysql-0 -- /bin/bash -c "mysqldump --host mysql.lilt.svc.cluster.local --user root -p$db_pass lilt_dev > /tmp/mysql_dump.sql"

# copy dump file to local directory
kubectl cp lilt/mysql-0:/tmp/mysql_dump.sql /mysql_dump/mysql_dump.sql
```

To restore the backup, copy the dump file from local machine to the upgraded MtSql pod and restore:

```bash theme={null}
# copy from local machine to new pod
kubectl cp /mysql_dump/mysql_dump.sql lilt/mysql-0:/tmp/mysql_dump.sql

# restore backup file
kubectl exec --stdin --tty mysql-0 -- /bin/bash -c "mysql --user root -p$db_pass -D lilt_dev < /tmp/mysql_dump.sql"

# after restore, file extensions need to be converted back to .js
kubectl exec --stdin --tty mysql-0 -- mysql -u root -p$db_pass lilt_dev -e 'INSERT INTO `SequelizeMeta` VALUES ("20211206161036-insertEnableWorkflowsSetting-only-on-prem.js"),("20211208093745-bootstrap-workflows-on-prem.js"),("20211208115112-setDefaultWorkflow-only-on-prem.js"),("20211214100926-filtered-tmx-enable.js"),("20211214102401-enable-filtered-tmx-on-prem-only.js"),("20220303094802-enableJobsListPaginated.js"),("20220311151105-project-list-indices.js"),("20220314130118-LinguistQualities-indices.js"),("20220325124422-drop-qaconfig-srclang-trglang.js"),("20220329223422-create-qa-checks.js"),("20220331145745-add-organizations_isserviceaccount.js"),("20220331223422-create-qa-check-results.js"),("20220418193734-add-connector-tokens.js"),("20220420090333-create-ProjectChangelogs.js"),("20220420092243-create-DocumentChangelogs.js"),("20220427104748-create-mandatoryErrorFlaggingSecondaryReviewCommercial-setting.js"),("20220427104803-create-showPreviousStageCommentsSecondaryReviewCommercial-setting.js"),("20220429223422-recreate-qa-check-results.js"),("20220502094431-translation-done-required-review-stage.js"),("20220504202653-create-SegmentWorkflowStages.js"),("20220505123749-apply-customer-reviewer-role.js"),("20220505132017-create-scheduled-events.js"),("20220509105002-create-enableNextWordSuggestions-setting.js"),("20220510132117-enable-reject-document.js"),("20220511054900-create-neural-batch-translation-reqs-and-jobs.js"),("20220531155509-lagging_linguist_query_indexes.js"),("20220601223420-create-async-checks.js"),("20220601223422-create-async-check-results.js"),("20220602154639-edit-lingustlanguagepairs-locale-constraints.js"),("20220609102900-add-lingustqualities-is-available.js"),("20220614133322-drop-ScheduledEvents_unique_index.js"),("20220627154145-create-SSConfigurations.js"),("20220729021311-create-OrganizationLangPairSamplePercentage.js"),("20220729103939-limit-assignments-to-onboarded-users.js"),("20220809081110-add-index-notifications.js"),("20220817123018-add-MemoryCheckResults_ConfirmedByUserId_createdAt.js"),("20220908205552-add-workflow-group.js"),("20220909010816-add-OrganizationLangPairSamplePercantage-isDeleted.js"),("20220914150537-update-workflow-group.js"),("20220922130312-add-MemoryCheckResults-issueTypes-index.js"),("20221013194559-linguistAvailabilityToggle.js"),("20221013195401-autoAssignDueDates.js"),("20221014115910-create-customer-role.js"),("20221021103702-limit-assignments-to-onboarded-users-defaulted-false.js"),("20221024120044-update-customer-and-cr-roles.js"),("20221114153435-update-default-role-permissions.js"),("20221125151839-create-user-fingerprints.js"),("20221129124055-add-customer-review-sampling-columns.js"),("20221129143620-LinguistOrganization-Review-Allowance.js"),("20221130010816-add-backend_version_to_neural_batch.js"),("20221201010816-add-backend_version_to_neural_update_manager_requests.js"),("20221202163800-preaccepted-segments-remove-segment-id-unique.js"),("20221209130938-control-copy-and-paste-settings.js"),("20221212185204-add-hourly-rate-to-linguist-language-pairs.js"),("20221214123158-create-connector-jobs-index.js"),("20221230143840-viewprojectstats.js"),("20230110140105-add-memory-hashes.js"),("20230127162613-create-workflow-task-reasons.js"),("20230302235253-viewContextualAI.js"),("20230306124148-longer-locales-varchar.js"),("20230306170809-demo-user-metadata-setting.js"),("20230319121009-create-LanguagePairsScalar.js"),("20230330222759-enableConnectorsView.js"),("20230425235845-create-SegmentMemoryMetadata.js"),("20230509155908-create-generated-content.js"),("20230510191322-OrgSettingForTmFuzzyMatchDisablement.js"),("20230516230606-create-JobParameters.js"),("20230606131341-generatedContentPreferences.js"),("20230613125129-create-lilt-create-role.js"),("20230621154632-connectors-organization-update-tables.js")'
```

### 3. Backup/Restore Minio

`Minio` uses basic object storage that is persisted with a PVC. The PVC data can simply be backed up and then copied to a new PVC:

```bash theme={null}
# get minio current PVC
kubectl get pvc -n lilt | grep minio

# output will be similar to this
pvc-f6c7eda9-e7bb-4975-bb40-80c808718d0d

# ssh to worker node and make a copy of the existing pvc
mkdir /backup
cp -a /liltdata/pvc-f6c7eda9-e7bb-4975-bb40-80c808718d0d /backup/pvc-f6c7eda9-e7bb-4975-bb40-80c808718d0d
```

All relevant data is stored in a subdirectory of the pvc `lilt`. To restore, copy the old `lilt` directory to the new pvc. NOTE: do NOT overwrite existing data in the new PVC:

```bash theme={null}
# get new minio pvc
kubectl get pvc -n lilt | grep minio

# output will be similar to this
pvc-sdj345-d85j-f56r-jgjt-fjhg89458g89df4

# copy old lilt dir to new pvc, DO NOT overwrite existing data
cp -an /backup/pvc-f6c7eda9-e7bb-4975-bb40-80c808718d0d/lilt /liltdata/pvc-sdj345-d85j-f56r-jgjt-fjhg89458g89df4/
```

### 4. Enable Front

Reenable LILT `front`:

```bash theme={null}
vi lilt/environments/seed_custom_values/values.yaml

# set to enable
front:
  enabled: true

# run install lilt charts
sh install_scripts/install-lilt-charts.sh
```