LILT relies on third-party apps for operation, a majority of which can be upgraded with no concerns. As of LILT version 5.1, the following third-party apps are utlized:
  • Clickhouse
  • Elasticsearch
  • Flannel
  • Grafana
  • Ingress-nginx
  • Istio
  • Localpv-provisioner
  • Mino
  • Mongodb
  • Rabbitmq
  • MySql
  • Prometheus
  • Redis
All apps are designed to upgrade in place with PVC (persistent storage) remaining bound to the new pods. The two apps of most concern are MySql and Minio, which retain all LILT data. The other third-party apps do have persistent data in cache but will be automatically rewritten if an upgrade fails.

​
Upgrade

Upgrade apps by running the following respective helm script (using Clickhouse as an example): 
sh install_scripts/install-clickhouse.sh
This runs a helm upgrade command that adds a new version of the app, allowing for helm rollback if necessary. Be cautious of using helm delete as this removes all previous versions and deletes all persistent data for the app..

​
Backup App Data

As a precaution, data can be backed up for MySql and Minio in case of data loss during upgrades

​
1. Disable Front

As best practice, disable LILT frontaccess so that no data is written to MySql or Minio during backup/restore. Disable front: 
vi lilt/environments/seed_custom_values/values.yaml

# set to disable
front:
  enabled: false

# run install lilt charts
sh install_scripts/install-lilt-charts.sh

​
2. Backup/Restore MySql

Exec to MySql pod and create a db dump file. That file can then be backed up on another local machine: 
# set db password
export db_pass="putthedbpasswordhere"

# create db dump file
kubectl exec --stdin --tty mysql-0 -- /bin/bash -c "mysqldump --host mysql.lilt.svc.cluster.local --user root -p$db_pass lilt_dev > /tmp/mysql_dump.sql"

# copy dump file to local directory
kubectl cp lilt/mysql-0:/tmp/mysql_dump.sql /mysql_dump/mysql_dump.sql
To restore the backup, copy the dump file from local machine to the upgraded MtSql pod and restore: 
# copy from local machine to new pod
kubectl cp /mysql_dump/mysql_dump.sql lilt/mysql-0:/tmp/mysql_dump.sql

# restore backup file
kubectl exec --stdin --tty mysql-0 -- /bin/bash -c "mysql --user root -p$db_pass -D lilt_dev < /tmp/mysql_dump.sql"

# after restore, file extensions need to be converted back to .js
kubectl exec --stdin --tty mysql-0 -- mysql -u root -p$db_pass lilt_dev -e 'INSERT INTO `SequelizeMeta` VALUES ("20211206161036-insertEnableWorkflowsSetting-only-on-prem.js"),("20211208093745-bootstrap-workflows-on-prem.js"),("20211208115112-setDefaultWorkflow-only-on-prem.js"),("20211214100926-filtered-tmx-enable.js"),("20211214102401-enable-filtered-tmx-on-prem-only.js"),("20220303094802-enableJobsListPaginated.js"),("20220311151105-project-list-indices.js"),("20220314130118-LinguistQualities-indices.js"),("20220325124422-drop-qaconfig-srclang-trglang.js"),("20220329223422-create-qa-checks.js"),("20220331145745-add-organizations_isserviceaccount.js"),("20220331223422-create-qa-check-results.js"),("20220418193734-add-connector-tokens.js"),("20220420090333-create-ProjectChangelogs.js"),("20220420092243-create-DocumentChangelogs.js"),("20220427104748-create-mandatoryErrorFlaggingSecondaryReviewCommercial-setting.js"),("20220427104803-create-showPreviousStageCommentsSecondaryReviewCommercial-setting.js"),("20220429223422-recreate-qa-check-results.js"),("20220502094431-translation-done-required-review-stage.js"),("20220504202653-create-SegmentWorkflowStages.js"),("20220505123749-apply-customer-reviewer-role.js"),("20220505132017-create-scheduled-events.js"),("20220509105002-create-enableNextWordSuggestions-setting.js"),("20220510132117-enable-reject-document.js"),("20220511054900-create-neural-batch-translation-reqs-and-jobs.js"),("20220531155509-lagging_linguist_query_indexes.js"),("20220601223420-create-async-checks.js"),("20220601223422-create-async-check-results.js"),("20220602154639-edit-lingustlanguagepairs-locale-constraints.js"),("20220609102900-add-lingustqualities-is-available.js"),("20220614133322-drop-ScheduledEvents_unique_index.js"),("20220627154145-create-SSConfigurations.js"),("20220729021311-create-OrganizationLangPairSamplePercentage.js"),("20220729103939-limit-assignments-to-onboarded-users.js"),("20220809081110-add-index-notifications.js"),("20220817123018-add-MemoryCheckResults_ConfirmedByUserId_createdAt.js"),("20220908205552-add-workflow-group.js"),("20220909010816-add-OrganizationLangPairSamplePercantage-isDeleted.js"),("20220914150537-update-workflow-group.js"),("20220922130312-add-MemoryCheckResults-issueTypes-index.js"),("20221013194559-linguistAvailabilityToggle.js"),("20221013195401-autoAssignDueDates.js"),("20221014115910-create-customer-role.js"),("20221021103702-limit-assignments-to-onboarded-users-defaulted-false.js"),("20221024120044-update-customer-and-cr-roles.js"),("20221114153435-update-default-role-permissions.js"),("20221125151839-create-user-fingerprints.js"),("20221129124055-add-customer-review-sampling-columns.js"),("20221129143620-LinguistOrganization-Review-Allowance.js"),("20221130010816-add-backend_version_to_neural_batch.js"),("20221201010816-add-backend_version_to_neural_update_manager_requests.js"),("20221202163800-preaccepted-segments-remove-segment-id-unique.js"),("20221209130938-control-copy-and-paste-settings.js"),("20221212185204-add-hourly-rate-to-linguist-language-pairs.js"),("20221214123158-create-connector-jobs-index.js"),("20221230143840-viewprojectstats.js"),("20230110140105-add-memory-hashes.js"),("20230127162613-create-workflow-task-reasons.js"),("20230302235253-viewContextualAI.js"),("20230306124148-longer-locales-varchar.js"),("20230306170809-demo-user-metadata-setting.js"),("20230319121009-create-LanguagePairsScalar.js"),("20230330222759-enableConnectorsView.js"),("20230425235845-create-SegmentMemoryMetadata.js"),("20230509155908-create-generated-content.js"),("20230510191322-OrgSettingForTmFuzzyMatchDisablement.js"),("20230516230606-create-JobParameters.js"),("20230606131341-generatedContentPreferences.js"),("20230613125129-create-lilt-create-role.js"),("20230621154632-connectors-organization-update-tables.js")'

​
3. Backup/Restore Minio

Minio uses basic object storage that is persisted with a PVC. The PVC data can simply be backed up and then copied to a new PVC: 
# get minio current PVC
kubectl get pvc -n lilt | grep minio

# output will be similar to this
pvc-f6c7eda9-e7bb-4975-bb40-80c808718d0d

# ssh to worker node and make a copy of the existing pvc
mkdir /backup
cp -a /liltdata/pvc-f6c7eda9-e7bb-4975-bb40-80c808718d0d /backup/pvc-f6c7eda9-e7bb-4975-bb40-80c808718d0d
All relevant data is stored in a subdirectory of the pvc lilt. To restore, copy the old lilt directory to the new pvc. NOTE: do NOT overwrite existing data in the new PVC: 
# get new minio pvc
kubectl get pvc -n lilt | grep minio

# output will be similar to this
pvc-sdj345-d85j-f56r-jgjt-fjhg89458g89df4

# copy old lilt dir to new pvc, DO NOT overwrite existing data
cp -an /backup/pvc-f6c7eda9-e7bb-4975-bb40-80c808718d0d/lilt /liltdata/pvc-sdj345-d85j-f56r-jgjt-fjhg89458g89df4/

​
4. Enable Front

Reenable LILT front: 
vi lilt/environments/seed_custom_values/values.yaml

# set to enable
front:
  enabled: true

# run install lilt charts
sh install_scripts/install-lilt-charts.sh