​
Overview

This article walks the installer through the process of upgrading an existing on-premise system. Given an available Kubernetes cluster, this document guides you through a partially-automated install process to deploy and configure LILT in your environment.

​
Required Tools

You will need to use SSH to connect to the systems.
  • On the Mac or Linux, you can open a terminal window and use ssh.
  • On Windows, you can use the PuTTY ssh client.
You will also need a web browser to browse the website post installation and verification.

​
Installer Privileges

Many sections of this article include text formatted to indicate console input and output. The commands are prefixed with a $ or # depending on if the administrator should run them as root or not. Additionally, to assist the user to know which machine to run commands on, the prefix node, master, and gpu will be given when the machine should be switched. All following commands after a switch should use those.

​
Run commands as root

sudo su -

​
Prerequisites

This upgrade document is only valid for an existing on-premise install previously installed using on-prem-installer scripts.

​
Data Location

Ensure that the new/updated Docker images and RPM packages are mounted and available to all of the nodes (master, worker, and GPU). In the typical installation, the system administrator will receive this data from LILT prior to installation.

​
Backup

Backup up the current running system prior to upgrade. This backup is highly recommended, and will be used to restore projects, documents and memory content in case of any errors.

​
Step 1: Master Node

Login to the k8s-master node and follow the below steps. Note that the following steps have to be performed as root (sudo su -). Backup Install Directory Backup previous Install Directory (structure might be slightly different than the following example): 
mkdir -p /install_dir_backup
cp -r /install_dir/ /install_dir_backup
Database Backup mysql database: 
# this will dump a file to the mysql pod temp memory
kubectl exec --stdin --tty mysql-0 -- /bin/bash -c "mysqldump --host mysql.lilt.svc.cluster.local --user root -p<password> lilt_dev > /tmp/mysql_dump.sql"

# copy dump file from mysql pod to local directory on master node
kubectl cp lilt/mysql-0:/tmp/mysql_dump.sql /mysql_dump/mysql_dump.sql
Minio (optional) Take note of the existing minio pvc (pvc-eiurty3794t679eryt495ytotoer) for future commands: 
# get minio pvc
kubectl get pvc | grep minio
Example output: 
minio  Bound  pvc-80e6a0a5-9861-42ac-9e3f-ee6c7d439914   400Gi      RWO   openebs-hostpath

​
Step 2: Worker Node

Login to the k8s-worker node and follow the below steps. Note that the following steps have to be performed as root (sudo su -). Persistant Volumes Backup storage volumes (pvcs) managed by openebs mount folders, default directory is /liltdata: 
# make backup dir
mkdir /lildtata_backup

# copy all pvcs
cp -r /liltdata/ /liltdata_backup

​
Upgrade Preparation

​
Step 1: Import New Images and Installer Packages

Download new images from remote s3 bucket and/or installer package. Details describing this process are documented in https://lilt.atlassian.net/wiki/spaces/SMD/pages/202997821/Install+System+Rocky+Linux+8#Prepare-the-LILT-Platform
NOTE: ensure that required images for each specific node are imported/loaded via containerd and that the installer package is correctly unpacked per steps in the documentation.

​
Step 2: Update Custom Values

If changes were made to the default values in an existing install, compare files and modify as required (see the next section). 
diff -y /old_releaase/lilt/values.yaml  /new_release/lilt/values.yaml
In previous versions of LILT, values were spread across lilt/values.yaml and lilt/environment/lilt/values.yaml, which made it hard to differentiate what is a customer-centric vs default. In subsequent release versions, default values lilt/values.yamlshould remain default, and custom-values set in lilt/environment/lilt/values.yaml. As an example, the first file is default values.yaml: 
# values.yaml
front:
  enabled: true
  onpremValues:
    # Image credentials secrets to fetch the docker image
    imagePullSecrets:
    - name: docker-registry-secret
    # Create new image credentials, used when we aren't using local docker registry
    createimageCredentials:
      enabled: false
      name: docker-registry-secret
    # Enable biConnector(analytics-api), verify the port from biConnector configs
    biconnector:
      enabled: false
      Port: 8080
    # tls enabled, using dummy certificate for bare-tmp.lilt.com, please use new certs if available
    tls:
      enabled: true
      # set to true if tls secret exists
      existingSecret: false
      # Enter secret name
      secretName: front-app-tls
    env:
      # Theme configurations
      THEME_HEADER_BGCOLOR: "f0f0f0"
      THEME_HEADER_COLOR: "333333"
      THEME_HEADER_FONTSIZE: "12px"
      THEME_HEADER_TEXT: "My Company"
      THEME_FOOTER_BGCOLOR: "f0f0f0"
      THEME_FOOTER_COLOR: "333333"
      THEME_FOOTER_FONTSIZE: "12px"
      THEME_FOOTER_TEXT: "Test Installation Footer"
Add custom values for THEME_HEADER_TEXT: "My Company", update lilt/environment/lilt/values.yaml as in the following: 
# lilt/environment/lilt/values.yaml
front:
  onpremValues:
    env:
      THEME_HEADER_TEXT: "My Company"
NOTE: this is a one-time migration, no necessary to repeat after custom values are set.

​
Step 3: Update Custom Secrets

Verify custom secrets, no changes required unless specifically mentioned. The customer-centric secrets file is located at: lilt/environment/lilt/secrets.yaml

​
Step 4: Environment Setup (Optional)

For multiple environments, create customer-centric values.yaml and secrets.yaml inside each respective environments folder as required.

​
Upgrade LILT

After all new images, install files and custom environment variables have been set, run install LILT script.

​
Step 1: Master Node

Login to the k8s-master node and follow the below steps. Note that the following steps have to be performed as root (sudo su -). Change to install directory determined above: 
$ cd /install_dir
Output should look similar to the following: 
api              install-lilt-docker-registry.sh  localpv-provisioner  nginx-ingress
clickhouse       install-lilt.sh                  metrics-server       nvidia-device-plugin
dev-scripts      install_scripts                  migration_utilities  output-container-images.py
docker-registry  istiod                           minio                prometheus
elasticsearch    istio-ingressgateway             mongodb              redis
flannel          jenkins                          mq-rabbitmq          uninstall-lilt.sh
grafana          lilt                             mysql
It is not necessary to re-install all third-party applications, just LILT. As an example, view the scripts included in install-lilt.sh: 
cat install-lilt.sh
Output should be similar to the following: 
# Install devops charts
sh install_scripts/install-nginx-ingress.sh
# sh install_scripts/install-docker-registry.sh
sh install_scripts/install-nvidia-device-plugin.sh
sh install_scripts/install-mongodb.sh
sh install_scripts/install-mysql.sh
sh install_scripts/install-clickhouse.sh
sh install_scripts/install-cache-redis.sh
sh install_scripts/install-rabbitmq.sh
# sh install_scripts/install-grafana.sh
sh install_scripts/install-prometheus.sh
sh install_scripts/install-metrics-server.sh
sh install_scripts/install-minio.sh
sh install_scripts/install-elasticsearch.sh
sh install_scripts/install-istio-ingressgateway.sh
sh install_scripts/install-api.sh
# Install LILT Charts
sh install_scripts/install-lilt-charts.sh
Only need to run the last script: 
sh install_scripts/install-lilt-charts.sh

​
Step 2: Verification

Ensure all the pods are healthy (running status), this can take up to 15-30 minutes as some images will have to be reloaded. 
$ kubectl get pods -n lilt
If you see Error: UPGRADE FAILED:timed out waiting for the condition, please continue as this can happen due to time taken by the pods to startup, apps deployment happens as expected.

​
Rollback (only if necessary)

If errors persist and LILT will not function, rollback to backup previous values/images and run sh install_scripts/install-lilt-charts.sh

​
Restore MySql DB (only if necessary)

Restore DB from the backup performed in the above section.

​
Step 1: Copy Dump File to MySql Pod

kubectl cp /mysql_dump/mysql_dump.sql lilt/mysql-0:/tmp/mysql_dump.sql

​
Step 2: Restore File

kubectl exec --stdin --tty mysql-0 -- /bin/bash -c "mysql --user root -p<password> -D lilt_dev < /tmp/mysql_dump.sql"

​
Step 3: Restore SequalizeMeta

During restore, .ts and .js files are mismatched, the following reestablishes correct file extensions: 
kubectl exec --stdin --tty mysql-0 -- mysql -u root -p<password> lilt_dev -e 'INSERT INTO `SequelizeMeta` VALUES ("20211206161036-insertEnableWorkflowsSetting-only-on-prem.js"),("20211208093745-bootstrap-workflows-on-prem.js"),("20211208115112-setDefaultWorkflow-only-on-prem.js"),("20211214100926-filtered-tmx-enable.js"),("20211214102401-enable-filtered-tmx-on-prem-only.js"),("20220303094802-enableJobsListPaginated.js"),("20220311151105-project-list-indices.js"),("20220314130118-LinguistQualities-indices.js"),("20220325124422-drop-qaconfig-srclang-trglang.js"),("20220329223422-create-qa-checks.js"),("20220331145745-add-organizations_isserviceaccount.js"),("20220331223422-create-qa-check-results.js"),("20220418193734-add-connector-tokens.js"),("20220420090333-create-ProjectChangelogs.js"),("20220420092243-create-DocumentChangelogs.js"),("20220427104748-create-mandatoryErrorFlaggingSecondaryReviewCommercial-setting.js"),("20220427104803-create-showPreviousStageCommentsSecondaryReviewCommercial-setting.js"),("20220429223422-recreate-qa-check-results.js"),("20220502094431-translation-done-required-review-stage.js"),("20220504202653-create-SegmentWorkflowStages.js"),("20220505123749-apply-customer-reviewer-role.js"),("20220505132017-create-scheduled-events.js"),("20220509105002-create-enableNextWordSuggestions-setting.js"),("20220510132117-enable-reject-document.js"),("20220511054900-create-neural-batch-translation-reqs-and-jobs.js"),("20220531155509-lagging_linguist_query_indexes.js"),("20220601223420-create-async-checks.js"),("20220601223422-create-async-check-results.js"),("20220602154639-edit-lingustlanguagepairs-locale-constraints.js"),("20220609102900-add-lingustqualities-is-available.js"),("20220614133322-drop-ScheduledEvents_unique_index.js"),("20220627154145-create-SSConfigurations.js"),("20220729021311-create-OrganizationLangPairSamplePercentage.js"),("20220729103939-limit-assignments-to-onboarded-users.js"),("20220809081110-add-index-notifications.js"),("20220817123018-add-MemoryCheckResults_ConfirmedByUserId_createdAt.js"),("20220908205552-add-workflow-group.js"),("20220909010816-add-OrganizationLangPairSamplePercantage-isDeleted.js"),("20220914150537-update-workflow-group.js"),("20220922130312-add-MemoryCheckResults-issueTypes-index.js"),("20221013194559-linguistAvailabilityToggle.js"),("20221013195401-autoAssignDueDates.js"),("20221014115910-create-customer-role.js"),("20221021103702-limit-assignments-to-onboarded-users-defaulted-false.js"),("20221024120044-update-customer-and-cr-roles.js"),("20221114153435-update-default-role-permissions.js"),("20221125151839-create-user-fingerprints.js"),("20221129124055-add-customer-review-sampling-columns.js"),("20221129143620-LinguistOrganization-Review-Allowance.js"),("20221130010816-add-backend_version_to_neural_batch.js"),("20221201010816-add-backend_version_to_neural_update_manager_requests.js"),("20221202163800-preaccepted-segments-remove-segment-id-unique.js"),("20221209130938-control-copy-and-paste-settings.js"),("20221212185204-add-hourly-rate-to-linguist-language-pairs.js"),("20221214123158-create-connector-jobs-index.js"),("20221230143840-viewprojectstats.js"),("20230110140105-add-memory-hashes.js"),("20230127162613-create-workflow-task-reasons.js"),("20230302235253-viewContextualAI.js"),("20230306124148-longer-locales-varchar.js"),("20230306170809-demo-user-metadata-setting.js"),("20230319121009-create-LanguagePairsScalar.js"),("20230330222759-enableConnectorsView.js"),("20230425235845-create-SegmentMemoryMetadata.js"),("20230509155908-create-generated-content.js"),("20230510191322-OrgSettingForTmFuzzyMatchDisablement.js"),("20230516230606-create-JobParameters.js"),("20230606131341-generatedContentPreferences.js"),("20230613125129-create-lilt-create-role.js"),("20230621154632-connectors-organization-update-tables.js")'