> ## Documentation Index > Fetch the complete documentation index at: https://support.lilt.com/llms.txt > Use this file to discover all available pages before exploring further. # Upgrade LILT ## Overview This article walks the installer through the process of upgrading an existing on-premise system. Given an available Kubernetes cluster, this document guides you through a partially-automated install process to deploy and configure LILT in your environment. #### Required Tools You will need to use SSH to connect to the systems. * On the Mac or Linux, you can open a terminal window and use ssh. * On Windows, you can use the PuTTY ssh client. You will also need a web browser to browse the website post installation and verification. #### Installer Privileges Many sections of this article include text formatted to indicate console input and output. The commands are prefixed with a \$ or # depending on if the administrator should run them as root or not. Additionally, to assist the user to know which machine to run commands on, the prefix node, master, and gpu will be given when the machine should be switched. All following commands after a switch should use those. #### Run commands as root ```bash theme={null} sudo su - ``` ## Prerequisites This upgrade document is only valid for an existing on-premise install previously installed using on-prem-installer scripts. #### Data Location Ensure that the new/updated Docker images and RPM packages are mounted and available to all of the nodes (master, worker, and GPU). In the typical installation, the system administrator will receive this data from LILT prior to installation. ## Backup Backup up the current running system prior to upgrade. This backup is **highly recommended**, and will be used to restore projects, documents and memory content in case of any errors. #### **Step 1: Master Node** Login to the k8s-master node and follow the below steps. Note that the following steps have to be performed as root (`sudo su -`). **Backup Install Directory** Backup previous Install Directory (structure might be slightly different than the following example): ```bash theme={null} mkdir -p /install_dir_backup cp -r /install_dir/ /install_dir_backup ``` **Database** Backup mysql database: ```bash theme={null} # this will dump a file to the mysql pod temp memory kubectl exec --stdin --tty mysql-0 -- /bin/bash -c "mysqldump --host mysql.lilt.svc.cluster.local --user root -p lilt_dev > /tmp/mysql_dump.sql" # copy dump file from mysql pod to local directory on master node kubectl cp lilt/mysql-0:/tmp/mysql_dump.sql /mysql_dump/mysql_dump.sql ``` **Minio (optional)** Take note of the existing minio pvc (`pvc-eiurty3794t679eryt495ytotoer`) for future commands: ```csharp theme={null} # get minio pvc kubectl get pvc | grep minio ``` Example output: ```powershell theme={null} minio Bound pvc-80e6a0a5-9861-42ac-9e3f-ee6c7d439914 400Gi RWO openebs-hostpath ``` #### **Step 2: Worker Node** Login to the k8s-worker node and follow the below steps. Note that the following steps have to be performed as root (`sudo su -`). **Persistant Volumes** Backup storage volumes (pvcs) managed by openebs mount folders, default directory is `/liltdata`: ```bash theme={null} # make backup dir mkdir /lildtata_backup # copy all pvcs cp -r /liltdata/ /liltdata_backup ``` ## Upgrade Preparation #### Step 1: Import New Images and Installer Packages Download new images from remote s3 bucket and/or installer package. Details describing this process are documented in [Install System (Rocky Linux 8)](/kb/install-system-amazon-linux-2023-or-rocky-8-9) NOTE: ensure that required images for each specific node are imported/loaded via containerd and that the installer package is correctly unpacked per steps in the documentation. #### Step 2: Update Custom Values If changes were made to the default values in an existing install, compare files and modify as required (see the next section). ```swift theme={null} diff -y /old_releaase/lilt/values.yaml /new_release/lilt/values.yaml ``` In previous versions of LILT, values were spread across `lilt/values.yaml` and `lilt/environment/lilt/values.yaml`, which made it hard to differentiate what is a customer-centric vs default. In subsequent release versions, default values `lilt/values.yaml`**should remain default**, and custom-values set in `lilt/environment/lilt/values.yaml`. As an example, the first file is default `values.yaml`: ```yaml theme={null} # values.yaml front: enabled: true onpremValues: # Image credentials secrets to fetch the docker image imagePullSecrets: - name: docker-registry-secret # Create new image credentials, used when we aren't using local docker registry createimageCredentials: enabled: false name: docker-registry-secret # Enable biConnector(analytics-api), verify the port from biConnector configs biconnector: enabled: false Port: 8080 # tls enabled, using dummy certificate for bare-tmp.lilt.com, please use new certs if available tls: enabled: true # set to true if tls secret exists existingSecret: false # Enter secret name secretName: front-app-tls env: # Theme configurations THEME_HEADER_BGCOLOR: "f0f0f0" THEME_HEADER_COLOR: "333333" THEME_HEADER_FONTSIZE: "12px" THEME_HEADER_TEXT: "My Company" THEME_FOOTER_BGCOLOR: "f0f0f0" THEME_FOOTER_COLOR: "333333" THEME_FOOTER_FONTSIZE: "12px" THEME_FOOTER_TEXT: "Test Installation Footer" ``` Add custom values for `THEME_HEADER_TEXT: "My Company"`, update `lilt/environment/lilt/values.yaml` as in the following: ```yaml theme={null} # lilt/environment/lilt/values.yaml front: onpremValues: env: THEME_HEADER_TEXT: "My Company" ``` NOTE: this is a one-time migration, no necessary to repeat after custom values are set. #### Step 3: Update Custom Secrets Verify custom secrets, no changes required unless specifically mentioned. The customer-centric secrets file is located at: `lilt/environment/lilt/secrets.yaml` #### Step 4: Environment Setup (Optional) For multiple environments, create customer-centric *values.yaml* and *secrets.yaml* inside each respective environments folder as required. ## Upgrade LILT After all new images, install files and custom environment variables have been set, run install LILT script. #### **Step 1: Master Node** Login to the k8s-master node and follow the below steps. Note that the following steps have to be performed as root (`sudo su -`). Change to install directory determined above: ```bash theme={null} $ cd /install_dir ``` Output should look similar to the following: ```powershell theme={null} api install-lilt-docker-registry.sh localpv-provisioner nginx-ingress clickhouse install-lilt.sh metrics-server nvidia-device-plugin dev-scripts install_scripts migration_utilities output-container-images.py docker-registry istiod minio prometheus elasticsearch istio-ingressgateway mongodb redis flannel jenkins mq-rabbitmq uninstall-lilt.sh grafana lilt mysql ``` It is not necessary to re-install all third-party applications, just LILT. As an example, view the scripts included in `install-lilt.sh`: ```powershell theme={null} cat install-lilt.sh ``` Output should be similar to the following: ```bash theme={null} # Install devops charts sh install_scripts/install-nginx-ingress.sh # sh install_scripts/install-docker-registry.sh sh install_scripts/install-nvidia-device-plugin.sh sh install_scripts/install-mongodb.sh sh install_scripts/install-mysql.sh sh install_scripts/install-clickhouse.sh sh install_scripts/install-cache-redis.sh sh install_scripts/install-rabbitmq.sh # sh install_scripts/install-grafana.sh sh install_scripts/install-prometheus.sh sh install_scripts/install-metrics-server.sh sh install_scripts/install-minio.sh sh install_scripts/install-elasticsearch.sh sh install_scripts/install-istio-ingressgateway.sh sh install_scripts/install-api.sh # Install LILT Charts sh install_scripts/install-lilt-charts.sh ``` Only need to run the last script: ```bash theme={null} sh install_scripts/install-lilt-charts.sh ``` #### Step 2: Verification Ensure all the pods are healthy (running status), this can take up to 15-30 minutes as some images will have to be reloaded. ```bash theme={null} $ kubectl get pods -n lilt ``` If you see `Error: UPGRADE FAILED:`timed out waiting for the condition, please continue as this can happen due to time taken by the pods to startup, apps deployment happens as expected. ## Rollback (only if necessary) If errors persist and LILT will not function, rollback to backup previous values/images and run `sh install_scripts/install-lilt-charts.sh` ## Restore MySql DB (only if necessary) Restore DB from the backup performed in the above section. #### Step 1: Copy Dump File to MySql Pod ```ruby theme={null} kubectl cp /mysql_dump/mysql_dump.sql lilt/mysql-0:/tmp/mysql_dump.sql ``` #### Step 2: Restore File ```bash theme={null} kubectl exec --stdin --tty mysql-0 -- /bin/bash -c "mysql --user root -p -D lilt_dev < /tmp/mysql_dump.sql" ``` #### Step 3: Restore SequalizeMeta During restore, `.ts` and `.js` files are mismatched, the following reestablishes correct file extensions: ```swift theme={null} kubectl exec --stdin --tty mysql-0 -- mysql -u root -p lilt_dev -e 'INSERT INTO `SequelizeMeta` VALUES ("20211206161036-insertEnableWorkflowsSetting-only-on-prem.js"),("20211208093745-bootstrap-workflows-on-prem.js"),("20211208115112-setDefaultWorkflow-only-on-prem.js"),("20211214100926-filtered-tmx-enable.js"),("20211214102401-enable-filtered-tmx-on-prem-only.js"),("20220303094802-enableJobsListPaginated.js"),("20220311151105-project-list-indices.js"),("20220314130118-LinguistQualities-indices.js"),("20220325124422-drop-qaconfig-srclang-trglang.js"),("20220329223422-create-qa-checks.js"),("20220331145745-add-organizations_isserviceaccount.js"),("20220331223422-create-qa-check-results.js"),("20220418193734-add-connector-tokens.js"),("20220420090333-create-ProjectChangelogs.js"),("20220420092243-create-DocumentChangelogs.js"),("20220427104748-create-mandatoryErrorFlaggingSecondaryReviewCommercial-setting.js"),("20220427104803-create-showPreviousStageCommentsSecondaryReviewCommercial-setting.js"),("20220429223422-recreate-qa-check-results.js"),("20220502094431-translation-done-required-review-stage.js"),("20220504202653-create-SegmentWorkflowStages.js"),("20220505123749-apply-customer-reviewer-role.js"),("20220505132017-create-scheduled-events.js"),("20220509105002-create-enableNextWordSuggestions-setting.js"),("20220510132117-enable-reject-document.js"),("20220511054900-create-neural-batch-translation-reqs-and-jobs.js"),("20220531155509-lagging_linguist_query_indexes.js"),("20220601223420-create-async-checks.js"),("20220601223422-create-async-check-results.js"),("20220602154639-edit-lingustlanguagepairs-locale-constraints.js"),("20220609102900-add-lingustqualities-is-available.js"),("20220614133322-drop-ScheduledEvents_unique_index.js"),("20220627154145-create-SSConfigurations.js"),("20220729021311-create-OrganizationLangPairSamplePercentage.js"),("20220729103939-limit-assignments-to-onboarded-users.js"),("20220809081110-add-index-notifications.js"),("20220817123018-add-MemoryCheckResults_ConfirmedByUserId_createdAt.js"),("20220908205552-add-workflow-group.js"),("20220909010816-add-OrganizationLangPairSamplePercantage-isDeleted.js"),("20220914150537-update-workflow-group.js"),("20220922130312-add-MemoryCheckResults-issueTypes-index.js"),("20221013194559-linguistAvailabilityToggle.js"),("20221013195401-autoAssignDueDates.js"),("20221014115910-create-customer-role.js"),("20221021103702-limit-assignments-to-onboarded-users-defaulted-false.js"),("20221024120044-update-customer-and-cr-roles.js"),("20221114153435-update-default-role-permissions.js"),("20221125151839-create-user-fingerprints.js"),("20221129124055-add-customer-review-sampling-columns.js"),("20221129143620-LinguistOrganization-Review-Allowance.js"),("20221130010816-add-backend_version_to_neural_batch.js"),("20221201010816-add-backend_version_to_neural_update_manager_requests.js"),("20221202163800-preaccepted-segments-remove-segment-id-unique.js"),("20221209130938-control-copy-and-paste-settings.js"),("20221212185204-add-hourly-rate-to-linguist-language-pairs.js"),("20221214123158-create-connector-jobs-index.js"),("20221230143840-viewprojectstats.js"),("20230110140105-add-memory-hashes.js"),("20230127162613-create-workflow-task-reasons.js"),("20230302235253-viewContextualAI.js"),("20230306124148-longer-locales-varchar.js"),("20230306170809-demo-user-metadata-setting.js"),("20230319121009-create-LanguagePairsScalar.js"),("20230330222759-enableConnectorsView.js"),("20230425235845-create-SegmentMemoryMetadata.js"),("20230509155908-create-generated-content.js"),("20230510191322-OrgSettingForTmFuzzyMatchDisablement.js"),("20230516230606-create-JobParameters.js"),("20230606131341-generatedContentPreferences.js"),("20230613125129-create-lilt-create-role.js"),("20230621154632-connectors-organization-update-tables.js")' ```