2025
Patch Updates
The configuration of GPU requirements has changed. Starting with this release, 24GB of VRAM is necessary to run the translate pod successfully. For customers running on T4 GPUs, this means that the GPU node must have at least two (2) T4s attached to it. Put another way, it is not sufficient to have two (2) nodes, each with one (1) T4 GPU attached.2024 Q4
Deprecation Notice
Theanalytics-api
application, part of the old analytics implementation, has been officially deprecated and removed.
Node Labeling
In order to support better utilization of clusters, we have adjusted the way we recommend labeling clusters. Node Labels describes the expected labeling of nodes.Troubleshooting
Updated the Troubleshooting guide to include the newly added CLI command to reset the AI models.Default Values
Updated resource defaults for services to optimize performance. These changes are documented within Resource Metrics.Flannel CNI
Flannel is now a helm chart and included in the overallinstall-lilt.sh
script and no longer a separate deployment. If upgrading LILT from a previous version and flannel
is already installed, comment out the flannel section of the install script:
flannel
for the first time via the helm chart, ensure that the podCidr
is consistent with K8S cluster settings:
Redis
Memory limits have been implemented to prevent pod restarts/crashes. This ensures that consumed memory does not exceed pod resource limits. Settings formaxmemory
must be slightly below pod mem limits
. Memory can be increased if required:
Istio
Additionalkernel
parameters are required to prevent ztunnel
pod restarts:
Firewall Ports
Additional ports are required forIstio
, api
, Clickhouse
, and Flannel
. Please ensure that the following are enabled:
Containerd
Additional workloads now run on theGPU
node in parallel with the Worker
node. If NOT using a centralized repository for all images, ensure that the following are loaded via containerd
on the GPU
node:
2024 Q2
Hardware Requirements
Due to the inclusion of V2 and V3 models by default, the hardware requirements have changed. Resource Metrics reflects the additional deployments that need to be considered, and the following recommendations have been updated:- Master node disk requirement has increased from 200 GB to 500 GB to accommodate for additional container images, configuration, and logging.
-
GPU Node instance type updated from
g4dn.2xlarge (8 vCPUs, 32 GB RAM)
tog4dn.8xlarge (32 vCPUs, 128 GB RAM)
, in order to be able to run the V2 and V3 services. - Due to the added models, the hard disk space requirements have been increased. See Installation Requirements for more information.
V2 and V3 Language Model Updates
As we introduce newer, more accurate language models into LILT, we’ve continually updated our hardware requirements. See Language Models for the latest in V2 and V3 model information. More information can be found in the Knowledge Base around Resource Requirements.Operating System Requirements
CentOS 7 → Rocky Linux 8
New software features of the LILT platform are incompatible with CentOS 7[1]. All installations still on CentOS 7 should migrate before adopting this release. Some previous LILT installations were done using CentOS 7, which reached End of Life (EOL) support as of June 30, 2024. The recommended base operating system, and the one being tested in our QA environment, is using Rocky Linux 8. Rocky Linux 8 provides a secure environment similar to CentOS 7, with an End of Life (EOL) support date of May 2029.Istio Module Support
Modules
All installations have updated modules needed to support Istio. See the section regarding kernel modules which now includes the following modules to install:Ports
All installations have updated firewall port changes needed to support Istio. See the section regarding Firewall Settings which now includes opening up ports15000,15001,15006,15008,15009,15010,15012,15014,15017,15020,15021,15090,15443,20001}/tcp
for Istio.
Configuration Updates
Custom Domains
New configurations should be done, as described in Set custom Domain and Certificates and Set Connectors Domain.Upgrading Process
The upgrading process, which involves Helm values files, has been updated for this release to make upgrading simpler in the future. See Q2 2024 Updates for more details.MinIO Resize
In previous releases, the PersistentVolume for MinIO was set to 200GB. With the release of V2 and V3 models, this is no longer large enough to support them. The default size has been updated from 200GB to 400GB, however, this will not automatically resize existing installations. If your backing MinIO PersistentVolume is resizable, please resize to 400GB. If it is not able to be resized, the recommended procedure is as follows:- Back up the MinIO data (if necessary)
- Delete the MinIO PersistentVolumeClaim and and PersistentVolume
- Restart the new MinIO deployment
- Restore MinIO data (if necessary)
WPA Metrics
WPA metrics, as described in Generate Evaluation Metrics (WPA, BLEU) , are now enabled by default.SMTP Notifications
See the new page around configuring SMTP Notifications here: SMTP Email NotificationsGuide on how to handle GPU worker counts
As GPU processing has become increasingly critical in LILT’s models, we’ve added Configuring GPU Worker Counts in LILT to assist system administrators in configuring the LILT application for GPU use.Vulnerability (CVE) Scan Results
LILT has conducted thorough scans of all services and components to confirm there are no components rated as High or Critical CVEs[2]. Self-Hosted customers can find further details in the CVE Scan PDF provided with the release.Known Issues
MongoDB Upgrade
The latest MongoDB version has a known issue that may cause it to fall into a CrashLoop upon upgrading. If this occurs, the recommended fix is as follows:- Back up the MongoDB data (if necessary)
- Delete the MongoDB PersistentVolumeClaim and and PersistentVolume
- Restart the new MongoDB deployment
- Restore MongoDB data (if necessary)
Known CVE issues
Vulnerability Reference | Application | Mitigation / Notes |
---|---|---|
CVE-2024-31580 | neural | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
CVE-2024-31583 | neural | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
CVE-2023-6378 | core-api | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
CVE-2023-6481 | core-api | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
CVE-2024-22257 | core-api | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
CVE-2016-1000027 | core-api | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
CVE-2024-22243 | core-api | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
CVE-2024-22259 | core-api | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
CVE-2024-22262 | core-api | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
CVE-2023-32697 | core-api | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
CVE-2022-1471 | core-api | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
CVE-2022-25857 | core-api | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
CVE-2024-21634 | core-api | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
GHSA-m425-mq94-257g | localpv-provisioner | Vulnerability exists in latest version of this application. Waiting on a newer release to fix. |
CVE-2024-24790 | localpv-provisioner | Vulnerability exists in latest version of this application. Waiting on a newer release to fix. |
CVE-2023-39325 | localpv-provisioner | Vulnerability exists in latest version of this application. Waiting on a newer release to fix. |
CVE-2023-45283 | localpv-provisioner | Vulnerability exists in latest version of this application. Waiting on a newer release to fix. |
CVE-2023-45287 | localpv-provisioner | Vulnerability exists in latest version of this application. Waiting on a newer release to fix. |
CVE-2023-45288 | localpv-provisioner | Vulnerability exists in latest version of this application. Waiting on a newer release to fix. |
CVE-2023-1370 | elasticsearch | Vulnerability exists in latest version of this application. Waiting on a newer release to fix. |
CVE-2021-40690 | elasticsearch | Vulnerability exists in latest version of this application. Waiting on a newer release to fix. |
CVE-2022-1471 | elasticsearch | Vulnerability exists in latest version of this application. Waiting on a newer release to fix. |
CVE-2024-41110 | istiod | Vulnerability exists in latest version of this application. Waiting on a newer release to fix. |