Infrastructure Release Notes

This article details the infrastructure changes required when installing this version of LILT.

6.1.0

This section is under active development and has not yet been verified — content was auto-generated from a diff between the 5.3.x and main branches of on-prem-installer and needs human review before this constitutes guidance.

In-Cluster Vault (Default ESO Backend)

EKS and on-prem installs now default to deploying a HashiCorp Vault instance in-cluster (ENABLE_VAULT=true) as the backend for External Secrets Operator (ESO), in place of direct secret seeding. The Vault server image ships in the delivery tarball with every other image. On EKS this is the working default: External Secrets reads its secrets from that Vault, and AWS KMS auto-unseal means the pod recovers on its own after a restart.

On-prem installs get Vault, but nothing that uses it. External Secrets Operator is installed only on the EKS path, so on the on-prem flavor the shipped Vault is deployed and then consumed by nothing, and SECRET_BACKEND has no effect. Secrets continue to come from the seed file exactly as before.An on-prem install that does not want the unused component can skip it:

# install.env
ENABLE_VAULT=false

Note also that on-prem Vault seals itself on every restart or node drain, because it uses Shamir rather than KMS auto-unseal and the bootstrap runs only at install time.

Customers pointing ESO at their own external Vault, rather than the shipped in-cluster one, continue to use the SECRET_BACKEND=vault and VAULT_AUTH=approle path, which is unchanged. See Secrets and Vault for the backends, the Vault path and key inventory, and the unseal behaviour.

Hardware Requirements

gemma-vllm-inference now requires compute capability ≥8.9 (Ada generation or later) on both EKS and on-prem. Gemma is FP8-quantised and native FP8 support starts with Ada, so this excludes every Ampere card, including the 80 GB A100, regardless of how much VRAM it has. The VRAM floor differs between the two:

EKS: the GPU VRAM floor has been raised from 21999 MiB to 43399 MiB, and the GPU count is 1. L4 GPUs (23034 MiB) no longer satisfy this requirement. L40, L40S, H100, or H200 cards are required going forward. A cluster whose only Ada hardware is the 24 GB class can size back down with the gemma-dual-medium GPU profile.
On-prem / classic: gemma-vllm-inference previously had no VRAM floor (only a generic capability: gpu node selector). It now requires ≥22000 MiB of GPU memory. This admits L4, L40, L40S, and H100; it excludes T4, A10G, A6000, and A100. GPU count is unchanged at 2.

See GPU profiles and values overlays for the profile tables and how to change an allocation.

Configuration Updates

The APRYSE keys (APRYSE_KEY, APRYSE_DOCX_KEY) moved from inline environment variables on the converter/file-translation services to the lilt-secrets Kubernetes Secret, referenced via envValueFrom. Before upgrading, ensure lilt-secrets contains apryseKey and apryseDocxKey, or the core-tmx-import WorkflowTemplate will fail with a duplicate-value error.
A new connectors-secrets Kubernetes Secret was introduced (managerUiSecretKey, webhookAuthToken). This must be seeded before upgrading.
The cluster-autoscaler Helm chart is now pulled from Lilt’s internal GAR OCI registry (CHART_BASE, defaults to oci://us-central1-docker.pkg.dev/lilt-service-48916b30/charts) instead of the public https://kubernetes.github.io/autoscaler repo. Air-gapped or firewall-restricted customers must ensure OCI chart pull access to this registry.
Customers setting SECRET_BACKEND=vault or SECRET_BACKEND=aws (ESO-backed secrets, distinct from the in-cluster Vault above) must provision each secret in the README’s “Secret provisioning (External Secrets)” table as a JSON object, either under BASE_VAULT_PATH or via a per-secret <NAME>_SECRET_PATH override — do not provision lilt-secrets directly, it’s composed by ESO from the other secrets. The external-Vault approle path (VAULT_AUTH=approle) additionally requires a pre-existing vault-approle Kubernetes Secret in the external-secrets namespace, provisioned out-of-band. The default in-cluster-Vault path (VAULT_AUTH=kubernetes) needs no such secret. The AWS path uses IRSA — leave S3/AWS credential keys empty or they’ll override IRSA and break S3 access.

Service Mesh (Istio)

Istio has been bumped two minor versions: the istiod chart from 1.26.2-rc4 to 1.28.0 (image tags 1.28 → 1.30), and istio-ingressgateway from 1.26.2-rc2 to 1.30.0. PeerAuthentication is no longer enabled by the istiod chart directly — it’s now managed by the lilt-networking chart instead.

Identity Provider (WSO2 SSO)

install-wso2.sh gained an ENABLE_WSO2 toggle (default true) and the chart was bumped from 7.4.0-3 to 7.4.1-3, which includes a patch disabling OpenTelemetry auto-instrumentation on the WSO2 pod to avoid a LinkageError.
EMBEDDED_OIDC_PROVIDER_ENABLED was flipped from true to false on front, av-scan, and auth-service.

TODO: assess customer-facing impact of the EMBEDDED_OIDC_PROVIDER_ENABLED flip for installs relying on the embedded OIDC provider rather than external WSO2/SSO.

Deprecation Notice

nonInteractiveMqPwd has been removed from the secrets seed. Installs seeded before this change should manually remove the key from their secrets.yaml before upgrading.
translatev4 has been disabled; its /2/translate routes are now served through Rayman. No customer action is expected, but flag if any override files still reference translatev4 directly.
Dead non-interactive RabbitMQ environment variables (MQ_NON_INTERACTIVE_HOST, _PORT, _PORT_SSL, _USER, _PASSWORD) were removed from all neural service blocks. Customers with these variables in override files can remove them.

Patch Updates

kuberay-operator bumped from 1.5.1 to 1.6.2.
Resource defaults changed for a few pre-existing services, relevant to capacity planning: converter memory request 26Gi → 27Gi and memory limit 28Gi → 36Gi; tm CPU request 4 → 6 cores; front’s 2-core CPU limit was removed (now unbounded); beehive gained a new 4-core CPU limit where none existed before. No replica-count changes were found for any pre-existing service. Firewall port requirements and the Flannel CNI chart version (v0.27.0) are unchanged.

Known CVE Issues

Package	Severity	Component	Fixed In
libcrypto3 / libssl3	CRITICAL (CVE-2026-34182)	rayman-artifacts base image	`rayman-artifacts_535c6c4`
wget	HIGH (CVE-2026-58469)	rayman-artifacts base image	`rayman-artifacts_535c6c4`

TODO: a full CVE scan is expected alongside the 6.1.0 release; add any additional findings here.

2025

Patch Updates

The configuration of GPU requirements has changed. Starting with this release, 24GB of VRAM is necessary to run the translate pod successfully. For customers running on T4 GPUs, this means that the GPU node must have at least two (2) T4s attached to it. Put another way, it is not sufficient to have two (2) nodes, each with one (1) T4 GPU attached.

2024 Q4

Deprecation Notice

The analytics-api application, part of the old analytics implementation, has been officially deprecated and removed.

Node Labeling

In order to support better utilization of clusters, we have adjusted the way we recommend labeling clusters. Node Labels describes the expected labeling of nodes.

Troubleshooting

Updated the Troubleshooting guide to include the newly added CLI command to reset the AI models.

Default Values

Updated resource defaults for services to optimize performance. These changes are documented within Resource Metrics.

Flannel CNI

Flannel is now a helm chart and included in the overall install-lilt.sh script and no longer a separate deployment. If upgrading LILT from a previous version and flannel is already installed, comment out the flannel section of the install script:

# install-lilt.sh

# Install flannel, on-prem customers only
kubectl label --overwrite ns kube-flannel pod-security.kubernetes.io/enforce=privileged
sh install_scripts/install-flannel.sh
# wait until pod ready
kubectl wait --namespace kube-flannel --for=condition=ready pod -l app=flannel --timeout=180s

If installing flannel for the first time via the helm chart, ensure that the podCidr is consistent with K8S cluster settings:

# flannel/on-prem-values.yaml

flannel:
  podCidr: "192.168.100.0/19"

Redis

Memory limits have been implemented to prevent pod restarts/crashes. This ensures that consumed memory does not exceed pod resource limits. Settings for maxmemory must be slightly below pod mem limits. Memory can be increased if required:

# redis/on-prem-values.yaml

global:
  redis:
    # maxmem needs to be just below resource limit
    maxmemory: "5.8gb"
    maxmemoryPolicy: "allkeys-lru"  # evict least recently used keys
#
master:
  resources:
    limits:
      memory: 6G

Persistent storage is now disabled by default. This increases performance and reduces storage requirements.

# redis/on-prem-values.yaml

master:
  persistence:
    enabled: false

If persistence is required for cache security logging/audit, reenable and ensure that the storage size is sufficient for estimated usage:

# redis/on-prem-values.yaml

master:
  persistence:
    enabled: true
    size: 20Gi   # up to 100Gi for heavy usage

Istio

Additional kernelparameters are required to prevent ztunnel pod restarts:

# avoid ztunnel container restarts due to load
# append to end of file
cat <<EOF >> /etc/security/limits.conf
soft nofile 131072
hard nofile 131072
EOF

cat <<EOF >> /etc/systemd/system.conf
DefaultLimitNOFILE=131072
EOF

Firewall Ports

Additional ports are required for Istio, api, Clickhouse, and Flannel. Please ensure that the following are enabled:

firewall-cmd --permanent --add-port={22,80,443,2379,2380,5000,6443,10250,10251,10252,10255}/tcp
# api
firewall-cmd --permanent --add-port={5005,8011,8080}/tcp
# istio
firewall-cmd --permanent --add-port={15000,15001,15006,15008,15009,15010,15012,15014,15017,15020,15021,15090,15443,20001}/tcp
# flannel
firewall-cmd --permanent --add-port=8472/udp
# clickhouse
firewall-cmd --permanent --add-port=8123/tcp

Containerd

Additional workloads now run on the GPU node in parallel with the Worker node. If NOT using a centralized repository for all images, ensure that the following are loaded via containerd on the GPU node:

pilot*
proxyv2*
install-cni*
ztunnel*
kiali*
flannel*     # (only if new install)
k8s-device-plugin*
metrics-server*
neural*   # (all neural from docker_images master/node)
llm*
batch*

2024 Q2

Hardware Requirements

Due to the inclusion of newer models by default, the hardware requirements have changed. Resource Metrics reflects the additional deployments that need to be considered, and the following recommendations have been updated:

Master node disk requirement has increased from 200 GB to 500 GB to accommodate for additional container images, configuration, and logging.
GPU Node instance type updated from g4dn.2xlarge (8 vCPUs, 32 GB RAM) to g4dn.8xlarge (32 vCPUs, 128 GB RAM), in order to be able to run the V4 neural services.
Due to the added models, the hard disk space requirements have been increased. See Installation Requirements for more information.

V4 Language Model Updates

As we introduce newer, more accurate language models into LILT, we’ve continually updated our hardware requirements. See Language Models for the latest in V4 model information. More information can be found in the Knowledge Base around Resource Requirements.

Operating System Requirements

CentOS 7 → Rocky Linux 8

New software features of the LILT platform are incompatible with CentOS 7[1]. All installations still on CentOS 7 should migrate before adopting this release. Some previous LILT installations were done using CentOS 7, which reached End of Life (EOL) support as of June 30, 2024. The recommended base operating system, and the one being tested in our QA environment, is using Rocky Linux 8. Rocky Linux 8 provides a secure environment similar to CentOS 7, with an End of Life (EOL) support date of May 2029.

Istio Module Support

Modules

All installations have updated modules needed to support Istio. See the section regarding kernel modules which now includes the following modules to install:

overlay
br_netfilter
nf_nat
xt_REDIRECT
xt_owner
iptable_nat
iptable_mangle
iptable_filter

These will need to be installed on all existing nodes.

Ports

All installations have updated firewall port changes needed to support Istio. See the section regarding Firewall Settings which now includes opening up ports 15000,15001,15006,15008,15009,15010,15012,15014,15017,15020,15021,15090,15443,20001}/tcp for Istio.

Configuration Updates

Custom Domains

New configurations should be done, as described in Set custom Domain and Certificates and Set Connectors Domain.

Upgrading Process

The upgrading process, which involves Helm values files, has been updated for this release to make upgrading simpler in the future. See Q2 2024 Updates for more details.

MinIO Resize

In previous releases, the PersistentVolume for MinIO was set to 200GB. With the release of newer models, this is no longer large enough to support them. The default size has been updated from 200GB to 400GB, however, this will not automatically resize existing installations. If your backing MinIO PersistentVolume is resizable, please resize to 400GB. If it is not able to be resized, the recommended procedure is as follows:

Back up the MinIO data (if necessary)
Delete the MinIO PersistentVolumeClaim and PersistentVolume
Restart the new MinIO deployment
Restore MinIO data (if necessary)

WPA Metrics

WPA metrics, as described in Generate Evaluation Metrics (WPA, BLEU) , are now enabled by default.

SMTP Notifications

See the new page around configuring SMTP Notifications here: SMTP Email Notifications

Guide on how to handle GPU worker counts

As GPU processing has become increasingly critical in LILT’s models, we’ve added Configuring GPU Worker Counts in LILT to assist system administrators in configuring the LILT application for GPU use.

Vulnerability (CVE) Scan Results

LILT has conducted thorough scans of all services and components to confirm there are no components rated as High or Critical CVEs[2]. Self-Hosted customers can find further details in the CVE Scan PDF provided with the release.

Known Issues

MongoDB Upgrade

The latest MongoDB version has a known issue that may cause it to fall into a CrashLoop upon upgrading. If this occurs, the recommended fix is as follows:

Back up the MongoDB data (if necessary)
Delete the MongoDB PersistentVolumeClaim and PersistentVolume
Restart the new MongoDB deployment
Restore MongoDB data (if necessary)

Known CVE issues

Vulnerability Reference	Application	Mitigation / Notes
CVE-2024-31580	neural	To be fixed in next release, requires upgrade of package that is used by internal dependencies.
CVE-2024-31583	neural	To be fixed in next release, requires upgrade of package that is used by internal dependencies.
CVE-2023-6378	core-api	To be fixed in next release, requires upgrade of package that is used by internal dependencies.
CVE-2023-6481	core-api	To be fixed in next release, requires upgrade of package that is used by internal dependencies.
CVE-2024-22257	core-api	To be fixed in next release, requires upgrade of package that is used by internal dependencies.
CVE-2016-1000027	core-api	To be fixed in next release, requires upgrade of package that is used by internal dependencies.
CVE-2024-22243	core-api	To be fixed in next release, requires upgrade of package that is used by internal dependencies.
CVE-2024-22259	core-api	To be fixed in next release, requires upgrade of package that is used by internal dependencies.
CVE-2024-22262	core-api	To be fixed in next release, requires upgrade of package that is used by internal dependencies.
CVE-2023-32697	core-api	To be fixed in next release, requires upgrade of package that is used by internal dependencies.
CVE-2022-1471	core-api	To be fixed in next release, requires upgrade of package that is used by internal dependencies.
CVE-2022-25857	core-api	To be fixed in next release, requires upgrade of package that is used by internal dependencies.
CVE-2024-21634	core-api	To be fixed in next release, requires upgrade of package that is used by internal dependencies.
GHSA-m425-mq94-257g	localpv-provisioner	Vulnerability exists in latest version of this application. Waiting on a newer release to fix.
CVE-2024-24790	localpv-provisioner	Vulnerability exists in latest version of this application. Waiting on a newer release to fix.
CVE-2023-39325	localpv-provisioner	Vulnerability exists in latest version of this application. Waiting on a newer release to fix.
CVE-2023-45283	localpv-provisioner	Vulnerability exists in latest version of this application. Waiting on a newer release to fix.
CVE-2023-45287	localpv-provisioner	Vulnerability exists in latest version of this application. Waiting on a newer release to fix.
CVE-2023-45288	localpv-provisioner	Vulnerability exists in latest version of this application. Waiting on a newer release to fix.
CVE-2023-1370	elasticsearch	Vulnerability exists in latest version of this application. Waiting on a newer release to fix.
CVE-2021-40690	elasticsearch	Vulnerability exists in latest version of this application. Waiting on a newer release to fix.
CVE-2022-1471	elasticsearch	Vulnerability exists in latest version of this application. Waiting on a newer release to fix.
CVE-2024-41110	istiod	Vulnerability exists in latest version of this application. Waiting on a newer release to fix.

Appendix

[1] The Analytics dashboard relies on Istio, which uses some underlying code libraries in the operating system kernel that are not present in CentOS 7. [2] Fixed High and Critical CVEs refer to vulnerabilities for which fixes are available and do not break the integration of the application and its dependencies. It should be noted that vulnerabilities are continuously discovered, and new CVEs may have been identified but not yet addressed.

​6.1.0

​In-Cluster Vault (Default ESO Backend)

​Hardware Requirements

​Configuration Updates

​Service Mesh (Istio)

​Identity Provider (WSO2 SSO)

​Deprecation Notice

​Patch Updates

​Known CVE Issues

​2025

​Patch Updates

​2024 Q4

​Deprecation Notice

​Node Labeling

​Troubleshooting

​Default Values

​Flannel CNI

​Redis

​Istio

​Firewall Ports

​Containerd

​2024 Q2

​Hardware Requirements

​V4 Language Model Updates

​Operating System Requirements

​CentOS 7 → Rocky Linux 8

​Istio Module Support

Modules

Ports

​Configuration Updates

​Custom Domains

​Upgrading Process

​MinIO Resize

​WPA Metrics

​SMTP Notifications

​Guide on how to handle GPU worker counts

​Vulnerability (CVE) Scan Results

​Known Issues

​MongoDB Upgrade

​Known CVE issues

​Appendix

6.1.0

In-Cluster Vault (Default ESO Backend)

Hardware Requirements

Configuration Updates

Service Mesh (Istio)

Identity Provider (WSO2 SSO)

Deprecation Notice

Patch Updates

Known CVE Issues

2025

Patch Updates

2024 Q4

Deprecation Notice

Node Labeling

Troubleshooting

Default Values

Flannel CNI

Redis

Istio

Firewall Ports

Containerd

2024 Q2

Hardware Requirements

V4 Language Model Updates

Operating System Requirements

CentOS 7 → Rocky Linux 8

Istio Module Support

Configuration Updates

Custom Domains

Upgrading Process

MinIO Resize

WPA Metrics

SMTP Notifications

Guide on how to handle GPU worker counts

Vulnerability (CVE) Scan Results

Known Issues

MongoDB Upgrade

Known CVE issues

Appendix