> ## Documentation Index
> Fetch the complete documentation index at: https://support.lilt.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Infrastructure Release Notes

This article details the infrastructure changes required when installing this version of LILT.

## 6.1.0

<Warning>
  This section is under active development and has not yet been verified — content was auto-generated from a diff between the `5.3.x` and `main` branches of `on-prem-installer` and needs human review before this constitutes guidance.
</Warning>

### In-Cluster Vault (Default ESO Backend)

EKS and on-prem installs now default to deploying a HashiCorp Vault instance in-cluster (`ENABLE_VAULT=true`) as the backend for External Secrets Operator (ESO), in place of direct secret seeding. The Vault server image ships in the delivery tarball with every other image.

On EKS this is the working default: External Secrets reads its secrets from that Vault, and AWS KMS auto-unseal means the pod recovers on its own after a restart.

<Note>
  **On-prem installs get Vault, but nothing that uses it.** External Secrets Operator is installed only on the EKS path, so on the on-prem flavor the shipped Vault is deployed and then consumed by nothing, and `SECRET_BACKEND` has no effect. Secrets continue to come from the seed file exactly as before.

  An on-prem install that does not want the unused component can skip it:

  ```bash theme={null}
  # install.env
  ENABLE_VAULT=false
  ```

  Note also that on-prem Vault seals itself on every restart or node drain, because it uses Shamir rather than KMS auto-unseal and the bootstrap runs only at install time.
</Note>

Customers pointing ESO at their own external Vault, rather than the shipped in-cluster one, continue to use the `SECRET_BACKEND=vault` and `VAULT_AUTH=approle` path, which is unchanged.

See [Secrets and Vault](/self-managed/v6.1/secrets-and-vault) for the backends, the Vault path and key inventory, and the unseal behaviour.

### Hardware Requirements

`gemma-vllm-inference` now requires **compute capability ≥8.9 (Ada generation or later) on both EKS and on-prem**. Gemma is FP8-quantised and native FP8 support starts with Ada, so this excludes every Ampere card, including the 80 GB A100, regardless of how much VRAM it has. The VRAM floor differs between the two:

* **EKS:** the GPU VRAM floor has been raised from 21999 MiB to 43399 MiB, and the GPU count is 1. **L4 GPUs (23034 MiB) no longer satisfy this requirement.** L40, L40S, H100, or H200 cards are required going forward. A cluster whose only Ada hardware is the 24 GB class can size back down with the `gemma-dual-medium` GPU profile.
* **On-prem / classic:** `gemma-vllm-inference` previously had no VRAM floor (only a generic `capability: gpu` node selector). It now requires ≥22000 MiB of GPU memory. This admits L4, L40, L40S, and H100; it excludes T4, A10G, A6000, and A100. GPU count is unchanged at 2.

See [GPU profiles and values overlays](/self-managed/v6.1/gpu-profiles-and-values-overlays) for the profile tables and how to change an allocation.

### Configuration Updates

* The APRYSE keys (`APRYSE_KEY`, `APRYSE_DOCX_KEY`) moved from inline environment variables on the `converter`/`file-translation` services to the `lilt-secrets` Kubernetes Secret, referenced via `envValueFrom`. Before upgrading, ensure `lilt-secrets` contains `apryseKey` and `apryseDocxKey`, or the `core-tmx-import` WorkflowTemplate will fail with a duplicate-value error.
* A new `connectors-secrets` Kubernetes Secret was introduced (`managerUiSecretKey`, `webhookAuthToken`). This must be seeded before upgrading.
* The `cluster-autoscaler` Helm chart is now pulled from Lilt's internal GAR OCI registry (`CHART_BASE`, defaults to `oci://us-central1-docker.pkg.dev/lilt-service-48916b30/charts`) instead of the public `https://kubernetes.github.io/autoscaler` repo. Air-gapped or firewall-restricted customers must ensure OCI chart pull access to this registry.
* Customers setting `SECRET_BACKEND=vault` or `SECRET_BACKEND=aws` (ESO-backed secrets, distinct from the in-cluster Vault above) must provision each secret in the README's "Secret provisioning (External Secrets)" table as a JSON object, either under `BASE_VAULT_PATH` or via a per-secret `<NAME>_SECRET_PATH` override — do **not** provision `lilt-secrets` directly, it's composed by ESO from the other secrets. The external-Vault approle path (`VAULT_AUTH=approle`) additionally requires a pre-existing `vault-approle` Kubernetes Secret in the `external-secrets` namespace, provisioned out-of-band. The default in-cluster-Vault path (`VAULT_AUTH=kubernetes`) needs no such secret. The AWS path uses IRSA — leave S3/AWS credential keys empty or they'll override IRSA and break S3 access.

### Service Mesh (Istio)

Istio has been bumped two minor versions: the `istiod` chart from `1.26.2-rc4` to `1.28.0` (image tags `1.28` → `1.30`), and `istio-ingressgateway` from `1.26.2-rc2` to `1.30.0`. `PeerAuthentication` is no longer enabled by the `istiod` chart directly — it's now managed by the `lilt-networking` chart instead.

### Identity Provider (WSO2 SSO)

* `install-wso2.sh` gained an `ENABLE_WSO2` toggle (default `true`) and the chart was bumped from `7.4.0-3` to `7.4.1-3`, which includes a patch disabling OpenTelemetry auto-instrumentation on the WSO2 pod to avoid a `LinkageError`.
* `EMBEDDED_OIDC_PROVIDER_ENABLED` was flipped from `true` to `false` on `front`, `av-scan`, and `auth-service`.

TODO: assess customer-facing impact of the `EMBEDDED_OIDC_PROVIDER_ENABLED` flip for installs relying on the embedded OIDC provider rather than external WSO2/SSO.

### Deprecation Notice

* `nonInteractiveMqPwd` has been removed from the secrets seed. Installs seeded before this change should manually remove the key from their `secrets.yaml` before upgrading.
* `translatev4` has been disabled; its `/2/translate` routes are now served through Rayman. No customer action is expected, but flag if any override files still reference `translatev4` directly.
* Dead non-interactive RabbitMQ environment variables (`MQ_NON_INTERACTIVE_HOST`, `_PORT`, `_PORT_SSL`, `_USER`, `_PASSWORD`) were removed from all neural service blocks. Customers with these variables in override files can remove them.

### Patch Updates

* `kuberay-operator` bumped from `1.5.1` to `1.6.2`.
* Resource defaults changed for a few pre-existing services, relevant to capacity planning: `converter` memory request 26Gi → 27Gi and memory limit 28Gi → 36Gi; `tm` CPU request 4 → 6 cores; `front`'s 2-core CPU limit was removed (now unbounded); `beehive` gained a new 4-core CPU limit where none existed before. No replica-count changes were found for any pre-existing service. Firewall port requirements and the Flannel CNI chart version (`v0.27.0`) are unchanged.

### Known CVE Issues

| Package | Severity | Component | Fixed In |
| - | - | - | - |
| libcrypto3 / libssl3 | CRITICAL (CVE-2026-34182) | rayman-artifacts base image | `rayman-artifacts_535c6c4` |
| wget | HIGH (CVE-2026-58469) | rayman-artifacts base image | `rayman-artifacts_535c6c4` |

TODO: a full CVE scan is expected alongside the 6.1.0 release; add any additional findings here.

## 2025

### Patch Updates

The configuration of GPU requirements has changed. Starting with this release, 24GB of VRAM is necessary to run the translate pod successfully.

For customers running on T4 GPUs, this means that the GPU node **must** have at least two (2) T4s attached to it. Put another way, it is **not sufficient** to have two (2) nodes, each with one (1) T4 GPU attached.

## 2024 Q4

### Deprecation Notice

The `analytics-api` application, part of the old analytics implementation, has been officially deprecated and removed.

### Node Labeling

In order to support better utilization of clusters, we have adjusted the way we recommend labeling clusters. [Node Labels](/self-managed/v6.1/node-labels) describes the expected labeling of nodes.

### Troubleshooting

Updated the [Troubleshooting](/self-managed/v6.1/deleting) guide to include the newly added CLI command to reset the AI models.

### Default Values

Updated resource defaults for services to optimize performance. These changes are documented within [Resource Metrics](/self-managed/v6.1/resource-metrics).

### Flannel CNI

Flannel is now a helm chart and included in the overall `install-lilt.sh` script and no longer a separate deployment. If upgrading LILT from a previous version and `flannel `is already installed, comment out the flannel section of the install script:

```bash theme={null}
# install-lilt.sh

# Install flannel, on-prem customers only
kubectl label --overwrite ns kube-flannel pod-security.kubernetes.io/enforce=privileged
sh install_scripts/install-flannel.sh
# wait until pod ready
kubectl wait --namespace kube-flannel --for=condition=ready pod -l app=flannel --timeout=180s
```

If installing `flannel` for the first time via the helm chart, ensure that the `podCidr` is consistent with K8S cluster settings:

```python theme={null}
# flannel/on-prem-values.yaml

flannel:
  podCidr: "192.168.100.0/19"
```

### Redis

Memory limits have been implemented to prevent pod restarts/crashes. This ensures that consumed memory does not exceed pod resource limits. Settings for `maxmemory` must be slightly below `pod mem limits`. Memory can be increased if required:

```python theme={null}
# redis/on-prem-values.yaml

global:
  redis:
    # maxmem needs to be just below resource limit
    maxmemory: "5.8gb"
    maxmemoryPolicy: "allkeys-lru"  # evict least recently used keys
#
master:
  resources:
    limits:
      memory: 6G
```

Persistent storage is now disabled by default. This increases performance and reduces storage requirements.

```cpp theme={null}
# redis/on-prem-values.yaml

master:
  persistence:
    enabled: false
```

If persistence is required for cache security logging/audit, reenable and ensure that the storage size is sufficient for estimated usage:

```cpp theme={null}
# redis/on-prem-values.yaml

master:
  persistence:
    enabled: true
    size: 20Gi   # up to 100Gi for heavy usage
```

### Istio

Additional `kernel`parameters are required to prevent `ztunnel` pod restarts:

```bash theme={null}
# avoid ztunnel container restarts due to load
# append to end of file
cat <<EOF >> /etc/security/limits.conf
soft nofile 131072
hard nofile 131072
EOF

cat <<EOF >> /etc/systemd/system.conf
DefaultLimitNOFILE=131072
EOF
```

### Firewall Ports

Additional ports are required for `Istio`, `api`, `Clickhouse`, and `Flannel`. Please ensure that the following are enabled:

```csharp theme={null}
firewall-cmd --permanent --add-port={22,80,443,2379,2380,5000,6443,10250,10251,10252,10255}/tcp
# api
firewall-cmd --permanent --add-port={5005,8011,8080}/tcp
# istio
firewall-cmd --permanent --add-port={15000,15001,15006,15008,15009,15010,15012,15014,15017,15020,15021,15090,15443,20001}/tcp
# flannel
firewall-cmd --permanent --add-port=8472/udp
# clickhouse
firewall-cmd --permanent --add-port=8123/tcp
```

### Containerd

Additional workloads now run on the `GPU` node in parallel with the `Worker` node. If NOT using a centralized repository for all images, ensure that the following are loaded via `containerd` on the `GPU` node:

```powershell theme={null}
pilot*
proxyv2*
install-cni*
ztunnel*
kiali*
flannel*     # (only if new install)
k8s-device-plugin*
metrics-server*
neural*   # (all neural from docker_images master/node)
llm*
batch*
```

## 2024 Q2

### **Hardware Requirements**

Due to the inclusion of newer models by default, the hardware requirements have changed. [Resource Metrics](/self-managed/v6.1/resource-metrics) reflects the additional deployments that need to be considered, and the following recommendations have been updated:

* Master node disk requirement has increased from 200 GB to 500 GB to accommodate for additional container images, configuration, and logging.

* GPU Node instance type updated from `g4dn.2xlarge (8 vCPUs, 32 GB RAM)` to `g4dn.8xlarge (32 vCPUs, 128 GB RAM)`, in order to be able to run the V4 neural services.

* Due to the added models, the hard disk space requirements have been increased. See [Installation Requirements](/self-managed/v6.1/install-system-amazon-linux-2023-or-rocky-8-9) for more information.

#### V4 Language Model Updates

As we introduce newer, more accurate language models into LILT, we’ve continually updated our hardware requirements. See [Language Models](/self-managed/v6.1/v2-language-models) for the latest in V4 model information. More information can be found in the Knowledge Base around Resource Requirements.

### **Operating System Requirements**

#### CentOS 7 → Rocky Linux 8

New software features of the LILT platform are incompatible with CentOS 7\[1]. All installations still on CentOS 7 should migrate before adopting this release.

Some previous LILT installations were done using CentOS 7, which reached End of Life (EOL) support as of June 30, 2024. The recommended base operating system, and the one being tested in our QA environment, is using Rocky Linux 8. Rocky Linux 8 provides a secure environment similar to CentOS 7, with an End of Life (EOL) support date of May 2029.

#### Istio Module Support

##### Modules

All installations have updated modules needed to support Istio. See the section regarding [kernel modules](/self-managed/v6.1/install-system-amazon-linux-2023-or-rocky-8-9) which now includes the following modules to install:

```
overlay
br_netfilter
nf_nat
xt_REDIRECT
xt_owner
iptable_nat
iptable_mangle
iptable_filter
```

These will need to be installed on all existing nodes.

##### Ports

All installations have updated firewall port changes needed to support Istio. See the section regarding [Firewall Settings](/self-managed/v6.1/install-system-amazon-linux-2023-or-rocky-8-9) which now includes opening up ports `15000,15001,15006,15008,15009,15010,15012,15014,15017,15020,15021,15090,15443,20001}/tcp` for Istio.

### **Configuration Updates**

#### Custom Domains

New configurations should be done, as described in [Set custom Domain and Certificates](/self-managed/v6.1/set-custom-domain-and-certificates) and [Set Connectors Domain](/self-managed/v6.1/connector-login-credentials).

#### Upgrading Process

The upgrading process, which involves Helm values files, has been updated for this release to make upgrading simpler in the future. See [Q2 2024 Updates](/self-managed/v6.1/upgrade-system#UpgradeSystem-Q22024Updates) for more details.

#### MinIO Resize

In previous releases, the PersistentVolume for MinIO was set to 200GB. With the release of newer models, this is no longer large enough to support them. The default size has been updated from 200GB to 400GB, however, this will not automatically resize existing installations.

If your backing MinIO PersistentVolume is resizable, please resize to 400GB. If it is not able to be resized, the recommended procedure is as follows:

* Back up the MinIO data (if necessary)

* Delete the MinIO PersistentVolumeClaim and PersistentVolume

* Restart the new MinIO deployment

* Restore MinIO data (if necessary)

#### WPA Metrics

WPA metrics, as described in [Generate Evaluation Metrics (WPA, BLEU)](/self-managed/v6.1/generate-evaluation-metrics-wpa-bleu) , are now enabled by default.

#### SMTP Notifications

See the new page around configuring SMTP Notifications here: [SMTP Email Notifications](/self-managed/v6.1/smtp-email-notifications)

#### Guide on how to handle GPU worker counts

As GPU processing has become increasingly critical in LILT’s models, we’ve added [Configuring GPU Worker Counts in LILT](/self-managed/v6.1/configuring-gpu-worker-counts-in-lilt) to assist system administrators in configuring the LILT application for GPU use.

### Vulnerability (CVE) Scan Results

LILT has conducted thorough scans of all services and components to confirm there are no components rated as High or Critical CVEs\[2]. Self-Hosted customers can find further details in the CVE Scan PDF provided with the release.

### **Known Issues**

#### MongoDB Upgrade

The latest MongoDB version has a known [issue](https://github.com/bitnami/charts/issues/27604) that may cause it to fall into a CrashLoop upon upgrading. If this occurs, the recommended fix is as follows:

* Back up the MongoDB data (if necessary)

* Delete the MongoDB PersistentVolumeClaim and PersistentVolume

* Restart the new MongoDB deployment

* Restore MongoDB data (if necessary)

#### Known CVE issues

| **Vulnerability Reference** | **Application** | **Mitigation / Notes** |
| - | - | - |
| CVE-2024-31580 | neural | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
| CVE-2024-31583 | neural | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
| CVE-2023-6378 | core-api | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
| CVE-2023-6481 | core-api | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
| CVE-2024-22257 | core-api | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
| CVE-2016-1000027 | core-api | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
| CVE-2024-22243 | core-api | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
| CVE-2024-22259 | core-api | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
| CVE-2024-22262 | core-api | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
| CVE-2023-32697 | core-api | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
| CVE-2022-1471 | core-api | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
| CVE-2022-25857 | core-api | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
| CVE-2024-21634 | core-api | To be fixed in next release, requires upgrade of package that is used by internal dependencies. |
| GHSA-m425-mq94-257g | localpv-provisioner | Vulnerability exists in latest version of this application. Waiting on a newer release to fix. |
| CVE-2024-24790 | localpv-provisioner | Vulnerability exists in latest version of this application. Waiting on a newer release to fix. |
| CVE-2023-39325 | localpv-provisioner | Vulnerability exists in latest version of this application. Waiting on a newer release to fix. |
| CVE-2023-45283 | localpv-provisioner | Vulnerability exists in latest version of this application. Waiting on a newer release to fix. |
| CVE-2023-45287 | localpv-provisioner | Vulnerability exists in latest version of this application. Waiting on a newer release to fix. |
| CVE-2023-45288 | localpv-provisioner | Vulnerability exists in latest version of this application. Waiting on a newer release to fix. |
| CVE-2023-1370 | elasticsearch | Vulnerability exists in latest version of this application. Waiting on a newer release to fix. |
| CVE-2021-40690 | elasticsearch | Vulnerability exists in latest version of this application. Waiting on a newer release to fix. |
| CVE-2022-1471 | elasticsearch | Vulnerability exists in latest version of this application. Waiting on a newer release to fix. |
| CVE-2024-41110 | istiod | Vulnerability exists in latest version of this application. Waiting on a newer release to fix. |

### Appendix

\[1] The Analytics dashboard relies on Istio, which uses some underlying code libraries in the operating system kernel that are not present in CentOS 7.

\[2] Fixed High and Critical CVEs refer to vulnerabilities for which fixes are available and do not break the integration of the application and its dependencies. It should be noted that vulnerabilities are continuously discovered, and new CVEs may have been identified but not yet addressed.