Install System (AWS EKS)

This article describes how to install LILT on Amazon Elastic Kubernetes Service from the EKS install package. It covers configuration, the phased install, secrets, verification, and recovery.

Overview

An EKS deployment has two layers:

The AWS infrastructure: a VPC and subnets, the EKS cluster and its node groups, an RDS MySQL instance, an S3 bucket, an SQS queue, a container registry, and the IAM roles the workloads assume.
The LILT application stack, deployed by install-lilt-eks.sh: Istio and the Gateway API routes, the data stores, and the LILT Helm charts.

Choose one of two paths for the first layer:

Path	Use it when	Guide
Terraform module	You want LILT to provision the infrastructure. The `lilt-aws-env` module ships in the install package.	Provision AWS infrastructure with Terraform
Bring your own	You already have a cluster, or you provision AWS resources with your own tooling.	Install on an existing EKS cluster

Both paths converge here for the second layer.

For a bare-metal or virtual-machine deployment, use the on-premises install package instead. See Install System (Amazon Linux 2023 or Rocky 8/9).

Prerequisites

An EKS cluster, version 1.32 or later, with the EBS CSI driver installed. Version 1.35 is the tested default, and the LILT Terraform module creates 1.35.
Helm 4, plus kubectl, aws, and jq on your PATH.
Check your Helm version before you start.
helm version --short # must report v4.x
Several component scripts use flags that exist only in Helm 4, among them --rollback-on-failure for ArgoCD and Argo Rollouts and --server-side for the Elasticsearch cluster and the LILT charts. On releases whose install-lilt-eks.sh does not check the version at startup, Helm 3 gets through the prerequisites and the data stores and then fails an hour in, reporting an unrecognised flag rather than the Helm version.
Add crane if you populate your registry with seed-registry.sh. The installer does not install it, and the script stops with ERROR: crane not on PATH. Skip it only if you mirror the release images by some other means.
An RDS MySQL instance, an S3 bucket, and an SQS queue that the cluster can reach.
A container registry the cluster can pull from, seeded with the release images. See Seed your container registry.
IAM roles for the workloads that need AWS access, bound with EKS Pod Identity or IRSA.
The extracted install package. See Install package.

Run the installer from a machine that can reach the Kubernetes API. Clusters created by the LILT Terraform module have a private API endpoint, so the install runs from inside the VPC. See Install from a bastion.

Node Kernel Settings

The EKS-optimized Amazon Linux 2023 AMI already carries the kernel modules, sysctl values and swap and SELinux settings that Kubernetes needs, so a cluster on the stock AMI needs nothing here. Two settings are not part of that baseline and matter to LILT specifically. Check them on your nodes whichever AMI you use:

# Istio's ztunnel restarts under load without a raised file-descriptor limit.
ulimit -n                        # want 131072

# Elasticsearch fails to start below this.
sysctl vm.max_map_count          # want 262144

To set them, on each node:

cat <<EOF >> /etc/security/limits.conf
soft nofile 131072
hard nofile 131072
EOF

echo "DefaultLimitNOFILE=131072" >> /etc/systemd/system.conf

echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
sysctl --system

Custom or non-EKS-optimized AMI: the full baseline

If you build your own AMI, or run a node OS that is not the EKS-optimized image, the Kubernetes baseline is yours to provide as well. Bake this into the image rather than applying it to running nodes, so replacements and scale-ups inherit it.

# Kernel modules for kube-proxy and the Istio redirect rules.
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
nf_nat
xt_REDIRECT
xt_owner
iptable_nat
iptable_mangle
iptable_filter
EOF

for m in overlay br_netfilter nf_nat xt_REDIRECT xt_owner \
         iptable_nat iptable_mangle iptable_filter; do
  modprobe "$m"
done

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

sysctl --system

# The kubelet does not start with swap enabled.
swapoff -a
sed -e '/swap/s/^/#/g' -i /etc/fstab

# SELinux, on distributions that enforce it.
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

An unset xt_REDIRECT or xt_owner is worth calling out: the mesh’s traffic redirection depends on them, and without them pods start and then fail to reach each other, which reads as a networking or DNS fault rather than a missing kernel module.

Configure the Install

Copy the template and edit it. Every variable the installer reads is documented there with its default.

cp install.env.example install.env
chmod 0600 install.env

The file is organised by intent: identity, pre-existing resources, DNS and TLS, image registry, feature toggles, and the secret backend. Leave a line commented to keep the default shown next to it. Every variable is listed in the install.env reference. The settings almost every install needs:

Variable	Purpose
`SUBDOMAIN`, `DNS_DOMAIN`	Every ingress hostname is `<name>-<SUBDOMAIN>.<DNS_DOMAIN>`. Both are required.
`AWS_REGION`, `AWS_ACCOUNT_ID`	Resource ARNs and registry hosts. The account ID defaults to your current caller identity.
`REGISTRY_BASE`	The registry to pull images from.
`S3_BUCKET`, `SQS_QUEUE_URL`, `DB_NAME`	The pre-existing resources to use.
`CONNECTORS_ADMIN_PASSWORD`	The `admin` login for the `/connectors-admin` console. The install stops with an error if this is unset, so choose one before you start.
`SECRET_BACKEND`	`seed`, `vault`, or `aws`. Defaults to `seed`.
`IAM_AUTH_METHOD`	`irsa` or `pod_identity`. Defaults to `irsa`. It must match how your IAM roles are actually bound, or every AWS call from a pod fails.
`PREFIX`	Optional. Set it to the Terraform module’s `prefix` and the installer derives the resource names for you.

How the Installer Works

install-lilt-eks.sh is a flat sequence of component scripts. Each one:

Prints --- Installing <Component> --- before it runs, so the last such line names the component that failed.
Checks its own ENABLE_<NAME> variable and exits without doing anything when that variable is not true.
Uses helm upgrade --install, which is a no-op when the chart and its values are unchanged.

Together these three properties make the whole script safe to re-run from the top. That is the intended recovery path.

Component Flags

Set any of these to false in install.env to skip that component. Optional components, enabled by default: ENABLE_CLUSTER_AUTOSCALER, ENABLE_MONGODB, ENABLE_CLICKHOUSE, ENABLE_DRAGONFLY_OPERATOR, ENABLE_CACHE_DRAGONFLY, ENABLE_RABBITMQ, ENABLE_ECK_OPERATOR, ENABLE_ECK_CLUSTER, ENABLE_ARGOCD, ENABLE_ARGO_ROLLOUTS, ENABLE_ARGO_WORKFLOWS, ENABLE_PROMETHEUS, ENABLE_METRICS_SERVER, ENABLE_ISTIO_INGRESSGATEWAY, ENABLE_QDRANT, ENABLE_PROXY_SQL, ENABLE_WSO2, ENABLE_MYSQL_CLIENT, ENABLE_RAY_OPERATOR_CRDS, ENABLE_RAY_OPERATOR, ENABLE_RAYMAN, ENABLE_LILT_CHARTS, ENABLE_EXTERNAL_SECRETS, ENABLE_VAULT, ENABLE_NVIDIA_GPU_OPERATOR. Optional components, disabled by default:

Flag	Component
`ENABLE_DNS`	Route53 record creation
`ENABLE_CERT_MANAGER`	cert-manager with Route53 DNS-01
`ENABLE_SEED_REGISTRY`	Copying the release images into your registry

install.env.example also lists ENABLE_DATADOG and ENABLE_OTELCOL. Neither entrypoint calls an installer for them, and neither component ships in the tarball, so setting these has no effect on a customer install.

Prerequisites, enabled by default: ENABLE_NAMESPACE, ENABLE_CLEANUP_STALE_PODS, ENABLE_CLUSTER_PREREQS, ENABLE_RDS_SECRETS, ENABLE_S3_IRSA, ENABLE_RDS_LILT_USER, ENABLE_DATABASE_BOOTSTRAP, ENABLE_PREINSTALL_ISTIO_CRDS, ENABLE_GATEWAY_API_CRDS, ENABLE_ISTIOD, ENABLE_LILT_NETWORKING.

Prerequisites are foundational, and everything after them depends on them. Disable one only on a cluster where that step has already completed.

Dependency order matters among the optional components too. The ECK operator has to exist before its cluster, the Ray operator before Rayman, and ProxySQL fronts the database.

Phased Bring-Up

Bringing the infrastructure and data stores up first, and the application afterwards, makes a failure much easier to attribute. Phase 1: infrastructure and data stores. Add these lines to install.env:

ENABLE_ARGOCD=false
ENABLE_ARGO_ROLLOUTS=false
ENABLE_ARGO_WORKFLOWS=false
ENABLE_PROMETHEUS=false
ENABLE_QDRANT=false
ENABLE_PROXY_SQL=false
ENABLE_WSO2=false
ENABLE_MYSQL_CLIENT=false
ENABLE_RAY_OPERATOR_CRDS=false
ENABLE_RAY_OPERATOR=false
ENABLE_RAYMAN=false
ENABLE_LILT_CHARTS=false

Run the installer, then confirm that the data stores are healthy:

sh install-lilt-eks.sh
kubectl get pods -n lilt

Phase 2: the full stack. Remove those lines so each component reverts to its default, and run the installer again.

Secrets

SECRET_BACKEND selects where the installer reads application secrets from: seed (the default, from the file in the tarball), vault, or aws. LILT also ships an in-cluster HashiCorp Vault as the default External Secrets Operator backend.

The seed file holds placeholder values, most of them the literal string dummypass. Do not run a production install with them. On EKS the file is overwritten from the seed on every run while SECRET_BACKEND=seed, so edits there do not persist.

See Secrets and Vault for the backend comparison, the full Vault path and key table, the per-secret path overrides, seed-vault-secrets.sh, and how to point LILT at a Vault or AWS Secrets Manager you already run.

Override Values

Every .yaml file under lilt/environments/lilt/values.d/ is applied after the environment overlay. Drop a whole override file in there rather than editing the values files the installer ships. Your overrides then survive an upgrade. GPU sizing and single sign-on are configured the same way, by copying a shipped profile into that directory. See GPU profiles and values overlays for the profile tables, the memory and compute-capability floors, and the rule for combining files.

Single Sign-On

LILT ships with external SSO disabled, so password sign-in works out of the box and no install points at an identity provider you do not control. To connect a provider, copy a file from lilt/sso-profiles/ into lilt/environments/lilt/values.d/, then add the client secret as the singleOidcClientSecret key of the lilt-secrets secret. See GPU profiles and values overlays for the full value list.

Set SINGLE_OIDC_PROVIDER_ENABLED=true only when every other SINGLE_OIDC_PROVIDER_* value is filled in. With one of them empty, front fails its startup validation and the pod crashes.

Run the Install

sh install-lilt-eks.sh 2>&1 | tee /var/log/lilt-install.log

Budget roughly two hours. Keep the log: when something fails 90 minutes in, it is the only account of what happened.

Verify

# Anything not Running or Completed needs a look.
kubectl get pods -n lilt | grep -v Running | grep -v Completed

# The ingress load balancer should have an address.
kubectl get svc -n istio-ingressgateway

# The application should answer through the ingress.
curl -sS -o /dev/null -w '%{http_code} verify=%{ssl_verify_result}\n' https://<SUBDOMAIN>.<DNS_DOMAIN>/

Read them outwards from the cluster. The first command filters to pods that need attention, so no output is the result you want. The second should show a hostname in EXTERNAL-IP, not <pending>. The third tests DNS, the load balancer, the gateway and the certificate in one request: any status code means it reached the gateway, and verify=0 means the certificate chain validated. A chain missing its intermediate still satisfies a browser that has cached it, and fails later in a stricter client. For the same checks with the failure modes spelled out, see Install from a bastion.

Resuming a Failed Install

The installer runs with set -e, so it stops at the first failing component and the last --- Installing <Component> --- line names it.

Inspect the component that failed:

kubectl get pods -n lilt
kubectl describe pod <pod> -n lilt
kubectl logs <pod> -n lilt

Fix the underlying cause: a missing value, an unreachable dependency, or a resource limit.
Run the installer again from the top. Components that already succeeded are no-ops.

To skip a component you have already dealt with, set its flag for that run only:

ENABLE_MONGODB=false sh install-lilt-eks.sh

Common Failures

A pod stays in ContainerCreating for more than 15 minutes. The image is usually still pulling. The neural images are tens of gigabytes. Check kubectl describe pod for a pull error before you intervene. UPGRADE FAILED: timed out waiting for the condition. A component took longer to become ready than its timeout allowed. Confirm the pods are healthy and run the installer again. A pod cannot reach S3. Check that the service account carries no eks.amazonaws.com/role-arn annotation if the cluster uses EKS Pod Identity, and that no static AWS credentials are set in the secrets file. Either one overrides workload identity.

Uninstall

To remove LILT and leave the data in place:

sh uninstall-lilt.sh

The script tears the stack down in dependency order: application, then data services, then platform. It also removes the objects Helm never owned, which a bare helm uninstall leaves behind. Data volumes are kept by default. To destroy them as well, opt in on the command line:

WIPE_DATA=true sh uninstall-lilt.sh

To remove the installer’s custom resource definitions too:

PURGE_CRDS=true sh uninstall-lilt.sh

Pass WIPE_DATA and PURGE_CRDS on the command line only. Never put them in install.env, which persists across installs.

To decommission an environment entirely, run the uninstall before you destroy the AWS resources. Destroying them first leaves the load balancer and the volumes behind persistent volume claims still attached to network interfaces, and the VPC deletion hangs.

Neither form removes the AWS resources themselves. Delete those separately, or with terraform destroy if you created them with the module.

​Overview

​Prerequisites

​Node Kernel Settings

​Configure the Install

​How the Installer Works

​Component Flags

​Phased Bring-Up

​Secrets

​Override Values

​Single Sign-On

​Run the Install

​Verify

​Resuming a Failed Install

​Common Failures

​Uninstall

​Related Articles

Overview

Prerequisites

Node Kernel Settings

Configure the Install

How the Installer Works

Component Flags

Phased Bring-Up

Secrets

Override Values

Single Sign-On

Run the Install

Verify

Resuming a Failed Install

Common Failures

Uninstall

Related Articles