Install from a Bastion

install-lilt-eks.sh has to run somewhere that can reach the Kubernetes API. When the EKS endpoint is private, which is the default for clusters created by the lilt-aws-env Terraform module, an operator’s laptop has no such route. The install runs from a small EC2 instance inside the VPC instead. This article covers what that machine needs: the tools, the IAM permissions, and the order to do things in. The Terraform module creates the instance when you set enable_bastion = true. Your own jump host works equally well. Nothing here depends on the module having created it.

Why a Bastion

The EKS API is private. helm, kubectl, and every install script need to reach it.
Seeding the registry moves a lot of data. seed-registry.sh pushes about 180 GB of images into your registry. From inside the VPC that traffic stays on the AWS network. From a laptop it crosses your uplink twice.
The install takes hours. A long-running unit on an instance survives a closed laptop lid.

Sizing

Give the instance a root volume with room for the largest image in the release, which is roughly 47 GB. A 120 GB volume is comfortable. seed-registry.sh streams one image at a time, so it does not need room for all 180 GB at once.

Tools

Install these before you start. The installer does not install them for you, and a missing tool usually surfaces late, after an hour of successful steps.

Tool	Version	Why
AWS CLI	2.x	Every script that touches AWS. Version 1 is not sufficient.
`kubectl`	Matching the cluster minor version	Applies manifests and waits on rollouts.
Helm	4.x	`get-helm-4` installs the current Helm 4. The shipped charts use OCI features and flags Helm 3 does not have.
`crane`	0.22 or later	`seed-registry.sh` copies images with it and does not install it.
`jq`	Any	Parsing AWS and Kubernetes JSON.
`tar`, `gzip`	Any	Unpacking the release tarball.
`openssl`	Any	Only if you seed a TLS certificate yourself.

On Amazon Linux 2023:

dnf install -y jq tar gzip

curl -fsSLo /usr/local/bin/kubectl \
  "https://dl.k8s.io/release/$(curl -fsSL https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x /usr/local/bin/kubectl

curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-4 \
  | HELM_INSTALL_DIR=/usr/local/bin bash

curl -fsSL "https://github.com/google/go-containerregistry/releases/download/v0.22.0/go-containerregistry_Linux_x86_64.tar.gz" \
  | tar xz -C /usr/local/bin crane
chmod +x /usr/local/bin/crane

The AWS CLI is preinstalled on Amazon Linux 2023. On other distributions, follow Install or update to the latest version of the AWS CLI in the AWS documentation. Version 1 is not sufficient.

IAM Permissions

The bastion’s instance profile needs whatever the install scripts call. The policy below is derived from the scripts themselves, rather than discovered one failed run at a time. Scope the resources to your own account, region, and names. Attach it to the module’s bastion with bastion_extra_policy_arns.

data "aws_iam_policy_document" "lilt_installer" {
  # seed-registry.sh authenticates, pushes the release images, and pulls them
  # back. Drop both ECR blocks if your registry is not ECR.
  statement {
    sid       = "EcrAuth"
    actions   = ["ecr:GetAuthorizationToken"]
    resources = ["*"]
  }

  statement {
    sid = "EcrReadWrite"
    actions = [
      "ecr:BatchGetImage",
      "ecr:GetDownloadUrlForLayer",
      "ecr:BatchCheckLayerAvailability",
      "ecr:InitiateLayerUpload",
      "ecr:UploadLayerPart",
      "ecr:CompleteLayerUpload",
      "ecr:PutImage",
      "ecr:DescribeRepositories",
      "ecr:DescribeImages",
      "ecr:ListImages",
      "ecr:CreateRepository",
    ]
    resources = ["arn:aws:ecr:${var.region}:${var.account_id}:repository/*"]
  }

  # install-rds-secrets.sh reads the endpoint, and the managed master secret
  # when the database was created with one.
  statement {
    sid       = "ReadDatabase"
    actions   = ["rds:DescribeDBInstances"]
    resources = ["*"]
  }

  statement {
    sid       = "ReadDatabaseSecret"
    actions   = ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"]
    resources = ["*"]
  }

  # install-s3-irsa.sh finds the app-workload role by name.
  statement {
    sid       = "ListRolesForRoleLookup"
    actions   = ["iam:ListRoles"]
    resources = ["*"]
  }

  # install-dns-records.sh writes the service records and waits for the change
  # to propagate. DescribeLoadBalancers resolves the load balancer's hosted
  # zone ID, which an alias record needs.
  statement {
    sid       = "WriteDnsRecords"
    actions   = ["route53:ChangeResourceRecordSets", "route53:ListResourceRecordSets"]
    resources = ["arn:aws:route53:::hostedzone/${var.hosted_zone_id}"]
  }

  statement {
    sid       = "ReadDnsChangeStatus"
    actions   = ["route53:GetChange"]
    resources = ["*"]
  }

  statement {
    sid       = "DescribeLoadBalancers"
    actions   = ["elasticloadbalancing:DescribeLoadBalancers"]
    resources = ["*"]
  }
}

Two notes on scoping:

ecr:GetAuthorizationToken, route53:GetChange, and the Describe* actions do not accept a resource ARN. "*" is the only valid value, and the API itself is the boundary.
Narrow ReadDatabaseSecret to the specific secret where you can. When RDS manages the master password, the secret’s ARN is not known until the database exists, so scoping by a resource tag is usually easier than scoping by ARN.

Beyond this list, add whatever your own delivery path needs, such as read access to the bucket that holds the release tarball.

Step 1: Get the Release onto the Bastion

Copy the tarball across however you normally move artifacts, then unpack it:

mkdir -p /opt/lilt && cd /opt/lilt
tar xzf on-prem-installer-eks_<hash>.tar.gz
test -f install-lilt-eks.sh   # confirms this is the EKS flavor

See Install package for how to download and verify the tarball.

Step 2: Point kubectl at the Cluster

aws eks update-kubeconfig --region <region> --name <cluster-name>
kubectl get nodes

If kubectl get nodes times out, the bastion has no route to the API endpoint. Fix that before you go further. Every later step depends on it, and the failures it produces are much harder to read than this one.

Step 3: Write install.env

cp install.env.example install.env
chmod 0600 install.env

Fill it in as described in Install System (AWS EKS). Keep the file at mode 0600. It holds credentials.

Step 4: Seed the Registry

sh install_scripts/seed-registry.sh --help

This is the long step. It skips images your registry already has, so a re-run after a failure resumes rather than restarting. See Seed your container registry for the full options.

Point SEED_REGISTRY_WORK_DIR at real disk. It defaults to /var/tmp for a reason: on most current distributions /tmp is a tmpfs sized at half of system RAM, and staging a large image there fails with No space left on device while df shows the root volume nearly empty.

Step 5: Run the Install

sh install-lilt-eks.sh 2>&1 | tee /var/log/lilt-install.log

Budget roughly two hours. Keep the log. When something fails at minute 90, it is the only account of what happened.

Optional: run the installer as a systemd unit

For an environment you expect to rebuild, running the installer as a unit rather than in a shell survives a dropped connection and keeps the log in one place.

[Service]
Type=oneshot
RemainAfterExit=yes
TimeoutStartSec=infinity
WorkingDirectory=/opt/lilt
ExecStart=/opt/lilt/install-lilt-eks.sh
StandardOutput=append:/var/log/lilt-install.log
StandardError=append:/var/log/lilt-install.log

With RemainAfterExit=yes, use systemctl restart to run the installer again. A systemctl start against a unit that already succeeded is a silent no-op, which reads exactly like a run that finished instantly.

Step 6: Verify

Three checks, working outwards from the cluster to the browser. Run them in order: each one only makes sense if the previous one passed. 1. Are the pods healthy?

kubectl get pods -n lilt | grep -v Running | grep -v Completed

The filter leaves only pods that need attention, so no output is the result you want. A pod in Init or ContainerCreating shortly after the install is usually still pulling an image; the neural images are tens of gigabytes. Anything in CrashLoopBackOff or Pending needs kubectl describe pod. 2. Did the load balancer get an address?

kubectl get svc -n istio-ingressgateway

The EXTERNAL-IP column should hold a hostname, not <pending>. Stuck at <pending> means no controller claimed the Service: on EKS, the AWS Load Balancer Controller is missing or lacks permissions. Nothing outside the cluster can reach LILT until this resolves. 3. Does the whole path work from outside?

curl -sS -o /dev/null -w '%{http_code} verify=%{ssl_verify_result}\n' https://<your-host>/

This tests DNS, the load balancer, the gateway, and the certificate in one request.

verify=0 is the part that matters. It means the certificate chain validated. A chain missing its intermediate still satisfies a browser that cached it, then fails in a stricter client later, so check it here rather than by loading the page.
Any status code means the request reached the gateway. A 200, 302, or 401 are all fine at this stage. A connection or TLS error is not.

If curl reports a TLS error, see Set custom domain and certificates.

Tear Down in the Right Order

To decommission an environment, uninstall LILT from inside the cluster before you run terraform destroy:

WIPE_DATA=true PURGE_CRDS=true sh uninstall-lilt.sh

Destroying the AWS resources first leaves the load balancer and the volumes that persistent volume claims created still attached to network interfaces in the VPC. The VPC deletion then hangs.

When It Fails

The install is re-runnable. Fix the cause and run it again. Every component uses helm upgrade --install, which is a no-op when the chart and values are unchanged. See Resuming a failed install.

​Why a Bastion

​Sizing

​Tools

​IAM Permissions

​Step 1: Get the Release onto the Bastion

​Step 2: Point kubectl at the Cluster

​Step 3: Write install.env

​Step 4: Seed the Registry

​Step 5: Run the Install

​Step 6: Verify

​Tear Down in the Right Order

​When It Fails

Why a Bastion

Sizing

Tools

IAM Permissions

Step 1: Get the Release onto the Bastion

Step 2: Point kubectl at the Cluster

Step 3: Write install.env

Step 4: Seed the Registry

Step 5: Run the Install

Step 6: Verify

Tear Down in the Right Order

When It Fails