Skip to main content
LILT reads its application secrets from one of three places. This article covers how to choose, what the shipped in-cluster Vault does, and how to point LILT at a secret manager you already run.

​
Choose a Backend

SECRET_BACKEND in install.env selects the source.
External Secrets Operator is installed only on the EKS path. install-lilt.sh never calls install-external-secrets.sh, and the external-secrets component is not in the on-premises tarball. On an on-premises install, SECRET_BACKEND has no effect and secrets always come from the seed file.

​
Seed Mode

With SECRET_BACKEND=seed, secrets come from lilt/environments/seed_custom_values/secrets.yaml on the on-premises flavor and eks-secrets.yaml on EKS. The installer copies that file to lilt/environments/<ENV_NAME>/secrets.yaml, where ENV_NAME defaults to lilt.
The shipped values are placeholders, most of them the literal string dummypass. They let a fresh cluster start without a secret manager. Do not run a production install with them.
The two flavors differ in how the copy behaves on a re-run:
  • On-premises seeds the file only if it does not already exist, so your edits survive.
  • EKS overwrites it from the seed file on every run while SECRET_BACKEND=seed. Edits do not persist. To manage EKS secrets outside the tarball, use vault or aws.
On EKS the AWS credential fields are intentionally empty. Workload identity supplies them. A static value there overrides the role and breaks S3 access.

​
The Shipped Vault

ENABLE_VAULT defaults to true, so both flavors install a single-node HashiCorp Vault into a namespace called vault, backed by a 10 GiB Raft volume. Override the seal mode with VAULT_SEAL_MODE. Setting shamir on EKS is supported: the installer strips the KMS seal stanza before it installs.

​
Initialisation and Unseal

vault-bootstrap.sh runs after the pod starts, gated by ENABLE_VAULT_BOOTSTRAP (default true). It initialises Vault, enables the KV v2 mount, configures Kubernetes auth, and unseals.
The initialisation output, which contains the unseal key and the root token, is written to a Kubernetes Secret named vault-init in the vault namespace. Nothing is printed and nothing is written to disk.Anyone who can read that Secret can unseal Vault and mint tokens. Restrict access to it, back it up somewhere outside the cluster, and treat losing it as losing the Vault. Once you have another way in, such as userpass or OIDC auth, revoke the root token.
On an on-premises install, a pod restart or a node drain brings Vault back sealed. A sealed pod is Running but not Ready. The bootstrap runs at install time only, so nothing unseals it automatically. Re-run sh install_scripts/vault-bootstrap.sh, which is idempotent, or unseal manually. On EKS, KMS auto-unseal handles this.

​
How ESO Reads It

On EKS the installer creates a ClusterSecretStore pointing at http://vault.vault.svc.cluster.local:8200, authenticating with the Kubernetes auth method as the external-secrets service account against a Vault role named eso. That role’s policy grants read-only access to the KV mount. Set BASE_VAULT_PATH to a prefix that suits your environment, for example lilt/prod/aws.

​
What Vault Must Contain

Paths are BASE_VAULT_PATH plus the suffix below. These are KV v2 mounts: the data/ infix appears in the API path but not in the vault kv CLI, so vault kv get lilt/prod/aws/mysql addresses what ESO calls lilt/data/prod/aws/mysql. datadog with key api_key is seeded too, but only matters if you enable Datadog. External Secrets Operator reads secrets. It never creates them. Every path must exist before you install.

​
Overriding a Single Path

Each secret has its own <NAME>_SECRET_PATH variable that overrides the derived path. The variable is named after the secret rather than the path suffix, which is easy to get wrong for the common/ entries.

​
Seed the Paths

install_scripts/seed-vault-secrets.sh creates every path with a placeholder, so a first install does not fail one missing key at a time. It derives the path and key list from the same ExternalSecret manifests the operator reads, so it cannot drift from what LILT actually asks for. Authenticate to the target Vault first, with VAULT_ADDR set, then run it from the tarball root:
The script is non-destructive. It writes only keys that are missing and leaves existing values alone, so re-running it never overwrites a real secret with a placeholder. Two behaviours worth knowing:
  • Values default to the literal REPLACE_ME. Override with PLACEHOLDER, or set SEED_RANDOM=true to generate a random value per key, for an install that has to come up unattended.
  • The mysql path is the exception. Its username, mysql-password, and host are taken from the database user the installer actually creates, because ESO feeds them to every application and they have to match.
After the install, replace every remaining placeholder with a real value.

​
Use Your Own Vault

To point LILT at a Vault you already run, set SECRET_BACKEND=vault and VAULT_AUTH=approle, with VAULT_ADDR pointing at your server. AppRole authentication needs a Kubernetes Secret named vault-approle in the external-secrets namespace, holding the keys role-id and secret-id. Create it before you install:
The installer never creates this Secret and stops if it is missing. There are no VAULT_ROLE_ID or VAULT_SECRET_ID variables: keeping the credentials out of install.env keeps them out of a file that could be committed.
Your own Vault and the shipped in-cluster Vault are mutually exclusive. Two secret stores of the same name must never both be active on one cluster. Set ENABLE_VAULT=false when you use your own.

​
Use AWS Secrets Manager

Set SECRET_BACKEND=aws. External Secrets authenticates with workload identity as the external-secrets service account in the external-secrets namespace. The same path and key table applies, under the prefix you set. If you provision with the LILT Terraform module, grant that access with enable_eso_aws_sm = true and scope it with eso_aws_sm_secret_prefix. That role binds the service account External Secrets actually runs as, and its trust policy follows iam_auth_method.
Do not use enable_external_secrets = true for this. It creates an older, overlapping role whose Pod Identity association binds a service account named external-secrets-sa, which is not the name the External Secrets chart creates, and whose trust policy is hard-coded to Pod Identity regardless of iam_auth_method. Nothing errors: the role and the association are created and simply never attach to a pod, and External Secrets then fails with an access-denied error that does not mention the mismatch.

​
Turn the Shipped Vault Off

ENABLE_VAULT=false skips the Vault install entirely. With SECRET_BACKEND left unset it falls back to seed mode, which is the safe default.
Nothing guards the combination ENABLE_VAULT=false with SECRET_BACKEND=vault and the default VAULT_AUTH=kubernetes. The installer then points External Secrets at an in-cluster Vault that was never installed, and the install fails at the point where it waits for the first ExternalSecret to become ready. If you disable the shipped Vault, set SECRET_BACKEND to seed or aws, or use VAULT_AUTH=approle with your own server.

​
The Vault UI

ENABLE_VAULT_UI defaults to false. Setting it to true publishes the Vault UI on its own hostname, vault.<SUBDOMAIN>.<DNS_DOMAIN> by default, overridable with VAULT_UI_HOST. It needs a hostname of its own because the Vault UI serves only at the root with absolute asset paths, so it cannot live under a path prefix. Enabling it requires a DNS record for that hostname and a TLS certificate that covers it. See Set custom domain and certificates. Without both, use a port-forward instead: