> ## Documentation Index
> Fetch the complete documentation index at: https://support.lilt.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Secrets And Vault

LILT reads its application secrets from one of three places. This article covers how to choose, what the shipped in-cluster Vault does, and how to point LILT at a secret manager you already run.

## Choose a Backend

`SECRET_BACKEND` in `install.env` selects the source.

| Value            | Source                                                 | Available on |
| ---------------- | ------------------------------------------------------ | ------------ |
| `seed` (default) | The secrets file that ships in the tarball             | Both flavors |
| `vault`          | HashiCorp Vault, through External Secrets Operator     | EKS          |
| `aws`            | AWS Secrets Manager, through External Secrets Operator | EKS          |

<Warning>
  External Secrets Operator is installed only on the EKS path. `install-lilt.sh` never calls `install-external-secrets.sh`, and the `external-secrets` component is not in the on-premises tarball. On an on-premises install, `SECRET_BACKEND` has no effect and secrets always come from the seed file.
</Warning>

## Seed Mode

With `SECRET_BACKEND=seed`, secrets come from `lilt/environments/seed_custom_values/secrets.yaml` on the on-premises flavor and `eks-secrets.yaml` on EKS. The installer copies that file to `lilt/environments/<ENV_NAME>/secrets.yaml`, where `ENV_NAME` defaults to `lilt`.

<Warning>
  The shipped values are placeholders, most of them the literal string `dummypass`. They let a fresh cluster start without a secret manager. Do not run a production install with them.
</Warning>

The two flavors differ in how the copy behaves on a re-run:

* **On-premises** seeds the file only if it does not already exist, so your edits survive.
* **EKS** overwrites it from the seed file on **every** run while `SECRET_BACKEND=seed`. Edits do not persist. To manage EKS secrets outside the tarball, use `vault` or `aws`.

<Warning>
  On EKS the AWS credential fields are intentionally empty. Workload identity supplies them. A static value there overrides the role and breaks S3 access.
</Warning>

## The Shipped Vault

`ENABLE_VAULT` defaults to `true`, so both flavors install a single-node HashiCorp Vault into a namespace called `vault`, backed by a 10 GiB Raft volume.

|                        | On-premises                            | EKS                              |
| ---------------------- | -------------------------------------- | -------------------------------- |
| Seal mode              | Shamir, one key share                  | AWS KMS auto-unseal              |
| Unseal after a restart | Manual, or re-run the bootstrap script | Automatic                        |
| Consumed by ESO        | No — ESO is not on this path           | Yes, this is the default backend |

Override the seal mode with `VAULT_SEAL_MODE`. Setting `shamir` on EKS is supported: the installer strips the KMS seal stanza before it installs.

### Initialisation and Unseal

`vault-bootstrap.sh` runs after the pod starts, gated by `ENABLE_VAULT_BOOTSTRAP` (default `true`). It initialises Vault, enables the KV v2 mount, configures Kubernetes auth, and unseals.

<Warning>
  The initialisation output, which contains the unseal key and the root token, is written to a Kubernetes Secret named `vault-init` in the `vault` namespace. Nothing is printed and nothing is written to disk.

  Anyone who can read that Secret can unseal Vault and mint tokens. Restrict access to it, back it up somewhere outside the cluster, and treat losing it as losing the Vault. Once you have another way in, such as userpass or OIDC auth, revoke the root token.
</Warning>

On an on-premises install, a pod restart or a node drain brings Vault back **sealed**. A sealed pod is `Running` but not `Ready`. The bootstrap runs at install time only, so nothing unseals it automatically. Re-run `sh install_scripts/vault-bootstrap.sh`, which is idempotent, or unseal manually. On EKS, KMS auto-unseal handles this.

### How ESO Reads It

On EKS the installer creates a `ClusterSecretStore` pointing at `http://vault.vault.svc.cluster.local:8200`, authenticating with the Kubernetes auth method as the `external-secrets` service account against a Vault role named `eso`. That role's policy grants read-only access to the KV mount.

| Setting                 | Variable          | Default         |
| ----------------------- | ----------------- | --------------- |
| KV mount                | `VAULT_KV_MOUNT`  | `lilt`          |
| Vault role for ESO      | `VAULT_ESO_ROLE`  | `eso`           |
| Auth method             | `VAULT_AUTH`      | `kubernetes`    |
| Path prefix for secrets | `BASE_VAULT_PATH` | `lilt/bare/aws` |

Set `BASE_VAULT_PATH` to a prefix that suits your environment, for example `lilt/prod/aws`.

## What Vault Must Contain

Paths are `BASE_VAULT_PATH` plus the suffix below. These are KV v2 mounts: the `data/` infix appears in the API path but not in the `vault kv` CLI, so `vault kv get lilt/prod/aws/mysql` addresses what ESO calls `lilt/data/prod/aws/mysql`.

| Path suffix         | Keys                                                                       | Where the value comes from                                                                          |
| ------------------- | -------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| `mysql`             | `username`, `mysql-password`, `host`                                       | Must match the `lilt` user the installer creates and your database endpoint. Derived, not invented. |
| `mongodb`           | `mongodb-password`, `mongodb-root-password`                                | Generate                                                                                            |
| `rabbitmq`          | `password`, `cookie`                                                       | Generate. `cookie` is the Erlang cluster cookie.                                                    |
| `clickhouse`        | `password`                                                                 | Generate                                                                                            |
| `cache-redis`       | `password`                                                                 | Generate                                                                                            |
| `common/auth`       | `auth_jwt_secret`, `passport_secret`, `SINGLE_OIDC_PROVIDER_CLIENT_SECRET` | Generate the first two. The third comes from your identity provider.                                |
| `common/api-keys`   | `apryse_key`, `apryse_docx_key`, `llm_api_key`, `oidc_client_secret`       | All external: your Apryse licence, your LLM provider, your identity provider.                       |
| `common/connectors` | `CONNECTORS_ADMIN_PASSWORD`, `manager_ui_secret_key`, `webhook_auth_token` | Generate                                                                                            |
| `common/smtp`       | `smtp_password`                                                            | Your SMTP provider                                                                                  |

`datadog` with key `api_key` is seeded too, but only matters if you enable Datadog.

External Secrets Operator reads secrets. It never creates them. Every path must exist before you install.

### Overriding a Single Path

Each secret has its own `<NAME>_SECRET_PATH` variable that overrides the derived path. The variable is named after the secret rather than the path suffix, which is easy to get wrong for the `common/` entries.

| Variable                   | Overrides                          |
| -------------------------- | ---------------------------------- |
| `MYSQL_SECRET_PATH`        | `mysql`                            |
| `MONGODB_SECRET_PATH`      | `mongodb`                          |
| `RABBITMQ_SECRET_PATH`     | `rabbitmq`                         |
| `CLICKHOUSE_SECRET_PATH`   | `clickhouse`                       |
| `CACHE_REDIS_SECRET_PATH`  | `cache-redis`                      |
| `DATADOG_SECRET_PATH`      | `datadog`                          |
| `AUTH_SECRET_PATH`         | `common/auth`                      |
| `API_KEYS_SECRET_PATH`     | `common/api-keys`                  |
| `CONNECTORS_SECRET_PATH`   | `common/connectors`                |
| `SMTP_SECRET_PATH`         | `common/smtp`                      |
| `RDS_CA_SECRET_PATH`       | The RDS CA bundle                  |
| `LILT_SECRETS_SECRET_PATH` | The composed `lilt-secrets` secret |

## Seed the Paths

`install_scripts/seed-vault-secrets.sh` creates every path with a placeholder, so a first install does not fail one missing key at a time. It derives the path and key list from the same ExternalSecret manifests the operator reads, so it cannot drift from what LILT actually asks for.

Authenticate to the target Vault first, with `VAULT_ADDR` set, then run it from the tarball root:

```bash theme={null}
BASE_VAULT_PATH=lilt/prod/aws sh install_scripts/seed-vault-secrets.sh           # report only
BASE_VAULT_PATH=lilt/prod/aws sh install_scripts/seed-vault-secrets.sh --apply   # write
```

The script is non-destructive. It writes only keys that are missing and leaves existing values alone, so re-running it never overwrites a real secret with a placeholder.

Two behaviours worth knowing:

* Values default to the literal `REPLACE_ME`. Override with `PLACEHOLDER`, or set `SEED_RANDOM=true` to generate a random value per key, for an install that has to come up unattended.
* The `mysql` path is the exception. Its `username`, `mysql-password`, and `host` are taken from the database user the installer actually creates, because ESO feeds them to every application and they have to match.

After the install, replace every remaining placeholder with a real value.

## Use Your Own Vault

To point LILT at a Vault you already run, set `SECRET_BACKEND=vault` and `VAULT_AUTH=approle`, with `VAULT_ADDR` pointing at your server.

AppRole authentication needs a Kubernetes Secret named `vault-approle` in the `external-secrets` namespace, holding the keys `role-id` and `secret-id`. Create it before you install:

```bash theme={null}
kubectl create secret generic vault-approle \
  --namespace external-secrets \
  --from-literal=role-id=<your-role-id> \
  --from-literal=secret-id=<your-secret-id>
```

The installer never creates this Secret and stops if it is missing. There are no `VAULT_ROLE_ID` or `VAULT_SECRET_ID` variables: keeping the credentials out of `install.env` keeps them out of a file that could be committed.

<Warning>
  Your own Vault and the shipped in-cluster Vault are mutually exclusive. Two secret stores of the same name must never both be active on one cluster. Set `ENABLE_VAULT=false` when you use your own.
</Warning>

## Use AWS Secrets Manager

Set `SECRET_BACKEND=aws`. External Secrets authenticates with workload identity as the `external-secrets` service account in the `external-secrets` namespace. The same path and key table applies, under the prefix you set.

If you provision with the LILT Terraform module, grant that access with `enable_eso_aws_sm = true` and scope it with `eso_aws_sm_secret_prefix`. That role binds the service account External Secrets actually runs as, and its trust policy follows `iam_auth_method`.

<Warning>
  Do not use `enable_external_secrets = true` for this. It creates an older, overlapping role whose Pod Identity association binds a service account named `external-secrets-sa`, which is not the name the External Secrets chart creates, and whose trust policy is hard-coded to Pod Identity regardless of `iam_auth_method`. Nothing errors: the role and the association are created and simply never attach to a pod, and External Secrets then fails with an access-denied error that does not mention the mismatch.
</Warning>

## Turn the Shipped Vault Off

`ENABLE_VAULT=false` skips the Vault install entirely. With `SECRET_BACKEND` left unset it falls back to seed mode, which is the safe default.

<Warning>
  Nothing guards the combination `ENABLE_VAULT=false` with `SECRET_BACKEND=vault` and the default `VAULT_AUTH=kubernetes`. The installer then points External Secrets at an in-cluster Vault that was never installed, and the install fails at the point where it waits for the first ExternalSecret to become ready. If you disable the shipped Vault, set `SECRET_BACKEND` to `seed` or `aws`, or use `VAULT_AUTH=approle` with your own server.
</Warning>

## The Vault UI

`ENABLE_VAULT_UI` defaults to `false`. Setting it to `true` publishes the Vault UI on its own hostname, `vault.<SUBDOMAIN>.<DNS_DOMAIN>` by default, overridable with `VAULT_UI_HOST`.

It needs a hostname of its own because the Vault UI serves only at the root with absolute asset paths, so it cannot live under a path prefix.

Enabling it requires a DNS record for that hostname and a TLS certificate that covers it. See [Set custom domain and certificates](/kb/set-custom-domain-and-certificates). Without both, use a port-forward instead:

```bash theme={null}
kubectl port-forward -n vault svc/vault-ui 8200:8200
```

## Related Articles

* [Install System (AWS EKS)](/kb/install-system-aws-eks)
* [Set custom domain and certificates](/kb/set-custom-domain-and-certificates)
* [GPU profiles and values overlays](/kb/gpu-profiles-and-values-overlays)