> ## Documentation Index > Fetch the complete documentation index at: https://support.lilt.com/llms.txt > Use this file to discover all available pages before exploring further. # GPU Profiles And Values Overlays LILT ships alternative GPU allocations and identity-provider configurations as files you copy into an overlay directory. Nothing is edited in place, so your choices survive an upgrade. ## The Overlay Directory `install-lilt-charts.sh` applies every `*.yaml` file under `lilt/environments//values.d/` on top of the base values file and the environment overlay. `ENV_NAME` defaults to `lilt`, so the directory is usually `lilt/environments/lilt/values.d/`. ```bash theme={null} mkdir -p lilt/environments/lilt/values.d cp lilt/gpu-profiles/gemma-single-large.yaml lilt/environments/lilt/values.d/ ``` The directory is yours. The installer reads it and never writes to it or prunes it, so one copy holds until you undo it. Delete the file to return to the default. This is the supported way to change values the installer ships. Editing `lilt/values.yaml` or `lilt/eks-values.yaml` directly produces the same result at deploy time, but you then have to remember to reapply the edit on every upgrade. Files are applied in alphabetical order, and a later layer replaces an earlier one. Two files that set the same top-level key conflict silently: the alphabetically last one wins and the other is discarded with no warning. Keep at most one profile per service. Helm replaces list values wholesale rather than merging them, so a profile's `nodeSelectorTerms` supersede the default entirely rather than adding to it. Files that set different top-level keys compose safely. A GPU profile and an SSO profile can sit in the directory together. ## GPU Profiles The base values file ships one GPU allocation per service, sized for the smallest hardware that service supports. On larger cards the same model needs fewer of them, and the surplus is not free: the device plugin assigns GPUs exclusively, so a card reserved by a pod that never uses it cannot be used by anything else. The profiles in `lilt/gpu-profiles/` are alternative allocations for hardware that allows a different one. They are optional. Installing nothing keeps the defaults, which work on every card the service supports. ### Read Your Cards First A profile is matched against the VRAM the driver reports, which is lower than the size the card is sold as. An L4 sold as 24 GB reports 23034 MiB. ```bash theme={null} kubectl get nodes -L nvidia.com/gpu.memory,nvidia.com/gpu.product ``` These labels come from GPU Feature Discovery, which the NVIDIA GPU Operator provides. They are a hard requirement of any GPU install, not something a profile introduces. If the labels are absent, GPU pods stay `Pending` because no node satisfies their affinity. ### Choosing a Profile Pick the smallest card count your hardware allows. | Service | Configuration | Cards | Memory floor | Compute capability | Sized for | | ---------------------- | ------------------------- | ----- | ------------------ | ------------------ | ------------------------------------ | | `gemma-vllm-inference` | on-premises default | 2 | 22000 MiB | 8.9 | L4 (23034 MiB) | | `gemma-vllm-inference` | EKS default | 1 | 43400 MiB | 8.9 | L40S (46068 MiB) | | `gemma-vllm-inference` | `gemma-dual-medium.yaml` | 2 | 22000 to 43399 MiB | 8.9 | L4 | | `gemma-vllm-inference` | `gemma-single-large.yaml` | 1 | 43400 MiB | 8.9 | L40, L40S, H100, H200 | | `llama-vllm-inference` | default | 4 | 15000 MiB | 7.5 | T4 (15360 MiB) and larger | | `llama-vllm-inference` | `llama-dual-medium.yaml` | 2 | 30000 to 57999 MiB | 7.5 | RTX 5000 Ada, A100 40GB, L40S, A6000 | | `llama-vllm-inference` | `llama-single-large.yaml` | 1 | 58000 MiB | 7.5 | A100 80GB, H100, H200 | The two defaults differ because the hardware does. An on-premises cluster typically has several smaller cards, so the default takes two of them. An AWS GPU node carries one large card, so the EKS default takes one and raises the floor high enough that the card can hold the whole model. Every gemma configuration requires compute capability 8.9 or higher, on both flavors. Gemma is FP8-quantised and native FP8 support starts with the Ada generation, so every Ampere card is excluded, including the 80 GB A100. This is a separate condition from the memory floor: a card with plenty of VRAM and an older architecture still does not qualify. The memory floors are exclusive comparisons in the values files. A floor written as `43399` admits cards reporting 43400 MiB or more. The numbers in the preceding table are the values that actually qualify. ### When to Use Each Profile **`gemma-dual-medium.yaml`** — your cards are the 24 GB class and gemma stays `Pending` because no single card clears the EKS default's floor. Two cards together hold what one cannot. The profile is bounded above, so it will not also match a larger card and reserve two where one would do. **`gemma-single-large.yaml`** — your cards are 48 GB or larger and the on-premises default is reserving two where one suffices. **`llama-dual-medium.yaml`** — your nodes have fewer than four cards, so the default never schedules. Here the profile is what makes llama runnable, not what makes it faster. **`llama-single-large.yaml`** — your cards are 80 GB or larger. `llama-vllm-inference` ships disabled on both flavors, as it is being retired in favour of Gemma. A llama profile only takes effect on a cluster that has turned it back on. ### Why a Profile Is a File Each profile changes two fields that are only correct together: the GPU count Kubernetes reserves, and the memory floor in the node affinity that decides which cards are eligible. Changing one without the other fails, and the two failures look nothing alike. A count lowered without raising the floor still schedules, then the pod lands on cards too small for the model and dies during startup with a VRAM error. A floor raised without lowering the count never schedules at all, and the pod sits `Pending` forever. Kubernetes cannot make this choice for you. The GPU count is a resource request, fixed when Helm renders the manifest, and the scheduler needs that number to pick a node, so it cannot depend on which node was picked. Keeping both fields in one file is what makes them impossible to apply by halves. ### Services Without a Profile `llm-inference-whisper`, `batch-worker-gpuv4`, and `translatev4` carry GPU floors of their own but ship no profile. To change their allocation you edit the base values file, or write your own overlay file setting the same two fields. ## Single Sign-On Profiles LILT ships with external SSO disabled, so password sign-in works out of the box and no install points at an identity provider you do not control. The files in `lilt/sso-profiles/` are complete configurations for one provider each. Apply one the same way: ```bash theme={null} cp lilt/sso-profiles/.yaml lilt/environments/lilt/values.d/ ``` Then set the client secret, which no profile carries, as the `singleOidcClientSecret` key of the `lilt-secrets` secret. Add it to the environment's `secrets.yaml`, or provision it through External Secrets Operator. The `panva-sandbox.yaml` profile targets a public sandbox provider with a shared test client. Anyone can authenticate as anyone. Use it to check that the SSO path works end to end, never on a customer install. ### Writing a Profile for Your Own Provider A profile sets all eight values on three services: `front`, `av-scan`, and `auth-service`. Each reads its own copy of the same block, so a profile covering only one of them leaves the other two disabled. | Value | Set it to | | --------------------------------------------------- | -------------------------------------- | | `SINGLE_OIDC_PROVIDER_ENABLED` | `true` | | `SINGLE_OIDC_PROVIDER_CLIENT_ID` | The client ID your provider issued | | `SINGLE_OIDC_PROVIDER_ID_TOKEN_SIGNED_RESPONSE_ALG` | The signing algorithm, usually `RS256` | | `SINGLE_OIDC_PROVIDER_ISSUER_NAME` | The issuer name or URL | | `SINGLE_OIDC_PROVIDER_ISSUER_AUTH_ENDPOINT` | The authorization endpoint | | `SINGLE_OIDC_PROVIDER_ISSUER_TOKEN_ENDPOINT` | The token endpoint | | `SINGLE_OIDC_PROVIDER_ISSUER_USERINFO_ENDPOINT` | The userinfo endpoint | | `SINGLE_OIDC_PROVIDER_ISSUER_JWKS_URI` | The JWKS URI | Set all of them together. `front` validates the block when the provider is enabled and exits on the first empty field, so a partial profile crashes the pod at startup rather than falling back to no SSO. To customise TLS for the calls LILT makes to your provider, populate the `single-oidc-provider-tls-options` secret and set the matching `SINGLE_OIDC_PROVIDER_TLS_OPTIONS_PATH_TO_*` values. ## Apply a Profile Copying a file changes nothing on its own. Re-run the installer: ```bash theme={null} sh install-lilt.sh # on-premises sh install-lilt-eks.sh # EKS ``` Confirm the layer was picked up. The installer prints a line for each file it applies: ``` Applying extra values layer: lilt/environments/lilt/values.d/gemma-single-large.yaml ``` Then check that the pods scheduled: ```bash theme={null} kubectl get pods -n lilt -l app=gemma-vllm-inference kubectl describe pod -n lilt | grep -A5 'Events' ``` A pod stuck in `Pending` with `didn't match Pod's node affinity` means no card in the cluster clears the profile's floor. Compare the profile you chose against the `nvidia.com/gpu.memory` labels on your nodes. ## Related Articles * [Install System (AWS EKS)](/kb/install-system-aws-eks) * [Self-managed hardware requirements](/kb/self-managed-hardware-requirements) * [Single sign-on (SSO)](/kb/single-sign-on-sso)