Using S3 instead of MinIO

Overview

By default, LILT comes with MinIO enabled. However, if an environment has access to S3 directly, this is the recommended approach. This document covers configuring LILT to use S3 instead of MinIO—it does not cover how to transition an existing MinIO deployment to S3. Please work with your program manager to facilitate this process. There are two options for authentication with s3 depending on the type of cluster, managed or unmanaged:

Unmanaged: Identity and Access Management (IAM) that utilizes a AWS access and secret key
Managed (EKS): IAM Roles for Service Accounts (IRSA) that attaches a service account to a pod

Uninstall MinIO

There is currently no mechanism in LILT to automatically remove MinIO from a deployment. This is a feature we are working to implement by the end of 2025. In the mean time, MinIO may be removed directly by:

helm uninstall minio

Configure LILT

Update Secrets

LILT ships with a default AWS credential. This needs to be removed. Edit lilt/environments/lilt/secrets.yaml and remove the following line:

front:
  onpremValues:
    config:
      front_secret_values:
        s3secretkey: testsecretkey      # Only remove this line

Option 1: Unmanaged IAM

Update Values

First, collect the following information for your bucket:

AWS_ACCESS_KEY: <AWS_ACCESS_KEY>
AWS_SECRET_KEY: <AWS_SECRET_KEY>
AWS_BUCKET: <AWS_BUCKET>
AWS_REGION: <AWS_REGION>
AWS_S3_ENDPOINT_URL: <AWS_S3_ENDPOINT_URL>
AWS_S3_ENDPOINT_URL_SSL: <AWS_S3_ENDPOINT_URL_SSL>

Example endpoint URLs in the AWS Commercial us-east-1 region would look like:

AWS_S3_ENDPOINT_URL: http://s3.us-east-1.amazonaws.com
AWS_S3_ENDPOINT_URL_SSL: https://s3.us-east-1.amazonaws.com

Edit your override values at lilt/environments/lilt/values.yaml. Add all of the following values, replacing values within <> with the values documented from above:

AWS_ACCESS_KEY: <AWS_ACCESS_KEY>
AWS_SECRET_KEY: <AWS_SECRET_KEY>
AWS_BUCKET: <AWS_BUCKET>
AWS_REGION: <AWS_REGION>
AWS_S3_ENDPOINT_URL: <AWS_S3_ENDPOINT_URL>
AWS_S3_ENDPOINT_URL_SSL: <AWS_S3_ENDPOINT_URL_SSL>

front:
  onpremValues:
    env:
      UPLOADER_EVENT_SOURCE_TYPE: s3
      MM_BUCKET: <AWS_BUCKET>
    config:
      front_secret_values:
        awsSecretAccessKey: <AWS_SECRET_KEY>
        s3secretkey: <AWS_SECRET_KEY>
      front_configmap_values:
        awsAccessKeyId: <AWS_ACCESS_KEY>
        bucket: <AWS_BUCKET>
        s3-region: <AWS_REGION>
        awsRegion: <AWS_REGION>
        s3accesskey: <AWS_ACCESS_KEY>
        awsS3Bucket: <AWS_BUCKET>
        awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
        awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>
        awsS3EndpointUrlPublic: <AWS_S3_ENDPOINT_URL_SSL>

lexicon:
  onpremValues:
    app:
      bucketname: <AWS_BUCKET>
    config:
      awsSecretAccessKey: <AWS_SECRET_KEY>
      awsAccessKeyId: <AWS_ACCESS_KEY>
      bucket: <AWS_BUCKET>
      s3-region: <AWS_REGION>
    init:
      filePaths: "s3://<AWS_BUCKET>/lexicon-data/panlex/panlex-20220201-json_compiled.scored-rev-filter.json.gz,s3://<AWS_BUCKET>/lexicon-data/ranking/linguee_en_de_es_fr_pt_nl_it_zh_esfr_itfr_da_sv_no_pl_ru_fi_20170513_fixes.csv,s3://<AWS_BUCKET>/lexicon-data/morphology/morph_da.json,s3://<AWS_BUCKET>/lexicon-data/morphology/morph_de.json,s3://<AWS_BUCKET>/lexicon-data/morphology/morph_en.json,s3://<AWS_BUCKET>/lexicon-data/morphology/morph_es.json,s3://<AWS_BUCKET>/lexicon-data/morphology/morph_fr.json,s3://<AWS_BUCKET>/lexicon-data/morphology/morph_it.json,s3://<AWS_BUCKET>/lexicon-data/morphology/morph_nl.json,s3://<AWS_BUCKET>/lexicon-data/morphology/morph_no.json,s3://<AWS_BUCKET>/lexicon-data/morphology/morph_pl.json,s3://<AWS_BUCKET>/lexicon-data/morphology/morph_pt.json,s3://<AWS_BUCKET>/lexicon-data/morphology/morph_sv.json"
    extraInitContainer:
      outputPath: "s3://<AWS_BUCKET>/lexicon-data/"

converter:
  onpremValues:
    config:
      awsSecretAccessKey: <AWS_SECRET_KEY>
      s3-privatekey: <AWS_SECRET_KEY>
      s3-accesskey: <AWS_ACCESS_KEY>
      awsAccessKeyId: <AWS_ACCESS_KEY>
      bucket: <AWS_BUCKET>
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>  
    app:
      bucketname: <AWS_BUCKET>
      args:
        s3-endpoint: <AWS_S3_ENDPOINT_URL_SSL>
        s3-region: <AWS_REGION>
        bucket: <AWS_BUCKET>

assignment:
  onpremValues:
    app:
      bucketname: <AWS_BUCKET>

auditlog:
  onpremValues:
    app:
      bucketname: <AWS_BUCKET>

batch-tb:
  onpremValues:
    app:
      bucketname: <AWS_BUCKET>

tb:
  onpremValues:
    app:
      bucketname: <AWS_BUCKET>

tm:
  onpremValues: 
    app:
      bucketname: <AWS_BUCKET>
      args:
        s3-endpoint: <AWS_S3_ENDPOINT_URL_SSL>
        s3-region: <AWS_REGION>
        bucket: <AWS_BUCKET>

indexer:
  onpremValues: 
    app:
      bucketname: <AWS_BUCKET>

linguist:
  onpremValues: 
    app:
      bucketname: <AWS_BUCKET>

memory:
  onpremValues: 
    app:
      bucketname: <AWS_BUCKET>

qa:
  onpremValues: 
    app:
      bucketname: <AWS_BUCKET>

search:
  onpremValues: 
    app:
      bucketname: <AWS_BUCKET>

segment:
  onpremValues: 
    app:
      bucketname: <AWS_BUCKET>

tag:
  onpremValues: 
    app:
      bucketname: <AWS_BUCKET>

watchdog:
  onpremValues: 
    app:
      bucketname: <AWS_BUCKET>

job:
  onpremValues: 
    app:
      bucketname: <AWS_BUCKET>
      args:
        s3-endpoint: <AWS_S3_ENDPOINT_URL_SSL>
        s3-region: <AWS_REGION>
        bucket: <AWS_BUCKET>

workflow:
  onpremValues: 
    app:
      bucketname: <AWS_BUCKET>
      args:
        s3-endpoint: <AWS_S3_ENDPOINT_URL_SSL>
        s3-region: <AWS_REGION>
        bucket: <AWS_BUCKET>

file-translation:
  onpremValues:
    config:
      awsSecretAccessKey: <AWS_SECRET_KEY>
      s3-privatekey: <AWS_SECRET_KEY>
      s3-accesskey: <AWS_ACCESS_KEY>
      awsAccessKeyId: <AWS_ACCESS_KEY>
      bucket: <AWS_BUCKET>
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>    
    app:
      bucketname: <AWS_BUCKET>
      args:
        s3-endpoint: <AWS_S3_ENDPOINT_URL_SSL>
        s3-region: <AWS_REGION>
        bucket: <AWS_BUCKET>
        s3-accesskey: <AWS_ACCESS_KEY>
        s3-privatekey: <AWS_SECRET_KEY>

file-job:
  onpremValues:
    config:
      awsSecretAccessKey: <AWS_SECRET_KEY>
      s3-privatekey: <AWS_SECRET_KEY>
      s3-accesskey: <AWS_ACCESS_KEY>
      awsAccessKeyId: <AWS_ACCESS_KEY>
      bucket: <AWS_BUCKET>
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
    init:
      filePaths: "s3://<AWS_BUCKET>/tesseract/eng.traineddata,s3://<AWS_BUCKET>/tesseract/deu.traineddata,s3://<AWS_BUCKET>/tesseract/fra.traineddata,s3://<AWS_BUCKET>/tesseract/spa.traineddata,s3://<AWS_BUCKET>/tesseract/ara.traineddata,s3://<AWS_BUCKET>/tesseract/nor.traineddata,s3://<AWS_BUCKET>/tesseract/ind.traineddata,s3://<AWS_BUCKET>/tesseract/chi_tra.traineddata,s3://<AWS_BUCKET>/tesseract/chi_sim.traineddata,s3://<AWS_BUCKET>/tesseract/por.traineddata,s3://<AWS_BUCKET>/tesseract/nld.traineddata,s3://<AWS_BUCKET>/tesseract/ita.traineddata,s3://<AWS_BUCKET>/tesseract/dan.traineddata,s3://<AWS_BUCKET>/tesseract/swe.traineddata,s3://<AWS_BUCKET>/tesseract/pol.traineddata,s3://<AWS_BUCKET>/tesseract/rus.traineddata,s3://<AWS_BUCKET>/tesseract/fin.traineddata,s3://<AWS_BUCKET>/tesseract/jpn.traineddata,s3://<AWS_BUCKET>/tesseract/kor.traineddata,s3://<AWS_BUCKET>/tesseract/ukr.traineddata,s3://<AWS_BUCKET>/tesseract/ron.traineddata,s3://<AWS_BUCKET>/tesseract/heb.traineddata,s3://<AWS_BUCKET>/tesseract/tha.traineddata,s3://<AWS_BUCKET>/tesseract/hin.traineddata,s3://<AWS_BUCKET>/tesseract/tur.traineddata,s3://<AWS_BUCKET>/tesseract/vie.traineddata,s3://<AWS_BUCKET>/tesseract/ell.traineddata,s3://<AWS_BUCKET>/tesseract/afr.traineddata,s3://<AWS_BUCKET>/tesseract/ces.traineddata,s3://<AWS_BUCKET>/tesseract/slk.traineddata,s3://<AWS_BUCKET>/tesseract/hun.traineddata,s3://<AWS_BUCKET>/tesseract/fas.traineddata,s3://<AWS_BUCKET>/tesseract/jav.traineddata,s3://<AWS_BUCKET>/tesseract/pus.traineddata,s3://<AWS_BUCKET>/tesseract/urd.traineddata,s3://<AWS_BUCKET>/tesseract/fas.traineddata,s3://<AWS_BUCKET>/tesseract/ben.traineddata,s3://<AWS_BUCKET>/tesseract/hrv.traineddata,s3://<AWS_BUCKET>/tesseract/swa.traineddata,s3://<AWS_BUCKET>/tesseract/slv.traineddata,s3://<AWS_BUCKET>/tesseract/bul.traineddata,s3://<AWS_BUCKET>/tesseract/fil.traineddata,s3://<AWS_BUCKET>/tesseract/kan.traineddata,s3://<AWS_BUCKET>/tesseract/guj.traineddata,s3://<AWS_BUCKET>/tesseract/pan.traineddata,s3://<AWS_BUCKET>/tesseract/mal.traineddata,s3://<AWS_BUCKET>/tesseract/mar.traineddata,s3://<AWS_BUCKET>/tesseract/tam.traineddata,s3://<AWS_BUCKET>/tesseract/tel.traineddata,s3://<AWS_BUCKET>/tesseract/sqi.traineddata"
    extraInitContainer:
      outputPath: "s3://<AWS_BUCKET>/tesseract/"
      BUCKETNAME: <AWS_BUCKET>
    app:
      bucketname: <AWS_BUCKET>
      args:
        s3-endpoint: <AWS_S3_ENDPOINT_URL_SSL>
        s3-region: <AWS_REGION>
        bucket: <AWS_BUCKET>

dataflow:
  onpremValues:
    env:
      MINIO_ENDPOINT: s3://<AWS_BUCKET>
      MINIO_ACCESS_KEY: <AWS_ACCESS_KEY>
      MINIO_SECRET_KEY: <AWS_SECRET_KEY>
      MINIO_REGION: <AWS_REGION>

translatev4:
  onpremValues:
    config:
      awsSecretAccessKey: <AWS_SECRET_KEY>
      awsAccessKeyId: <AWS_ACCESS_KEY>
      bucket: <AWS_BUCKET>
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>
    init:
      bucket: <AWS_BUCKET>
      outputPath: "s3://<AWS_BUCKET>/trained/"

updatev4:
  onpremValues:
    config:
      awsSecretAccessKey: <AWS_SECRET_KEY>
      awsAccessKeyId: <AWS_ACCESS_KEY>
      bucket: <AWS_BUCKET>
      awsRegion: <AWS_REGION>
      wpaDataStoreBasePath: "s3://<AWS_BUCKET>/wpa/"
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>

langid:
  onpremValues:
    config:
      awsSecretAccessKey: <AWS_SECRET_KEY>
      awsAccessKeyId: <AWS_ACCESS_KEY>
      bucket: <AWS_BUCKET>
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>

update-managerv4:
  onpremValues:
    config:
      awsSecretAccessKey: <AWS_SECRET_KEY>
      awsAccessKeyId: <AWS_ACCESS_KEY>
      bucket: <AWS_BUCKET>
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>

routing:
  onpremValues:
    config:
      awsSecretAccessKey: <AWS_SECRET_KEY>
      awsAccessKeyId: <AWS_ACCESS_KEY>
      bucket: <AWS_BUCKET>
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>

batchv4:
  onpremValues:
    config:
      awsSecretAccessKey: <AWS_SECRET_KEY>
      awsAccessKeyId: <AWS_ACCESS_KEY>
      bucket: <AWS_BUCKET>
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>

batch-worker-gpuv4:
  onpremValues:
    config:
      awsSecretAccessKey: <AWS_SECRET_KEY>
      awsAccessKeyId: <AWS_ACCESS_KEY>
      bucket: <AWS_BUCKET>
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>

llm-inference:
  onpremValues:
    config:
      awsSecretAccessKey: <AWS_SECRET_KEY>
      awsAccessKeyId: <AWS_ACCESS_KEY>
      bucket: <AWS_BUCKET>
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>
    init:
      bucket: <AWS_BUCKET>
      outputPath: "s3://<AWS_BUCKET>/trained/"

automqm:
  onpremValues:
    config:
      awsSecretAccessKey: <AWS_SECRET_KEY>
      awsAccessKeyId: <AWS_ACCESS_KEY>
      bucket: <AWS_BUCKET>
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>

alignment:
  onpremValues:
    config:
      awsSecretAccessKey: <AWS_SECRET_KEY>
      awsAccessKeyId: <AWS_ACCESS_KEY>
      bucket: <AWS_BUCKET>
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>
    init:
      bucket: <AWS_BUCKET>
      outputPath: "s3://<AWS_BUCKET>/trained/"

tag-projection:
  onpremValues:
    config:
      awsSecretAccessKey: <AWS_SECRET_KEY>
      awsAccessKeyId: <AWS_ACCESS_KEY>
      bucket: <AWS_BUCKET>
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>
    init:
      bucket: <AWS_BUCKET>
      outputPath: "s3://<AWS_BUCKET>/trained/"

nncache:
  onpremValues:
    config:
      awsSecretAccessKey: <AWS_SECRET_KEY>
      awsAccessKeyId: <AWS_ACCESS_KEY>
      bucket: <AWS_BUCKET>
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>

Option 2: Managed (EKS) IRSA

Create AWS IAM Policy and Role

In the AWS console or via terraform, create IAM role and policy for accessing the s3 bucket and then attach them to each other. The following assumes that you are using OIDC provider.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::<my-bucket-name>",
        "arn:aws:s3:::<my-bucket-name>/*"
      ]
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<ACCOUNT_ID>:oidc-provider/<OIDC_PROVIDER>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "<OIDC_PROVIDER>:sub": "system:serviceaccount:<SERVICE_ACCOUNT_NAMESPACE>:<SERVICE_ACCOUNT_NAME>"
        }
      }
    }
  ]
}

Once the role and policy are attached, copy the arn. Example:

arn:aws:iam::<ACCOUNT_ID>:role/<role_name>

Create Service Account

Either via terraform or kubectl, create a service account for use by the pods. Example using kubectl:

kubectl create serviceaccount <service-account-name> -n <your-namespace>
kubectl annotate serviceaccount <service-account-name> \
  -n <your-namespace> \
  eks.amazonaws.com/role-arn=<your-arm>

Update RBAC

Pod can only reference ONE service account. This overrides any default cluster permissions so any additional requirements must be included within RBAC roles. For example, neural requires access to pods and deployments. Create role yaml and apply to cluster:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: <role-name>
  namespace: <your-namespace>
rules:
  - apiGroups: [""]
    resources: ["deployments", "deployments/scale", "jobs", "pods", "pods/status", "pods/log", "configmaps", "secrets"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: ["apps"]
    resources: ["deployments/scale"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

Bind role to the previously created service account:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: <bind-role-name>
  namespace: <your-namespace>
subjects:
- kind: ServiceAccount
  name: <service-account-name>
  namespace: <your-namespace>
roleRef:
  kind: Role
  name: <role-name>
  apiGroup: rbac.authorization.k8s.io

Update Values

First, collect the following information for your bucket:

AWS_BUCKET: <AWS_BUCKET>
AWS_REGION: <AWS_REGION>
AWS_S3_ENDPOINT_URL: <AWS_S3_ENDPOINT_URL>
AWS_S3_ENDPOINT_URL_SSL: <AWS_S3_ENDPOINT_URL_SSL>
STORAGETYPE : s3

Example endpoint URLs in the AWS Commercial us-east-1 region would look like:

AWS_S3_ENDPOINT_URL: http://s3.us-east-1.amazonaws.com
AWS_S3_ENDPOINT_URL_SSL: https://s3.us-east-1.amazonaws.com

Service Account:

<SERVICE_ACCOUNT>: <service-account-name>

Edit your override values at lilt/environments/lilt/values.yaml. Add all of the following values, and replacing values within <> with the values documented from above:

front:
  enabled: true
    env:
      MM_BUCKET: <AWS_BUCKET>
      UPLOADER_EVENT_SOURCE_TYPE: <STORAGETYPE>
      UPLOADER_STORAGETYPE: <STORAGETYPE>
    config:
      front_configmap_values:
        awsAccessKeyId: ""
        awsRegion: <AWS_REGION>
        awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
        awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>
        awsS3EndpointUrlPublic: <AWS_S3_ENDPOINT_URL>
        awsS3Bucket: <AWS_BUCKET>
        s3accesskey: ""
      front_secret_values:
        awsSecretAccessKey: ""
        s3secretkey: ""
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>

converter:
  enabled: true
  onpremValues:
    config:
    app:
      bucketname: <AWS_BUCKET>
      args:
        s3SslVerificationMode: NONE
        storagetype: <STORAGETYPE>
        bucket: <AWS_BUCKET>
        s3-endpoint: <AWS_S3_ENDPOINT_URL>
        s3-region: <AWS_REGION>
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>

qa:
  enabled: true
  onpremValues:
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>

linguist:
  enabled: true
  onpremValues:
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>

search:
  enabled: true
  onpremValues:
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>

tm:
  enabled: true
  onpremValues:
    app:
      bucketname: <AWS_BUCKET>
      args:
        storagetype: <STORAGETYPE>
        bucket: <AWS_BUCKET>
        s3-endpoint: <AWS_S3_ENDPOINT_URL>
        s3-region: <AWS_REGION>
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>

tb:
  enabled: true
  onpremValues:
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>

batch-tb:
  enabled: true
  onpremValues:
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>

indexer:
  enabled: true
  onpremValues:
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>

lexicon:
  enabled: true
  onpremValues:
    config:
      awsSecretAccessKey: ""
      awsAccessKeyId: ""
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsRegion: <AWS_REGION>
    app:
      bucketname: <AWS_BUCKET>
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>
    init:
      filePaths: "s3://<AWS_BUCKET>/lexicon-data/panlex/panlex-20220201-json_compiled.scored-rev-filter.json.gz,s3://<AWS_BUCKET>/lexicon-data/ranking/linguee_en_de_es_fr_pt_nl_it_zh_esfr_itfr_da_sv_no_pl_ru_fi_20170513_fixes.csv,s3://<AWS_BUCKET>/lexicon-data/morphology/morph_da.json,s3://<AWS_BUCKET>/lexicon-data/morphology/morph_de.json,s3://<AWS_BUCKET>/lexicon-data/morphology/morph_en.json,s3://<AWS_BUCKET>/lexicon-data/morphology/morph_es.json,s3://<AWS_BUCKET>/lexicon-data/morphology/morph_fr.json,s3://<AWS_BUCKET>/lexicon-data/morphology/morph_it.json,s3://<AWS_BUCKET>/lexicon-data/morphology/morph_nl.json,s3://<AWS_BUCKET>/lexicon-data/morphology/morph_no.json,s3://<AWS_BUCKET>/lexicon-data/morphology/morph_pl.json,s3://<AWS_BUCKET>/lexicon-data/morphology/morph_pt.json,s3://<AWS_BUCKET>/lexicon-data/morphology/morph_sv.json"
    extraInitContainer:
      outputPath: "s3://<AWS_BUCKET>/lexicon-data/"

segment:
  enabled: true
  onpremValues:
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>

file-translation:
  enabled: true
  onpremValues:
    app:
      bucketname: <AWS_BUCKET>
      args:
        s3-accesskey: ""
        s3-privatekey: ""
        storagetype: <STORAGETYPE>
        bucket: <AWS_BUCKET>
        s3-endpoint: <AWS_S3_ENDPOINT_URL>
        s3-region: <AWS_REGION>
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>

job:
  enabled: true
  onpremValues:
    env:
      BUCKETNAME: <AWS_BUCKET>
    app:
      bucketname: <AWS_BUCKET>
      args:
        storagetype: <STORAGETYPE>
        bucket: <AWS_BUCKET>
        s3-endpoint: <AWS_S3_ENDPOINT_URL>
        s3-region: <AWS_REGION>
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>

tag:
  enabled: true
  onpremValues:
    env:
      BUCKETNAME: <AWS_BUCKET>
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>

auditlog:
  enabled: true
  onpremValues:
    env:
      BUCKETNAME: <AWS_BUCKET>

assignment:
  enabled: true
  onpremValues:
    env:
      BUCKETNAME: <AWS_BUCKET>
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>

workflow:
  enabled: true
  onpremValues:
    env:
      BUCKETNAME: <AWS_BUCKET>
    app:
      bucketname: <AWS_BUCKET>
      args:
        storagetype: <STORAGETYPE>
        bucket: <AWS_BUCKET>
        s3-endpoint: <AWS_S3_ENDPOINT_URL>
        s3-region: <AWS_REGION>
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>

dataflow:
  enabled: true
  onpremValues:
    env:
      MINIO_ENDPOINT: <AWS_S3_ENDPOINT_URL>
      MINIO_ACCESS_KEY: ""
      MINIO_SECRET_KEY: ""
      MINIO_REGION: <AWS_REGION>
      MINIO_SSL_CERT_FILE: ""
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>

file-job:
  enabled: true
  onpremValues:
    config:
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsRegion: <AWS_REGION>
      awsSecretAccessKey: ""
      awsAccessKeyId: ""
    env:
      BUCKETNAME: <AWS_BUCKET>
    app:
      bucketname: <AWS_BUCKET>
      args:
        storagetype: <STORAGETYPE>
        bucket: <AWS_BUCKET>
        s3-endpoint: <AWS_S3_ENDPOINT_URL>
        s3-region: <AWS_REGION>
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>
    init:
      filePaths: "s3://<AWS_BUCKET>/tesseract/eng.traineddata,s3://<AWS_BUCKET>/tesseract/deu.traineddata,s3://<AWS_BUCKET>/tesseract/fra.traineddata,s3://<AWS_BUCKET>/tesseract/spa.traineddata,s3://<AWS_BUCKET>/tesseract/ara.traineddata,s3://<AWS_BUCKET>/tesseract/nor.traineddata,s3://<AWS_BUCKET>/tesseract/ind.traineddata,s3://<AWS_BUCKET>/tesseract/chi_tra.traineddata,s3://<AWS_BUCKET>/tesseract/chi_sim.traineddata,s3://<AWS_BUCKET>/tesseract/por.traineddata,s3://<AWS_BUCKET>/tesseract/nld.traineddata,s3://<AWS_BUCKET>/tesseract/ita.traineddata,s3://<AWS_BUCKET>/tesseract/dan.traineddata,s3://<AWS_BUCKET>/tesseract/swe.traineddata,s3://<AWS_BUCKET>/tesseract/pol.traineddata,s3://<AWS_BUCKET>/tesseract/rus.traineddata,s3://<AWS_BUCKET>/tesseract/fin.traineddata,s3://<AWS_BUCKET>/tesseract/jpn.traineddata,s3://<AWS_BUCKET>/tesseract/kor.traineddata,s3://<AWS_BUCKET>/tesseract/ukr.traineddata,s3://<AWS_BUCKET>/tesseract/ron.traineddata,s3://<AWS_BUCKET>/tesseract/heb.traineddata,s3://<AWS_BUCKET>/tesseract/tha.traineddata,s3://<AWS_BUCKET>/tesseract/hin.traineddata,s3://<AWS_BUCKET>/tesseract/tur.traineddata,s3://<AWS_BUCKET>/tesseract/vie.traineddata,s3://<AWS_BUCKET>/tesseract/ell.traineddata,s3://<AWS_BUCKET>/tesseract/afr.traineddata,s3://<AWS_BUCKET>/tesseract/ces.traineddata,s3://<AWS_BUCKET>/tesseract/slk.traineddata,s3://<AWS_BUCKET>/tesseract/hun.traineddata,s3://<AWS_BUCKET>/tesseract/fas.traineddata,s3://<AWS_BUCKET>/tesseract/jav.traineddata,s3://<AWS_BUCKET>/tesseract/pus.traineddata,s3://<AWS_BUCKET>/tesseract/urd.traineddata,s3://<AWS_BUCKET>/tesseract/fas.traineddata,s3://<AWS_BUCKET>/tesseract/ben.traineddata,s3://<AWS_BUCKET>/tesseract/hrv.traineddata,s3://<AWS_BUCKET>/tesseract/swa.traineddata,s3://<AWS_BUCKET>/tesseract/slv.traineddata,s3://<AWS_BUCKET>/tesseract/bul.traineddata,s3://<AWS_BUCKET>/tesseract/fil.traineddata,s3://<AWS_BUCKET>/tesseract/kan.traineddata,s3://<AWS_BUCKET>/tesseract/guj.traineddata,s3://<AWS_BUCKET>/tesseract/pan.traineddata,s3://<AWS_BUCKET>/tesseract/mal.traineddata,s3://<AWS_BUCKET>/tesseract/mar.traineddata,s3://<AWS_BUCKET>/tesseract/tam.traineddata,s3://<AWS_BUCKET>/tesseract/tel.traineddata,s3://<AWS_BUCKET>/tesseract/sqi.traineddata"
    extraInitContainer:
      outputPath: "s3://<AWS_BUCKET>/tesseract/"

memory:
  enabled: true
  onpremValues:
    env:
      BUCKETNAME: <AWS_BUCKET>
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>

notification:
  enabled: true
  onpremValues:
    env:
      BUCKETNAME: <AWS_BUCKET>
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>

translatev4:
  enabled: true
  onpremValues:
    config:
      awsSecretAccessKey: ""
      awsAccessKeyId: ""
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>
      caBundlePath: ""
      storageType: <STORAGETYPE>
      btcJobBucket: <AWS_BUCKET>
      bucket: <AWS_BUCKET>
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>
    # Init container to copy neural trained v4 models
    init:
      enabled: true
      outputPath: "s3://<AWS_BUCKET>/trained/"
      
updatev4:
  enabled: true
  onpremValues:
    config:
      awsSecretAccessKey: ""
      awsAccessKeyId: ""
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>
      caBundlePath: ""
      storageType: <STORAGETYPE>
      btcJobBucket: <AWS_BUCKET>
      bucket: <AWS_BUCKET>
      wpaDataStoreBasePath: "s3://<AWS_BUCKET>/wpa/"
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>
      
langid:
  enabled: true
  onpremValues:
    config:
      awsSecretAccessKey: ""
      awsAccessKeyId: ""
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>
      caBundlePath: ""
      storageType: <STORAGETYPE>
      bucket: <AWS_BUCKET>
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>

update-managerv4:
  enabled: true
  onpremValues:
    config:
      awsSecretAccessKey: ""
      awsAccessKeyId: ""
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>
      caBundlePath: ""
      storageType: <STORAGETYPE>
      bucket: <AWS_BUCKET>
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>

routing:
  enabled: true
  # All messages from other teams' services go through the routing service.
  # Increase the number of replicas in environments with high load.
  onpremValues:
    config:
      awsSecretAccessKey: ""
      awsAccessKeyId: ""
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>
      caBundlePath: ""
      storageType: <STORAGETYPE>
      bucket: <AWS_BUCKET>
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>

batchv4:
  enabled: true
  onpremValues:
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>
    config:
      awsSecretAccessKey: ""
      awsAccessKeyId: ""
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>
      storageType: <STORAGETYPE>
      bucket: <AWS_BUCKET>

batch-worker-gpuv4:
  enabled: true
  onpremValues:
    config:
      awsSecretAccessKey: ""
      awsAccessKeyId: ""
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>
      caBundlePath: ""
      storageType: <STORAGETYPE>
      bucket: <AWS_BUCKET>
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>
      
llm-inference:
  enabled: true
  onpremValues:
    config:
      awsSecretAccessKey: ""
      awsAccessKeyId: ""
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>
      caBundlePath: ""
      storageType: <STORAGETYPE>
      bucket: <AWS_BUCKET>
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>
    init:
      outputPath: "s3://<AWS_BUCKET>/trained/"

automqm:
  enabled: true
  onpremValues:
    config:
      awsSecretAccessKey: ""
      awsAccessKeyId: ""
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>
      caBundlePath: ""
      storageType: <STORAGETYPE>
      bucket: <AWS_BUCKET>
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>

alignment:
  enabled: true
  onpremValues:
    config:
      awsSecretAccessKey: ""
      awsAccessKeyId: ""
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>
      caBundlePath: ""
      storageType: <STORAGETYPE>
      bucket: <AWS_BUCKET>
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>
    init:
      outputPath: "s3://<AWS_BUCKET>/trained/"

tag-projection:
  enabled: true
  onpremValues:
    config:
      awsSecretAccessKey: ""
      awsAccessKeyId: ""
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>
      caBundlePath: ""
      storageType: <STORAGETYPE>
      bucket: <AWS_BUCKET>
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>
    init:
      outputPath: "s3://<AWS_BUCKET>/trained/"

nncache:
  enabled: true
  onpremValues:
    config:
      awsSecretAccessKey: ""
      awsAccessKeyId: ""
      awsRegion: <AWS_REGION>
      awsS3EndpointUrl: <AWS_S3_ENDPOINT_URL>
      awsS3EndpointUrlSsl: <AWS_S3_ENDPOINT_URL_SSL>
      caBundlePath: ""
      storageType: <STORAGETYPE>
      bucket: <AWS_BUCKET>
    serviceAccount: 
      enabled: true
      name: <SERVICE_ACCOUNT>

​Overview

​Uninstall MinIO

​Configure LILT

​Update Secrets

​Option 1: Unmanaged IAM

​Update Values

​Option 2: Managed (EKS) IRSA

​Create AWS IAM Policy and Role

​Create Service Account

​Update RBAC

​Update Values

Overview

Uninstall MinIO

Configure LILT

Update Secrets

Option 1: Unmanaged IAM

Update Values

Option 2: Managed (EKS) IRSA

Create AWS IAM Policy and Role

Create Service Account

Update RBAC

Update Values