Supported Languages
V3 Languages
Starting in V3, our models will be trained in groups as opposed to single language pairs.Language Grouping | Explanation | Codes |
---|---|---|
Asian: English, Japanese, Korean, Chinese (Simplified), Chinese (Traditional) | Any two languages in this grouping | EN,JA,KO,ZH,ZT |
Eastern European: English, Belarusian, Bulgarian, Czech, Croatian, Lithuanian, Latvian, Polish, Romanian, Russian, Slovak, Slovenian, Albanian, Serbian, Ukrainian | Any two languages in this grouping | EN,BE,BG,CS,HR,LT,LV,PL,RO,RU,SK,SL,SQ,SR,UK |
Eastern Hemisphere: English, Arabic, Bengali, Central Kurdish, Persian (Farsi), Dari, Gujarati, Hebrew, Hindi, Kannada, Malayalam, Marathi, Nepali, Panjabi, Pushto, Tamil, Telugu, Tajik, Urdu | Any two languages in this grouping | EN,AR,BN,CKB,FA,FS,GU,HE,HI,KN,ML,MR,NE,PA,PS,TA,TE,TG,UR |
European Group 2: English, Afrikaans, Danish, German, Greek, Amharic, Finnish, Hungarian, Armenian, Dutch, Norwegian, Swedish, Turkish | Any two languages in this grouping | EN,AF,DA,DE,EL,ET,FI,HU,HY,NL,NO,SV,TR |
European: English, German, Spanish, French, Italian, Portuguese | Any two languages in this grouping | EN,DE,ES,FR,IT,PT |
**Other: **English, Indonesian, Igbo, Javanese, Malay, Burmese, Thai, Tagalog, Samoan, Somali, Swahili, Vietnamese | Any two languages in this grouping | EN,ID,IG,JV,MS,MY,TH,TL,SM,SO,SW,VI |
Unsupported Languages
Enlarges the table by opening it in a full screen dialogOpenSource Language | Target Language | Contextual AI Support | Baseline Model | On-Premise |
---|---|---|---|---|
Amharic | English | No | N/A | Not available |
Azeri | English | No | N/A | Not available |
Catalan | English | No | N/A | Not available |
English | Amharic | No | N/A | Not available |
English | Azeri | No | N/A | Not available |
English | Bosnian | No | N/A | Not available |
English | Catalan | No | N/A | Not available |
English | Creole | No | N/A | Not available |
English | English | No | N/A | Not available |
English | Icelandic | No | N/A | Not available |
English | Kazakh | No | N/A | Not available |
English | Khmer | No | N/A | Not available |
English | Kinyarwanda | No | N/A | Not available |
English | Lao | No | N/A | Not available |
English | Maori | No | N/A | Not available |
English | Mongolian | No | N/A | Not available |
English | Sinhala | No | N/A | Not available |
English | Zulu | No | N/A | Not available |
Khmer | English | No | N/A | Not available |
Kinyarwanda | English | No | N/A | Not available |
Lao | English | No | N/A | Not available |
Maori | English | No | N/A | Not available |
Mongolian | English | No | N/A | Not available |
Sinhala | English | No | N/A | Not available |
Zulu | English | No | N/A | Not available |
Large Language Models
LILT is shipped with:- META: Llama-3-70B-Chat-AWQ
- OpenAI: Whisper-large-v3
GPU Requirements
V3
language models require three dedicated T4 GPUs. Two GPUs for Translate and One GPU for Batch requests, or two A10 GPUs, one for translate and one for batch
Number of GPU(s) | GPU Support |
---|---|
3 | V3 |
If you are using split GPUs, please ensure that each split is allocated 16GB of memory.
Configuring V3 Models
This is no longer relevant to any releases done in 2025 and beyond, and will be removed from the documentation sometime in the future.
Updating V3 Language Models
-
Choose desired
V3
language pairsSetrouting:onpremValues:config:selfManagedV3LanguagePairs
to required lang-pairs. Sample config: -
Update GPU count for
translatev3
service as per the compatibility metricsSettranslatev3:onpremValues:config:resources:nvidia.com/gpu
-
Run installer script
Disabling All V3 Language Models
If you wish to completely disable all V3 languages, follow the guide below.-
Edit custom
values.yaml
and enable referred services by settingenabled: false
-
Services:
- batch-tb [Leave it enabled if you wish to use v2]
- translatev3
- updatev3
- update-managerv3
- batchv3
- batch-worker-gpuv3
-
Services:
-
Comment or remove below parameter:
-
Run installer script
Upgrading from V1 to V2/V3
If you’d like to upgrade to V2/V3 models from V1, please ensure the following resources exist in your cluster:Master | Node* | GPU Node* |
---|---|---|
No changes required | * 128 vCPUs |
- 1024G RAM
- 1TB Disk space | - > 1 GPU
- 16 vCPUs
- 120G RAM
- 1TB Disk space |
Costs
For accurate figures, make use of the AWS cost calculator or refer to your cloud provider’s infrastructure cost tool.New Language Support
We add new AI-supported languages to the LILT Platform continuously based on three criteria:- community and customer demand
- availability of training data
- baseline translation quality that is competitive with Google Translate and Microsoft Translator