Data Source Memory Matching

Each Data Source has memory entries containing source/target pairs, called TM entries. LILT improves translation speed and consistency by matching TM entries to segments that need translating.

Where TM matching is used in LILT

• TM segment suggestions: When working in LILT Translate, translation suggestions are provided to linguists for the active segment. If there are any TM matches above 75%, the best match will be displayed as the suggestion. However, if there are no such TM matches, LILT uses the associated Contextual AI model to provide suggestions to the linguist. Note that if the language pair does not have Contextual AI, no Contextual AI suggestions will be shown.

• Pretranslation: When Pretranslation is run, any segments that have 100% or 101% matches with TM entries in the associated Data Source are translated.

• TM Leveraging: LILT automatically leverages TM entries in the associated Data Source to populate the target text of empty, unreviewed segments with 100% or 101% matches. This is performed across all documents that share the Data Source.

• Auto-Propagation: When a linguist confirms or accepts a segment in LILT Translate, Auto-Propagation updates matching segments (100% and 101%) with the same changes.

TM match ordering

LILT leverages TM matches in the following order:

1. By score. TM entries with higher match scores are prioritized over TM entries with lower match scores. For example, a TM entry that is a 100% match is prioritized over a TM entry that is a 95% match.

2. By recency: More recently updated TM entries are prioritized over entries that weren't updated as recently.

Note that the reviewed/unreviewed status of a TM entry does not affect match prioritization.

Prioritize recency setting

If enabled in Organization settings, this will disable 101% match priority and allow for recency-based match prioritization. 101% matches will be treated as equivalent to 100% matches, prioritizing recency over context precision.

TM match scores

The table below outlines the various match scores in LILT.

The lowest match percentage available is 75%. This is a set threshold and not configurable.

If the Organization Setting for Case-sensitive matching is enabled, TM match scoring takes into account any differences in letter casing; otherwise, casing is ignored.

In LILT, 100% matches that have whitespace differences or tag differences are downgraded to 99% matches. In the example below, the bottom segment doesn't have a new line whitespace like the top segment does, so the TM match for the bottom segment is displayed as a 99% match.

How TM match percentages are calculated

LILT uses an Edit Distance algorithm to calculate match percentages.  However, because TMs can be large, it is not practical to compare every source segment to every entry in the TM. To simplify this problem, LILT first uses a Locality-Sensitive Hashing algorithm to find a subset of the TM that is roughly similar to the source string, and then calculates the match percentage against each of those possibilities. This process is not 100% guaranteed to always produce every possible fuzzy match candidate, but rather allows for an optimized balance between quality of TM matches with speed/time to match.

In cases where our algorithm misses a fuzzy match, this is likely due to having multiple languages present in the same segment.

How TM entries are added to and removed from Data Sources

TM entry metadata: When a TM entry is added or modified, its metadata is updated:

• Entry author: Email address of the author. Authorship is not always changed when an entry is updated. The sections below describe how and when authorship is updated.

• Entry timestamp: Date and time the TM entry was last updated. The time is displayed in the viewer's time zone.

Linguist actions that update TM entries

The last person to confirm a segment is attributed as the TM entry author.

• When a reviewer accepts a confirmed segment without modifying the target text, the author is left as the translator who confirmed the segment.

• When a reviewer accepts an unconfirmed segment, they are given authorship.

• Reviews have the ability to unconfirm segments, meaning that if a reviewer unconfirms a segment and then accepts the segment, they will be given authorship.

Project manager actions that update TM entries

TM entries uploaded to the Data Source by a project manager are attributed to the TM entry authors listed in the TM file.

When a project manager manually adds a TM entry on the Manage entries page, they are made the author of the entry.

When a project manager manually modifies a TM entry on the Manage entries page, authorship is not changed.

Automated processes that update TM entries

TM Leveraging and Auto-Propagation attribute authorship to the user in the document when TM Leveraging or Auto-Propagation is run.

Pretranslation attributes authorship to either the project manager or the author of the matching entries (depending on the Pretranslation settings).