Review Sampling

LILT’s review sampling settings allow managers to find their team’s perfect balance between reviewing more segments to catch errors and reviewing fewer segments to save on time and money.

By default, a reviewer will review 100% of each document within a Project. However, it is possible to assign only a sample of the Project to be reviewed. Reducing the sample rate can help reduce the time and costs of translation projects.

The default settings for review sampling are managed at the organization level, but they can also be adjusted on a per-Project basis (either when creating a new Project, or in the Project settings page of existing Projects).

The sections below discuss the functionality of the different review sample settings and where the settings are managed within LILT.

Review sampling workflow

A document must be marked as Done in the translation stage for review sampling to take effect. If the document is not marked as Done by the translator, the reviewer will be given the entire document to review.

When reviewing a document and review sampling is less than 100%, once the document has been marked as done in Translate mode, a portion of the segments in the document become locked (for both translators and reviewers) equal to the review sampling rate. Reviewers are shown the entire document, but they are only able to work on unlocked segments.

Reviewers will go through all the unlocked segments to accept any segments with correct translations and edit any segments that need to be corrected.

When all documents in the Project are marked as done in Review mode, the manager will be notified.

The manager is notified the project is completed via email. The email includes the number of segments reviewed and the error rate of those segments.

How to unlock segments

Once segments have become locked in a document, if you need to unlock any segments, follow these steps:

1. A project manager needs to open the document in Translate mode and click on File > Unlock document translation.

2. In Review mode, the locked segments can now be unaccepted to unlock them in both Translate and Review mode.

3. Once the necessary segments have been modified, marked as done in Translate mode again to lock the segments and proceed with the document review.

Review sampling settings

Review Sample Rate

The review sample rate is a value from 10% to 100% and can be set in increments of 10%. This value controls the percent of segments that will be assigned for review. The review percentage applies to all documents in the Project.

For example, in a Project with a 20% review sample rate, a document with 100 segments would have 20 segments assigned for review.

Include exact matches

If the “include exact matches” setting is enabled, the following segment types are included in the pool of segments that can be selected for review:

• 100% TM matches

• Repeated segments

If this setting is disabled, the above segment types are excluded from the pool of segments that can be selected for review. There are two separate settings to apply this to the Review Sample and the Secondary Review Sample.

For example, if a document is 100 segments long and has 40 repeated segments, the remaining 60 segments will be included in the pool of segments that can be selected for review. If this document is in a project with a review sampling rate of 20%, then 20% of the remaining 60 segments would be sampled (12 segments).

Managing review sample settings

The review sample settings are set in the Organization Settings and are used to populate the settings of new Projects. Review sample settings can also be set on a per-Project basis in Project settings.

Organization setting

To set the review sample settings for your organization, navigate to the Settings page, open the Workflow tab, and scroll down to the Review section. From there, you can set the settings for:

• Review sampling rate (two separate settings for Review and Secondary Review samples)

• Include exact matches (two separate settings for Review and Secondary Review samples)

Note: Changing the default setting for review sampling in the Organization Settings does not update the settings of existing Projects.

The “Include exact matches” setting is toggled off by default. Turning on “Include exact matches” at an org level would then turn it on for all new projects, or you can turn it on on a project basis depending on their needs.

Project setting

Once a Project has been created, you can change the review sample settings by opening the Project, navigating to the Project settings page, and scroll down to the Workflow settings section.

After changing the settings, click the Save settings button to apply the changes. Making changes here will not affect documents that have already been sampled for review, nor will it affect the Organization workflow settings.

Review sampling prioritization

The setting for Review sampling prioritization is available in LILT Org Settings. If review sampling prioritization is applied, segments selected for review will be prioritized based on the severity of errors found through AI Review instead of being selected randomly. AI Review will run with LILT’s standard MQM error categories, sub-categories, severities, and weighting. However, in some cases, LILT users may customize AI Review score weighting that factors into a document’s overall AI Review quality score in Project Board, or add sub-categories from the additional pre-defined sub-categories.

If the Review sampling prioritization setting is toggled on, the analysis will trigger right after the Translate (Instant or verified) step is complete. The UI will indicate in the projects page a new workflow state that shows AI Review is running, and it will prevent users from translating/reviewing segments during this time. Any user will be able to see this workflow state running if they have access to this page. AI Review can take up to 3 hours to run. If it gets to 3 hours and has not run, it will automatically progress to the Review stage.

With this setting on, 20% of all segments are randomly selected and reviewed, with the remaining 80% of segments reviewed in order of stackranking based on error severity (Critical → Major → Minor → Neutral (etc)) until the review % sampling is met.

• For example, let’s say an org has review sampling set to 50%, and a document has 100 segments. If Review sampling prioritization is turned on, it will prioritize 50 segments through review sampling. 10 of these segments will be randomly selected, and the remaining 40 will be selected based on the results of the AI Review error severity stackranking.

Revision reports will display additional review sampling prioritization metrics:

1. Accepted random sampled segments with errors. The error rate on the randomly selected segments for review sampling.

2. % Accepted random sampled segments with errors. The % of random sampled segments with errors.

3. AI Review error rate. The error rate among the segments that were selected for AI Review.