Generative AI has been top of mind for the better part of two years. Now, executives across industries and functions are asking their teams to find ways to use it to reduce costs or increase efficiencies. But the question is, does AI actually work in translation?
Over the past 18 months, Smartling has focused our Research & Development efforts on figuring out an answer. Specifically, we wanted to know if AI could help us deliver translation solutions with the best quality, cost, speed, and experience in the world.
The answer? A resounding yes! AI is working and it is working at scale. But, not every application is a winner.
Where does AI work today in translation?
When Large Language Models (LLMs) first came out, companies started testing them as a direct replacement for machine translation. However, although this use case has potential, there are still a few kinks to work to ensure consistent performance.
Our R&D team evaluated several LLMs and machine translation across multiple languages and quality metrics (i.e. BLEU, MQM, TER, etc). This experiment showed that using LLMs for translations across all languages, automated quality evaluation metrics indicated that machine translation engines still outperformed LLMs on translation quality. On top of that, machine translation is a much more affordable approach for translation than using LLMs.
While LLMs are not yet the optimal option for translation, our team started to see them shine in automatic post-editing and evaluation of machine translation outputs. Rather than being a replacement for machine translation, AI is proving to be a powerful addition to machine translation.
In particular, the following are proving to be strong applications for AI:
- Fuzzy match repair
- Glossary term insertion
- Edit effort estimation
- Formality
- Gender de-biasing
With this knowledge, we began developing and deploying new AI solutions for our customers. It started with AI-Powered Human Translation, a translation solution that brings together Smartling’s expert linguists and AI technology to deliver high-quality translations at half the cost and twice as fast.
Then, customers began to ask us whether they could use our AI features with their third-party language service providers. So we said yes.
Introducing the AI Translation Toolkit
Smartling’s AI Translation Toolkit delivers a suite of AI translation features designed to significantly reduce translation costs and improve linguist performance without sacrificing quality. We’re making our AI technology that has delivered incredible outputs for our AIHT customers available to everyone, including:
- AI Fuzzy Match Repair: Increase translation memory leverage by as much as 35 percentage points, leading to substantial cost savings and improved quality.
- AI Glossary Term Insertion: Seamlessly embed your glossary terms into content, guaranteeing contextual and grammatical accuracy.
- Edit Effort Estimation: Groups translations based on expected effort to achieve human quality. Insights can be used in our workflow automation to assign strings to specific linguists or bypass human review entirely.
Let’s dive into each of these features and the research supporting them.
Deep Dive: AI Fuzzy Match Repair
Today, leveraging translation memory is a standard practice. However, AI Fuzzy Match Repair takes the traditional approach a step further by using LLMs to repair and smooth strings that would not usually be leveraged for savings.
For example, in the picture below you can see that the source text and the translation memory match are just one word apart. With regular translation memory, the source text would typically have to be completely retranslated. But, with fuzzy match repair, the LLM is able to identify the difference and provide a repaired translation. The suggested translation would now be eligible for fuzzy match discounts giving customers more leverage of their TMs and reducing the workload for linguists.
While in concept, this sounds like a really cool feature - our R&D team wanted to make sure that it worked. More importantly, they wanted to make sure that it provided real value in the form of cost, quality, or speed improvements.
To do this, they translated the same content twice - once with AI fuzzy match repair on and once with it off. They evaluated both scenarios based on the HTER of the output translation and the final human translation. We wanted to see how many strings would need no human intervention after Fuzzy Match Repair.
The research found that AI fuzzy match repair increased the translation memory matches by up to 35 percentage points and did so fairly consistently across languages. This means that customers can use translation memory leverage for much more of their content, leading to substantial cost savings and improved consistency.
Deep Dive: AI Glossary Term Insertion
Another standard practice in translation is glossary term insertion. The traditional approach is to simply rip and replace terms in the source and target text with the relevant terms from the glossary. However, this tends to introduce new fluency issues when those glossary terms don’t agree with the numerical count, inflections, or other grammatical elements of the overall sentence.
With AI Glossary Term Insertion, an LLM is used to “smooth” the insertion of the glossary term, ensuring contextual and grammatical accuracy. For example, in the picture below, you can see that the traditional application of the glossary term “publicación” has introduced a grammatical error into the overall string where the associated adjective now has the incorrect inflection (primer vs. primera). With the help of an LLM, the appropriate adjective form is inserted and the error is eliminated.
Based on our research, LLMs can have a huge impact on the number of morphologically correct glossary insertions. When tested for Russian and Spanish content, the AI Glossary Term Insertion feature increased the number of correct insertions by 25% and 17.5% respectively. That is content that no longer requires human effort to fix.
Deep Dive: Edit Effort Estimation
Our Edit Effort Estimation technology groups translations based on expected effort to achieve human quality. All strings are sent with our proprietary prompt to an LLM which evaluates strings based on:
- Grammatical correctness
- Fluency
- Semantic coherence
- Lexical accuracy
- Quality checks
- Style guide (unique to each customer)
These groupings (Level 1, Level 2, and Level 3) can be used within dynamic workflow automation to treat strings based on their expected effort. For example, a company could assign strings requiring extensive edits to more experienced linguists or bypass human review entirely on strings requiring little to no edits. Customers can also use our Edit Effort Estimation discounts to create discount tiers for high performing stings.
This has the potential to deliver significant time and cost savings to companies, especially to those heavily utilizing human-based translation workflows. It also gives teams deeper insights into their translation process.
Revolutionize your translation strategy
We have seen AI working to expand leverage, improve the accuracy of glossary term insertion, apply formality, and more. However, it takes a lot of time and research to identify the places where AI can be the most effective and deliver a strong return on investment. That’s where the help of a partner, like Smartling, comes in, so you don’t have to do it on your own.
A critical component of our strategy is our investment in R&D to make sure that we're able to provide the most comprehensive LanguageAI™ solution in the industry. We have multiple patents on our work with generative AI, and our engineering team has delivered over 70 major features or enhancements across an astounding 3300 production releases just in the past year.
That's what it takes to truly ensure that AI solutions deliver the best quality, cost, speed, and experience in the world.
Learn more about our latest research and what we’re exploring next in the Global Ready Conference session: Research Revealed - now on demand.
