About two months ago, we switched our article translation from Google Translate to LLM.
Why Claude? How does the system work? And why did we choose Claude over the more cost-effective Gemini?
While still in the experimental phase, we're documenting our current setup and approach.
Google Translate works fine for translation
To be clear upfront: Google Translate wasn't the problem.
It's fast and sufficient for everyday use. We hadn't encountered major issues until now, but when translating entire articles, simply substituting Japanese with another language felt entirely inadequate compared to what modern AI can deliver in terms of accuracy.
Looking at individual sentences, they're not wrong, but when you read the full article, something feels slightly unnatural. Word translations shift halfway through, and design elements like headlines or copy that use characters as single-character delimiters fall apart.
We need translation that doesn't just convey meaning—it reads naturally as an article.
So, better late than never, we decided to switch to machine translation using LLM.
Last year, we expanded to multiple languages using Google Translate, and our Domain Rating dropped from 30 to 19.
The reason we switched to LLM wasn't just about translation quality.
When we expanded our articles to multiple languages using Google Translate, we also made various changes to our SEO architecture. However, since we added support for 8 languages all at once, our link juice diminished significantly.
Our Ahrefs Domain Rating dropped from 30 to 19. 😨
That's an 11-point drop.
It was quite a dramatic decline.
Of course, it's not simply that using Google Translate alone caused the DR to drop. The real issue was our SEO architecture on the multilingual site side, including URLs and internal links. We urgently needed to rethink the entire multilingual system, including how we approach translation and structure translated articles!
Our DR should be back above 30 by now, but since we're planning a few more experiments, it will probably drop again. lol
Up and down and up and down.
After comparing the results, Claude performed best for our articles.
When switching to LLM translation, we researched multiple models and had them translate actual articles to compare.
We primarily evaluated the following aspects:
- Whether translated sentences read naturally
- Whether context is maintained across lengthy articles
- Whether terminology and conventions are consistent
- Whether the relationship between headings and body text is understood
- Whether specified translation rules are followed
- Whether content not present in the source is added without permission
Based on the comparison, Claude's translation appeared to be the best fit for our articles.
This does not mean that Claude is the superior choice for all translation work in the world.
These results are based on testing with our evaluation criteria and articles, but the outcome will vary depending on the type and length of text to be translated and the desired tone.
This time we chose Claude API to prioritize naturalness as an article.
Cost-wise, Gemini is cheaper.
While Claude performed better in translation quality, Gemini is more cost-effective when looking at price alone.
We're not committed to using Claude from the start, but we'll first operate with Claude, which showed better translation quality.
If Gemini can also meet the required quality, we'll switch to it later.
Or we might use multiple models depending on the type of article.
We're considering that kind of approach as well.
LLMs evolve rapidly, so the best model right now may not be the best in six months.
Both model performance and pricing change.
Use what works well now, and switch when conditions change.
I think that's the right mindset for AI adoption these days!
Call the Claude API only for untranslated articles
We don't call the Claude API every time an article is displayed.
At build time, we cache translated articles. From the next time onward, we return the saved translation. We call the Claude API only for articles that haven't been translated yet.
Translating the same article every time it's accessed incurs API charges each time and increases the wait time before display.
Translate only the articles that are needed, and reuse translations that have already been created.
With this approach, we can keep both API usage costs and display speed in check.
Retranslate when an article is updated
When caching translation results, we need to handle the case when the original article is updated.
It would be problematic if we fixed a Japanese article but the translated version still shows the old content.
Therefore, when the source text changes, we retranslate it with the Claude API instead of using the saved translation.
We're currently in the process of refining this area as we use it in practice.
Summary
Regarding translation quality, we're satisfied with Claude for now, but given cost and accuracy considerations, we may naturally switch to other models like Chat GPT or Gemini at some point.
Experimentation continues! We'll cover the details around SEO in a separate article!!
A CEO who always acts as a true counterpart. Someone who loves understanding new technologies, finds joy in those moments when something becomes more convenient, and is a hands-on person who dives deep into projects. Excited about the technologies of tomorrow, enjoying new experiences at every stage of life.
Morimoto
Project Manager / Director / Founded in 2007