Translation research and data prep in the age of AI

Translation research is the gathering of information, terminology, sources and context before or during translation. Although it increasingly happens with the help of language models, it must always be backed by external verification of sources.

Traditionally it rested on four pillars: physical sources (dictionaries, industry publications), domain expert knowledge, electronic resources (translation memories, terminology databases) and the internet. Language models joined that list relatively recently, and they arrived with enough force to tempt many translators into discarding everything that came before.

That would be a mistake.

The problem: LLM as the only source of truth

Language models are impressive. They can generate terminology across dozens of domains, translate context, summarise articles. But they have one fundamental problem, one we cover in more depth in our article on hallucinations: they don't know when they're wrong, and they can't tell on their own.

In translation research, that translates into concrete failure modes: a term that sounds plausible but is never actually used in the field; a link to an article that doesn't exist or contains entirely different content; a "quote" that never appeared in the cited source.

From a non-deterministic to a deterministic process

The answer follows the same pattern as any deterministic process: the LLM is just one of many steps, and it's never the step that decides on its own whether the data can be trusted.

External verification, meaning something that doesn't rely on yet another query to the model, has to be deterministic here. In practice, that means a script that checks whether a given URL exists and whether the phrase you're looking for is actually on it. Confirming the URL exists isn't enough: you also need to confirm the cited content is actually there.

AI agents in practice: what doesn't work the way you think

AI agents, systems capable of autonomously searching the internet and taking consecutive steps without user intervention, sound like the perfect research tool. In part, they are. But with important caveats.

Many valuable sources block automated access through robots.txt or paywalls. An agent will not report this gracefully; it will simply return nothing, or return an error that is easy to overlook. A separate issue is fake sites and AI-generated content: estimates suggest that more than half of content on the internet is now produced with the help of language models (not this article, we promise). Agents cannot distinguish text "written by an expert" from text "generated on a content farm." Without additional quality criteria, the entire pipeline is only as trustworthy as its weakest source.

Hence the principle of intermediate formats: instead of asking an agent to "describe the terminology of domain X," ask for structured data: JSON, CSV, a table with specific columns. Intermediate formats force precision and give you something that can be checked and filtered without another model query.

If you don't want to write scripts, Excel or CSV works well as a manual checkpoint: you can quickly review the intermediate output by eye before it moves further down the pipeline. If you can verify something with a script instead of an LLM, don't think twice.

One caveat is worth making here: the newest model generations (the Fable/Mythos family, or Opus) already handle this noticeably better. Across multiple iterations and with sub-agents in play, the model increasingly filters reliable sources from questionable ones on its own. That doesn't let us off the hook for designing the process, though: we're still the ones deciding which steps need external verification, no matter how good the underlying model is.

In practice: building a glossary step by step

Here is a concrete example. We want to build a glossary of dental instruments (EN↔PL) with verified sources.

Step 1: prompt for a web-search model

A well-constructed prompt specifies not just what you want but how to verify it at the data-collection stage:

Build a glossary of dental instruments. For each term: a) the term in English, b) a link to a medical site or article where the term is used, c) an exact quote of the sentence in which the term appears, d) the Polish translation, e) a link and quote for the Polish version, f) synonyms in separate rows. Save the result as CSV and JSON. Minimum 30 terms. Finally, assess the reliability of each source and discard any term whose only source is Wikipedia.

Two elements are critical here. First, the quote: not just the link, but the passage of text in which the term actually appears. Second, the rejection criterion: Wikipedia is not enough; we need peer-reviewed articles, medical databases, official glossaries. This is not snobbery towards Wikipedia; the point is to bake logic into the prompt that will eliminate some of the noise before we even look at the results.

Step 2: verification by script

Even the best prompt cannot guarantee that links exist and that the quotes actually appear in them. This is where a script comes in, generated via any code-capable environment (Cursor, ChatGPT with interpreter, Claude):

Write a script that: 1) checks whether the URLs in the JSON file exist (200 OK response), 2) searches for the quoted passage from the file on that specific page, 3) removes terms with a low reliability score. The result for each term is pass/fail.

We run the script on the output from step 1. Terms that fail are dropped. This is not another language model assessing reliability; it is deterministic code that either finds the quote on the page or does not. The difference is fundamental.

Step 3: clean-up and enrichment

After filtering, what remains are terms with confirmed sources. Only now is it worth manually supplementing them, adding metadata (domain, specialisation level) and converting to the target format: XLIFF, TMX or TBX if the data is heading into a CAT environment. The critical rule: do not pipe the data through an LLM for conversion. A model working directly on your data may silently alter term content in ways that are hard to catch. The safer approach: ask the model to generate a conversion script, then run the script on the data. The script won't change a term's meaning; the model might.

Summary

The principle is simple: the less language model in the verification step, the more predictable the result. LLMs are great for generating initial data and for tasks where errors are acceptable. They are not a good tool for checking their own answers.

A solid translation research pipeline looks like this: clear goal → structured data collection by an agent → external verification (a script, not another prompt) → clean-up and metadata → use. At every stage we know what goes in and what comes out. AI stops being a lottery and becomes a production tool.