data-designer-nemotron-indic-translation¶

A document translation column generator plugin for Nemotron Indic pretraining data pipelines.

Installation¶

uv add data-designer data-designer-nemotron-indic-translation

Column types¶

document-translation¶

Translates a column into a single fixed target language, scores the result with an MQM-based evaluator, and optionally self-corrects via a rewrite loop. Supports an optional two-stage pivot through an intermediate language (e.g. translating into native script first, then romanizing).

Translation itself can run two ways, controlled by enable_windowing:

• False (default): one model call translates the whole document.
• True: the document is split into overlapping token-bounded windows (sized by the window_* fields below), each window is translated independently, then each window's own non-overlapping segment is extracted (with its neighbours shown for context, so extraction calls run concurrently) and the segments are stacked back into one document. This is for long documents that degrade whole-document translation quality (omissions, splicing, timeouts). Evaluation always runs on the whole (merged) document either way -- only the initial translation step is windowed. The rewrite loop is incompatible with windowing (it would rewrite the whole merged document in one shot, defeating the point of windowing it) -- setting enable_windowing=True alongside nonzero max_rewrites/max_rewrites_intermediate forces both to 0 and emits a UserWarning.

When keep_window_traces is also True (the default), one dict per window -- window (source text), translated_window (that window's raw translation before extraction), extracted_window/extracted_translated_window (its own non-overlapping source/translated segment after extraction, both null for a window that contributes nothing to the merged document) -- is written to <name>_window_traces/<name>_intermediate_window_traces. Both are None when enable_windowing is False, or when keep_window_traces is False (windowing/merging still run either way -- only whether the trace data is kept changes).

When target_language_script is set (a 4-letter ISO 15924 script code, e.g. Mtei), the final stage's translation is deterministically remapped into that script once the translate-evaluate-rewrite procedure finishes -- e.g. translate Manipuri into Bengali script (target_language="mni-Beng-IN") and remap into Meetei Mayek (target_language_script="Mtei"), since no translation model in this pipeline produces Meetei Mayek directly. Only the script pairs in script_remap.SCRIPT_REMAPPERS are supported (currently Bengali -> Meetei Mayek only); an unsupported pair raises a validation error when the config is constructed. The pre-remap translation/evaluation are pushed onto <name>_translation_history/ <name>_evaluation_history (same as a superseded rewrite attempt), the translation is remapped, and a fresh evaluation scores the remapped text against the same source text -- that becomes <name>/<name>_evaluation/ <name>_overall_score. Remapping only ever applies to the final stage; the intermediate stage (if any) is unaffected. The remapper is character-level and has no notion of code vs. prose, so by default (remap_preserves_protected_spans=True) fenced code blocks, inline code, and URLs are left untouched during remapping -- only the surrounding prose is remapped -- so that source-script characters inside a code literal or URL (which the translation step already promised to preserve byte-for-byte) aren't silently corrupted. Set remap_preserves_protected_spans=False to remap the whole text unconditionally.

native_numeral_probability (default 0.0, matching the source prototype's NATIVE_NUMERAL_PCT = 0.5 pattern but opted out by default) is the per-row chance of translating with translation_prompt_native_numerals (numbers in prose written in the target language's native numeral script instead of ASCII digits) instead of the plain translation_prompt -- drawn independently per row, but shared between the intermediate and final stage within that row. Digits inside code, formulas, URLs, and LaTeX stay ASCII either way. Evaluation always uses evaluation_prompt, regardless of which style was drawn for translation -- there is no native-numeral evaluation prompt variant. Only "generic" translation_mode supports native numerals at all -- bodhan-ai/indic-translate's minimal prompt has no native-numeral variant, so translation_mode="bodhan" forces native_numeral_probability to 0.0 (with a UserWarning). Requires target_language (and intermediate_language, if set) to have an entry in prompts.NATIVE_NUMBERS_DOC; an uncovered language raises a validation error when the config is constructed. translation_prompt_native_numerals (defaulting to GENERIC_TRANSLATION_PROMPT_NATIVE_NUMERALS) lets you override the native-numeral prompt the same way translation_prompt overrides the plain one; it must keep the literal {native_numbers_doc} placeholder (resolved via str.replace() before Jinja rendering) or config construction raises a validation error. Overriding translation_prompt without also overriding translation_prompt_native_numerals emits a UserWarning, since the native-numeral row would otherwise silently fall back to the generic native-numeral prompt instead of your custom one. Which style was drawn for a row is recorded in <name>_numeral_style ("native" or "none"), shared between the intermediate and final stage.

For the full plugin authoring guide, see the main repository docs.