FlowTranslator: Training-Free In-Image Machine Translation with Flow-Based Editing

| Source: arXiv AI

Tags: FlowTranslator, in-image machine translation, diffusion models, flow-based editing, UniTranslator, OCR

FlowTranslator translates text inside images with no task-specific training, pairing OCR and an LLM with flow-based diffusion editing and beating the trained UniTranslator baseline on BLEU and COMET.

Details

In-image machine translation replaces text in an image with its translation while keeping the visual context. Existing models need large paired before/after training sets. FlowTranslator skips training: an OCR model and an LLM build a translation prompt that is injected into a text-to-image diffusion model. Two trajectory controls, Region Coupling Release and Coupling Expansion, steer where and how strongly the text region is edited. Under an EasyOCR read-back protocol it reports average PRIM BLEU of 20.6 and COMET of 57.2, ahead of UniTranslator, and 40.1% exact-match accuracy on ScenePair photos translated into Chinese.