AI Video Localization Without Re-Rendering

AI Video Localization releases one short story video in up to 17 languages without touching the original render.

17 languages from one finished video

4–20× cheaper than a dubbing API, by the team’s estimate

1 click to generate every language variant

Partner's Profile

Background

The client runs a platform that produces short AI-generated story videos for Instagram Reels end to end: script, images, clips, voice-over, lip-sync, editing and publishing. Every story was produced in a single language. The client wanted to release the same story in several languages, three, five or seven at first, without a new render each time. Some target languages are only weakly covered by multilingual voice models, so a simple translate-and-regenerate approach would not work.

Objective

Add language variants to the existing production pipeline so one finished story can be released in many languages. The visuals stay untouched; only the text, voice-over and subtitles change. Operators need to review translations, set default voices per language for a campaign and override them for a single video. Cost per video mattered as well, which meant a deliberate choice between a dubbing service and our own re-voicing pipeline.

Solution

We extended the platform's Django and Celery backend with language variants: one variant per language, attached to the original story. An LLM translates the script, the operator edits it, and the text is re-voiced with per-language voice defaults that are set per campaign and can be overridden for a single video. Subtitles are recomputed with Whisper, and a single “Generate all languages” action builds every variant in the background. Support grew to 17 languages, from English and German to Arabic, Hindi, Japanese, Korean and Chinese.

The hard part is timing. Translated speech is rarely as long as the original, and the scene video is fixed. Audio is fitted to the picture by a cascade with a tolerance of 0.15 s:

trim pauses;
time-stretch up to 1.4× while preserving pitch;
trim the tail of the video;
freeze a frame only as a last resort.

Montage stories need the opposite logic. Our first version rendered a 91.08 s video for 61.75 s of audio, leaving a silent tail. We now recompute the edit list so that clip durations add up exactly to the audio length. A separate mode synthesizes the whole track in one pass and aligns words with Whisper; if drift exceeds 0.75 s, it falls back to per-segment synthesis. This alignment work sits at the intersection of AI software development and ML engineering.

We also made an explicit build-vs-buy call. By the team's calculation, a 60-second video in five languages costs about $0.45 with TTS re-voicing versus $1.65–8+ through a dubbing API, which makes it 4–20× cheaper, and the dubbing API was not programmatically available at the time. The accepted trade-off: in lip-synced scenes, lip movements do not match the new language.

Language Variants, Same Render

Each story gets one variant per language, attached to the original. Only the script, voice-over and subtitles change; images, clips and editing stay exactly as rendered. A single "Generate all languages" action builds every variant in the background.

Translation with Operator Review

An LLM translates the script and the operator edits it before voicing. Voice defaults are set per language for each campaign and can be overridden for a single video, so every language can use the voice that suits it.

Audio Fitted to Fixed Video

Translated speech rarely matches the original length. A cascade fits it within 0.15 s: trim pauses, time-stretch up to 1.4× with preserved pitch, trim the video tail, and freeze a frame only as a last resort.

Exact Montage Timing

For montage stories, the edit list is recomputed so clip durations add up exactly to the audio length. This replaced a first version that rendered 91.08 s of video for 61.75 s of audio, leaving a silent tail.

Whisper Subtitles and Alignment

Subtitles are recomputed with Whisper for every language. In whole-track mode, words are aligned against the synthesized audio; if drift exceeds 0.75 s, the pipeline falls back to per-segment synthesis automatically.