Dubbing video content believably demands a delicate balance between semantic fidelity, syllable pacing, and natural vocal timbre. Commercial cloud services operate as rigid subscription black boxes that ignore lip-sync timing constraints and require uploading proprietary footage to third-party servers.

DubAID was created to provide a fully local, modular, and reproducible auto-dubbing pipeline, translating and adapting video from Italian to English while faithfully respecting original pauses and speech cadence.

The workflow begins with clean dialogue isolation and background stem separation using MelBand-RoFormer or Demucs, backed by automated RMS, LUFS, and clipping analysis before moving to downstream stages.

Speech recognition uses Whisper large-v3 with word-level timestamps, re-segmented into natural cadence units based on punctuation and pauses. A local LLM (LM Studio / Ollama running Qwen or Gemma) translates under strict syllable budget constraints (target 5.5–6.5 syllables/s): if length overflows the slot, the model iteratively condenses lines with fallback to human review.

Voice generation is powered by zero-shot neural voice cloning with Fish Audio S2 Pro via ComfyUI, regulated by a time-stretch guard that caps dilation or compression to an unnoticeable ±8% without cutting audio.

The pipeline concludes with a deterministic master audio mix via FFmpeg: dynamic sidechain ducking (-6 dB), de-essing, and two-pass loudness normalization adhering to -16 LUFS (web) or -23 LUFS (broadcast), paired with shot-split face QC ready for facial lip-sync inference.