Showcase

🔊 Please turn on sound for the best experience

Abstract

Audio-video (AV) generation has recently made strong progress in perceptual quality and multimodal coherence, yet generating content with plausible motion-sound relations remains challenging. Existing methods often produce object motions that are visually unstable and sounds that are only loosely aligned with salient motion or contact events, largely because they lack an explicit motion-aware structure shared by video and audio generation. We present Tora3, a trajectory-guided AV generation framework that improves physical coherence by using object trajectories as a shared kinematic prior. Rather than treating trajectories as a video-only control signal, Tora3 uses them to jointly guide visual motion and acoustic events. Specifically, we design a trajectory-aligned motion representation for video, a kinematic-audio alignment module driven by trajectory-derived second-order kinematic states, and a hybrid flow matching scheme that preserves trajectory fidelity in trajectory-conditioned regions while maintaining local coherence elsewhere. We further curate PAV, a large-scale AV dataset emphasizing motion-relevant patterns with automatically extracted motion annotations. Extensive experiments show that Tora3 improves motion realism, motion-sound synchronization, and overall AV generation quality over strong open-source baselines.

Pipeline

The overall architecture of Tora3. Object trajectories provide a shared kinematic prior for both video and audio generation. The video branch uses trajectory-aligned motion injection, while the audio branch incorporates trajectory-derived kinematic features to better align sound with motion. Hybrid flow matching further improves trajectory-guided generation by balancing motion fidelity and local coherence.

Experiments

• Comparison on video quality, audio quality, text alignment, motion control, and audio-video synchronization, including event-level and intensity-level motion-sound coherence metrics.

Comparison

• Qualitative comparison. Tora3 produces more stable visual motion and more coherent audio in both examples, including motion-consistent loudness variation for the moving motorcycle and better synchronized rolling and collision sounds for the rolling ball.

🔊 Please turn on sound for the best experience

BibTeX

@misc{liao2026tora3trajectoryguidedaudiovideogeneration,
      title={Tora3: Trajectory-Guided Audio-Video Generation with Physical Coherence}, 
      author={Junchao Liao and Zhenghao Zhang and Xiangyu Meng and Litao Li and Ziying Zhang and Siyu Zhu and Long Qin and Weizhi Wang},
      year={2026},
      eprint={2604.09057},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2604.09057}, 
}