當用戶給出一句簡單提示詞時，當前的音視頻生成模型往往已經能夠生成具有不錯質量的視聽內容。然而，一旦提示詞變得復雜，問題便開始暴露出來。

例如，用戶可能要求模型生成這樣一個場景：一個男孩先完成運球訓練，隨后開始講話；或者兩個人在互動過程中依次說出不同內容；又或者某個動作發生后，對應的聲音才逐漸出現并增強。

這類包含多階段動作、復雜人物交互以及明確時序關系的指令，不僅要求模型理解「發生了什么」，還要求模型準確推理「什么時候發生」「誰在發生」以及「聲音應該如何與畫面對應」。

遺憾的是，對于這類需要長程語義理解的復雜場景，目前大多數開源音視頻生成模型仍然表現不佳。生成結果中經常出現人物動作與聲音錯位、多角色對白對應錯誤、音畫節奏不同步等問題。

其根本原因在于，現有方法大多將文本提示編碼為單個全局語義向量，并將其同時作為視頻與音頻生成過程的條件信號。雖然這種方式能夠提供場景級別的語義指導，但卻難以進一步拆解復雜事件之間的時序關系，也無法明確描述不同角色、動作和聲音之間應如何對應。

圍繞這一問題，研究社區已經進行了大量探索。例如，Ovi 率先構建了原生視頻 — 音頻聯合生成框架，并采用雙分支 DiT 架構同時建模視覺與聽覺信號；LTX-2.3 進一步提升了模型規模與訓練數據質量；MOVA 等工作則增強訓練策略與跨模態協同機制入手。與此同時，隨著多模態大語言模型的發展，一些研究開始引入 Qwen3、Qwen3-VL、Qwen3-Omni 等模型，希望利用其更強的語義理解與推理能力，對用戶提示進行擴展、重寫或增強，從而為生成模型提供更豐富的條件信息。

然而，上述方法大多仍然遵循同一種范式：先將復雜提示壓縮為統一的全局語義表示，再將其作為條件注入視頻與音頻擴散過程。

這樣的設計雖然能夠告訴模型「場景中有哪些內容」，卻難以進一步描述「哪些事件應該先發生、哪些事件應該后發生，以及對應的視覺與聲音信號應當如何在時間軸上保持一致」。由于缺乏顯式的跨模態語義規劃機制，視頻與音頻往往只能依據同一個模糊條件獨立完成生成，最終在復雜場景下逐漸形成彼此偏離甚至相互沖突的生成軌跡。

因此，當面對多階段動作鏈條、復雜人物互動、長程因果關系乃至多說話人對話等任務時，現有方法仍然難以生成穩定、高保真且高度同步的視聽內容。

為了解決這一問題，復旦大學與騰訊混元團隊提出了 Baton。與現有方法直接利用全局文本特征驅動擴散生成不同，Baton 的核心思想是將語義推理與內容生成解耦：模型首先構建一份跨模態共享的語義藍圖（Semantic Blueprint），隨后再依據這份藍圖同步生成視頻與音頻。

在這一框架下，視頻和音頻不再各自獨立地理解用戶提示，而是共享同一份包含事件、角色、時序關系和跨模態對應關系的中間規劃結果。借助這一顯式規劃過程，模型能夠在生成開始之前完成復雜語義關系的推理，從而為后續擴散過程提供穩定且一致的指導信號。

為實現這一目標，Baton 設計了 VA-Planner 和 Relative Semantic RoPE 兩項關鍵技術。其中，VA-Planner 負責生成跨模態語義藍圖，而 Relative Semantic RoPE 則負責將藍圖中的規劃信息準確映射到擴散模型的生成空間中，從而實現視頻與音頻的精細協同生成。

• 論文標題：Baton: Explicit Semantic Blueprints for Joint Video-Audio Generation
• 論文地址：https://arxiv.org/pdf/2605.25195
• 項目地址：https://francis-rings.github.io/Baton/

方法簡介

如下圖所示，Baton 通過顯式解耦語義推理與內容生成兩個階段，構建了一套具備模態感知能力的語義藍圖（Blueprint）機制，用于統一協調視頻與音頻的擴散去噪過程。

具體而言，用戶輸入的文本提示首先被送入多模態大語言模型（MLLM）進行語義推理，從中預測出一對分別對應視頻和音頻模態的 planned tokens。這些 planned tokens 充當跨模態共享的語義藍圖，為后續生成過程提供明確的內容規劃和時序指導。為了將語義藍圖融入生成過程，planned tokens 進一步通過 cross-attention 注入擴散 Transformer（DiT）中。這里采用的 DiT 延續了 Ovi 中的雙分支架構，分別負責視頻與音頻的生成與去噪。

值得注意的是，planned tokens 與擴散模型中的 latents 分布在不同的時空網格上，兩者天然存在位置對應關系不一致的問題。

為了解決這一問題，Baton 提出了Relative Semantic RoPE（RS-RoPE）機制，通過構建統一的相對位置編碼空間，實現 planned tokens 與 diffusion latents 之間的精確語義對齊，從而確保語義藍圖能夠有效指導視頻和音頻的聯合生成過程。

如下圖所示，Baton 將語義理解和內容生成拆分成兩個獨立階段，并通過一種跨模態的規劃機制來保持視頻與音頻的協調一致。

具體來說，系統首先對用戶提供的文本指令進行深度解析，由多模態大語言模型（MLLM）生成兩組專門的 Planned Tokens，分別對應視頻和音頻模態。這些規劃信息可以看作一份「生成藍圖」，明確規定了內容應該呈現什么，以及事件在時間上的先后關系，為接下來的生成步驟提供導航。

在生成過程中，Planned Tokens 通過跨注意力機制注入擴散 Transformer（DiT），使得生成模型在每一步去噪時都能參考這份語義藍圖。Baton 沿用 Ovi 的雙分支設計，視頻和音頻各自擁有獨立的生成路徑，但通過藍圖保持同步。

由于藍圖與擴散潛變量在空間和時間上存在天然的不匹配，為保證精確對應，Baton 引入了Relative Semantic RoPE（RS-RoPE）。這一機制為規劃信息和潛變量建立了統一的相對位置編碼，使生成模型能夠準確地將語義指導映射到每個生成單元上，從而實現音畫高度同步的聯合生成。

1. VA-Planner：跨模態語義規劃

與直接使用全局文本嵌入不同，VA-Planner 首先利用多模態大語言模型對用戶提示進行顯式語義推理，并生成一組分別對應視頻和音頻模態的 Planned Tokens。這些 Tokens 不再僅僅表示整體場景，而是進一步編碼了局部事件的語義信息，包括發生了什么、發生在何處以及發生在什么時間。

具體而言，Baton 將視頻規劃區域與音頻規劃區域共同組織到同一個自回歸推理序列中，并利用 MLLM 逐步預測對應的語義表示。

由于視頻與音頻規劃共享同一上下文，同時位于統一的推理過程中，因此模型能夠在生成階段之前就建立跨模態關聯關系。最終得到的 Planned Tokens 可以被視為一份跨模態共享的語義藍圖，為后續視頻與音頻生成提供統一且細粒度的規劃信息。

2. 雙語義對齊塔：構建視頻與音頻共享的語義藍圖

雖然 VA-Planner 已經能夠生成視頻和音頻對應的語義規劃，但這些表示仍然位于 MLLM 的語言空間中，與擴散模型實際使用的視覺和音頻特征空間之間存在明顯差異。

因此，Baton 進一步設計了Dual Semantic Alignment Towers（雙語義對齊塔），負責將規劃結果轉換為更適合生成模型理解的感知語義表示。

具體而言，Baton 分別構建視頻塔和音頻塔，并采用 SigLip2 與 WavTokenizer 作為對應模態的感知監督目標。每個對齊塔內部都包含一組可學習查詢（Learnable Queries），用于從視頻規劃表示和音頻規劃表示中提取最關鍵的語義信息。

更重要的是，雙塔引入了雙向跨模態注意力機制。由于 MLLM 的自回歸結構天然具有單向依賴關系，視頻規劃無法直接感知音頻規劃的信息。為了解決這一問題，視頻塔在提取視覺語義的同時還會主動吸收音頻信息，音頻塔則同步引入視覺信息，從而實現雙向語義交互。最終得到的視頻與音頻 Planned Tokens 不再是兩份獨立規劃，而是共享同一時間軸和語義結構的統一藍圖。

為了進一步建立跨模態時序對應關系，Baton 還引入了Timestamp-based RoPE，將視頻關鍵幀與音頻片段映射到統一時間坐標系中，使模型能夠準確理解不同模態事件之間的時間對應關系。具體實現細節和詳細公式推導請閱讀原論文。

3. RS-RoPE：讓語義藍圖真正「落地」到生成過程

在實際生成階段，Planned Tokens 與擴散模型中的潛變量（Latents）位于不同的時空網格上。前者描述的是關鍵事件和語義結構，而后者對應的是視頻和音頻在擴散過程中的具體表示，兩者之間并不存在天然的一一對應關系。如果直接進行跨注意力交互，模型很難準確判斷某個潛變量應該關注哪部分語義規劃信息。

為了解決這一問題，Baton 提出了Relative Semantic RoPE（RS-RoPE）。與傳統位置編碼僅描述 Token 絕對位置不同，RS-RoPE 構建了一套統一的相對語義坐標系，將 Planned Tokens 與擴散潛變量映射到同一參考空間中。

借助這一機制，擴散模型能夠在去噪過程中準確找到與當前時空位置最相關的規劃信息，使語義藍圖真正參與到每一步生成過程之中。

換句話說，RS-RoPE 相當于為 Blueprint 和擴散生成之間建立了一座精確的「導航系統」，確保視頻和音頻始終沿著預先規劃好的語義路徑協同演化。

訓練策略

Baton 的訓練采用三階段策略：

1. VA-Planner 預訓練

在第一階段，模型學習將用戶提示轉化為跨模態語義規劃（Planned Tokens）。利用真實視頻和音頻數據作為監督，VA-Planner 學會生成能夠反映視覺和音頻感知結構的連續特征，而不僅僅依賴自然語言嵌入，從而獲得更豐富的語義信息。

2. DiT 適配訓練

第二階段旨在讓擴散模型（DiT）學習這些語義特征的分布。此時，DiT 以真實特征作為條件進行訓練，能夠熟悉視頻與音頻的生成規律，同時避免被 VA-Planner 預測誤差干擾。

3. 聯合微調

最后，VA-Planner 與 DiT 組合成完整系統，VA-Planner 參數凍結，DiT 接收規劃器預測的 Planned Tokens 作為輸入進行訓練。這一步能夠彌合理想特征與實際預測之間的差距，緩解曝光偏差問題，使生成過程更穩定、魯棒。

實驗

在定量試驗對比上，Baton 與以前的開源模型在 Verse-Bench 和 Sem100 上進行指標對比，其中 Verse-Bench 為開源的音畫一致生成的測試集，Sem100 為內部收集的 100 條測試視頻樣例，相比于以前的開源測試集，Sem100 的 text prompt 具備更加復雜的描述，包括人物與周圍環境的多次連續性交互動作，涉及多人的復雜交互，涉及多個連續指定性質的復雜組合動作描述。對比結果如下表所示：

評測指標涵蓋視頻質量（AQ、IQ、DD、ID）、音頻質量（PQ、CU）、音視頻同步性（Sync-C、Sync-D、DeSync）以及提示詞遵循能力（P-Acc）等多個維度。

實驗結果表明，在以簡單場景為主的 Verse-Bench 上，Baton 與當前領先開源模型 LTX-2 整體表現接近；而在更具挑戰性的 Sem100 上，Baton 展現出明顯優勢。

相比 LTX-2，Baton 的提示詞遵循準確率（P-Acc）提升 32%，多說話人詞錯誤率（M-WER）提升 76%，音畫不同步指標（DeSync）提升 30%。

其中，M-WER 的提升尤為突出。多說話人場景不僅要求模型理解說話內容，更要求準確判斷「誰在什么時候說了什么」。這一能力恰恰依賴于 Baton 所構建的細粒度時序語義規劃，而傳統全局文本嵌入難以提供這樣的時間對齊信息。這也進一步驗證了顯式語義規劃對于復雜指令生成的重要性。

此外，團隊還將 Baton 與多款閉源商業模型進行了對比。盡管在視覺質量和音頻美感方面，Baton 與頂級商業系統仍存在一定差距，但在復雜指令遵循能力上已經展現出較強競爭力。

生成結果展示

視頻鏈接：https://mp.weixin.qq.com/s/UobG7nWamiWMt45L62tCWA

Video Prompt:On a vast barren beach under a pale overcast sky with haze obscuring the flat horizon, a young man with dark messy hair lies face down on the sand, wearing a thick brown hooded wool coat, sand clinging to his clothes and skin. He props himself on his elbows, looking forward. In the far background, a column of sand explodes upward among silhouetted soldiers. The young man flinches in terror and clutches his head while successive blasts draw closer, sending towering columns of sand into the air. A harsh, aggressive soundscape with deep rumbles and piercing screeches builds as the bombardment rapidly approaches his position. Close-up, low angle, rule-of-thirds composition. Natural diffused overcast daylight with soft shadows. Desaturated, monochromatic palette of beige, tan, and muted olive green with a cool color temperature. Gritty realism, tense tone.

Audio Prompt:On a windswept open beach, continuous artillery explosions rumble and crash, growing progressively louder and closer. The blasts intensify from distant muffled thuds into deafening concussive roars, each one shaking the ground harder than the last. Sand and debris scatter and rain down with increasing violence. Beneath the relentless bombardment, rapid shallow breathing and a choked gasp of terror from a soldier [Speaker A] are barely audible.

視頻鏈接：https://mp.weixin.qq.com/s/UobG7nWamiWMt45L62tCWA

Video Prompt:In a indoor martial arts gym with yellow padded bars along the wall, and cool fluorescent overhead lighting, two bald men of Middle Eastern descent stand facing each other. The first, with a short beard, wears a black chest protector over a white t-shirt and remains stationary. The second, with a darker skin tone in a dark polo shirt, stands to his right as the instructor. The instructor delivers a quick punch to the first man's upper body. The camera shifts focus to the first man, who absorbs the hit. The instructor asks him a brief question; the first man nods and responds. The instructor then resumes speaking, using hand gestures to continue his explanation as the instructor looks toward the camera. Medium shot, eye-level, rule-of-thirds composition. Cool-toned overhead lighting with high contrast between brightly lit faces and deep surrounding shadows. Neutral palette of black, grey, and white accented by yellow padded bars and cool blue ambient light. Observational documentary style, focused tone.

Audio Prompt:In a gym with faint ambient echo and a low-level room tone, a mature man [Speaker A] speaks in a steady, instructional tone: \"Think about the idea of short distance power. If someone like this suddenly tries to headbutt me, one, I can hit him." A sharp, percussive thud of a fist striking a padded chest protector rings out immediately after. A brief pause, then [Speaker A] asks in a calm, slightly concerned tone: \"You okay?" Another man [Speaker B] replies affirmatively: \"Yeah." [Speaker A] continues in the same steady, instructional tone: \"And he's not headbutting."

視頻鏈接：https://mp.weixin.qq.com/s/UobG7nWamiWMt45L62tCWA

Video Prompt:At dusk in a desolate clearing beside a rustic log cabin with a thatched roof, shuttered windows, and barrels leaning against its side, under bare trees and a dim overcast sky, a bearded white man with short dark hair, wearing a loose brown shirt and matching pants, squats before a small crackling campfire within a stone ring on dry grass. To his right stands a slender white teenage boy with curly dark hair in an off-white Henley shirt and light trousers, holding a crumpled grey cloth. The man rises from his squat, reaches out with both hands, and takes the cloth from the boy. He turns toward the fire, steps forward, and bends down to drape the cloth over the burning logs. Smoke rises. The boy stands still, watching intently. The man then picks up a long wooden poker from the ground and pokes the smoldering bundle beneath the cloth.

Audio Prompt:A quiet outdoor dusk atmosphere with faint wind rustling dry grass. A small campfire crackles and pops within a stone ring. The soft rustle of cloth being handled. The crackling intensifies briefly as the cloth is draped over the fire, then dulls into a low, muffled smolder. A wooden poker scrapes against stone and prods the embers, stirring hissing smoke.

視頻鏈接：https://mp.weixin.qq.com/s/UobG7nWamiWMt45L62tCWA

Video Prompt:In a dimly lit interior, a close-up shows hands using a knife and fork to slice through a medium-rare steak on a white square plate. After cutting off a piece, the camera tilts upward to reveal a Caucasian woman with long wavy reddish-blonde hair, blue eyes, and fair skin, wearing dark clothing over a collared shirt. She lifts the piece of steak with the fork, brings it to her mouth, and chews slowly. After swallowing, her gaze lowers briefly, then she raises her head and stares intensely forward. Cinematic realism.

Audio Prompt:A knife sawing through steak with a soft, wet slicing sound against the plate. A fork scrapes briefly. Quiet, slow chewing follows. After a pause, a single melancholic piano note rings out.

視頻鏈接：https://mp.weixin.qq.com/s/UobG7nWamiWMt45L62tCWA

Video Prompt:Inside an old car, a girl wearing a grey-white t-shirt first looks down, then smiles slightly while steering along a rural road. A small figurine sits on the dashboard. The camera then pans left to reveal a passenger wearing a colorful wrestling mask.

Audio Prompt:A dramatic orchestral score with sweeping strings. The music is layered with the sounds of a vehicle engine starting and revving. A dog barks repeatedly in the background, its voice echoing slightly as if in an open space. A boy [Speaker A] shouts: \”Ah"

視頻鏈接：https://mp.weixin.qq.com/s/UobG7nWamiWMt45L62tCWA

Video Prompt:On a sunny suburban backyard with green lawn and tall hedges, a woman in a ribbed sweater and black skirt rallies a shuttlecock with a boy across a badminton net. He jogs off-screen to fetch it; she turns toward camera, striding forward with arms raised in playful triumph. A second boy charges in from behind, tackling her into wrestling on the grass — she lifts him, spins him around, both laughing joyously. Handheld tracking shot follows the action, shifting from eye-level to low-angle during the lift.

Audio Prompt:A fast-paced electronic dance music track with a driving beat and synthesized melodies plays throughout the clip. A boy [Speaker A] shouts excitedly in a energetic voice: \"Oh no! Ten points! I'm scared! She's the winner!" A girl [Speaker B] shouts back with equal excitement: \"We're the winners!"

視頻鏈接：https://mp.weixin.qq.com/s/UobG7nWamiWMt45L62tCWA

Video Prompt:On a residential street corner, a young Asian boy in bright blue shorts stands holding a brown Spalding basketball in one hand and a yellow-orange ball in the other. The camera slowly orbits from behind him onto a concrete patio beside a dense green hedge. He drops into a low stance and begins simultaneously dribbling both balls side by side, bouncing them in rhythm as he moves forward step by step along the road. Medium close-up widening to a tracking shot, eye-level shifting to slightly high-angle.

Audio Prompt:Set in an outdoor environment, a young boy [Speaker A] speaks in a clear, instructional tone: \"This is two ball basketball drill.". Immediately after he finishes speaking, the rhythmic, percussive sound of a basketball being dribbled on a hard surface begins and continues for the rest of the clip.

視頻鏈接：https://mp.weixin.qq.com/s/UobG7nWamiWMt45L62tCWA

Video Prompt:A young Caucasian man stands at an outdoor shooting range, holding a scoped AR-15 rifle, he fires several shots at a nearby pine tree, then reloads.

Audio Prompt:In a quiet, open outdoor environment, a sharp gunshot rings out, followed by a male voice [Speaker A] saying \"Ah\" in a neutral tone. Immediately after, another gunshot is fired. After a brief pause, a mechanical click is heard, as if a weapon is being reloaded.

視頻鏈接：https://mp.weixin.qq.com/s/UobG7nWamiWMt45L62tCWA

Video Prompt:On a sunlit outdoor asphalt basketball court, bordered by dense green trees under a clear blue sky, a young man with short brown hair wearing dark sunglasses, a grey baseball cap, a black t-shirt and black Nike athletic shorts. He picks up a red basketball. He stands, turns away from the camera, and walks along the baseline, dribbling the ball between his legs. As he nears the free-throw line he takes a jump shot; the ball arcs over the rim and drops through the net. Medium-to-tracking shot, eye-level, with leading room guiding the viewer toward the basket. Natural high-key golden-hour daylight casts soft shadows across the grey court. Observational tone capturing the action in real time.

Audio Prompt:Set in an outdoor environment with birdsong and faint rustling sounds, a young man [Speaker A] speaks in a calm, encouraging tone: \"Easy peasy, baby.\" The sound of a ball being dribbled on a hard surface is heard, followed by a sharp impact as it hits a backboard or wall. The dribbling resumes, accompanied by the soft thud of the ball bouncing on the ground.