时间接地语言生成：迈向实时视觉-语言模型
Temporally-Grounded Language Generation: Towards Real-Time Vision-Language Models
AI总结：
针对实时交互环境对视觉-语言模型的时间精确性要求，提出TGLG基准任务及TRACE指标，构建VLM-TSI模型，实验显示其性能优于强基线但仍有提升空间。
摘要：
视觉-语言模型（VLMs）在图像字幕、视频问答等离线任务中已取得显著进展。然而，实时交互环境对VLMs提出了新要求，需生成语义准确且时间精确的语句。我们明确了此类场景所需的两项核心能力——感知更新与应急感知，并提出新基准任务：时间接地语言生成（TGLG）以评估这两项能力。TGLG要求模型响应流式视频生成语句，使内容与时间均匹配动态视觉输入。为支撑该基准，我们从体育转播和人类交互的第一视角领域整理了评估数据集，并引入新指标TRACE，通过联合测量语义相似度与时间对齐度来评估TGLG。最后，我们提出视觉-语言时间同步交错模型（VLM-TSI），该模型以时间同步方式交错视觉与语言令牌，无需依赖回合制假设即可实现实时语言生成。实验结果表明，VLM-TSI显著优于强基线模型，但整体表现仍有限，凸显TGLG任务的难度，为实时VLMs的进一步研究提供了动力。我们的代码可在this https URL获取。
英文摘要：
Vision-language models (VLMs) have shown remarkable progress in offline tasks such as image captioning and video question answering. However, real-time interactive environments impose new demands on VLMs, requiring them to generate utterances that are not only semantically accurate but also temporally precise. We identify two core capabilities necessary for such settings---\textit{perceptual updating} and \textit{contingency awareness}---and propose a new benchmark task, \textbf{Temporally-Grounded Language Generation (TGLG)}, to evaluate them. TGLG requires models to generate utterances in response to streaming video such that both content and timing align with dynamic visual input. To support this benchmark, we curate evaluation datasets from sports broadcasting and egocentric human interaction domains, and introduce a new metric, \textbf{TRACE}, to evaluate TGLG by jointly measuring semantic similarity and temporal alignment. Finally, we present \textbf{Vision-Language Model with Time-Synchronized Interleaving (VLM-TSI)}, a model that interleaves visual and linguistic tokens in a time-synchronized manner, enabling real-time language generation without relying on turn-based assumptions. Experimental results show that VLM-TSI significantly outperforms a strong baseline, yet overall performance remains modest---highlighting the difficulty of TGLG and motivating further research in real-time VLMs. Our code is available at https://github.com/yukw777/tglg.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
