StreamSense: 基于选择性视觉-语言模型路由的流式社交任务检测
StreamSense: Streaming Social Task Detection with Selective Vision-Language Model Routing
AI总结：
StreamSense通过轻量级编码器与选择性路由结合VLM专家，提升流式社交任务检测的准确性和效率。
摘要：
实时流媒体平台需要实时监控和响应社会信号，利用视频、文本和音频中的部分异步证据。我们提出了StreamSense，一种结合轻量级流式编码器和选择性路由的流式检测器，与视觉-语言模型（VLM）专家耦合。StreamSense使用轻量级流式编码器处理大多数时间戳，将困难/模糊的情况升级到VLM，并在上下文不足时推迟决策。编码器通过（i）交叉模态对比项对齐视觉/音频线索与文本信号，以及（ii）IoU加权损失，该损失通过降低重叠目标段的权重来减轻段边界之间的标签干扰。我们在多个社交流式检测任务（如情感分类和仇恨内容审核）上评估了StreamSense，结果表明，StreamSense在仅偶尔调用VLM的情况下，比仅使用VLM的流式检测器具有更高的准确性，从而减少了平均延迟和计算成本。我们的结果表明，选择性升级和延迟是理解流式社交任务的有效原始构件。代码已在GitHub上公开。
英文摘要：
Live streaming platforms require real-time monitoring and reaction to social signals, utilizing partial and asynchronous evidence from video, text, and audio. We propose StreamSense, a streaming detector that couples a lightweight streaming encoder with selective routing to a Vision-Language Model (VLM) expert. StreamSense handles most timestamps with the lightweight streaming encoder, escalates hard/ambiguous cases to the VLM, and defers decisions when context is insufficient. The encoder is trained using (i) a cross-modal contrastive term to align visual/audio cues with textual signals, and (ii) an IoU-weighted loss that down-weights poorly overlapping target segments, mitigating label interference across segment boundaries. We evaluate StreamSense on multiple social streaming detection tasks (e.g., sentiment classification and hate content moderation), and the results show that StreamSense achieves higher accuracy than VLM-only streaming while only occasionally invoking the VLM, thereby reducing average latency and compute. Our results indicate that selective escalation and deferral are effective primitives for understanding streaming social tasks. Code is publicly available on GitHub.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
