SEDS: Semantically Enhanced Dual-Stream Encoder for Sign Language Retrieval
University of Science and Technology of China(中国科学技术大学);Hefei Comprehensive National Science Center(合肥综合性国家科学中心);Merchants Union Consumer Finance Company Limited(招商局消费金融有限公司)
Longtao Jiang, Min Wang, Zecheng Li, Yao Fang, Wengang Zhou, Houqiang Li
Different from traditional video retrieval, sign language retrieval is more biased towards understanding the semantic information of human actions contained in video clips. Previous works typically only encode RGB videos to obtain high-level semantic features, resulting in local action details drowned in a large amount of visual information redundancy. Furthermore, existing RGB-based sign retrieval works suffer from the huge memory cost of dense visual data embedding in end-to-end training, and adopt offline RGB encoder instead, leading to suboptimal feature representation. To address these issues, we propose a novel sign language representation framework called Semantically Enhanced Dual-Stream Encoder (SEDS), which integrates Pose and RGB modalities to represent the local and global information of sign language videos. Specifically, the Pose encoder embeds the coordinates of keypoints corresponding to human joints, effectively capturing detailed action features. For better context-aware fusion of two video modalities, we propose a Cross Gloss Attention Fusion (CGAF) module to aggregate the adjacent clip features with similar semantic information from intra-modality and inter-modality. Moreover, a Pose-RGB Fine-grained Matching Objective is developed to enhance the aggregated fusion feature by contextual matching of fine-grained dual-stream features. Besides the offline RGB encoder, the whole framework only contains learnable lightweight networks, which can be trained end-to-end. Extensive experiments demonstrate that our framework significantly outperforms state-of-the-art methods on various datasets.
OASIS: On-Demand Hierarchical Event Memory for Streaming Video Reasoning
Sun Yat-sen University(中山大学);OPPO AI Center(OPPO人工智能中心);Shenzhen Loop Area Institute(深圳河套学院);Guangdong Key Laboratory of Big Data Analysis and Processing(广东大数据分析与处理重点实验室)
SR-Clustering: Semantic Regularized Clustering for Egocentric Photo Streams Segmentation
Universitat de Barcelona(巴塞罗那大学);Computer Vision Center(计算机视觉中心);University of Groningen(格罗宁根大学);Imagga Technologies Ltd(Imagga技术有限公司);Digital Spaces Living Lab(数字空间生活实验室)
Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)(中国科学院智能信息处理重点实验室);Key Laboratory of AI Safety, Chinese Academy of Sciences(中国科学院人工智能安全重点实验室);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学)
Response-G1: Explicit Scene Graph Modeling for Proactive Streaming Video Understanding
Northwestern Polytechnical University(北华大学);Tsinghua University(清华大学);The Hong Kong University of Science and Technology(香港科技大学);Harbin Engineering University(哈尔滨工程大学)
Visual-aware Attention Dual-stream Decoder for Video Captioning
Wuhan University of Technology(武汉理工大学);ZhongQianLiYuan Engineering Consulting Co., Ltd(中前立源工程咨询有限公司)
视频描述是一项具有挑战性的任务,它需要捕捉不同的视觉部分并将其描述成句子,因为它需要视觉和语言的连贯性。当前视频描述方法中的注意力机制学习为每个帧分配权重,从而动态地促进解码器。为了生成语义连贯的句子,本文提出了一种新的视觉感知注意力(VA)模型,它将时间序列帧的动态变化与前一时刻的单词连接起来,作为注意力机制的输入来提取序列特征。流行的方法在训练期间广泛使用教师强迫(TF)学习,其中下一个标记是以先前的地面实况标记为条件而生成的。先前生成的标记中的语义信息丢失。因此,我们设计了一种自强制(SF)流,它将前一个标记的概率分布中的语义信息作为输入,以增强当前标记。双流解码器(DD)架构将TF和SF流统一起来,生成句子,以促进两个流的注释字幕。同时,双流解码器的使用,缓解了曝光偏差问题,通过在微软视频描述(MSVD)语料库和MSR-Video to Text(MSR-VTT)数据集上的实验研究,验证了本文提出的视觉感知注意力双流解码器(VADD)的有效性。
Event-VStream: Event-Driven Real-Time Understanding for Long Video Streams
University of Houston(德克萨斯大学休斯顿分校);The University of Texas at Arlington(德克萨斯理工大学);Indiana University Bloomington(印第安纳大学布卢明顿分校);Temple University(特拉华大学)
Semantic-E2VID: a Semantic-Enriched Paradigm for Event-to-Video Reconstruction
organization= Department of Electrical;Electronic Engineering, The University of Hong Kong , city= Hong Kong , country= China;organization= School of Artificial Intelligence, Beijing University of Posts;organization= State Key Laboratory of Multimedia Information Processing;National Engineering Research Center of Visual Technology, School of Computer Science, Peking University , city= Beijing , postcode= 100871 , country= China
Contrastive Language-Image Learning with Augmented Textual Prompts for 3D/4D FER Using Vision-Language Model
Information & Computer Science Department, King Fahd University of Petroleum & Minerals(国王法赫德石油与矿物大学信息与计算机科学系);Center for Machine Vision and Signal Analysis, University of Oulu(奥卢大学机器视觉与信号分析中心)
physfusion: A Transformer-based Dual-Stream Radar and Vision Fusion Framework for Open Water Surface Object Detection
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Dual-Stream Cross-Modal Representation Learning via Residual Semantic Decorrelation
School of Information Science & Engineering, Shandong Normal University(信息科学与工程学院,山东师范大学);Tajikistan State University of Law, Business Sughd(塔吉克斯坦法律、商业大学,苏赫德);Tajik State University of Law, Business and Politics Sughd(塔吉克斯坦法律、商业与政治大学,苏赫德)
Audio-Sync Video Generation with Multi-Stream Temporal Control
Beijing Academy of Artificial Intelligence(北京人工智能研究院);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院);Nat’l Key Lab of General AI, School of Intelligence Science and Technology, Peking University(国家通用人工智能实验室,北京大学智能科学与技术学院);Nat’l Eng. Research Ctr. of Visual Tech., School of Computer Science, Peking University(国家视觉技术工程研究中心,北京大学计算机学院)
本文提出 MTV 框架,通过分离语音、音效和音乐多流音频实现细粒度音视频同步控制,并构建 DEMIX 数据集,在六项指标上达到最先进性能。
Learning Enriched Features for Real Image Restoration and Enhancement
Inception Institute of Artificial Intelligence(inception人工智能研究院);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);University of California, Merced(加州大学默塞德分校);Google Research(谷歌研究院)
Event-Causal RAG: A Retrieval-Augmented Generation Framework for Long Video Reasoning in Complex Scenarios
Tianjin Key Lab of Intelligent Unmanned Swarm Tech & System(天津智能集群技术与系统重点实验室);Tianjin University(天津大学);Institute of Computing and Intelligence(计算与智能研究所);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Tianjin Artificial Intelligence Innovation Center(天津人工智能创新中心);Defense Innovation Institute Academy of Military Sciences(国防科技创新研究院);School of Future Technology(未来技术学院);Shanghai University(上海大学)
Learning Enriched Features for Fast Image Restoration and Enhancement
Inception Institute of Artificial Intelligence(起源人工智能研究院);Mohammed Bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);University of California at Merced(加州大学默塞德分校);Google(谷歌);Terminus Group(特斯联集团)