The emerging video LMMs (Large Multimodal Models) have achieved significant improvements on generic video understanding in the form of VQA (Visual Question Answering), where the raw videos are captured by cameras. However, a large portion of videos in real-world applications are edited videos, \textit{e.g.}, users usually cut and add effects/modifications to the raw video before publishing it on social media platforms. The edited videos usually have high view counts but they are not covered in existing benchmarks of video LMMs, \textit{i.e.}, ActivityNet-QA, or VideoChatGPT benchmark. In this paper, we leverage the edited videos on a popular short video platform, \textit{i.e.}, TikTok, and build a video VQA benchmark (named EditVid-QA) covering four typical editing categories, i.e., effect, funny, meme, and game. Funny and meme videos benchmark nuanced understanding and high-level reasoning, while effect and game evaluate the understanding capability of artificial design. Most of the open-source video LMMs perform poorly on the EditVid-QA benchmark, indicating a huge domain gap between edited short videos on social media and regular raw videos. To improve the generalization ability of LMMs, we collect a training set for the proposed benchmark based on both Panda-70M/WebVid raw videos and small-scale TikTok/CapCut edited videos, which boosts the performance on the proposed EditVid-QA benchmark, indicating the effectiveness of high-quality training data. We also identified a serious issue in the existing evaluation protocol using the GPT-3.5 judge, namely a "sorry" attack, where a sorry-style naive answer can achieve an extremely high rating from the GPT judge, e.g., over 4.3 for correctness score on VideoChatGPT evaluation protocol. To avoid the "sorry" attacks, we evaluate results with GPT-4 judge and keyword filtering. The dataset is released at https://github.com/XenonLamb/EditVid-QA.
Can VLMs Detect and Localize Fine-Grained AI-Edited Images?
Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Ant Group(蚂蚁集团);Tsinghua University(清华大学);Shandong University(山东大学);Wuhan University(武汉大学)
Uncovering Entity Identity Confusion in Multimodal Knowledge Editing
New Laboratory of Pattern Recognition (NLPR)(模式识别新实验室);State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS)(多模态人工智能系统国家重点实验室);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Beijing University of Posts and Telecommunications(北京邮电大学);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);Huazhong University of Science and Technology(华中科技大学);Shandong University(山东大学)
Multimedia Verification Through Multi-Agent Deep Research Multimodal Large Language Models
University of New Brunswick(新 Brunswick大学);Institute for Clarity in Documentation(文档清晰研究所);Inria Paris-Rocquencourt(巴黎-罗克琴特研究所);Rajiv Gandhi University(拉贾·甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕勒尔研究实验室)
Edited Media Understanding Frames: Reasoning About the Intent and Implications of Visual Misinformation
Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Michigan(密歇根大学);Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院);Stanford University(斯坦福大学)
Beyond Textual CoT: Interleaved Text-Image Chains with Deep Confidence Reasoning for Image Editing
Shanghai Jiaotong University(上海交通大学);Huawei Noah’s Ark Lab(华为诺亚实验室);East China Normal University(华东师范大学);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究院,中国科学院)
LMM4Edit: Benchmarking and Evaluating Multimodal Image Editing with LMMs
Institute of Image Communication and Network Engineering, Shanghai JiaoTong University(上海交通大学图像通信与网络工程研究所);University of Electronic and Science Technology of China(电子科技大学);Nanyang Technological University(南洋理工大学)
End2You -- The Imperial Toolkit for Multimodal Profiling by End-to-End Learning
Department of Computing(计算机系);Imperial College London(帝国理工学院);London, UK(伦敦(英国)——注意:根据提取规则,“伦敦,英国”是地点,不应作为机构,此处原数据可能有误,正确应为帝国理工学院的部分。但严格按原文,作者Björn W. Schuller的Affiliation写的是“London, UK”,但根据规则,需提取机构,若“London, UK”不是机构,则可能此处无对应机构。但根据前两位作者的Affiliation,Stefanos Zafeiriou的是“Imperial College London”(帝国理工学院),Panagiotis Tzirakis的是“Department of Computing”(计算机系,属于帝国理工学院)。Björn W. Schuller的“Affiliation: London, UK”可能是笔误,实际应为帝国理工学院。但按规则,仅提取明确的机构,所以正确应为:{)
MultiVENT-Raw: A Benchmark for Retrieval and Reasoning over Raw Videos
Human Language Technology Center of Excellence(人类语言技术卓越中心);Johns Hopkins University(约翰斯·霍普金斯大学);Georgetown University(乔治城大学);George Mason University(乔治梅森大学);University of Rochester(罗切斯特大学)
ReEdit: Multimodal Exemplar-Based Image Editing with Diffusion Models
Indian Institute of Technology, Roorkee(鲁尔基印度理工学院);Adobe MDSR;Microsoft Research(微软研究院);Indian Institute of Technology, Bombay(孟买印度理工学院);Stanford University(斯坦福大学);Carnegie Mellon University(卡内基梅隆大学)
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
Columbia University(哥伦比亚大学);Google(谷歌);Cornell University(康奈尔大学)
我们提出了一个框架,学习多模态表示从未标记的数据使用卷积自由Transformer架构。具体来说,我们的视频-音频-文本Transformer(VATT)将原始信号作为输入,并提取多模态表示,这些表示足够丰富,可以使各种下游任务受益。我们使用多模态对比损失从头开始训练VATT端到端,并通过视频动作识别、音频事件分类、图像分类和文本到视频检索等下游任务来评估其性能。此外,我们研究了一个模态不可知的,单骨干Transformer共享三个模态之间的权重。我们证明了无卷积VATT在下游任务中的性能优于最先进的基于ConvNet的架构。特别是,VATT的视觉Transformer在Kinetics-400上达到了82.1%的顶级准确率,在Kinetics-600上达到了83.6%,在Kinetics-700上达到了72.7%,在Moments in Time上达到了41.1%,这些都是新的记录,同时避免了监督式的预训练。转移到图像分类导致ImageNet上的top-1准确率为78.7%,而从头开始训练相同的Transformer的准确率为64.7%,这表明尽管视频和图像之间存在领域差距,但我们的模型具有普遍性。VATT的音频Transformer还在基于波形的音频事件识别方面创造了新纪录,在AudioSet上实现了39.4%的mAP,而无需任何监督的预训练。VATT的源代码是公开的。
Multimodal feature fusion for CNN-based gait recognition: an empirical comparison
University of Malaga(马拉加大学);University of Cordoba(科尔多瓦大学);University of Granada(格拉纳达大学);Department of Computer Architecture, University of Malaga, Spain(西班牙马拉加大学计算机架构系);Department of Computing and Numerical Analysis, University of Cordoba, Spain(西班牙科尔多瓦大学计算与数值分析系);Department of Computer Science and Artificial Intelligence, University of Granada, Spain(西班牙格拉纳达大学计算机科学与人工智能系)
MM-ContextFold: Context Folding for Multimodal Agentic Retrieval
Shandong University(山东大学);Southeast University(东南大学);Kuaishou(快手);Hong Kong University of Science and Technology(香港科技大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
Beyond Visual Safety: Jailbreaking Multimodal Large Language Models for Harmful Image Generation via Semantic-Agnostic Inputs
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications(网络与交换技术国家重点实验室,北京邮电大学);School of Cyberspace Security, Beijing University of Posts and Telecommunications(网络安全学院,北京邮电大学)
LumiVideo: An Intelligent Agentic System for Video Color Grading
Northwestern University(西北大学);Northeastern University(东北大学);South China University of Technology(华南理工大学);Hong Kong Baptist University(香港浸会大学);Beijing Normal - Hong Kong Baptist University(北京师范大学-香港浸会大学)