Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Baichuan Inc.（百川智能）
Zijia Zhao, Haoyu Lu, Yuqi Huo, Yifan Du, Tongtian Yue, Longteng Guo, Bingning Wang, Weipeng Chen, Jing Liu
视频理解是多模态大语言模型（MLLMs）的关键下一步。为了更好地评估MLLMs，人们引入了各种基准。然而，由于构建数据集的高成本以及难以隔离特定技能，当前的视频基准在迭代开发过程中评估视频模型时仍然效率低下。在本文中，我们提出了VideoNIAH（Video Needle In A Haystack），一个通过合成视频生成来构建基准的框架。VideoNIAH通过将不相关的视觉“针”插入原始视频中，将视频内容与其查询-响应对解耦。该框架使用预定义规则自动生成查询-响应对，从而最大限度地减少人工劳动。查询聚焦于视频理解的特定方面，使得评估更具技能针对性。视频内容与查询之间的分离还允许增加视频多样性，并在不同长度下进行评估。利用VideoNIAH，我们构建了一个视频基准VNBench，其中包括检索、排序和计数等任务，以评估视频理解的三个关键方面：时间感知、时间顺序排列和时空一致性。我们对专有模型和开源模型进行了全面评估，揭示了它们在不同任务上的视频理解能力存在显著差异。此外，我们对测试结果和模型配置进行了深入分析。基于这些发现，我们为改进视频MLLM训练提供了一些建议，为未来研究和模型开发提供了有价值的见解。代码和数据可在https://github.com/joez17/VideoNIAH获取。
英文摘要：
Video understanding is a crucial next step for multimodal large language models (MLLMs). Various benchmarks are introduced for better evaluating the MLLMs. Nevertheless, current video benchmarks are still inefficient for evaluating video models during iterative development due to the high cost of constructing datasets and the difficulty in isolating specific skills. In this paper, we propose VideoNIAH (Video Needle In A Haystack), a benchmark construction framework through synthetic video generation. VideoNIAH decouples video content from their query-responses by inserting unrelated visual 'needles' into original videos. The framework automates the generation of query-response pairs using predefined rules, minimizing manual labor. The queries focus on specific aspects of video understanding, enabling more skill-specific evaluations. The separation between video content and the queries also allow for increased video variety and evaluations across different lengths. Utilizing VideoNIAH, we compile a video benchmark VNBench, which includes tasks such as retrieval, ordering, and counting to evaluate three key aspects of video understanding: temporal perception, chronological ordering, and spatio-temporal coherence. We conduct a comprehensive evaluation of both proprietary and open-source models, uncovering significant differences in their video understanding capabilities across various tasks. Additionally, we perform an in-depth analysis of the test results and model configurations. Based on these findings, we provide some advice for improving video MLLM training, offering valuable insights to guide future research and model development. The code and data are available at https://github.com/joez17/VideoNIAH.
Fudan University（复旦大学）；OpenGVLab（开放视觉实验室）；Shanghai AI Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）；The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）；Nanjing University（南京大学）；SenseTime Research（商汤科技研究院）
The growing use of convolutional neural networks (CNN) for a broad range of visual tasks, including tasks involving fine details, raises the problem of applying such networks to a large field of view, since the amount of computations increases significantly with the number of pixels. To deal effectively with this difficulty, we develop and compare methods of using CNNs for the task of small target localization in natural images, given a limited "budget" of samples to form an image. Inspired in part by human vision, we develop and compare variable sampling schemes, with peak resolution at the c…
Barco Healthcare（巴可医疗）；BARCO N.V.（巴可公司）；University of Pennsylvania（宾夕法尼亚大学）
Within the framework of a virtual clinical trial for breast imaging, we aim to develop numerical observers that follow the same detection performance trends as those of a typical human observer. In our prior work, we showed that by including spatiotemporal contrast sensitivity function (stCSF) of human visual system (HVS) in a multi-slice channelized Hotelling observer (msCHO), we can correctly predict trends of a typical human observer performance with the viewing parameters of browsing speed, viewing distance and contrast. In this work we further improve our numerical observer by modeling co…
University of Amsterdam（阿姆斯特丹大学）；The University of Tokyo（东京大学）
Detecting tiny objects in a high-resolution video is challenging because the visual information is little and unreliable. Specifically, the challenge includes very low resolution of the objects, MPEG artifacts due to compression and a large searching area with many hard negatives. Tracking is equally difficult because of the unreliable appearance, and the unreliable motion estimation. Luckily, we found that by combining this two challenging tasks together, there will be mutual benefits. Following the idea, in this paper, we present a neural network model called the Recurrent Correlational Netw…
Transfer learning has been recently popularized as a data-efficient alternative to training models from scratch, in particular for computer vision tasks where it provides a remarkably solid baseline. The emergence of rich model repositories, such as TensorFlow Hub, enables the practitioners and researchers to unleash the potential of these models across a wide range of downstream tasks. As these repositories keep growing exponentially, efficiently selecting a good model for the task at hand becomes paramount. We provide a formalization of this problem through a familiar notion of regret and in…
The advent of the internet, followed shortly by the social media made it ubiquitous in consuming and sharing information between anyone with access to it. The evolution in the consumption of media driven by this change, led to the emergence of images as means to express oneself, convey information and convince others efficiently. With computer vision algorithms progressing radically over the last decade, it is become easier and easier to study at scale the role of images in the flow of information online. While the research questions and overall pipelines differ radically, almost all start wit…
Imperial College London（帝国理工学院）；Facebook AI Research（脸书人工智能研究院）；Radboud University Medical Center（拉德堡德大学医学中心）
In some important computer vision domains, such as medical or hyperspectral imaging, we care about the classification of tiny objects in large images. However, most Convolutional Neural Networks (CNNs) for image classification were developed using biased datasets that contain large objects, in mostly central image positions. To assess whether classical CNN architectures work well for tiny object classification we build a comprehensive testbed containing two datasets: one derived from MNIST digits and one from histopathology images. This testbed allows controlled experiments to stress-test CNN …
NeMo: Needle in a Montage for Video-Language Understanding
The Chinese University of Hong Kong（香港中文大学）；Phoenix TV（凤凰电视台）；Stanford University（斯坦福大学）；University of Liverpool（利物浦大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
