Shots are key narrative elements of various videos, e.g. movies, TV series, and user-generated videos that are thriving over the Internet. The types of shots greatly influence how the underlying ideas, emotions, and messages are expressed. The technique to analyze shot types is important to the understanding of videos, which has seen increasing demand in real-world applications in this era. Classifying shot type is challenging due to the additional information required beyond the video content, such as the spatial composition of a frame and camera movement. To address these issues, we propose a learning framework Subject Guidance Network (SGNet) for shot type recognition. SGNet separates the subject and background of a shot into two streams, serving as separate guidance maps for scale and movement type classification respectively. To facilitate shot type analysis and model evaluations, we build a large-scale dataset MovieShots, which contains 46K shots from 7K movie trailers with annotations of their scale and movement types. Experiments show that our framework is able to recognize these two attributes of shot accurately, outperforming all the previous methods.
MILA（米拉计算学习研究所）；University of Montreal（蒙特利尔大学）；University of Amsterdam（阿姆斯特丹大学）；CU Boulder（科罗拉多大学博尔德分校）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Tübingen（蒂宾根大学）
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Recent work has suggested that a good embedding is all we need to solve many few-shot learning benchmarks. Furthermore, other work has strongly suggested that Model Agnostic Meta-Learning (MAML) also works via this same method - by learning a good embedding. These observations highlight our lack of understanding of what meta-learning algorithms are doing and when they work. In this work, we provide empirical results that shed some light on how meta-learned MAML representations function. In particular, we identify three interesting properties: 1) In contrast to previous work, we show that it is…
Visual relationship detection aims to capture interactions between pairs of objects in images. Relationships between objects and humans represent a particularly important subset of this problem, with implications for challenges such as understanding human behaviour, and identifying affordances, amongst others. In addressing this problem we first construct a large-scale human-centric visual relationship detection dataset (HCVRD), which provides many more types of relationship annotation (nearly 10K categories) than the previous released datasets. This large label space better reflects the reali…
I Seek You in Videos: Identity-Conditioned Queries for Person-Centric Video Reasoning
Beijing Jiaotong University（北京交通大学）；HUJING Digital Media & Entertainment Group（汇晶数字媒体与娱乐集团）；MAIS Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所MAIS）
Objects appear to scale differently in natural images. This fact requires methods dealing with object-centric tasks (e.g. object proposal) to have robust performance over variances in object scales. In the paper, we present a novel segment proposal framework, namely FastMask, which takes advantage of hierarchical features in deep convolutional neural networks to segment multi-scale objects in one shot. Innovatively, we adapt segment proposal network into three different functional components (body, neck and head). We further propose a weight-shared residual neck module as well as a scale-toler…
University of Sydney（悉尼大学）；Monash University（莫纳什大学）；University of Melbourne（墨尔本大学）；University of Technology Sydney（悉尼科技大学）
Few-shot learning is currently enjoying a considerable resurgence of interest, aided by the recent advance of deep learning. Contemporary approaches based on weight-generation scheme delivers a straightforward and flexible solution to the problem. However, they did not fully consider both the representation power for unseen categories and weight generation capacity in feature learning, making it a significant performance bottleneck. This paper proposes a multi-level weight-centric feature learning to give full play to feature extractor's dual roles in few-shot learning. Our proposed method con…
Current machine learning models achieve super-human performance in many real-world applications. Still, they are susceptible against imperceptible adversarial perturbations. The most effective solution for this problem is adversarial training that trains the model with adversarially perturbed samples instead of original ones. Various methods have been developed over recent years to improve adversarial training such as data augmentation or modifying training attacks. In this work, we examine the same problem from a new data-centric perspective. For this purpose, we first demonstrate that the ex…
Human Pose Descriptions and Subject-Focused Attention for Improved Zero-Shot Transfer in Human-Centric Classification Tasks
University of Kaiserslautern-Landau（凯泽斯劳滕-兰道大学）；Deutsches Forschungszentrum für Künstliche Intelligenz（德国人工智能研究中心）；ETH Zürich（苏黎世联邦理工学院）；Technical University of Munich（慕尼黑工业大学）
FUNCTO: Function-Centric One-Shot Imitation Learning for Tool Manipulation
Southern University of Science and Technology（南方科技大学）；National University of Singapore（新加坡国立大学）；Smart Systems Institute, National University of Singapore（新加坡国立大学智能系统研究院）
Geometric-Guided Few-Shot Dental Landmark Detection with Human-Centric Foundation Model
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China（电子与计算机工程系，香港科学与技术大学）；Division of Paediatric Dentistry and Orthodontics, Faculty of Dentistry, The University of Hong Kong, Hong Kong SAR, China（牙科学院儿童牙科与正畸学系，香港大学）
Towards Blind Lens Aberration Correction via Large LensLib Pre-training and Discrete Degradation Priors
National Research Center for Optical Instrumentation, Zhejiang University（浙江省光学仪器研究中心，浙江大学）；INSAIT, Sofia University "St. Kliment Ohridski"（INSAIT，索菲亚大学"圣克莱门特·欧弗里迪斯基"）；School of Artificial Intelligence and Robotics, Hunan University（人工智能与机器人学院，湖南大学）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University（机器人视觉感知与控制技术国家工程研究中心，湖南大学）
University of California, Berkeley（加州大学伯克利分校）；Orange Coast College（奥兰治海岸学院）
Compressive lensless imagers enable novel applications in an extremely compact device, requiring only a phase or amplitude mask placed close to the sensor. They have been demonstrated for 2D and 3D microscopy, single-shot video, and single-shot hyperspectral imaging; in each of these cases, a compressive-sensing-based inverse problem is solved in order to recover a 3D data-cube from a 2D measurement. Typically, this is accomplished using convex optimization and hand-picked priors. Alternatively, deep learning-based reconstruction methods offer the promise of better priors, but require many tho…
We propose a method to learn image representations from uncurated videos. We combine a supervised loss from off-the-shelf object detectors and self-supervised losses which naturally arise from the video-shot-frame-object hierarchy present in each video. We report competitive results on 19 transfer learning tasks of the Visual Task Adaptation Benchmark (VTAB), and on 8 out-of-distribution-generalization tasks, and discuss the benefits and shortcomings of the proposed approach. In particular, it improves over the baseline on all 18/19 few-shot learning tasks and 8/8 out-of-distribution generaliz…
Optimizing Latent Graph Representations of Surgical Scenes for Zero-Shot Domain Transfer
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；West China Hospital of Sichuan University（四川大学华西医院）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；Fondazione Policlinico Universitario A. Gemelli IRCCS（Gemelli IRCCS大学综合医院基金会）
Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning
Hacettepe University（哈杰泰佩大学）；Aselsan Research（阿塞尔桑研究院）；Koç University（科克大学）；Amsterdam UMC, University of Amsterdam（阿姆斯特丹大学医学中心，阿姆斯特丹大学）；Amsterdam Public Health（阿姆斯特丹公共卫生研究院）
The objective of this work is to learn an object-centric video representation, with the aim of improving transferability to novel tasks, i.e., tasks different from the pre-training task of action classification. To this end, we introduce a new object-centric video recognition model based on a transformer architecture. The model learns a set of object-centric summary vectors for the video, and uses these vectors to fuse the visual and spatio-temporal trajectory 'modalities' of the video clip. We also introduce a novel trajectory contrast loss to further enhance objectness in these summary vecto…
Detecting Children with Autism Spectrum Disorder based on Script-Centric Behavior Understanding with Emotional Enhancement
School of Computer Science, Wuhan University（武汉大学计算机学院）；Digital Innovation Research Center, Duke Kunshan University（杜克昆山大学数字创新研究中心）；School of Electronics and Information Technology, Sun Yat-sen University（中山大学电子与信息学院）；Child Development and Behavior Center, Third Affiliated Hospital of Sun Yat-sen University（中山大学第三附属医院儿童发展与行为中心）
