Video processing has become a popular research direction in computer vision due to its various applications such as video summarization, action recognition, etc. Recently, deep learning-based methods have achieved impressive results in action recognition. However, these methods need to process a full video sequence to recognize the action, even though most of these frames are similar and non-essential to recognizing a particular action. Additionally, these non-essential frames increase the computational cost and can confuse a method in action recognition. Instead, the important frames called keyframes not only are helpful in the recognition of an action but also can reduce the processing time of each video sequence for classification or in other applications, e.g. summarization. As well, current methods in video processing have not yet been demonstrated in an online fashion.
Motivated by the above, we propose an online learnable module for keyframe extraction. This module can be used to select key-shots in video and thus can be applied to video summarization. The extracted keyframes can be used as input to any deep learning-based classification model to recognize action. We also propose a plugin module to use the semantic word vector as input along with keyframes and a novel train/test strategy for the classification models. To our best knowledge, this is the first time such an online module and train/test strategy have been proposed.
The experimental results on many commonly used datasets in video summarization and in action recognition have shown impressive results using the proposed module.
Large Model based Sequential Keyframe Extraction for Video Summarization
China University of Geosciences（中国地质大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Tsinghua University（清华大学）；Beihang University（北京航空航天大学）
TriPSS: A Tri-Modal Keyframe Extraction Framework Using Perceptual, Structural, and Semantic Representations
Computer and Information Science, University of Arkansas - Little Rock（计算机与信息科学，亚拉荷马州立大学）；ICSI, University of California, Berkeley（ICSI，加州大学伯克利分校）；COSMOS Research Center, University of Arkansas - Little Rock（COSMOS研究中心，亚拉荷马州立大学）
线性判别分析（LDA）是一种众所周知的降维和分类方法。以前的研究也扩展到多类的二进制类的情况下。然而，许多应用，如目标检测和关键帧提取不能提供一致的实例-标签对，而LDA需要在实例级别上的标签用于训练。因此，它不能直接应用于半监督分类问题。在本文中，我们克服了这一限制，并提出了一个潜在的变量Fisher判别分析模型。我们放松的实例级标签到袋级，是一种半监督（视频级标签的事件类型需要语义帧提取），并结合了数据驱动的潜在变量的先验。因此，我们的方法结合了潜在变量的推理和降维在一个统一的baidu框架。我们在MUSK和Corel数据集上测试了我们的方法，并与基线方法相比产生了有竞争力的结果。我们还展示了其在具有挑战性的TRECVID MED 11数据集上进行语义关键帧提取的能力，并进行了基于人为因素排名的实验评估，这清楚地表明我们提出的方法始终提取比具有挑战性的基线更有语义意义的关键帧。
Robust RGB-T Tracking via Learnable Visual Fourier Prompt Fine-tuning and Modality Fusion Prompt Generation
Key Laboratory of Education Blockchain and Intelligent Technology, Ministry of Education, Guangxi Normal University（教育区块链与智能技术重点实验室，教育部，广西师范大学）；Guangxi Key Lab of Multi-Source Information Mining and Security, Guangxi Normal University（多源信息挖掘与安全广西重点实验室，广西师范大学）
Fusionista2.0: Efficiency Retrieval System for Large-Scale Datasets
Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI), UAE（Mohamed Bin Zayed人工智能大学（MBZUAI））；AISIA Research Lab（AISIA研究实验室）；University of Information Technology（信息技术大学）；Ho Chi Minh University of Science（胡志明科学大学）；Vietnam National University, Ho chi Minh City（越南国家大学，胡志明市）
A Tactical Behaviour Recognition Framework Based on Causal Multimodal Reasoning: A Study on Covert Audio-Video Analysis Combining GAN Structure Enhancement and Phonetic Accent Modelling
Dhurakij Pundit University（易三仓大学）；The University of Western Australia（西澳大利亚大学）；Association for Computing Machinery（美国计算机协会）；Royal Anthropological Institute（英国皇家人类学学会）
