Feature extraction is a critical component of many applied data science workflows. In recent years, rapid advances in artificial intelligence and machine learning have led to an explosion of feature extraction tools and services that allow data scientists to cheaply and effectively annotate their data along a vast array of dimensions---ranging from detecting faces in images to analyzing the sentiment expressed in coherent text. Unfortunately, the proliferation of powerful feature extraction services has been mirrored by a corresponding expansion in the number of distinct interfaces to feature extraction services. In a world where nearly every new service has its own API, documentation, and/or client library, data scientists who need to combine diverse features obtained from multiple sources are often forced to write and maintain ever more elaborate feature extraction pipelines. To address this challenge, we introduce a new open-source framework for comprehensive multimodal feature extraction. Pliers is an open-source Python package that supports standardized annotation of diverse data types (video, images, audio, and text), and is expressly with both ease-of-use and extensibility in mind. Users can apply a wide range of pre-existing feature extraction tools to their data in just a few lines of Python code, and can also easily add their own custom extractors by writing modular classes. A graph-based API enables rapid development of complex feature extraction pipelines that output results in a single, standardized format. We describe the package's architecture, detail its major advantages over previous feature extraction toolboxes, and use a sample application to a large functional MRI dataset to illustrate how pliers can significantly reduce the time and effort required to construct sophisticated feature extraction workflows while increasing code clarity and maintainability.
Nowadays, metadata information is often given by the authors themselves upon submission. However, a significant part of already existing research papers have missing or incomplete metadata information. German scientific papers come in a large variety of layouts which makes the extraction of metadata a non-trivial task that requires a precise way to classify the metadata extracted from the documents. In this paper, we propose a multimodal deep learning approach for metadata extraction from scientific papers in the German language. We consider multiple types of input data by combining natural la…
Visually rich documents (VRDs) are ubiquitous in daily business and life. Examples are purchase receipts, insurance policy documents, custom declaration forms and so on. In VRDs, visual and layout information is critical for document understanding, and texts in such documents cannot be serialized into the one-dimensional sequence without losing information. Classic information extraction models such as BiLSTM-CRF typically operate on text sequences and do not incorporate visual features. In this paper, we introduce a graph convolution based model to combine textual and visual information prese…
We introduce a novel approach for scanned document representation to perform field extraction. It allows the simultaneous encoding of the textual, visual and layout information in a 3-axis tensor used as an input to a segmentation model. We improve the recent Chargrid and Wordgrid \cite{chargrid} models in several ways, first by taking into account the visual modality, then by boosting its robustness in regards to small datasets while keeping the inference time low. Our approach is tested on public and private document-image datasets, showing higher performances compared to the recent state-of…
Multimodal named entity recognition and relation extraction (MNER and MRE) is a fundamental and crucial branch in information extraction. However, existing approaches for MNER and MRE usually suffer from error sensitivity when irrelevant object images incorporated in texts. To deal with these issues, we propose a novel Hierarchical Visual Prefix fusion NeTwork (HVPNeT) for visual-enhanced entity and relation extraction, aiming to achieve more effective and robust performance. Specifically, we regard visual representation as pluggable visual prefix to guide the textual representation for error …
Zhejiang University（浙江大学）；AZFT Joint Lab for Knowledge Engine（AZFT知识引擎联合实验室）；Hangzhou Innovation Center, Zhejiang University（浙江大学杭州国际科创中心）；Alibaba Group（阿里巴巴集团）
Multimodal relation extraction is an essential task for knowledge graph construction. In this paper, we take an in-depth empirical analysis that indicates the inaccurate information in the visual scene graph leads to poor modal alignment weights, further degrading performance. Moreover, the visual shuffle experiments illustrate that the current approaches may not take full advantage of visual information. Based on the above observation, we further propose a strong baseline with an implicit fine-grained multimodal alignment based on Transformer for multimodal relation extraction. Experimental r…
M-MRE: Extending the Mutual Reinforcement Effect to Multimodal Information Extraction
Yokohama National University（Yokohama国立大学）；Monash University（墨尔本大学）；Southern University of Science and Technology（南方科技大学）；Hong Kong University of Science and Technology（香港科学大学）；University of Chinese Academy of Sciences（中国科学院大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Université de Montréal（蒙特利尔大学）；University of Adelaide（阿德莱德大学）；Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）
Knowledge-based visual question answering requires the ability of associating external knowledge for open-ended cross-modal scene understanding. One limitation of existing solutions is that they capture relevant knowledge from text-only knowledge bases, which merely contain facts expressed by first-order predicates or language descriptions while lacking complex but indispensable multimodal knowledge for visual understanding. How to construct vision-relevant and explainable multimodal knowledge for the VQA scenario has been less studied. In this paper, we propose MuKEA to represent multimodal k…
Memes are one of the most ubiquitous forms of social media communication. The study and processing of memes, which are intrinsically multimedia, is a popular topic right now. The study presented in this research is based on the Memotion dataset, which involves categorising memes based on irony, comedy, motivation, and overall-sentiment. Three separate innovative transformer-based techniques have been developed, and their outcomes have been thoroughly reviewed.The best algorithm achieved a macro F1 score of 0.633 for humour classification, 0.55 for motivation classification, 0.61 for sarcasm cl…
SAVER: Selective As-Needed Vision Evidence for Multimodal Information Extraction
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
DongbaMIE: A Multimodal Information Extraction Dataset for Evaluating Semantic Understanding of Dongba Pictograms
College of Information and Engineering, Minzu University of China（中国民族大学信息与工程学院）；Key Laboratory of Ethnic Language Intelligent Analysis and Security Governance of MOE, Minzu University of China（教育部民族语言智能分析与安全治理重点实验室）；College of Information and Communication Engineering, Harbin Engineering University（哈尔滨工程大学信息与通信工程学院）；Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University（清华大学人工智能研究院计算机科学与技术系）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）
Columbia University（哥伦比亚大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Visual and textual modalities contribute complementary information about events described in multimedia documents. Videos contain rich dynamics and detailed unfoldings of events, while text describes more high-level and abstract concepts. However, existing event extraction methods either do not handle video or solely target video while ignoring other modalities. In contrast, we propose the first approach to jointly extract events from video and text articles. We introduce the new task of Video MultiMedia Event Extraction (Video M2E2) and propose two novel components to build the first system t…
University of Science and Technology of China（中国科学技术大学）；iFLYTEK Research（讯飞研究院）
Table of contents (ToC) extraction aims to extract headings of different levels in documents to better understand the outline of the contents, which can be widely used for document understanding and information retrieval. Existing works often use hand-crafted features and predefined rule-based functions to detect headings and resolve the hierarchical relationship between headings. Both the benchmark and research based on deep learning are still limited. Accordingly, in this paper, we first introduce a standard dataset, HierDoc, including image samples from 650 documents of scientific papers wi…
In this paper, we present our solutions for the Multimodal Sentiment Analysis Challenge (MuSe) 2022, which includes MuSe-Humor, MuSe-Reaction and MuSe-Stress Sub-challenges. The MuSe 2022 focuses on humor detection, emotional reactions and multimodal emotional stress utilizing different modalities and data sets. In our work, different kinds of multimodal features are extracted, including acoustic, visual, text and biological features. These features are fused by TEMMA and GRU with self-attention mechanism frameworks. In this paper, 1) several new audio features, facial expression features and …
UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documents
School of Computer Science and Engineering, Northeastern University, Shenyang, China（东北大学计算机科学与工程学院）；Department of Computer Science and Technology, Tsinghua University, Beijing, China（清华大学计算机科学与技术系）；Alibaba Group, Hangzhou, China（阿里巴巴集团）
YH-MINER: Multimodal Intelligent System for Natural Ecological Reef Metric Extraction
South China Sea Institute of Oceanology, Chinese Academy of Sciences（中国科学院南海海洋研究所）；National Astronomical Observatories of the Chinese Academy of Sciences（中国科学院国家天文台）；University of Chinese Academy of Sciences（中国科学院大学）；International Department of Affiliated High School of South China Normal University（华南师范大学附属中学国际部）；Alibaba Cloud（阿里云）
LMOD+: A Comprehensive Multimodal Dataset and Benchmark for Developing and Evaluating Multimodal Large Language Models in Ophthalmology
School of Medicine, Yale University（耶鲁大学医学院）；School of Computing, Australian National University（澳大利亚国立大学计算机学院）；School of Engineering, Imperial College London（伦敦帝国理工学院工程学院）；School of Computing, University of Georgia（佐治亚大学计算机学院）；Yong Loo Lin School of Medicine, National University of Singapore（新加坡国立大学杨秀隆医学学院）；National Eye Institute, National Institutes of Health（美国国立卫生研究院眼科研究所）；National Library of Medicine, National Institutes of Health（美国国立卫生研究院国家医学图书馆）；School of Computing Technologies, RMIT University（皇家墨尔本理工大学计算机技术学院）
In this paper, we present a novel Image Fusion Model (IFM) for ECG heart-beat classification to overcome the weaknesses of existing machine learning techniques that rely either on manual feature extraction or direct utilization of 1D raw ECG signal. At the input of IFM, we first convert the heart beats of ECG into three different images using Gramian Angular Field (GAF), Recurrence Plot (RP) and Markov Transition Field (MTF) and then fuse these images to create a single imaging modality. We use AlexNet for feature extraction and classification and thus employ end to end deep learning. We perfo…
VectorLLM: Human-like Extraction of Structured Building Contours vis Multimodal LLMs
School of Remote Sensing and Information Engineering（遥感与信息工程学院）；College of Oceanography and Space Informatics（海洋学与空间信息学院）；School of Surveying and Geoinformation Engineering（测绘与地理信息工程学院）
Shanghai University（上海大学）；University of Leicester（莱斯特大学）
Multimodal sentiment analysis has a wide range of applications due to its information complementarity in multimodal interactions. Previous works focus more on investigating efficient joint representations, but they rarely consider the insufficient unimodal features extraction and data redundancy of multimodal fusion. In this paper, a Video-based Cross-modal Auxiliary Network (VCAN) is proposed, which is comprised of an audio features map module and a cross-modal selection module. The first module is designed to substantially increase feature diversity in audio feature extraction, aiming to imp…
MDD-Net: Multimodal Depression Detection through Mutual Transformer
Centre for Pattern Analysis and Machine Intelligence, Department of Electrical and Computer Engineering, University of Waterloo（模式分析与机器智能中心，电气与计算机工程系，滑铁卢大学）；School of Engineering and Computing, Department of Computer Science and Engineering, American University of Ras Al Khaimah（工程与计算学院，计算机科学与工程系，阿联酋拉线哈姆市美国大学）
Reliable Object Tracking by Multimodal Hybrid Feature Extraction and Transformer-Based Fusion
Clinical Hospital of Chengdu Brain Science Institute（成都脑科学研究院临床医院）；School of Life Science and Technology, University of Electronic Science and Technology of China（电子科技大学生命科学与技术学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Sichuan Academy of Medical Sciences and Sichuan Provincial People’s Hospital（四川省医学科学院四川省人民医院）
Beijing University of Posts and Telecommunications（北京邮电大学）；University of Technology Sydney（悉尼科技大学）；Australian AI Institute（澳大利亚人工智能研究所）
For multimodal tasks, a good feature extraction network should extract information as much as possible and ensure that the extracted feature embedding and other modal feature embedding have an excellent mutual understanding. The latter is often more critical in feature fusion than the former. Therefore, selecting the optimal feature extraction network collocation is a very important subproblem in multimodal tasks. Most of the existing studies ignore this problem or adopt an ergodic approach. This problem is modeled as an optimization problem in this paper. A novel method is proposed to convert…
Balancing Multimodal Training Through Game-Theoretic Regularization
Department of Electrical Engineering, KU Leuven（KU Leuven 电子工程系）；Department of Development and Regeneration, KU Leuven（KU Leuven 发展与再生系）；Media Lab and EECS, MIT（MIT 媒体实验室和电子工程与计算机科学系）
Singapore University of Technology and Design（新加坡科技设计大学）；Institute for Infocomm Research（新加坡资讯通信研究院）
With the increasing availability of wearable devices, research on egocentric activity recognition has received much attention recently. In this paper, we build a Multimodal Egocentric Activity dataset which includes egocentric videos and sensor data of 20 fine-grained and diverse activity categories. We present a novel strategy to extract temporal trajectory-like features from sensor data. We propose to apply the Fisher Kernel framework to fuse video and temporal enhanced sensor features. Experiment results show that with careful design of feature extraction and fusion algorithm, sensor data c…
To improve users' experience as they navigate the myriad of options offered by online marketplaces, it is essential to have well-organized product catalogs. One key ingredient to that is the availability of product attributes such as color or material. However, on some marketplaces such as Rakuten-Ichiba, which we focus on, attribute information is often incomplete or even missing. One promising solution to this problem is to rely on deep models pre-trained on large corpora to predict attributes from unstructured data, such as product descriptive texts and images (referred to as modalities in …
Anhui University（安徽大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
The continuous improvement of human-computer interaction technology makes it possible to compute emotions. In this paper, we introduce our submission to the CVPR 2023 Competition on Affective Behavior Analysis in-the-wild (ABAW). Sentiment analysis in human-computer interaction should, as far as possible Start with multiple dimensions, fill in the single imperfect emotion channel, and finally determine the emotion tendency by fitting multiple results. Therefore, We exploited multimodal features extracted from video of different lengths from the competition dataset, including audio, pose and im…
