Towards Human-Like Machine Comprehension: Few-Shot Relational Learning in Visually-Rich Documents
School of Computer Engineering and Science, Shanghai University(上海大学计算机工程与科学学院);Graduate School of Informatics, Kyoto University(京都大学信息学研究科);Department of Computer Science and Engineering, Shanghai Jiao Tong University(上海交通大学计算机科学与工程系)
Key-value relations are prevalent in Visually-Rich Documents (VRDs), often depicted in distinct spatial regions accompanied by specific color and font styles. These non-textual cues serve as important indicators that greatly enhance human comprehension and acquisition of such relation triplets. However, current document AI approaches often fail to consider this valuable prior information related to visual and spatial features, resulting in suboptimal performance, particularly when dealing with limited examples. To address this limitation, our research focuses on few-shot relational learning, specifically targeting the extraction of key-value relation triplets in VRDs. Given the absence of a suitable dataset for this task, we introduce two new few-shot benchmarks built upon existing supervised benchmark datasets. Furthermore, we propose a variational approach that incorporates relational 2D-spatial priors and prototypical rectification techniques. This approach aims to generate relation representations that are more aware of the spatial context and unseen relation in a manner similar to human perception. Experimental results demonstrate the effectiveness of our proposed method by showcasing its ability to outperform existing methods. This study also opens up new possibilities for practical applications.
Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University(中国教育部多媒体可信感知与高效计算重点实验室,厦门大学);Nanjing University(南京大学);University of Rochester(罗切斯特大学)
Multimodal Graph RAG for Long-range Visually Rich Document Understanding
Department of Computer Science and Information Engineering, National Taiwan University(国家台湾大学计算机科学与信息工程系);FinTech Center, National Taiwan University(国家台湾大学金融科技中心)
The University of Sydney(悉尼大学);SenseTime Research(商汤科技研究院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Huazhong University of Science and Technology(华中科技大学);University of Science and Technology of China(中国科学技术大学)