Currently, vision encoder models like Vision Transformers (ViTs) typically excel at image recognition tasks but cannot simultaneously support text recognition like human visual recognition. To address this limitation, we propose UNIT, a novel training framework aimed at UNifying Image and Text recognition within a single model. Starting with a vision encoder pre-trained with image recognition tasks, UNIT introduces a lightweight language decoder for predicting text outputs and a lightweight vision decoder to prevent catastrophic forgetting of the original image encoding capabilities. The training process comprises two stages: intra-scale pretraining and inter-scale finetuning. During intra-scale pretraining, UNIT learns unified representations from multi-scale inputs, where images and documents are at their commonly used resolution, to enable fundamental recognition capability. In the inter-scale finetuning stage, the model introduces scale-exchanged data, featuring images and documents at resolutions different from the most commonly used ones, to enhance its scale robustness. Notably, UNIT retains the original vision encoder architecture, making it cost-free in terms of inference and deployment. Experiments across multiple benchmarks confirm that our method significantly outperforms existing methods on document-related tasks (e.g., OCR and DocQA) while maintaining the performances on natural images, demonstrating its ability to substantially enhance text recognition without compromising its core image recognition capabilities.
SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognition
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
CMFN: Cross-Modal Fusion Network for Irregular Scene Text Recognition
Intelligent Software Research Center, Institute of Software Chinese Academy of Sciences（中国科学院软件研究所智能软件研究中心）；University of Chinese Academy of Sciences（中国科学院大学）；The Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所复杂系统认知与决策智能实验室）；State Key Laboratory of Computer Science, Institute of Software Chinese Academy of Sciences（中国科学院软件研究所计算机科学国家重点实验室）
Multilingual Scene Character Recognition System using Sparse Auto-Encoder for Efficient Local Features Representation in Bag of Features
REsearch Groups in Intelligent Machines Laboratory（智能机器研究组实验室）；ENIS-Sfax（斯法克斯国立工程师学院）；Higher Institute of Computer Science and Communication Techniques（高等计算机科学与通信技术学院）；University of Sousse（苏塞大学）；Al Baha University（巴哈大学）；Laboratoire d’InfoRmatique en Images et Systmes d’information (LIRIS)（图像与信息系统信息学实验室（LIRIS））；INSA of Lyon（里昂国立应用科学学院）
Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition
The University of Sydney（悉尼大学）；SenseTime Computer Vision Research Group（商汤科技计算机视觉研究组）；University of Chinese Academy of Sciences（中国科学院大学）；CASIA（中国科学院自动化研究所）
GenRec: Unifying Video Generation and Recognition with Diffusion Models
Fudan University（复旦大学）；University of Maryland（马里兰大学）；Shanghai Key Lab of Intell. Info. Processing（上海智能信息处理重点实验室）；Shanghai Collaborative Innovation Center of Intelligent Visual Computing（上海智能视觉计算协同创新中心）
本文提出首个基于 Stable Video Diffusion 的统一框架 GenRec，通过随机帧条件化过程联合优化视频生成与识别，在多个数据集上取得优异性能，并在输入信息有限时表现出色。
Unifying Specialized Visual Encoders for Video Language Models
Department of Computer Science, Princeton University, Princeton, NJ, United States（普林斯顿大学计算机科学系）；Salesforce Research, Palo Alto, CA, United States（Salesforce研究）
LAVENDER: Unifying Video-Language Understanding as Masked Language Modeling
Microsoft（微软）
近年来，统一视觉语言框架有了很大的发展，其中大多数采用编码器-解码器架构，将图像-文本任务统一为序列到序列生成。然而，现有的视频语言（VidL）模型仍然需要在模型架构和每个任务的训练目标的任务特定的设计。在这项工作中，我们探索了一个统一的VidL框架LAVENDER，其中Masked Language Modeling（MLM）被用作所有预训练和下游任务的公共接口。这样的统一导致简化的模型架构，其中在多模态编码器之上仅需要轻量级MLM头，而不是具有更多参数的解码器。令人惊讶的是，实验结果表明，这个统一的框架实现了竞争力的性能在14个VidL基准，涵盖视频问答，文本到视频检索和视频描述。广泛的分析进一步证明了LAVENDER优于现有VidL方法的优点：（i）在多任务微调时仅使用一组参数值支持所有下游任务;（ii）对各种下游任务的Few-Shot泛化;以及（iii）对视频问答任务启用zero-shot评估。代码可在https://github.com/microsoft/LAVENDER上获得。
SphereFace Revived: Unifying Hyperspherical Face Recognition
University of Cambridge（剑桥大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；The Alan Turing Institute（阿兰·图灵研究所）；Carnegie Mellon University（卡内基梅隆大学）
Unifying Graph Embedding Features with Graph Convolutional Networks for Skeleton-based Action Recognition
University of California, Merced（加州大学默塞德分校）；Bishop’s University（毕索大学）；The Education University of Hong Kong（香港教育大学）；City University of Hong Kong（香港城市大学）；Hefei University of Technology（合肥工业大学）
AdapNet: Adaptability Decomposing Encoder-Decoder Network for Weakly Supervised Action Recognition and Localization
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；University of Electronic Science and Technology of China（电子科技大学）；University of Science and Technology Beijing（北京科技大学）；College of Information and Control Engineering, China University of Petroleum (East China)（中国石油大学（华东）信息与控制工程学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Stroke Constrained Attention Network for Online Handwritten Mathematical Expression Recognition
University of Science and Technology of China（中国科学技术大学）；National Engineering Laboratory for Speech and Language Information Processing（国家语音与语言信息处理工程实验室）
In-Context Translation: Towards Unifying Image Recognition, Processing, and Generation
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University（上海交通大学图像通信与网络工程研究所）；School of Information Systems, Singapore Management University（新加坡管理大学信息系统学院）；MoE Key Laboratory of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院（教育部人工智能重点实验室））；School of Electronics and Computer Science, University of Southampton（南安普顿大学电子与计算机科学学院）
InstructSeq: Unifying Vision Tasks with Instruction-conditioned Multi-modal Sequence Generation
The Chinese University of Hong Kong（香港中文大学）；Fudan University（复旦大学）；Sensetime Research Institute（商汤研究院）；Tsinghua University（清华大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
EDSL: An Encoder-Decoder Architecture with Symbol-Level Features for Printed Mathematical Expression Recognition
East China Normal University（华东师范大学）
印刷体数学表达式识别（PMER）旨在将印刷体数学表达式图像转录为结构化表达式，例如LaTeX表达式。它是一个至关重要的任务，许多应用，包括自动推荐问题，自动解决问题和分析的学生等。目前，主流的解决方案依赖于解决图像描述任务，所有解决图像摘要。因此，这些方法对于解决MER问题可能是次优的。本文提出了一种新的识别印刷体数学表达式的方法EDSL（encoder-decoder with symbol-level features）。EDSL的符号级图像编码器由分割模块和重建模块组成。通过执行分割模块，我们以无监督的方式从图像中识别所有符号及其空间信息。然后，我们设计了一个新的重建模块，以恢复符号分割后的符号依赖关系。特别地，我们采用位置校正注意力机制来捕捉符号之间的空间关系。为了减轻长输出带来的负面影响，我们采用Transformer模型将编码后的图像转换为序列化和结构化的输出。我们在两个真实数据集上进行了广泛的实验，以验证我们提出的EDSL方法的有效性和合理性。实验结果表明，EDSL在评价指标Match上分别达到了92.7%和89.0%，比现有方法提高了3.47%和4.04%。我们的代码和数据集可以在https://github.com/abcAnonymous/EDSL上找到。
Dynamic Inter-Class Confusion-Aware Encoder for Audio-Visual Fusion in Human Activity Recognition
School of Automation Science and Engineering, Xi’an Jiaotong University, Xi’an, China（自动化科学与工程学院，西安交通大学，西安，中国）；School of Software Engineering, Xi’an Jiaotong University, Xi’an, China（软件工程学院，西安交通大学，西安，中国）
Unifying Latent and Lexicon Representations for Effective Video-Text Retrieval
MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；University of Science and Technology of China（中国科学技术大学）；Alibaba Group（阿里巴巴集团）；School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）
