This study aims to systematically evaluate the performance of large language models (LLMs) in abstract visual reasoning problems. We examined four LLM models (GPT-4.1-Mini, Claude-3.5-Haiku, Gemini-1.5-Flash, Llama-3.3-70b) utilizing four different reasoning architectures (single-shot, embedding-controlled repetition, self-reflection, and multi-agent) on the RAVEN-FAIR dataset. Visual responses generated through a three-stage process (JSON extraction, LLM reasoning, and Tool Function) were evaluated using SSIM and LPIPS metrics; Chain-of-Thought scores and error types (semantic hallucination, numeric misperception) were analyzed. Results demonstrate that GPT-4.1-Mini consistently achieved the highest overall accuracy across all architectures, indicating a strong reasoning capability. While the multi-agent architecture occasionally altered semantic and numeric balance across models, these effects were not uniformly beneficial. Instead, each model exhibited distinct sensitivity patterns to architectural design, underscoring that reasoning effectiveness remains model-specific. Variations in response coverage further emerged as a confounding factor that complicates direct cross-architecture comparison. To estimate the upper-bound performance of each configuration, we report the best of five independent runs, representing a best-case scenario rather than an averaged outcome. This multi-run strategy aligns with recent recommendations, which emphasize that single-run evaluations are fragile and may lead to unreliable conclusions.
Abstract reasoning refers to the ability to analyze information, discover rules at an intangible level, and solve problems in innovative ways. Raven's Progressive Matrices (RPM) test is typically used to examine the capability of abstract reasoning. The subject is asked to identify the correct choice from the answer set to fill the missing panel at the bottom right of RPM (e.g., a 3$\times$3 matrix), following the underlying rules inside the matrix. Recent studies, taking advantage of Convolutional Neural Networks (CNNs), have achieved encouraging progress to accomplish the RPM test. However, …
The task of answering natural language questions over knowledge bases has received wide attention in recent years. Various deep learning architectures have been proposed for this task. However, architectural design choices are typically not systematically compared nor evaluated under the same conditions. In this paper, we contribute to a better understanding of the impact of architectural design choices by evaluating four different architectures under the same conditions. We address the task of answering simple questions, consisting in predicting the subject and predicate of a triple given a q…
Tel Aviv University（特拉维夫大学）；Facebook AI Research（Facebook人工智能研究院）
We consider the abstract relational reasoning task, which is commonly used as an intelligence test. Since some patterns have spatial rationales, while others are only semantic, we propose a multi-scale architecture that processes each query in multiple resolutions. We show that indeed different rules are solved by different resolutions and a combined multi-scale approach outperforms the existing state of the art in this task on all benchmarks by 5-54%. The success of our method is shown to arise from multiple novelties. First, it searches for relational patterns in multiple resolutions, which …
Xiaomi AI Lab（小米人工智能实验室）；Minzu University of China（中央民族大学）
Differentiable Architecture Search (DARTS) is now a widely disseminated weight-sharing neural architecture search method. However, it suffers from well-known performance collapse due to an inevitable aggregation of skip connections. In this paper, we first disclose that its root cause lies in an unfair advantage in exclusive competition. Through experiments, we show that if either of two conditions is broken, the collapse disappears. Thereby, we present a novel approach called Fair DARTS where the exclusive competition is relaxed to be collaborative. Specifically, we let each operation's archi…
The Hidden Evolution of Disguised Visual Context inside the VLM
Surrey Institute for People-Centred AI, University of Surrey（萨里大学以人为本人工智能研究所）；Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey（萨里大学视觉、语音与信号处理中心）
ModuLM: Enabling Modular and Multimodal Molecular Relational Learning with Large Language Models
University of Science and Technology of China（中国科学技术大学）；Suzhou Institute for Advanced Research, USTC（先进研究所）；Hunan University（湖南大学）；Monash University（墨尔本大学）；State Key Laboratory of Precision and Intelligent Chemistry, USTC（精密与智能化学国家重点实验室）
Convolutional neural networks (CNNs) have become commonplace in addressing major challenges in computer vision. Researchers are not only coming up with new CNN architectures but are also researching different techniques to improve the performance of existing architectures. However, there is a tendency to over-emphasize performance improvement while neglecting certain important variables such as simplicity, versatility, the fairness of comparisons, and energy efficiency. Overlooking these variables in architectural design and evaluation has led to research bias and a significantly negative envi…
Understanding LLM Agent Behaviours via Game Theory: Strategy Recognition, Biases and Multi-Agent Dynamics
Faculty of Information and Technology, Ho Chi Minh City University of Science (HCMUS), Vietnam（信息科技学院，胡志明市科学大学（HCMUS））；Vietnam National University - Ho Chi Minh City (VNU-HCM), Vietnam（越南国家大学-胡志明市（VNU-HCM））；Faculty of Computer Science and Engineering, Ho Chi Minh City University of Technology (HCMUT), Vietnam（计算机科学与工程学院，胡志明市技术大学（HCMUT））
A Unified Benchmark of Deep Learning Models for Multi-task 3D Brain Tumor Segmentation from Magnetic Resonance Imaging
Centro de Tecnologías de la Imagen (CTIM)（图像技术中心（CTIM））；Instituto Universitario de Cibernética, Empresas y Sociedad (IUCES)（网络、企业与社会大学研究所（IUCES））；University of Las Palmas de Gran Canaria（拉斯帕尔马斯大加那利大学）
Helmholtz-Zentrum Dresden-Rossendorf (HZDR) - Helmholtz Institute Freiberg for Resource Technology (HIF)（亥姆霍兹德累斯顿罗森多夫研究中心（HZDR）- 弗莱贝格亥姆霍兹资源技术研究所（HIF））；University of Antwerp（安特卫普大学）；National School of Applied Sciences of Oujda（乌季达国立应用科学学院）
