节点分类是图分析中的一项基础任务，在诸多领域有着广泛应用。大语言模型（Large Language Models，LLMs）领域的最新突破使得基于LLM的方法得以应用于该任务。尽管已有诸多研究展现了基于LLM的方法的出色性能，但清晰设计指南的缺失可能会阻碍其实际应用。在本研究中，我们旨在通过对这些算法进行公平、系统的比较来建立此类指南。作为第一步，我们开发了LLMNodeBed——一个用于基于LLM的节点分类的综合代码库与测试平台，它包含10个同配性数据集、4个异配性数据集、8种基于LLM的算法、8个经典基线以及3种学习范式。随后，我们开展了大量实验，训练并评估了超过2700个模型，以确定影响性能的关键设置（例如学习范式、同配性）与组件（例如模型规模、提示词）。我们的研究得出了8项洞见，例如：（1）在半监督设置下，基于LLM的方法能显著优于传统方法，而在监督设置下其优势十分有限；（2）在零样本设置下，图基础模型（Graph Foundation Models）能胜过开源LLM，但仍不及GPT-4o这类强性能LLM。我们希望LLMNodeBed的发布连同我们的洞见，能推动可复现研究的开展，并为该领域的未来研究提供启发。代码与数据集已发布于https://llmnodebed.github.io/。
英文摘要：
Node classification is a fundamental task in graph analysis, with broad applications across various fields. Recent breakthroughs in Large Language Models (LLMs) have enabled LLM-based approaches for this task. Although many studies demonstrate the impressive performance of LLM-based methods, the lack of clear design guidelines may hinder their practical application. In this work, we aim to establish such guidelines through a fair and systematic comparison of these algorithms. As a first step, we developed LLMNodeBed, a comprehensive codebase and testbed for node classification using LLMs. It includes 10 homophilic datasets, 4 heterophilic datasets, 8 LLM-based algorithms, 8 classic baselines, and 3 learning paradigms. Subsequently, we conducted extensive experiments, training and evaluating over 2,700 models, to determine the key settings (e.g., learning paradigms and homophily) and components (e.g., model size and prompt) that affect performance. Our findings uncover 8 insights, e.g., (1) LLM-based methods can significantly outperform traditional methods in a semi-supervised setting, while the advantage is marginal in a supervised setting; (2) Graph Foundation Models can beat open-source LLMs but still fall short of strong LLMs like GPT-4o in a zero-shot setting. We hope that the release of LLMNodeBed, along with our insights, will facilitate reproducible research and inspire future studies in this field. Codes and datasets are released at \href{https://llmnodebed.github.io/}{\texttt{https://llmnodebed.github.io/}}.
Exploring Graph Learning Tasks with Pure LLMs: A Comprehensive Benchmark and Investigation
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Michigan State University（密歇根州立大学）；The Hong Kong Polytechnic University（香港理工大学）；Rensselaer Polytechnic Institute（纽约理工学院）
Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey
State Key Lab of Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；School of Computing and Mathematical Sciences, Birkbeck, University of London（伦敦大学伯克贝克学院计算与数学科学学院）
LLMs for Drug-Drug Interaction Prediction: A Comprehensive Comparison
Software Engineering (SeSa) Lab（软件工程（SeSa）实验室）；University of Salerno（萨勒诺大学）；Department of Epidemiology and Data Science（流行病学与数据科学系）；Amsterdam UMC Locatie AMC（阿姆斯特丹大学医学中心（AMC））；Amsterdam Public Health Research Institute（阿姆斯特丹公共卫生研究院）；University of Amsterdam（阿姆斯特丹大学）；University of Amsterdam Data Science Center（阿姆斯特丹大学数据科学中心）
ChemEval: A Comprehensive Multi-Level Chemical Evaluation for Large Language Models
University of Science and Technology of China（中国科学技术大学）；State Key Laboratory of Cognitive Intelligence（认知智能国家重点实验室）；Artificial Intelligence Research Institute, iFLYTEK Co., Ltd.（讯飞股份有限公司人工智能研究院）；iFLYTEK AI Research (Hebei)（讯飞人工智能研究院（河北））
