Learning a model of a stochastic setting often involves learning both general structure rules and specific properties of the instance. This paper investigates the interplay between learning the general and the specific in various learning methods, with emphasis on sample efficiency. We design a framework called {\sc LeverWorlds}, which allows the generation of simple physics-inspired worlds that follow a similar generative process with different distributions, and their instances can be expressed in natural language. These worlds allow for controlled experiments to assess the sample complexity of different learning methods. We experiment with classic learning algorithms as well as Transformer language models, both with fine-tuning and In-Context Learning (ICL). Our general finding is that (1) Transformers generally succeed in the task; but (2) they are considerably less sample efficient than classic methods that make stronger assumptions about the structure, such as Maximum Likelihood Estimation and Logistic Regression. This finding is in tension with the recent tendency to use Transformers as general-purpose estimators. We propose an approach that leverages the ICL capabilities of contemporary language models to apply simple algorithms for this type of data. Our experiments show that models currently struggle with the task but show promising potential.
How Does Sequence Modeling Architecture Influence Base Capabilities of Pre-trained Language Models? Exploring Key Architecture Design Principles to Avoid Base Capabilities Degradation
Research Center for Social Computing and Interactive Robotics(社会计算与交互机器人研究中心);Harbin Institute of Technology(哈尔滨工业大学);iFLYTEK Co., Ltd(iFLYTEK公司)
MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generation
University of Edinburgh(爱丁堡大学);The University of Hong Kong(香港大学);Harbin Institute of Technology(哈尔滨工业大学);Singapore University of Technology and Design(新加坡科技设计大学);South China University of Technology(华南理工大学);City University of Hong Kong(香港城市大学);University of Cambridge(剑桥大学)
Atomic Thinking of LLMs: Decoupling and Exploring Mathematical Reasoning Abilities
Sun Yat-sen University(中山大学);Tsinghua University(清华大学);ByteDance Inc(字节跳动公司);Arizona State University(亚利桑那州立大学);Tencent Youtu Lab(腾讯优图实验室);Fudan University(复旦大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Guangdong Provincial Key Laboratory of Fire Science and Intelligent Emergency Technology(广东省消防科学与智能应急技术重点实验室);University of Illinois Chicago(伊利诺伊大学香槟分校)
Exploring In-Context Learning Capabilities of Foundation Models for Generating Knowledge Graphs from Text
Peoples' Friendship University of Russia (RUDN University)(俄罗斯人民友谊大学(卢蒙巴大学));Joint Institute for Nuclear Research(联合核研究所);Vrije Universiteit Amsterdam(阿姆斯特丹自由大学);University of Skövde(舍夫德大学);University of Lübeck(吕贝克大学);IBM Research(IBM研究院)
How Far Can In-Context Alignment Go? Exploring the State of In-Context Alignment
School of Computer Science and Technology, Beijing Institute of Technology(北京理工大学计算机科学与技术学院);Beijing Institute of Technology Southeast Academy of Information Technology(北京理工大学东南信息技术研究院)
Exploring the Multilingual NLG Evaluation Abilities of LLM-Based Evaluators
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);School of Data Science and Intelligent Media, Communication University of China(中国传媒大学数据科学与智能媒体学院)
MIRAGE: Exploring How Large Language Models Perform in Complex Social Interactive Environments
Institute of Big Data, Fudan University(复旦大学大数据研究院);Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University(复旦大学计算机学院);School of Computer Science, Fudan University(复旦大学计算机学院);Xiaohongshu Inc.(小红书公司)
Exploring Recommendation Capabilities of GPT-4V(ision): A Preliminary Case Study
Hong Kong University of Science and Technology(香港科技大学);Alibaba(阿里巴巴);Harvard T.H. Chan School of Public Health(哈佛大学陈曾熙公共卫生学院);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));International Digital Economy Academy (IDEA)(国际数字经济学院);Peking University(北京大学)