从文本到数值:你的大语言模型在给定上下文示例时实则是能力出众的回归器
From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples
AI总结:
该研究发现预训练大语言模型无需额外训练,仅靠上下文示例就能完成回归任务,性能可匹敌甚至超越传统监督方法,且能实现亚线性悔值。
摘要:
我们分析了预训练大语言模型(如Llama2、GPT-4、Claude 3等)在仅给定上下文示例、无需额外训练或梯度更新的情况下,执行线性与非线性回归的效果。研究结果显示,多款大语言模型(如GPT-4、Claude 3)的回归任务表现可匹敌(甚至超越)随机森林、Bagging、梯度提升等传统监督方法。例如在颇具难度的Friedman #2回归数据集上,Claude 3的表现优于AdaBoost、SVM、随机森林、KNN、梯度提升等多种监督方法。我们进一步探究了大语言模型的性能随上下文示例数量的扩展规律,借鉴在线学习中的regret(悔值)概念,通过实验证明大语言模型可实现亚线性悔值。
英文摘要:
We analyze how well pre-trained large language models (e.g., Llama2, GPT-4, Claude 3, etc) can do linear and non-linear regression when given in-context examples, without any additional training or gradient updates. Our findings reveal that several large language models (e.g., GPT-4, Claude 3) are able to perform regression tasks with a performance rivaling (or even outperforming) that of traditional supervised methods such as Random Forest, Bagging, or Gradient Boosting. For example, on the challenging Friedman #2 regression dataset, Claude 3 outperforms many supervised methods such as AdaBoost, SVM, Random Forest, KNN, or Gradient Boosting. We then investigate how well the performance of large language models scales with the number of in-context exemplars. We borrow from the notion of regret from online learning and empirically show that LLMs are capable of obtaining a sub-linear regret.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。