Language model agents (LMA) recently emerged as a promising paradigm on muti-step decision making tasks, often outperforming humans and other reinforcement learning agents. Despite the promise, their performance on real-world applications that often involve combinations of tasks is still underexplored. In this work, we introduce a new benchmark, called CompWoB -- 50 new compositional web automation tasks reflecting more realistic assumptions. We show that while existing prompted LMAs (gpt-3.5-turbo or gpt-4) achieve 94.0% average success rate on base tasks, their performance degrades to 24.9% success rate on compositional tasks. On the other hand, transferred LMAs (finetuned only on base tasks) show less generalization gap, dropping from 85.4% to 54.8%. By balancing data distribution across tasks, we train a new model, HTML-T5++, that surpasses human-level performance (95.2%) on MiniWoB, and achieves the best zero-shot performance on CompWoB (61.5%). While these highlight the promise of small-scale finetuned and transferred models for task compositionality, their performance further degrades under different instruction compositions changing combinational order. In contrast to the recent remarkable success of LMA, our benchmark and detailed analysis emphasize the necessity of building LMAs that are robust and generalizable to task compositionality for real-world deployment.
Exposing propaganda: an analysis of stylistic cues comparing human annotations and machine classification
Airbus Defence and Space（空客防务与空间公司）；Université Paris-Saclay（巴黎-萨克雷大学）；CentraleSupélec（中央理工-高等电力学院）；Institut Jean-Nicod（让·尼科研究所）；CNRS（法国国家科学研究中心）；ENS-PSL（巴黎文理研究大学高等师范学院）；EHESS（社会科学高等研究院）；LIP6（巴黎第六大学计算机实验室）；Sorbonne Université（索邦大学）；Université de Rennes（雷恩大学）；Inria（法国国家信息与自动化研究所）；IRISA（雷恩信息与自动化研究所）；Mondeca（蒙德卡公司）；Freedom Partners（自由伙伴公司）；Observatoire des Médias（媒体观察机构）
An Asymptotically Optimal Batched Algorithm for the Dueling Bandit Problem
我们研究了$K$ -武装决斗土匪问题，传统的多臂土匪问题，其中反馈的形式成对比较的变化。以前的学习算法集中在$\textit{fully adaptive}$设置上，其中算法可以在每次比较后进行更新。“批量”决斗强盗问题的动机是大规模的应用程序，如网络搜索排名和推荐系统，其中执行顺序更新可能是不可行的。在这项工作中，我们要求：$\textit{is there a solution using only a few adaptive rounds that matches the asymptotic regret bounds of the best sequential algorithms for $ K $-armed dueling bandits?}$我们的回答是肯定的$\textit{under the Condorcet condition}$，这是$K$ -武装决斗强盗问题的标准设置。我们得到了$O(K^2\log^2(K)) + O(K\log(T))$在$O(\log(T))$轮中的渐近后悔，其中$T$是时间范围.我们的遗憾边界几乎匹配最好的遗憾边界已知的完全连续设置下孔多塞条件。最后，在各种现实世界的数据集的计算实验中，我们观察到，我们的算法使用$O(\log(T))$轮实现了几乎相同的性能完全顺序算法（使用$T$轮）。
JEF-Hinter: Leveraging Offline Knowledge for Improving Web Agents Adaptation
ServiceNow AI Research（ServiceNow AI研究机构）；Chandar Research Lab（Chandar研究实验室）；Mila -- Quebec AI Institute（魁北克人工智能研究院）；Dalhousie University（达尔豪斯大学）；Canada CIFAR AI Chair（加拿大CIFAR人工智能主席）
Does The Way You Plan Matter? An Empirical Study of Planning Representations for LLM Web Agents
Concordia University（康科德大学）；Mila - Quebec AI Institute（魁北克人工智能研究所）；University of Copenhagen（哥本哈根大学）；Universite Claude Bernard Lyon（克莱尔蒙特-伯恩大学）；McGill University（麦吉尔大学）
