大模型推理能力

Reactor Mk.1 performances: MMLU, HumanEval and BBH test results

TJ Dunham, Henry Syahputra

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Lynx: An Open Source Hallucination Evaluation Model

Selvan Sunitha Ravi, Bartosz Mielczarek, Anand Kannappan, Douwe Kiela, Rebecca Qian

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding

Haoning Wu, Dongxu Li, Bei Chen, Junnan Li

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.LG

Comments 29 pages

Can MLLMs Perform Text-to-Image In-Context Learning?

Yuchen Zeng, Wonjun Kang, Yicong Chen, Hyung Il Koo, Kangwook Lee

专题命中 推理评测 ：chain-of-thought(abstract)；分类 cs.CL、cs.LG

Comments Accepted at COLM 2024

Rapid Biomedical Research Classification: The Pandemic PACT Advanced Categorisation Engine

Omid Rohanian, Mohammadmahdi Nouriborji, Olena Seminog, Rodrigo Furst, Thomas Mendy, Shanthi Levanita, Zaharat Kadri-Alabi, Nusrat Jabin, Daniela Toale, Georgina Humphreys, Emilia Antonio, Adrian Bucher, Alice Norton, David A. Clifton

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

SciCode: A Research Coding Benchmark Curated by Scientists

Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, Shengzhu Yin, Minhui Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yufeng Du, Tianhua Tao, Ofir Press, Jamie Callan, Eliu Huerta, Hao Peng

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 25 pages, 9 figures, 7 tables

Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs

Andries Smit, Paul Duckworth, Nathan Grinsztajn, Thomas D. Barrett, Arnu Pretorius

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 2 pages, 13 figures

CHOPS: CHat with custOmer Profile Systems for Customer Service with LLMs

Jingzhe Shi, Jialuo Li, Qinwei Ma, Zaiwen Yang, Huan Ma, Lei Li

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Accepted by COLM 2024

EmoBench: Evaluating the Emotional Intelligence of Large Language Models

Sahand Sabour, Siyang Liu, Zheyuan Zhang, June M. Liu, Jinfeng Zhou, Alvionna S. Sunaryo, Juanzi Li, Tatia M. C. Lee, Rada Mihalcea, Minlie Huang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments ACL 2024 Main Conference

MOCA: Self-supervised Representation Learning by Predicting Masked Online Codebook Assignments

Spyros Gidaris, Andrei Bursuc, Oriane Simeoni, Antonin Vobecky, Nikos Komodakis, Matthieu Cord, Patrick Pérez

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Comments Published in Transactions on Machine Learning Research (TMLR) 2024. Code at https://github.com/valeoai/MOCA

LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model

Dilxat Muhtar, Zhenshi Li, Feng Gu, Xueliang Zhang, Pengfeng Xiao

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Comments 36 pages, 10 figures. Github https://github.com/NJU-LHRS/LHRS-Bot

Graphusion: Leveraging Large Language Models for Scientific Knowledge Graph Fusion and Construction in NLP Education

Rui Yang, Boming Yang, Sixun Ouyang, Tianwei She, Aosong Feng, Yuang Jiang, Freddy Lecue, Jinghui Lu, Irene Li

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 24 pages, 11 figures, 13 tables. arXiv admin note: substantial text overlap with arXiv:2402.14293

Interfacing Foundation Models' Embeddings

Xueyan Zou, Linjie Li, Jianfeng Wang, Jianwei Yang, Mingyu Ding, Junyi Wei, Zhengyuan Yang, Feng Li, Hao Zhang, Shilong Liu, Arul Aravinthan, Yong Jae Lee, Lijuan Wang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments CODE: https://github.com/UX-Decoder/FIND

Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency

Sakib Shahriar, Brady Lund, Nishith Reddy Mannuru, Muhammad Arbab Arshad, Kadhim Hayawi, Ravi Varma Kumar Bevara, Aashrith Mannuru, Laiba Batool

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

SciQAG: A Framework for Auto-Generated Science Question Answering Dataset with Fine-grained Evaluation

Yuwei Wan, Yixuan Liu, Aswathy Ajith, Clara Grazian, Bram Hoex, Wenjie Zhang, Chunyu Kit, Tong Xie, Ian Foster

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Testing AI on language comprehension tasks reveals insensitivity to underlying meaning

Vittoria Dentella, Fritz Guenther, Elliot Murphy, Gary Marcus, Evelina Leivada

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 18 pages, 4 figures, 1 table

Towards Optimizing and Evaluating a Retrieval Augmented QA Chatbot using LLMs with Human in the Loop

Anum Afzal, Alexander Kowsik, Rajna Fani, Florian Matthes

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Some Issues in Predictive Ethics Modeling: An Annotated Contrast Set of "Moral Stories"

Ben Fitzgerald

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments This project was a runner-up to the Novel Research prize for the BlueDot Impact course AI Safety Fundamentals. View my contrast set as JSONL, the UI used to generate it, and Emelin et. al."s initial paper and code at https://github.com/bfitzgerald3132/MoralStoriesContrastSet

MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?

Zhaorun Chen, Yichao Du, Zichen Wen, Yiyang Zhou, Chenhang Cui, Zhenzhen Weng, Haoqin Tu, Chaoqi Wang, Zhengwei Tong, Qinglan Huang, Canyu Chen, Qinghao Ye, Zhihong Zhu, Yuqing Zhang, Jiawei Zhou, Zhuokai Zhao, Rafael Rafailov, Chelsea Finn, Huaxiu Yao

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.LG

Comments 42 pages, 13 figures, 33 tables

Automated Assessment of Encouragement and Warmth in Classrooms Leveraging Multimodal Emotional Features and ChatGPT

Ruikun Hou, Tim Fütterer, Babette Bühler, Efe Bozkir, Peter Gerjets, Ulrich Trautwein, Enkelejda Kasneci

专题命中 推理评测 ：reasoning(abstract)；分类 cs.AI、cs.LG

Comments Accepted as a full paper by the 25th International Conference on Artificial Intelligence in Education (AIED 2024)

Journal ref Proceedings of the 25th International Conference on Artificial Intelligence in Education (AIED 2024)

BLINK: Multimodal Large Language Models Can See but Not Perceive

Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, Ranjay Krishna

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Multimodal Benchmark, Project Url: https://zeyofu.github.io/blink/, ECCV 2024

Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awareness

Khyathi Raghavi Chandu, Linjie Li, Anas Awadalla, Ximing Lu, Jae Sung Park, Jack Hessel, Lijuan Wang, Yejin Choi

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 26 pages

HGOT: Hierarchical Graph of Thoughts for Retrieval-Augmented In-Context Learning in Factuality Evaluation

Yihao Fang, Stephen W. Thomas, Xiaodan Zhu

专题命中 推理评测 ：planning(abstract)；分类 cs.CL、cs.AI

Can GPT-4 Help Detect Quit Vaping Intentions? An Exploration of Automatic Data Annotation Approach

Sai Krishna Revanth Vuruma, Dezhi Wu, Saborny Sen Gupta, Lucas Aust, Valerie Lookingbill, Wyatt Bellamy, Yang Ren, Erin Kasson, Li-Shiun Chen, Patricia Cavazos-Rehg, Dian Hu, Ming Huang

专题命中 推理评测 ：chain-of-thought(abstract)；分类 cs.CL、cs.AI

Comments Accepted for the AI Applications in Public Health and Social Services workshop at the 22nd International Conference on Artificial Intelligence in Medicine (AIME 2024)

MedCalc-Bench: Evaluating Large Language Models for Medical Calculations

Nikhil Khandekar, Qiao Jin, Guangzhi Xiong, Soren Dunn, Serina S Applebaum, Zain Anwar, Maame Sarfo-Gyamfi, Conrad W Safranek, Abid A Anwar, Andrew Zhang, Aidan Gilson, Maxwell B Singer, Amisha Dave, Andrew Taylor, Aidong Zhang, Qingyu Chen, Zhiyong Lu

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments Github link: https://github.com/ncbi-nlp/MedCalc-Bench HuggingFace link: https://huggingface.co/datasets/nsk7153/MedCalc-Bench

Muffin or Chihuahua? Challenging Multimodal Large Language Models with Multipanel VQA

Yue Fan, Jing Gu, Kaiwen Zhou, Qianqi Yan, Shan Jiang, Ching-Chen Kuo, Xinze Guan, Xin Eric Wang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments ACL 2024

Telecom Language Models: Must They Be Large?

Nicola Piovesan, Antonio De Domenico, Fadhel Ayed

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.LG

Is the Pope Catholic? Yes, the Pope is Catholic. Generative Evaluation of Non-Literal Intent Resolution in LLMs

Akhila Yerukola, Saujas Vaduguru, Daniel Fried, Maarten Sap

专题命中 推理评测 ：chain-of-thought(abstract)；分类 cs.CL、cs.AI

FinBen: A Holistic Financial Benchmark for Large Language Models

Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, Yijing Xu, Haoqiang Kang, Ziyan Kuang, Chenhan Yuan, Kailai Yang, Zheheng Luo, Tianlin Zhang, Zhiwei Liu, Guojun Xiong, Zhiyang Deng, Yuechen Jiang, Zhiyuan Yao, Haohang Li, Yangyang Yu, Gang Hu, Jiajia Huang, Xiao-Yang Liu, Alejandro Lopez-Lira, Benyou Wang, Yanzhao Lai, Hao Wang, Min Peng, Sophia Ananiadou, Jimin Huang

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 26 pages, 11 figures

Attributions toward Artificial Agents in a modified Moral Turing Test

Eyal Aharoni, Sharlene Fernandes, Daniel J. Brady, Caelan Alexander, Michael Criner, Kara Queen, Javier Rando, Eddy Nahmias, Victor Crespo

专题命中 推理评测 ：reasoning(abstract)；分类 cs.CL、cs.AI

Comments 23 pages, 0 figures, in press

Journal ref Scientific Reports (2024)