语言大模型 / LLM

The Backpropagation of the Wave Network

Xin Zhang, Victor S. Sheng

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.AI

MSWA: Refining Local Attention with Multi-ScaleWindow Attention

Yixing Xu, Shivank Nag, Dong Li, Lu Tian, Emad Barsoum

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.AI

Syzygy: Dual Code-Test C to (safe) Rust Translation using LLMs and Dynamic Analysis

Manish Shetty, Naman Jain, Adwait Godbole, Sanjit A. Seshia, Koushik Sen

专题命中 长上下文与记忆 ：LLM(abstract)；分类 cs.AI、cs.LG

Comments Project webpage at https://syzygy-project.github.io/. Preliminary version accepted at LLM4Code 2025, 34 pages

Memory Layers at Scale

Vincent-Pierre Berges, Barlas Oğuz, Daniel Haziza, Wen-tau Yih, Luke Zettlemoyer, Gargi Ghosh

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.AI

An Adversarial Example for Direct Logit Attribution: Memory Management in GELU-4L

Jett Janiak, Can Rager, James Dao, Yeu-Tong Lau

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.AI、cs.LG

Journal ref BlackboxNLP Workshop 2024, pages 232-237

Test Time Learning for Time Series Forecasting

Panayiotis Christou, Shichu Chen, Xupeng Chen, Parijat Dube

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.AI、cs.LG

2D Matryoshka Sentence Embeddings

Xianming Li, Zongxi Li, Jing Li, Haoran Xie, Qing Li

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.LG

Comments Decoupled with ESE

Selective Attention: Enhancing Transformer through Principled Context Control

Xuechen Zhang, Xiangyu Chang, Mingchen Li, Amit Roy-Chowdhury, Jiasi Chen, Samet Oymak

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.LG

M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding

Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, Mohit Bansal

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.AI

Comments Project webpage: https://m3docrag.github.io

SAM 2: Segment Anything in Images and Videos

Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, Christoph Feichtenhofer

专题命中 长上下文与记忆 ：foundation model(abstract)；分类 cs.AI、cs.LG

Comments Website: https://ai.meta.com/sam2

On the token distance modeling ability of higher RoPE attention dimension

Xiangyu Hong, Che Jiang, Biqing Qi, Fandong Meng, Mo Yu, Bowen Zhou, Jie Zhou

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted to EMNLP 2024 Findings

MoEUT: Mixture-of-Experts Universal Transformers

Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber, Christopher Potts, Christopher D. Manning

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.AI、cs.LG

Comments Accepted to NeurIPS 2024

Automatic Summarization of Long Documents

Naman Chhibbar, Jugal Kalita

专题命中 长上下文与记忆 ：LLM(abstract)；分类 cs.CL、cs.AI

Comments 9 pages (including bibliography) with 6 figures. ACL 2023 proceedings format

AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies

Xiao Ye, Andrew Wang, Jacob Choi, Yining Lu, Shreya Sharma, Lingfeng Shen, Vijay Tiyyala, Nicholas Andrews, Daniel Khashabi

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted to EMNLP 2024 (Main)

SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention

Róbert Csordás, Piotr Piękos, Kazuki Irie, Jürgen Schmidhuber

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.LG

Comments Accepted to NeurIPS 2024

A Controlled Study on Long Context Extension and Generalization in LLMs

Yi Lu, Jing Nathan Yan, Songlin Yang, Justin T. Chiu, Siyu Ren, Fei Yuan, Wenting Zhao, Zhiyong Wu, Alexander M. Rush

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.LG

ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigation

Abrar Anwar, John Welsh, Joydeep Biswas, Soha Pouya, Yan Chang

专题命中 长上下文与记忆 ：LLM(abstract)；分类 cs.CL、cs.AI

Benchmarking General-Purpose In-Context Learning

Fan Wang, Chuan Lin, Yang Cao, Yu Kang

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.AI、cs.LG

Great Memory, Shallow Reasoning: Limits of $k$NN-LMs

Shangyi Geng, Wenting Zhao, Alexander M Rush

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.AI

xLSTMTime : Long-term Time Series Forecasting With xLSTM

Musleh Alharthi, Ausif Mahmood

专题命中 长上下文与记忆 ：prompting(abstract)；分类 cs.AI、cs.LG

Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms

Tian Meng, Yang Tao, Wuliang Yin

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.LG

Scaling Granite Code Models to 128K Context

Matt Stallone, Vaibhav Saxena, Leonid Karlinsky, Bridget McGinn, Tim Bula, Mayank Mishra, Adriana Meza Soria, Gaoyuan Zhang, Aditya Prasad, Yikang Shen, Saptha Surendran, Shanmukha Guttula, Hima Patel, Parameswaran Selvam, Xuan-Hong Dang, Yan Koyfman, Atin Sood, Rogerio Feris, Nirmit Desai, David D. Cox, Ruchir Puri, Rameswar Panda

专题命中 长上下文与记忆 ：pretraining(abstract)；分类 cs.CL、cs.AI

Memoria: Resolving Fateful Forgetting Problem through Human-Inspired Memory Architecture

Sangjun Park, JinYeong Bak

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.AI、cs.LG

Comments ICML 2024 Spotlight. 29 pages, 15 figures, 11 tables

Journal ref Proceedings of the 41st International Conference on Machine Learning, PMLR 235:39587-39615, 2024

AutoGRAMS: Autonomous Graphical Agent Modeling Software

Ben Krause, Lucia Chen, Emmanuel Kahembwe

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.AI

Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs

Ben Athiwaratkun, Sujan Kumar Gonugondla, Sanjay Krishna Gouda, Haifeng Qian, Hantian Ding, Qing Sun, Jun Wang, Jiacheng Guo, Liangfu Chen, Parminder Bhatia, Ramesh Nallapati, Sudipta Sengupta, Bing Xiang

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.AI、cs.LG

On the Power of Convolution Augmented Transformer

Mingchen Li, Xuechen Zhang, Yixiao Huang, Samet Oymak

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.LG

M5: A Whole Genome Bacterial Encoder at Single Nucleotide Resolution

Agust Egilsson

专题命中 长上下文与记忆 ：foundation model(abstract)；分类 cs.AI、cs.LG

Comments 13 pages, 5 figures

Promoting Exploration in Memory-Augmented Adam using Critical Momenta

Pranshu Malviya, Gonçalo Mordido, Aristide Baratin, Reza Babanezhad Harikandeh, Jerry Huang, Simon Lacoste-Julien, Razvan Pascanu, Sarath Chandar

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.AI、cs.LG

Comments Published in Transactions on Machine Learning Research

MATTER: Memory-Augmented Transformer Using Heterogeneous Knowledge Sources

Dongkyu Lee, Chandana Satya Prakash, Jack FitzGerald, Jens Lehmann

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.AI

Comments ACL2024-Findings

Discrete Prompt Compression with Reinforcement Learning

Hoyoun Jung, Kyung-Joong Kim

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.AI