This paper introduces and evaluates two novel Hierarchical Attention Network models [Yang et al., 2016] - i) Hierarchical Pruned Attention Networks, which remove the irrelevant words and sentences from the classification process in order to reduce potential noise in the document classification accuracy and ii) Hierarchical Sparsemax Attention Networks, which replace the Softmax function used in the attention mechanism with the Sparsemax [Martins and Astudillo, 2016], capable of better handling importance distributions where a lot of words or sentences have very low probabilities. Our empirical evaluation on the IMDB Review for sentiment analysis datasets shows both approaches to be able to match the results obtained by the current state-of-the-art (without, however, any significant benefits). All our source code is made available athttps://github.com/jmribeiro/dsl-project.
Attention mechanisms in deep learning architectures have often been used as a means of transparency and, as such, to shed light on the inner workings of the architectures. Recently, there has been a growing interest in whether or not this assumption is correct. In this paper we investigate the interpretability of multi-head attention in abstractive summarization, a sequence-to-sequence task for which attention does not have an intuitive alignment role, such as in machine translation. We first introduce three metrics to gain insight in the focus of attention heads and observe that these heads s…
A Novel Hierarchical Integration Method for Efficient Model Merging in Medical LLMs
Deakin Cyber Research（德肯网络研究）；Innovation Centre Deakin University / Universal Higher Education Melbourne, Australia（德肯大学创新中心/通用高等教育墨尔本，澳大利亚）；School of IT（信息科技学院）；Engineering Melbourne Institute of Technology Melbourne, Australia（工程墨尔本技术学院墨尔本，澳大利亚）；Innovation Centre Deakin University Geelong, Australia（德肯大学创新中心吉朗，澳大利亚）
HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exit
Institute of Digital Twin, Eastern Institute of Technology（数字孪生研究所，东部技术研究所）；Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative（宁波空间智能与数字衍生关键实验室）；University of Science and Technology of China（中国科学技术大学）；Shanghai Jiao Tong University（上海交通大学）；Munich Center for Machine Learning, LMU Munich（慕尼黑大学机器学习中心，慕尼黑大学）
Token Caching for Diffusion Transformer Acceleration
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所）；School of Information and Electronic Engineering, Shandong Technology and Business University（山东科技职业大学信息与电子工程学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；PeopleAl, Inc.（PeopleAl公司）；School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Terminal Technology Department, Alipay, Ant Group（蚂蚁集团支付宝终端技术部）
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；The University of Sydney（悉尼大学）；University of California, Irvine（加州大学欧文分校）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Enhancing Knowledge Density via Hierarchical Context Pruning for Retrieval-Augmented Repository-Level Code Completion
MIIT Key Laboratory of Pattern Analysis and Machine Intelligence（信息科技模式分析与机器智能国家重点实验室）；College of Computer Science and Technology（计算机科学与技术学院）；Huaneng Information Technology Co Ltd（华能信息技术有限公司）；Hunan Vanguard Group Corporation Co Ltd（湖南 Vanguard集团有限公司）；Tsinghua University（清华大学）；Zhejiang University（浙江大学）；PowerChina Huadong Engineering Co Ltd（国家电网华东工程有限公司）
Ant Group（蚂蚁集团）；Shandong University（山东大学）；Hasso Plattner Institute（哈索·普拉特纳研究所）；University of Potsdam（波茨坦大学）
Recently CKY-based models show great potential in unsupervised grammar induction thanks to their human-like encoding paradigm, which runs recursively and hierarchically, but requires $O(n^3)$ time-complexity. Recursive Transformer based on Differentiable Trees (R2D2) makes it possible to scale to large language model pre-training even with complex tree encoder by introducing a heuristic pruning method. However, the rule-based pruning approach suffers from local optimum and slow inference issues. In this paper, we fix those issues in a unified method. We propose to use a top-down parser as a mo…
