脆弱的精通:领域特定权衡是否正在削弱端侧语言模型?
Fragile Mastery: Are Domain-Specific Trade-Offs Undermining On-Device Language Models?
AI总结:
本研究提出广义边缘模型(GEM),通过稀疏交叉注意力路由器平衡端侧语言模型的领域特化与跨领域泛化,并引入三个新指标揭示压缩强度与脆弱性的关联。
摘要:
在资源受限的边缘设备上应用端侧语言模型(ODLMs)是一个多维问题,需要在异构任务中精细平衡计算效率、内存、功耗和语言能力。这项整体性研究深入探讨了领域特定优化与跨领域鲁棒性之间的权衡,最终提出了广义边缘模型(GEM),这是一种旨在和谐平衡专业化与泛化的新架构。通过严格的实验方法,我们在八个领域——医疗保健、法律、金融、STEM、常识、对话式AI、多语言和领域自适应任务——测试了47个精心挑选的基准,结果表明,传统优化技术将目标任务的困惑度降低了18-25%,但导致通用任务性能急剧下降,F1分数降低了12-29%,正如Liu等人所报告的那样。GEM采用稀疏交叉注意力路由器(SCAR)动态分配计算资源,在树莓派4、Pixel 6、iPhone 13和定制神经处理单元(NPUs)上,跨领域F1准确率达到0.89,延迟低于100毫秒。与GPT-4 Lite相比,GEM在通用任务水平上提升了7%,同时在领域特定性能上保持同等水平。我们提出了三个新的测量工具——领域特化指数(DSI)、泛化差距(GG)和跨领域迁移比(CDTR)——它们显示出模型压缩强度与脆弱性之间的强相关性。
英文摘要:
The application of on-device language models (ODLMs) on resource-constrained edge devices is a multi-dimensional problem that strikes a fine balance between computational effectiveness, memory, power usage, and linguistic capacity across heterogeneous tasks. This holistic study conducts a thorough investigation of the trade-offs between domain-specific optimization and cross-domain robustness, culminating in the proposal of the Generalized Edge Model (GEM), a new architecture that aims to balance specialization and generalization in a harmonious manner. With a rigorous experimental approach testing 47 well-chosen benchmarks in eight domains--healthcare, law, finance, STEM, commonsense, conversational AI, multilingual, and domain-adaptive tasks--we show that conventional optimization techniques decrease target task perplexity by 18-25% but result in a precipitous decline in general-task performance with F1 scores decreasing by 12-29%, as reported by Liu et al. GEM employs a Sparse Cross-Attention Router (SCAR) to dynamically allocate computation to a variable number of computing resources with a cross-domain F1 accuracy of 0.89 on less than 100ms latency across Raspberry Pi 4, Pixel 6, iPhone 13, and bespoke custom neural processing units (NPUs). Compared to GPT-4 Lite, GEM enhances the general-task level by 7% with respect and parity in domain-specific performance. We propose three new measurement tools--Domain Specialization Index (DSI), Generalization Gap (GG), and Cross-Domain Transfer Ratio (CDTR)--which show strong correlation between model compression intensity and brittleness.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。