CharED：用于大型语言模型的字符级集成解码
CharED: Character-wise Ensemble Decoding for Large Language Models
AI总结：
CharED是一种推理时的字符级集成解码算法，通过加权平均多个LLM的字符边际分布来生成输出，在编程、数学和毒性基准上优于单个模型，且不依赖共享词汇或分词。
摘要：
大型语言模型（LLMs）在问题解决方面展现出显著的潜力，开源模型在衡量从逻辑推理到数学能力等领域的基准测试中取得了越来越令人印象深刻的性能。集成模型可以进一步提升跨多个领域的能力。然而，在推理时组合模型的传统方法（如浅层融合）需要共享词汇表和分词方式，而针对特定领域性能进行微调等替代方案既耗时又计算成本高昂。因此，我们提出了一种推理时的集成算法，旨在对多个LLM的输出进行“平均”，并展示了其与单独使用各组成模型相比在多个领域中的性能提升。字符级集成解码（CharED）为单个模型计算每个字符的边际分布，并执行加权平均以逐字符生成输出。在编程、数学和毒性基准测试中，我们发现我们提出的模型能够结合多个LLM的互补优势，无论词汇表、分词方式或模型大小如何。
英文摘要：
Large language models (LLMs) have shown remarkable potential for problem solving, with open source models achieving increasingly impressive performance on benchmarks measuring areas from logical reasoning to mathematical ability. Ensembling models can further improve capabilities across a variety of domains. However, conventional methods of combining models at inference time such as shallow fusion necessitate a shared vocabulary and tokenization, and alternatives like fine-tuning for domain-specific performance are both time consuming and computationally expensive. We therefore present an inference-time ensembling algorithm aimed at "averaging" outputs from multiple LLMs and illustrate its improved performance across multiple domains compared to its constituent models alone. Character-wise ensemble decoding, CharED, finds the marginal distribution of each character for an individual model and performs a weighted average to generate an output, character by character. In coding, math, and toxicity benchmarks, we find our proposed model able to combine complimentary strengths of multiple LLMs, regardless of vocabulary, tokenization, or model size.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
