Inferflow：一个高效且高度可配置的大语言模型推理引擎
Inferflow: an Efficient and Highly Configurable Inference Engine for Large Language Models
AI总结：
Inferflow 是一个高效且高度可配置的大语言模型推理引擎，通过模块化框架、3.5-bit 量化和混合模型划分，实现无需改代码即可服务多种模型并提升多 GPU 推理性能。
摘要：
我们提出 Inferflow，一个高效且高度可配置的大语言模型（LLMs）推理引擎。借助 Inferflow，用户只需在相应配置文件中修改若干行，无需编写一行源代码，即可服务大多数常见 transformer 模型。与大多数现有推理引擎相比，Inferflow 具有若干关键特性。首先，通过实现原子构建块和技术的模块化框架，Inferflow 在组合上可泛化到新模型。其次，Inferflow 引入 3.5-bit 量化，作为 3-bit 与 4-bit 量化之间的折中方案。第三，Inferflow 引入面向多 GPU 推理的混合模型划分，相比现有的按层划分和按张量划分策略，能更好地平衡推理速度与吞吐量。
英文摘要：
We present Inferflow, an efficient and highly configurable inference engine for large language models (LLMs). With Inferflow, users can serve most of the common transformer models by simply modifying some lines in corresponding configuration files, without writing a single line of source code. Compared with most existing inference engines, Inferflow has some key features. First, by implementing a modular framework of atomic build-blocks and technologies, Inferflow is compositionally generalizable to new models. Second, 3.5-bit quantization is introduced in Inferflow as a tradeoff between 3-bit and 4-bit quantization. Third, hybrid model partitioning for multi-GPU inference is introduced in Inferflow to better balance inference speed and throughput than the existing partition-by-layer and partition-by-tensor strategies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
