ComplianceGate: 分类器门控的多层LLM路由用于受监管行业的推理
ComplianceGate: Classifier-Gated Multi-Tier LLM Routing for Inference in Regulated Industries
AI总结：
提出分类器门控路由架构，在推理前评估查询复杂度和数据敏感性，将含PII的查询路由到本地端点，简单查询路由到小型模型，实现合规与成本效率。
摘要：
部署在受监管行业的大型语言模型在两种约束下运行：合规执行和成本效率。用户查询中的个人身份信息（PII）可能在系统确定该数据是否应离开其管辖边界之前到达模型端点。通过单个大型模型服务所有查询会消耗全部GPU容量，无论查询复杂度如何，同时没有提供地理路由机制。混合专家架构没有解决这个问题——路由发生在数据已经到达端点后的模型内专家层之间，所有专家都加载在内存中，无论查询复杂度如何。我们提出了一种分类器门控路由架构，通过设计强制合规。一个经过训练的编码器分类器位于任何解码器推理之前，评估每个查询的复杂度和数据敏感性，然后将其路由到适当大小的密集模型和适当的地理位置。包含PII的查询在任何LLM计算开始之前路由到本地端点，使得数据驻留违规在结构上不可能。简单查询以一小部分成本到达小型、快速的模型。我们在600个查询上的评估显示，中位延迟降低39%，根据查询分布节省33-52%的成本，生成吞吐量为122-200 tokens/秒，而基线为50-64 tokens/秒。编码器分类器实现了99.2%的准确率，近乎完美的PII召回率，推理开销为7毫秒，确立了推理前分类作为实现合规设计的LLM部署的实用途径。
英文摘要：
Large language models deployed in regulated industries operate under two constraints: compliance enforcement and cost efficiency. Personally identifiable information (PII) in user queries can reach model endpoints before the system determines whether that data should leave its jurisdictional boundary. Serving all queries through a single large model consumes full GPU capacity regardless of query complexity while offering no mechanism for geographic routing. Mixture-of-Experts architectures do not address this routing occurs between expert layers within the model after data has already arrived at the endpoint, with all experts loaded in memory regardless of query complexity. We propose a classifier-gated routing architecture that enforces compliance by design. A trained encoder classifier sits before any decoder inference, evaluating each query for complexity and data sensitivity, then routing it to an appropriately sized dense model in the appropriate geographic location. PII-containing queries route to local endpoints before any LLM computation begins, making data residency violations structurally impossible. Simple queries reach small, fast models at a fraction of the cost. Our evaluation on 600 queries demonstrates 39% median latency reduction, 33-52% cost savings depending on query distribution, and generation throughput of 122-200 tokens/second versus 50-64 for the baseline. The encoder classifier achieves 99.2% accuracy with near-perfect PII recall at 7ms inference overhead, establishing pre-inference classification as a practical path to compliance-by-design LLM deployment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
