LLM-FACETS：一个保护隐私的评估LLM透明度和问责制的框架
LLM-FACETS: A Privacy-Preserving Framework for Evaluating LLM Transparency and Accountability
AI总结：
提出一个开源框架LLM-FACETS，通过浏览器界面和插件架构，为技术专家、领域专家和合规官员提供隐私保护的LLM评估，实现透明度与问责制。
摘要：
评估大型语言模型的输出是否事实准确、认知校准和方法可复现，是负责任AI部署的前提。然而，审计LLM对非技术从业者仍然难以实现：现有工具需要编程专业知识和非平凡的环境设置，云托管平台将评估数据传输到外部服务，为法律上负责AI监督的领域专家和合规官员设置了障碍。我们介绍LLM-FACETS（LLM事实交叉评估系统）：一个开源框架，具有浏览器可访问的界面和插件架构，围绕三个从业者画像（技术专家、领域专家、合规官员）构建，这些画像反映了EU AI法案和NIST AI风险管理框架中识别的利益相关者类别。该架构使数据流明确：确定性指标（BLEU、ROUGE、BERTScore）完全在自托管服务器内运行，无出站传输；LLM评判指标显式联系外部API，用户保留完全凭据控制。该框架通过三种机制实现透明度：用于认知不确定性的token级对数概率可视化、多评判共识以减轻评判偏差，以及RAG Triad指标（忠实性、答案相关性、上下文相关性）以检测和定位幻觉。插件架构允许在不修改评估管道的情况下集成任何新指标或数据集。开源实现支持针对同一属性的多个指标进行交叉检查，确保可复现性，并将AI问责制与评估系统的构建团队解耦。我们通过18个指标实现与规范参考库的交叉验证来验证该框架。
英文摘要：
Assessing whether Large Language Models outputs are factually grounded, epistemically calibrated, and methodologically reproducible is a prerequisite for responsible AI deployment. Yet auditing LLMs remains inaccessible to non-technical practitioners: existing tools require programming expertise and non-trivial environment setup, and cloud-hosted platforms transmit evaluation data to external services, creating barriers for domain experts and compliance officers legally responsible for AI oversight. We introduce LLM-FACETS (LLM FActuality Cross-EvaluaTion System): an open-source framework with a browser-accessible interface and a plugin architecture, structured around three practitioner profiles (technical experts, domain experts, compliance officers) that mirror the stakeholder categories identified in the EU AI Act and the NIST AI Risk Management Framework. The architecture makes data flows explicit: deterministic metrics (BLEU, ROUGE, BERTScore) run entirely within the self-hosted server with no outbound transmission; LLM-judge metrics contact external APIs explicitly, with users retaining full credential control. The framework operationalizes transparency through three mechanisms: token-level log-probability visualization for epistemic uncertainty, multi-judge consensus to mitigate judge bias, and RAG Triad metrics (Faithfulness, Answer Relevance, Context Relevance) to detect and localize hallucinations. A plugin architecture allows any new metric or dataset to be integrated without modifying the evaluation pipeline. The open-source implementation enables cross-checking across multiple metrics targeting the same property, ensuring reproducibility and decoupling AI accountability from the teams building the systems assessed. We verify the framework through cross-validation of 18 metric implementations against canonical reference libraries.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
