A central feature of many deliberative processes, such as citizens' assemblies and deliberative polls, is the opportunity for participants to engage directly with experts. While participants are typically invited to propose questions for expert panels, only a limited number can be selected due to time constraints. This raises the challenge of how to choose a small set of questions that best represent the interests of all participants. We introduce an auditing framework for measuring the level of representation provided by a slate of questions, based on the social choice concept known as justified representation (JR). We present the first algorithms for auditing JR in the general utility setting, with our most efficient algorithm achieving a runtime of $O(mn\log n)$, where $n$ is the number of participants and $m$ is the number of proposed questions. We apply our auditing methods to historical deliberations, comparing the representativeness of (a) the actual questions posed to the expert panel (chosen by a moderator), (b) participants' questions chosen via integer linear programming, (c) summary questions generated by large language models (LLMs). Our results highlight both the promise and current limitations of LLMs in supporting deliberative processes. By integrating our methods into an online deliberation platform that has been used for over hundreds of deliberations across more than 50 countries, we make it easy for practitioners to audit and improve representation in future deliberations.
AQuA -- Combining Experts' and Non-Experts' Views To Assess Deliberation Quality in Online Discussions Using LLMs
Heinrich Heine University Düsseldorf(杜塞尔多夫海因里希·海涅大学);Technical University Ilmenau(伊尔默瑙工业大学);Johannes Gutenberg University Mainz(美因茨约翰内斯·古腾堡大学);Technical University Dortmund(多特蒙德工业大学)
MemAudit: Post-hoc Auditing of Poisoned Agent Memory via Causal Attribution and Structural Anomaly Detection
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Qiyuan Tech(齐元科技);Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(北京大学计算机科学学院多媒体信息处理实验室);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院)
Inform, Coach, Relate, Listen: Auditing LLM Caregiving Support Roles
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);OSF HealthCare(OSF医疗集团);Indiana University Indianapolis(印第安纳大学印第安纳波利斯分校)
Fighting Numerical Hallucinations via Data-centric Compilation for Online Financial QA
Shenzhen Technology University(深圳科技大学);FiT, Tencent(腾讯金融科技部);South China University of Technology(华南理工大学);The Hong Kong University of Science and Technology(香港科学与技术大学);Huazhong University of Science and Technology(华中科技大学);McGill University(麦吉尔大学)