Wisdom from Diversity: Bias Mitigation Through Hybrid Human-LLM Crowds
Machine Learning Group, Université Libre de Bruxelles（布鲁塞尔自由大学机器学习组）；AI Lab, Vrije Universiteit Brussel（布鲁塞尔自由大学人工智能实验室）；FARI, AI for the Common-Good Institute, ULB-VUB（ULB-VUB共同利益人工智能研究所）；Center for Human-Compatible AI, UC Berkeley（伯克利人类兼容人工智能中心）
Axel Abels, Tom Lenaerts
备注：
Accepted for publication in the Proceedings of the 34th International Joint Conference on Artificial Intelligence (IJCAI 2025)
Despite their performance, large language models (LLMs) can inadvertently perpetuate biases found in the data they are trained on. By analyzing LLM responses to bias-eliciting headlines, we find that these models often mirror human biases. To address this, we explore crowd-based strategies for mitigating bias through response aggregation. We first demonstrate that simply averaging responses from multiple LLMs, intended to leverage the "wisdom of the crowd", can exacerbate existing biases due to the limited diversity within LLM crowds. In contrast, we show that locally weighted aggregation methods more effectively leverage the wisdom of the LLM crowd, achieving both bias mitigation and improved accuracy. Finally, recognizing the complementary strengths of LLMs (accuracy) and humans (diversity), we demonstrate that hybrid crowds containing both significantly enhance performance and further reduce biases across ethnic and gender-related contexts.
Wisdom of the AI Crowd (AI-CROWD) for Ground Truth Approximation in Content Analysis: A Research Protocol & Validation Using Eleven Large Language Models
University of Alcalá（阿尔卡拉大学）；Carlos III University of Madrid（马德里卡洛斯三世大学）
Explainable Fake News Detection With Large Language Model via Defense Among Competing Wisdom
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；Hong Kong Baptist University（香港浸会大学）；Guangdong Institute of Smart Education, Jinan University（暨南大学广东智慧教育研究院）
Sentiment Analysis of microblog feeds has attracted considerable interest in recent times. Most of the current work focuses on tweet sentiment classification. But not much work has been done to explore how reliable the opinions of the mass (crowd wisdom) in social network microblogs such as twitter are in predicting outcomes of certain events such as election debates. In this work, we investigate whether crowd wisdom is useful in predicting such outcomes and whether their opinions are influenced by the experts in the field. We work in the domain of multi-label classification to perform sentime…
In open-ended natural-language generation, existing text decoding methods typically struggle to produce text which is both diverse and high-quality. Greedy and beam search are known to suffer from text degeneration and linguistic diversity issues, while temperature, top-k, and nucleus sampling often yield diverse but low-quality outputs. In this work, we present crowd sampling, a family of decoding methods based on Bayesian risk minimization, to address this diversity-quality trade-off. Inspired by the principle of "the wisdom of the crowd," crowd sampling seeks to select a candidate from a po…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Harvard University（哈佛大学）；University of Chicago（芝加哥大学）
As political polarization in the United States continues to rise, the question of whether polarized individuals can fruitfully cooperate becomes pressing. Although diversity of individual perspectives typically leads to superior team performance on complex tasks, strong political perspectives have been associated with conflict, misinformation and a reluctance to engage with people and perspectives beyond one's echo chamber. It is unclear whether self-selected teams of politically diverse individuals will create higher or lower quality outcomes. In this paper, we explore the effect of team poli…
The Strategic Foresight of LLMs: Evidence from a Fully Prospective Venture Tournament
Ross School of Business University of Michigan（密歇根大学罗斯商学院）；NYU Stern School of Business New York University（纽约大学 Stern 商学院）；Kelley School of Business Indiana University（印第安纳大学凯利商学院）
Many educational technologies use artificial intelligence (AI) that presents generated or produced language to the learner. We contend that all language, including all AI communication, encodes information about the identity of the human or humans who contributed to crafting the language. With AI communication, however, the user may index identity information that does not match the source. This can lead to representational harms if language associated with one cultural group is presented as "standard" or "neutral", if the language advantages one group over another, or if the language reinforc…
M-GWAP is a multimodal game with a purpose of that leverages on the wisdom of crowds phenomenon for the annotation of multimedia data in terms of mental states. This game with a purpose is developed in WordPress to allow users implementing the game without programming skills. The game adopts motivational strategies for the player to remain engaged, such as a score system, text motivators while playing, a ranking system to foster competition and mechanics for identify building. The current version of the game was deployed after alpha and beta testing helped refining the game accordingly.
Dissecting vocabulary biases datasets through statistical testing and automated data augmentation for artifact mitigation in Natural Language Inference
