Machine learning (ML) components are increasingly integrated into software products, yet their complexity and inherent uncertainty often lead to unintended and hazardous consequences, both for individuals and society at large. Despite these risks, practitioners seldom adopt proactive approaches to anticipate and mitigate hazards before they occur. Traditional safety engineering approaches, such as Failure Mode and Effects Analysis (FMEA) and System Theoretic Process Analysis (STPA), offer systematic frameworks for early risk identification but are rarely adopted. This position paper advocates for integrating hazard analysis into the development of any ML-powered software product and calls for greater support to make this process accessible to developers. By using large language models (LLMs) to partially automate a modified STPA process with human oversight at critical steps, we expect to address two key challenges: the heavy dependency on highly experienced safety engineering experts, and the time-consuming, labor-intensive nature of traditional hazard analysis, which often impedes its integration into real-world development workflows. We illustrate our approach with a running example, demonstrating that many seemingly unanticipated issues can, in fact, be anticipated.
AssistantX: An LLM-Powered Proactive Assistant in Collaborative Human-Populated Environment
Department of Computer Science and Technology, Tsinghua University(计算机科学与技术系,清华大学);School of Artificial Intelligence, Beijing University of Posts and Telecommunications(人工智能学院,北京邮电大学)
When Reasoning Is Not Enough: Proactive Inquiry for Cross-Modal Safety in Multimodal Large Language Models
Institute of Artificial Intelligence (TeleAI), China Telecom(中国电信人工智能研究院(TeleAI));Peking University(北京大学);School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University(西北工业大学人工智能与光电子学院(iOPEN))
SafeMCP: Proactive Power Regulation for LLM Agent Defense via Environment-Grounded Look-Ahead Reasoning
Beijing Institute of Technology(北京理工大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院)
Better Adherence, Richer Context: A Field Evaluation of LLM-Powered Conversational Voice Diaries for Sleep
The Johns Hopkins University(约翰霍普金斯大学);Department of Psychiatry and Behavioral Sciences, The Johns Hopkins University School of Medicine(精神病学与行为科学系,约翰霍普金斯大学医学院)
LLM-Powered Detection of Price Manipulation in DeFi
The Hong Kong University of Science and Technology(香港科技大学);MegaETH;McGill University(麦吉尔大学);Nankai University(南开大学);Monash University(莫纳什大学);Southern University of Science and Technology(南方科技大学)
Tampere University(坦佩雷大学);University of Jyväskylä(于韦斯屈莱大学);University of Helsinki(赫尔辛基大学);Free University of Bozen Bolzano(博尔扎诺自由大学);University of South Eastern Norway(挪威东南大学)
Georgia Institute of Technology(佐治亚理工学院);Carnegie Mellon University(卡内基梅隆大学);The Hong Kong University of Science and Technology(香港科技大学);North Carolina State University(北卡罗来纳州立大学)