Formal specifications play a central role in ensuring software reliability, yet automatically synthesizing high-quality specifications remains difficult and often requires domain expertise. Recent work has applied large language models to generate specifications in the Java Modeling Language (JML), reporting high verifier pass rates. But passing a verifier only confirms that an implementation is consistent with a specification, not that the specification is meaningful: a trivial postcondition such as ensures true satisfies any verifier while saying nothing about the code. Mutation-based completeness scores go further, but they return a single number and cannot tell a generator which clause to fix. In this work, we first compare classical and prompt-based JML synthesis under a unified setup, and find that 22.7% of the verified specifications have no postcondition at all, and that prompt optimization through verification feedback repairs weak prompts but does not go beyond the best hand-written one. We then introduce Spec-Harness, a framework that measures the behavioral adequacy of a specification along four dimensions of precondition and postcondition correctness and completeness, using Hoare-triple based symbolic verification and input/output mutation on systematically built test suites. Spec-Harness reveals that many verifier-accepted specifications are behaviorally weak, over- or under-constraining inputs and outputs in ways the verifier cannot see. Finally, we show that Spec-Harness works as a feedback signal that helps coding agents synthesize specifications with higher behavioral adequacy on held-out tests. This holds for the general-purpose agents Codex and Claude Code and for a CodeAct-style JML synthesis agent that we built to use the verifier and Spec-Harness as feedback.
Ecdysis: Efficient and Effective Training of Runtime Harnesses for LLM Agents
Chengdu Institute of Computer Applications, Chinese Academy of Sciences(中国科学院成都计算机应用研究所);University of Chinese Academy of Sciences(中国科学院大学);Beijing Institute of Technology(北京理工大学);Beijing University of Technology(北京工业大学);Yangtze Delta Region Institute of Tsinghua University, Zhejiang(浙江清华大学长三角研究院);Jiaxing Key Laboratory of Artificial Intelligence and Cyber Resilience(嘉兴人工智能与网络安全重点实验室)
Grow the Harness, Not the Context: From Strategy-Free Scaffolds to Reusable Specialist Agents
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);Shenzhen University of Advanced Technology(深圳理工大学);University of Macau(澳门大学)
More Programs or More Rolls? Separating Coverage from Specialization in LLM Harnesses
The Chinese University of Hong Kong(香港中文大学);Zhongguancun Academy(中关村学院);SINOPEC Research Institute of Petroleum Processing(中国石化石油化工科学研究院);Beijing Normal-Hong Kong Baptist University(北京师范大学-香港浸会大学联合国际学院);Guangzhou Nanfang College(广州南方学院)
SHE: Trajectory-driven Safety Harness Evolution for LLM Agents
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Fudan University(复旦大学);Shanghai Jiao Tong University(上海交通大学);The Hong Kong University of Science and Technology(香港科技大学)
From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws
State Key Laboratory of Complex System Modeling and Simulation Technology(复杂系统建模与仿真技术国家重点实验室);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);University of Chinese Academy of Sciences(中国科学院大学);School of Computer Science and Technology, Tianjin University(天津大学计算机科学与技术学院)