The advent of large language models (LLMs) has paved the way for a new era of programming tools with both significant capabilities and risks, as the generated code lacks guarantees of correctness and reliability. Developers using LLMs currently face the difficult task of optimizing, integrating, and maintaining code generated by AI. We propose an embedded domain-specific language (DSL), Pythoness, to address those challenges. In Pythoness, developers program with LLMs at a higher level of abstraction. Rather than interacting directly with generated code, developers using Pythoness operate at the level of behavioral specifications when writing functions, classes, or an entire program. These specifications can take the form of unit tests and property-based tests, which may be expressed formally or in natural language. Guided by these specifications, Pythoness generates code that both passes the tests and can be continuously checked during execution. We posit that the Pythoness approach lets developers harness the full potential of LLMs for code generation while substantially mitigating their inherent risks. We describe our current prototype implementation of Pythoness and demonstrate that it can successfully leverage a combination of tests and code generation to yield higher quality code than specifications alone.
Towards Reliable LLM-Driven Fuzz Testing: Vision and Road Ahead
Institute of Information Engineering, CAS(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);University of Sydney(悉尼大学);Singapore Management University(新加坡管理大学)
HalluGuard: Demystifying Data-Driven and Reasoning-Driven Hallucinations in LLMs
CS Department Virginia Tech(弗吉尼亚理工学院计算机科学系);EECS Department MIT(麻省理工学院电子工程与计算机科学系);CS Department Dartmouth College(达特茅斯学院计算机科学系);Statistics Department Virginia Tech(弗吉尼亚理工学院统计学系);CS Department Michigan State University(密歇根州立大学计算机科学系)
JudgeAgent: Beyond Static Benchmarks for Knowledge-Driven and Dynamic LLM Evaluation
DataArc Tech Ltd.(DataArc科技有限公司);IDEA Research, International Digital Economy Academy(国际数字经济学院IDEA研究所);School of Advanced Interdisciplinary Sciences, UCAS(北京大学交叉学科学院);State Key Lab of AI Safety, Institute of Computing Technology, CAS(中国科学院计算技术研究所人工智能安全国家重点实验室)
From Parameters to Performance: A Data-Driven Study on LLM Structure and Development
School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院);School of Computer Science, Wuhan University(武汉大学计算机学院);School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机学院);Cognitive AI Lab, Shanghai Huawei Technologies, China(上海华为技术有限公司认知人工智能实验室)
Formula-R1: Incentivizing LLM Reasoning over Complex Tables with Numerical Computation via Formula-Driven Reinforcement Learning
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Nankai University(南开大学);Tsinghua University(清华大学);Shandong University(山东大学);Microsoft Research(微软研究院)
MeLA: A Metacognitive LLM-Driven Architecture for Automatic Heuristic Design
School of Computer Science, University of Nottingham Ningbo China(诺丁汉大学宁波分校计算机科学学院);College of Teacher Education, Zhejiang Normal University(浙江师范大学教师教育学院)
PANDA: An LLM-Enhanced Performance-Driven Analog Design Framework Bridging Design Intent and Layout Generation
School of Integrated Circuits, Peking University(集成电路学院,北京大学);Beijing Advanced Innovation Center for Integrated Circuits(北京集成电路先进创新中心);Institute of Electronic Design Automation, Peking University(电子设计自动化研究所,北京大学)
A Scalable Multi-LLM Collaboration System with Retrieval-based Selection and Exploration-Exploitation-Driven Enhancement
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);Fudan University(复旦大学);Shanghai Jiao Tong University(上海交通大学)