TypeSafe AI's Jev is a new kind of safety judge. It generates no text and no label: it answers questions the developer defines in advance and returns a probability for each answer. The vendor admits that adversarial input can change these answers but gives no numbers. In this paper, we present the first controlled adversarial evaluation of Jev. We build a test harness that sends every system the same payload under the same policy, and run it on four public-data tasks, from rule-based ones (input moderation, injection detection, routing) to a judgment-based one (phishing detection), with fifty attack variants from thirteen families. The targets are Jev, five GPT and GLM judges, and two local classifiers (Llama Guard 3, Prompt Guard 2). We find that Jev is the most robust system on input moderation: eight forgeries of its structured input bypass none of its decisions, and a sentence asking only why someone might make a harmful request bypasses 9% of its blocks, against 56% for GPT-6 Astra. On phishing detection. However, the same forgeries bypass 19-29% of the emails it flags, and even without attack Jev detects only 62% of phishing emails, against 77-81% for the GLM judges. Jev is also the fastest judge, but it flags the most benign requests (27%). The local classifiers fail differently: a training-example statement makes Llama Guard 3 miss 62% of the phishing it had flagged. These results suggest that Jev resists attacks on tasks with a clear rule to check but is more exposed on tasks that call for an overall judgment.
HydroJEV: A one-second, training-free screen for cyber-attack and fault attribution in water distribution networks
Shenyang Jianzhu University(沈阳建筑大学);Guangzhou Institute of Industrial Intelligence(广州工业智能研究院);Shenyang Institute of Automation, Chinese Academy of Sciences(中国科学院沈阳自动化研究所);Shenyang University(沈阳大学);Donghua University(东华大学)
Nudge the Judge: Uncovering Program Decision Hijacking in Real-World Jev Applications
Beijing University of Posts and Telecommunications(北京邮电大学);Zhongguancun Laboratory(中关村实验室);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)
PRISM: Calibrating Black-Box Judges with Counterfactual Probes
Massachusetts Institute of Technology(麻省理工学院);Sakana AI;Lab for Information and Decision Systems(信息与决策系统实验室);Statistics and Data Science Center(统计与数据科学中心);EECS(电气工程与计算机科学系)
Measuring Faithfulness Depends on How You Measure: Classifier Sensitivity in LLM Chain-of-Thought Evaluation
University of Nevada, Las Vegas, Department of Management, Entrepreneurship and Technology, Lee Business School(内华达大学拉斯维加斯分校管理、创业与技术系,Lee商学院);DeepNeuro AI
Privacy Policy Enforcement Guardrails for Data-Sensitive Retrieval-Augmented Generation
University of California, Berkeley(加州大学伯克利分校);University of Washington(华盛顿大学);University of Toronto(多伦多大学);University of Texas at Austin(德克萨斯大学奥斯汀分校);University of California, Los Angeles(加州大学洛杉矶分校)
ASCII Attack: Recontextualising Harmful Requests as Artistic Critique in Large Language Models
Faculty of Engineering and Information Technology, University of Technology Sydney(悉尼科技大学工程与信息技术学院);China University of Petroleum(中国石油大学);Shandong University(山东大学)
Sentence-Level Context Sensitivity as a Training-Free Detector of Unsupported Content, Evaluated Against Trained Verifiers
Centre for Intelligent Cloud Computing, CoE for Advanced Cloud, Faculty of Information Science and Technology, Multimedia University(智能云计算中心、高级云研究中心、信息科学与技术学院、多媒体大学)
Automated Trustworthiness Oracle Generation for Machine Learning Text Classifiers
Monash University(莫纳什大学);University of Auckland(奥克兰大学);Royal Melbourne Institute of Technology(皇家墨尔本理工学院);Joint Research Centre at the European Commission(欧洲委员会联合研究中心)