arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.18953cs.HC
收藏

评估人机自动化交互应用的前端与后端:Δ-EVAL 一个假设性基准

Evaluating Front-end & Back-end of Human Automation Interaction Applications Δ-EVAL A Hypothetical Benchmark

Gonçalo Hora de Carvalho

AI总结:

本文提出一个假设性基准框架Δ-EVAL,通过基于认知工程和AI评估范式的度量指标,统一评估人机自动化交互应用的前端与后端,以推动可复现、通用且面向未来的HAI系统设计。

摘要:

人因工程、认知工程与人机自动化交互(HAI)构成了一个三足鼎立的领域,其中用户与自主控制能力不断增强的技术系统占据核心位置。然而,强大的自主性伴随着重大的责任。正是在这一背景下,我们基于人工智能(AI)中已知的评估范式,提出了一组度量指标和一个基准框架。基准是指针对特定测试或任务所进行的一组测试以及在这些测试或任务上实施的度量或测量。我们假设了一系列可能设计的任务,用于评估操作员与系统之间的交互,以及 HAI 应用的前端和后端组件。其中,前端涉及用户界面以及用户与系统之间的直接交互,而后端则由支撑前端体验的底层流程和机制构成。通过基于认知工程中关于判断与预测研究的度量指标来评估 HAI 系统,我们尝试在一个基准中统一众多已知的 HAI 系统分类法和设计指南。为此,我们提供了一种结构化方法,以受近期人工智能基准测试技术快速发展启发的形式化方式,量化这些系统的有效性和可靠性。因此,我们尝试将设计原则导向一个可测试的基准,该基准能够产生可复现的结果,并且面向未来、具有通用性,同时在任意 HAI 应用的认知栈和技术栈层面都具有洞察力。

英文摘要:

Human Factors, Cognitive Engineering, and Human-Automation Interaction (HAI) form a trifecta, where users and technological systems of ever increasing autonomous control occupy a centre position. But with great autonomy comes great responsibility. It is in this context that we propose metrics and a benchmark framework based on known regimes in Artificial Intelligence (AI). A benchmark is a set of tests and metrics or measurements conducted on those tests or tasks. We hypothesise about possible tasks designed to assess operator-system interactions and both the front-end and back-end components of HAI applications. Here, front-end pertains to the user interface and direct interactions the user has with a system, while the back-end is composed of the underlying processes and mechanisms that support the front-end experience. By evaluating HAI systems through the proposed metrics, based on Cognitive Engineering studies of judgment and prediction, we attempt to unify many known taxonomies and design guidelines for HAI systems in a benchmark. This is facilitated by providing a structured approach to quantifying the efficacy and reliability of these systems in a formal way inspired by the recent fast developments in AI benchmarking techniques, thus, we attempt to guide designing principles towards a testable benchmark capable of reproducible results that is future-proof, general, and insightful both in the cognitive and technological stacks of any HAI application.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑