Automated annotation of pedagogical dialogue is a high-stakes task where LLMs often fail without sufficient domain grounding. We present a domain-adapted RAG pipeline for tutoring move annotation. Rather than fine-tuning the generative model, we adapt retrieval by fine-tuning a lightweight embedding model on tutoring corpora and indexing dialogues at the utterance level to retrieve labeled few-shot demonstrations. Evaluated across two real tutoring dialogue datasets (TalkMoves and Eedi) and three LLM backbones (GPT-5.2, Claude Sonnet 4.6, Qwen3-32b), our best configuration achieves Cohen's $κ$ of 0.526-0.580 on TalkMoves and 0.659-0.743 on Eedi, substantially outperforming no-retrieval baselines ($κ= 0.275$-$0.413$ and $0.160$-$0.410$). An ablation study reveals that utterance-level indexing, rather than embedding quality alone, is the primary driver of these gains, with top-1 label match rates improving from 39.7\% to 62.0\% on TalkMoves and 52.9\% to 73.1\% on Eedi under domain-adapted retrieval. Retrieval also corrects systematic label biases present in zero-shot prompting and yields the largest improvements for rare and context-dependent labels. These findings suggest that adapting the retrieval component alone is a practical and effective path toward expert-level pedagogical dialogue annotation while keeping the generative model frozen.
How Real Is AI Tutoring? Comparing Simulated and Human Dialogues in One-on-One Instruction
Shanghai Institute of Artificial Intelligence for Education, East China Normal University（华东师范大学上海人工智能教育研究院）；Lab of Artificial Intelligence for Education, East China Normal University（华东师范大学人工智能教育实验室）
MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems
ETH AI Center（苏黎世联邦理工学院人工智能中心）；ETH Zurich（苏黎世联邦理工学院）；TU Darmstadt（达姆施塔特工业大学）；Hessian Center for AI (hessian.AI)（黑森人工智能中心）；National Institute of Education, Nanyang Technological University（南洋理工大学国立教育学院）
Rewarding How Models Think Pedagogically: Integrating Pedagogical Reasoning and Thinking Rewards for LLMs in Education
Chosun University（昌原大学）；Korea University（韩国大学）；Seoul National University（首尔国立大学）；Korea Institute for Curriculum and Evaluation（韩国课程与评价研究院）；Upstage；Indiana University Bloomington（印第安纳大学布卢明顿分校）；Nanyang Technological University（南洋理工大学）
How can we test whether state-of-the-art generative models, such as Blender and GPT-3, are good AI teachers, capable of replying to a student in an educational dialogue? Designing an AI teacher test is challenging: although evaluation methods are much-needed, there is no off-the-shelf solution to measuring pedagogical ability. This paper reports on a first attempt at an AI teacher test. We built a solution around the insight that you can run conversational agents in parallel to human teachers in real-world dialogues, simulate how different agents would respond to a student, and compare these c…
SafeTutors: Benchmarking Pedagogical Safety in AI Tutoring Systems
Eindhoven University of Technology（埃因霍温理工大学）；Indian Institute of Technology Kharagpur（印度理工学院Khargpur分校）；Cisco Research（思科研究）；Taras Shevchenko National University of Kyiv（塔拉斯·谢甫琴科基辅国立大学）；National Technical University of Ukraine（乌克兰国家技术大学）；New York University（纽约大学）
Sun Yat-sen University（中山大学）；Alibaba Group（阿里巴巴集团）
Generating fluent and informative responses is of critical importance for task-oriented dialogue systems. Existing pipeline approaches generally predict multiple dialogue acts first and use them to assist response generation. There are at least two shortcomings with such approaches. First, the inherent structures of multi-domain dialogue acts are neglected. Second, the semantic associations between acts and responses are not taken into account for response generation. To address these issues, we propose a neural co-generation model that generates dialogue acts and responses concurrently. Unlik…
University of Colorado Boulder（科罗拉多大学博尔德分校）；University of California Santa Cruz（加州大学圣克鲁兹分校）
In this paper, we introduce Dependency Dialogue Acts (DDA), a novel framework for capturing the structure of speaker-intentions in multi-party dialogues. DDA combines and adapts features from existing dialogue annotation frameworks, and emphasizes the multi-relational response structure of dialogues in addition to the dialogue acts and rhetorical relations. It represents the functional, discourse, and response structure in multi-party multi-threaded conversations. A few key features distinguish DDA from existing dialogue annotation frameworks such as SWBD-DAMSL and the ISO 24617-2 standard. Fi…
From Words to Wisdom: Discourse Annotation and Baseline Models for Student Dialogue Understanding
Department of Electrical and Computer Engineering, Tufts University（电气与计算机工程系，塔夫茨大学）；Department of Computer Science and Information Technology, Patuakhali Science and Technology University（计算机科学与信息技术系，帕图阿哈利科学与技术大学）；Department of Electrical and Computer Engineering, Computer Science and Biomedical Engineering, Tufts University（电气与计算机工程系，计算机科学与生物医学工程系，塔夫茨大学）；Department of Mechanical Engineering and Education, Tufts University（机械工程与教育系，塔夫茨大学）
Hong Kong University of Science and Technology（香港科技大学）
Dialogue policy transfer enables us to build dialogue policies in a target domain with little data by leveraging knowledge from a source domain with plenty of data. Dialogue sentences are usually represented by speech-acts and domain slots, and the dialogue policy transfer is usually achieved by assigning a slot mapping matrix based on human heuristics. However, existing dialogue policy transfer methods cannot transfer across dialogue domains with different speech-acts, for example, between systems built by different companies. Also, they depend on either common slots or slot entropy, which ar…
Microsoft Corporation（微软公司）；Beihang University（北京航空航天大学）
We study open domain dialogue generation with dialogue acts designed to explain how people engage in social chat. To imitate human behavior, we propose managing the flow of human-machine interactions with the dialogue acts as policies. The policies and response generation are jointly learned from human-human conversations, and the former is further optimized with a reinforcement learning approach. With the dialogue acts, we achieve significant improvement over state-of-the-art methods on response quality for given contexts and dialogue length in both machine-machine simulation and human-machin…
The recognition of emotion and dialogue acts enriches conversational analysis and help to build natural dialogue systems. Emotion interpretation makes us understand feelings and dialogue acts reflect the intentions and performative functions in the utterances. However, most of the textual and multi-modal conversational emotion corpora contain only emotion labels but not dialogue acts. To address this problem, we propose to use a pool of various recurrent neural models trained on a dialogue act corpus, with and without context. These neural models annotate the emotion corpora with dialogue act …
Neural abstractive summarization has been increasingly studied, where the prior work mainly focused on summarizing single-speaker documents (news, scientific publications, etc). In dialogues, there are different interactions between speakers, which are usually defined as dialogue acts. The interactive signals may provide informative cues for better summarizing dialogues. This paper proposes to explicitly leverage dialogue acts in a neural summarization model, where a sentence-gated mechanism is designed for modeling the relationship between dialogue acts and the summary. The experiments show t…
IBM-Research, Almaden（IBM阿尔马登研究中心）；UC Santa Cruz（加州大学圣克鲁兹分校）
In the last several years, Twitter is being adopted by the companies as an alternative platform to interact with the customers to address their concerns. With the abundance of such unconventional conversation resources, push for developing effective virtual agents is more than ever. To address this challenge, a better understanding of such customer service conversations is required. Lately, there have been several works proposing a novel taxonomy for fine-grained dialogue acts as well as develop algorithms for automatic detection of these acts. The outcomes of these works are providing steppin…
In this work, we investigate the human perception of coherence in open-domain dialogues. In particular, we address the problem of annotating and modeling the coherence of next-turn candidates while considering the entire history of the dialogue. First, we create the Switchboard Coherence (SWBD-Coh) corpus, a dataset of human-human spoken dialogues annotated with turn coherence ratings, where next-turn candidate utterances ratings are provided considering the full dialogue context. Our statistical analysis of the corpus indicates how turn coherence perception is affected by patterns of distribu…
AI Pedagogy: Dialogic Social Learning for Artificial Agents
University of Milan-Bicocca（米兰-比科卡大学）；Technical University of Munich（慕尼黑技术大学）；Institute for Cognitive Sciences and Technologies (ISTC-CNR)（认知科学与技术研究所（ISTC-CNR））
受维果茨基社会文化理论启发，本研究提出AI Social Gym动态环境，通过AI师生智能体间的二元教学对话进行本体获取，证明混合方向交互显著提升LLM知识获取与应用能力，为AI训练提供补充路径。
We present an annotation schema as part of an effort to create a manually annotated corpus for Arabic dialogue language understanding including spoken dialogue and written "chat" dialogue for inquiry-answer domain. The proposed schema handles mainly the request and response acts that occurs frequently in inquiry-answer debate conversations expressing request services, suggests, and offers. We applied the proposed schema on 83 Arabic inquiry-answer dialogues.
Foviatech GmbH（Foviatech有限公司）；University of Hamburg（汉堡大学）
Many socio-linguistic cues are used in conversational analysis, such as emotion, sentiment, and dialogue acts. One of the fundamental cues is politeness, which linguistically possesses properties such as social manners useful in conversational analysis. This article presents findings of polite emotional dialogue act associations, where we can correlate the relationships between the socio-linguistic cues. We confirm our hypothesis that the utterances with the emotion classes Anger and Disgust are more likely to be impolite. At the same time, Happiness and Sadness are more likely to be polite. A…
Medical Dialogue Generation via Dual Flow Modeling
The Hong Kong Polytechnic University（香港理工大学）；Southern University of Science and Technology（南方科技大学）；Research Institute of Trustworthy Autonomous Systems（可信自主系统研究院）
University of Hamburg（汉堡大学）；Foviatech GmbH（Foviatech 有限公司）
Many socio-linguistic cues are used in the conversational analysis, such as emotion, sentiment, and dialogue acts. One of the fundamental social cues is politeness, which linguistically possesses properties useful in conversational analysis. This short article presents some of the brief findings of polite emotional dialogue acts, where we can correlate the relational bonds between these socio-linguistics cues. We found that the utterances with emotion classes Anger and Disgust are more likely to be impolite while Happiness and Sadness to be polite. Similar phenomenon occurs with dialogue acts,…
University of California, Santa Cruz（加州大学圣克鲁斯分校）；IBM Research, Almaden（IBM阿尔马登研究中心）
Given the increasing popularity of customer service dialogue on Twitter, analysis of conversation data is essential to understand trends in customer and agent behavior for the purpose of automating customer service interactions. In this work, we develop a novel taxonomy of fine-grained "dialogue acts" frequently observed in customer service, showcasing acts that are more suited to the domain than the more generic existing taxonomies. Using a sequential SVM-HMM model, we model conversation flow, predicting the dialogue act of a given turn in real-time. We characterize differences between custom…
As an essential component in task-oriented dialogue systems, dialogue state tracking (DST) aims to track human-machine interactions and generate state representations for managing the dialogue. Representations of dialogue states are dependent on the domain ontology and the user's goals. In several task-oriented dialogues with a limited scope of objectives, dialogue states can be represented as a set of slot-value pairs. As the capabilities of dialogue systems expand to support increasing naturalness in communication, incorporating dialogue act processing into dialogue model design becomes esse…
