AI 中文总结
PartInteractor是支持语音、草图、图像输入的部件感知三维创作框架，将语义部件层级作为可操控单元，提升生成后控制与意图一致性，助力XR中持续人机协同创作。
AI 中文摘要
随着扩展现实（XR）演变为沉浸式计算媒介，交互式三维创作对创意和功能工作流而言至关重要。然而，现有的生成式XR系统会产生缺乏显式语义结构的整体式输出，限制了生成后的控制能力。我们引入PartInteractor，这是一种从表示到交互的框架，研究如何将语义部件层级融入生成式XR创作，并将其作为可直接操控的一级单元，将一次性的“提示到物体”生成转变为持续的部件级协同创作。PartInteractor支持语音、草图和图像输入，将大语言模型（LLM）解释器与检索-生成策略相结合，在三维生成前对用户意图进行支撑。该系统不会生成整体式物体，而是生成带有显式部件层级的语义分解三维资产，支持对物体结构和组成进行丰富的部件级交互。我们的评估表明，部件感知表示提升了生成后的控制能力，减少了对整体物体重新生成的依赖；而意图支撑缓解了歧义，提升了意图与结果的一致性，共同支持更具表现力和可控性的人机协同创作工作流。这些结果凸显，部件感知表示和意图支撑是未来生成式XR创作系统值得关注的设计考量因素。
英文摘要
As Extended Reality (XR) evolves into an immersive computing medium, interactive 3D authoring becomes essential for creative and functional workflows. However, existing generative XR systems produce monolithic outputs lacking explicit semantic structure, limiting post-generation control. We introduce PartInteractor, a representation-to-interaction framework that investigates how semantic part hierarchies can be incorporated into generative XR authoring, and exposed as first-class, directly manipulable units, turning one-shot prompt-to-object generation into continuous component-level co-creation. PartInteractor supports speech, sketch, and image inputs, integrating an LLM interpreter with a retrieval-generation strategy to scaffold user intent prior to 3D generation. Instead of producing monolithic objects, our system generates semantically decomposed 3D assets with explicit part hierarchies, enabling rich component-level interaction over object structure and composition. Our evaluations suggest that part-aware representation increases post-generation control and reduces reliance on whole-object regeneration, while intent scaffolding mitigates ambiguity and improves intent-result alignment, together supporting more expressive and controllable human-AI co-creation workflows. These results highlight part-aware representation and intent scaffolding as promising design considerations for future generative XR authoring systems.
CommentsAccepted to ACM UIST 2026
