Personalized Scene-Aware Chatbots in AR

Abstract

The Game Technology Center at ETH Zurich is currently developing tools for creating intelligent characters that inhabit augmented reality worlds within their ongoing project Intelligent Content Creation Tools for Augmented and Virtual Reality. Our work aims to explore the potential application of Large Language Models to generate and enact interactive story experiences and to connect the Augmented Reality playspace to this virtual world using ICCTARVR’s scene understanding module. We present dynamic story generation, progression, and interaction in an interactive text-based adventure with an LLM-based virtual character serving as a gamemaster. We implement personalization of generated content using adjective-based personality descriptions. We conduct a qualitative user study to evaluate the produced systems and gather story, personalization, and experience assessments. The results demonstrate optimistic results towards the possibilities of LLM-based story generation and personalization but show certain limitations where more complex handling of story memory and consistency is needed. The study shows that the AR aspects were not developed enough, but we remark on the potential of future progression towards 3D-Asset generation and its potential for immersion.