Scene-Aware AR Assistant

Abstract

Virtual assistants are currently available on almost every platform, however they generally are only disembodied voices, with no awareness of the user’s surroundings. Using Augmented Reality (AR) and Natural Language Processing (NLP) technologies, this thesis describes the implementation of an application that lets you have a conversation with an AR assistant. The assistant can interact with its surroundings by looking out the window and telling you the weather, setting reminders or sitting down when requested. We employ the language model GPT-4 to generate responses consisting of fitting sets of actions. With the help of a weather forecast API the assistant is able to answer questions about current data. We conduct a quantitative user study to assess the application’s usability, believability and feeling of co-presence. Further, we draw promising conclusions about the potentials and limitations of using a Large Language Model (LLM) in this context in future works.