Virtual Mapping of Real-Objects for AR Experiences

Abstract

While Augmented Reality (AR) has matured over the recent years and is being used in many different applications these days, the field of realistic interaction with autonomous virtual characters in AR is largely unexplored.

The goal of this thesis is to gather information about the real scene surrounding the user. This information should then be usable to improve the interaction with autonomous virtual characters. For that, we aim to find instances of interactable real objects and virtually map them, such that the virtual characters can interact with these objects realistically. By getting the virtual characters to interact with the real world in a realistic way, the overall user experience will improve.

To accomplish this, we created a pipeline that detects and visualizes 3D bounding boxes of real objects using RGB, depth and depth confidence images as input. Additionally, the mesh inside the bounding boxes is also captured leading to a more precise virtual map of the real world objects. To make use of images captured from different angles, we proposed and implemented an approach for merging two observations of bounding boxes that belong to the same real object.

Finally, an evaluation shows that the performance of the object detection algorithms is influenced substantially by the blurriness of the input images, highlighting the importance of capturing images when the camera is stable.