Real-time Hiding of Physical Objects in Augmented Reality
Niclas Scheuing
Master's Thesis, August 2018
Supervisors: Dr. Stéphane Magnenat, Henry Raymond, Prof. Dr. Bob Sumner
Abstract
Image inpainting, the process of reconstructing missing pixels, has been intensely studied for almost two decades and many solutions to the problem have been proposed. Inpainting of videos has also been discussed, mostly in the context of post-processing and there are very few publications on real-time video inpainting. The widespread availability of mobile devices opens a new range of applications of image and video inpainting. Especially in the context of augmented reality, a solution to the video inpainting problem that runs in real-time on mobile devices is most useful for a wide range of applications.
We therefore propose a video inpainting algorithm and present a GPU implementation thereof. We embed this implementation into an augmented reality application that removes a tracked object in real-time and runs on iOS, Android and Windows. Our algorithm is based on the PixMix approach introduced by Herling and Broll, which is inspired by patch-matching algorithms, but matches individual pixels instead of entire patches. We use a gradient descent optimization to minimize a cost function composed of a term enforcing color consistency and one preserving texture patches.
In this thesis we will show that real-time video inpainting on mobile devices is feasible without any prior knowledge on the setting, provided that the computational power of the GPU is exploited. We demonstrate how this can be used in combination with image tracking to remove a known physical object.