Embodied Interaction with Hand Gestures for AR Games

Abstract

Hand gestures can provide a natural way to interact with Augmented Reality (AR) and Virtual Reality (VR) systems. Previous work shows how we can use the expressiveness of our hands to retrieve and manipulate objects in AR and VR, for example by mimicking them. In our work, we explore what we call embodied hand gesture interaction: an interaction technique where the hand gestures represent objects in an embodied manner.

In the context of games, there are few examples of embodied hand gesture interaction, and most of them are played in VR. Therefore, our thesis aims to fill this research gap by exploring the use of embodied hand gesture interaction in AR games. Using a research through design approach, we designed and implemented three mobile AR games with different types of embodied hand gesture interaction: Dragon Snacks explores the use of direct embodied hand gesture interaction, Puppet Dance uses indirect embodied hand gesture interaction, and Ocean Clean-Up uses a vocabulary of gestures to interact with the game.

We conducted a qualitative user study and performed an inductive thematic analysis. Based on the five identified themes, we formulated design recommendations for AR games using em- bodied hand gesture interaction: 1) provide multimodal feedback about performed gestures and tracking information, 2) use meaningful hand gestures, 3) carefully balance the difficulty of the gestures and game mechanics, 4) consider arising physical discomfort due to the possibly unnatural use of the hands, 5) leverage the potential of AR. In conclusion, our results show that this new interaction method has the potential to provide a challenging and fun way to interact with AR games.