Generative Conditional Modeling for Motion

Motion and environment interactions are a complex set of actions that our brains solve easily. They are influenced by internal factors such as body proportions and shape, external factors such as furniture placement, and social factors such as place or time. We decipher all these conditions and know the course of action. However, encoding the same for a virtual character is much harder due to various reasons. Firstly, these conditions are not easily modeled as metrics to optimize. Furthermore, ambiguity in possible actions means that a recorded motion capture may not be the only feasible or correct motion at the time. Secondly, the set of future actions changes depending on current pose and environment interactions. For example, controlling the center of gravity becomes impossible if the character is in mid-air. Thirdly, modeling all of the joints and muscles in the human body can be very expensive. Hence, only a sparse subset of joints is used to represent human motion. As a result, some of the joints used can be an average of a number of the physical joints. This further adds ambiguity on appropriate joint-limits.

Current methods have attempted using adversarial losses [Aberman et al., 2020, Peng et al., 2021, Peng et al., 2022] and probabilistic modeling [Henter et al., 2020] in order to circumvent the metric definition and ambiguity problem. Environment based interactions have also been studied [Starke et al., 2019, Hassan et al., 2023].

While current state of the art models generate good unconditional motion, imposing conditions on the speed, position or style of the character has significant penalty on motion fidelity. This is compounded by the absence of a large homogeneous motion dataset. Unlike images and text datasets, multiple smaller animation datasets cannot be easily combined due to variations in body proportions. Homogenizing the animations to a uniform skeleton causes artifacts such as foot sliding or even completely changing the final animation.

In this project, we tackle both these problems: conditional modeling and homogenizing datasets. The first two work packages focus on conditional modeling for human poses and motion respectively. Conditioning on a secondary pose, sparse keyframes, style or velocity will be explored. The next project focuses on retargeting between different skeletal hierarchy and proportion. Finally, the last work package introduces alternative input modalities such as images, text or gestures to condition the generated motion.