Interactive Avatars
This research project focuses on creating photorealistic digital humans that faithfully mirror real people. We combine high-resolution 3D face scans and performance captures retargeted to MetaHuman in Unreal Engine, and integrate real-time speech synthesis with voice cloning. A Large Language Model (LLM) drives interactive dialogue using a biography imported from our Virtual Biographer project, which includes memories and personality, to simulate the avatar’s brain.
Project Status
Ongoing
GTC Team
Henning Metzmacher, Fraser Rothnie, Carolina Ferrari, Prof. Robert W. Sumner, Dr. Fabio Zünd
Collaborators
Prof. Markus Gross (ETH CGL)
Introduction
Digital human avatars have become a central aspiration across interactive media, education, cultural heritage, and mixed reality. While offline pipelines in cinematic productions achieve striking visual quality, they do so without the constraints of live interaction. Our ambition is different: we target real-time, embodied interaction in which animation, rendering, perception, language understanding, and speech generation operate within tight latency budgets. This setting imposes a stringent research question: how far can we push realism, both visual and behavioral, while preserving real-time performance and the sense that the avatar mirrors a specific person rather than a generic character.
Approach
Constructing a digital human entails three orthogonal decisions. First, developers must choose between offline and real-time operation. Offline production tolerates expensive rendering and animation, while real-time systems must execute all tasks within a few milliseconds, and are therefore technically the most challenging. Second, one must decide between 3D mesh-based characters and image- or video-based avatars. The latter can achieve striking fidelity on 2D displays but are poorly suited to integration in interactive 3D worlds such as games, metaverse, or augmented- or virtual reality environments. Third, designers weigh stylized against realistic appearance. Stylization can avoid the uncanny valley, whereas realism demands higher precision and robustness.
Our project aims for real-time operation, 3D mesh-based representation, and a non-stylized realism goal aligned with mirroring a specific individual.
To create an avatar, we capture high-resolution facial geometry and textures, which are ingested into Unreal Engine’s MetaHuman framework. A set of facial performance takes is recorded and retargeted to the MetaHuman rig to offer realistic facial animations, which can be blended with lip animations. At runtime, incoming speech is detected and transcribed by an LLM, which is conditioned on the subject’s curated biography. The LLM then generates the response and triggers a voice-cloning synthesizer. Real-time lip synchronization and rendering are executed in Unreal. The runtime pipeline is optimized for low latency.