[3D5-GS-2] Machine learning

[3D5-GS-2-03] Face Reenactment with Diffusion Model and Its Application to Video Compression

〇Wataru Iuchi¹, Yuya Umeda¹, Kazuaki Harada¹, Hayato Yunoki¹, Koki Mukai¹, Shun Yoshida¹, Toshihiko Yamasaki¹ (1. The University of Tokyo)

Keywords:diffusion model, video compression, face reenactment

With the advancement of information technology, the use of images and videos has become common. However, the capacity of storagedevices and communication bandwidth is finite, so the demand forcompression has been increasing. In addition to conventional frequency-based compression, deeplearning-based compression methods such as generative adversarial networks (GAN) have been emerging in recentyears. According to the existing FaR-GAN, a face image with a certainfacial expression can be reconstructed from a reference face image of a person andthe coordinate data of 68 landmarks representing the facial expression, which can be used forefficient facial image compression. However, suchexisting methods have problems in terms of reconstruction accuracy and smoothnessbetween frames.In this study, we propose a method that reconstructs an image from theprevious frame using a diffusion model recurrently for smooth inter-framerepresentation while optimizing the trade-off between person identificationand facial expression generation in diffusion model-based face imagereconstruction.