Language Log

« previous post | next post »

EMO, by Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo from Alibaba's Institute for Intelligent Computing, is "an expressive audio-driven portrait-video generation framework. Input a single reference image and the vocal audio, e.g. talking and singing, our method can generate vocal avatar videos with expressive facial expressions, and various head poses".

As far as I know, there's no interactive demo so far, much less code — just a github demo page and an arXiv.org paper.

Their demo clips are very impressive — a series of X posts from yesterday has gotten 1.1M views already. Here's Leonardo DiCaprio artificially lip-syncing Eminem:

(There are plenty more where that came from — I uploaded that sample to YouTube because I was unable to persuade the html <video> element to modify the display width without distortion.)

I'm always skeptical of hand-selected synthesis demos, since we have no way of knowing how many problematic attempts were discarded, or what sorts of side-information might have been provided. Still, their examples are impressive. The arXiv.org paper explains:

To train our model, we constructed a vast and diverse audio-video dataset, amassing over 250 hours of footage and more than 150 million images. This expansive dataset encompasses a wide range of content, including speeches, film and television clips, and singing performances, and covers multiple languages such as Chinese and English. The rich variety of speaking and singing videos ensures that our training material captures a broad spectrum of human expressions and vocal styles, providing a solid foundation for the development of EMO. We conducted extensive experiments and comparisons on the HDTF dataset, where our approach surpassed current state-of-the-art (SOTA) methods, including DreamTalk, Wav2Lip, and SadTalker, across multiple metrics such as FID, SyncNet, F-SIM, and FVD. In addition to quantitative assessments, we also carried out a comprehensive user study and qualitative evaluations, which revealed that our method is capable of generating highly natural and expressive talking and even singing videos, achieving the best results to date.

…and provides this graphical overview:

Here's AI Mona Lisa lip-syncing Miley Cyrus covered by Yuqi — because the syllabic pace is slower, it's easier to visually evaluate the (generally excellent) phonetic lip/face synchrony:

Update — I was a bit puzzled by their account of the training set size: "over 250 hours of footage and more than 150 million images". If the images came from the standard digital video frame rate of 30 fps, that would be 250*60*60*30 = 27 million. The standard movie rate of 24 fps would yield a lower total. So did they actually use more "footage"? or did they do some interpolation? or did they multiply wrong?

The paper says in another section that

We collected approximately 250 hours of talking head videos from the internet and supplemented this with the HDTF [34] and VFHQ [31] datasets to train our models. As the VFHQ dataset lacks audio, it is only used in the first training stage.

That would explain the difference, but it means that the training was actually based on a lot more than "250 hours of footage".

Update #2 — There are quite a few recent experiments in similar directions, mostly following up on Wav2Lip (K.R. Prajwal, et al.,  "A lip sync expert is all you need for speech to lip generation in the wild", 2020), whose authors provided an interactive demo and source code.

For example, some recent work at NVidia provides an interesting set of videos comparing their  system's outputs to those several others.

February 29, 2024 @ 6:40 am · Filed by Mark Liberman under Computational linguistics, Phonetics and phonology

Permalink