Emotion Dependent Domain Adaptation for Speech Driven Affective Facial Feature Synthesis August 26, 2020
Although speech driven facial animation has been studied extensively in the literature, works focusing on the affectivecontent of the speech are limited. This is mostly due to the scarcity of affective audio-visual data. In this paper, we improve the affectivefacial animation using domain adaptation by partially reducing the data scarcity. We first define a domain adaptation to map affectiveand neutral speech representations to a common latent space in which cross-domain bias is smaller. Then the domain adaption is used to augment affective representations for each emotion category, including angry, disgust, fear, happy, sad, surprise, and neutral, so that we can better train emotion-dependent deep audio-to-visual (A2V) mapping models. Based on the emotion-dependent deepA2V models, the proposed affective facial synthesis system is realized in two stages: first, speech emotion recognition extracts softemotion category likelihoods for the utterances; then a soft fusion of the emotion-dependent A2V mapping outputs form the affectivefacial synthesis. Experimental evaluations are performed on the SAVEE audio-visual dataset. The proposed models are assessed withobjective and subjective evaluations. The proposed affective A2V system achieves significant MSE loss improvements in comparison to the recent literature. Furthermore, the resulting facial animations of the proposed system are preferred over the baseline animations in the subjective evaluations.
When you visit a website, information is stored in your browser, mostly in the form of cookies. This information may be about you, your preferences or your device, and is often used to make the site work as you expect it to. The information does not usually identify you directly, it is meant to provide you with a more personalized web experience.
You can choose not to allow some cookies. Click on the different category headings to learn more and change our default settings.
These cookies are necessary for the website to function and cannot be turned off in our systems.
These cookies are used to provide insight into how we can improve our service to all our users and to understand how you interact with our website as an anonymous user.
These cookies are used to create your profile and provide ads relevant to your interests. It is also used to limit the number of times you see an ad, as well as help measure the effectiveness of the ad campaign.
Cookies are used to personalize content and ads, to provide social media features and to analyze our traffic. You can accept all cookies with the "Allow All" option or you can edit the settings with the "Customize Settings" option.