Do Deep Learning Models Generalize Facial Emotion Recognition in Different Age Groups?

Neusa Liberato Evangelista; Thiago Lopes Trugillo da Silveira

doi:10.22456/2175-2745.144479

Authors

Neusa Liberato Evangelista Serviço Federal de Processamento de Dados (SERPRO)
Thiago Lopes Trugillo da Silveira Universidade Federal do Rio Grande do Sul (UFRGS) https://orcid.org/0000-0001-6788-2667

DOI:

https://doi.org/10.22456/2175-2745.144479

Keywords:

FER, CNN, Pre-trained models fine-tuning, Age bias in FER datasets

Abstract

The prevalence of images of adults in facial emotion recognition (FER) databases is an evident bias in emotion classification. Although there are sets of FER images exclusively of children or adults aged 60 and over, the capture scenario is generally controlled and has staged emotions. Many studies in this area focus on increasing task performance by using a single data collection for training and testing; however, due to the diversity of individuals portrayed and photographs collected, this strategy generally fails when applied to new images or in real-world circumstances. In this paper, we assess the generalization potential of deep learning models that have been pre-trained in general-purpose databases such as ImageNet and refined using the databases CK+, DEFSS, FACES (60+), MUG, and NIMH-ChEFS with cross-databases in tests. Through this study, which assesses six different datasets involving three distinct age groups through cross-database evaluation using a clear experimental protocol, we were able to verify that models fine-tuned with data sets composed exclusively of children's images have accuracy above 80%, reaching 88% when used to predict images of children; models fine-tuned with images of young adults are more accurate in classifying young adults than children or adults aged 60 and over, and the dataset of adults aged 60 and over in FACES dataset demonstrated accuracy above 80% in predicting images from other datasets and 96% in predicting images within its own test set.

Downloads

Download data is not yet available.

References

[1] BHATTI, Y. et al. Facial expression recognition of instructor using deep features and extreme learning machine. Computational Intelligence and Neuroscience, v. 2021, p. 1–17, 2021.

[2] MIT MEDIA LABORATORY. Multimodal Affect Recognition in Learning Environments. p. 677–682.

[3] EDWARDS, C. et al. Human-machine communication: What does/could communication science contribute to HRI? In: ACM/IEEE International Conference on Human-Robot Interaction. [S.l.: s.n.], 2019. p. 673–674.

[4] SENIOR, A. W.; PANKANTI, S. Handbook of Face Recognition. 2. ed. London: Springer London, 2011. 699 p.

[5] LIU, P. et al. Facial expression recognition via a boosted deep belief network. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition. [S.l.: s.n.], 2014. p. 1805–1812.

[6] RAMIS, S. et al. A novel approach to cross dataset studies in facial expression recognition. Multimedia Tools and Applications, v. 81, p. 39507–39544, 2022.

[7] GOODFELLOW, I. J. Papers with Code - FER2013 Dataset. 2013.

[8] KHAN, R. A. et al. A novel database of children's spontaneous facial expressions (LIRIS-CSE). Image and Vision Computing - Elsevier, v. 83-84, p. 61–69, 2019.

[9] WANG, K. et al. A database for emotional interactions of the elderly. In: 2016 IEEE/ACIS 15th International Conference on Computer and Information Science (ICIS). [S.l.: s.n.], 2016. p. 1–6.

[10] CANAL, F. Z. et al. A survey on facial emotion recognition techniques: A state-of-the-art literature review. Information Sciences, v. 582, p. 593–617, 2022.

[11] LOPES, A. T.; AGUIAR, E. de; OLIVEIRA-SANTOS, T. A facial expression recognition system using convolutional networks. In: Conference on Graphics, Patterns and Images. [S.l.: s.n.], 2015. p. 273–280.

[12] TAMAYO-MONSALVE, M. A. et al. Coffee maturity classification using convolutional neural networks and transfer learning. IEEE Access, v. 10, p. 42971–42982, 2022.

[13] UTAMI, P.; HARTANTO, R.; SOESANTI, I. The EfficientNet performance for facial expressions recognition. In: 2022 International Conference on Electrical and Information Technology (IEIT). [S.l.]: Institute of Electrical and Electronics Engineers Inc., 2022. p. 756–762. ISBN 9781665455121.

[14] LOPES, A. T. et al. Facial expression recognition with convolutional neural networks: Coping with few data and the training sample order. Pattern Recognition, v. 61, p. 610–628, 2017.

[15] SAJJANHAR, A.; WU, Z.; WEN, Q. Deep learning models for facial expression recognition. Digital Image Computing: Techniques and Applications, p. 1–6, 2018.

[16] KHAIREDDIN, Y.; CHEN, Z. Facial Emotion Recognition: State of the Art Performance on FER2013. 2021.

[17] MOLLAHOSSEINI, A.; CHAN, D.; MAHOOR, M. H. Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter Conference on Applications of Computer Vision. [S.l.: s.n.], 2016. p. 1–10.

[18] AGRAWAL, A.; MITTAL, N. Using CNN for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy. The Visual Computer, v. 36, p. 405–412, 2020.

[19] WILSON, G.; COOK, D. J. A survey of unsupervised deep domain adaptation. ACM Transactions on Intelligent Systems and Technology, v. 11, n. 5, 2020.

[20] RUSSAKOVSKY, O. et al. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, v. 115, n. 3, p. 211–252, 2015.

[21] LUCEY, P. et al. The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops. [S.l.: s.n.], 2010. p. 94–101.

[22] MEUWISSEN, A. S.; ANDERSON, J. E.; ZELAZO, P. D. The creation and validation of the Developmental Emotional Faces stimulus set. Behavior Research Methods, v. 49, n. 3, p. 960–966, 2017.

[23] EBNER, N. C.; RIEDIGER, M.; LINDENBERGER, U. FACES—A database of facial expressions in young, middle-aged, and older women and men: Development and validation. Behavior Research Methods, v. 42, n. 1, p. 351–362, 2010.

[24] AIFANTI, N.; PAPACHRISTOU, C.; DELOPOULOS, A. The MUG facial expression database. In: 11th International Workshop on Image Analysis for Multimedia Interactive Services. [S.l.: s.n.], 2010. p. 1–4.

[25] EGGER, H. et al. The NIMH Child Emotional Faces Picture Set (NIMH-CheFS): A new set of children’s facial emotion stimuli. International Journal of Methods in Psychiatric Research, v. 20, n. 3, p. 145–156, 2011.

[26] GUO, Y. et al. SpotTune: Transfer learning through adaptive fine-tuning. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [S.l.: s.n.], 2019. p. 4805–4814.

[27] NGUYEN, D. et al. Meta-transfer learning for emotion recognition. Neural Computing and Applications, v. 35, p. 10535–10549, 2023.

[28] TAN, M.; LE, Q. V. EfficientNet: Rethinking model scaling for convolutional neural networks. In: 36th International Conference on Machine Learning. [S.l.: s.n.], 2019. p. 6105–6114.

[29] LYONS, M. J.; KAMACHI, M.; GYOBA, J. Coding Facial Expressions with Gabor Wavelets (IVC Special Issue). [S.l.]: Zenodo, 2020.

[30] YIN, L. et al. A 3D facial expression database for facial behavior research. In: 7th International Conference on Automatic Face and Gesture Recognition (FGR 2006). [S.l.: s.n.], 2006. p. 211–216.

[31] SIMONYAN, K.; ZISSERMAN, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. In: 3rd International Conference on Learning Representations (ICLR 2015). [S.l.: s.n.], 2015.

[32] SZEGEDY, C. et al. Rethinking the Inception Architecture for Computer Vision. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). [S.l.: s.n.], 2016. p. 2818–2826.

[33] CHEN, H.; HAOYU, C. Face recognition algorithm based on VGG network model and SVM. Journal of Physics: Conference Series, v. 1229, p. 012015, 2019.

[34] HUANG, G. et al. Densely connected convolutional networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). [S.l.: s.n.], 2017. p. 2261–2269.

[35] SANDLER, M. et al. MobileNetV2: Inverted residuals and linear bottlenecks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [S.l.: s.n.], 2018. p. 4510–4520.

[36] HE, K. et al. Identity mappings in deep residual networks. In: 14th European Conference on Computer Vision (ECCV 2016). [S.l.: s.n.], 2016. p. 630–645.

[37] HE, K. et al. Deep Residual Learning for Image Recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). [S.l.: s.n.], 2016. p. 770–778.

[38] GERON, A. Hands-on machine learning with Scikit-Learn and TensorFlow: concepts, tools, and techniques to build intelligent systems. Sebastopol, CA: O’Reilly Media, 2017. ISBN 978-1491962299.

[39] BROWNLEE, J. A Gentle Introduction to Pooling Layers for Convolutional Neural Networks. 2019. Disponível em: ⟨https://machinelearningmastery.com/pooling-layers-for-convolutional-neural-networks/⟩. Acessado em agosto de 2023.

[40] BARSOUM, E. et al. Training deep networks for facial expression recognition with crowd-sourced label distribution. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). [S.l.: s.n.], 2016.

[41] KAGGLE. 2024. Disponível em: ⟨https://www.kaggle.com⟩.

[42] NELSON, S. G. Emotion Recognition using Facial Expressions. 2023. Disponível em: ⟨https://www.kaggle.com/code/sandragracenelson/emotion-recognition-using-facial-expressions⟩.

[43] KHRYASHCHEV, V.; IVANOVSKY, L.; PRIOROV, A. Deep learning for real-time robust facial expression analysis. In: Proceedings of the International Conference on Machine Vision and Applications. Association for Computing Machinery, 2018. p. 66–70. Disponível em: ⟨https://doi.org/10.1145/3220511.3220518⟩.

[44] KERAS APPLICATIONS. Keras Applications. 2023. Disponível em: ⟨https://keras.io/api/applications/⟩.

[45] BIRHANE, A.; PRABHU, V. U. Large image datasets: A pyrrhic win for computer vision? In: 2021 IEEE Winter Conference on Applications of Computer Vision (WACV). Los Alamitos, CA, USA: IEEE Computer Society, 2021. p. 1536–1546. Disponível em: ⟨https://doi.ieeecomputersociety.org/10.1109/WACV48630.2021.00158⟩.

[46] BELLAMY, R. et al. AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias. IBM Journal of Research and Development, v. 63, n. 4/5, p. 4:1–4:15, 2019.

[47] MCSTAY, A. Emotional AI: The Rise of Empathic Media. London: SAGE Publications, 2018. ISBN 9781473971103.

[48] MCSTAY, A.; PAVLISCAK, P. Emotional Artificial Intelligence: Guidelines For Ethical Use. 2019. Disponível em: ⟨https://drive.google.com/file/d/1frAGcvCYv25V8ylqgPF2brTK9UVj_5Z/view⟩.