Towards Causal Effect Estimation of Emotional Labeling of Watched Videos
Keywords:affective computing, causal inference, pattern recognition, multimedia
AbstractEmotions play a crucial role in human life, they are measured using many approaches. There are also many methodologies for emotion elicitation. Emotion elicitation through video watching is one important approach used to create emotion datasets. However, the causation link between video content and elicited emotions was not well explained by scientific research. In this article, we present an approach for computing the causal effect of video content on elicited emotion. The Do-Calculus theory was employed for computing causal inference, and a SCM (Structured Causal Model) was proposed considering the following variables: EEG signal, age, gender, video content, like/dislike, and emotional quadrant. To evaluate the approach, EEG data were collected from volunteers watching a sample of videos from the LIRIS-ACCEDE dataset. A total of 48 causal effects was statistically evaluated in order to check the causal impact of age, gender, and video content on liking and emotion. The results show that the approach can be generalized for any dataset that contains the variables of the proposed SCM. Furthermore, the proposed approach can be applied to any other similar dataset if an appropriate SCM is provided.
KOELSTRA, S. et al. Deap: A database for emotion analysis; using physiological signals.IEEE Transactions on AffectiveComputing, IEEE, v. 3, n. 1, p. 18–31, 2012.
SOLEYMANI, M. et al. A multimodal database for affect recognition and implicit tagging.IEEE Transactions onAffective Computing, IEEE, v. 3, n. 1, p. 1–14, 2012.
ZHENG, W.-L.; LU, B.-L. Investigating critical frequency bands and channels for eeg-based emotion recognition withdeep neural networks.IEEE Transactions on Autonomous Mental Development, v. 7, n. 3, p. 162–175, 2015.
PEARL, J.Causality: Models, Reasoning, and Inference. Cambridge, England: Cambridge University Press, 2009.
LANZA, S. T.; MOORE, J. E.; BUTERA, N. M. Drawing causal inferences using propensity scores: A practical guide forcommunity psychologists.American Journal of Community Psychology, v. 52, n. 0, p. 380–392, 2013.
PEARL, J. et al.Causal Inference in Statistics: A primer. United Kingdom: Wiley, 2016.
NGUYEN, P. et al. Age and gender classification using eeg paralinguistic features. In:6th Annual International IEEEEMBS Conference on Neural Engineering. San Diego, CA, USA: IEEE, 2013. p. 1295–1298.
PUTTEN, M. J. A. M. van; OLBRICH, S.; ARNS, M. Predicting sex from brain rhytms with deep learning.NatureScientific Reports, v. 1, n. 8, p. 1–7, 2018.
PETRANTONAKIS, P. C.; HADJILEONTIADIS, L. J. Emotion recognition from eeg using higher order crossings.IEEETransactions on Information Technology in Biomedicine, v. 14, n. 2, p. 186 – 197, 2010.
PEREIRA, E. T. et al. Empirical evidence relating eeg signal duration to emotion classification performance.IEEETransactions on Affective Computing, p. 1–12, 2018.
JENKE, R.; PEER, A.; BUSS, M. Feature extraction and selection for emotion recognition from eeg.IEEE Transactionsof Affective Computing, v. 5, n. 3, p. 327 – 339, 2014.
BRADLEY, M. M.; LANG, P. J. International affective picture system. In: ZEIGLER-HILL, V.; SHACKELFORD, T. K.(Ed.).Encyclopedia of Personality and Individual Differences. Switzerland: Springer International Publishing, 2017. p. 1–4.
FOURATI, R. et al. Unsupervised learning in reservoir computing for eeg-based emotion recognition.IEEE Transactionson Affective Computing, p. 1–1, 2020.
PEREIRA, E. T.; GOMES, H. M. The role of data balancing for emotion classification using eeg signals. In:DigitalSignal Processing Conference. Beijing, China: IEEE, 2016. p. 1–6.
LIU, Y.-J. et al. Real-time movie-induced discrete emotion recognition from eeg signals.IEEE Transactions on AffectiveComputing, v. 9, n. 4, p. 550–562, 2017.
SONG, T. et al. Eeg emotion recognition using dynamical graph convolutional neural networks.IEEE Transactions onAffective Computing, p. 1–10, 2018.
ZHENG, W.-L.; ZHU, J.-Y.; LU, B.-L. Identifying stable patterns over time for emotion recognition from eeg. IEEETransactions of Affective Computing, v. 10, n. 3, p. 417 – 429, 2019.
HUNTER, M. et al. The australian eeg database.Clinical EEG and neuroscience, v. 36, n. 2, p. 76–81, 2005.
TOMESCU, M. et al. From swing to cane: Sex differences of eeg resting-state temporal patterns during maturation andaging.Developmental Cognitive Neuroscience, n. 31, p. 58–66, 2018.
VANDENBOSCH, M. M. L. J. Z. et al. Eeg-based age-prediction models as stable and heritable indicators of brainmaturational level in children and adolescents.Human Brain Mapping, Wiley, n. 40, p. 1919–1926, 2018.
HU, J. An approach to eeg-based gender recognition using entropy measurement methods.Knowledge-Based Systems,p. 1–8, 2018.
KAUR, B.; SINGH, D.; ROY, P. P. Age and gender classification using brain-computer interface.Neural Computing andApplications, Springer, v. 31, p. 5887–5900, 2019.
KAUSHIK, P. et al. Eeg-based age and gender prediction using deep blstm-lstm network model.IEEE Sensors Journal,p. 1–8, forthcoming.
SIDIROPOULOS, P. et al. Temporal video segmentation to scenes using high-level audiovisual features.IEEETransactions on Circuits and Systems for Video Technology, IEEE, v. 21, n. 8, p. 1163–1177, 2011.
SIDIROPOULOS, P.; MEZARIS, V.; KOMPATSIARIS, I. Video tomographs and a base detector selection strategy forimproving large-scale video concept detection.IEEE Transactions on Circuits and Systems for Video Technology, IEEE, v. 24,n. 7, p. 1251–1264, 2014.
TONOMURA, Y.; AKUTSU, A. Video tomography: An efficient method for camerawork extraction and motion analysis.In:International Conference on Multimedia. San Francisco, California, USA: ACM, 1994. p. 349–356.
APOSTOLIDS, E.; MEZARIS, V. Fast shot segmentation combining global and local visual descriptors. In:InternationalConference on Acoustics, Speech and Signal Processing (ICASSP). Florence, Italy: IEEE, 2014. p. 1–5.
MARKATOPOULOU, F.; MEZARIS, V.; PATRAS, I. Cascade of classifiers based on binary, non-binary and deepconvolutional network descriptors for video concept detection. In:International Conference on Image Processing (ICIP).Quebec City, QC, Canada: IEEE, 2015. p. 1–5.
VIOLA, P.; JONES, M. Rapid object detection using a boosted cascade of simple features. In:Conference on ComputerVision and Pattern Recognition. Kauai, HI, USA: IEEE Computer Society, 2001. p. 511–518.
SIMONYAN, K.; ZISSERMAN, A. Very deep convolutional networks for large-scale image recognition. In:InternationalConference on Learning Representations. San Diego, CA, USA: ICLR, 2015. p. 1–15.
MARKATOPOULOU, F.; MEZARIS, V.; PATRAS, I. Online multi-task learning for semantic concept detection in video.In:International Conference on Image Processing (ICIP). Phoenix, AZ, USA: IEEE, 2016. p. 1–5.
EATON, E.; RUVOLO, P. L. Ella: An efficient lifelong learning algorithm. In:International Conference on MachineLearning. Atlanta, Georgia, USA: PMLR, 2013. p. 507–515.
MARKATOPOULOU, F.; MEZARIS, V.; PATRAS, I. Implicit and explicit concept relations in deep neural networksfor multi-label video/image annotation.Transactions on Circuits and Systems for Video Technology, IEEE, v. 29, n. 6, p.1631–1644, 2018.
GALLES, D.; PEARL, J. Testing identifiability of causal effects. In:Conference on Uncertainty in Artificial Intelligence.Montreal, Quebec, Canada: ACM, 1995. p. 185–195.
PEARL, J.do-calculus revisited. In: FREITAS, N. de; MURPHY, K. (Ed.).Proceedings of the Twenty-Eighth Conferenceon Uncertainty in Artificial Intelligence. Corvallis, OR: AUAI Press, 2012. p. 4–11.
GONZALES, C.; TORTI, L.; WUILLEMIN, P.-H. aGrUM: a Graphical Universal Model framework. In:InternationalConference on Industrial Engineering, Other Applications of Applied Intelligent Systems. Arras, France: Springer, 2017.
BAVEYE, Y. et al. Liris-accede: A video database for affective content analysis.IEEE Transactions on AffectiveComputing, v. 6, n. 1, p. 43–55, 2015.
LEITE, N. M. N. et al. Deep convolutional autoencoder for eeg noise filtering. In:International Conference onBionformatics and Biomedicine (BIBM). Madrid, Spain: IEEE, 2018. p. 2605–2612.
BAREINBOIM, E.; PEARL, J. Causal inference and the data-fusion problem.Proceedings of the National Academy ofSciences, v. 113, n. 27, p. 7345 – 7352, 2016.
How to Cite
Copyright (c) 2022 Eanes Torres Pereira, Geovane do Nascimento Silva
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.Autorizo aos editores a publicação de meu artigo, caso seja aceito, em meio eletrônico de acordo com as regras do Public Knowledge Project.