Evaluating the Causal Effect of Multimedia and Affective Temperament in Felt Emotion and Liking

Rafaela de Amorim Barbosa Silva; Eanes Torres Pereira

doi:10.22456/2175-2745.116017

Authors

Rafaela de Amorim Barbosa Silva Unidade Acadêmica de Sistemas e Computação, Centro de Engenharia Elétrica e Informática, Universidade Federal de Campina Grande https://orcid.org/0000-0003-2568-3218
Eanes Torres Pereira Unidade Acadêmica de Sistemas e Computação, Centro de Engenharia Elétrica e Informática, Universidade Federal de Campina Grande https://orcid.org/0000-0002-9717-794X

DOI:

https://doi.org/10.22456/2175-2745.116017

Keywords:

Affective Computing --- Causality --- Multimedia --- Affective Temperament

Abstract

In this paper, we propose an approach to evaluate the causal effect of videos on subjects who watched movies from the LIRIS-ACCEDE dataset and from whom the following information was collected: affective temperaments, gender, and electroencephalography (EEG) signals. The affective temperament was obtained by analyzing their answers to the AFECT questionnaire. Evidence was collected from specialized literature to design a Structural Causal Model to be subjected to Do-Calculus Causal Inference. Video concepts were extracted to characterize the major video content after k-means clustering. Information from 15 volunteers was analyzed and the effects of video content, affective temperament, and gender on emotion response and liking were computed. Higher Order Crossings (HOC) were extracted from EEG signals and the features were clustered and used as intermediate evidence of affective influence. This research provides answers for the following questions about the specific watched videos: (i) How does gender affect the felt emotion and liking? (ii) How does the affective temperament of a person affect felt emotion and liking? and (iii) How does the content of a video affect felt emotion and liking? The main contribution of this paper is in the proposed methodology which can be applied to any similar dataset to investigate the causal relationships of video content and affective temperament on the emotion of the audience.

Downloads

Download data is not yet available.

Author Biography

Eanes Torres Pereira, Unidade Acadêmica de Sistemas e Computação, Centro de Engenharia Elétrica e Informática, Universidade Federal de Campina Grande

Departamento de Sistemas e Computação

References

CISCO.Cisco Annual Internet Report. [S.l.], 2020. 1–35 p.

WANG, S.; JI, Q. Video affective content analysis: A survey of state-of-the-art methods.IEEE Transactions of AffectiveComputing, v. 6, n. 4, p. 410–430, 2015.

EKMAN, P. Universal facial expressions of emotions.California Mental Health Research Digest, v. 8, n. 4, p. 151–158,1970.

POSNER, J.; RUSSELL, J. A.; PETERSON, B. S. The circumplex model of affect: An integrative approach to affectiveneuroscience, cognitive development, and psychopathology.Development and Psychopathology, v. 17, n. 3, p. 715–734, 2005.

PEARL, J. Bayesianism and causality, or, why i am only a half-bayesian. In: CORFIELD, D.; WILLIAMSON, J. (Ed.).Foundations of Bayesianism. Germany: Springer International Publishing, 2001. p. 19–36.

PEARL, J.Causality: Models, Reasoning and Inference. 32 Avenue of the Americas, New York, NY 10013-2473, USA:Cambridge University Press, 2009.

LARA, D. R. et al. The affective and emotional composite temperament (afect) model and scale: A system-basedintegrative approach.Journal of Affective Disorders, v. 140, p. 14–37, 2012.

LARA, D.Temperamento e Humor: uma Abordagem Integrada da Mente. Brazil: Observat ́orio Gr ́afico, 2011.

KAUR, B.; SINGH, D.; ROY, P. P. Eeg based emotion classification mechanism in bci.Procedia Computer Science,v. 132, p. 752–758, 2018.

CHAKLADAR, D. D.; CHAKRABORTY, S. Eeg based emotion classification using correlation based subset selection.Biollogically Inspired Cognitive Architectures, v. 24, p. 98–106, 2018.

RAZUMNIKOVA, O. M. Gender differences in hemispheric organization during divergent thinking: an eeg investigationin human subjects.Neuroscience Letters, v. 362, n. 3, p. 193–195, 2004.

BARRY, R. J. et al. Age and gender effects in eeg coherence: I. developmental trends in normal children.ClinicalNeurophysiology, v. 115, n. 10, p. 2252–2258, 2004.

ALARCAO, S. M.; FONSECA, M. J. Emotions recognition using eeg signals: A survey.IEEE TRANSACTIONS ONAFFECTIVE COMPUTING, v. 10, n. 3, p. 374–393, 2019.

LANZA, S. T.; MOORE, J. E.; BUTERA, N. M. Drawing causal inferences using propensity scores: A practical guide forcommunity psychologists.American Journal of Community Psychology, v. 52, n. 0, p. 380–392, 2013.

PEARL, J.do-calculus revisited. In: FREITAS, N. de; MURPHY, K. (Ed.).Proceedings of the Twenty-Eighth Conferenceon Uncertainty in Artificial Intelligence. Corvallis, OR: AUAI Press, 2012. p. 4–11.

PEREIRA, E. T. et al. Empirical evidence relating eeg signal duration to emotion classification performance.IEEETransactions on Affective Computing, v. 12, n. 1, p. 154–164, 2021.

PETRANTONAKIS, P. C.; HADJILEONTIADIS, L. J. Emotion recognition from eeg using higher order crossings.IEEETransactions on Information Technology in Biomedicine, v. 14, n. 2, p. 186 – 197, 2010.

JENKE, R.; PEER, A.; BUSS, M. Feature extraction and selection for emotion recognition from eeg.IEEE Transactionsof Affective Computing, v. 5, n. 3, p. 327 – 339, 2014.

PEREIRA, E. T.; GOMES, H. M. The role of data balancing for emotion classification using eeg signals. In:2016 IEEEInternational Conference on Digital Signal Processing (DSP). [S.l.: s.n.], 2016. p. 555–559.

BAVEYE, Y. et al. Liris-accede: A video database for affective content analysis.IEEE Transactions on AffectiveComputing, v. 6, n. 1, p. 43–55, 2015.

MARKATOPOULOU, F.; MEZARIS, V.; PATRAS, I. Implicit and explicit concept relations in deep neural networksfor multi-label video/image annotation.IEEE Transactions on Circuits and Systems for Video Technology, v. 29, n. 6, p.1631–1644, 2019.

MARKATOPOULOU, F.; MEZARIS, V.; PATRAS, I. Deep multi-task learning with label correlation constraint forvideo concept detection. In:Proceedings of the 24th ACM International Conference on Multimedia. New York, NY, USA:Association for Computing Machinery, 2016. (MM ’16), p. 501–505.

APOSTOLIDIS, E.; MEZARIS, V. Fast shot segmentation combining global and local visual descriptors. In: . [S.l.: s.n.],2014.

SIDIROPOULOS, P. et al. Temporal video segmentation to scenes using high-level audiovisual features.IEEETransactions on Circuits and Systems for Video Technology, v. 21, p. 1163–1177, 08 2011.

GLYMOUR, C.; ZHANG, K.; SPIRTES, P. Review of causal discovery methods based on graphical models.Frontiers inGenetics, v. 10, p. 524, 2019.

BAREINBOIM, E.; PEARL, J. Causal inference and the data-fusion problem.Proceedings of the National Academy ofSciences, v. 113, n. 27, p. 7345 – 7352, 2016.