Author(s)

Toshiya Akiyama¹, Kyoko Osaka², Hirokazu Ito³, Ryuichi Tanioka⁴, Allan Paulo Blaquera¹, Leah Anne Christine Bollos¹, Tetsuya Tanioka^5*

¹Graduate School of Health Sciences, Tokushima University, Tokushima, Japan.
²Department of Psychiatric Nursing, Nursing Course of Kochi Medical School, Kochi University, Kochi, Japan.
³Department of Nursing Art and Sciences, Tokushima University, Tokushima, Japan.
⁴Department of Physical Therapy, Hiroshima Cosmopolitan University, Hiroshima, Japan.
⁵Department of Nursing Outcomes Management, Institute of Biomedical Sciences, Tokushima University, Tokushima, Japan.

In this case study, we hypothesized that sympathetic nerve activity would be higher during conversation with PALRO robot, and that conversation would result in an increase in cerebral blood flow near the Broca’s area. The facial expressions of a human subject were recorded, and cerebral blood flow and heart rate variability were measured during interactions with the humanoid robot. These multimodal data were time-synchronized to quantitatively verify the change from the resting baseline by testing facial expression analysis, cerebral blood flow, and heart rate variability. In conclusion, this subject indicated that sympathetic nervous activity was dominant, suggesting that the subject may have enjoyed and been excited while talking to the robot (normalized High Frequency < normalized Low Frequency: 0.22 ± 0.16 < 0.78 ± 0.16). Cerebral blood flow values were higher during conversation and in the resting state after the experiment than in the resting state before the experiment. Talking increased cerebral blood flow in the frontal region. As the subject was left-handed, it was confirmed that the right side of the brain, where the Broca’s area is located, was particularly activated (Left < right: 0.15 ± 0.21 < 1.25 ± 0.17). In the sections where a “happy” facial emotion was recognized, the examiner-judged “happy” faces and the MTCNN “happy” results were also generally consistent.

Humanoid Robots, Multimodal Data, Emotion Analysis

Akiyama, T. , Osaka, K. , Ito, H. , Tanioka, R. , Blaquera, A. , Bollos, L. and Tanioka, T. (2023) Analysis of Emotions Using Multimodal Data: A Case Study. Journal of Biosciences and Medicines, 11, 54-68. doi: 10.4236/jbm.2023.1112006.