The Optimization of Humanoid Robot’s Dialog in Improving Communication between Humanoid Robot and Older Adults ()
1. Introduction
As the aging society with low birth rate progresses [1] , elderly populations with dementia are increasing in Japan [2] [3] , as well as the increasing burden of care for the elderly with dementia [4] . This phenomenon is also observed in many developed countries all over the world [5] . Solitary aged-person increased [6] and many people experience loneliness and depression in old age, either as a result of living alone or due to lack of close family ties and reduced connections with their culture of origin, which results in an inability to actively participate in community activities [7] .
Less opportunities for elderly dialogue cause risk for developing dementia [8] . Therefore, supportive communication is essential to promote good quality dementia care. Also, person-centered approach focusing on supporting an older adult is critical to use and retain their skills and abilities [9] [10] . Nurses and other care workers are decreasing or are in short supply [11] [12] ; for this reason, “communication robots” have been introduced into nursing settings as an alternative for the shortage of nurses and other care workers to support caring for elderly [13] .
For examples, there are small animal type robots that can respond to voice and stroking [14] [15] , and mobile phones with sensors that can detect sound and touch, and can answer back [16] . These technologies have limitations that do not support the dialogue of older adults. Even if there is a development of the elderly communication support system such as remote communication using telenoid etc., and the autonomous interactive robot [17] [18] , this current speech recognition technology is not well-developed and could be challenging for some researchers [19] [20] due to its imperfect two-way dialogue.
Our research group is currently developing a humanoid nursing robot with caring functions (HNR) [21] [22] specifically for the functional and practical use of older adults with dementia.
2. Purpose
The purpose of this study was to clarify improvement points of the current function of a humanoid robot’s dialog pattern for improving optimal communication between humanoid robot and older adults.
3. Methods of This Study
3.1. Method
This descriptive study utilized the observation method. The older adults in a hospital and a Long-Term Care Health facility were the participants of this study. The researchers employed the dialogical programs which were installed in the humanoid robot, Pepper (SoftBank Robotics), including the application program Care Prevention Gymnastics Exercises for Pepper (Pepper-CPGE) that was made by Xing Company, Japan. Dialogues between older adult participants and Pepper-CPGE were recorded by video camera. In addition, the facial expressions and behaviors of older adults with dementia during dialogues with Pepper-CPGE were observed, and field notes were done. The collected data were the descriptions of the interactions, transcriptions of verbalizations, facial expressions, and behaviors. In the analysis, transcripts and video recordings were analyzed from the interaction scenarios.
3.2. The Theoretical Framework
The Transactive Relationship Theory of Nursing (TRETON) was used as a theoretical framework to observe communication between/among humanoid robot and older adults [23] [24] (Figure 1). The transactions between human beings and HNRs are perceived as a means of promoting interaction which helps to improve patitnts’ experience. In essence, the integration of HNR-to-human relationships and the science of human caring have been carefully considered within
![]()
Figure 1. Conceptual diagram of optimal communication between/among humanoid robot and older adults [23] [24] .
Tanioka’s TRETON. This theory envisages the mutual engagement and the technological engagement that transpire between that HNRs and the human being. This process involved in establishing interactions among older adults with dementia, the nurse as a mediator, and the HNRs.
This theory describes that HNRs need to have functions that correspond according to older adult’s mood, feeling, and emotion (e.g. anxiety, depression or suffering). This HNRs’ functions require these following elements such as verbal and nonverbal communication; facial expression; eye contact; knowing patient; listening empathically, and sharing patient’s lived experiences. These functions will be able to provide a caring as “healing process” to older adults.
3.3. Data Collection Period
Data collection period was in May, 2018.
3.4. Ethical Consideration
The data collection procedure was approved by the Ethics Committee of the University of Tokushima Hospital (application number 3046). In addition, approval was also obtained from Mifune Hospital Clinical Research Ethics Review Committee (application number 201180502) where the data collection and analysis were conducted. The purpose of the study and the methods used in data generation were explained to all participants and their guardians. Participants confirmed that they fully understood that their personal information were protected, and that reports were compiled and used only for research purposes. Informed consent to participate in the study and to be audio-visually recorded were obtained from every participant before the start of the study.
4. Findings
From the recorded scenes and conversations (Tables 1-3), the following were points to be further developed: 1) Intonation of the words vocalized by Pepper-CPGE was different what was expected by the older adult, so there was no appropriate response from the older adult; 2) The timing between Pepper-CPGE’s questions and the responses of the older adult were not timely and did not match (differences in the question-response time), therefore, the older adult exhibited confusion; and 3) Improving Pepper-CPGE’s ability to communicate with multiple older adults as an older adult’s dialogue with Pepper-CPGE can cause other older adults to also interact not only with the older adult that Pepper-CPGE is interacting with but also with Pepper-CPGE.
5. Discussion
5.1. Problem of the Intonation of the Word Spoken by Pepper-CPGE
Prior to starting a dialogue, the older adult said her name was “Mak-chan”. When the dialogue started, Pepper-CPGE called her “Mak-Chan-San”, however,
![]()
Table 1. Conversation contents (1).
![]()
Table 2. Conversation contents (2).
![]()
Table 3. Conversation contents (3).
the older adult could not understand the name and did not respond. For smooth communication, it is important to correctly recognize the other person, and it is essential to call the other person’s name correctly. The older adult was expecting Pepper-CPGE to call her “Mak-chan ⤵”, however, Pepper-CPGE called her respectfully with the name “Mak-chan-san”. Pepper’s vocalization sounded flat and mechanical, without intonation. In addition, “Mr./Ms. is like ‘San’ in Japanese” was automatically said after the name. It was concluded that the use of these words may pose problems and obstacles for older adults because they may fail to recognize that her name was called.
In this first scene, it was considered that the difference in intonation between the Pepper-CPGE and the older adult and the honorific name given to her, deviated from the expectation of the older adult, thereby causing delayed recognition. In communicating with older adults with dementia, it is important to successfully include and integrate verbal communication and non-verbal communication functions with declining cognitive function, along with the decline of cognitive function [25] . Therefore, it seemed effective to call older adults with dementia with the expression as desired by them. It was considered necessary to have a mechanism for Pepper-CPGE to repeat the name of the other person appropriately.
Also, from the perspective of occupational ethics, it is not appropriate to call the older adult “XX chan” or “grandma” [26] . Even if the older adult wishes to be called “XX chan”, in order to make function as a nursing caring robot, Pepper-CPGE needs to use appropriate words that emphasize the dignity of older adults. Therefore, it was considered that Pepper-CPGE needs a function to select (judge) whether or not to call a person using the proper (courtesy) title according to the level of cognitive function of the older adults.
5.2. Scenes Where the Timing between Pepper-CPGE’s Question and the Response of the Older Adult Did Not Match
When the older adult tried to speak or while talking, Pepper-CPGE spoke out of topic or the topic changed suddenly. The timing between Pepper-CPGE’s question and the response from the older adult did not match. Therefore, the older adult was confused. In Pepper-CPGE’s dialogue program, timeliness was deficient, causing inadequate interactive conversation.
In the previous study, the average time of “pause” that a human takes for a robot was about 2000 ms. The time required for a reply in the case of a question that needs to be considered was 1600 ms for male and 2200 ms for female [27] . The older adult with dementia often takes time to understand the other person’s words and reaching accurate and appropriate verbal responses [28] . Koike [29] pointed out that when care workers talk to patients with dementia, their timing should be slow as it is the characteristic way of nursing care staff when talking to a patient with dementia. In conversation with an elderly with dementia, a slow and polite way of speaking to attunement with the elderly is important [30] . For that purpose, it is necessary to have a particular function in a Pepper-CPGE machine that allows facial expressions recognition of the elderly and selective dialogue content is necessary. In addition, there is a need to look at adjusting the speed of conversation according to the pace of elderly. Moreover, when Pepper-CPGE changes the content of the story, it is important that continuity of communication with the older adult is ensured, such as confirming the content with the older adult like “May I move on to the next topic?”
Also, it is necessary to make Pepper-CPGE’s dialogue pattern from the current question interactive type to the discourse type speech pattern. In dealing with the characteristics of the older adults, it is important to encourage elderly patients to speak and observe their non-verbal cues to ensure that they are also listening. Therefore, instead of Pepper-CPGE asking people one after another, a mechanism is required that accepts elderly people, such as “nodding with a smile laughing” or “Oh, I see”, “Was that so” and encouraging speech. Furthermore, it is needed a dialogue program with a diagnostic function of dementia in natural conversation, such as “I am getting cold, what month is it today?” A dialogue program with the function of accurately diagnosing the cognitive function of elderly people in natural conversation was considered to lead to the effective evaluation of dialogue rehabilitation.
5.3. Scenes Where the Older Adult Who Was Around Joined the Dialogue with Pepper-CPGE
This scene exhibits the attraction of Pepper-CPGE to other persons, especially other older adults. When Pepper-CPGE asked, “Have you ever experienced strawberry picking?” The older adult A who was watching the dialogue with Pepper-CPGE responded to the word “strawberry picking” and loudly exclaimed, “Strawberries are very delicious.” Furthermore, the older adult B who was interested in the dialogue with Pepper-CPGE had a reaction to this occurrence that he was disturbed by A and was upset by person A’s behavior.
In this scene, it was affirmed that interaction and/or transaction occurred not only between Pepper-CPGE and the older adult with dementia, but also with other older adults who were attracted to the conversation with Pepper-CPGE in the surrounding area [23] . These conversations showed a bigger response than the usual dialogue may exist between patients and between nurses and patient. This seemed to be caused by having a strong interest in the dialogue between the older adults and the robot. In addition, it can be said that the older adult A who was watching the dialogue between Pepper-CPGE and her, recognized Pepper-CPGE as a conversation partner. As a result, the older adult A joined the dialogue. Unfortunately, Pepper-CPGE can only have one-on-one conversation with the target who did face recognition. If Pepper-CPGE has the ability to talk with multiple people, Pepper-CPGE can respond to surrounding reactions, spreading and deepening further conversation becomes possible, and it becomes possible to expand opportunities of speech of the older adults. It is important to develop a dialog pattern of humanoid robots that sympathize with older adults.
5.4. Limitations of This Study
In the first implementation of the Pepper-CPGE, it does not only assist older adults to increase communication, but also to be able to improve the whole person producing good outcomes related to health and well-being among the elderly group. However, the observation was only conducted on one occasion. Further observation and evaluation should be made in the intermediate phase and extended phase.
6. Conclusion
This study analyzed the interaction/transaction dialogues in the recorded scenes between older adults with dementia and Pepper-CPGE. It became clear that dialogues with Pepper-CPGE enhance transactive relations between humanoid robots and older adults. Furthermore, structured dialogue with the Pepper-CPGE promotes interaction not only with the older adult but also with surrounding older adults. This shows the development of transactive relations between humanoid robots and older adults. For a humanoid robot to communicate optimally with the older adults, it is necessary to create caring dialogue databases focused on knowing patients/clients. In essence, the structured dialogues from a humanoid robot provide not only interaction but also an aesthetic experience for older adults. Moving forward, it is essential to develop a dialog pattern that humanoid robots who express emotions (e.g. sympathy) that can be used with older adults.
Acknowledgements
This work was supported in part by JSPS KAKENHI Grant Number JP 17H01609.