Monitoring students’ level of engagement during learning activities is an important challenge in the development of tutoring interventions. In this paper, we explore the feasibility of using electroencephalographic signals (EEG) as a tool to monitor the mental engagement index of novice medicine students during a reasoning process. More precisely, the objectives were first, to track students’ mental engagement evolution in order to investigate whether there were particular sections within the learning environment that aroused the highest engagement level among the students, and, if so, did these sections have an impact on learners’ performance. Experimental analyses showed the same trends in the different resolution phases as well as across the different regions of the environments. However, we noticed a higher engagement index during the treatment identification phase since it aroused more mental effort. Moreover statistically significant effects were found between mental engagement and students’ performance.
Endowing computer-based environments with the faculty of monitoring users’ experience is an important step toward improving human-computer interactions and understanding users’ needs [
In particular, EEG data provide valuable quantitative and unbiased information on brain activity in a millisecond time-frame. It has become a low-cost and a non-invasive tool that is increasingly being used in a wide range of applications, including: medical diagnosis (e.g. patient coma monitoring and epilepsy detection) [
Monitoring learners’ mental state is of primary interest particularly in computer-based learning environments [
In this paper, we recorded learners’ electrophysiological activity as they interact with a medical serious game in order to track their mental state and assess their learning performance. We aim first to monitor EEG signals in order to explore how students’ mental engagement evolve across the different phases of the learning environment. Second, we examine whether being engaged during particular phases of the game or paying attention to specific relevant areas of the environment have an impact on the learners’ outcomes.
The remainder of the paper is organized as follows: Section 2 outlines some previous work concerning students’ engagement assessment within computer-based learning environments. Section 3, describes our experimental setup. Section 4, discusses the obtained results and Section 5 presents a conclusion and future works.
The key to a successful learning experience is the ability for educators to provide adequate assistance to learners. In this context, many researchers are currently devoting a growing interest in assessing students’ mental engagement in an effort to keep students engaged during computer-based learning interactions [
The quantitative approaches are the most common measures. They are based on self-assessment tools such as questionnaires and surveys to be completed by the student. Self-report measures are used to assess students’ emotional and cognitive engagement [
Qualitative methods measure students’ engagement by means of various techniques such as observations, interviews and educators’ ratings [
Another alternative method to measure students’ engagement was observed during these last years, which is the use of physiological sensors. The prime advantage of using such sensing technologies in learning environments is that they can provide valuable quantitative data about the cognitive behavior of the learner, which cannot be directly observable. The use of physiological sensors such as skin conductance, heart rate and electroencephalography proved its effectiveness in monitoring changes in learners’ mental state [
Among all the existing sensors, researchers reported promising results when using EEG to measure students’ engagement. In fact, EEG can be a viable indicator of moment-to-moment changes in learners’ attention. In their tutoring system, Chaouachi and his colleagues (2015) automatically adapt the learning material according to learners’ attention and workload, measured using EEG signals. For instance, when the learner is disengaged or overloaded/underloaded, a worked example is given as a next activity in order to keep the learner engaged. In another study, EEG technology was used in an e-learning environment that detects individual mental effort using a mental state classification system [
In this paper, we propose to use EEG to track learners’ mental engagement while they are reasoning and solving different medical cases. We aim to assess how students’ engagement varies between the three phases of the problem-solving task and across the different areas of interest within the environment. Then we investigate whether learners’ engagement can have an impact on their reasoning outcomes.
Fifteen undergraduate medicine students (7 females) with a mean age of 21.8 ± 2.73 years ranging from 20 to 27 years participated in the experiment with a compensation of 20 dollars.
Upon arrival at the laboratory, participants were asked to sign a consent form explaining the experiment and the material. They were then outfitted with the Emotiv EEG headset and placed in front of the eye tracker to record respectively their brain activity and eye movements while interacting with our medical learning environment called Amnesia. During the session, 30 - 45 minutes were dedicated to the game and at the end, participants were invited to complete a post-game questionnaire in order to collect feedbacks about the ergonomics and usability of the game.
Amnesia is a learning environment developed for novice medicine students to assess their clinical abilities through different problem-solving tasks. The game features a virtual hospital where the player who plays the role of a doctor is mistakenly diagnosed with amnesia and trapped within the hospital. In order to get out of the trap, the students need to resolve first some cognitive tasks such as logic tests and second they have to prove their clinical skills by resolving six medical cases that were designed and validated with the help of a medical professional. Each medical case represents a different disease: flu, bacterial pneumonia, measles, Ebola, mumps and whooping cough.
The resolution task is divided into three phases: exploration, diagnostic and treatment and in each phase, they can also collect additional data such as analyses and antecedents to establish a diagnosis. The exploration phase represents the first part of the clinical case resolution process where the student should analyze the patient’s demographic information as well as his/her clinical data (e.g. symptoms, antecedents, etc.). Once all the useful data gathered, the objective of the diagnostic phase is to identify the correct disease from a list of six proposed ones. The student has up to three trials to find out the correct response. After completing the diagnosis, the student is shown a list of different treatments and he/she is asked to find out the adequate one(s). Participants are also given up to three attempts to discover the appropriate treatment.
Furthermore, in each medical case, we divided the resolution environment into different sections in order to analyze the level of engagement according to different areas of interest (AOI). For that purpose, we defined six specific sections, representing six task-relevant regions of the screen, as follows: Information (I), Antecedents (A), Symptoms (S), Analyses (N), Diagnosis (D) and Treatment (T). The I area of interest includes the demographic information of the patient (e.g. name, origin, weight and height). In the antecedents AOI, we introduce the diseases that the patient has had before (e.g. allergies). The S region includes all the symptoms related to the specific disease. In the analyses AOI, we present other clinical data (e.g. temperature, heart rate and blood pressure). The D area shows the different diagnoses offered to the student among which he/she has to choose the correct answer. Finally, the treatment AOI presents different proposals among which the student should select the appropriate treatments.
Participants’ brain activity was recorded using the Emotiv headset that contains 16 electrodes placed according to the 10-20 international standard [
The three frequency bands were extracted by multiplying one second of the EEG signal by a Hamming window and applying a Fast Fourrier Transform. Then, a combined value of the θ, α and β was computed by summing their values over all the 14 measured regions. Finally, as in [
In this study, EEG data were gathered and analyzed among all participants to investigate students’ mental state using the engagement brain index. The experimental results are mainly divided into three parts: 1) We assess the evolution of students’ brain activity during the three phases of the clinical problem-solving task (exploration, diagnostic and treatment), 2) we analyze the distribution of the engagement index across the different AOIs, and 3) we investigate the impact of the engagement level on performance.
Our first objective was to analyze how the engagement index evolves in the different phases of the resolution process across all participants, in order to investigate whether there was a particular period of time within the game that aroused the most attention. From
As to have detailed comparisons, we dressed a bar chart to study the variation of the engagement index across the exploration, diagnostic and treatment phases.
Case 1 | Case 2 | Case 3 | Case 4 | Case5 | Case 6 | |
---|---|---|---|---|---|---|
Number of participants who succeeded | 15 | 10 | 10 | 8 | 5 | 5 |
Number of total attempts | 47 | 36 | 39 | 31 | 13 | 18 |
Number of failed attempts | 22 | 16 | 21 | 19 | 3 | 9 |
with repeated measures. No statistically significant results (p = n.s) were found across the three phases suggesting that the engagement index is not related to a particular phase or a case. These findings underline the fact that the students were engaged in the same way through all the reasoning process from the exploration to the treatment identification.
In the previous analyses, it was found that the engagement index varies in a nearly similar way in all phases of the game all cases combined. In our next investigation, we performed further analyses to examine how the engagement index is distributed across the AOIs and similarly whether there is a specific area among those identified (i.e. Information, Antecedents, Symptoms, Analyses, Diagnosis and Treatment) that may have an impact on students’ engagement.
Preliminary statistical comparisons were made in terms of engagement index.
In the first case, the highest value was obtained for the Information area. Indeed, this corresponds to the first time the participants interacted with the clinical environment. That is why they were more engaged and focused to
Case 1 | Case 2 | Case 3 | Case 4 | Case 5 | Case 6 | |
---|---|---|---|---|---|---|
I | 67.26 (64.33) | 29.80 (7.47) | 46.03 (61.20) | 27.08 (13.56) | 34.02 (10.26) | 34.10 (8345) |
A | 61.55 (56.31) | 26.57 (4.53) | 32.24 (12.37) | 26.36 (8.69) | 29.11 (7.18) | 33.97 (13.99) |
S | 51.11 (43.88) | 34.76 (19.92) | 41.53 (33.92) | 31.06 (0.6) | 31.10 (9.62) | 44.39 (15.40) |
N | 46.75 (40.14) | 33.49 (22.96) | 29.53 (11.35) | 29.07 (10.97) | 34.72 (11.29) | 41.81 (20.79) |
D | 44.64 (34.37) | 36.25 (23.11) | 42.57 (35.37) | 28.29 (7.30) | 34.80 (14.58) | 40.15 (18.49) |
T | 40.73 (28.75) | 38.45 (19.86) | 48.96 (32.03) | 28.77 (7.94) | 35.69 (11.08) | 47.72 (15.36) |
understand the tutoring material compared to the other regions. The Antecedents area was the least engaging in almost all cases since it did not contain specific relevant material, except in the fourth medical case where the engagement index was high. Indeed, in order to correctly resolve this case and to be able to identify the appropriate disease, the students needed to pay a particular attention to the important information contained in this area. As in this case, the diagnosis was Ebola, the learners needed to focus on the Antecedents area to successfully identify the disease, which includes an important clue (“recently travelled to Guinea”) that could help them provide the correct answer. In the remaining areas, the Treatment AOI has sparked the highest engagement level. Indeed, once the diagnosis is made, the students had to establish the appropriate treatments for the identified illness. A list of six different treatments was presented to the participants, and they were asked to choose the right ones (two or three depending on the case). Hence, the students had to pay attention to all the suggestions in order not to make mistakes. As a result, this step required more concentration than the others especially that the students were informed that the game will be over after three errors.
Based upon the obtained results, one-way ANOVAs were performed to examine whether there were significant differences among all AOIs in terms of engagement index. Three main effects were found respectively in the first case (F (5, 1312) = 7.905, p < 0.01), the third case (F (5, 738) = 4.495, p < 0.01) and the fifth one (F (5, 292) = 2.559, p < 0.05). This suggests that statistically significant differences exist between the six regions of interest in terms of mental engagement.
Post hoc tests with a Bonferroni adjustment were run in each case to show which specific regions raised the highest attention. Results indicate that in the first case, the engagement level was statistically significantly higher in the Information area than in the Symptoms (p = 0.021), Diagnosis (p = 0.006) and Treatment (p = 0.000) areas. Significant differences were also found in case 3 for the Treatment area compared to the Symptoms (p = 0.010) and Analyses (p = 0.002). Finally, in case 5 the pairwise comparisons demonstrated no significant effect indicating that there were no differences between the AOIs. Indeed, in this case, and as shown in
These results demonstrated that there were no particular AOIs that aroused the most the students’ attention. These findings are contradictory with our previous work [
One way ANOVAs were first conducted to compare two groups of learners (group 1: success; group 2: failure) in terms of engagement level throughout the case resolution. For cases 2 and 5, the analysis was not conducted since the number of subjects was not fair between the two groups (all the participants succeeded in resolving the cases). For the remaining cases, the analysis yielded a main effect (p < 0.05) showing a significant difference of the students’ engagement index in both groups as depicted in
Moreover, within-subject repeated measure ANOVAs were performed to investigate whether there is a specific region that may have an effect on students’ performance. Yet, no significant differences (p = n.s.) were found between the AOIs in terms of success/failure over all cases suggesting that being engaged when focusing on a specific AOI did not have an impact on students’ performance.
These results confirm our hypothesis in [
Case 1 F(1, 1319) | Case 3 F(1, 742) | Case 4 F(1, 475) | Case 6 F(1, 222) | |
---|---|---|---|---|
p | 0.025 | 0.000 | 0.000 | 0.039 |
F | 5.019 | 70.976 | 81.630 | 4.334 |
no effect on learners’ performance. One can explain these results in different ways: the first reason can be that the fixated AOI may not contain important information, i.e. the student is admittedly engaged but not on the right direction, as is in case 4 where learners had to pay particular attention to the Antecedents area of interest. However, based on the statistical results, it is shown that this was the area with the lowest engagement index (see
To sum up, the first experimental analyses are in line with our assumptions. In case 1 and during the exploration phase, students were more likely to experience high mental engagement since it is the first medical case they had to resolve. Also, the displayed interface differed from the other scenes of the game, which makes them more attentive in order to understand how they would solve the case. No particular variation was identified for the other cases in terms of engagement index: the three resolution phases aroused almost the same level of attention. When solving the medical cases, participants needed to focus on particular regions of the screen. Therefore, we were interested to examine whether there was a specific area(s) of interest whither the mental engagement was high. Similar trends have been observed in most cases: learners were mostly engaged when focusing on the Treatment AOI. In fact, this part requires more mental effort knowing that for each identified disease in each medical case, two treatments at least are needed. On the other side, the Antecedents area is the least engaging, as it does not contain significant information to support the students’ reasoning process. Finally, in the second part of the experimental design, we developed the hypothesis that a change in the students’ engagement level across the fixated AOIs can have an impact on learners’ performance. Results showed statistically significant relationships between students’ performance and engagement index. Nevertheless, based on the remaining analyses we cannot conclude that there is a unique area that has an effect on students’ outcomes: i.e. being engaged when fixating an AOI, important or not, cannot necessarily lead to their success or failure in the medical case resolution. Indeed in clinical problem-solving tasks, all the steps of the resolution are mandatory. For instance, we cannot only be engaged when reading the symptoms and forget about the antecedents that may involve relevant information as well.
In this study, an experimental protocol was established to measure the mental engagement index of fifteen participants using EEG signals. Participants were shown six medical cases and asked to resolve them by identifying the correct diagnosis and treatment. The objective of this research was to draw a general overview of the students’ brain activity changes when reasoning in order to subsequently develop appropriate help strategies. Indeed, tracking learners’ mental engagement is very important especially in high-risk medical learning environments in order to constantly have an updated monitoring of their progress and level of knowledge acquisition.
First, we assessed the evolution of the engagement index across the different phases of the resolution process and the different areas of interest, with the aim to identify potential time frames or regions that could yield a high level of engagement. We found out that the first case aroused a high level of engagement especially during the exploration phase, which was the first scene exhibited to the participants. Then, we analyzed the relationship between mental engagement and learners’ performance. The results showed that engagement has overall a significant positive impact on students’ outcomes, however, we found that this relationship was not modulated by paying more attention to a particular section of the game or to a particular area of interest.
In our future research, we propose to incorporate eye tracking data with EEG in order to have a multimodal sensor-based assessment of students’ learning behavior. In the long term, we plan to provide novice medicine students with timely interventions to foster their analytical reasoning process according to both their mental and visual behaviors.
This work was supported by NSERC (National Science and Engineering Research Council) and SSHRC (Social Science and Human Research Council) through the LEADS project.
The authors declare no conflicts of interest regarding the publication of this paper.
Ben Khedher, A., Jraidi, I. and Frasson, C. (2019) Tracking Students’ Mental Engagement Using EEG Signals during an Interaction with a Virtual Learning Environment. Journal of Intelligent Learning Systems and Applications, 11, 1-14. https://doi.org/10.4236/jilsa.2019.111001