The Relation between Mental Workload and Face Temperature in Flight Simulation ()
1. Introduction
Mental workload is a crucial concept in cognitive psychology and human factors research, as it indicates the amount of cognitive effort and attention required to carry out a task [1] . It is a multifaceted concept that involves cognitive, perceptual, and motor processes. The Mental workload is crucial for human performance in different fields and includes a significant impact on the safety and health of workers [2] . Comprehending and controlling mental workload is essential for enhancing job efficiency, improving task performance, user experience, and minimizing the mistake rate [3] . Mental effort may be affected by many elements, such as user-friendliness, mistake avoidance, and consistency of the job or system being used [4] . Additionally, it is worth noting that professionals working in intensive care units may face negative consequences if their moral sensitivity is compromised due to an excessive mental workload [5] . The correlation between facial temperature and workload has attracted considerable interest as a result of the capability of thermal imaging to capture physiological and emotional reactions. The reliability of facial thermal imaging as a technique for obtaining facial temperature has been established through its independence from illumination conditions and skin color [6] . The study conducted by Ganesh et al. [6] , observed that emotional states and emotions might affect face thermal imaging, suggesting a possible connection between workload and facial thermal patterns. The aviation industry emphasizes understanding the cognitive workload of pilots to ensure the highest safety and optimal pilot performance. The relationship between workload and face temperature has attracted considerable attention within the air transportation domain. A potential tool for assessing cognitive workload and emotional states is facial thermal imaging, offering a real-time and non-invasive method for monitoring physiological responses [7] . Research has shown that changes in face temperature may be used as indicators of emotional arousal. Emotional states are associated with higher thermal compared to neutral states [8] . Thermal image analysis can accurately differentiate between individuals who are being deceptive and those who are not during cognitive tasks [9] . This suggests that thermal image analysis has the potential to be used in assessing the workload of pilots and non-pilots in aviation. Although the potential of using face thermal imaging to evaluate cognitive stress is clear, there are difficulties in extracting relevant features from facial thermal images, as pointed out by Wang et al. [10] . Liu et al. [11] emphasized the relevance of methodological concerns when using face thermal imaging to assess workload. They highlighted the necessity to segment facial areas of interest to enhance accuracy and pointed out the dependability of thermal image analysis. In addition, researchers have investigated the practicality of utilizing face skin temperature to categorize cognitive workload. This research suggests that thermal imaging might be included into aircraft systems to estimate degrees of cognitive stress [7] . To summarize, the compilation of existing research indicates that face thermal imaging has potential as a non-intrusive technique for evaluating cognitive workload and emotional states in flight. However, challenges such as standardization and representation of temperature distribution need to be addressed to fully leverage the potential of facial thermal imaging in workload assessment. However, challenges such as standardization and representation of temperature distribution, measuring the exact variation of facial temperature, and finding precise similarity between workload and facial temperature regions in detail need to be addressed to fully leverage the potential of facial thermal imaging in workload assessment. In order to completely use face thermal imaging as a valid technique for workload evaluation in aviation settings, it is crucial to conduct more research and make methodological breakthroughs. In previous research [12] we have established a correlation between workload and Electroencephalogram (EEG) measures. The objective of this paper is to correlate EEG measures and thermal imaging technology.
This is a novel approach to studying mental workload, which includes using thermal imaging technologies. Thermal images acquire the infrared radiation released by humans’ body, offering a distinct viewpoint on physiological reactions that might serve as indicators of mental activity. The relation between mental workload and face thermal imaging has the potential to deepen our comprehension of cognitive processes and improve the development of systems and settings that facilitate human performance.
2. Related Work
Research has shown that EEG analysis, together with other physiological indicators, may accurately categorize various degrees of exertion [13] . Moreover, the integration of EEG with other physiological signals has been used to evaluate workload, making it a viable device for measuring workload in authentic settings [14] . Furthermore, Rebsamen et al. [15] have suggested using EEG measures of spectral powers at various cortical sites to evaluate cognitive effort, highlighting the potential of EEG in workload assessment. In addition, EEG has been used to quantify brainwave activity in several situations, including the assessment of cognitive effort during mental arithmetic tasks [16] . The results emphasize the importance of EEG in measuring workload and its ability to provide useful insights into cognitive processes.
Studies assessing and monitoring mental workload are essential in transportation, such as subway train operations, to comprehend the influence of mental workload on operational performance [17] . Furthermore, the examination of mental workload has been conducted within the framework, revealing evidence that the combination of activities might result in cognitive overload and diminished performance [18] . Moreover, researchers have investigated the use of electroencephalography (EEG) to identify mental effort associated with multitasking, emphasizing the need of using objective metrics to comprehend cognitive workload [19] . The concept of cognitive workload theory has been put out as a theoretical framework to comprehend the cognitive strain, with a specific focus on the psychological capacity to process information [20] . Furthermore, researchers have investigated the connection between facial expression and mental stress in arithmetic tasks, revealing that increased mental workload might result in a higher frequency of lapses and mistakes [21] . The study of user interfaces has explored the optimization of web interface design by considering the cognitive load of users. This research highlights the significance of incorporating cognitive load in human-computer interaction [22] .
Hernández-Sabaté et al. developed a convolutional neural network for EEG features classification in different level of pilot mental workloads in the Cockpit during a continuous performance task test that assessed working memory and working memory capacity to some extent [23] .
Wang et al. [24] investigated the potential of using infrared facial thermography to assess mental workload in indoor thermal environments including slightly cool, neutral, and slightly warm. They segmented the facial region, including the forehead, nose, cheeks, ears, mouth, and neck. Their findings demonstrate that the relationship between face temperature and mental workload varies under various thermal environments. Significant connections are seen in the neutral environment and specifically in the areas of the ears, mouth, and neck.
The study by Hassoumi et al. [25] discusses the use of thermal imaging of the face as a method for detecting mental workload during flight simulation. Their research modulated cognitive workload by varying the difficulty of two landing scenarios and rest time. The results showed that changes in mental workload were reflected in the thermal patterns of the participants’ faces. Specifically, increased mental workload was associated with increased temperatures in the nose tip and nose area, and no significant change in forehead temperature was observed. In this experiment, they also did not measure workload precisely in real-time using EEG and, they only measured the variation of face temperature in three facial regions including nose tip, nose area, and forehead.
3. Physiological Measures
Physiological metrics are essential for comprehending physical activities and mental states. The use of these metrics is crucial for the examination of physiological reactions and cognitive functions [26] . The connection between physical and emotional experiences is shown by the correlation between empathy and the expression of emotional emotions via physiological reactions [27] . Moreover, the ability to perceive internal physiological signals, known as interoception, is linked to effective regulation of emotions, underscoring the importance of physical awareness in controlling emotional states [28] . Furthermore, physical gestures play a crucial role in social perception, underscoring the significance of comprehending physiological cues when reading emotions and social exchanges [29] .
· Heart Rate
Heart Rate (HR), measured through pulse or electrocardiogram (ECG), represents the count of heart beats per minute. It is a crucial physiological metric, offering insights into an individual’s health, such as current diseases or warnings about impending cardiac conditions [30] . Heart rate variability (HRV) is a non-invasive approach used to analyze the control of the autonomic nervous system on the heart [31] . Significant for assessing cardiac well-being [32] , HRV also serves as an indicator of stress levels and reflects autonomic nervous system influence [33] .
· Electroencephalogram (EEG)
The electroencephalogram (EEG) is a non-invasive technique that utilizes electrodes strategically placed on the scalp to quantify the electrical activity produced by the brain [34] . EEG is among the most precise and reactive measures which has been extensively used to evaluate mental effort by measuring brainwave activity. EEG has shown efficacy in deciphering degrees of human Workload and is regarded as one of the principal physiological indicators for assessing mental strain [35] [36] [37] .
· Facial temperature variation
Facial temperature variation is a relevant metric of both physiological and psycho-physiological measures. Studies have shown that face skin temperature serves as a physiological indicator that fluctuates in response to changes in skin blood flow, which is regulated by the activity of the autonomic nervous system [38] . Research has shown that face skin temperature demonstrates distinctive characteristics that are closely related to emotional arousal and concurrent measurements of typical physiological indicators of sympathetic activity [39] . Further, studies have shown that face skin temperature may be altered by psychological factors such as shame or pain, resulting in consistent fluctuations that can be seen using thermal imaging [40] . Furthermore, face skin temperature has been used as an indicator of stress and emotional reactions in multiple research investigations [41] . Moreover, Research has investigated the correlation between stress hormones, and face skin temperature. The findings reveal that stress may impact both facial temperature and emotional reactions [42] .
4. Experiment
An experiment was conducted aiming to gather in real time participants’ workload and facial thermal image during a takeoff procedure in an Airbus A320. It consisted of six different scenarios with varying weather, time, and conditions as well as the possibility of failure during the takeoff procedure. The first to third scenarios relate to standard takeoff sessions, while scenarios four to six relate to failure sessions. A participant monitor and a participant pilot are required to operate the Airbus A320. In this experiment, the participants served as pilots and the experimenter as monitor. The experiment was conducted at the University of Montreal and with participants who all signed written consent forms following an ethics certificate.
4.1. Participants
In this study, ten (10) individuals were recruited, ensuring a gender-balanced group with ages ranging between 25 and 35 years old. Before participating in the experiment, all participants received a briefing that explained the objectives and purpose of the study. Additionally, they were required to provide informed consent by signing consent forms, indicating their willingness to participate voluntarily.
Among the ten participants, five of them had prior experience and knowledge in piloting and aviation simulation. The remaining five participants did not have any previous exposure to aviation simulation. This division allowed us to have a balanced representation of both experienced and inexperienced individuals, which can help in drawing more comprehensive conclusions from the study.
4.2. Flight Scenarios
To simulate realistic flying circumstances, a variety of flight scenarios were utilized in the airplane simulator during the design and implementation of this experiment [12] . The scenarios varied in time, weather conditions, and whether a failure will occur. We will use scenarios one through three for the regular takeoff sessions and four through six for the failure sessions. Table 1 shows the details of different scenarios.
4.3. Procedure
The experiment environment was comprised of a participant as the pilot and the experimenter as the pilot monitor. Participants received a detailed description of the A320 takeoff procedure two weeks before their experiment to familiarize themselves with its handling characteristics. During this experiment, participants performed a sequence of flight simulations using the X-plane flight simulator. Meanwhile, equipment carefully measured and documented several physiological and cognitive data. The main goal was to investigate the correlation between mental effort and physiological reactions during flying duties. The experiment involved 10 participants, each exposed to a series of six distinct flying situations that are randomly replicated. In order to guarantee sufficient data gathering, every scenario was replicated at least twice during a one-hour testing session. This variety was intended to include a wide spectrum of cognitive demands and stresses that are associated with aviation activities and various flying circumstances.
Before starting the experiment, thorough calibration and setup procedures were performed on all the equipment. The monitoring platform included an X-plane logger to log the airplane simulator parameters, an EEG headset (electroencephalogram) for measuring mental workload, a heart rate monitor for detecting heart rate fluctuations, and an infrared camera that captured facial thermal images and temperatures at a frequency of five images per second. Consequently, a total of 18,000 thermal images and temperature data of each participant’s face were recorded. All devices continuously and non-stop capture and record data throughout the entire flight, including different flight scenarios and the participants’ rest time. To generate a more realistic cognitive workload for every scenario, participants were not made aware of the scenarios or the possibility of failure.
![]()
Table 1. An overview of the different scenarios [12] .
4.4. Heart Rate Monitoring
During the experiment, we used the Polar H10 to record the participants’ heart rates. The Polar H10 is a heart rate sensor mounted on a chest strap designed and manufactured by Polar.
4.5. EEG and Workload
In this experiment, we utilized the EEG headset from BMU, manufactured by OpenBCI, to measure the cognitive workload of the participants and accurately record real-time brain activity and emotions. The NCO software, also built by “BMU Augmented Intelligence”, enabled us to extract real-time cognitive workload data.
5. Infrared Camera and Thermal Image
Infrared cameras, often known as thermal or thermographic cameras, have the capability to detect and record infrared light. Infrared radiation is a kind of electromagnetic radiation that cannot be seen by the human eye. Infrared cameras function by detecting infrared radiation using specialized sensors capable of perceiving the thermal energy released by objects. This energy is then transformed into a visual representation that may be shown on a display or stored for further examination. The thermograms generated by the process exhibit temperature discrepancies represented by distinct hues or tints, enabling users to visually perceive heat distribution and see changes in temperature across items or displays within a given period.
Thermal imaging is capable of measuring and visualizing the temperature of an individual’s facial region. Thermal cameras have the ability to absorb the heat radiated from the surface of the skin. Consequently, the thermal picture produced may reveal discrepancies in temperature across various facial areas. The experiment used an ICI-7640 infrared camera and IR Flash software to extract csv files including facial temperature data.
5.1. Facial Temperature Extraction Process from Face Thermal Image
Figure 1 illustrates the sequential processing stages involved in converting a raw
![]()
Figure 1. Procedure of extract Temperature from face thermal image.
thermal image of the face into a temperature change graph for each individual point. First, the infrared camera records and saves unprocessed images at a resolution of 640 × 480. Afterwards, each of these images is subjected to processing in order to extract the CSV file that contains the temperature data for each individual pixel. Subsequently, the pictures undergo pre-processing to optimize face detection accuracy by converting the original raw data, resulting in four distinct types of thermal images. We employed a cascade function of all available thermal image types to detect the participants’ faces. Following the process of face recognition, we proceeded to extract the facial landmark points from the thermal picture. The precise locations of the eyes, nose, mouth, and other facial features are essential for accurately identifying and delineating the separate parts of a face. Next, we get the temperature associated with each landmark location and present its variations.
5.2. Thermal Image
A thermal image is a visual representation that illustrates fluctuations in temperature within a given scene. It uses a color gradient to represent various temperatures. Thermal images often represent warmer parts with warmer colors, such as red, orange, and yellow, whereas colder parts are shown with cooler colors, such as blue and purple. The raw image captured by the infrared camera is shown in Figure 2(a).
Figure 2(b) illustrates the color spectrum associated with the thermal image. In this scale, white represents the warmest (highest temperature), while black indicates the lowest temperature within the image (It’s important to note that this color spectrum is specific to the corresponding thermal photo and colors don’t necessarily represent the overall highest and lowest temperatures found in nature.
5.3. Thermal Image Color Palettes
Investigating specific areas of the face and accurately extracting signals from IRI data is difficult because of the presence of head motion artifacts. Harnessing its potential, therefore, relies on advancements in analytical techniques. Within this section, a comprehensive review has been conducted on different color palettes used in thermal imaging cameras to represent temperature of facial thermal images with the following (Figure 3).
1) Rainbow:
This palette uses a spectrum of colors, typically ranging from cool to warm colors like violet, blue, green, yellow, orange, and red, plus white(hottest) and black (coldest).
2) Gray
Represents temperatures using various shades of gray, with black for colder areas and white for warmer areas.
3) Inverse Gray
Like the gray palette but with reversed colors, where black represents warmer areas and white represents colder areas.
![]()
Figure 2. (a) Raw face thermal image (b) Color Temperature Chart for Infrared.
4) Iron
Utilizes a color palette mostly consisting of tones of black and white, complemented with nuanced hues of red and yellow, creating a visual likeness to hot iron.
5) Research
Customized color schemes designed for particular scientific, or research aims.
6) IR11
The thermal imaging color palette is a distinct set of 11 colors, with each hue corresponding to a certain temperature range.
7) IR16
Another specific color palette, consisting of 16 colors for more detailed temperature representation.
8) IR256
A high-resolution color palette with 256 colors, providing a finer level of detail in temperature representation.
9) Medical
Specifically designed for medical use, with a focus on detecting temperature variations for diagnostic reasons.
10) Medical16
A medical-specific color palette with 16 colors which is designed to enhance temperature differences for medical imaging purposes.
We utilized all 10 color palettes of thermal images to assess the capability of detecting faces. Each participant had a total of 18,000 images for each color palette, resulting in 180,000 thermal images for each participant (18,000 × 10 = 180,000). In total, for 10 participants, 1,800,000 thermal images were analyzed for facial detection. Following the analysis, we have selected the top four-color palettes—rainbow, iron, gray, and research—based on the obtained results for further investigation. Following that, a thorough evaluation and comparison study of these four kinds has been carried out to accurately locate face landmark points.
5.4. Face Landmark Points
Face landmark points are a set of specific, anatomically significant points detected on a human face. These points serve as crucial reference locations for various computer vision and facial analysis applications [4] . By pinpointing these landmarks, it becomes easier to analyze and understand facial expressions, emotions, and facial structures. Here’s a description of some commonly detected face landmark points [4] [5] :
· Left Eye (LE) and Right Eye (RE):
The LE and RE landmarks represent the corners of the left and right eyes, respectively. They are crucial for tracking eye movement and determining gaze direction.
· Left Eyebrow (LB) and Right Eyebrow (RB):
These landmarks denote the highest points of the left and right eyebrows. They help in identifying the eyebrow shape and its orientation.
· Nose Tip (N):
The Nose Tip landmark marks the tip of the nose. It assists in analyzing the nose’s angle and position on the face.
· Nose Base (NB):
The Nose Base landmark represents the bottom center of the nose. It helps in assessing the nose’s length and tilt.
· Left Mouth Corner (LMC) and Right Mouth Corner (RMC):
These landmarks identify the corners of the mouth. They are crucial for measuring the width of the mouth and analyzing expressions like smiles or frowns.
· Upper Lip Top (ULT) and Lower Lip Bottom (LLB):
These landmarks indicate the highest point of the upper lip and the lowest point of the lower lip, respectively. They are used to measure lip curvature and analyze expressions.
· Chin (C):
The Chin landmark represents the center of the chin and aids in understanding facial symmetry and structure.
· Left Cheek (LC) and Right Cheek (RC):
These landmarks denote the cheek’s highest points on the left and right sides of the face, assisting in analyzing facial contours.
· Left Ear (L_Ear) and Right Ear (R_Ear):
These landmarks indicate the top corners of the left and right ears. They can be useful in estimating the face’s orientation and pose.
· Forehead Top (FT):
The Forehead Top landmark marks the highest point on the forehead and can provide insights into the forehead’s size and shape.
5.5. Facial Muscles
The human face is a complex structure comprising several different muscles that play a crucial role in various functions, including appearance, movement, and facial expressions. In Figure 4, facial muscles are organized based on facial landmarks and indexes, accompanied by labels denoting the names of each muscle. The following are the facial muscles:
1) Frontalis: The frontalis muscle is located in the forehead region and its primary function is to raise the eyebrows and horizontal wrinkles on the forehead [43] .
2) Corrugator: The corrugator muscle is located between the eyebrows and is responsible for pulling the eyebrows down and toward the medial, resulting in a frowning expression [43] .
3) Procerus: The procerus muscle is a tiny muscle located between the eyebrows and its function is to help with the formation of wrinkles on the bridge of the nose [44] .
![]()
Figure 4. Facial muscles grouped from facial landmarks and indexes and muscle names [55] [56] .
4) Orbicularis Oculi: The Orbicularis Oculi muscle surrounds the eye and is primarily responsible for the closure of the eyelids, narrowing the eyes (squinting), and other eyes movements [45] .
5) Levator Labii Superioris: The Levator Labii Superioris muscle is responsible for raising the upper lip, thereby playing a role in facial expressions such as snarling or sneering [46] .
6) Nasalis: The Nasalis muscle consists of two components, namely the Transverse and Alar. Its primary function is to control the movement of the nostrils and dilating or compressing the nasal openings [47] .
7) Nose Tip: This term includes the several muscles located around the nose that they involve in expression and movement [48] .
8) Orbicularis Oris: The Orbicularis Oris muscle encircles the mouth and is responsible for the protruding of the lips and actions of closing, puckering, and kissing [49] .
9) Mentalis: The Mentalis muscle creates the formation of a wrinkled chin by raising and protruding the lower lip [50] .
10) Depressor Anguli Oris: The Depressor Anguli Oris muscle plays a role for the downward movement of the corners of the mouth, which contributes to a facial expression associated with frowning [51] .
11) Zygomaticus Minor and Major: Both muscles have a role in the facial expression of smiling. The zygomaticus major muscle raises the corners of the mouth, whereas the zygomaticus minor muscle assists in this movement [43] .
12) Buccinator: The buccinator muscle helps in blowing air, compressing the cheek, and performing actions such as whistling and sucking [52] .
13) Risorius: The risorius muscle is for laterally stretching the lips, which helps create a grinning smile [51] .
14) Platysma: The platysma muscle is in the neck, pulling down the corners of the mouth and tensing the neck’s skin [53] .
15) Masseter: The masseter is located in the jaw and is a muscle involved in the process of mastication, or chewing [54] .
16) Temporalis: The Temporalis muscle is another masticatory muscle located on the side of the head, contributing to the movement of the jaw [54] .
The comprehensive Table 2 presents a detailed subset list of facial landmarks that have been precisely identified within the defined facial muscle group.
5.6. Face Landmark Point in Face Thermal Image
Detecting face landmark points in thermal images can be challenging due to the absence of color information. In the context of face thermal images, face landmark point detection involves the precise localization of key facial features, such as the eyes, nose, and mouth, despite the absence of color information. Utilizing advanced computer vision techniques, including deep learning-based methods and pre-trained models fine-tuned on thermal face datasets, it becomes possible to identify and map the crucial landmarks on a person’s face in thermal images.
![]()
Table 2. A subset of facial landmarks in the defined facial muscle group [7] .
In the experiment, the utilization of 478 face landmark points proved to be a comprehensive and powerful approach for facial feature detection and analysis. By employing advanced computer vision techniques and state-of-the-art deep learning models, these abundant landmark points allowed for a highly accurate and detailed representation of facial structures and expressions in the dataset. The use of such a large number of landmark points provided valuable insights into the subtle variations and nuances of facial features, enabling precise measurements and comparisons. Figure 5 shows an example of the 478 face landmark points detected in a thermal image of a participant during the flight.
5.7. Mental Workload and Face Temperature
The relation between face thermal image temperature and workload is a subject that has been explored in some research studies, especially in the context of occupational health and safety. The idea behind this relationship is that an individual’s workload can impact their body’s thermoregulation, which may be reflected in the thermal patterns of their face. Increased mental workload can lead to heightened stress and cognitive load, which may trigger physiological responses in the body, including changes in facial temperature.
6. Results and Discussion
In this section, the Results and Discussion are presented, delving into the obtained findings, and providing a thorough analysis of their implications and significance.
6.1. Comparison of Thermal Image Types for Face Detection
The subsequent crucial stage involves facial detection. For this purpose, the four types of processed images are utilized to improve the accuracy of detecting and identifying facial features and key facial points. Table 3 and Figure 6, present the statistics of detected faces for each type of thermal camera face image. The results indicate that the “Gray type exhibited the highest number of face detections, while the Rainbow type showed the lowest number.
The illustrative example of Figure 7 reveals that in cases where only a segment of the face is visibly clear, the “Gray” type excels in face recognition quality, while other types fail to detect the face.
Figure 8 outlines a facial temperature detection subsystem that has been designed to ensure accurate temperature measurements with various types of thermal images. The subsystem involves three crucial steps: Face Detection, Extracting Face Landmark Points, and Extracting Temperature from these Points. The goal is to provide a comprehensive solution for accurately detecting faces and measuring face temperature using thermal imaging.
Step 1: Face Detection
The first step in the flowchart is Face Detection, a pivotal stage that initiates the entire process. To enhance accuracy, four different types of thermal images are utilized: Rainbow, Iron, Research, and Gray. Each image type serves as a unique perspective for detecting facial features. The process is as follows:
![]()
Table 3. Comparison of participants average face detection for each type of face thermal image.
![]()
Figure 5. Thermal Imaging Reveals Facial Landmark Points: An example of 478 face landmark points detected on a participant’s thermal image during a flight.
![]()
Figure 6. Comparison of average face detection for each type of face thermal image.
![]()
Figure 7. Comparison of face detection for each type of face thermal image on Head Movement.
![]()
Figure 8. Face detection cascade flowchart for face thermal image types.
1) Rainbow Thermal Image:
· If the Rainbow thermal image successfully detects a face, the system proceeds to the next step.
· If no face is detected, the system moves to the next thermal image type.
2) Iron Thermal Image:
· If the Iron thermal image detects a face, the system advances to the next step.
· If no face is detected, the system proceeds to the next thermal image type.
3) Research Thermal Image:
· Like the previous steps, if the Research thermal image detects a face, the system proceeds to the next step.
· If no face is found, the system moves on to the final thermal image type.
4) Gray Thermal Image:
· If the Gray thermal image detects a face, the system progresses to the next step.
· If no face is found, the system ignores that image.
Step 2: Extract Face Landmark Points
Upon successful face detection in Step 1, the system advances to the second step, which involves extracting facial landmark points. These points provide a detailed map of facial features necessary for accurate temperature measurement.
Step 3: Extract Temperature of the Points
The third and final step is to extract the temperature of the facial landmark points. Leveraging the information obtained in the previous steps, the system precisely measures the temperature at specific points on the face, ensuring a reliable and accurate assessment. Figure 8 depicts the cascade flowchart for achieving more accurate face detection, illustrating the process of measuring facial landmark points temperatures.
We performed this analysis on a subset of participants’ thermal face photographs, which were picked randomly. A landmark point was deemed accurately recognized if the discrepancy between the anticipated and actual position was within a range of five pixels. According to the conducted investigations on various thermal camera image types, the facial landmark point detection accuracy can be ranked in the following order: Rainbow, Iron, Research, and Gray. Among these, Rainbow and Iron demonstrate the highest accuracy, while Gray shows the lowest accuracy (Figure 9). When the participant’s face is directly facing the camera, the Rainbow and Iron types of thermal image achieved an accuracy 98.2% and 98.1%, respectively, in detecting facial landmarks points within a 5-pixel margin. On average, across all head and face orientations, the accuracy is 95%. The framework achieved a mean accuracy of 95%, meaning that 95% of all face points were accurately recognized within a 5-pixel margin of error compared to the ground truth in the test photos.
![]()
Figure 9. Comparison of face detection for each type of face thermal image on extracted Face Landmark points.
6.2. Data Preparation
Normalization is a data preprocessing technique that scales features to a standardized range, ensuring they are on a similar scale and improving the effectiveness of various data analysis and machine learning tasks [9] . In this study, the following list of normalizations is used for temperature of face landmark points (T), workload (W) and heart rate (HR):
· Robust Scaler
· Quantile Transformer
o Uniform output
o Gaussian output
· Power Transformer
o Yeo-Johnson
o Box-Cox
The results depicted in Figure 10 clearly demonstrate that Quantile Transformer with Gaussian output and Power Transformer with Yeo-Johnson transformer excel in normalizing the data and significantly improve the probability of accurate outcomes.
6.3. Time Series Correlations
Quantifying synchrony between time series data involves assessing the degree of similarity or correlation in their temporal patterns. There are several methods to measure synchrony, and the choice of the appropriate method depends on the specific characteristics of the data and the research question.
Dynamic Time Warping:
Dynamic Time Warping (DTW) is a technique used to calculate the optimal alignment path between two signals, minimizing the distance between them. Its primary advantage lies in its ability to handle signals of different lengths. Initially developed for speech analysis DTW calculates the Euclidean distance between each frame and every other frame to determine the minimum path that aligns the two signals [10] . However, a limitation is that it cannot handle missing values, necessitating interpolation beforehand if there are any gaps in the data points. By aligning and comparing the temporal patterns of facial temperature changes with corresponding workload variations (Figure 11), DTW accommodates differences in data lengths and captures temporal synchronizations, which are crucial for analyzing non-linear and time-delayed relationships.
In the present experiments, the intricate relationship between temperature changes in face landmark points and workload is investigated, leading to the unveiling of intriguing results. Through meticulous data collection and cutting-edge analysis techniques, particularly utilizing DTW, we have revealed a fascinating inverse correlation between facial temperature fluctuations and variations in workload levels. The experiments have revealed the presence of a millisecond delay between changes in workload and subsequent temperature responses in specific facial landmarks. These findings suggest that the physiological
![]()
Figure 10. Comparison of Normalization method results.
![]()
Figure 11. DTW Analysis of Workload and Temperature Variations of some Facial Landmark Points.
responses reflected in facial temperature changes may not be immediate but rather exhibit a slight time delay in response to shifts in workload demands.
Figure 12 is a heatmap illustrating the similarity between workload and facial muscles’ temperature across 10 participants and 17 facial muscles. The purpose of this analysis is to visually convey the relationship between the intensity of workload and the temperature of various facial muscles, quantified using the Pearson correlation coefficient. The color-coded heatmap highlights this relationship, ranging from blue to red. The metric employed to quantify this similarity is the Pearson correlation coefficient. The heatmap is color-coded to highlight the nature of the relationship between workload and facial muscles’ temperature. The color spectrum ranges from blue to red, with specific color associations. Blue hues signify an inverse relation, indicating that as workload increases, the temperature of the corresponding facial muscles decreases. On the other hand, red hues indicate a straight relation, suggesting that as workload intensifies, the temperature of the associated facial muscles also increases. The 10 participants underwent varying levels of workload, and the 17 facial muscles were monitored to observe how their temperatures responded to the workload. The intensity of the colors in the heatmap allows for a quick and intuitive interpretation of the overall patterns and trends in the relationship between workload and facial muscles’ temperature. The numbers within the heatmap indicate that, in each participant, changes in the temperature of 17 facial muscles corresponded to the workload. For instance, in participant 1, muscle number 7 exhibited the most inverse relationship with the workload. A larger value implies that the corresponding facial muscle showed no significant relationship with the workload. For example, in participant 1, the temperature changes in facial muscle No. 10 had the least correlation with workload. In summary, the numbers in the rank heatmap represent the degree of the relationship between temperature changes in each facial muscle and workload, with 1 indicating the strongest relationship and 17 indicating the weakest.
The heatmap (Figure 12) specifically highlights those facial muscles 7, 3, and 5 exhibit the highest inverse relation in their temperature changes concerning cognitive workload. This indicates that these specific facial muscles consistently exhibit dissimilarity in their responses to variations in cognitive demands across the 10 participants. In contrast, facial muscles 17, 16, 15, 10, and 8 are highlighted for their lowest similarity in temperature changes concerning cognitive workload. This implies that these facial muscles demonstrate less consistent or weaker correlations with cognitive demands. In addition, facial muscles 1 and 2, associated with the frontal areas of the face, exhibit a direct and minimal relationship with workload.
Figure 13 presents a Heatmap visualizing the Similarity of Workload and Nose Area Facial Muscles Temperature. The analysis includes 10 participants and considers facial landmark points temperatures in the Nose Area of specific facial muscles: facial muscle 7 (Nose tip), facial muscle 6 (Nasalis), and facial muscle 3 (Procerus). The color-coded heatmap is designed to depict the nature of the relationship between the temperature of these facial landmarks and muscles and the workload of the participants. In this visual representation, blue colors are indicative of an inverse relationship between the temperature of the facial landmark points and muscles and the workload, while red colors signify a straight (positive) relationship. The entire color spectrum employed in the heatmap ranges from light blue to dark blue, reflecting the strength and direction of the inverse correlation.
![]()
Figure 12. Heatmap Similarity of Workload and Facial Muscles Temperature-Ranked Muscles.
![]()
Figure 13. Heatmap similarity of workload and nose area temperature.
The metric used to quantify this similarity is the Pearson correlation coefficient, a statistical measure that assesses the linear relationship between two variables. In this context, the coefficient provides a numerical representation of the degree of correlation between the workload and the temperature of both the Nose Area facial landmark points and the specified facial muscles (7, 6, and 3). Given that the relationship between the nose area temperature and workload is identified as reverse, all colors in the heatmap fall within the blue range, spanning from light blue to dark blue. This consistent blue coloration emphasizes the prevailing pattern of inverse correlation between the temperature of both the Nose Area facial landmark points and the participants’ workload. Additionally, the figure provides valuable insights by pinpointing specific facial landmark points with the highest and lowest similarity. The highest similarity, as indicated by the Pearson correlation coefficient, is observed for facial landmark points 6, 122, and 196, implying a strong and positive correlation between the temperature of these specific facial landmark points in the Nose Area and the workload. On the other hand, the lowest similarity is found for facial landmark points 274, 275, and 4, suggesting a weaker correlation or a possible inverse correlation between the temperature of these facial muscles and the workload. In summary, the figure effectively communicates the intricate relationships between both the Nose Area facial landmark points temperatures (specified facial muscles), and the workload of participants. The color-coded heatmap and the quantitative measure of the Pearson correlation coefficient enhance the clarity of these complex associations.
Figure 14 provides a comprehensive overview of the changes in workload and the temperature of the face under scenario 5 (with Engine Failure after V1 speed). The red line represents the average temperature of the face, while the blue line represents the workload. Green and yellow dash vertical lines indicate the time of start and end of the scenario, pink dash lines mean engine failure time. Additionally, a dash blue line indicates the maximum workload. The figure illustrates how the workload increases upon starting the scenario and decreases after its end, leading to fluctuations in average face temperature. The plot depicts a weak inverse correlation between the average face temperature and workload, with a delay time. After one second following the engine failure (as indicated by the pink dash line), the workload reaches its highest value (Blue Point, indicating the moment of the highest workload). However, there is a noticeable delay before the average face temperature responds, subsequently reaching its lowest value (Red Point, representing the lowest average face temperature) during this period. This delayed response highlights the physiological dynamics of the face and its sensitivity to workload changes. Moreover, during the flight in scenario 5, the heart rate fluctuated, but overall, it increased gradually, reaching its peak after the engine failure. The average heart rate after the engine failure was higher than before the failure.
The heart rate increased multiple times before the plane’s engine failed because, in the participant’s previous experience with scenario 4, an engine failure occurred at a speed of 80 knots. The participant anticipated a similar failure in subsequent instances after surpassing the 80-knot threshold. Due to this, there has been increased stress, resulting in a higher heart rate. Additionally, the mental workload has been on the rise, accompanied by a decrease in facial temperature. On the other hand, in scenario 5, the engine failure occurs at a higher speed (V1 = 140 knots). At the time of the engine failure, the airplane is at high speed. Since the remaining distance to the end of the runway is not sufficient for braking, participants must skillfully control the plane and execute a takeoff, making this scenario difficult. As can be seen, following the engine failure, the heart rate suddenly increased.
Figure 15 illustrates the variations in workload and temperature of the nose area. The red line represents the temperature of the nose area, while the blue
![]()
Figure 14. Relation between workload and temperature.
![]()
Figure 15. Relation between Workload and Temperature of Nose Area, Nose tip, and Forehead.
line represents the workload. The vertical green dashed line indicates the start of the scenario, the pink dashed line denotes the occurrence of an engine failure, and the yellow dashed line indicates the end of the scenario. Additionally, the dash blue line signifies the maximum workload experienced during the scenario. The purple line represents the temperature changes of the face Frontalis muscle (Face Muscle number 1), which corresponds to the forehead, showing that the forehead temperature remains relatively stable and does not show any significant correlation with workload. In contrast, the green line, representing the temperature changes of face muscle 7 (Nose tip), exhibits an inverse relationship with workload. As the workload increases, the temperature of the nose tip decreases, suggesting a noteworthy response in this specific area during high workload conditions. Additionally, the red line depicts temperature changes associated with the average temperature of facial muscles, including Procerus, Nasalis, and Nose Tip (corresponding to numbers 3, 6, and 7, respectively), highlighting the most inverse relationship with workload.
6.4. Limitations
The study’s limitations encompass various facets, including potential experimental constraints and technology (hardware and software)-related errors that could impact the accuracy of results. Individual factors, such as differing skill levels among participants, pose challenges to the generalizability of findings, especially concerning their applicability to licensed pilots. Environmental factors, notably the absence of authentic physical sensations in the simulation, may introduce variations in participant responses compared to real-flight scenarios. Time constraints and ethical considerations, particularly safety restrictions, further influence the research’s scope, guiding the extent to which certain scenarios can be realistically replicated and studied. These limitations collectively underscore the need for a cautious interpretation of the study’s outcomes and suggest areas for potential improvement in future research endeavors.
7. Conclusions
With the increase of traffic piloting operations are an important subject of securing. The augmentation of flow of information leads to an increase of pilot’s workload with dangerous consequences of possible confusion and mistakes. In a previous study we have highlighted the links between the workload and the brainwaves measured by EEG activity. In the same context of aviation, our present study aimed to explore the relationship between face temperature and workload. The idea behind this relationship is that an individual’s workload can impact their body’s thermoregulation, which may be reflected in the thermal patterns of their face. Increased mental workload can lead to heightened stress and cognitive load, which may trigger physiological responses in the body, including changes in facial temperature.
We conducted the experiment with 10 participants, in six different aviation takeoff scenarios including normal and emergency situations, at least two flights for each scenario, totally taken 120 hours takeoff, and one hour per participant collect (9 hours totally) time series data without stop continuously including workload, heart rate, and facial thermal images and temperatures to examine the robustness of our results and to ensure the applicability of our findings in real-world aviation contexts. Our experiment conducted the relation between workload and face muscles and landmark points temperature during takeoff.
We found that face temperature fluctuations correlated with the increasing workload experienced by participants during extended flight durations. Monitoring pilots’ face temperature in takeoff, landing and long-haul flights could help identify fatigue and stress, prompting timely interventions to pilot training, ensure aviation safety and well-being. By understanding the factors influencing workload, aviation authorities and training institutions can develop more effective strategies to support pilots and enhance aviation operations in diverse scenarios.
As future work, the ongoing project aims to integrate the implemented and developed machine learning model to focus on calculating workload based on non-invasive physiological data. This model, designed to leverage the collected EEG, heart rate, eye-tracking, and facial thermal imaging data, holds the potential to provide a nuanced and real-time assessment of cognitive workload in aviation scenarios. By incorporating machine learning, we anticipate enhancing the accuracy and reliability of workload predictions, ultimately contributing to a more robust understanding of pilots’ cognitive states. The experimentation process is currently underway, involving continuous refinement of the machine learning model. This phase includes gathering additional data to expand the model’s training dataset and evaluating its performance across a spectrum of scenarios and individual differences. As the number of participants in the experiment increases, an improvement in discerning the correlation between mental workload and face temperature is anticipated. This iterative process is crucial for fine-tuning the model’s predictive capabilities and ensuring its generalizability to diverse operational conditions in aviation.
Acknowledgements
We acknowledge the support from NSERC-Alliance, CRIAQ, CAE, Bombardier, and BMU for funding this work. We also thank the participants of the experiment and particularly Massimo Pietracupa and Marc-Antoine Courtemanche for their help.