ROC and SAT Analysis of Different Grayscale Test Images (Distractors L and Target T) to Customize a Visual-Search Attention Task ()
1. Introduction
Fatigue is defined as a condition or phenomenon of decreased ability and efficiency of mental and/or physical activities caused by excessive mental and/or physical activities, or illness. It is characterized by decreased energy, motivation and difficult concentrating or difficulty in maintaining sustained attention and remaining alert to relevant stimuli appearing at unpredictable time points over long periods of time. Fatigue is usually accompanied by a sense of discomfort, desire to rest and reduced motivation [1]. There are many types of different works such as driving, monitoring tasks which require operators to maintain sustained attention on multiple objects for prolonged periods of time. One of the principal causes of decrements in sustained attention is mental fatigue [2] which was defined as the state of reduced mental alertness that impairs performance. There are many studies which showed the adverse effects of mental fatigue on sustained attention as shown by reduced behavioural performance. For instance, in sustained attention to response task [3] found that participants’ performance declined in both accuracy and speed in time. It was found that as the duration of the mental fatiguing induced task increases, the number of misses and false alarms increase, and the response speed decreases [4]. In these studies as well as those from research [5], participants were instructed to execute a task continuously for a long period of time, and then their behavioural performances, reactions times and accuracy rates, were compared between the former and the latter of the mental fatigue induced experiment [6]. Moreover, factors that can induce fatigue in a short period of time can be a very boring/monotonous task, a bad/dark illuminating, night time, combination with physical fatigue and low-performance phases according to circadian rhythm.
To date, in most fatigue studies, the methods used to induce fatigue are either through sleep deprivation or the performance of long-term, tedious and repetitive tasks. Moreover, there are many researches study which employ and support long-time repetitive visual search tasks to induce mental fatigue [7]. For instance, authors [7] [8] used a visual search task to induce mental fatigue and they found promising electroencephalogram indicators for mental fatigue and evaluation. Moreover, researchers [9] found that visual search is a promising instrument for the assessment of cognitive functions and cognitive changes in patients with multiple sclerosis because of its good discriminatory power and insusceptibility to practice effects. Therefore, this research developed and attempted to customize a visual search algorithm task which consisted of distractors and a target.
2. Methodology
The visual search algorithm was developed using GNU Octave programming language running on Linux (Debian8). The visual search algorithm displays a number of rotated distractors (L) and a rotated target (T) presented randomly on a Cathode Ray Tube (CRT) monitor’s screen with width 600 and height 480. The number of distractors including the target (N) was varied from 12 to 28 in increments of 4. The image size of the distractor L and that of the target T were varied from [25 × 25] pixels to [40 × 40] pixels in increments of 5 to produce 4 different image sizes. Then, the image intensity for each particular distractor image size as well as the target image size was varied in the following manner: 220, 230, 240 and 255. This produces in total 16 types of grayscale test images for the distractor L and 16 types of grayscale test images for the target T. Some examples of the images are shown in Figures 1-3. The distance between the participants and the monitor screen was kept at constant at 60 cm.
![]()
Figure 1. Displays target T of size 40 × 40 pixels in different intensity of 220, 230, 240 and 255. (a) Size 40Int220; (b) Size 40Int230; (c) Size 40Int240; (d) Size40Int255.
![]()
Figure 2. Test image of target T at different image size in respective size [Pixel × Pixel]. (a) 40 × 40; (b) 35 × 35; (c) 30 × 30; (d) 25 × 25.
![]()
Figure 3. A screenshot of the visual search task with a random set size of 28 including the target and distractors. The distractors as well as the target are randomly rotated throughout and spread across the screen.
2.1. Participant Recruitment
The whole research study ethics was approved by Tubitak’s Ethics committee. Prior to performing the visual search task, an informed consent form was provided to each participant to demonstrate their willingness to take part in this psychophysics experiment. The psychophysics experiment took about 20 minutes, and each participant was required to press the right-arrow key on the keyboard when he or she sees a target T among randomly shown distractors L on the screen or to press the left-arrow key on the keyboard when he or she do not see a target T among randomly shown distractors L on the screen. The participants were required to press the appropriate key as quickly and accurately as possible. Participants could withdraw from the experiment at any time while performing the psychophysics experiment if they did not feel comfortable to continue, and their results were eventually withdrawn from the recorded database. All participants’ responses data were coded and no names were be divulged during this dissemination. A total number of 18 participants took part in this study of which 10 were males and 8 were females. The healthy participants comprised of medical students of different years of study (1st year to 6th year of their medical studies), and also research staff of the biophysics, physiology and biology department of Necmettin Erbakan University. Their ages range from 19 to 51 years old (24.8 ± 8.2).
2.2. Data Collection
Both reaction times (response times) as well as the responses of the participants were recorded. The responses of the participants were compared to the presence or absence of target in a particular screen. Moreover, the different set sizes of the number of distractors including target for each screen were recorded.
2.3. ROC Analysis
A Receiver Operating Characteristics (ROC) graph is a technique for visualizing, organizing and selecting classifiers based on their performance. ROC graphs have been employed in signal detection theory to depict the tradeoff between hit rates and false alarm rates of classifiers. ROC analysis has been extended for use in visualizing and also analyzing the behavior of diagnostic systems. One of the earliest adopters of ROC graphs in machine learning was Spackman (1989), who demonstrated the value of ROC curves in evaluating and comparing algorithms.
Classifier Performance
A classification model or classifier is a mapping from instances to predicted classes. Given a classifier and an instance, there are four possible outcomes. If the instance is positive, and it is classified as positive, it is counted as a true positive; if it is classified as negative, it is counted as a false negative. If the instance is negative and it is classified as negative, it is counted as true negative; if it is classified as positive, it is counted as a false positive. Given a classifier and a set of instances (the test set), the following terms are defined [10] [11]:
The true positive rate (also called the hit rate and recall) is estimated as:
1) Tp-rate = positives correctly classified/total positives
The false positive rate (also called the false alarm rate) of the classifier is:
2) Fp-rate = Negatives incorrectly classified/Total negatives
Other terms derived from these two rates are:
3) Specificity = 1 − fp rate
Specificity measures the proportion of negatives that are correctly identified.
4) Sensitivity = tp rate
Sensitivity measures the proportion of positives that are correctly identified.
In this research, the Receiver Operator Characteristics (ROC) analysis was applied to the reaction times of the participants that were affected by the different image size and image intensity as well as the number of distractors on a particular screen. In this psychophysics experiment, for each participant, the target was present at 50% of the screen presentations in order to observe how well the reaction times could be used to predict the two different classes of presence of target on a screen and the absence of target on a particular screen. The Area under Curve (AUC) of an ROC curve will pinpoint the best test image(s) for this particular visual task in terms of specificity and sensitivity.
2.4. SAT Analysis
There is normally a negative relationship between reaction times and accuracy of responses. It means that the faster one reacts in a particular task, the worse is one’s performance (accuracy). The confusion matrix summarizes the different types of true positives, false positives, true negatives as well as false negative that might occur during an experiment (Table 1). Therefore, for the Speed Accuracy trade-off analysis, the ratio of d-prime (d’) or sensitivity index to reaction time was investigated for each test image and it is also called as the Speed-Accuracy Tradeoff Function (SATF). The d’ (d-prime) is a statistic used in signal detection theory and it provides the separation between the means of the signal and the noise distributions, compared against the standard deviation of the signal or noise distribution. An estimate of the d’ can be found from the measurement of the hit rate and the false alarm rate based on the confusion matrix (Table 1). It is calculated as d’ = Z (hit rate) − Z (false alarm rate) where function Z (p), p ϵ [0, 1], is the inverse of the cumulative distribution function of the Gaussian distribution. A higher d’ indicates that the signal can be more readily detected.
![]()
Table 1. Confusion matrix to describe the target present and target absent responses to differentiate between true positive, false positive, false negative and true negative.
Based on the SATF function, low constant d’ for low reaction times mean the Task is too difficult and if high constant d’ for high reaction time, this means performance saturates.
3. Results
A detailed summary of the results produced from the ROC analysis, d’, the mean reaction times of the participants for each test image as well as the corresponding responses’ accuracy are shown in Table 2. The percentage accuracy of the responses is computed by dividing the number of correct responses with the total number of questions posed to the participants. The reaction times of the participants’ responses are classified using ROC technique (see Figure 4).
SATF analysis was conducted on the test image Size35Int255 based on its high d’ and highest AUC values to observe the change in sensitivity with increasing reaction times of the participants (see Figure 5).
![]()
Figure 4. ROC analysis of reaction times of the participants in successfully being classified in two classes (screen with target and screen without target) to display the behaviour of the selected test images. The reference line shows an imaginary line that divides the graph into equal percentages (50%).
![]()
Figure 5. The SATF shows how the sensitivity changes with increasing reaction times.
![]()
Table 2. ROC analysis, dprime, mean RT and accuracy.
4. Discussion and Conclusion
ROC analysis proved to be a useful tool in distinguishing the effect of the different types of images (distractors and targets) on the subjects’ reaction times. The highest value of AUC was the test image with size of 35 × 35 pixels and with an image intensity of 255 and it represented the best test image in terms of specificity and sensitivity. Moreover, the results based on the d-prime alone showed that the test image Size35Int255 ranked among the first three best images to represent a readily detected visual stimulus. Speed-accuracy trade-off function analysis on the test image Size35Int255 showed that the sensitivity increases sharply between the reaction times of 0.8 to 1.1 seconds, then sensitivity rises gradually until 2.8 seconds and then sensitivity remains constant beyond this point which insinuates that performance saturates at this point onward. The ongoing analysis is also focusing on the effect of a varied number of targets and distractors on the participants’ performance.
Acknowledgements
This research work is supported by The Scientific and Technological Research Council of Turkey (TÜBİTAK) and the Marie Curie Action Cofund of the 7th Framework Program (FP7) for me to pursue my postdoctoral research work. I would like to sincerely thank the medical students as well as the various research staffs from the Biophysics, Biology and Physiology Departments of the Necmettin Erbakan University for their precious participation as well as for helping me in the organization (scheduling and translation) process of this psychophysics experiment. My sincere thanks go also to my mentors Assoc. Professor Barkin Ilhan for his fruitful advice and guidance about performance algorithms, and the concept of the rotation of the targets and distractors, and Prof. Nizamettin Dalkılıç about the support of this work.