Tolerance of the ERP Signatures of Unfamiliar versus Familiar Face Perception to Spatial Quantization of Facial Images *

Processing of faces as stimuli is known to be associated with a conspicuous ERP component N170. Processing of familiar faces is found to be associated with an increased amplitude of the ERP components N250r and P300, including when a subject wishes to conceal face familiarity. Leaving facial images without high spatial frequency content by low pass spatial filtering does not eliminate face-perception signatures of ERP. Here, for the first time, we tested whether these facial-processing ERP-signatures can be recorded also when facial images are spatially quantized by pixelation, a procedure where in addition to impoverishment of face-specific information by spatial-frequency filtering a competing masking structure is generated by the square-shaped pixels. We found dependence of N170 expression on level of pixelation and P300 amplitudes dependent on familiarity with 21 pixels-per-face and 11 pixels-per-face images, but not with 6 pixels-per-face images. ERP signatures of facial information processing tolerate image degradation by spatial quantization down to about 11 pixels per face and this holds despite the subject’s wish to conceal his or her familiarity with some of the faces.


Introduction
The ability to identify and discriminate faces is a major research field in cognitive neuroscience, cognitive psychology, artificial pattern recognition and forensic research [1][2][3][4][5][6][7][8][9][10][11][12].Advancement of knowledge in this area promises considerable developments and gains in technology, economy, security-state of society, etc.Among several urgent tasks, finding objective and reliable brainprocess signatures of face recognition and familiar versus unfamiliar face discrimination can be especially emphasised.Inter alia, electroencephalographic (EEG) event related potentials (ERPs) based methods have shown their good applicability for the above-mentioned purposes.EEG/ERP are a relatively cheap, non-invasive, well standardised and internationally quite widespread means to study brain-process signatures of processing meaningful object information, supported by an impressive amount of documented psychophysiological facts and regularities from basic and applied research.
In practical applications of face recognition research, many directions have emerged and many important results obtained.However, quite many unsolved or unexplored problems remain [2,11].For example, it may be the case that the images of facial stimuli that are to be shown to perceiving subjects (e.g., in order to evaluate if the subject recognises a face or identifies a familiar face) are degraded due to some technical problems or imperfections.Often the available facial information is represented as a pixelized image with poor resolution.It is useful to know whether these stimuli can be nevertheless used as critical stimuli for testing and expertise and what is the scale of degradation tolerated by the automatic face-processing routines in the brain so that meaningful and actually sensitive ERP signatures of face recognition and/or face familiarity can be still registered and evaluated.Up to now there is no face-identification ERP-research using poor-quality pixelated images.
The three ERP components registered from the human scalp that are strongly involved in face processing are N170, N250r, and P300 [13][14][15][16][17][18].N170 is a quite robust signature of facial image processing found in many studies and under a wide variety of facial stimuli, spectral contents of face-images and perceptual tasks [13,16,17,[19][20][21][22][23].It is a negative potential deflection appearing about 130-200 ms after presentation of a facial stimulus, peaking at about 170 ms.N170 can be best registered from the occipito-temporal, temporal and temporal-parietal electrodes [15,19,24].It appears that face familiarity does not influence N170 [16,25,26].N250r is a negative-polarity ERP component that has been related to image-independent representations of familiar faces aiding person recognition [27].P300 as a positive-polarity potential that appears about 300-500 ms after stimulus presentation is widely accepted as a signature of working-memory analysis involving categorical cognitive processing and comparisons, context updating, resource allocation and meaningfulness evaluation [28,29].A variety of P300 called P3b which is a signature of categorical, memory dependent processing is best expressed over parietal and temporal-parietal areas.Importantly, the amplitude of the late positive ERP components may be significantly increased when familiar, relevant or attended stimuli (e.g., faces) as opposed to unfamiliar or nonrelevant stimuli are presented [18,[30][31][32][33].Because there are many brain sites that increase breadth and amplitude of activity in reponse to highly meaningful or attention-demanding faces as opposed to less significant faces [34] it is not unanimously agreed upon what are the exact brain sites maximally contributing to the increase in the brain responses to significant faces.Importantly, the increased brain response to more highly meaningful stimuli occurs even when a subject tries to conceal familiarity of a particular stimulus item that reliably has a capacity to lead to an enhanced response such as the P300 amplitude [33].With familiar faces, P300 may be transformed so that a face-specific negative deflection, N400f precedes typical positivity at about 300-500 ms post stimulus [16].As mentioned above, it is important to know whether and to what extent ERPs that are sensitive to faces and facial familiarity can be present when facial information is degraded.
Some studies have manipulated facial stimulus-images by filtering out detailed (facial) information by spatial low-pass filtering and then measured subjects' ERP responses [17,20,21,35].It appears that if only coarse-scale face related spatial information is present, ERPs still differentiate between faces and non-faces and/or between different categorization tasks, with coarse-scale information sometimes leading to relatively better expressed N170 compared to fine-scale faces [20,21,35,36].However, simple spatial filtering may bring in confounds between image spatial frequency components and lower level features such as luminance and contrast (see, e.g., [37,38] on how to circumvent these problems].It is therefore important for new studies to use experimental controls over these factors (see our text below).
In practice, security surveillance recordings also often produce facial images that are impoverished, degraded and/or distorted, which makes obstacles for high-quality and reliable evidence-gathering and eyewitness reports [2,11,31].However, a typical degradation of such images involves not only and not so much spatial low-pass filtering per se, but also often these images are spatially quantized (pixelized) so that in addition to the filtering out of higher spatial frequencies of image content (its fine detail), the mosaic-like structure of the squares produced by the image-processing algorithms that are used in producing pixelised images represents an additional image structure besides the authentic facial low-frequency content [39][40][41][42][43].This extra image content (squares with vertical and horizontal sharp edges and orthogonal corners; see, e.g., Figure 1) provides a competing structure for the perceptual systems of image feature extraction, figure-ground discrimination and visual-categorical interpretation.In a sense, this procedure, in addition to filtering out virtually all of the useful fine-scale information does also something else -it adds also a newly formed masking structure.It is important to know whether brain systems of facial information processing can be immune to this kind of complication or not.Equally important, it would be useful to know whether spatial quantization could change an image in a way that different cues of diagnosticity become to be used, but the ERP signatures of face processing, by using these new cues, may show sensitivity to categorical facial differences (e.g., familiarity).Hypothetically, this may lead to increased categorical sensitivity of ERPs compared to the absence of this kind of sensitivity which has been the case when unquantized, but otherwise filtered face-images have been used [16,25,26].The existing research literature does not provide an answer to these questions.
Most of the studies of spatially quantised image recognition have been strictly psychophysical -e.g., [39][40][41][42].Up to our knowledge, the only psychophysiological study where spatially quantized images of faces were used was that by Ward [44], but because monkeys were used as perceivers and because only very coarse quantized images with 8 pixels/face or less were used, her findings showing that discrimination between quantized face and nonface stimuli was not possible cannot be strongly generalized.
Coincidentally, spatial manipulation by quantization is free of the methodological problem that accompanies the traditional standard spatial filtering where selectively filtering out certain frequencies may also lead to filtering out luminance and/or local contrast information to a different extent.Because spatial pixelation is based on calculating average luminances within precisely defined square-shaped areas of the original image, spatially quantized images do not bring in artifacts of unequal luminance filtering.
Face-sensitive bioelectrical signatures of processing heavily rely on configural attributes of facial images, with three main types of configural cues involved [6]: 1) first-order relational processing allowing to specify a stimulus as a face as such, 2) holistic (Gestalt-) processing leading to a mutually supportive, integrated structural set of features, 3) second-order relational processing that uses metric information about spacing of facial features and thus enables discrimination of individual faces.By spatially quantizing faces, and beginning from a relatively coarse level of quantization, we eliminate local featural information and seriously disturb second-order configural processing, at the same time introducing relatively less distortions into first-order and into holistic processing.If it would happen that intermediate level (or even coarse level) spatial quantization does not eliminate face-sensitive ERP signatures and maybe even does not eliminate EEG-sensitivity to the familiarity of faces, then we would show that coarse-scale configural information in the conditions where it is presented within the context of a competing and conflicting structural cues is processed to the extent that allows one to carry out instrumental procedures of detecting (familiar) face detection and discrimination with quantized images.
The present study has two main aims.First, it is to test if spatially quantized images of faces can carry perceptual information sufficient for brain processes to discriminate different classes of facial images and if the answer to this question is positive -to see what is the approximate spatial scale of pixelation coarseness that allows to carry this information.The second aim is to test whether spatially quantized facial images when they can help lead to ERP signatures of face discrimination enable to differentiate familiar face image processing and unfamiliar face image processing in the conditions where the perceiver tries to conceal his/her familiarity with some of the faces.We put forward three general hypotheses.1. Spatially quantized images of faces as stimuli carry configural information that can be used by brain processes to generate ERP signatures typical for facial image processing (e.g., N170) and can therefore lead to reliable ERP differences as a function of the scale of spatial quantization.2. Spatially quantized images of faces lead to ERP signatures that are sensitive to face familiarity (e.g., P300) despite that local featural information is filtered out, second-order configural information is distorted and masked and despite that subjects try to conceal their familiarity with some of the stimuli faces.3.There is a critical level of coarseness of spatial quantization beyond which ERP signatures of processing facial images do not anymore discriminate between familiar and unfamiliar faces.

Participants
Six female subjects (age range 20-25 years) who were naïve about the research hypotheses of the present study participated.All had normal or corrected-to-normal vision.The subject sample was selected opportunistically from the pool of bachelor-level students of Tallinn University.

Experimental Setup and Procedure
Frontal images of human faces were used as the visual stimuli.Each subject was presented repeatedly with 6 versions of the facial images of the 2 persons well familiar to them and repeatedly with 30 versions of the facial images of 10 unfamiliar persons.The images subtended 3.8° × 5.7°.All images were achromatic gray scale images.They varied between three levels of spatial quantization (pixelation by a mosaic transform): 8 × 8 screenpixels (corresponding to about 21 pixels per face width within the image), 16 × 16 screen-pixels (about 11 pixels per face width), and 32 × 32 screen-pixels (about 6 pixels per face width (Figure 1).(The intermediate level quantization value at 11 pixels/face which is an approximate equivalent of 5.5 cycles/face was chosen to be slightly lower than the 8 cycles/face spatial low-pass filtering used in [20,21] as a border value between high-and low spatial frequency filtered facial images.)The space-average luminances of all stimuli images were set equal at about 40 cd/m 2 .Stimuli were presented on EIZO Flex-Scan T550 monitor (85 Hz refresh rate).
Stimuli were presented on a computer monitor con- trolled by a custom made VB program at a viewing distance equal to 150 cm.The program and computer regimen allowed necessary synchronization so that no splitting of facial images ocurred.Synchronized with face presentation, a trigger signal was sent to the EEG recording system to mark the time each stimulus face was presented.All stimuli were presented in random order, each of them 10 times.(The fact that the probability of seeing a particular familiar face is different from the probability of seeing an unfamiliar face is acceptable because ERP signatures showing tuning to meaningful stimuli are not sensitive to the probability of a stimulus, but are sensitive to the probability of the stimulus class - [28].)Duration of the stimuli was set at 480 ms.There were 360 trials per subject.(As it is known that facesensitive responses may decrease with stimuli repetition, our design can be acceptable provided that facial stimuli that have different significance and/or meaning for the subject are all similarly susceptible to this decrease.Research based on fMRI and MEG methods has shown this to be the case - [45,46].)Subjects were instructed to "play a game", meaning that they knew that experimenters tried to use brain EEG-imaging to see whether they can catch if a familiar face was seen, but subjects had to conceal any possible signs of familiarity.Thus subjects were also forced to respond to each face by saying "unfamiliar".The experiment was run in a double blind protocol so that experimenters who were standing by during the experiment did not know whether a familiar or unfamiliar face was shown at each particular trial.

Data Analysis
For EEG data processing, Brain Vision Analyzer 1.05 was used.For processing the raw EEG data for ERPs, a high-cutoff 30-Hz filter was used.To obtain ERPs, EEG signal was segmented according to 900 ms peristimulus epochs (from -200 ms pre-stimulus to +700 ms poststimulus).Eye-movement artefacts were eliminated using Brain Vision Analyzer custom Gratton and Coles algorithm.The EEG data for obtaining ERPs was pooled for selected regional electrodes and thus 3 conditional regional ERPs were computed: O (pooled electrodes O1, O2, Oz), T (electrodes T3, T4), and TP (electrodes TP7, TP8, P3, P4).ERPs were baseline corrected (-100-0 ms).
In the analysis, we concentrated on ERP components N170 and P300.

Statistical Analysis
ERP components' mean amplitude data gathered from different subjects and different experimental conditions was subjected to ANOVA, with factors spatial quantization (3 levels) and familiarity (2 levels).Main effects as well as interactions were tested for significance.

Results
There are no behavioral results to be reported separately from ERP results because subjects equally and systematically answered "No" to each of the presented quantized faces and tried not to display any signs of possible familiarity with some of the faces.

N170 Amplitude
ERPs from all recording sites showed distinctive N170 in responses to faces.However, there were only few statistically significant effects involving our experimental factors.Measured from the occipital electrodes, the effect of level of pixelation proved to be significant

P300 Amplitude
As measured from occipital electrodes, the effect of pixelation level on P300 amplitude was highly significant [F(2, 34) = 10.644,P = 0.002] while the effect of familiarity was expressed as a trend [F(1, 17) = 3.871, P = 0.066].Familiar faces lead to higher P300 amplitude.There was a highly significant interaction between level of pixelation and familiarity [F(2, 46) = 10.366,P < 0.001.With familiar faces, the finest level of pixelation lead to P300 amplitude that was distinctly larger than amplitudes for intermediate level and coarse level quantized images; with unfamiliar faces the finest scale and intermediate scale quantized images lead to relatively similar amplitudes of P300 whereas the P300 amplitude value stood apart from the other two quantization levels.

Discussion
Our results support our hypotheses: 1) Spatially quantized images of faces do carry configural information which is used by brain processes to generate ERP signatures typical for facial image processing (e.g., N170).The coarseness range of spatial quantization capable of communicating facial configuration includes 11 pixels/face images (an equivalent of 5.5 cycles/face) or finer.
2) Spatially quantised images of faces lead to ERP signatures that are sensitive to face familiarity (e.g., P300); this is despite that local featural information is filtered out, second-order configural information is distorted and that subjects try to conceal their familiarity with some of the stimuli-faces.However, the familiarity effect is reliably expressed when measured from the temporal-parietal electrode locations, but could not be easily obtained from the occipital and temporal electrodes.3) There is a critical level of coarseness of spatial quantization beyond which ERP signatures of processing facial images do not anymore discriminate between familiar and unfamiliar faces.The familiarity effect does not tolerate coarseness of quantization set at less than 11 pixels/face.If the square-shaped pixel size in our images was 8 × 8 screen-pixels, this amounted to about 21 pixels per face quantization (an equivalent of about 10.5 cycles/face).With this level of image detail, all three basic varieties of configural information (first-order, holistic, second-order - [6]) are kept present.(See also Figure 1.)If the pixel size was 16 × 16 screen pixels, this corresponded to about 11 pixels per face quantization (roughly 5.5 cycles/face).According to our evaluation, this is sufficient in order to filter out local featural information, appropriate for strong distortion of second-order configural information, but allowing holistic information to remain present in the image.If pixel size of 32 × 32 screen pixels was used, a quantized face image with about 6 pixels per face was created (roughly 3 cycles/face).In that case, first-order configural information is considerably degraded, holistic information is severely degraded, and second-order configural information is maximally degraded if not eliminated.Because familiarity effects were obtained with 21-and 11-pixels-per-face images and not with 6-pixels-per-face images and because there was an interaction between ERP P300 amplitudes and familiarity, we can conclude that facial familiarity information was carried primarily by second-order configural cues.Whereas it is likely that a face is categorized as belonging to the class of familiar faces only after the cues that allow face individuation had been discriminated, the dependence of the familiarity effect on second-order configural processing is a viable theoretical conclusion.On the other hand, the absence of main effects of familiarity on N170 together with the sensitivity of N170 to the change of spatial quantization between 11 pixels/face and 6 pixels/face levels altogether indicate that this ERPcomponent is especially sensitive to the first-order configural cues.Some other works have supported both of these ideas [6,16,25].
It has been usually accepted that N170 is insensitive to face familiarity [16,25,26].Our results are consistent with this in general terms.One minor exception to this rule can be noticed when we remember that there was a significant interaction between familiarity and pixelation level with temporal-parietal electrodes.Unfamiliar faces produced expected effects, showing higher N170 amplitude with systematically finer facial stimuli.This can be explained as better detection of facial first-order configural cues and also holistic templates when image detail gets finer and the competing structure of the square-shaped pixels' mosaic gradually loses its distracting power.However, with familiar faces the finest quantization did not lead to a highest N170 peak amplitude.One possible explanation could assume that 11 pixels/face and 21 pixels/face quantization levels in case of familiar faces are equally efficient for individual face recognition because of equal ease with which first-order facial configural representations are activated.This may be a result of formation of some habitual, automatic link between second-order featural configuration representation and first-order facial templates.This speculation should be tested in specific experiments in future.
Somewhat surprisingly, the finest level of quantization when applied to familiar faces lead to highest P300 amplitude also as measured from the occipital electrodes (with unfamiliar faces, the finest and the intermediate level quantization yielded equal P300 amplitudes).Although the cortical site of this effect was surprising, the direction of the effect supported the conclusion about second-order and featural information being the basis of familiarity effects.Fine quantization level allows visual system to recognize a familiar face with high certainty, which in turn can capture attentional processes to a stronger degree.Indeed, as shown by [30], focusing attention on certain facial cues enhances the P300 amplitude.
Our design presupposed repetitive presentation of familiar and unfamiliar faces, appearing in random order and varying in the low-level attributes which was caused by the varying levels of quantization.This means at least two things.First, while sometimes familiar faces appeared successively, but even in the more often occurring cases they appeared after a few unfamiliar faces had intervened.Thus, this design may be appropriate for finding a certain definite signature of face processing that seems to be sensitive to face familiarity of successively presented faces and that presupposes parietal involvement --the N400f [16].Yet, our statistical analyses did not succeed in disentangling this component as a statistically significant one (see also Figure 2).Secondly, because in our experiment the same original faces, when quantized at varying levels, were depicted as different low-level images, they should have enabled generation of ERP components that are sensitive to invariant face recognition with varying low-level attributes of the corresponding facial images.Because familiarity presupposes recognition (in addition to detection) and because some of the ERP signatures that are specific to individual face processing are image-independent (in terms of image low-level characteristics) and explicable when no more than only a few items intervene, we should be able to observe such signatures in our ERPs also.The ERP component N250r is known to satisfy the above criteria [18,27].Unfortunately, our statistical analyses did not succeed in finding any reliable effects of N250r.On the other hand, if we observe Figure 3 where ERPs with strong parietal and temporal involvement are depicted, we see that familiar face perception is associated with a visibly stronger negativity between 200 ms and 350 ms post-stimulus (and only with fine and intermediate scale pixelation, but not with coarse quantized images).Hopefully, subsequent studies when especially targeted on this observation could produce reliable statistical effects.The ERPs that discriminated between familiar and unfamiliar faces were found with face-image pixelation at 11 pixels/face and above, but not with 6 pixels/face images.This specific value of difference when it sets the images with above 10 pixels/face quantization apart from the rest approximately corresponds to the critical pixelation values found in behavioural studies of face identification [39,40].This may mean that processing familiar facial information from the spatially quantized images requires that subjects can explicitly discriminate these quantised images in terms of their facial identity.On the other hand, when gathering introspective reports from the subjects after they completed the experiment, it appeared that some of the actually familiar faces, when quantized at the intermediate level were not recognized as familiar.It should be important to carry out further studies in order to ascertain if ERPs could reflect face familiarity even with explicitly unrecognized facial images.In addition to theoretical significance of this question it may be valuable to solve it also for applied purposes.For example, a need may emerge to test whether a person is familiar with some individuals whose low-quality photographs are available and where, therefore, this person has no explicit awareness of what is depicted in the picture.Another applied aspect related to our results is even simpler: we have shown that spatially quantized (pixelated) images can be used for registration and analysis of facesensitive ERPs.This in itself is encouraging.

Conclusions
As was stated in the introductory part, spatial quantization is an image transform with effects ranging beyond simple spatial-frequency filtering.The structure of the square-shaped pixels with their square-corners, squareedges and formal aspects of the mosaic of square-shapes provides a visual structure that 1) masks facial configural cues and 2) sets visual system at the competing demands of image interpretation -a face versus a mosaic.In these circumstances there are no strong a priori foundations to expect an inevitable capability of the visual processing system to extract face-specific information sufficient for generation of known face-specific and/or face-sensitive ERP signatures on the face of the pixelised masking structure.Our study showed that spatial quantization does not make an obstacle for the emergence of ERPsigns of facial processing, including the ones sensitive to face familiarity.However, this sensitivity has its limits so that with pixelation coarseness approaching 6 pixels per face, familiarity effects on ERP disappear.

Figure 1 .
Figure 1.Examples of stimuli: (a) pixel size 8 × 8 (approximately 21 pixels/face); (b) pixel size 16 × 16 (approximately 11 pixels/face); (c) pixel size 32 × 32 (approximately 6 pixels/face).In (a), all three basic varieties of configural information (first-order, holistic, second-order) are kept present; in (b), local featural information is filtered our, second-order configural information is strongly distorted, but holistic information kept present; in (c), first-order configural information is considerably degraded, holistic information is severely degraded, and second-order configural information is maximally degraded if not eliminated

Figure 2
depicts grand average ERPs obtained from generalized regions O, T, and TP as a function of level of pixelation; ERPs are shown separately for unfamiliar faces (ERP functions on the left) and for familiar faces (ERP functions on the right).As seen from Figure2, distinct ERP components P100, N170 and P300 are produced for quantized faces as the visual stimuli.

Figure 2 .
Figure 2. Grand average ERPs registered from occipital, temporal and temporal-parietal pooled electrodes (negativity up).Distinct P100, N170 and P300 can be seen.Left column -unfamiliar; right column -familiar.For 21 pixels/face images condition ERPs drawn in black; for 11 pixels/face ERPs in red; for 6 pixels/face ERPs in green

Figure 3 .
Figure 3. ERPs recorded from TP7, depicted for fine-scale quantization; (a) intermediate level quantization; (b) and coarse-scale quantization; (c) conditions.Familiar faces -ERPs in black; unfamiliar faces -ERPs in red.With (a) and (b), familiar faces lead to some N250f-like ERP deflections