Neuromarketing : Neurocode-Tracking in Combination with Eye-Tracking for Quantitative Objective Assessment of TV Commercials

The term “neuromarketing” suggests a combination of marketing research and brain research. Within marketing research, objective assessment of TV commercials is performed by use of EyeTracking devices. Brain research has been dominated by NMRI technologies during the last years. However, information content and time resolution are much better when monitoring electric activity by spectral analysis. The current experimental approach was initiated in order to test the combination of a newly developed EEG technology (Neurocode-Tracking) with commercially available Eye-Tracking. Ten male subjects were exposed to a reference recording condition (watching a fixed cross on the screen) followed by 5 TV commercials representing 5 different banks. Video films obtained from Eye-Tracking (so-called heat maps) were synchronized with the results of Neurocode-Tracking in form of a second video with a time resolution of 364 ms per image. Spectral power analysis of the EEG in different brain regions allowed for assessment of single scenes for cognitive and emotional responses. Averaging of data over the whole time course gave information on mental activation, attention, visual perception, memory, pleasure and tension. Comparison of the 5 commercials to reference recording revealed the clip of the Targo-Bank having the highest values with respect to mental activation and attention.These experiments have proven that the combination of Neurocode-Tracking with Eye-Tracking can be successfully used in advertisement research and gives valid information on individual as well as group dependent cognitive and emotional responses leading to objective assessment of TV commercials.


Introduction
A common definition of neuromarketing is "that it is a field of study concerning the application of neuroscience methods to analyse and understand human behavior related to markets and marketing exchanges" [1].A more extensive and critical review on the definition of the term "neuromarketing" and its historical aspects has been published earlier [2].
TV Commercials dominate broadcasting and TV programs to an ever-increasing extent.However, since broadcasting time is very expensive, many efforts are undertaken to assess the quality of the video clips and the resonance by viewers.Besides personal impressions and opinion polling more and more technical equipment is used to find out if the clip under investigation will draw the attention of the viewer.A crucial question is for example how much time one will need to communicate a certain message, since every second to be transmitted by TV increases the financial burden.Ideally, each of the scenes should therefore be assessed in separate and several versions of a clip should be pretested in an objective manner.
One of the currently applied technologies consists in the use of Eye-Tracking [3].This method allows for following continuously the eye gazes throughout the viewing of a video clip by projecting a coloured spot on to the momentarily focussed location in a separate video clip.At the end of the application of the Eye-Tracking a movie is produced, where the eye gazes can be traced image by image in order to learn where and how long gazes fix a certain part of the video clip.Currently available software also allows for averaging data from several subjects.The result is a so-called "heat-map", where spots are generated to indicate the number of subjects looking at the same time at a particular location represented by a colour code.This information is quite valuable since regions of major interest can be diagnosed very well.However, no information with respect to cognitive or emotional responses can be gathered by use of this technology alone.
The question now arose how this information can be achieved in order to extend the Eye-Tracking data.On the base of recent progress in brain research a new option emerged in looking at the electrical activity of the brain.Since the discovery of the EEG by Hans Berger [4] visual analysis of the signal curves has only provided limited success with respect to understanding brain function or deviation from normality.However, Hans Berger published a paper together with Dietsch [5] proposing quantitative assessment of the EEG by frequency analysis.Common use of this approach had to await help by computers for faster calculation.For this type of analysis, time periods of 2 s and longer were used in order to be able to evaluate also the extent of slow waves.Successful application of classic EEG technologies in advertisement research has been reported earlier [6].
But the brain obviously works faster than that.Processing time for a visual or auditory stimulus in the brain is about 300 to 400 ms.Based on certain preconditions [7], I now describe a new approach for fast dynamic quantitative analysis of the EEG including documentation of frequency changes by means of electric brain maps (also called "enkephaloglyphs").Interpretation of single short-term maps of 364 ms duration is achieved by use of Neurocode-Tracking (fast dynamic real time EEG analysis) in combination with conventional Eye-Tracking as used in market research.

Methods
Recording of the EEG was performed as published earlier [7].In short, a 17 channel EEG (neo-CATEEM ® ; MEWICON CATEEM-Tec GmbH, 4164 Schwarzenberg, Austria: www.mewicon.atand www.cateem-tec.com) in combination with Eye-Tracker (from Tobii Technology GmbH, Frankfurt, Germany) was taken by using a conventional electrode cap, and signals were amplified using a commercially available device (DeMeTec GmbH, Langgöns, Germany).Electrode positions were based on the classic 10/20 system [8].Frequency content of the spectra after Fast Fourier Transformation (FFT) was converted into spectral colors.Besides numerical calculation of 6 frequency ranges at 17 electrode positions (102 parameters) data are coded into color maps by additive color mixture according to the RGB mode as used in TV using spectral colours representing frequencies from 1.25 to 35.00 Hz.Slow frequencies are thus represented by red color, medium ones by yellow and green colors and fast frequencies by blue colors.
Map data are here presented in the so-called "global median mode".This mode represents an individual normalization, since spectral power values from all electrode positions are collected for each frequency range followed by calculation of the median value, which is set to 100%.Spectral power values of different brain regions are now calculated in % of this median and used to show the relative focal distribution of spectral power for each cortical region of the brain.
For offline analysis of the quantitative EEG data and synchronization with the eye-track data a screen grabber (Adobe Captivate) was used to produce a video containing all successively calculated EEG maps.A second video was obtained from the eye-tracker software running on a separate computer (TobiiTechnology GmbH, Frankfurt, Germany).It is called a "gaze overlay" movie depicting the presented picture, task or video film.In this gaze overlay video the momentary gaze of the subject is documented by a colored spot.Since the presentation always starts with an audio signal (gong) and this audio signal is also registered by screen capture of the EEG computer, it is used for synchronization of both videos by means of a video cut software (Adobe Premiere Pro).Due to the processing time of the brain (300 to 400 ms) plus that of the computer (depends individually on the type and number of active processors!) the gaze overlay video is shifted in our case (quad core, 3.4 GigaHz) for one second in order to obtain synchronized images between gaze and the particular EEG epoch of 364 ms.For detailed offline documentation a movie was exported and analyzed image by image.Single images containing the gaze and corresponding EEG map (called an "Enkephaloglyph"; [9]) were cut from the screen by a software tool available on all computers.
For quantitative objective assessment with respect to cognition and emotion, spectral power from 6 regions of interest (ROI) were selected on the base of earlier experience: Spectral delta power in the left lateral frontal cortex for mental activation [10]; spectral theta power in the left lateral frontal cortex for attention [11]; spectral theta power in the left visual cortex for visual perception; spectral alpha2 power in the right temporal lobe for memory processing [12]; spectral delta power in the central region for pleasure detection and spectral beta power in the left temporal lobe for evaluation of tension [13].
Ten male subjects (aged 36 ± 4.4 years) entered the study.They were asked to view TV commercials of five bank institutions: ComDirect (duration 22 s), DKB (21 s), IngDiBa (42 s), Postbank (25 s) and Targo-Bank (27 s) within a study aiming at surfing on the webpages of these particular banks [14].

Results
Synchronization of the Neurocode-Tracking data with the Eye-Tracking data (for details s. methods) allowed step-by-step analysis of single scenes.Selected examples are shown in Figure 1 and Figure 2, each representing a highlight of mental activation.These enkephaloglyphs differ from scene to scene.Especially, fronto-temporal areas are obviously involved in processing the audio-visual information.Since the maps are not constructed by false color-coding but represent the result of the mathematical transformation according to Fourier, differences represent true quantitative changes.Time dependent oscillations of the raw signal leading to changes of single spectral maps are analysed quantitatively with respect to cognitive and emotional features according to different regions of interest.Figure 3 documents these changes for the duration of one minute looking at a fixation cross at the screen with respect to mental activation, attention, visual perception, memory, pleasure and tension.The extent of the response fluctuates on a low level (% global median) except for tension, which is quite high during the whole time of viewing the fixation cross.The result is quite different for the time dependent fluctuation of these responses during viewing the clip of the commercial of the Targo-Bank as depicted in Figure 4 and the IngDiBa bank in Figure 5. Here, mental activation and attention reach rather large values at particular scenes as can be recognized looking at particular scenes in the movie clip at the lowest part of the figure.This type of analysis relates single scenes to the pattern of electric activity with a time resolution of about 3 images/per second.Thus, scenes with eminent cognitive resonances are recognized.This type of assessment has been performed for all commercials, but only data from viewing the commercial of the Targo-Bank and the IngDiBa bank are depicted in order to explain the type of approach.More information can be gathered from my video clip on YouTube (http://youtu.be/Dgx4Nv2aqaM).
The next step in quantitative assessment of TV commercials is check of the attraction of clips against a standard condition like watching the fixation cross or against other commercials.In order to initiate this evaluation spectral data are averaged over the whole duration of the clip for each of the cognitive or emotional features.The result is documented for all bank commercials in Figure 6.Largest differences-when the whole clip is assessed-are seen with respect to the cognitive features mental activation and attention.Highest values are observed with respect to the clip of the Targo-Bank in a statistically significant manner in comparison to the recording condition "watching fixation cross".Lowest values were seen with respect to the ComDirect bank.All together, the objective assessment of TV commercials allowed a ranking of the overall attractiveness of all 5 bank presentations as documented in Figure 6.

Discussion
These preliminary results clearly indicate a very specific activation of electric circuits during sensory processing, much more complex than has been suggested by fMRI experiments measuring blood flow as indirect representation of neuronal activity.Nevertheless, fMRI studies have contributed very important information on brain regions involved in the processing of marketing relevant sensory inputs like logos [15].However, fMRI measure- ments are not suitable for routine use because of the technical effort and pricing.Opposite to this, the current measurement of cortical electric activity probably reflects brain activity to a better degree and in more detail, since electric activity can be related much better to cognition and emotion than in NMRI measurements [16].
With respect to the assessment of TV commercials high-resolution electro-encephalogram recordings were performed in healthy subjects during the observation of TV commercials [17].These authors were able to discriminate cortical activity from subjects who remembered the commercial from those who had forgotten them.Three questions arise now, which are most importantly to be answered: First of all, one would like to know, which scene within a video clip is most appreciated and induces the strongest change of the electric pattern representing cognitive or emotional reactions.This is important in order to verify that the message is documented in an attractive manner.By it, the clip can be reduced to the best reactions, which saves time and by it money within a so-called pre-testing.Secondly, one would like to know the individual and group reaction with respect to the whole time course of a clip.Thirdly, one would like to compare the video under production with reference or competitive commercials produced by other companies.The present assessment of TV commercials using the described combination of Neurocode-Tracking and Eye-Tracking provides the answers.A good example with respect to the first question is one scene from the ComDirect Bank commercial.A very strong reaction emerges in the lateral frontal brain, when it comes to the topic of money (Heat map of Eye-Tracking in Figure 1).The answer to the second and third question is provided by data listed in Figure 6.Here the commercials have been assessed by evaluation of different brain areas.
It is very well known from earlier work with respect to EEG data [10] as well as from NMRI data that the lat- eral frontal brain is involved in mental activation.Accordingly, electric changes of the spectral power in the lateral frontal brain serve for quantitative evaluation with respect to mental activation and attention.Spectral changes in this area have also been observed earlier during watching TV commercials [18].Likewise changes within the occipital area are related to perception, since visual input from the eyes is communicated to this area passing the corpus geniculatum.If this occipital area does not show a change in electric activity, no visual perception has occurred.Thus, spectral analysis of this area codes for visual perception.With respect to memory frontal areas as well as temporal areas are involved during information processing.Changes in the temporal lobe are related to the hippocampus, which is located just beneath this area, and which has been shown to be one of the most important areas for time and space related memory.Spectral changes in the temporal lobe therefore reflect activation of memory processes like retrieval of memory contents.Emotional features like pleasure are represented in central areas of the brain.For this reason spectral changes in this area indicate emotional engagement and can be quantitatively assessed.Finally, earlier experiments have shown that particular frequency changes within the temporal lobe indicate stress and/or tension [13].This emotional feature turned out to be the only one, which showed rather high values during fixation of the cross on the screen representative for the reference recording condition "Open Eyes".Obviously, the concentration at the fixation cross during one minute had produced some tension, presumably because of the dark screen, since the same enhancement of tension was observed in the second part of the commercial of the IngDiBa bank as documented in Figure 5.This pilot experimental series of experiments has shown that the combination of Neurocode-Tracking with Eye-Tracking can be used in advertisement research and gives valid information on individual as well as group dependent cognitive and emotional reactions leading to objective assessment of TV commercials in a similar fashion as reported for the assessment of websites [14].A combination of Eye-Tracking and EEG has also been used during natural reading [19].A more general discussion on neuromarketing techniques has been published earlier [20].With Ohme and Matukin one can state: "As the majority of consumer mental processes occur below the level of conscious awareness, observations of the brain reactions enable researchers to reach very core (which is consciously inaccessible) foundations of consumer decisions, emotions, motivations, and preferences" [21].Future research will show how emotion can be even better related to spectral EEG oscillations by special classification algorithms [22].

Figure 1 .
Figure 1.Documentation of single brain maps from Neurocode-Tracking together with heat maps from Eye-Tracking for viewing the fixation cross or TV commercials from several banks.Enkephaloglyphs representing a recording epoch of 364 ms averaged from ten male subjects.Please note spectral power increases in frontal brain (F7, F8 (lower bar-chart left and right) and Fz, upper bar-chart second from right).Map: frontal brain oriented to the middle of the image.Heat map (right side): The more of the subjects look at the same place, the more green/yellow color emerges.Bar-chart: C = central, F = frontal, P = parietal, T = temporal, O = occipital.

Figure 2 .
Figure 2. Documentation of single brain maps from Neurocode-Tracking together with heat maps from Eye-Tracking for viewing TV commercials from several banks.Enkephaloglyphs representing a recording epoch of 364 ms averaged from ten male subjects.Please note spectral power increases in frontal brain (F7, F8 (lower bar-chart left and right).Map: frontal brain oriented to the middle of the image.Heat map (right side): The more subject's look at the same place at the same timing, the more of green/yellow/red color emerges.Bar-chart: C = central, F = frontal, P = parietal, T = temporal, O = occipital.

Figure 3 .
Figure 3.Time courses of cognitive and emotional changes during viewing the fixation cross for 1 minute.Condensed video clip is documented in the last lower line showing only the fixation cross (marked red).(Single epochs are rather dense due to longer duration of clip).

Figure 4 .
Figure 4. Time courses of cognitive and emotional changes during viewing the clip of the Targo-Bank.Condensed commercial is documented in the last lower line.Scaling of the ordinate corresponds to Figure 3. Duration of clip = 27 s.Averaged data from n = 10 subjects.

Figure 5 .
Figure 5.Time courses of cognitive and emotional changes during viewing the clip of the IngDiBa-Bank.Condensed commercial is documented in the last lower line.Scaling of the ordinate corresponds to Figure 3. Duration of clip = 42 s.Averaged data from n = 10 subjects.

Figure 6 .
Figure 6.Overview on quantitative assessment of 6 different cognitive and emotional reactions averaged from 10 subjects viewing different bank TV commercials.The extent of the reaction is given on the ordinate as % global median based on local frontal, temporal, central and occipital spectral power, averaged for the duration of the challenge (for details of assignments s. method).Statistical evaluation is marked as * representing p < 0.05 in comparison to viewing the fixation cross for one minute.