ABSTRACT

Event-related potential (ERP) studies demonstrated that emotional state immediately affects the processing of different linguistic mismatches (e.g., semantic and syntactic mismatches) in sentences. Furthermore, mood has been shown to impact discourse processing. In these studies, a strong expectancy was evoked by a linguistic context. In the present study, a strong expectancy was induced by a visual (non-linguistic) context paired with a sentence. A spatial array was followed by a sentence that matched (p¡―The square stands in front of the circle) or mismatched the picture (intra-dimensional mismatch: p¡―The square stands behind the circle; extra-dimensional mismatch: p¡―The square stands above the circle). The main question was whether mood effects on the processing of visually induced expectancies are similar or different from mood effects on the processing of linguistically induced expectancies. To this aim, we presented conceptual (mis)matches that differed in saliency: intra-dimensional vs. extra-dimensional mismatches which are both reported to elicit an N2b/N400 and P600, reflecting reanalysis. EEG was recorded while participants read for comprehension. Mood (happy vs. sad) was effectively induced by film clips. Neither for N2b/N400 nor for P600, an interaction between mood and match was observed. Thus, in contrast with studies investigating the effects of pure linguistic mismatches (linguistic context paired with unexpected lexical item), mood did not modulate the processing of conceptual mismatches. However, a main effect of match revealed different ERP patterns for the two mismatches. While an increase in N2b/ N400 occurred for the intra-dimensional and extra-dimensional mismatches, a P600 only occurred to the extra-dimensional (more salient) mismatches. This finding is taken as support that being in an emotional state (positive or negative mood) influences processes of reanalysis. That is, when being in an emotional state, a reanalysis (monitoring) process is exclusively triggered by salient mismatches.

Keywords:

Event-Related Potentials, Mood, Conceptual Processing, Picture-Sentence Mismatches

1. Introduction

In several fields of psychology, it has been shown that a participant’s background emotional state influences how information is processed. Emotional state, also referred to as mood, refers to a long-term psychological state. In contrast with an emotion which is triggered by a specific event in the environment, it often occurs without awareness of an evoking event [1] . It has been shown that emotional state―in particular being in a happy or sad mood―impacts our behavior, colors our perception, influences decision making and determines the scope of our attention (for a review see [2] ). The focus of the present study is on the effects of emotional state on language, specifically on the processing of conceptual mismatches in language comprehension.

Only recently, the relation between emotional state and language comprehension has been examined. According to traditional theories of comprehension, language processes―especially processing of word meaning and syntactic pro- cessing―take place automatically and are organized in independent modules. A core assumption is that language takes places in isolation and without interaction, neither within the language system nor with other psychological processes [3] . This modular view, however, has been challenged by more interactive models. A good example are embodied theories which posit that language is grounded in perception and action and external information is integrated by simulation [4] [5] [6] . The advantage of embodied approaches is that they naturally account for interactions of language comprehension with emotion.

The first study that investigated the effect of emotional state on sentential processing used behavioral measures [7] . A positive or negative emotional state was induced via manipulation of facial expression (i.e., “forcing” or blocking a smile by holding a pen between the teeth; for details see [8] ). Participants were asked to judge the valence of sentences (positive vs. negative). Reaction times were shorter when the valence of the sentence matched the facial expression (e.g. when participants were “forced” to smile while reading positive sentences) than when the valence of the sentence was contrary to the facial expression (e.g. when a smile was blocked while reading a positive sentence). Relevant for the present article, recently, the Event-Related Potential (ERP) technique has been used to study the on-line interplay between language and emotional state. ERPs are electrical brain potentials extracted from the electroencephalogram (EEG) that occur time- and phase-locked to a certain event, within the language context often the presentation of a word. Two well established language-relevant ERP components are the N400 and the P600. The N400 is a negative slow wave that peaks at around 400 ms following word onset. It has been shown to be highly sensitive to semantic processing. Its amplitude is an inverse function of the degree to which a word is expected, the less expected a word the larger the N400 amplitude. This is referred to as the standard N400 cloze probability effect [9] . The P600 is a positive slow shift starting around 500 ms and typically extending up to 800 ms after word onset. It has been shown to be reliably elicited by syntactically anomalous and ambiguous linguistic constructions [10] . More recently, it has also been shown to be elicited by some kinds of semantic anomalies, for example, semantic reversal anomalies [11] . Another ERP component is the N2 which can be divided into different sub-components. The N2b is of direct relevance for the present study because it has been reported to be sensitive to picture-sentence mismatches (see [12] ). One major advantage of the ERP technique is that it allows tracking cognitive processing in real time at the level of milliseconds. A second advantage is that ERPs can be recorded in the absence of an artificial judgment task [13] that does not form part of normal language comprehension. There is a wealth of articles showing that reliable modulations in ERPs occur when participants do nothing else than what they do in daily life, that is, reading or listening for comprehension (see [14] for a review).

What follows is a brief overview of the main results of the ERP studies that examined the effects of emotional state on different aspects of language comprehension. The few studies that investigated the influence of emotional state on language at the sentence level have consistently shown that mood affects semantic processing. One study investigated the impact of emotional state on the standard N400 cloze probability effect [15] . A happy vs. sad mood was induced by presenting film clips. EEG was recorded while participants read sentences that elicited a strong expectancy for a particular word. In half of the sentences the highly expected word was presented (high cloze sentence: “The pillows are stuffed with feathers which make them feel soft.”) and in the other half an unexpected word was presented (low cloze sentence: “The pillows are stuffed with books which make them feel hard.”). The intended mood was induced successfully. For N400, an interaction between emotional state and cloze probability was found. The interaction reflected the presence of a clear and broadly distributed N400 effect for the happy mood condition and a strong reduction of the N400 effect (restricted to the right hemisphere) for the sad mood condition. The modulation of the N400 as a function of emotional state was presented as support for embodied theories of language comprehension.

In two studies the effects of emotional state on the access to semantic memory were examined. Both studies presented sentences like “They wanted to make the hotel look more like a tropical resort. So along the driveway they planted rows of palms/pines/tulips.” In the first study [16] , pictures from the International Affective Picture System [17] were used to induce a positive or a neutral mood. Participants read for comprehension while their EEG was recorded. The results indicated that emotional state modulates the N400 effect. For the neutral mood condition, the N400 effect for a within-category violation (pines) relative to a match (palms) was smaller than that for a between-category violation (tulips) compared to the match condition. In contrast, for the happy mood condition, there was no difference in the size of the N400 effect for within-category violations and between-category violations. In the second study [18] , a neutral vs. happy vs. sad mood was induced by pictures. For the neutral mood condition, the same N400 pattern as in the Federmeier et al. study [16] was found. Within-category violations were modulated by emotional state: for the happy mood condition, a decrease in N400 effect for within-category violations was found compared to the neutral mood condition. For the sad mood condition, an increase in N400 effect for within-category violations was found compared to the neutral mood condition. Another study [19] explored the effect of emotional state on the processing of semantic reversal anomalies. A happy vs. sad mood was induced by film clips. For scenarios that conflict with general world knowledge (e.g., “The cat that fled from the mice”), a P600 effect was elicited for the happy mood condition but not for the sad mood condition. The studies reported above show that emotional state immediately affects semantic processing. In particular, the processing of different kinds of semantic mismatches―an unexpected word in the presence of a highly expected word [15] , a within- or between-category violation [16] [18] , an unexpected scenario based on world knowledge [19] ―gave rise to language by emotion interactions.

The results regarding the effect of mood on the processing of syntactic mismatches are less clear. Vissers and colleagues [20] were the first to investigate the effect of emotional state on the standard P600 effect to subject-verb agreement violations. Mood (happy vs. sad) was successfully induced by film clips. EEG was recorded while participants read sentences that were syntactically correct or incorrect. For P600, an interaction between emotional state and correctness revealed a strong reduction of the P600 effect for the sad as compared to the happy mood condition. In contrast, two other studies did not report an effect of emotional state on the P600 effect to syntactic violations. The first study [21] investigated the effect of emotional state on noun-adjective gender disagreements. Mood was varied by presenting context paragraphs with different emotional valences (happy, sad and neutral) that preceded neutral sentences containing a violation. No modulation of the P600 as a function of emotional state was found. However, as pointed out by the authors themselves, this outcome is likely due to the fact that their mood induction was less effective than that used by Vissers and colleagues [20] . The second study [22] examined the effect of emotional state on subject-verb agreement errors. Mood (happy vs. sad) was manipulated by film clips (like in [20] ). Again, P600 amplitude was not influenced by emotional state. In light of these inconsistencies, Verhees and colleagues [23] conducted a study to determine the reliability of the mood-related modulation of the standard P600 effect to subject-verb agreement errors [20] . For the present purposes, the most important finding of this study was that it replicated a mood-related modulation of the P600. In other words, this study demonstrates the robustness of the sensitivity of the P600 to emotional state.

Next to these emotion effects at the sentential level, an effect of emotional state on language comprehension has also been found at the discourse level. Egidi and Nusbaum [24] investigated the combined effect of emotional state, induced by film clips, and the valence of sentence endings (happy, sad, neutral) on semantic processing during discourse comprehension. They observed a larger N400 amplitude when the valence of the sentence ending and the participant’s emotional state did not match as compared to when they did match. More important for the present study, they showed that the effects of emotional state generalize to the discourse level. A second study [22] examined the effects of mood on the processing of verb-based expectancies. In the example “Peter annoyed Mary because he” vs. “Peter annoyed Mary because she”, the pronoun in the first sentence is highly expected whereas the pronoun in the second sentence is not. It was shown that mood affects referential anticipation. In particular, a positivity was elicited by bias-inconsistent pronouns (time window: 400 - 600 ms) relative to consistent pronouns in the happy mood condition, but not in the sad mood condition. This was taken to indicate that readers in the positive mood condition anticipated information about a specific person while readers in the sad mood condition refrained from making predictions about the pronoun.

To sum up, emotional state has been demonstrated to influence language processing at the sentence and the discourse level. Of importance for the goals of the present article, in all the studies reported above participants formed high expectancies about which linguistic events should occur. A mismatch between a highly expected representation―set up by a language context―and an actually encountered unexpected lexical item immediately led to a mood-related modulation of language-relevant ERP components, in particular, N400 and positivities in the P600 latency range.

So far, expectancies were generated by a mismatch between a linguistic context and an unexpected lexical item (noun, verb inflection or pronoun). Some researchers have claimed that expectancy plays a key role in mediating the inter- actions between language and emotional state [15] [19] [20] [22] [23] . That is, happy people seem to be more inclined to a top down processing strategy, following their expectations when processing sentences. Sad people though, adopt a bottom up processing strategy; they are not misled by expectancies and process sentences more systematically with greater focus on detail. Following this functional explanation of the mood effects on linguistic processing, similar effects of mood may be predicted when a high expectancy is elicited in a different modality. To our knowledge, the effects of emotional state on the integration of conceptually unexpected events, induced by a visual context and a sentence that either matches or mismatches the meaning of the visual context, has not been investigated.

¹It is relevant to mention that it has already been shown that expectancies can be effectively induced by other nonlinguistic contexts. This was demonstrated for the first time by Van Petten and Rheinfelder [26] . These researchers investigated the effects of environmental nonspeech sounds on auditory word recognition. To this aim they presented amongst others environmental nonspeech sounds like a barking sound and paired this with a related or unrelated spoken word (e.g., the word dog or horse). The main results were as follows: 1) The conceptual relationship between spoken words and meaningful nonspeech sounds affected the processing of both words and sounds, and 2) ERP context effects elicited by words and sounds were similar in terms of timing and morphology but differed in scalp distribution (greater involvement of the right hemisphere in the processing of words than sounds). A full discussion of the effects of nonlinguistic context effects on single-word processing is outside of the scope of this paper. For this we refer the interested reader to N400 review papers (e.g, [9] [14] ).

The goal of the present study was to fill this gap. The main question was whether mood effects on the processing of visually set up expectancies are similar or different from mood effects on the processing of linguistically set up expectancies. To this aim, we presented a visual (pictorial) context paired with a written sentence that either matched or mismatched the information provided by the visual context. Relevant for the present study, it has been shown that pictures are an effective means to induce a high expectation [25] ¹. Furthermore, asking participants to match combinations of pictures and sentential materials― referred to as picture-sentence matching task―is a standard paradigm to test language comprehension in patients with aphasia (e.g., [12] ). Hence, in the present experiment we used the picture-sentence matching task to generate a high expectancy for a particular event. It is generally assumed that when people see a picture, they immediately form some kind of mental representation of the displayed event. There is strong evidence from eye tracking studies that pictures are immediately integrated into context [25] [27] . Also, readers expect sentences to be true. The pictures utilized in the present study showed two geometric line drawings located either in a horizontal dimension (next to each other) or in a vertical dimension (above each other). They were followed by a sentence describing the locative relation of the figures by using a preposition of one of the two dimensions (horizontally or vertically). The picture elicited the strong expectation to subsequently read a correct description of the picture (i.e., a match: e.g., p¡―The square stands in front of the circle). A sentence giving an incorrect description of the picture (i.e., a mismatch) is very rare and thus highly unexpected. Incorrect descriptions contained a wrong preposition either within the same dimension (intra-dimensional mismatch: e.g., p¡―The square stands behind the circle) or a wrong preposition of the other dimension (extra-dimen- sional mismatch: e.g., p¡―The square stands above the circle.). Importantly, the extra-dimensional mismatches are more salient, in that they cannot be resolved by reassigning thematic roles [12] , as it is the case for the intra-dimen- sional mismatches. A clear advantage of using these specific materials was that the ERP signatures to these picture-sentence mismatches had already been examined. Important for this study, these conceptual mismatches have been shown to elicit robust ERP effects [28] , which is a prerequisite for finding potential mood by language interactions. Specifically, a biphasic ERP pattern was reported to conceptual mismatches: both intra-dimensional and extra-dimensional mismatches relative to matches elicited an early negativity around 200 to 400 ms (taken to reflect an N2b or N400) that was followed by a late positivity around 500 to 700 ms (taken to reflect a P600).

On the condition that the mood manipulation is effective, it is possible to address the effects of emotional state on the processing of conceptual mismatches triggered by a visual pictorial context. The predictions for the present study were as follows: if emotional state affects the processing of conceptual picture-sentence mismatches, this should be reflected by different ERP patterns for the happy vs. sad mood condition. Emotional state could have an effect on the early negativity interpreted as either an N2b (reflecting the detection of the mismatch [12] ) or N400 (taken to reflect processes of lexical access [29] or problems with post- lexical integration [30] ) or the late positivity (P600: taken to reflect general processes of reanalysis or more specifically, monitoring for processing errors [31] ). In contrast, if emotional state does not affect the processing of picture- sentence mismatches, a similar biphasic ERP pattern for both mismatch conditions, namely an early negativity followed by a late positivity, should be observed as in the Vissers et al. [28] article.

2. Methods

2.1. Participants

Thirty-eight (29 female) Dutch native speakers aged between 18 and 30 years (M = 21.03, SD = 3.28) participated in this study. All participants were right-handed as determined by an abridged Dutch version of The Edinburgh Inventory [32] . They had normal or corrected-to-normal vision and did not report any language disabilities or neurological, mental, or chronically physical impairment. Participants were compensated with 25?in cheques or an equal amount of course credit points. Each participant gave written informed consent.

2.2. Materials

The experimental material consisted of 288 picture-sentence pairs that were adopted from Vissers et al. [28] . Each picture displayed two different line drawings of a geometrical figure (a star, square, circle or triangle). The figures were arranged either horizontally next to or vertically above each other. Six different figure combinations―in two directions and along two dimensions―resulted in a total of 24 different pictures. Each sentence contained two noun phrases (one for each figure) and, in between, a verb phrase with a spatial preposition. The preposition described the location of one figure in relation to the other (horizontal: voor (in front of) and achter (behind); vertical: boven (above) and onder (below). All sentences were written in syntactically correct and unambiguous Dutch.

Pictures and sentences were combined in three experimental conditions. 1) In the match condition, the preposition in the sentence resulted in a correct description of the picture. 2) In a mismatch condition, the preposition in the sentence resulted in an incorrect description of the picture. 2a) If the wrong preposition was within the same dimension, we called it an intra-dimensional mismatch (e.g., voor (in front of) instead of achter (behind)). 2b) If the wrong preposition was of the other dimension, we called it an extra-dimensional mismatch (e.g., voor (in front of) instead of boven (above)). Table 1 provides an example for each condition.

Two ways of describing the 24 pictures correctly (based on which figure made the subject in the sentence), resulted in 48 trials in the match condition. The same number of trials was used for each of the two mismatch conditions, resulting in a total of 144 experimental trials. A complete list of experimental trials for each condition is provided in Tables A1-A3 in the Appendix.

An equal amount of similar picture-sentence pairs was included as filler trials. The prepositions in these trials were moved to the beginning of the sentence or replaced by less specific prepositions (naast(next to)/bij(at)). Still, all sentences were written in syntactically correct Dutch. In order to have an equal distribution of matches and mismatches in the experiment, 48 of the filler trials were mismatches and 96 matches.

The 288 trials were distributed across 4 blocks, each consisting of 72 trials of which the first was a filler item. Two different experimental lists were created, both with a pseudo-random order of trials. We ensured that the following events did not occur more than three times in a row: matches, mismatches in general, intra-dimensional mismatches, extra-dimensional mismatches, experimental trials, filler trials, the same preposition, and the same dimension of the picture. Identical pictures were divided by at least three other pictures.

2.3. Procedure

The experiment took place in a dimly lit Faraday cage in an EEG lab at the Donders Institute for Brain, Cognition and Behavior in Nijmegen. Participants were seated in a comfortable chair in front of a 23” LED screen (Samsung Sync

Master SA 700) with an approximate viewing distance of 90 cm. The screen had a refresh rate of 100 Hz. The stimuli were presented in white on a black background screen by a Dell computer (Dell Precision T3500) using the software Presentation (Neurobehavioral Systems, www.neurobs.com).

Participants were instructed to carefully look at the picture-sentence pairs. To ensure that they understood the two dimensions of the pictures (horizontally and vertically) and whether the description of the picture was correct or incorrect, we showed them examples of 6 filler items at the beginning of the experiment. Importantly, an earlier reaction time (RT) pilot study (N = 15) pointed out that participants are able to detect these picture?sentence mismatches successfully [28] . In this RT pilot study the participants had to press a button to indicate whether the sentence matched or mismatched the picture with their right (match) or left (mismatch) index finger.

The four experimental blocks followed, divided by short breaks. Each trial started with the presentation of a fixation cross for 510 ms to indicate the start of a trial. It was followed by a 500 ms blank screen after which the picture was presented for 2000 ms at the middle of the screen. The picture had an approximate size of 2.5 by 1.5 cm. Then, the sentence was presented in a serial visual presentation mode with a word duration of 345 ms and a stimulus onset asynchrony (SOA) of 645 ms, resulting in a 300 ms blank screen between the words. Words had an approximate height of 1 cm. The sentence final word was indicated by a full stop after the word. The length of the inter-trial-interval (ITI) was 1500 ms. To avoid eye movement artifacts, the participants were asked to only blink during the ITI.

2.3.1. Mood Induction Procedure

We presented short film clips (duration between 4.13 and 12.07 minutes) that have been shown to effectively induce differences in emotional state between participants (e.g., [15] [20] [23] [33] ). Four film clips of the Warner Brothers’ movie Happy feet (mean duration: 7.17 minutes) were used to induce a happy mood. Four film clips of the Second World War drama Sophie’s choice (mean duration: 7.42 minutes) were used to induce a sad mood. We asked the participants to empathize with the feelings and thoughts of the figures in the film clips, in order for them to enter a happy or sad mood.

As mood is a long-term psychological state [1] , it is unlikely to change within one experimental session. Presenting the language materials a second time is problematic, too, because it would elicit repetition effects, which complicate the interpretation of the online ERP effects. Therefore, mood was manipulated between participants. Half of the participants watched four happy film clips and the other half watched four sad film clips. In order to maintain the effect of the mood induction throughout the experiment, the four film clips were distributed such that each block of picture-sentence pairs was preceded by one of the four film clips. We asked the participants to rate their mood before the experiment as a baseline measure and after each of the four film clips on a rating scale from −10 (very sad) to +10 (very happy).

2.3.2. EEG Data Acquisition

For the recording of the EEG, 26 electrodes were mounted into an elastic cap on the participant’s head (Acticap system). The electrodes were placed at four midline sites (Fz, Cz, Pz, and Oz), 11 sites over the left hemisphere (F7, F3, Fc5, Fc1, T7, C3, Cp5, Cp1, P7, P3, O1), and 11 sites over the right hemisphere (F8, F4, Fc6, Fc2, T8, C4, Cp6, Cp2, P8, P4, O2). Additionally, electrodes were placed at the left and right mastoid, with the right mastoid serving as reference during recording. For a schematic overview of the electrode configuration see Figure 1. The electro-oculogram (EOG) was recorded bipolarly. Horizontal eye-movements were measured by placing electrodes at the right and left outer canthi. Vertical eye-movements were measured by placing one electrode above and one below the right eye. Impedances of all electrodes were below 20 kΩ. The signals were amplified with a time constant of 10 seconds, a bandpass filter (between 0.016 and 125 Hz) and were digitized online at 500 Hz. The signals were recorded by the program Brain Vision Recorder (Brain Products GmbH).

2.3.3. EEG Data Analysis

Pre-processing was performed in Brain Vision Analyzer (Brain Products GmbH). The recordings were re-referenced to the mean of the right and left mastoid. EEG and EOG signals were semi-automatically rejected as artifacts, if the amplitude between −100 ms and +1000 ms around critical word onset exceeded ±100 µV. The 100 ms epoch before stimulus onset was used as baseline and averages

were aligned to it. For each participant and condition, the mean amplitudes for consecutive time windows of 100 ms each were calculated starting from 0 ms (critical word onset) up to 1000 ms.

Based on visual inspection of the waveforms and consistent with the Vissers et al. study [28] , the following time-windows were used to quantify the relevant ERP components: 200 - 400 ms for the early time windows spanning the early negativity (N2b/N400) and 500 - 800 ms for the time windows spanning the late positivity (P600). Separately for the midline and the lateral electrodes, repeated measure ANOVAs were conducted with match (match, intra-dimensional mismatch, extra-dimensional mismatch) as within-subject factor and emotional state (happy, sad) as between-subject factor. The midline analysis included the additional within-subject factor site (Fz, Cz, Pz, Oz). The lateral analyses included the additional factors site and hemisphere (F7, F3, Fc5, Fc1, T7, C3, Cp5, Cp1, P7, P3, O1 for left, F8, F4, Fc6, Fc2, T8, C4, Cp6, Cp2, P8, P4, O2 for right). Relevant interactions of site, hemisphere or hemisphere by site involving the factors mood and/or match were followed up by step-down analyses.

3. Results

3.1. Mood Induction Procedure

The baseline mood ratings did not differ significantly between people in the happy mood condition (M = 5.11, SD = 2.11) and people in the sad mood condition (M = 5.26, SD = 2.00; t (36) = −0.24, p = 0.814). An ANOVA was carried out with the within-subject factor film clip (baseline, film clip 1, 2, 3, and 4) and the between-subject factor mood (happy, sad). Importantly, a significant interaction between mood and film clip (F(4,144) = 33.90, p < 0.001) was present. The interaction indicates that the mood induction in the present study was effective. In particular, people watching happy film clips were significantly happier than people watching sad film clips (p < 0.001). Likewise, people watching sad film clips were significantly sadder than people watching happy film clips (p < 0.001). Figure 2 shows a graphical overview of the results of the mood induction procedure.

3.2. Event-Related Potentials

The grand averages for intra-dimensional mismatches and extra-dimensional mismatches, each relative to matches, averaged across mood conditions and time-locked to the onset of the critical word, are presented in Figure 3 and Figure 4. Inspection of the waveforms suggests that both types of mismatches elicited an early negativity relative to the match condition between 200 and 400 ms. The negativity at some electrodes was followed by a late positivity starting around 500 and extending up to 800 ms. The positivity was larger for the mismatch conditions than the match condition and seemed more pronounced for the extra-dimensional mismatches than for the intra-dimensional mismatches.

3.2.1. Overall Statistical Analyses

After artifact rejection, the remaining 87.13% of the trials were included into the analyses. The percentage of rejected trials did not differ across experimental conditions (4.15%, 4.33% and 4.39% respectively). At least 25 trials remained per condition and participant, resulting in no further exclusion of data. Table 2 displays an overview of the results of the repeated measure ANOVAs for the different 100 ms epochs in the early and late time window.

*p < 0.05; **p< 0.01; ***p < 0.001.

Importantly, our analyses revealed neither a main effect of mood (all p values > 0.05) nor an interaction of match by mood (all p values > 0.05). For the time-windows spanning the early negativity (200 - 400 ms), a main effect of match was found at the midline sites (200 - 300 ms: p < 0.05 and 300 - 400 ms: p < 0.001) and at the lateral sites for the 300 - 400 ms epoch (p < 0.001). For the time-windows spanning the late positivity (500 - 800 ms), no main effect of match was found at the midline (all p values > 0.05). For the lateral electrodes, a significant main effect of match was present for the three consecutive time windows (500 - 600 ms: p < 0.05, 600 - 700 ms: p < 0.01, and 700 - 800 ms: p < 0.05).

Based on the main effects of match in the overall analysis, follow-up analyses were carried out separately for the intra-dimensional mismatches relative to the match condition and for the extra-dimensional mismatches relative to the match condition. The main results of these analyses are presented in Table 3.

3.2.2. Intra-Dimensional Mismatches

Again, the analyses of the intra-dimensional mismatches yielded no main effect of mood (all p values > 0.05) or an interaction between match and mood for any of the time windows (all p values > 0.05). A main effect of match was found for the early window (200 - 400 ms), but only in the 300 - 400 ms epoch for both the midline and the lateral electrodes (both p values < 0.001). The main effect of match reflected significantly more negative-going mean amplitudes for the intra-dimensional mismatch as opposed to the match condition for all electrodes in the early window spanning N2b/N400 activity. For the late window covering the late positivity (500 - 800 ms; spanning P600 activity), no significant effects of match were found (all p values > 0.05).

3.2.3. Extra-Dimensional Mismatches

As for the intra-dimensional mismatches, no main effect of mood (all p values > 0.05) nor an interaction between match and mood was found for midline or lateral sites (all p values > 0.05). Main effects of match were found for the early time window (200 - 400 ms) at the midline for both epochs (200 - 300 ms: p < 0.01 and 300 - 400 ms: p < 0.001) and at the lateral sites for the 300 - 400 ms epoch (p < 0.001). These main effects of match indicated more negative-going mean amplitudes for the extra-dimensional mismatch condition compared to the match condition for the N2b/N400 time window distributed over all sites. Note, that this early effect was longer lasting for extra-dimensional than intra- dimensional mismatches. Additionally, in contrast to the intra-dimensional mismatches, a main effect of match was found for the late window (500 - 800 ms) at the midline sites for the 600 - 700 ms epoch (p < 0.05) and at the lateral

*p < 0.05; **p < 0.01; ***p < 0.001.

sites for all three epochs (500 - 600 ms: p < 0.05, 600 - 700 ms: p < 0.01 and 700 - 800 ms: p < 0.05). These effects reflected significantly more positive-going mean amplitudes for the extra-dimensional mismatch condition than the match condition for the P600 window distributed over all sites.

3.3. Summary of the Main Results

The mood induction was effective: Participants watching happy film clips were significantly happier than participants watching sad film clips. Regarding the ERP results, no effect of mood on the processing of conceptual mismatches was found, neither for the early window (200 - 400 ms; spanning N2b and N400 activity) nor for the late window (500 - 800 ms; spanning P600). Interestingly, however, intra-dimensional mismatches showed a monophasic ERP pattern whereas extra-dimensional mismatches showed a different, biphasic ERP pattern. In the early time window (200 - 400 ms), both mismatches, but more prolonged for the extra-dimensional mismatches, elicited more negative going mean amplitudes as compared to matches. In the late positivity window (500 - 800 ms), more positive going mean amplitudes for mismatches as compared to matches were only found for the more salient extra-dimensional mismatches and not for the intra-dimensional mismatches.

4. Discussion

ERP studies at the sentence level have shown that a participant’s background emotional state―like being in a happy or sad mood―has an immediate influence on the processing of semantic and syntactic information (e.g., [15] [16] [20] ). In addition, it has been demonstrated that emotional state impacts on-line processes of language comprehension, in particular processing of word meaning and verb-based expectancies, at the discourse level [22] [24] . These studies have in common that a strong expectancy was evoked by a linguistic context (written/spoken language) which triggered a mismatch with a highly expected item (e.g., “The pillows are stuffed with feathers…”) and the actually encountered lexical item (e.g., “The pillows are stuffed with books…”). The goal of the present study was to investigate whether, and if so how, mood affects the processing of a visual context paired with a written sentence that either matches or mismatches the meaning depicted in the picture. Here, in contrast with previous studies, pictures were used to induce a strong expectancy for a particular event. A spatial array was followed by a sentence that matched or mismatched the picture (intra-dimensional mismatch: p¡―The square stands behind the circle; extra-dimensional mismatch: p¡―The square stands above the circle). Extra-dimensional mismatches, in contrast with intra-dimensional mismatches, do not involve possible role switches. Hence, these mismatches are more salient than intra-dimensional mismatches because they can in no way be repaired by reassigning thematic roles. The main question we asked was whether emotional state influences the processing of conceptual mismatches in a similar or different way compared to purely linguistic mismatches.

A necessary precondition for investigating the effects of emotional state is an effective mood induction. In the current study this was the case: after watching film clips from the happy movie, participants were significantly happier compared to the sad mood condition. Likewise, after viewing film clips from the sad movie, participants were significantly sadder compared to the happy mood condition. With these data at hand we can determine the effects of emotional state on the processing of picture-sentence mismatches.

The main finding of this ERP study was that for the present conceptual mismatches no interaction between emotional state and match was found. In other words, being in a happy or sad mood did not yield different ERP patterns in the picture-sentence matching task. The absence of an interaction between emotional state and match indicates that the effect of mood―previously reported to several kinds of language mismatches triggered by a linguistic context―does not generalize to situations in which a high conceptual expectancy is induced by a picture.

As sketched in the introduction, the investigation of interactions of language comprehension and other systems, like emotion, perception, and attention, is of theoretical relevance. This is the case because theories make different predictions regarding the effects of emotional state on language comprehension. According to modular (abstract symbol) theories of language ( [3] , and e.g., [34] ), processing of word meaning and syntactic information is encapsulated, and, therefore, should not be influenced by other systems like emotion, attention or action. In contrast, embodied theories, in which comprehension entails simulating actions, predict an effect of emotional state on language processing [5] [6] . In line with this, the presence of language by emotional state interactions for N400, have been presented as support for embodied theories (e.g., [15] ) over modular (abstract symbol) theories. The absence of a modulation of the processing of the conceptual mismatches in the picture-sentence task reported in this article, therefore, seems to support modular theories of language and challenges embodied theories of language comprehension. Regarding possible theoretical implications of the present ERP results, one aspect of the present materials deserves special attention. The visual context consisted of pictures depicting the combination of two geometrical figures (squares, triangles, stars and circles). According to embodied theories, to compute meaning, we project ourselves into a situation and simulate possible actions. On this view, the structure of the body is very important in that it determines the range of effective actions. The use of inanimate geometrical figures from an embodied perspective, therefore, could be criticized because it hampers an involvement of the emotional system when simulating the actions. This, in combination with the abstract scenarios depicted in the present study (e.g., The star stands behind the circle), may explain the absence of an interaction between language and emotion for these specific pictorial mismatches. To address this issue, future studies should examine the effects of emotional state on the processing of conceptual mismatches using animate (human-like) creatures involved in more natural actions (see for example for the use of more natural materials [12] [35] ).

The question remains whether mood did not have any influence on the on-line comprehension process. To answer this question we compared the ERP effects in the present study (in which emotional state was successfully manipulated) with those of Vissers and colleagues ( [28] ; without a mood manipulation). A comparison of the ERP patterns across mismatch conditions revealed differences in the ERP effects between studies. In the previous study [28] , a biphasic ERP pattern (N2b/N400 followed by a P600) was reported for both intra-dimen- sional and extra-dimensional mismatches. In contrast, in the present study a monophasic N2b/N400 effect was found for the less salient intra-dimensional mismatches whereas a biphasic ERP pattern (N2b/N400 followed by a P600) was found for the extra-dimensional mismatches. To check whether the observed ERP patterns in the present study differ significantly from those of Vissers et al. [28], a global analysis was carried out to directly compare the P600 results (for the 500 - 700 time-window) across studies. Given that P600 effects are largest at centroparietal midline sites (i.e., Cz, Pz), a repeated measures ANOVA with the between-participant factor experiment (previous study, present study), and the within-participant factors match (match, intra-dimensional mismatch, extra- dimensional mismatch) and midline electrodes (Fz, Cz, Pz, Oz) was conducted. This global analysis yielded a main effect of match (F(2,132) = 19.95, p < 0.001), and importantly, an interaction between match and experiment (F(2,132) = 5.21, p < 0.01). The match by experiment interaction indicated that in the study without mood manipulation, a P600 effect was present across the mismatch conditions whereas in the present study with mood manipulation, a P600 effect was only found for extra-dimensional mismatches and not for intra-dimensional mismatches. Although we miss information about the participants’ emotional state in the Vissers et al. study [28] , it seems reasonable to assume that the subjects in that study were in a more neutral state than in the present study in which a sad or happy mood was successfully induced. In line with this, we take the interaction between experiment and match in the global analysis to suggest that emotional state had an impact on the P600.

So far, we did not elaborate on the functional interpretation of the ERP effects reported in this study. Lining up to the interpretation of the ERP effects in the previous study, there are two possibilities to interpret the early negative effect for all mismatches (see [28] ). First, it could be an N2b, reflecting the mismatch between the conceptual representation based on the picture and the conceptual representation derived from the actual sentence. The early negativity could also reflect an N400-like effect. The early onset of the N400 could be explained by the fact that the semantic expectancy induced by the picture is not confirmed (see also [36] ). Assuming that readers typically expect a sentence to be true, they will expect a sentence which describes the picture correctly. A sentence giving a false description is thus unexpected, which could be reflected by N400. We take the P600 effect to reflect a process of reanalysis. While most researchers would agree that the P600 indexes processes of reanalysis, there is a debate about the purpose of the reanalysis (see [37] [38] for reviews). Specifically, we have proposed that the function of the reanalysis is to check the input for possible processing errors. This view has been referred to as the monitoring hypothesis of P600. On this view, a conflict exists for the mismatches between the conceptual representation based on the picture and the conceptual representation derived from the actual sentence. Specifically, the strong expectation emanating from the picture conflicts with the preposition which describes the picture incorrectly. In line with this, the P600 to extra-dimensional mismatches observed in this study is taken to reflect a monitoring response to check for processing errors after this conflict between two conceptual representations.

The presence of a P600 to extra-dimensional vs. absence of a P600 for intra-dimensional mismatches can be taken to reflect differences in the situations that lead to reanalysis in a more neutral mood (previous study without mood induction) vs. emotional state (present study). A possible explanation for the P600 differences reported in the present article is that being in an emotional state (be it positive or negative) influences the processing of conceptual mismatches in the picture-sentence matching task. In particular, we take the present P600 results to suggest that people in an emotional state detect conceptual anomalies only when they are salient. This is the case for extra-dimensional mismatches because―opposite to intra-dimensional mismatches―they can in no way be repaired by reassigning thematic roles.

But how come that in the present study reanalysis is exclusively triggered after salient conceptual mismatches in an emotional state, while in earlier studies from our laboratory all linguistic violations mostly triggered reanalysis in a happy mood condition? A possible explanation for this finding could be found in the comparison of linguistic expectations (at the semantic and syntactic level from earlier studies) with the conceptual expectations in the present study.

While the conceptual expectations in the present experiment are exclusively induced by the visual context of this task, the semantic and syntactic expectations set up by the linguistic context from earlier studies are always driven by both word and world knowledge developed during a lifetime. After all, despite that language comprehension and speech errors are highly complex processes, comprehension and speech errors are very rare. Therefore, a reader’s default expectation is for sentences to be grammatical and in accordance with their semantic and syntactic knowledge [39] [40] [41] [42] . Generally, building on these linguistic expectations, that is, following ones rules of thumb or heuristics, is a highly effective and economical strategy (e.g., [43] ). However, for the semantic and syntactic anomalies, the strong expectations are based on heuristic processing conflicts with the actual sentence representation. This conflict between an expected representation based on heuristic processing and an unexpected representation is taken to trigger the P600 effect. In other words, these studies point out that the P600 evoked by violations, reflects heuristic processing and reanalysis based on the degree of expectation set up by the context. The observed modulation of P600 to syntactic and semantic violations by mood could thus be due to the strong reliance on heuristics. Since happy people are known to rely more on heuristics than sad people [44] , it is not unsurprising that the P600 effect is more pronounced for linguistic violations in a happy mood. Based on heuristics, the reader has a strong expectation for a semantically and syntactically correct word; a conflict arises when an unexpected word is encountered. This conflict is taken to trigger the brain to reanalyze the linguistic input which is reflected by P600 (see also [20] ). In the present study, the visually induced expectation, set up by geometrical figures, is only current during the task performance and deviates from natural settings. Hence, the conceptual expectation is far less strong than the linguistic expectations in earlier studies which are fuelled by a rich source of wor(l)d knowledge. From here we can explain that in the present study, the P600 effect is more pronounced after salient conceptual mismatches in an emotional state. After all, given the weaker conceptual expectation, monitoring is only necessary when the conceptual violation is highly salient and cannot be repaired by thematic role reassignment.

5. Conclusion

To conclude, the effect of a happy mood versus a sad mood on language processing of linguistic mismatches does not generalize to the processing of conceptual mismatches. Possible explanations for this finding are discussed above. Nevertheless, emotional state seems to matter. In an emotional state, a monitoring response is exclusively triggered after highly salient conceptual mismatches that cannot be repaired easily. The characteristics of this effect have to be investigated in more detail by future investigations on the processing of language at the conceptual level using paradigms in which conceptual expectations are set up in a more ecologically valid way, imitating language processing in natural settings.

Acknowledgements

This research was supported by the Radboud Honours Academy and the Donders Centre of Cognition (DCC) in Nijmegen. Parts of this research have been presented at the Annual Meeting of The Cognitive Neuroscience Society 2015 in San Francisco. We thank Mathieu Koppen for the excellent supervision of this honours project, Daniel Fitzgerald for providing us with the mood induction procedure, and the TSG at the DCC for their technical support, in particular Miriam Kos and Pascal de Water. We are grateful for the fruitful feedback on the results we received from Eleonora Rossi, Judith Kroll and other members of the Center for Language Science (CLS) at Penn State. Furthermore, we thank the reviewers for their constructive comments on a previous version of this article.

Cite this paper

Goertz, R.I., Vissers, C.T.W.M., Verhees, M.W.F.T. and Chwilla, D.J. (2017) Effects of Emotional State on Processing Conceptual Mismatches: ERPs in a Picture-Sentence Matching Task. Journal of Behavioral and Brain Science, 7, 557-584. https://doi.org/10.4236/jbbs.2017.712040

References

Appendix