The Representation of Affect in Chinese Visual-Verbal News Commentary: A Corpus-Based Study ()
1. Introduction
Affect, as one of the three semantic concepts within the sub-system of attitude in appraisal theory, is concerned with emotions that people react to behavior, text/process, and phenomena [1] . Previous studies showed that affect can primarily be divided into five typological pairs of positive-negative groups: un/happiness, in/security, dis/satisfaction, dis/inclination and not/surprise [1] [2] [3] . As one of the most common forms of multimodal discourse, visual-verbal discourses are usually loaded with affect of these different kinds. However, compared to studies of verbal affect, the analysis of affect in visual-verbal discourse has not attracted attention until recently, and is mainly discussed together with the other two kinds of attitude, i.e., judgment and appreciation. Economou conducts a system of visual appraisal based on the analysis of verbal-visual news media texts and finds that inscribed attitude in corpus is mainly related to primary human emotions [4] [5] [6] . Bednarek & Caple undertake a case study of an on-line multimodal reporting, which illustrates that news values were constructed by choices in language and image [7] . Pounds employs Appraisal Theory and analyzes the multimodal expression of authorial affect in a British television, which reveals that authorial affect can be conveyed through the audio-visual and dialogic dimensions [8] . As for researches on visual/verbal resources for the representation of attitude, Hood summarizes four different kinds of explicit verbal resources [9] . Martin & White exemplify how adjectival and the selection of ideational meaning can attribute to affect [1] . Bednarek provides a list of common English emotion terms [3] . Liu classifies eight different structural potentials of affect in English [10] . Lemke summarizes that there were 8 types of visual/verbal resources for evaluative meaning in political cartoon [11] . Martinec analyzes how affect could be expressed through actions [12] . Forceville discusses the visual representations of the idealized cognitive model of anger, one typical primary affect [13] . Feng & O’Halloran present how the semiotic resources of facial expression, touch and body orientation can be used to represent emotive meaning [14] . Swain exemplifies that visual affect is easily inscribed in the faces, gestures and body postures of depicted human participants [15] . These studies gave us a better understanding of visual/verbal resources for representing affect and other attitudes. However, few has tried to make use of a Mandarin Chinese visual-verbal corpus to undertake a quantitative analysis of the representation of affect in such discourse, which leaves large space for further investigation. Considering this, this study endeavors to examine the representational characteristics of affect in Chinese visual-verbal news commentary based on a quantitative analysis of the corpus so as to provide a possible analytical modal of the quantitative examination of multimodal attitude in general including affect, judgment and appreciation, especially in Chinese corpus.
2. Representational Factors and Resources of Affect in Chinese Visual-Verbal New Commentary
2.1. Representational Factors
Chinese visual-verbal new commentary is a kind of multimodal discourse that is composed of a text and a cartoon, the two of which together shows the authorial stance toward a certain negative social phenomenon. In this study, a distinction between macro-attitude and micro-attitude (and corresponding, between macro-affect and micro-affect) is made. The former refers to the overall authorial stance, the latter refers to the specific attitudes that can be identified in specific verbal expressions or in the cartoon. Generally speaking, the macro-attitude is realized by the designing of specific micro-attitudes, among which there are a lot cases of micro-affect. From the perspective of Systemic-Functional Linguistics, affect, like other kinds of attitudinal meaning, is not randomly realized in discourse, either. On the contrary, generally the representation of affect can be regarded as a complex process of systematic choices-making of different representational factors, which involves at least five aspects as followed:
1) Affect host. Pre-sample analysis shows that in general, there are two types of affect host in Chinese visual-verbal new commentary, i.e., the author and the third party on the one hand, and the depicted participants on the other hand. The latter further consists of two parts, the disadvantaged participant such as the victims of a certain event, etc. and the negative participant such as a bribe, etc.
2) Affect target. Nine types of affect target were found in the corpus: the negative behavior, the negative thing, the negative figure, the negative value, the positive behavior, the positive thing, the positive figure, the positive value, and the poor encounter of the disadvantaged participant.
3) Polarity. Although the macro-attitude is basically negative, for micro-attitude such as affect, there are both positive (positive, good, optimistic, encouraging, etc.) and negative affects (negative, bad, pessimistic, discouraging, etc.).
4) Semantic types of affect. Specific cases of affect may show one kind of concept out of the five semantic groups: un/happiness, in/security, dis/satisfaction, dis/inclination and not/surprise.
5) Semiotic resources. The representation of affect in Chinese visual-verbal new commentary makes use of both verbal and visual, both implicit and explicit resources.
2.2. Representational Resources
In Chinese visual-verbal new commentary, the representation of affect makes use of both verbal and visual resources. On the one hand, Martin & White (2005: 58) pointed out that the canonical grammatical realization for attitude is adjectival. Actually, other content words such as attitudinal nouns like 热情 (rè qíng, ambition), verbs like 钟情 (zhōng qíng, deeply love) and adverbs like 快快乐乐地 (kuài kuài lè lè, happily) are also potential explicit resources. In other words, explicit verbal representational resources are attitudinal content words. As for the implicit resources, there are mainly three types: the selection of ideational meaning (Martin & White, 2005: 62), the selection of grammatical mood, and the use of rhetorical devices.
On the other hand, on the basis of previous findings and further large scale of corpus analysis, Wang summarized that explicit visual resources as embodied behavioral processes which include reactional processes (e. g, eye gaze of the depicted participant(s), etc.), ideological behavioral processes (e.g., smiling, laughing, crying, tearing, etc. of the depicted participant(s)), emotive gestural processes (e.g., dancing, singing of the depicted participant(s)), and the appendant of embodied emotive processes (e.g., tears, beads of sweat, etc.) [2] . As for the implicit visual resources, sometimes non-embodied behavioral processes like the depicted participant’s behaviors of cheering-jump, hands-back, hands-in-pocket, etc. can also represent affect. Besides, depicted circumstances in cartoons of Chinese visual-verbal new commentary as well as the use of visual metaphor can also implicitly trigger affect.
3. Corpus Collection and Annotation
3.1. Corpus Collection
As is pointed out above, Chinese visual-verbal new commentary mainly shows the authorial stance toward a certain negative social phenomenon. To make the corpus collection as representative as possible, the process of corpus collection was in the first place designed to cover all of the five main types of social problems of contemporary China: problems related to social norms (corruption, moral abnormality, etc.), social equality (rural-urban disparity, rich-poor disparity, etc.), social organization (educational problems, policy issues, etc.), public safety (food safety, medical issues, etc.), and living and ecological environment (water and air pollution, etc.) [16] [17] . Next, after a careful and thorough investigation of Chinese journals that publish visual-verbal news commentary, the magazine 杂文选刊 (Essays Digest) was selected as the source of corpus due to the reason that on the one hand, this magazine, consisting of different columns, is rich in target discourse, that is, the discourse of visual-verbal news commentary. On the other hand, as a magazine founded for 30 years, news commentary published on it cover all the topics of social problems, which makes it a qualified source for corpus collection. Third, a corpus-selection process was designed to ensure that corpus collected is appropriate target discourse. On the one hand, cartoons of some news commentary are not related to the social problems being discussed but only appeared on the page for the sake of layout design. Therefore, discourses of this kind were excluded. On the other hand, cartoons of some other news commentary are of foreign origin, whereas this study focuses on the analysis of Chinese visual-verbal news commentary. Correspondingly, discourses of this kind were also excluded.
Finally, a corpus of 100 cases of Chinese visual-verbal new commentary was established, which consist of 100 verbal commentaries and 100 cartoons. The rough total number of Chinese characters of the former is 130,000.
3.2. Corpus Annotation
The corpus was systematically annotated to assist further quantitative analysis of the data. To begin with, an initial processing, putting every single target visual-verbal discourse into an individual document, was undertaken, which was followed by an initial identification of micro-attitude. Any clause in which micro-affect was identified was put into a single line. And for those clauses in which more than one micro-affect were identified, clauses were repeatedly listed in a single line by corresponding times. The same processing is applied to cartoons. Second, the labels “TITILE-text”, “TITILE-cartoon”, “BALLOONS-talk” (if any), “BALLOONS-thought” (if any), and “CARTOON” were added to the beginning of respective part of every single visual-verbal discourse for the sake of upcoming statistical analysis. Third, all the corpora were inserted into Microsoft Excel 2010 for annotation. Here, factors of the semantic types of affect, of affect host, of polarity, of semiotic resources, and also of affect target were annotated and inserted into individual rows. The whole annotating process was finished by the co-work and checking of the research group and special attention was paid to the following points: 1) The correct identification of which specific concept a case of affect belongs to relies deeply on the analysis of the role context plays. Therefore, context was always examined before a final annotation was made about the type of a single case of affect. This is also true with regard to the identification of the polarity of affect. For example, the same adjective 快乐 (kuài lè, happy) is used in both sentences. But it represents a positive evaluation of the target (school skipping) in the latter sentence, whereas in the former one no affect is represented but only an ideational meaning of happy childhood is expressed.
Case 1:
唯独忽略其中最为重要的一点，那就是还孩子们应有的快乐童年。
Only what is neglected is the most significant part, that is, children should be given back their deserved happy childhood.
Case 2:
逃学才快乐。逃学真妙不可言，算是人生一大幸福！
It is nothing but school skipping that is happy. The experience is extremely too wonderful for words. It can be regarded as one’s big moment in his life!
2) To ensure consistent annotation of the visual corpus, i.e., cartoons, this study first examined the narrative meaning of a cartoon according to the context, on the basis of what the semantic type of affect was decided. Then, other representative factors about the affect were annotated. For those cartoons in which there were more than one visual affect, the affect represented by talk-balloons and thought balloons was analyzed first, and those represented by visual resources including the depicted participant(s), visual transitive processes, and the depicted circumstance were analyzed successively.
4. Results and Discussion
4.1. Overall Analysis
Table 1 reveals that there are 410 cases of affect found in the corpus with 386 verbal ones and 24 visual ones. Although the amount of visual affect is much less than that of verbal ones, visual affect attributes to a much higher percentage (20.3%) in all the visual attitude than verbal ones (9.5%), which implies that visual affect occurs much more frequently than verbal ones in the corpus. This imbalance is actually understandable. On the one hand, affect is concerned with emotions that people react to behavior, text/process, phenomena. In other words, the subjectivity of affect is relatively stronger compared with the attitude of judgment and appreciation. However, Chinese visual-verbal news commentary aims to persuade readers to accept its authorial stance toward the being discussed social issue. The achievement of this requires that the designing of such discourse should help build a discourse that reads as objective as possible and show as little subjectivity as possible, that’s the possible reason why only a few cases of verbal affect were designed. On the other hand, cartoons as visual signs can vividly represent the critical figures, moment and happening of the social issue being discussed. For instance, by showing the depicted participant’s visual affect, it makes it easier to arouse a resonating emotion on the side of readers. This is in consistence with the overall need of Chinese visual-verbal new commentary. From these two perspectives, the statistical findings in Table 1 are not difficult to understand.
Table 2 shows the distribution of affect according to the types of affect host. As is listed, there are 199 cases of affect from the depicted disadvantaged participant (62.8%), 46 cases of affect from the depicted negative participant (27.7%), 45 cases of affect from the third party (9.7%), and 120 cases of authorial affect (3.7%). It reveals that on the one hand, the frequency of occurrence of visual affect from the depicted sides (the disadvantaged and the negative participant) is much higher than that of visual affect from the authorial and the third-party sides. On the other hand, the most common visual affect in the corpus, whether in amount (199) or by percentage (62.8%), is the visual affect from the disadvantaged participant, which covers even more than half of all the affect from the disadvantaged participant. Visual affect from the negative participant is also
![]()
Table 1. Overall distribution of affect in the corpus.
![]()
Table 2. Distribution of affect according to the types of affect host.
more common compared with those from the author and the third party. This statistical result is actually not difficult to understand, either. For one thing, the visual representation of affect from the depicted sides provides the readers with a reproduction of the event without hurting the degree of narrative objectivity of the discourse. However, the objectivity of authorial emotive evaluation is easily expressed if too many cases of authorial and third-party affect were visually expressed, which further breaks the degree of reliability of authorial stance. Table 3 illustrates the distribution of affect according to the types of affect target. It is revealed that no affect toward the positive figure and positive values was found in the corpus. And those cases of affect toward the positive things, the negative values, and negative figures are also quite rare (the respective percentage is below 3% for all.). On the contrary, nearly half of all the cases of affect in the corpus are toward the negative behaviors.
Take the finding of Tables 1-3 into consideration together, we can come to an overall picture of the affect in the corpus: affect only attributes to a low percentage of attitude but the frequency of occurrence of visual affect is much higher than that of verbal affect. According to the types of affect host, most cases of affect belong to affect from the disadvantaged participant, and most are emotively evaluation of the negative behaviors.
4.2. Detailed Analysis of the Visual-Verbal Representation of Affect
4.2.1. Representation of Affect from the Depicted Disadvantaged Participant
Among the 199 cases of affect from the depicted disadvantaged, there are respectively 97, 61, 20, 14, and 6 cases that are affect toward the poor encounter of the disadvantaged, of the negative behavior, of the negative thing, of the positive behavior, and of the positive thing, and there is only one case that is affect toward the negative value. As far as the semantic types of affect are concerned, most cases are either affect-in/security or affect-un/happiness. In addition, 17 out of 199 are visual affects with others being verbal affects. Next, 97 cases of affect toward the poor encounter of the disadvantaged will be exemplified in detail. Specifically speaking, there are 55, 36, 3 and 3 cases which respectively are affect-insecurity, unhappiness, disinclination and surprise.
Case 3:
这个家庭的母亲被随时可能夺去生命的疾病折磨，为无法支付的医疗费用发愁。
![]()
Table 3. Distribution of affect according to the types of affect target.
The mother in the family suffers from a fatal disease and is anxious about being uncapable of paying the medical cost.
Cases of affect-insecurity are all verbal affect. For example, in Case 3, the use of the attitudinal adjective 发愁 (fā chou, anxious) explicitly represents the affect of insecurity of the depicted disadvantaged, that is, the badly sick mother who worries a lot about how to pay the huge medical cost. Cases of affect-unhappiness were found both in visual and verbal corpora. For instance, Case 4 below expresses the depicted disadvantaged farmer’s strong unhappiness by the use of the attitudinal adverb 难过得 (nán guò de, sadly).
Case 4:
济南那位农民六亩田的卷心菜蚀大本，难过得喝农药寻了短见。
The six acres of cabbage was a great loss for the farmer from Jinan, who sadly killed himself by drinking pesticide.
Cases of affect-disinclination were all verbal ones. Take Case 5 below for instance. The difficulty of getting medical service is still quite huge for many ordinary people in China. The affect-disinclination of Gao Yan, the depicted disadvantaged working-class father whose daughter was suffering from severe illness, is easily represented through the attitudinal adverb 只好 (zhǐ hǎo, have no choice but to). There are both visual and verbal cases of affect-surprise. Case 6 is from the same discourse as that of Case 5. The father Gao Yan and his wife were quite unexpected and shocked (the affect-surprise) when they noticed that there were lots of people who were waiting for check-in before them. Similarly, this emotive evaluation was represented by the attitudinal adverb 没想到 (méi xiǎng dào, unexpectedly).
Case 5:
为了尽快让女儿就诊，高岩只好多花钱。
To have his daughter’s medical treatment begin as soon as possible, Gao Yan had no choice but to spend more money.
Case 6:
高岩夫妻俩早七点赶到北京儿童医院去挂号，没想到人山人海。
Gao Yan and his wife got up early around 7 to 8 am to check in the Beijing Children’s Hospital, but unexpectedly they found that there were a crowd of people.
In Case 7 (Figure 1), there are two depicted participants, one left boy who was laughing and one right boy who lowered his head and showed a sad face with frowned eyebrows, a typical embodied emotive behavioral process. Take its verbal context into consideration, this process explicitly represents the boy’s affect-unhappiness, i.e., he was sad thinking that his father was not as rich as fathers of other children. In Case 8 (Figure 2), the depicted disadvantaged man, that is, the old grandfather on the left, is equipped with a widely opened mouth, which is an ideological behavioral process, expresses the grandfather’s strong surprise (affect-surprise) he felt when he realized how difficult it was to check in a good hospital for medical service.
![]()
Figure 1. Cartoon “I have a good father”.
![]()
Figure 2. Cartoon “Getting medical care is difficult”.
4.2.2. Representation of Affect from the Depicted Negative Participant
Among the 46 cases of affect from the depicted negative participant, there are 7 cases of visual affect and 39 cases of verbal affect. As far as the type of affect target is concerned, more than 36% of the cases are affect toward the positive behavior. Examination of these cases demonstrates that they mainly express the inappropriate negative affect of the negative participant toward good behaviors. Take Case 9 and Case 10 for example.
Case 9:
而获救者吝于言谢，给这起悲剧画上黯淡的结尾。
However, the rescued man was too mean to express his gratitude, which drew a dark end of that tragedy (the saver drowned himself).
Case 10:
女副调研员面对律师“对医院出现医托的现象监督了没有”的质问，当场发飙。
The female vice investigator, when was questioned “is the problem of hospital scalper supervised or not?”, burst into great anger on the spot.
In the context of Case 9, the saver died of drowning when managing to save a man. However, the saved man just left the spot without showing any gratitude toward the saver. 吝于言谢 (lìn yú yán xiè, reluctant to express one’s gratitude) explicitly reproduced the negative participant (the saved man)’s inappropriate emotive reaction toward the saver (disinclination). It’s quite possible that clear resonation like this will make it easier to arouse resonation between readers and the author. In other words, the design of such affect in news commentary results in a stronger expression of authorial stance. Similarly, in Case 10, 发飙 (fā biāo, burst into great anger) also represents the inappropriate disinclination.
Among those 7 cases of visual affect, 5 are affect toward negative behavior and 2 toward the poor encounter of the disadvantaged participant.
In Case 11 (Figure 3), a group of civil servants, as the depicted negative participant, were designed as all showing broad smile, an embodied emotive behavioral process that represents their positive affect-happiness toward the controversial issue of travel at public expense. Similarly, the three embodied emotive behavioral processes in Case 12 (Figure 4) represent the three depicted negative participants’ affect-disinclination toward the wounded, the design of which helps build the resonation between the author and readers about their criticism against the depicted negative participants.
4.2.3. Representation of Authorial and Third-Party Affect
Different from affect from the depicted participants, all cases of authorial and third-party affect found in the corpus are verbal affect. And neither authorial affect nor third-party affect is common in the corpus. Only about 3.7% of authorial attitude and 9.7% of third-party attitude are affect. This distribution is consistent with the genre characteristics of news commentary, as is discussed in 4.1. In detail, 80 out of the 120 cases of authorial affect are affect about negative behavior under discussion in corresponding news commentary. As far as the semantic types of affect are concerned, the three most common types are affect-in/security, affect-un/surprise, and affect-un/happiness. As for the third-party affect, it is also the truth that most cases are affect toward the negative behavior (23 out of 45), and the three most common semantic types are affect-dis/satisfaction, affect-in/security and affect-un/happiness. The following two examples show respectively how authorial and third-party affect are represented in the corpus.
Case 13 is an example of typical authorial affect in which the use of 令我们心痛 (lìng wǒ men xīn tòng, makes us heartbroken) explicitly shows the authorial criticizing stance toward the negative social phenomenon of people’s indifference. Similarly, the negative emotive attitude of the public, i.e., the third party, is represented by the expression of 公众愤慨 (gong zhòng fèn kǎi, the public are indignant).
Case 13:
更令我们心痛的是，两个青年死亡六天了，没有人出来公开道歉。
What makes us heartbroken is that no one came out to apologize even six days after the death of the two young man.
Case 14:
然而，各地发生的“吃空饷”怪象，令公众非常愤慨。
However, the weird phenomenon of freeloading, happening in quite a few places, made the public very indignant.
5. Conclusions
This study addresses the issue of how the attitude of affect is visually and verbally represented in the corpus of Chinese news commentary. A thorough quantitative analysis reveals that the visual-verbal representation presents three main characteristics. First, compared to judgment and appreciation, the frequency of occurrence of affect is low and the most frequently occurred affect is the affect from the depicted disadvantaged participant. Second, while authorial and third-party affects are mainly negative affect toward negative behaviors, affects from the depicted disadvantaged participant mainly express their negative emotion toward the poor encounter, and affects from the depicted negative participant mainly show their negative emotion toward positive behaviors. Third, the majority of affects are verbal affects. While there are both visual and verbal affects from the depicted participants, no visual authorial or third-party affect is found.
Within the theoretical framework of Systemic Functional Linguistics, the representation of affect can be regarded as a complex process of systematic choices-making of different representational factors. Actually, the findings above exactly demonstrate such process. For one thing, the design of authorial and third-party affect directly manifests the macro-attitude (the writer’s critical attitude toward the issue under discussion), which is effective in building a convincing argument and creating possible resonation between the writer and the readers. In the meantime, the degree of objectivity of the argument is not threatened due to the reason that only a small amount of authorial and third-party is presented. For another, the expression of affect from the depicted disadvantaged participant can provide readers with a vivid reproduction of the poor encounter, the bad influences of negative behaviors etc., which may create an empathic effect on the readers and again makes the overall argument of the discourse stronger. On the contrary, the representation of the inappropriate affect from the depicted negative participant toward positive issues can establish a sharp comparison of this affect and the macro-attitude, which helps constructing the rationality of the latter. As a corpus-based study of the visual-verbal representation of affect, this paper paves the way for the researchers to further investigate the visual-verbal representation of other kinds of attitude rather than affect as well as the role of visual-verbal interaction in attitude representation. In addition, this paper also indicates that designers and writers of multimodal news commentary should pay special attention to the integrated use of both verbal and visual materials so as to effectively arise critical resonance on the side of readers and makes the discourse as persuasive as possible.
Acknowledgements
This study is supported by the 2018 National Social Science Fund of China (18CYY022), 2017 Shanxi Provincial Educational Reform Project (J2017009), 2016 University Philosophical and Social Science Project of Shanxi Province (2016216), and the National Scientific Project of University Foreign Language Teaching (01050216030019).