A Psychometric Study of Cognitive Self-Regulation: Are Self-Report Questionnaires and Behavioural Tasks Measuring a Similar Construct? ()
1. Introduction
Being successful at work and having fulfilling relationships, or staying in good mental health, all depend on self- regulatory capacities ( Heatherton & Wagner, 2011 ). Self-regulation refers to “the ability to monitor and modulate cognition, emotion and behavior, to accomplish one’s goals and/or to adapt to the cognitive and social demands of specific situations” ( Berger, Kofman, Livneh, & Henik, 2007: p. 257 ). Thus, in addition to the capacity for emotion-regulation, a healthy personality relies on cognition and includes those behaviours that rely on frontal cortex driven executive functions that are subject to voluntary control ( Lewis & Todd, 2007 ). As stated by Hofmann et al. (2012) , “self-regulating organisms have to solve trade-offs between the rigid pursuit of a focal goal and the possibility of being open to alternative courses of action” (p. 177). To solve such trade-offs successfully, goal directedness and two specific cognitive self-regulatory capacities are mandatory: impulse control (inhibition of prepotent impulses) and cognitive flexibility (mental set shifting).The exertion of impulse control enables us to guide voluntary complex actions and overcome habitual responses. In addition, cognitive flexibility enables us to monitor and adjust a performance whenever necessary. Therefore, in addition to the ability to stay focused on what’s currently important, cognitive flexibility implies being alert for environmental changes that create the need to rethink one’s strategy as well as acting upon these changes so that behaviour remains goal directed.
Many different terms that describe cognitive self-regulatory capacities have arisen in the literature, making comparisons of constructs rather difficult. For example self-control ( Baumeister, Vohs, & Tice, 2007 ; Diekhof et al., 2012 ; Muraven & Baumeister, 2000 ; Muraven, Tice, & Baumeister, 1998 ), delay of gratification ( Mischel, Shoda, & Rodriguez, 1989 ), effortful control ( Posner & Rothbart, 2009 ), willpower ( Mischel et al., 2011 ) and ego strength ( Duckworth & Kern, 2011 ) all resemble the construct of impulse control, while cognitive flexibility ( Armbruster, Ueltzhoffer, Basten, & Fiebach, 2012 ; Kehagia, Murray, & Robbins, 2010 ) resembles attention control ( Diehl, Semegon, & Schwarzer, 2006 ) and attentional flexibility ( Cools, Barker, Sahakian, & Robbins, 2001 ). To complicate the matter even further, many different questionnaires and tasks are used to operationalize self-regulatory capacities, yet a recent meta-analysis showed only moderate convergent validity ( Duckworth & Kern, 2011 ). Self-report questionnaires, informant-report questionnaires, observations, behavioural tasks, and functional neuroimaging can all be informative, but beg the question to what extent different operationalizations measure the same aspects of the self-regulation construct, and whether the use of different questionnaires will lead to the same conclusions.
The contributions of the current study to the psychometrics of cognitive self-regulation are three-fold: first, by means of factor analysis, we replicate the factor structure of two self-report questionnaires (Adult Temperament Questionnaire (ATQ) and shortened Temperament and Character Inventory (TCI)) commonly used to investigate impulse control and cognitive flexibility and we additionally show that they load on one “cognitive self-regulation” factor. Second, we analyze the convergent and discriminant validity of these commonly used self-regulation assessment tools. We investigate the extent to which scores of different self-report measures that assess impulse control and cognitive flexibility are correlated with each other and to what extent they diverge from self-report measures that are unrelated or detrimental to cognitive self-regulation. Third, we compare self-report data with performance on two different tasks known to challenge impulse control and cognitive flexibility. By correlating self-report with behaviour, we assess the predictive validity of the selected questionnaires.
2. Self-Report Questionnaires to Asses Cognitive Self-Regulatory Abilities
Self-report questionnaires vary in the extent to which they either assess a particular executive function (like impulse control) or attempt to provide a generalized measure of self-regulation. Therefore, it is not obvious which questionnaires overlap and in which particular areas. In this study we will investigate the convergent validity of three questionnaires that supposedly assess different aspects of cognitive self-regulation in young adults (overview in Table 1). We specifically opted for young adults because their temperamental traits already have become a stable part of their personality or character. The effortful control scale of the ATQ1 quantifiesthe ability to inhibit a dominant response to perform a subdominant response, to detect errors, and to engage in planning. The effortful control scale itself consists of the average score of three subscales. First, the activation control subscale describes the capacity to initiate a behaviour in order to achieve long-term goals. This subscale assesses goal directedness. Examples are “I can keep performing a task even when I would rather not do it” or “I usually finish doing things (paying bills, finishing homework) before they are actually due.” Second, attentional control describes the skills that are needed to stay focused, as well as the ability to shift attention when needed. This subscale thus mostly assesses cognitive flexibility. Examples of items are: “It’s often hard for me to alternate between two different
Table 1. Overview of self-report questionnaires and expected relations to cognitive self-regulatory constructs.
Note: subscales that are investigated in this study are bold printed.
tasks” (reversed) or “When interrupted or distracted, I usually can easily shift my attention back to whatever I was doing before”. Third, inhibitory control describes the ability to suppress behaviour and equals impulse control. Examples are “I can easily resist talking out of turn, even when I’m excited and want to express an idea”, or “When I see an attractive item in a store, it’s usually very hard for me to resist buying it” (reversed).
Second, the shortened version of the TCI ( Duijsens, Spinhoven, Goekoop, Spermon, & Eurelings-Bontekoe, 2000 ) builds on the psychobiological model of temperament and character first described by Cloninger et al. (1993) . The difference between temperamental and character traits is that the former represent traits that are mostly innate and biologically grounded, while the latter involve a maturation process, develop later in life, and can be more influenced by social learning ( Cloninger et al., 1993 ). The shortened TCI consists of seven subscales. Four scales assess temperamental traits (novelty seeking, harm avoidance, reward dependence, persistence) and three scales assess character traits (self-directedness, cooperativeness, and self-transcendence). Persistence (the ability to persevere and be ambitious) and self-directedness (“the ability to control, regulate and adapt behavior to fit the situation in accord with individually chosen goals and values”, Cloninger et al., 1993: p. 979 ) encompass many cognitive self-regulatory items2. Both scales have previously been used in studies on leadership ( O’Connor & Jackson, 2010 ) and were shown to surpass intelligence in predicting academic achievement ( Moreira et al., 2012 ). High persistence has predictive value with respects to overcoming substance abuse ( Sizoo, van den Brink, van Eenige, & van der Gaag, 2009 ), while low self-directedness has been associated with depressed ( de Winter, Wolterbeek, Spinhoven, Zitman, & Goekoop, 2007 ) and anorectic populations ( Vervaet, van Heeringen, & Audenaert, 2004a ). While it is difficult to pin a particular executive function such as “impulse control” or “cognitive flexibility” to subscales of the TCI, these previous results qualify these subscales as a measuring tool to assess capacities that are necessary to achieve long term goals.
Finally, the internal-external locus of control scale (LOC) ( Rotter, 1966 ), which originated in theories on social learning, can also be considered as a generalized measure of cognitive self-regulatory capacities ( Declerck, Boone, & De Brabander, 2006 ; Previc, 2009 ). This personality trait reflects the degree to which people attribute the outcome of events in life to be contingent on their own behaviour. Individuals with an internal LOC feel that they have control over what happens in their lives, (e.g., “coincidence and luck don’t exist in human life”), whereas externals attribute the causes of events to external factors, such as, luck, fate, or powerful others (e.g., “most people don’t realize to what extent their lives are determined by coincidental facts”). The essence of the LOC construct is that people vary in their individual learning experiences, leading to differences in the degree to which people will perceive rewards and punishments as contingent on their own behaviour. As such, the LOC is reminiscent of Cloninger’s more mature character traits that can change during the course of life. Yet there are likely also biological roots that predispose people to either an internal or external LOC. In a theoretical review, Declerck, Boone and De Brabander (2006) argue that the biological roots of LOC that allow people to perceive control are the same ones that substantiate the frontal executive functions necessary for cognitive self-regulation. Their review furthermore indicates that an internal LOC correlates significantly to strategic planning, impulse control, and selective attention, while an external LOC is overrepresented in Attention Deficit Hyperactivity Disorder (ADHD) populations and individuals with brain lesions leading to a reduced executive function capacity. Thus, an internal locus of control can be seen as one of the building blocks of successful cognitive self-regulation. As LOC correlates moderately with a number of other personality measures, the generalized construct of “perceived control” may well comprise a blend of desirable traits that are all related to psychological well-being and successful performance ( Declerck et al., 2006 ). The LOC has been well-researched and shown to have much predictive value in psychological health, goal achievement, and career and academic success, further justifying including the LOC as a measure of cognitive self-regulation ( Declerck et al., 2006; Judge, 2009 ).
In order to assess the discriminant validity of these three questionnaires, the current study includes two other scales that measure traits which should hamper cognitive self-regulation. To do this, we rely on two other subscales of the TCI: harm avoidance and novelty-seeking. Harm avoidance “is viewed as an heritable bias in the inhibition or cessation of behaviors, such as pessimistic worry in anticipation of future problems, passive avoidant behaviors such as fear of uncertainty and shyness of strangers and rapid fatigability” ( Cloninger et al., 1993: p. 977 ). Harm avoidance functions as an antipode of cognitive self-regulation by distracting people from goal directed thought. Novelty seeking is defined as “a heritable bias in the activation or initiation of behaviors such as frequent exploratory activity in response to novelty, impulsive decision making, extravagance in approach to cues of reward, and quick loss of temper and active avoidance of frustration” ( Cloninger, et al., 1993: p. 977 ). Novelty seeking impedes cognitive self-regulation by interfering with sustained attention. As such it is indicative of compulsive buying ( Black, Shaw, McCormick, Bayless, & Allen, 2012 ) and bulimia nervosa ( Vervaet, van Heeringen, & Audenaert, 2004b ). Furthermore, deficits in self-regulatory capacities in children, assessed by the Child Behavior Checklist?Dysregulation Profile, are associated with a temperamental profile characterized by high novelty seeking, high harm avoidance, and low persistence ( Althoff et al., 2012 ).
As control variables, we include scores on intelligence and social desirability in answering. Because matrix analogy tasks have been shown to be reliable estimates for general intellectual abilities ( Zhu, Weiss, Prifitera, & Coalson, 2003 ), we rely on Matrix Reasoning from WAIS-III as an estimate for intelligence. Since self-regulation is a desired capacity and social desirability is a relevant variable in self-report research ( Kline, Sulsky, & Rever- Moriyama, 2000 ), results can be subject to altered self-presentation (faking good). Therefore, we used 14 items that were added to the Rotter LOC scale by Boone and De Brabander (1993) especially designed to measure social desirability. Examples of such social desirable answers are: “one may not brag about his capacities when another does not have them” or “Exercising sports is an interesting way to harden one’s character”. The social desirability score controls for transparency of the questionnaires. If cognitive self-regulation is truly a construct that relies on executive functions independent of intelligence, we should find no correlations between the questionnaires and the intelligence estimate. However, the issue surrounding the role of intelligence in executive functions other than working memory ( Engle, Tuholski, Laughlin, & Conway, 1999 ) is not resolved yet.
3. Behavioural Tasks to Asses Cognitive Self-Regulatory Abilities
An alternative way of measuring cognitive self-regulatory capacities is to look at individual performance on tasks that measure executive functions. The advantage of using behavioural tasks is that they are less prone to self-representation in a socially desirable way. Because we know more about how performance on particular tasks is related to brain functions thanks to lesion studies and neuro imaging techniques, we can also associate particular tasks with distinct executive functions ( Stuss & Levine, 2002 ). Impulse control is typically challenged by means of a delay of gratification task ( Casey et al., 2011 ; Mischel et al., 1989 ) or a time discounting task ( McClure, Laibson, Loewenstein, & Cohen, 2004 ) whereas Wisconsin Card Sorting Tests ( Nagano-Saito et al., 2008 ; Smillie, Cooper, Tharp, & Pelling, 2009 ) or probabilistic reversal learning tasks ( Cools, Clark, Owen, & Robbins, 2002 ; Dodds et al., 2008 ) are often used to test cognitive flexibility. Although these tasks measure moment-to-moment performance and are not designed to assess generalized self-regulatory abilities, at least some self-control tasks have shown to have much predictive value with respect to success in life, even decades later ( Casey et al., 2011 ; Mischel et al., 1989 ). These classical studies show that the ability of preschoolers to resist to eat a marshmallow right away and instead wait for a moment to receive a bigger reward (for example two marshmallows), is predictive for their self-control four decades later.
In the current study, we challenge impulse control with a time discounting task in which high cognitive self- regulation is defined as resisting small, immediate financial rewards in favor of larger, postponed rewards. We assess cognitive flexibility with a probabilistic reversal learning task which tests the combined ability of 1) learning contingencies based on feedback as well as 2) switching behaviour when previously learned contingencies have changed. We define high cognitive self-regulation in this task as a superior ability to correctly adapt to new contingencies. Both tasks (time discounting and probabilistic reversal learning) have previously been used together as indications of cognitive self-regulatory ability by Camchong et al. (2011) who found that performance on time discounting and reversal learning tasks in cocaine dependent patients were associated with abnormalities in functional brain connectivity. Both tasks are shown to be valid and reliable ( Freyer et al., 2009 ; Hurst, Kepley, McCalla, & Livermore, 2011 ; Kirby, 2009 ).
To summarize the intent of the study, we first investigate whether the effortful control subscales (activation-, attentional-, and impulse control) and relevant TCI subscales (persistence and self-directedness) come forward as a single factor indicative of cognitive self-regulation in factor analysis. Second we look at how these two subscales of questionnaires are related to one another and to LOC. To investigate the discriminant validity of these self- report questionnaires, we include harm avoidance and novelty-seeking of the TCI and a proxy for intelligence into the analysis. Finally, to explore the predictive validity of these questionnaires, we correlate the scores to the results of a time discounting and probabilistic reversal learning task that challenge impulse control and cognitive flexibility respectively. Knowing how questionnaire data relate to performance should reveal whether or not these different types of instruments measure the same construct of cognitive self-regulation.
4. Study 1
4.1. Methods
A total of 569 first year economics students enrolled in an introductory psychology course were invited and volunteered to participate in exchange for partial course credit. By means of the electronic learning platform, participants were asked to complete the ATQ ( Rothbart, Ahadi, & Evans, 2000 ), the shortened TCI ( Duijsens et al., 2000 ) and the internal-external LOC scale ( Rotter, 1966 ). The progressive matrices subtest from the Wechsler Adult Intelligence Scale III ( Wechsler, 1997 ) was completed under supervision during class time. 77 of them did not correctly complete the questionnaires and were excluded from the sample. Analysis was conducted on 492 participants of which 42% were females (n = 204). Their ages ranged from 17 to 43 (mean = 19.28, standard deviation = 2.2).
4.2. Measures
The ATQ scale consists of 77 items that have to be rated on a 7-point Likert scale in terms of how applicable the statement is to the self, ranging from 1 “not applicable” to 7 “most applicable”. An extra option “X” is included in case the participant does not find the item applicable to him/her. We focus on the effortful control subscale and averaged the scores on activation control (7 items, Cronbach’s α = .739), attentional control (5 items, Cronbach’s α = .639) and inhibitory control (7 items, Cronbach’s α = .620) for further analysis. A high score indicates successful effortful control. The shortened TCI poses 105 questions that have to be answered with “wrong” or “wright”. All seven scales consist of 15 items. We averaged the scores for persistence (Cronbach’s α = .774), novelty seeking (Cronbach’s α = .728), harm avoidance (Cronbach’s α = .834) and self-directedness (Cronbach’s α = .745) for further analysis. High scores coincide with high persistence, novelty seeking, harm avoidance and self-direc- tedness respectively.The locus of control scale is a 37-item forced choice scale of which 23 items measure LOC (Cronbach’s α = .680) and 14 items test the social desirability of answering. A higher LOC score means a more internal orientation of locus of control. The reliability of these scales has been described in Evans and Rothbart (2007) for ATQ, in Duijsens et al. (2000) for the shortened TCI and in Boone et al. (2002) for LOC.
4.3. Results
Table 2 summarizes the results of three factor analyses. For all factor analyses, we rely on Principal Component Analysis with Varimax rotation. This type of rotation maximizes the dispersion of loadings within factors resulting in a smaller number of variables loading highly onto each factor, making clusters more interpretable ( Field, 2009 ). A first factor analysis validates the effortful control construct. All three subscales of the effortful control scale load onto one single factor explaining 61.55% of the variance in the data. A second analysis on persistence and self- directedness also results into one factor explaining 62.7% of the variance in the data (see Table 2). Combining all five subscales, shows only one “cognitive self-regulation” factor explaining 49.84% of the variance, see Table 2.
Table 3 reports the mean values as well as the Pearson’s correlation coefficients for all variables. All subscales of self-regulation (activation control, attentional control, inhibitory control, self-directedness, persistence) and the factor scores of the three factor analyses correlated positively with each other and with an internal LOC score. They correlated negatively with harm avoidance and novelty seeking. Scores on the progressive matrices (a proxy for intelligence) do not correlate with the three factor scores, corroborating that intelligence and self-regulation are independent of each other. However, while expected, the proxy measure for intelligence correlated positively at p < .05 with internal LOC (Pearson’s r = .094) and negatively with novelty seeking (Pearson’s r = −.114). Social desirability correlated positively with inhibitory control, persistence and self-directedness and consequently, also correlates with the factor scores of each factor analysis. This suggests that people do realize that self-regulation is a desirable trait and that they may manage their answers accordingly.
5. Study 2
The purpose of this behavioural experiment is to test how the self-report questionnaires used in study 1 relate to behavioural measures of cognitive self-regulation. For this, we chose a time discounting task (assessing impulse control) and a probabilistic reversal learning task (assessing cognitive flexibility). In the former task impulse control is needed to resist immediate gratification and opt for a larger but delayed reward. In the latter, cognitive flexibility is required to adjust behavior when it is no longer adaptive. Optimal flexibility is finding the balance between adjusting your behaviour too soon (impulsive reaction) and too late (perseverative reaction).
5.1. Methods
A total of 89 students, of which 57 (64%) females participated. Age ranges from 18 to 24 years (mean = 20.3, standard deviation = 1.88). Students in economics, psychology and architecture were recruited by e-mail and were sent a description of the study and an invitation to participate in exchange for monetary remuneration. The e-mail included a link to a website which provided additional information about the experiment and offered the possi-
Table 2. Result of three factor analyses on self-report questionnaires.
Note: (1) shows the factor scores of a first factor analysis, revealing that all subscales of the effortful control scale load onto one single factor. A second factor analysis (2) shows a similar finding for two subscales of TCI; both load onto one factor. Combining all five subscales into one analysis, shows that effortful control subscales and TCI subscales also load onto one factor (3). ATQ = Adult Temperament Questionnaire, TCI = shortened Temperament Character Inventory. Extraction Method: Principal Component, Varimax Rotation, N = 492.
Table 3. Descriptives and correlation matrix to depict convergent and discriminant validity of self-report measures, social desirability, proxy for intelligence and factor scores from factor analysis.
Note: Factor score (1), (2) and(3) refer to the factor scores from factor analysis containing all three subscales of the effortful control scale (1), persistence and self-directedness from TCI (2) and all five cognitive self-regulatory measures (3). Pearson Correlation coefficients are shown. ATQ = Adult Temperament Questionnaire, TCI = shortened Temperament Character Inventory, N = 492, Significance = * <0.05, ** < 0.01 (two tailed).
bility to register for one of 12experimental sessions. They were requested to complete an online version of the ATQ ( Rothbart et al., 2000 ) to finalize their registration. Upon arrival for the experiment participants signed an informed consent form and filled out the LOC scale ( Rotter, 1966 ) and the TCI ( Duijsens et al., 2000 ). Due to computer failure during the experiment, we lost behavioural data of 2 participants. ATQ data was incomplete for another 9 participants. Analysis was performed on 78 participants. The experimental tasks (explained below) were programmed in Visual Basic and presented on a 17" desktop screen. Left mouse clicking was the only way to respond during the tasks. The order of presentation of the tasks was fixed. Time discounting preceded probabilistic reversal learning because impulse control is prone to the depletory effect of a previously demanding task ( Joireman, Balliet, Sprott, Spangenberg, & Schultz, 2008 ). A pilot study (n = 8) showed that the probabilistic reversal learning task was perceived as more difficult and more attentionally demanding than the time discounting task. At the end of the experiment, participants were paid in truth the money earned during the time discounting task (see below), plus a ? show-up fee. The entire session lasted 2 hours.
5.2. Measures
5.2.1. Time Discounting Task
The time discounting task follows the procedures used by McClure et al. (2004) . Participants chose between an amount of money to be given now or in the near future (today, or within 2 or 4 weeks) or a larger amount which was always offered at a later time (minimal 2 weeks after the smaller amount with a maximum of 6 weeks from today). The larger amount (max ?0) was computed by raising the small amount (min ? by) a percentage (1%, 3%, 5%, 10%, 15%, 25%, 35% or 50%). The entire task was subdivided into 3 runs of 7 minutes to allow a short resting period in between runs. The length of the break was determined by the participant himself, but never lasted longer than 3 minutes. The participants were instructed that they were going to be confronted with a series of choices, and that for each choice they would have to indicate their preference. For each trial, the two options were presented on either side of the screen, with the smaller, earlier reward always presented on the left. The percentage of delayed choices over the total experiment is the dependent variable in the study. The higher this percentage, the more control the participant is believed to have developed to overrule impulsive desires. Participants were allowed as much time as needed to make their selection. Given that the total experimental time was fixed, this meant that slower participants made fewer choices. The number of decisions made by the participants varied between 56 and 81 (mean 73) decisions. In order to remunerate according to actual choice, each choice of a participant was recorded and total amounts were divided by a factor of 200. The total amount earned varied from ?0 to ?0 and was deposited in their bank account in instalments on the actual day that was indicated during the experiment.
5.2.2. Probabilistic Reversal Learning Task
In this task participants have to continuously learn and unlearn the contingencies between an action and an outcome ( Cools et al., 2002 ; Cools, Lewis, Clark, Barker, & Robbins, 2007 ). A choice between 2 options, A and B, is presented with nofurther information. Feedback (correct or incorrect) followed the choice. One option, A, was correct in 80% of the cases. Without informing the participant, the feedback could suddenly reverse after 10 to 15 correct answers into anew series of trials in which B became the stimulus with 80% correct feedback. The participant had to monitor this switch and adapt his/her choice to the new contingency. Every instance that the options A and B switched in reward contingencies denotes a reversal stage. The sooner an individual responds correctly to the contingency changes, the more reversal stages can be achieved during the course of the experiment. The dependent variable is the total number of reversal stages an individual achieves. We use this variable as a proxy for successful cognitive flexibility.
5.3. Results
Factor analysis confirms the factor structure found in study 1. All subscales of effortful control, persistence and self-directedness load onto one factor explaining 51.47% of the variance in the data, see Table 4.
To explore the predictive value of self-report measures towards behavioural performance we correlated the obtained factor score with the scores on the behavioural tasks and find that neither the percentage of delayed choices in the time discounting task (impulse control), nor the number of successful reversals in the probabilistic reversal task (cognitive flexibility) correlates with the factor score. Correlations with separate self-report measures
Table 4. Factor analysis of self-report measures in the behavioural study.
Note: (1) shows factor loadings of self-report measures loading onto one factor. ATQ = Adult Temperament Questionnaire, TCI = shortened Temperament Character Inventory, Extraction Method: Principal Component, Varimax Rotation, N = 89.
show that only internal LOC is predictive for the total percentage of delayed choices in the time discounting task (see Table 5).
6. General Discussion
Three major conclusions can be drawn from these data. First, results of both studies corroborate the construct validity of the self-regulatory subscales of two commonly used self-report questionnaires. Using independent data sets, activation control, attentional control, inhibitory control, persistence and self-directedness appear to measure the same cognitive self-regulatory construct. As evidence for its convergent validity we find that, in both studies, the factor score(s) from factor analysis correlate significantly with LOC, a personality trait which heavily relies on strong cognitive self-regulatory capabilities (see Declerck et al., 2006 , for review). With respect to discriminant validity, we found that the factor scores in our first study all correlate negatively with harm avoidance and novelty seeking. Finally, our data also suggest that cognitive self-regulatory capacities are independent of intelligence but that they are prone to a self-representation bias as people tend to answer in a social desirable way (which is impossible to do with matrix reasoning).
A second conclusion is that the predictive validity of cognitive self-regulation, assessed by relating self-report data to performance on behavioural tasks, is rather poor. We chose two tasks, time discounting and probabilistic reversal learning, to represent two important subsets of cognitive self-regulation, namely impulse control and cognitive flexibility, but only the former showed a significant correlation with internal locus of control.
Cognitive flexibility, captured with a probabilistic reversal learning task, correlated neither with the scores of any of the cognitive self-regulation scales, nor with the factor scores. This poses the question whether or not the “switching” capacity of the task and the “flexibility” assessed by questionnaires are in fact measuring the same thing. Possibly, they are assessing switching in a different time frame: whereas the probabilistic reversal learning task requires a short-term switching capacity (30 behavioural switches in half an hour), the cognitive flexibility assessed by self-report may be referring to behavioural switches that take place over a much longer term.
As a third conclusion, we note that the correlation between LOC and impulse control suggests that the ability to exert control to overcome impulsive choices in a time discounting task is better captured with questionnaires that assess people’s character and mature cognitive self-regulatory processes rather than with temperamental traits indicative of cognitive self-regulation. Indeed, performance on a time discounting task is known to be age-de- pendent ( Mischel et al., 1989 ) since it relies on frontal brain structures that still mature into young adulthood ( Christakou, Brammer, & Rubia, 2011 ). This suggests that LOC?of all analyzed self-report measures?best predicts behaviour due to its sensitivity to maturational growth of an individual; something that cannot be captured by other self-report questionnaires quantifying temperamental building blocks of one’s personality.
With only the LOC score correlating with a behavioural measure of impulse control, and none of the other variables correlating with task performance in cognitive flexibility, these data put the predictive validity of the ATQ
Table 5. Descriptives and correlation matrix of self-report measures, behavioural data and the factor score from factor analysis.
Note: Factor score (1) refers to the factor score from factor analysis containing all five cognitive self-regulatory measures loading onto one factor. Pearson Correlation coefficients are shown, ATQ = Adult Temperament Questionnaire, TCI = shortened Temperament Character Inventory, PRL = probabilistic reversal learning, TD = time discounting, N = 78, Significance = * < .05 , ** < .01 (two tailed).
and TCI into question, at least when it comes to differentiating individual differences in a normal, non-psychiatric population. This conclusion of the study is not peculiar to the current data but in fact echoes the results of other studies that have compared questionnaire data with tasks performance in the domain of self-control. In a recent review, Dalley and Roiser (2012) note that correlation coefficients between behaviour and self-report rarely exceed .4. Consistently, the most significant correlation in this study (between LOC and delay of gratification in a time discounting task) is below r = .3. Reynolds et al. (2006) report in their analysis on impulsivity measures that “self-report measures were generally unrelated to the task measures” (p. 312), leaving the possibility that self- report measures are not sufficiently robust to assess differences in normal populations (but see Sripada, Gonzalez, Phan and Liberzon (2011) for an exception). Most studies that find reliable individual differences in self-reported self-regulation investigate psychiatric out patient populations ( Crean, de Wit, & Richards, 2000 ) or substance dependent individuals ( Bickel, Jarmolowicz, Mueller, Gatchalian, & McClure, 2012 ; Ersche et al., 2011 ; Moeller et al., 2002 ). But even in the latter group, absence of significant correlations between alcohol addiction and self- report has been reported ( Dom, De Wilde, Hulstijn, & Sabbe, 2007 ).
In the literature on self-regulation several methodological issues have been raised which may explain why it is so difficult to find reliable construct validity of self-report measures of self-regulation. Dalley and Roiser (2012) note that correlations between behaviour and questionnaire data in healthy populations are stronger when sample sizes are very large and/or when a median split of the data is used to divide their participants into a high self- regulation and a low self-regulation group. Duckworth and Kern (2011) referring to recent work by Rothbart and colleagues, recommend combining several behavioural measures. They report “that average performance across three vs. six executive function tasks correlated r = .22 and r = .30, respectively, with informant-report questionnaire measures of self-control” (p. 266). A post hoc correlation analysis of the data in the current study indeed showed slightly higher correlation coefficients (but still not significant) when correlating the self-report subscales with the average of the standardized scores of the two behavioural tasks (time discounting and probabilistic reversal learning).
Next, in addition to the population characteristics (normal versus psychiatric) and the methodological issues that affect the relation between self-report and behaviour in the domain of self-regulation, there may be other considerations. Self-report data may be tainted with self-representation. Because self-regulation is a desirable trait, people may be tempted to overstate this skill. This would be particularly true among college students who? perhaps more than other segments of the population-depend on self-regulation to succeed. Results indeed indicate that social desirability correlates positively with all three cognitive self-regulatory factor scores in the first study, suggesting that it is an important factor in accounting for the lack of consistency between self-report and actual behaviour.
Furthermore, task performance may not always represent a temporally stable trait and may be subject to state influences and fluctuations in preferences. How motivated a participant is to perform optimally in an executive function task may not simply be a function of trait cognitive self-regulation. State influences such as boredom or feeling hungry can also affect task performance.
7. Conclusion
To summarize, we note that trait cognitive self-regulation is a complex construct which would benefit from multiple assessment methods. We have shown that construct validity of self-report cognitive self-regulation questionnaires is robust but the present data could not attest to its predictive validity. We suggest that social desirability answering patterns in particular can taint self-report scores.
In order to reliably assess stable individual differences in cognitive self-regulation, future research might benefit from including neuroimaging techniques. Investigating how behaviour on executive function tasks is on the one hand related to self-report data, and on the other hand to neuroimaging results, may reveal if individual differences in brain organization associated with self-report data are stable enough to be generalized to trait-differ- ences in executive functioning. Furthermore, by better understanding how the brain is wired to self-regulate, state and trait influences on behaviour might be disentangled.
NOTES
1In addition to effortful control, the ATQ comprises scales to measure negative affectivity, orienting sensitivity, and extraversion.
2Due to contractual restraints, no examples of questionnaire items can be described.