_{1}

^{*}

The application of CFA to Multitrait-Multimethod Matrices (MTMM) is an elaborated method for the evaluation of construct validity in terms of the discriminant and convergent validity as well as method effects. It is implemented to evaluate the acceptability of a questionnaire as a measure of constructs or latent variables, during the structural and external stages of an instrument validation process. CFA can be carried out in MTMM matrices, like in any other covariance matrix, to study the latent variables of traits and methods factors. CFA-MTMM models can distinguish systematic trait or method effects from unsystematic measurement error variance, thus offering the possibility of hypotheses testing on the measurement model, while controlling possible effects. A plethora of CFA-MTMM approaches have been proposed, but the Correlated Methods Models and the Correlated Uniqueness Models are two CFA-MTMM methods most widely used. This work discusses how these two approaches can be parametrized, and how inferences about construct validity and method effects can be drawn on the matrix level and on parameters level. Then, the two methods are briefly compared. Additional CFA MTMM parameterizations are also discussed and their advantages and disadvantages are summarized.

Test developers ought to publish only measurement instruments meeting the guidelines specified by the APA standards (Franzen, 2002) . Inaccurate measurements may lead to incorrect diagnoses, suboptimal treatment decisions or biased treatment effects estimates (Koch, Eid, & Lochner, 2018; Courvoisier, Nussbeck, Eid, Geiser, & Cole, 2008) . Additionally, the adequacy of inferences made based on test scores is a central issue in social and behavioral sciences and it is related to validity (Messick, 1980, 1989, 1995; APA/AERA & NCME, 1999, 2014) . Construct validity is the extent to which a measuring tool “reacts” the way the construct purports to react when compared with other, well-known measures of different constructs (DeVellis, 2017) and it consists the central focus of each measurement process (Kline, 2009) and an all-embracing principle of validity (Messick, 1995; Brown, 2015) . Construct validity (Cronbach & Meehl, 1955) examines the theoretical relationship of a variable (like the scale score) to other variables (Kyriazos, 2018) and it incorporates the internal scale structure (Zinbarg, Yovel, Revelle, & McDonald, 2006; Revelle, 2018) or the correct measurement of variables intended to be examined (Kline, 2009) . Construct validation is an ongoing process (Franzen, 2002) .

To this end, Campbell and Fiske (1959) introduced the Multitrait-Multimethod (MTMM) Matrix, an evaluation method of the acceptability of questionnaires as measures of constructs or latent variables (Price, 2017) . This method involves a correlation matrix customized to enhance the evaluation of construct validity^{1} in terms of the discriminant and convergent validity (Brown, 2015) . The method that Campbell and Fiske (1959) operationalize is based on the idea that theoretically related scores of the same or similar traits should also show high positive correlations with each other whereas theoretically unrelated scores of different constructs should also show low or insignificant correlations. The former refers to convergent validity and the latter refers to discriminant validity (Koch, Eid, & Lochner, 2018; Franzen, 2002) .

The purpose of this study is to describe the application of Factor Analysis―EFA and mainly CFA―into the Multitrait-Multimethod Matrices (MTMM), as an elaborated approach of studying convergent validity, discriminant validity, and method effects during the estimation of construct validity.

^{1}Construct Validity can be assessed by other methods along with MTMM. See Kyriazos et al. (2018a,2018b,2018c,2018d,2018e) for applied examples of construct validation studies.

The Multitrait-multimethod (Campbell &Fiske,1959) is “an analytic method that includes evaluation of construct validity relative to multiple examinee traits in relation to multiple (different) methods for measuring such traits” (Price,2017: p.140) .MTMM arranges the analysis of validity evidence in a manner that transcends the common “set of associations” approach, permitting the examination of associations among several psychological variables, each being measured through several methods (Furr,2011) . More specifically, two or more traits are measured with two or more methods. Traits could be hypothetical constructs about stable characteristics, like cognitive abilities. On the other hand, methods can encompass different occasions, or means of data collection (e.g., self-report, structured interview and observation), or multiple-test forms, e.g. for different informants (teachers,parents and peers). Note that, trait interpretation suggests that a latent attribute or construct explains the consist patterns observed in the scores (Price,2017; Kline,2016) .The MTMM method is carried out during the structural and external stages of the validation process (Price,2017) , see

For example, during the development of a new self-report scale on social anxiety, the researcher could also measure three constructs (e.g., affect, self-esteem and depression) by three methods for each construct (i.e. self-report, report by partner, and report by peer). Correlations among all measures for all domains are calculated, and they are arranged in a “multitrait-multimethod” correlation matrix (Furr, 2011) , like the one presented in

For the simplest case of MTMM, at least two methods and two constructs are required (2 methods × 2 traits) to generate a multitrait-multimethod design. Each score is a trait-method unit or TMU (Campbell & Fiske, 1959: p. 81; Koch et al., 2018) . The MTMM correlation matrix (see

The monomethod blocks emerge from correlations of measurements by the same method. Their number equals the methods used. In order to account for measurement error (Koch et al., 2018) , the reliability coefficients can be inserted

Stage MTMM is used | External stage/component |
---|---|

Purpose of MTMM | Study the external relationships of the target construct with other constructs and/or traits of the respondents |

Question Asked | Do the target construct as measured by observed variables fit within the nomological network (Cronbach & Meehl, 1955) of expected construct relations? |

Other Methods Used in Tandem | Group measurement invariance, Structural equation modeling, Correlation of construct with other constructs |

Benson (1988) adapted by Price (2017) page 129.

METHOD 1 | METHOD 2 | METHOD 3 | ||||||||
---|---|---|---|---|---|---|---|---|---|---|

Traits | X1-1 | X2-1 | X3-1 | X1-2 | X2-2 | X3-2 | X1-3 | X2-3 | X3-3 | |

METHOD 1 | X1-1 | (0.91) | ||||||||

X2-1 | 0.53 | (0.91) | ||||||||

X3-1 | 0.4 | 0.39 | (0.78) | |||||||

METHOD 2 | X1-2 | 0.59 | 0.24 | 0.11 | (0.95) | |||||

X2-2 | 0.24 | 0.59 | 0.12 | 0.7 | (0.96) | |||||

X3-2 | 0.13 | 0.13 | 0.48 | 0.61 | 0.6 | (0.86) | ||||

METHOD 3 | X1-3 | 0.58 | 0.24 | 0.13 | 0.69 | 0.44 | 0.35 | (0.96) | ||

X2-3 | 0.25 | 0.6 | 0.14 | 0.45 | 0.68 | 0.36 | 0.69 | (0.94) | ||

X3-3 | 0.13 | 0.13 | 0.47 | 0.36 | 0.34 | 0.6 | 0.6 | 0.62 | (0.87) |

Notes: Numbers in body of table are correlation coefficients except those in parentheses. Data is artificial. Cells in light Gray indicate Discriminant validity (different traits measured by same methods―should be lowest of all). Cells in bold typeface indicate Convergent validity (same trait measured by different methods―should be strong and positive; also termed Validity Diagonal). Cells in dark gray indicate Discriminant validity (different traits measured by different methods―should be lowest of all). Reliability coefficients for each test are in parentheses (Reliability Diagonal). Monomethod blocks are marked with a dotted line. Heteromethod blocks are marked by a thick line. The

in the diagonal of these blocks called Reliability Diagonal (Campbell & Fiske, 1959: p. 81) . The reliability diagonal should preferably comprise the largest coefficients in the matrix (Campbell & Fiske, 1959; Price, 2017) , i.e. the measure should show the strongest correlations with itself than with any other construct in the matrix (Trochim, 2002; Brown, 2015) . For example, in

The heteromethod blocks emerge from correlations of measurements by different methods. There number equals to k (k − 1)/2, where k is the number of methods used (Trochim, 2002) . The correlations in their diagonals represent the convergent validity of the same trait measured by different methods. Thus, they should be strong and positive (Price, 2017; Brown, 2015) . For example, in

Campbell and Fiske (1959: pp. 82-83) proposed four rules for the evaluation of MTMM matrices. The correlations in the monotrait-heteromethod block have to be statistically significant and strong. The correlations in the heterotrait-heteromethod block should be weak or should equal the correlations in the heterotrait-monomethod block. The correlations in the heterotrait-monomethod block should be smaller than the correlations in the monotrait-heteromethod block. The pattern of the trait intercorrelations in all heterotrait triangles should be similar for the mono- as well as the heteromethod blocks. This property also refers to the discriminant validity of the measures (summarized by Koch et al., 2018 ). For details refer also to Eid (2010) . Finally, Trochim (2002) in line with Campbell and Fiske (1959) also suggests the following guidelines for the interpretation of an MTMM matrix: 1) Reliability coefficients must be the highest values in the matrix because the correlations between a trait and itself are expected to be the strongest than with any other construct (also Brown, 2015 ). 2) The coefficients presented in the validity diagonal must be statistically different from zero and preferably moderately strong. 3) Every coefficient in the validity diagonal must be the greatest column and row value of the method block it belongs. Recall that all off-diagonal elements must be the smallest to evidence discriminant validity. 4) Each validity coefficient must be greater than all the off-diagonal elements in the heterotrait monomethod triangles. If this is not true, then a method effect may be present. 5) The relationships of the different traits must be coherent in all triangles, i.e. relationships between every trait dyad must be cohesive, without areas of dramatic changes (Trochim, 2002) . In spite of the ease of application of these rules, in practice, the application of MTMM method was criticized as having several limitations (Marsh, 1988, 1989; Schmitt & Stults, 1986; Eid, 2010) .

Despite the fact that the Campbell and Fiske (1959) technique was a significant progress for examining construct validity, the MTMM approach was not widely accepted the years immediately following its inception (Brown, 2015) . Additionally, several limitations were noted first by Jackson (1969) . Jackson’s criticisms as summarized by Franzen (2002) , mainly focused on the fact that the MTMM method juxtaposes distinct patterns without taking into account the overall structure of the relationships (i.e. between-network; Cronbach & Meehl, 1955 ; Martin & Marsh, 2006 ). This argument is based on the idea that the correlation pattern may be affected by the way variance is distributed when measuring the target traits. To alleviate the MTMM disadvantages, Jackson (1969) suggested to carry out an EFA (PCA actually) in the monomethod matrix. Specifically, the correlation matrix is orthogonalized and then submitted to Principal Components Analysis, with a varimax rotation. The extracted factors equal the number of matrix traits. However, Jackson’s suggestion while offering some advantages, it assumes no relationships exist among the traits and this is rarely the case in psychology. Additionally, the influence of different measurement methods cannot be taken into account when the monomethod matrix is used (Franzen, 2002) . Finally, EFA failed to obtain meaningful solutions of MTMM data due to the inherent restrictions of EFA to specify correlated errors (Brown, 2015) .

Later, Franzen (2002) continues Cole, Howard, and Maxwell (1981) compared the MTMM results with constructs mono-operationalization suggesting that as a rule, the validity coefficients were misleadingly deflated. Therefore, the MTMM method and factor analysis may be used in tandem (Franzen, 2002) . Finally, Cole (1987) repeated to a certain extent Jackson’s criticisms, i.e. absence of concrete guidelines regarding the size of the zero-order correlation coefficients. Additionally, Cole argued that the cost of the MTMM application in clinical settings deems the method inapplicable (Franzen, 2002) . Crucially, Cole noted that the MTMM is prone to correlated errors due to factors like the time of day, or the instrument administrator. Cole suggested the use of CFA instead to examine discriminant and convergent validity (Franzen, 2002) . For a detailed description of MTMM limitations see also Marsh (1988, 1989) and Schmitt & Stults (1986) .

At the same time, non-factorial oriented attempts to find an appropriate method of statistical analysis for the data in the matrix were numerous (Eid et al., 2008; Eid, 2010) . These methods were mainly based on analysis of variance models or nonparametric alternatives and the reason for the lack of success was attributed to conflicting data in the diagonals in relation to the data in the triangles (Sawilowsky, 2007) . For example, Sawilowsky (2002, 2007) proposed a statistical test for analysis MTMM data where the null hypothesis is that the coefficients in the matrix are unordered from high to low. This is tested against the alternative hypothesis of an increasing trend from the lowest level (heterotrait heteromethod) to the highest level i.e. reliability coefficients (Sawilowsky, 2007: p. 180) . See Sawilowsky (2002, 2007) for more details and an applied example.

Despite the several alternatives proposed―like Sawilowsky’s (2002, 2007) ―the analysis of MTMM data within the CFA framework (a.k.a. CFA-MTMM models) has been a widely accepted methodological strategy (Eid et al., 2008; Byrne, 2012; Koch et al., 2018) .

Actually, MTMM matrix regained interest upon the realization that EFA procedures (Jackson, 1969) or CFA (Cole et al., 1981; Cole, 1987; Flamer, 1983; Marsh & Hocevar, 1983; Widaman, 1985) are equally applicable for MTMM data analysis. Specifically, CFA can be carried out in MTMM matrices, like in any other covariance matrix, to study the latent dimensions of traits and methods factors (Brown, 2015) .

More specifically, CFA-MTMM models have many advantages as Koch et al., (2018) comment, because they offer a plethora of possibilities like: 1) distinguishing systematic trait or method effects from unsystematic measurement error variance (see Brown, 2015 ; Brown & Moore, 2012 ); 2) associating latent trait and/or method variables to other variables in order to study trait or method effects; and 3) testing specific hypotheses on the measurement model (Dumenci, 2000; Eid et al., 2006) , while controlling possible effects (Franzen, 2002) . Besides that, many different MTMM approaches have been suggested, such as variance component models or multiplicative correlation models ( Koch et al., 2018 also quoting Browne, 1984 ; Dudgeon, 1994 ; Millsap, 1995a, 1995b ; Wothke, 1995 ; Wothke & Browne, 1990 ).

Among them (see Marsh & Grayson, 1995 ; Widaman, 1985 ; Koch et al., 2018 ) two have been most widely used (Brown, 2015; Byrne, 2012) : 1) the Correlated Methods Models; and 2) the Correlated Uniqueness Models.

This method was among the first significant developments in the field of CFA-MTMM models (Koch et al., 2018) and was developed by Keith Widaman (1985) proposing a nested model comparison.

The model comparison presented by Widaman (1985) had (t) traits and (m) methods where (t) is the number of trait factors and (m) is the number of method factors, abiding by five key specifications rules (Brown, 2015) : 1) for model identification, at least three traits (t) and three methods (m) are required; 2) (t) × (m) indicators are specified to define (t) + (m) factors; 3) each indicator loads on two factors, i.e. a trait factor and a method factor and all other cross-loadings are set to zero; 4) trait factors inter-correlations and method factors inter-correlations are freely estimated; 5) trait-method factors correlations are typically constrained to zero; and 6) indicator uniquenesses are estimated freely but cannot be specified to correlate with uniquenesses of other indicators. Each indicator in this specification is a function of a trait, method, and unique factors, Brown (2015) explains.

The specified MTMM model becomes a baseline model, compared against a set of nested models in which parameters are either removed or added. Model comparison criteria used are the χ^{2} (Δχ^{2}) and other fit indicators (see Byrne, 2012 ), in a procedure similar to the multi-group CFA. Based on this comparison, evidence of convergent and discriminant validity emerges, both at the matrix level and at the individual parameter level (Byrne, 2012) .

Byrne (2012) describes the steps for comparing the nested models as follows.

Step 1―The CTCM Model: Test the first model (see path diagram in

solutions are provided by Marsh et al. (1992) and Byrne (2012) . Hopefully, upon model re-specification, the Correlated traits/Correlated Methods model (CTCM) results in a solution and the fit of this CTCM model must be considered (Byrne, 2012) .

Step 2―The NTCM Model: The next hypothesized model of correlated methods specified is a no traits/correlated methods (NTCM) model (see path diagram in

2012) . Because these two models are nested, a comparison of this model to the CTCM provides a statistical evaluation of whether effects associated with the different traits emerge (Brown, 2015) . First, the goodness-of-fit for this NTCM model is examined in relation to the CTCM model.

Step 3―The PCTCM Model: The specification of a hypothesized Perfectly Correlated Traits/Freely Correlated Methods (PCTCM) MTMM model follows. The path diagram of this model is presented in

Step 4―The CTUM Model: The last correlated methods MTMM model is specified which is a Freely Correlated Traits/Uncorrelated Methods (CTUM) see

Step 5―Evaluation of Construct Validity at the Matrix Level: After the evaluation of goodness-of-fit of the four MTMM models and with the assumption that the fit is acceptable (within the acceptability criteria used for typical CFA models, e.g. Hu & Bentler, 1999 ; Brown, 2015 ), evidence of construct and discriminant validity is examined. This information at the matrix level derives through the comparison of particular pairs of models. Note that typically, a summary of goodness-of-fit statistics of all four MTMM models is presented in a table, and a summary of model comparisons in a second table, see an example of both in

Step 5a―Convergent Validity at the Matrix Level: Generally, the principles applied when testing for invariance are also applied when comparing models in this step (see ^{2} (Δχ^{2}) between these models is highly significant and the fit difference (ΔCFI) is also large evidence of convergent validity is suggested (Byrne, 2012) .

Step 5b―Discriminant Validity at the Matrix Level: Discriminant validity is usually evaluated considering both traits and methods. Evidence of discriminant validity among traits emerges from the comparison of the CTCM Model in which traits are freely correlated (see Model in ^{2} and CFI, the stronger the evidence of discriminant validity. The procedure is repeated to show evidence of discriminant validity regarding method effects. Specifically, the CTCM Model in which methods are freely correlated―specified in step 1 and presented as path diagram in ^{2} (and/or considerable ΔCFI) suggests common method bias, therefore the lack of discriminant validity across measurement methods (Byrne, 2012) . For an applied example see Byrne (2012) . See Goodness of fit criteria used in

Step 6―Evaluation of Construct Validity at the Parameter Level: Trait- and method-related variance can be examined more thoroughly by examining individual parameter estimates, i.e. the factor loadings and factor correlations of

Goodness of fit Index for CFA MTMM Models 1 - 4 (Correlated Methods) | Models tested in Correlated Methods |
---|---|

Chi-square (χ^{2}) Degrees of Freedom Comparative Fit Index (CFI) Root Mean Square Error of Approximation (RMSEA) RMSEA 90% Confidence Interval (CI) Standardized Root Mean Square Residual (SRMR) | Model 1: CTCM Model (see |

Notes: Based on a table used by Byrne (2012: p. 299) ; Goodness of fit criteria are the same as standard CFA.

CFA MTMM Models 1 - 4 (Correlated Methods) | Differential Goodness-of-Fit |
---|---|

Model Comparisons | Difference in |

Test of Convergent Validity | Chi-square (χ^{2}) |

Model 1 (CTCM)versus Model 2 (NTCM) | Degrees of Freedom |

Test of Discriminant Validity | Comparative Fit Index (CFI) |

Model 1 (CTCM)versus Model 3 (PCTCM) | Significance level (p value) |

Model 1 (CTCM)versus Model 4 (CTUM) |

Notes: Based on a table used by Byrne (2012: p. 300) ; Byrne suggests difference criteria adopted to be identical to measurement Invariance Criteria (see Cheung and Rensvold, 2002 ; Chen, 2007 for cutoff values).

the CTCM model (the Model of step 1, see path diagram in

Step 6a―Convergent Validity at the Parameter Level: The size of the trait loadings indicates convergent validity. First, all loadings are examined to see if they are statistically significant. Then all factor loadings across traits and factor loadings across methods are compared, to see whether method variance or trait variance is higher over all the available ratings (Byrne, 2012) .

Step 6b―Discriminant Validity at the Parameter Level: Factor correlation matrices contain the evidence of discriminant validity regarding particular traits and methods. Generally, correlations among traits should theoretically be small to show evidence of discriminant validity, but such findings are rare in psychology. Regarding the correlations of the method factors, discriminability of these parameters values suggests the extent to which the methods are dissimilar, testing a major assumption of the MTMM strategy (Campbell & Fiske, 1959; Byrne, 2012) . For an applied example of the correlated method refer to Byrne (2012) and Brown (2015) .

Some simpler alternatives to CTCM models have been proposed, and the correlated-uniqueness (CU) model described next is one of them (Kline, 2016) .

The CU models signify a special case of the general CFA model based on the work of Kenny (1976, 1979) and Marsh (1988, 1989) who proposed this alternative MTMM model parametrization to avoid the common estimation and convergence problems arising when analyzing CTCM models, like the one in

The CU model identification must abide by the following rules as reported by Brown (2015) . There must be at least two traits (t) and three methods (m). However, a 2 t × 2 m model can be also specified in the factor loadings of indicators that belong to the same trait factor are fixed to be equal. Rules regarding the CU trait model are the same as in the correlated methods approach. Specifically: 1) Each indicator must load on one trait factor with all other cross-loadings fixed to zero; and 2) The trait factor correlations are freely estimated. Consequently, the major difference between the CU method in comparison to the correlated methods is the estimation of method effects. In the CU model, method effects are assessed by adding correlated uniquenesses (errors) among the indicators of the same assessment method instead of specifying method factors (Brown, 2015) .

After the model specification, the model is tested and the fit of the solution to the data is evaluated. Except fit the inspection of standardized residuals, the modification indices must be also evaluated (as proposed by Brown, 2015 ). The parameter estimates Brown (2015) continues are then examined. That is, when loadings of traits factors are substantial and statistically significant, this is an evidence supporting convergent validity. Conversely, when correlations among the trait factors are substantial, this is indicative of low discriminant validity. Method effects are assessed when correlated uniquenesses among indicators of the same method are moderate or large in magnitude (Brown, 2015) .

For example, Brown (2015) explains, when trait factor loadings are large this suggests that the relation of the indicators to their purported latent constructs is strong thus, convergent validity can be supported while adjusting for method effects. When modest correlations emerge among trait factors this is indicative of adequate discriminant validity. The significance of method effects is evaluated by the size of the correlated uniquenesses among indicators. As in the classic MTMM matrix, the method effects in the CFA model must be smaller than those of the indicators. For an applied example of the CU, method sees Byrne (2012)

and Brown (2015) . Saris and Aalberts (2003) also examined different interpretations of CU models.

Some recommend the correlated uniquenesses (CU) method (Kenny, 1976, 1979; Kenny & Kashy, 1992; Marsh, 1989) as superior, whereas others support the general CFA method (Conway, Scullen, Lievens, & Lance, 2004; Lance, Noble, & Scullen, 2002) . Nonetheless, Byrne (2012) comments that the general CFA model is as a rule preferable.

Specifically, an advantage of the CT-CM model is that it permits fragmenting observed variance into a trait, method, and error variance in comparison to the CU model that is more restrictive (Eid et al., 2006; Geiser et al., 2008; Koch et al., 2018) . Kenny and Kashy (1992) argue that this fragmentation of the correlated methods model is the reason why this method corresponds directly to Campbell and Fiske’s (1959) original conceptualization of the MTMM matrix. Moreover, the parameter estimates of the correlated methods models offer direct evidence of construct validity. For instance, substantial trait factor loadings show convergent validity; small and/or non-significant method factor loadings suggest that method effects are absent and modest trait factor correlations suggest good discriminant validity. The addition of method factors permits interpretation of method effects (Brown, 2015) .

Nevertheless, a drawback of the correlated methods model is that it is prone to underidentification errors, or Heywood cases and large standard errors. Because of these problems with the correlated methods model, methodologists suggested (Kenny & Kashy, 1992; Marsh & Grayson, 1995) using the correlated uniqueness model for the analysis of MTMM data. On the other hand, others (Lance, Noble, & Scullen, 2002) argue that given the substantive strengths of the correlated methods model, the CU model should be preferred only as last resort, i.e. when the correlated methods model fails to be identified (Brown, 2015) .

For a summary of CFA-MTMM models see Dumenci (2000) , Eid, Lischetzke, and Nussbeck (2006) , as well as Shrout and Fiske (1995) and Koch et al. (2018) . For applied examples of the above-mentioned procedure both for the Correlated methods and the Correlated Uniquenesses you can refer to Byrne and Goffin (1993) , Byrne and Bazana (1996) as well as to Meyer, Frost, Brown, Steketee, and Tolin (2013) or Koch et al. (2018) . Byrne (2012) refers readers who wish to read more on comparisons of the correlated uniquenesses, composite direct product, and general CFA models to other sources (Bagozzi, 1993; Bagozzi & Yi, 1990; Coenders & Saris, 2000; Hernández & González-Romá, 2002; Marsh & Bailey, 1991; Marsh, Byrne, & Craven, 1992) .

More recent, Koch et al. (2018) comment, MTMM research has gained insight on many aspects of the CTCM minus one (CT-C[M-1]) model coined by Eid, 2000 ; Eid, Lischetzke, Nussbeck, & Trierweiler, 2003 ; see also Geiser, Eid, & Nussbeck, 2008 ; Maydeu-Olivares & Coffman, 2006 ; Pohl & Steyer, 2010 . These comprise the inspection of specifically correlated residuals (Cole, Ciesla, & Steiger, 2007; Saris & Aalberts, 2003) , and extension of the method to longitudinal data (Courvoisier et al. 2008; Grimm, Pianta, & Konold, 2009; LaGrange & Cole, 2008) as well as to multilevel data (Hox & Kleiboer, 2007) .

More specifically, the CT-C (M-1) model is analogous to the correlated methods model (presented in

Latent difference models, Koch et al. (2018) argue, offer a possibility to contrast methods. Initially, they were developed for longitudinal research (Steyer, Partchev, & Shanahan, 2000) but they can successfully use in multimethod research (see Pohl, Steyer & Kraus, 2008 ). This technique also involves a reference method. Nevertheless, the latent method variables are not specified as residuals regarding the true-score variables of the reference method. It is hypothesized that each indicator taps a trait variable, but method variables are unidimensional. Therefore, a common method factor is specified for all indicators of the same method and the same trait (Koch et al., 2018) .

One more alternative approach is the direct product model (Browne, 1984; Cudeck, 1988; Wothke & Browne, 1990; Verhees & Wansbeek, 1990) . In this approach, the method factors relate to the trait factors in a multiplicatively instead of additively. That is, method effects may increase the correlations of strongly correlated traits, more than they increase the correlations of weakly correlated traits. In other words, the higher the correlation between the traits, the greater the method effects. However, the data must be suitable for the direct product model parameterization. Additionally, the implementation and interpretation of the method are rather complex and prone to improper solutions, but less prone than the correlated methods (Brown, 2015; Lievens & Conway, 2001; Campbell & O’Connell, 1967) .

On the other hand, methodologists disagree on the merits of the multiplicative versus the additive models. Some support the additive ones as more successful (Bagozzi & Yi, 1990) , whereas others either disagreed (Goffin & Jackson, 1992; Coovert et al., 1997) or present mixed evidence (Byrne & Goffin, 1993) . For example, Corten et al. (2002) reported better results in the additive models in 71 out of 79 datasets used (Loehlin & Beaujean, 2017; Loehlin, 2004) .

CFA-MTMM models have received additional criticism because the meaning of the trait and method factors often remains unclear. This happens because CFA-MTMM analysis just assumes that trait and method factors are present. In order to properly interpret trait and method effects, what the latent factors mean must be clear (Koch et al., 2018) . Koch et al., argue that developments in stochastic measurement theory (see Steyer, 1989 ; Zimmerman, 1975 ) proposed a technique to define the MTMM model factors in terms of random variables in a well-defined random experiment. Koch et al. (2018) add that a wide range of models for different purposes and data structures can be defined with this method but not all the CFA MTMM models.

With so many alternative approaches, is difficult for the researcher to choose the most appropriate. One of the most important criterion for selecting the appropriate model is the type of method to use. The researcher must examine: 1) the methods used in the study; 2) the data and the structure of the random experiment implied by the measurement design, and 3) the possible conceptualization of method effects. For a study with interchangeable methods, a classic CFA modelling strategy can be chosen. For a study with structurally different methods, several other modelling alternatives can be used. Each model has distinct features that should be appropriate for the research question of the study. This means, there is no optimal model for all types of CFA MTMM measurement designs. The selection of the model must be driven by theory and hypotheses studied (Koch et al., 2018) . For comparisons of these alternatives, readers can refer to other sources (Eid et al., 2008; Pohl & Steyer, 2010; Saris & Aalberts, 2003) .

Regarding CFA-MTMM research in general, Loehlin and Beaujean (2017) and Loehlin (2004) review that the implementation of CFA MTMM models within and across groups has also been discussed (Marsh & Byrne, 1993) . Marsh and Hocevar, (1988) reported using CFA MTMM with higher-order structures (Loehlin & Beaujean, 2017; Loehlin, 2004) . For applied examples the reader can refer to Bagozzi and Yi (1990) , Coovert, Craiger, and Teachout, (1997) and Lievens and Conway (2001) . Next we will focus on the longitudinal CFA-MTMM research.

Generally, longitudinal measurement designs permit to focus on: 1) change and/or variability over time; 2) measurement invariance (MI), or indicator-specific effects; and 3) examine potential causal relationships ( Coulacoglou & Saklofske, 2017

Longitudinal CFA-MTMM models (or multitrait-multimethod-multioccasion; MTMM-MO) present numerous advantages. For example, they permit researchers to study the true stability (i.e., without measurement error) and the true change of the target constructs in tandem with method effects inherent to different measurement levels (for example rater and target-construct level). In other words, researchers can focus on the change of the target construct over time (e.g. positive affect) taking also into account the change of different method effects over time. Furthermore, each measurement occurrence can be analyzed with regards to the convergent and discriminant validity of measures used. The concept of change of convergent validity over time may be particularly pertinent in intervention studies. Furthermore, the effect of external covariates can be regressed into the model like gender, age, SES or personality variables (Koch et al., 2018) .

Koch, Schultze, Eid, and Geiser (2014) introduced a longitudinal MTMM modeling frame for numerous MTMM methods termed Latent State-Combination-Of-Methods model (LS-COM). The LS-COM modeling frame incorporates the advantages of SEM, multilevel modeling, longitudinal modeling, and MTMM modeling with the flexibility of interchangeable and structurally different methods (Coulacoglou & Saklofske, 2017) . For details refer also to Koch et al., 2018 .

MTMM was developed in an attempt to assess: 1) the relationship between the same construct (trait) and the same methods of measurement (with the reliabilities in the diagonal in

The classic MTMM theory encompasses the following concepts. The monotrait-monomethod blocks comprise the correlations between the traits, measured by one common method. The monotrait-heteromethod block contains the correlations between the traits measured by different methods, to examine convergent validity. The higher the correlations in the monotrait-heteromethod block the higher the convergent validity. The heterotrait-monomethod block refers to correlations between different traits measured by the one common method to evaluate the discriminant validity of the measures. The higher the correlations in the heterotrait-monomethod block the lowest the discriminant validity. Finally, the heterotrait-heteromethod block includes correlations of different traits assessed by different methods, also indicating discriminant validity (Koch et al., 2018) . These four data arrangements, show different correlations levels, that can be ranked from the highest to the lowest as follows: 1) the reliability coefficients in the diagonals of the monomethod blocks; 2) the validity coefficients in the diagonals of the heteromethod blocks; 3) the heterotrait monomethod coefficients; and 4) heterotrait heteromethod coefficients; Note that 3) and 4) are off-diagonal elements (Sawilowsky, 2007) . See for example the multitrait-multimethod matrix in

CFA methods to MTMM matrices are to some extent underutilized in the empirical literature, and the same is true for the “classic” MTMM approach in general. Probably, this is to some extent due to the lack of availability of multiple measures for a given construct (Brown, 2015) . For example, as a minimum, three assessment methods must be typically available for the CU method. Additionally, MTMM methods can be also employed for constructs assessed by a single assessment method, e.g. parenting dimensions measured by different self-report questionnaires (c.f. Green, Goldman and Salovey, 1993 as quoted by Brown, 2015 ). CFA MTMM models can be easily used in tandem with other methods. As Brown (2015) , comments they can be integrated into structural equation models or MIMIC models. In this way, trait factors could be related to background variables in the role of external validators establishing, for example, the predictive validity of traits. A second possible extension is to integrate an MTMM model across different groups to test whether trait factors show measurement invariance (Brown, 2015) .

As originally theorized, evidence of convergence and discriminant validity can be shown by correlations between scores of various instruments. In Campbell and Fiske’s own words, (1959: p. 104) , “Convergence between independent measures of the same trait and discrimination between measures of different traits”. In a similar vein, the current version of the APA standards (APA/AERA, 1999, 2014) indicates that discriminant and convergence validity evidence should be established through correlations of test scores with scores from different instruments and not with different subscales, as noted by Gunnell et al. (2014) . Almost 60 years after, the CFA framework renders the MTMM method an equally valuable tool to the classic MTMM when first introduced (Koch et al., 2018) .

The author declares no conflicts of interest regarding the publication of this paper.

Kyriazos, T. A. (2018). Applied Psychometrics: The Application of CFA to Multitrait-Multimethod Matrices (CFA-MTMM). Psychology, 9, 2625-2648. https://doi.org/10.4236/psych.2018.912150