Applied Psychometrics : The Application of CFA to Multitrait-Multimethod Matrices ( CFA-MTMM )

The application of CFA to Multitrait-Multimethod Matrices (MTMM) is an elaborated method for the evaluation of construct validity in terms of the discriminant and convergent validity as well as method effects. It is implemented to evaluate the acceptability of a questionnaire as a measure of constructs or latent variables, during the structural and external stages of an instrument validation process. CFA can be carried out in MTMM matrices, like in any other covariance matrix, to study the latent variables of traits and methods factors. CFA-MTMM models can distinguish systematic trait or method effects from unsystematic measurement error variance, thus offering the possibility of hypotheses testing on the measurement model, while controlling possible effects. A plethora of CFA-MTMM approaches have been proposed, but the Correlated Methods Models and the Correlated Uniqueness Models are two CFA-MTMM methods most widely used. This work discusses how these two approaches can be parametrized, and how inferences about construct validity and method effects can be drawn on the matrix level and on parameters level. Then, the two methods are briefly compared. Additional CFA MTMM parameterizations are also discussed and their advantages and disadvantages are summarized.


Introduction
Test developers ought to publish only measurement instruments meeting the How to cite this paper: Kyriazos, T. A. (2018).Applied Psychometrics: The Application of CFA to Multitrait-Multimethod guidelines specified by the APA standards (Franzen, 2002).Inaccurate measurements may lead to incorrect diagnoses, suboptimal treatment decisions or biased treatment effects estimates (Koch, Eid, & Lochner, 2018;Courvoisier, Nussbeck, Eid, Geiser, & Cole, 2008).Additionally, the adequacy of inferences made based on test scores is a central issue in social and behavioral sciences and it is related to validity (Messick, 1980(Messick, , 1989(Messick, , 1995;;APA/AERA & NCME, 1999, 2014).Construct validity is the extent to which a measuring tool "reacts" the way the construct purports to react when compared with other, well-known measures of different constructs (DeVellis, 2017) and it consists the central focus of each measurement process (Kline, 2009) and an all-embracing principle of validity (Messick, 1995;Brown, 2015).Construct validity (Cronbach & Meehl, 1955) examines the theoretical relationship of a variable (like the scale score) to other variables (Kyriazos, 2018) and it incorporates the internal scale structure (Zinbarg, Yovel, Revelle, & McDonald, 2006;Revelle, 2018) or the correct measurement of variables intended to be examined (Kline, 2009).Construct validation is an ongoing process (Franzen, 2002).
To this end, Campbell and Fiske (1959) introduced the Multitrait-Multimethod (MTMM) Matrix, an evaluation method of the acceptability of questionnaires as measures of constructs or latent variables (Price, 2017).This method involves a correlation matrix customized to enhance the evaluation of construct validity 1 in terms of the discriminant and convergent validity (Brown, 2015).The method that Campbell and Fiske (1959) operationalize is based on the idea that theoretically related scores of the same or similar traits should also show high positive correlations with each other whereas theoretically unrelated scores of different constructs should also show low or insignificant correlations.The former refers to convergent validity and the latter refers to discriminant validity (Koch, Eid, & Lochner, 2018;Franzen, 2002).
The purpose of this study is to describe the application of Factor Analysis-EFA and mainly CFA-into the Multitrait-Multimethod Matrices (MTMM), as an elaborated approach of studying convergent validity, discriminant validity, and method effects during the estimation of construct validity.

Basic Concepts & Overview of the "Classic" MTMM
The Multitrait-multimethod (Campbell & Fiske, 1959) is "an analytic method that includes evaluation of construct validity relative to multiple examinee traits in relation to multiple (different) methods for measuring such traits" (Price, 2017: p. 140).MTMM arranges the analysis of validity evidence in a manner that transcends the common "set of associations" approach, permitting the examination of associations among several psychological variables, each being measured through several methods (Furr, 2011).More specifically, two or more traits are measured with two or more methods.Traits could be hypothetical constructs about stable characteristics, like cognitive abilities.On the other hand, methods 1 Construct Validity can be assessed by other methods along with MTMM.See Kyriazos et al. (2018aKyriazos et al. ( , 2018bKyriazos et al. ( , 2018cKyriazos et al. ( , 2018d, 2018e) , 2018e) for applied examples of construct validation studies.Psychology can encompass different occasions, or means of data collection (e.g., self-report, structured interview and observation), or multiple-test forms, e.g. for different informants (teachers, parents and peers).Note that, trait interpretation suggests that a latent attribute or construct explains the consist patterns observed in the scores (Price, 2017;Kline, 2016).The MTMM method is carried out during the structural and external stages of the validation process (Price, 2017), see Table 1 and Figure 1 for an overview.
For example, during the development of a new self-report scale on social anxiety, the researcher could also measure three constructs (e.g., affect, self-esteem and depression) by three methods for each construct (i.e.self-report, report by partner, and report by peer).Correlations among all measures for all domains are calculated, and they are arranged in a "multitrait-multimethod" correlation matrix (Furr, 2011), like the one presented in Table 2.
For the simplest case of MTMM, at least two methods and two constructs are required (2 methods × 2 traits) to generate a multitrait-multimethod design.
Each score is a trait-method unit or TMU (Campbell & Fiske, 1959: p. 81;Koch et al., 2018).The MTMM correlation matrix (see Table 2) consists of four sets of correlations: 1) the monotrait-monomethod block correlations; 2) the monotrait-heteromethod block correlations; 3) the heterotrait-monomethod block correlations; and 4) the heterotrait-heteromethod block correlations (Koch et al., 2018;Furr, 2011).The correlations are divided into the two different method blocks of the table : 1) The monomethod blocks; and 2) the heteromethod blocks (Trochim, 2002;Brown, 2015).Additionally, two different sets of triangles are nested in the two blocks (Campbell & Fiske, 1959: p. 82;Trochim, 2002): 1) The heterotrait-monomethod triangles; and 2) The heterotrait-heteromethod triangles (Table 2).The general layout of an MTMM matrix is presented in Table 2.This example layout involves three different traits, each measured by three methods, generating nine separate variables.The interpretation of the coefficients contained in the body of the table, is included in the table note.
The monomethod blocks emerge from correlations of measurements by the same method.Their number equals the methods used.In order to account for measurement error (Koch et al., 2018), the reliability coefficients can be inserted  in the diagonal of these blocks called Reliability Diagonal (Campbell & Fiske, 1959: p. 81).The reliability diagonal should preferably comprise the largest coefficients in the matrix (Campbell & Fiske, 1959;Price, 2017), i.e. the measure should show the strongest correlations with itself than with any other construct in the matrix (Trochim, 2002;Brown, 2015).For example, in Table 2 the reliability coefficients range from 0.78 to 0.96.Next, to the reliability diagonal, the off-diagonal elements are the heterotrait-monomethod triangles (Campbell & Fiske, 1959: p. 82;Trochim, 2002) presenting correlations evidencing the discriminant validity of different traits measured by same methods, thus they correla- tions should be the lowest of all (Price, 2017).
The heteromethod blocks emerge from correlations of measurements by different methods.There number equals to k (k − 1)/2, where k is the number of methods used (Trochim, 2002).The correlations in their diagonals represent the convergent validity of the same trait measured by different methods.Thus, they should be strong and positive (Price, 2017;Brown, 2015).For example, in Different constructs (not associated)

= Strongest Monotrait-Monomethodcorrelations
Table 2 correlations suggesting convergent validity are ranging from r = 0.47 to r = 0.69.The off-diagonal elements in the heteromethod blocks called heterotrait-heteromethod triangles (Campbell & Fiske, 1959: p. 82;Trochim, 2002).They present correlations evidencing discriminant validity because different traits are measured by different methods.They also should be the lowest of all (Price, 2017).Brown (2015) explains that discriminant validity is demonstrated when weaker correlations emerge between different traits measured by different methods (i.e., heterotrait-heteromethod coefficients) in comparison to the correlations of the validity diagonal (i.e., that is monotrait-heteromethod coefficients).For example, in Table 2, discriminant validity is supported because the heteromethod block correlations in the off-diagonal elements are lower (from r = 0.11 to r = 0.45) than the validity coefficients (from r = 0.47 to 0.69).Finally, Brown (2015) explains that method effects may be evaluated by comparing the differential magnitude of correlations in the off-diagonal elements of the monomethod blocks.Campbell and Fiske (1959: pp. 82-83) proposed four rules for the evaluation of MTMM matrices.The correlations in the monotrait-heteromethod block have to be statistically significant and strong.The correlations in the heterotrait-heteromethod block should be weak or should equal the correlations in the heterotrait-monomethod block.The correlations in the heterotrait-monomethod block should be smaller than the correlations in the monotrait-heteromethod block.The pattern of the trait intercorrelations in all heterotrait triangles should be similar for the mono-as well as the heteromethod blocks.This property also refers to the discriminant validity of the measures (summarized by Koch et al., 2018).For details refer also to Eid (2010).Finally, Trochim (2002) in line with Campbell and Fiske (1959) also suggests the following guidelines for the interpretation of an MTMM matrix: 1) Reliability coefficients must be the highest values in the matrix because the correlations between a trait and itself are expected to be the strongest than with any other construct (also Brown, 2015).2) The coefficients presented in the validity diagonal must be statistically different from zero and preferably moderately strong.3) Every coefficient in the validity diagonal must be the greatest column and row value of the method block it belongs.Recall that all off-diagonal elements must be the smallest to evidence discriminant validity.4) Each validity coefficient must be greater than all the off-diagonal elements in the heterotrait monomethod triangles.If this is not true, then a method effect may be present.5) The relationships of the different traits must be coherent in all triangles, i.e. relationships between every trait dyad must be cohesive, without areas of dramatic changes (Trochim, 2002).In spite of the ease of application of these rules, in practice, the application of MTMM method was criticized as having several limitations (Marsh, 1988(Marsh, , 1989;;Schmitt & Stults, 1986;Eid, 2010).Psychology

Criticism of the "Classic" MTMM
Despite the fact that the Campbell and Fiske (1959) technique was a significant progress for examining construct validity, the MTMM approach was not widely accepted the years immediately following its inception (Brown, 2015).Additionally, several limitations were noted first by Jackson (1969).Jackson's criticisms as summarized by Franzen (2002), mainly focused on the fact that the MTMM method juxtaposes distinct patterns without taking into account the overall structure of the relationships (i.e.between-network; Cronbach & Meehl, 1955;Martin & Marsh, 2006).This argument is based on the idea that the correlation pattern may be affected by the way variance is distributed when measuring the target traits.To alleviate the MTMM disadvantages, Jackson (1969) suggested to carry out an EFA (PCA actually) in the monomethod matrix.Specifically, the correlation matrix is orthogonalized and then submitted to Principal Components Analysis, with a varimax rotation.The extracted factors equal the number of matrix traits.However, Jackson's suggestion while offering some advantages, it assumes no relationships exist among the traits and this is rarely the case in psychology.Additionally, the influence of different measurement methods cannot be taken into account when the monomethod matrix is used (Franzen, 2002).
Finally, EFA failed to obtain meaningful solutions of MTMM data due to the inherent restrictions of EFA to specify correlated errors (Brown, 2015).
Later, Franzen (2002) continues Cole, Howard, and Maxwell (1981) compared the MTMM results with constructs mono-operationalization suggesting that as a rule, the validity coefficients were misleadingly deflated.Therefore, the MTMM method and factor analysis may be used in tandem (Franzen, 2002).Finally, Cole (1987) repeated to a certain extent Jackson's criticisms, i.e. absence of concrete guidelines regarding the size of the zero-order correlation coefficients.Additionally, Cole argued that the cost of the MTMM application in clinical settings deems the method inapplicable (Franzen, 2002).Crucially, Cole noted that the MTMM is prone to correlated errors due to factors like the time of day, or the instrument administrator.Cole suggested the use of CFA instead to examine discriminant and convergent validity (Franzen, 2002).For a detailed description of MTMM limitations see also Marsh (1988Marsh ( , 1989) ) and Schmitt & Stults (1986).
At the same time, non-factorial oriented attempts to find an appropriate method of statistical analysis for the data in the matrix were numerous (Eid et al., 2008;Eid, 2010).These methods were mainly based on analysis of variance models or nonparametric alternatives and the reason for the lack of success was attributed to conflicting data in the diagonals in relation to the data in the triangles (Sawilowsky, 2007).For example, Sawilowsky (2002Sawilowsky ( , 2007) ) proposed a statistical test for analysis MTMM data where the null hypothesis is that the coefficients in the matrix are unordered from high to low.This is tested against the alternative hypothesis of an increasing trend from the lowest level (heterotrait heteromethod) to the highest level i.e. reliability coefficients (Sawilowsky, 2007: Psychology p. 180).See Sawilowsky (2002Sawilowsky ( , 2007) ) for more details and an applied example.

CFA Approaches to Analyzing the MTMM Matrix (CFA-MTMM Models)
Actually, MTMM matrix regained interest upon the realization that EFA procedures (Jackson, 1969) or CFA (Cole et al., 1981;Cole, 1987;Flamer, 1983;Marsh & Hocevar, 1983;Widaman, 1985) are equally applicable for MTMM data analysis.Specifically, CFA can be carried out in MTMM matrices, like in any other covariance matrix, to study the latent dimensions of traits and methods factors (Brown, 2015).

Correlated Methods Models
This method was among the first significant developments in the field of CFA-MTMM models (Koch et al., 2018) and was developed by Keith Widaman (1985) proposing a nested model comparison.
The model comparison presented by Widaman (1985)  The specified MTMM model becomes a baseline model, compared against a set of nested models in which parameters are either removed or added.Model comparison criteria used are the χ 2 (Δχ 2 ) and other fit indicators (see Byrne, 2012), in a procedure similar to the multi-group CFA.Based on this comparison, evidence of convergent and discriminant validity emerges, both at the matrix level and at the individual parameter level (Byrne, 2012).Figure 2 is a path diagram for an MTMM matrix of 3 traits and 3 methods to describe the steps of the correlated methods.Testing for evidence of convergent and discriminant validity involves comparisons between the model in Figure 2 and three alternative nested MTMM models in Figures 3-5 (Byrne, 2012;Brown, 2015).
The Steps of the Correlated Methods Byrne (2012) describes the steps for comparing the nested models as follows.
Step 1-The CTCM Model: Test the first model (see path diagram in Figure 2) is the hypothesized baseline model against which the three alternative CFA models are compared.A common problem in this step of the process is a non-positive matrix error (see Kenny & Kashy, 1992;Marsh, 1989).Usually, a negative variance is associated, either with a residual or a factor and this problem is generally attributed to overparameterization of the MTMM models, due to their inherent complexity (Byrne, 2012; quoting also Wothke, 1993).Possible   results in a solution and the fit of this CTCM model must be considered (Byrne, 2012).
Step Step 4-The CTUM Model: The last correlated methods MTMM model is specified which is a Freely Correlated Traits/Uncorrelated Methods (CTUM) see Figure 5.It is different from Model 1 only in the absence of correlations in the method factors.The goodness of-fit for this model is evaluated (see also Byrne, 2012).
Step 5-Evaluation of Construct Validity at the Matrix Level: After the evaluation of goodness-of-fit of the four MTMM models and with the assumption that the fit is acceptable (within the acceptability criteria used for typical CFA models, e.g.Hu & Bentler, 1999;Brown, 2015), evidence of construct and discriminant validity is examined.This information at the matrix level derives through the comparison of particular pairs of models.Note that typically, a Psychology summary of goodness-of-fit statistics of all four MTMM models is presented in a table, and a summary of model comparisons in a second table, see an example of both in Table 3 adapted by Byrne (2012).
Step 5a-Convergent Validity at the Matrix Level: Generally, the principles applied when testing for invariance are also applied when comparing models in this step (see Table 3).In line with the procedure proposed by Widaman (1985), evidence of convergent validity is tested by comparing the CTCM model (Model in Figure 2) to the NTCM model (Model in Figure 3).When the difference in χ 2 (Δχ 2 ) between these models is highly significant and the fit difference (ΔCFI) is also large evidence of convergent validity is suggested (Byrne, 2012).
Step 5b-Discriminant Validity at the Matrix Level: Discriminant validity is usually evaluated considering both traits and methods.Evidence of discriminant validity among traits emerges from the comparison of the CTCM Model in which traits are freely correlated (see Model in Figure 2) with the PCTCM model, specified in step 3 in which traits are perfectly correlated (see Model in Figure 4).The larger the difference between the values of χ 2 and CFI, the stronger the evidence of discriminant validity.The procedure is repeated to show evidence of discriminant validity regarding method effects.Specifically, the CTCM Model in which methods are freely correlated-specified in step 1 and presented as path diagram in Figure 2-, is compared to the CTUM model in which methods are uncorrelated-specified in step 4 and presented as path diagram in Figure 5.A large Δχ 2 (and/or considerable ΔCFI) suggests common method bias, therefore the lack of discriminant validity across measurement methods (Byrne, 2012).For an applied example see Byrne (2012).See Goodness of fit criteria used in Table 3 and the Nested Models Compared in Table 4. Byrne suggests that the difference criteria for the comparison of the nested models to be identical to measurement Invariance criteria (Cheung & Rensvold, 2002) "as an evaluative base upon which to determine evidence of convergent and discriminant validity" (Byrne, 2012: p. 300).For a more detailed description and an applied example see Byrne (2012).
Step 6-Evaluation of Construct Validity at the Parameter Level: Traitand method-related variance can be examined more thoroughly by examining individual parameter estimates, i.e. the factor loadings and factor correlations of  Step 6a-Convergent Validity at the Parameter Level: The size of the trait loadings indicates convergent validity.First, all loadings are examined to see if they are statistically significant.Then all factor loadings across traits and factor loadings across methods are compared, to see whether method variance or trait variance is higher over all the available ratings (Byrne, 2012).
Step 6b-Discriminant Validity at the Parameter Level: Factor correlation matrices contain the evidence of discriminant validity regarding particular traits and methods.Generally, correlations among traits should theoretically be small to show evidence of discriminant validity, but such findings are rare in psychology.Regarding the correlations of the method factors, discriminability of these parameters values suggests the extent to which the methods are dissimilar, testing a major assumption of the MTMM strategy (Campbell & Fiske, 1959;Byrne, 2012).For an applied example of the correlated method refer to Byrne (2012) and Brown (2015).
Some simpler alternatives to CTCM models have been proposed, and the correlated-uniqueness (CU) model described next is one of them (Kline, 2016).
The CU model identification must abide by the following rules as reported by Brown (2015).There must be at least two traits (t) and three methods (m).
However, a 2 t × 2 m model can be also specified in the factor loadings of indicators that belong to the same trait factor are fixed to be equal.Rules regarding Psychology the CU trait model are the same as in the correlated methods approach.Specifically: 1) Each indicator must load on one trait factor with all other cross-loadings fixed to zero; and 2) The trait factor correlations are freely estimated.Consequently, the major difference between the CU method in comparison to the correlated methods is the estimation of method effects.In the CU model, method effects are assessed by adding correlated uniquenesses (errors) among the indicators of the same assessment method instead of specifying method factors (Brown, 2015).Figure 6 presents the path diagram of the correlated uniqueness CFA specification for an MTMM matrix with 3 traits and 3 methods.
After the model specification, the model is tested and the fit of the solution to the data is evaluated.Except fit the inspection of standardized residuals, the modification indices must be also evaluated (as proposed by Brown, 2015).The parameter estimates Brown (2015) continues are then examined.That is, when loadings of traits factors are substantial and statistically significant, this is an evidence supporting convergent validity.Conversely, when correlations among the trait factors are substantial, this is indicative of low discriminant validity.
Method effects are assessed when correlated uniquenesses among indicators of the same method are moderate or large in magnitude (Brown, 2015).and Brown (2015).Saris and Aalberts (2003) also examined different interpretations of CU models.
Specifically, an advantage of the CT-CM model is that it permits fragmenting observed variance into a trait, method, and error variance in comparison to the CU model that is more restrictive (Eid et al., 2006;Geiser et al., 2008;Koch et al., 2018).Kenny and Kashy (1992) argue that this fragmentation of the correlated methods model is the reason why this method corresponds directly to Campbell and Fiske's (1959) original conceptualization of the MTMM matrix.Moreover, the parameter estimates of the correlated methods models offer direct evidence of construct validity.For instance, substantial trait factor loadings show convergent validity; small and/or non-significant method factor loadings suggest that method effects are absent and modest trait factor correlations suggest good discriminant validity.The addition of method factors permits interpretation of method effects (Brown, 2015).
Nevertheless, a drawback of the correlated methods model is that it is prone to underidentification errors, or Heywood cases and large standard errors.Because of these problems with the correlated methods model, methodologists suggested (Kenny & Kashy, 1992;Marsh & Grayson, 1995) using the correlated uniqueness model for the analysis of MTMM data.On the other hand, others (Lance, Noble, & Scullen, 2002) argue that given the substantive strengths of the correlated methods model, the CU model should be preferred only as last resort, i.e. when the correlated methods model fails to be identified (Brown, 2015).
More specifically, the CT-C (M-1) model is analogous to the correlated methods model (presented in Figure 2) but it comprises one method factor less than the traits included (M-1).For example, the model presented in Figure 2 could be transformed into a CT-C (M-1) model by eliminating the Method 1 latent variable.This specification answers to identification problems of the correlated methods model (Brown, 2015).They can be defined in three steps: 1) Selection of a reference method (standard), on the basis of theory and research questions (Geiser, Eid, & Nussbeck, 2008).2) Selection of general trait variables (common or indicator-specific) and M-1 method factors.3) Define additional constraints as a function of how non-correlationally measurement errors are, how non-correlationally method variables with trait variables of the same TMU are, and what is the homogeneity of the variables.Note, that if parameters of the model are restricted, the model data is equivalent to the common factor TMU model (Koch et al., 2018also quoting Geiser et al., 2008;Geiser, Eid, West, Lischetzke, & Nussbeck, 2012).
Latent difference models, Koch et al. (2018) argue, offer a possibility to contrast methods.Initially, they were developed for longitudinal research (Steyer, Partchev, & Shanahan, 2000) but they can successfully use in multimethod research (see Pohl, Steyer & Kraus, 2008).This technique also involves a reference method.Nevertheless, the latent method variables are not specified as residuals regarding the true-score variables of the reference method.It is hypothesized that each indicator taps a trait variable, but method variables are unidimensional.Therefore, a common method factor is specified for all indicators of the same method and the same trait (Koch et al., 2018).
One more alternative approach is the direct product model (Browne, 1984;Cudeck, 1988;Wothke & Browne, 1990;Verhees & Wansbeek, 1990).In this approach, the method factors relate to the trait factors in a multiplicatively instead of additively.That is, method effects may increase the correlations of strongly correlated traits, more than they increase the correlations of weakly correlated traits.In other words, the higher the correlation between the traits, the greater the method effects.However, the data must be suitable for the direct product model parameterization.Additionally, the implementation and interpretation of the method are rather complex and prone to improper solutions, but less prone than the correlated methods (Brown, 2015;Lievens & Conway, 2001;Campbell & O'Connell, 1967).
CFA-MTMM models have received additional criticism because the meaning of the trait and method factors often remains unclear.This happens because CFA-MTMM analysis just assumes that trait and method factors are present.In order to properly interpret trait and method effects, what the latent factors mean must be clear (Koch et al., 2018).Koch et al., argue that developments in stochastic measurement theory (see Steyer, 1989;Zimmerman, 1975) proposed a technique to define the MTMM model factors in terms of random variables in a well-defined random experiment.Koch et al. (2018) add that a wide range of models for different purposes and data structures can be defined with this method but not all the CFA MTMM models.
With so many alternative approaches, is difficult for the researcher to choose the most appropriate.One of the most important criterion for selecting the appropriate model is the type of method to use.The researcher must examine: 1) the methods used in the study; 2) the data and the structure of the random experiment implied by the measurement design, and 3) the possible conceptualization of method effects.For a study with interchangeable methods, a classic CFA modelling strategy can be chosen.For a study with structurally different methods, several other modelling alternatives can be used.Each model has distinct features that should be appropriate for the research question of the study.This means, there is no optimal model for all types of CFA MTMM measurement designs.The selection of the model must be driven by theory and hypotheses studied (Koch et al., 2018).For comparisons of these alternatives, readers can refer to other sources (Eid et al., 2008;Pohl & Steyer, 2010;Saris & Aalberts, 2003).

Summary and Conclusion
MTMM was developed in an attempt to assess: 1) the relationship between the same construct (trait) and the same methods of measurement (with the reliabilities in the diagonal in Table 3); 2) the relationship between the same construct (trait) using different measurement (i.e., convergent validity); and 3) the relationship between different constructs using different methods of measurement (i.e., discriminant validity), as commented by Price (2017).
The classic MTMM theory encompasses the following concepts.The monotrait-monomethod blocks comprise the correlations between the traits, measured by one common method.The monotrait-heteromethod block contains the correlations between the traits measured by different methods, to examine convergent validity.The higher the correlations in the monotrait-heteromethod block the higher the convergent validity.The heterotrait-monomethod block refers to correlations between different traits measured by the one common method to evaluate the discriminant validity of the measures.The higher the correlations in the heterotrait-monomethod block the lowest the discriminant validity.Finally, the heterotrait-heteromethod block includes correlations of different traits assessed by different methods, also indicating discriminant validity (Koch et al., 2018).These four data arrangements, show different correlations levels, that can be ranked from the highest to the lowest as follows: 1) the reliability coefficients in the diagonals of the monomethod blocks; 2) the validity coefficients in the diagonals of the heteromethod blocks; 3) the heterotrait monomethod coefficients; and 4) heterotrait heteromethod coefficients; Note that 3) and 4) are off-diagonal elements (Sawilowsky, 2007).See for example the multitrait-multimethod matrix in Table 2. Psychology CFA methods to MTMM matrices are to some extent underutilized in the empirical literature, and the same is true for the "classic" MTMM approach in general.Probably, this is to some extent due to the lack of availability of multiple measures for a given construct (Brown, 2015).For example, as a minimum, three assessment methods must be typically available for the CU method.Additionally, MTMM methods can be also employed for constructs assessed by a single assessment method, e.g.parenting dimensions measured by different self-report questionnaires (c.f.Green, Goldman and Salovey, 1993 as quoted by Brown, 2015).CFA MTMM models can be easily used in tandem with other methods.
As Brown (2015), comments they can be integrated into structural equation models or MIMIC models.In this way, trait factors could be related to background variables in the role of external validators establishing, for example, the predictive validity of traits.A second possible extension is to integrate an MTMM model across different groups to test whether trait factors show measurement invariance (Brown, 2015).
As originally theorized, evidence of convergence and discriminant validity can be shown by correlations between scores of various instruments.In Campbell and Fiske's own words, (1959: p. 104), "Convergence between independent measures of the same trait and discrimination between measures of different traits".
In a similar vein, the current version of the APA standards (APA/ AERA, 1999AERA, , 2014) ) indicates that discriminant and convergence validity evidence should be established through correlations of test scores with scores from different instruments and not with different subscales, as noted by Gunnell et al. (2014).Almost 60 years after, the CFA framework renders the MTMM method an equally valuable tool to the classic MTMM when first introduced (Koch et al., 2018).
had (t) traits and (m) methods where (t) is the number of trait factors and (m) is the number of method factors, abiding by five key specifications rules(Brown, 2015): 1) for model identification, at least three traits (t) and three methods (m) are required; 2) (t) × (m) indicators are specified to define (t) + (m) factors; 3) each indicator loads on two factors, i.e. a trait factor and a method factor and all other cross-loadings are set to zero; 4) trait factors inter-correlations and method factors inter-correlations are freely estimated; 5) trait-method factors correlations are typically constrained to zero; and 6) indicator uniquenesses are estimated freely but cannot be specified to correlate with uniquenesses of other indicators.Each indicator in this specifi-Psychology cation is a function of a trait, method, and unique factors, Brown (2015) explains.

Figure 3 .
Figure 3. Model 2-The No Traits/ Correlated Methods Model (NTCM).Example of Correlated Methods CFA MTMM with 3 Traits and 3 Methods.Model 2 of 4 Nested models that are specified (Nested model to test Convergent Validity).

Figure 4 .
Figure 4. Model 3-Perfectly Correlated Traits/Freely Correlated Methods Model (PCTCM).Example of Correlated Methods CFA MTMM with 3 Traits and 3 Methods.Model 3 of 4 Nested models that are specified (Nested model to test Discriminant Validity).
Figure 5. Freely Correlated Traits/Uncorrelated Methods Model (CTUM).Example of Correlated Methods CFA MTMM with 3 Traits and 3 Methods.Model 4 of 4 Nested models that are specified (Nested model to test Discriminant Validity).

For
Figure 6.The Correlated Uniquenesses Model (CU).Path diagram of the correlated uniquenesses CFA specification for an MTMM matrix of 3 traits and 3 methods.Method effects are evaluated by specifying correlated uniquenesses (deltas or δ) among the indicators of the same assessment method.

Table 1 .
Summary of the big picture of the place of MTMM in the construct validation process.

Table 2 .
Sample MTMM Matrix of three Traits (Constructs) and 3 Methods.
Notes: Based on a table used byByrne (2012: p. 299); Goodness of fit criteria are the same as standard CFA.

Table 4 .
Correlated methods Comparison of Nested Models in CFA MTMM (Correlated methods).
the CTCM model (the Model of step 1, see path diagram in Figure2) as Byrne (2012) explains.
comment, MTMM research has gained insight on many aspects of the CTCM minus one (CT-C[M-1]) model coined by Eid, Koch et al., 2018.stability (i.e., without measurement error) and the true change of the target constructs in tandem with method effects inherent to different measurement levels (for example rater and target-construct level).In other words, researchers can focus on the change of the target construct over time (e.g.Saklofske, 2017).For details refer also toKoch et al., 2018.