Advancing Measures in Advertising, Communication, and Public Relations: A Comprehensive Review of Scale Development Studies, 1960-2023 ()
1. Introduction
Developing reliable, valid measures is crucial for high-quality research across disciplines, as such development enables the effective testing and advancement of theories. Recognizing the importance of scale reliability and validity for academic progress, scholars have considered scale development in discussions spanning the social/behavioral sciences, including psychology (e.g., Dawis, 1987; Worthington & Whittaker, 2006), marketing (e.g., Churchill, 1979), and management (e.g., Hinkin, 1995).
In advertising, communication, and public relations, scholars apply scales to explore various phenomena, such as opinions, attitudes, and behaviors. A selected group of scholars has pioneered discussions on the topic. McCroskey and Young (1979) highlighted issues with the application of factor analysis (FA) in communication research, whereas Park, Dailey, and Lemus (2002) examined exploratory FA (EFA) and principal component analysis (PCA) methodologies. They observed that reduction methods, retention criteria, and rotational techniques were often improper or inadequately documented, underscoring the need for meticulous FA application. Morrison (2009) built on the foundational analysis of Park et al. (2002) by scrutinizing the application of EFA in scales across 51 communication journals and found that the methodological choices pertaining to factoring within communication research lacked rigor.
Although previous studies have provided insight regarding FA in advertising, communication, and public relations research, they have often overlooked the full-scale development process and the comprehensive statistical analysis involved. In addition, they have mainly focused on EFA (e.g., Morrison, 2009) or both EFA and PCA (e.g., Park et al., 2002) while neglecting the growing importance of CFA for theory-based scale development.
With a view to addressing these gaps, our research contributes to the literature by detailing scale-development procedural steps, assessing methodologies in scale development studies, and offering recommendations for robust EFA and CFA applications. We analyze 240 articles on novel scale development across 38 journals in advertising, communication, and public relations with a publication date up to 2023. We aim to provide a comprehensive blueprint for creating refined, rigorous scales, thereby advancing scale-development discourse and practice.
2. Overview of New Scale Development
Developing a new scale is a daunting and time-consuming task. While the steps involved in constructing a solid new scale differ from scholar to scholar, researchers have offered a few steps as guidelines for developing a new scale (e.g., Anastasi, 1988; Churchill, 1979; Dawis, 1987; DeVellis, 2003; Kyriazos & Stalikas, 2018; Worthington & Whittaker, 2006; Spector, 1992; MacKenzie et al, 2011): 1) define the construct to be measured, 2) generate an item pool, 3) have experts test the content validity, 4) administer the measurement items to a developed sample, and 5) validate and norm the scale. Figure 1 illustrates the steps in developing a new measure.
2.1. Step 1. Define the Construct
The first step in developing a quality scale is to clearly and concretely define the
Source: (Spector, 1992: p. 8).
Figure 1. Steps in developing a measurement scale.
construct to be measured based on the literature. As the definition of a construct informs the development of items for the scale (Schwab, 1980), a scale developer should be precise in the conceptual specification of the construct, especially regarding what to include or exclude from the domain (Churchill, 1979). Moreover, the researcher should discuss how the construct to be developed is different from other related ones (MacKenzie et al., 2011). Consulting the literature is a prerequisite in conceptualizing the construct and narrowing the domain.
2.2. Step 2. Generate Items to Characterize the Construct
Once the construct of interest is conceptually defined, the next step involves generating a sample of items that fully capture all the essential aspects of the construct to be measured. Churchill (1979) outlines a few methods for generating representative items, including reviewing the literature, making deductions based on the theoretical definition of the construct or theoretical and empirical research on the focal construct, asking for suggestions from field experts, and conducting interviews or focus groups. To achieve the goal of this step, a scale developer must write items that are clear, concise, distinctive, and comprehensible, while avoiding items that are ambiguous or unfamiliar. Doing so will help yield a stable set of underlying factors that correctly represent the construct in the factor-analysis procedure (Worthington & Whittaker, 2006).
2.3. Step 3. Evaluate Content Validity
The third step in developing a new high-quality scale is to evaluate content validity. Straub, Boudreau, and Gefen (2004) define content validity as “the degree to which items in an instrument reflect the content universe to which the instrument will be generalized” (p. 424). In evaluating content validity, content representativeness and content relevance to the domain are the two key elements to be judged. Content representativeness concerns whether “the individual item is representative of an aspect of the content domain of the construct,” while content relevance measures whether “the items as a set collectively representative of the entire content domain of the construct” (MacKenzie et al., 2011: p. 304).
In this stage, a scale developer recruits a group of individuals with expertise in the domain to review the item pool of the construct and thus measure content validity. Expert review can achieve three objectives: 1) evaluate the item pools for clarity, conciseness, technical errors, readability, face validity, and redundancy; 2) analyze content validity and the extent to which a set of items represents the construct to be measured; and 3) make recommendations for the addition or removal of items.
2.4. Step 4. Administer the Developed Measure
The next step in the scale-development process involves fully administering the measurement items to a sample of 100 to 500 individuals, followed by a subsequent item analysis (Comrey & Lee, 1992). Moreover, it is recommended that the minimum ratio of the number of respondents to the number of items in the developed measure be between 3:1 and 10:1 (Cattell, 1978; Everitt, 1975). The internal consistency of the measurement items is determined according to the data collected, and the alpha coefficient is frequently applied to assess the degree to which the items are internally consistent and reliable. Cronbach’s alpha has been widely applied to calculate internal consistency. Fornell and Larcker (1981), however, have suggested a construct reliability index that is based on the ratio of the variance accounted for by the latent construct to the total variance in the measures. Overall, .70 or higher is generally considered acceptable for a newly developed measure across these two indices (Nunnally & Bernstein, 1994). When acceptable internal consistency is achieved, a scale developer can then move on to the final step. If it is not achieved, however, the researcher should return to an earlier step and revisit the measurement items.
2.5. Step 5. Validate and Norm the Scale
The final step in the process of developing the scale is validation. A series of validation procedures should be performed to identify whether the items used measure the construct of interest as predicted. By evaluating the evidence collected, the researcher can determine whether the newly developed scale measures the construct as intended. This step requires empirical tests, including exploratory factor analysis (EFA), confirmatory factor analysis (CFA), and reliability estimates.
2.6. EFA and CFA in Scale-Development Research
Factor analysis is a statistical technique that can be used to identify a small number of factors (=latent constructs) from a large number of observed items (Worthington & Whittaker, 2006). With reference to scale development, factor analysis consists of two main categories: exploratory factor analysis and confirmatory factor analysis (Kahn, 2006).
Exploratory Factor Analysis (EFA). Exploratory factor analysis is used to identify a latent structure on the part of observed variables by extracting common factors that influence the measured variables when a hypothesis about those factors is not proposed (Park et al., 2002). When no theory or prior literature is applied, researchers can use factor loading scores to establish the factor structure of the dataset. This process can indicate how many factors are needed to explain the relationships among a set of indicators. In short, the primary purpose of EFA is to identify the factors for observed variables (items). Among the several rotation methods available, principal component analysis (PCA) with an orthogonal rotation has been well received (Spector, 1992). A scale developer may apply EFA to each sub-scale separately or to the entire dataset with a specific number of factors (Spector, 1992).
Confirmatory Factor Analysis (CFA). Confirmatory factor analysis is often used when researchers intend to develop the factor structure for the data based on a theory or prior literature (Bollen, 1989; Spector, 1992). In particular, CFA is used to test how well the developed factor structure actually fits the data. Confirmatory factor analysis is typically conducted after assessing an instrument using EFA, and it allows a scale developer to evaluate whether the factor structure, which is extracted by EFA, fits the data using a new sample. Alternately, without employing EFA, researchers can perform CFA to confirm the factor structure that is established based on a theory.
To examine the status of scale development in the advertising, communication, and public relations disciplines, this study poses the following research questions:
RQ1: Have scale-development studies taken adequate steps in developing new measures?
RQ2: Have EFA and CFA been applied correctly in scale-development studies in the field of advertising, communication, and public relations?
3. Method
3.1. Population
We conducted a detailed content analysis of peer-reviewed articles on new scale development up to 2023, examining 240 articles from 38 journals recognized by the National Communication Association and other major associations in advertising and public relations (Barry, 1990; Henthorne et al., 1998; Ki & Ye, 2017). This broad selection aimed to encapsulate comprehensive insights across closely allied fields.
3.2. Article Extraction
The inclusion criteria focused on studies that reported the creation of new scales, excluding those that tested existing ones. Instead of relying solely on key search terms, the researchers manually reviewed all issues of the selected journals from 1960 to 2023 to identify relevant scale development studies. Articles were thoroughly extracted based on their titles and abstracts. Prior to the main coding process, we carefully reviewed the titles and abstracts of all 38 selected journals to ensure their relevance to scale development. This rigorous selection process was essential for capturing the essence of scale innovation in the fields of advertising, communication, and public relations. Ultimately, 240 articles were identified.
3.3. Measures
In delineating the measures for scale development, our study highlighted several critical stages, which were as follows: initially defining the construct, ensuring content validity through expert review, conducting pilot tests before main data collection, and validating the scale via statistical analyses, including EFA and CFA. From item generation to validation, each stage was scrutinized for methodological rigor. We examined EFA procedures across multiple dimensions, namely, sample characteristics, factorability assessment, extraction and rotation methods, and item retention criteria. Similarly, for CFA, our analysis encompassed the use of structural equation modeling (SEM), sample size adequacy, fit indices, and model-modification practices.
3.4. Intercoder Reliability
Two trained doctoral students independently coded 10.3% of the articles (n = 18) to assess intercoder reliability. Since all variables were dummy coded, reliability was tested across 92 variables. The Cohen’s kappa values ranged from .712 to 1, indicating acceptable reliability. For instance, Cohen’s kappa was .90 for the definition of the construct, .93 for content validity, .71 for test-retest reliability, .85 for internal consistency, 1.0 for orthogonal rotation, .93 for commonalities, and .71 for item analysis.
4. Results
This study was designed to examine the status of scale development in the field of advertising, communication, and public relations. Table 1 lists the journal names and the number of articles derived from each journal. Of all the coded articles on scale-development studies, 75.8% were in the communications discipline (n = 182), followed by 15% in public relations (n = 36) and 9.2% in advertising (n = 22).
Table 1. Journals included in the analysis.
Journal |
Number of Articles |
Journal of Advertising |
4 |
Journal of Advertising Research |
9 |
Journal of Current Issues and Research in Advertising |
3 |
International Journal of Advertising |
5 |
Communication Research |
9 |
Human Communication Research |
9 |
Journal of Communication |
5 |
Journal of Broadcasting & Electronic Media |
5 |
Journalism & Mass Communication Quarterly |
16 |
Journal of Public Relations Research |
8 |
Public Relations Review |
14 |
Atlantic Journal of Communication |
5 |
Communication Methods and Measures |
14 |
Communication Education |
18 |
Communication Management Quarterly |
8 |
Communication Monographs |
8 |
Communication Quarterly |
15 |
Communication Reports |
3 |
Communication Research Report |
16 |
Communication Studies |
4 |
Environment Communication |
2 |
Health Communication |
10 |
Howard Journal of Communication |
1 |
Information, Communication & Society |
4 |
International Journal of Media Management |
1 |
Journal of Applied Communication Research |
4 |
Journal of Communication Management |
3 |
Journal of Computer-Mediated Communication |
1 |
Journal of Family Communication |
3 |
Journal of Health Communication |
10 |
Journal of Intercultural Communication Research |
3 |
Journal of International and Intercultural Communication |
2 |
Mass Communication and Society |
|
Media Psychology |
10 |
New Media & Society |
4 |
Southern Communication Journal |
2 |
Technical Communication Quarterly |
1 |
Western Journal of Communication |
1 |
The first research question asked whether the scale-development studies followed adequate steps in developing a new measure. Among the 240 studies, 67.1% explicitly defined the targeted construct (n = 161), whereas 32.9% mentioned previous definitions only briefly or did not discuss definitions at all (n = 79). About 32.9% of the studies for which the aim was to develop a new measure failed to define the construct.
Regarding face and content validity, it was found that 25.4% of the studies invited experts to review the initial pool of items (n = 61), whereas 74.6% did not cover this process (n = 179). Prior to the main administration, 53.8% of the studies collected a small sample to pilot test the items (n = 129); in contrast, 46.3% directly implemented the main data collection (n = 111). Regarding the number of datasets collected to validate a scale, 237 studies implemented one round of data collection (98.8%), 143 implemented two rounds (59.6%), 40 involved three rounds (16.7%), and two involved more than four rounds of data collection (0.1%).
Regarding the process of validating the scale, 71 studies used EFA only, whereas 46 used CFA only; 119 used both EFA and CFA. As displayed in Figure 2, there has been a remarkable increase in the quantity of studies using EFA and CFA since the 1990s. More than 79% of studies using EFA and 93% of studies using CFA were published after 1990. However, the number of studies using PCA and FA did not change much from 1960 to 2023.
Figure 2. Statistical analyses to validate scales by year.
4.1. Exploratory Factor Analysis (EFA)
Among the 240 new-scale-development articles evaluated, approximately 98.3% described conducting CFA or SEM and EFA (n = 236). About 29.6% used only EFA (n = 71), whereas 19.2% used only CFA (n = 46). The results indicate that most studies conducted data analysis based on more than one sample. Fifty-nine percent of the studies (n = 143) involved more than one round of data collection. Just over 16% of the studies (n = 40) involved more than three rounds of data collection, and approximately 1% (n = 2) involved more than four rounds of data collection.
Testing factorability. About 79.1% of the studies (n = 190) employed EFA to evaluate dimensionality. While the dimensionality of a dataset is related to the sizes of the correlations in the matrix, only four studies assessed the factorability of the correlation matrix before conducting EFA.
To justify performing an FA, it is necessary to determine whether the number of significant correlations existing among the observed items is sufficient. The three following indices are often used to detect the factorability of observed items: Kaiser–Meyer–Olkin (KMO), Bartlett’s test of sphericity (Bartlett, 1950), and individual measures of sampling adequacy (MSA).
The current study found that a large portion of the articles analyzed (n = 64, 26.7%) did not report any criteria that were used to evaluate the factorability of the correlation matrix. About 17.1% of studies used Bartlett’s test of sphericity (n = 41), while 16.3% of studies reported KMO results (n = 39), and 4.6% assessed MSA (n = 11). Based on this finding, scale developers in the communication domain are encouraged to provide greater evidence of scale factorability prior to performing EFA.
Extraction methods. In the current study, it was found that most studies performing EFA reported the extraction method used (n = 164, 86.3%). Principal component analysis was the most commonly used method (n = 91, 37.9%), followed by principal axis factoring (PAF) (n = 42, 17.5%) and the machine learning (ML) method (n = 23, 9.6%). Four studies used multiple extraction methods. Examining the publication dates of the studies reporting extraction methods indicates that most studies using PCA were published prior to the majority of those using FA.
Rotation method criteria. For factor rotation, most of the articles analyzed discussed the rotation method (n = 160, 84.2%). Orthogonal rotation (n = 87) was found to be a more popular method compared with oblique rotation (n = 71). Specifically, varimax rotation (n = 80) was identified as the dominant method among orthogonal rotations, although some studies did not specify which kind of orthogonal rotation they used (n = 7). In oblique rotation, 37 studies employed promax rotation, whereas 11 used direct oblimin rotation. Of the remaining studies, 23 did not explain their oblique rotation method, and seven employed more than one rotation method to achieve ideal factor loading. Although the studies used widely different rotation methods, most did not provide any specific reason for choosing the selected methods. Only 16 studies (6.9%) mentioned their selection criteria.
Factor-retention criteria. While there is no precise solution available for determining the number of factors to retain, researchers have applied several criteria, including the eigenvalue, scree plot, parallel analysis, and Velicer’s minimum average partial (MAP) test.
Of the articles examined in this study, 84.7% (n = 161) mentioned their criteria for how many factors should be extracted. The minimum proportion of variance was, most commonly, accounted for by the factor of eigenvalue (n = 105), followed by retention standard (n = 76), scree plot (n = 59), parallel analysis (n = 11), and the MAP test (n = 1). Three studies used other retention standards.
Item deletion or retention criteria. When using EFA, examining item loading values and cross-loadings on the factors is the most common practice used to determine whether items should be deleted or retained. Most articles reviewed for this study explicated their deletion criteria (91.1%, n = 173); the majority of those studies made decisions based on loadings (n = 134), followed by cross-loadings (n = 88), commonalities (n = 17), and item analysis (n = 9). Fifteen studies used other criteria. Table 2 provides the characteristics of EFA in scale-development studies.
Table 2. Characteristics of exploratory factor analysis employed in scale development studies.
|
Number of Studies (N = 240) |
Percentage |
Criteria Used to Assess Factorability |
|
|
KMO |
39 |
16.3% |
Bartlett’s Test of Sphericity |
41 |
17.1% |
Measures of Sampling Adequacy |
11 |
4.6% |
Not Mentioned |
64 |
26.7% |
Extraction Method |
|
|
Principal Component Analysis |
91 |
37.9% |
Common Factor Analysis |
70 |
29.2% |
Principal Axis Factoring |
42 |
17.5% |
Maximum Likelihood |
23 |
9.6% |
Multiple Extraction Methods |
4 |
1.7% |
Not Reported |
26 |
10.8% |
Rotation Method |
|
|
Orthogonal Rotation |
87 |
36.3% |
Varimax |
80 |
33.3% |
Unspecified Orthogonal |
7 |
2.9% |
Oblique |
71 |
29.6% |
Promax |
37 |
15.4% |
Direct Oblimin |
11 |
4.6% |
Unspecified Oblimin |
23 |
9.6% |
Both Orthogonal and Oblique |
7 |
2.9% |
Not Reported |
30 |
12.5% |
Criteria for Factor Retention |
|
|
Eigenvalues |
105 |
43.8% |
Scree Plot |
59 |
24.6% |
Variance Accounted for by Factor |
76 |
31.7% |
MAP Test |
1 |
0.4% |
Parallel Analysis |
11 |
4.6% |
Unspecified |
29 |
12.1% |
Criteria for Item Retention |
|
|
Loadings |
134 |
55.8% |
Cross loadings |
88 |
36.7% |
Commonalities |
17 |
7.1% |
Item Analysis |
9 |
3.8% |
Others |
15 |
7.9% |
Unspecified |
19 |
7.9% |
4.2. Confirmatory Factor Analysis (CFA)
Structural equation modeling (SEM) versus common factor analysis (FA). Among the studies, 38.8% (n = 93) employed SEM-based CFA, whereas 30.4% (n = 73) only used FA to confirm the final scale (See Table 3). Three types of SEM approaches were mentioned in these articles: single model (n = 80), competing model (n = 53), and nested model (n = 30).
Overall fit. As shown in Table 3, RMSEA (n = 131) was the most frequently reported fit statistic, followed by CFI (n = 130), Chi-square (n = 117), NNF/TLI (n = 63), RMSEA with confidence interval (n = 48), SRMR (n = 58), Chi-square/df ratio (n = 46), GFI (n = 21), NFI (n = 25), AIC (n = 13), AGFI (n = 12), IFI (n = 11), RMR (n = 5), BIC (n = 3), ECVI (n = 1), and RNI (n = 1). Forty percent of studies that employed CFA (n = 75) provided recommended cutoffs from previous research.
Table 3. Characteristics of confirmatory factor analysis employed in scale development studies.
|
Number of Studies (N = 240) |
Percentage |
Confirmatory Approaches |
|
|
SEM |
93 |
38.8% |
Factor Analysis |
73 |
30.4% |
Not Applicable |
74 |
36.6% |
Typical SEM Approaches |
|
|
Single-Model Approach |
80 |
33.3% |
Competing-Model Approach |
53 |
22.1% |
Nested Model Compared |
30 |
12.5% |
Not Reported |
77 |
32.1% |
Overall Model Fit |
|
|
Chi-square |
117 |
48.8% |
Chi-square/df ratio |
46 |
19.2% |
Incremental Fit Indexes |
CFI |
130 |
54.2% |
IFI |
11 |
4.6% |
NFI |
25 |
10.4% |
NNF/TLI |
63 |
26.3% |
RNI |
1 |
0.4% |
Absolute Fit Indexes |
|
|
GFI |
21 |
8.8% |
AGFI |
12 |
5% |
RMSEA |
131 |
54.6% |
RMSEA with Confidence Intervals |
48 |
20% |
RMR |
5 |
2.1% |
SRMR |
58 |
24.2% |
Predictive Indexes |
|
|
AIC |
13 |
5.4% |
BIC |
3 |
1.3% |
Fit Indexes Criteria |
|
|
Recommended Cutoffs |
75 |
31.3% |
4.3. Validity and Reliability
Table 4 presents that various types of validity were discussed, including the following: construct validity (n = 102), discriminant validity (n = 86), convergent validity (n = 84), face validity (n = 56), content validity (n = 55), predictive validity (n = 46), and concurrent validity (n = 37). In terms of reliability, the overwhelming majority of studies reported internal consistency (n = 216), followed by test-retest reliability (n = 9) and alternative-form reliability (n = 2).
4.4. Summary of Key Findings
Our study examined 240 scale development studies across the disciplines of communication, public relations, and advertising. The majority of research articles on scale development were in communication (75.8%), followed by public relations (15%) and advertising (9.2%). Most of the studies (67.1%) clearly defined the constructs being measured, while the remaining 32.9% lacked clarity. Only 25.4% of the studies included expert review of items, and just over half (53.8%) performed pilot testing. Most studies used only one round of data collection, with a few conducting multiple rounds to validate the scale.
The majority of studies employed both exploratory and confirmatory factor
Table 4. Tests of reliabilities and validities.
|
Number of Studies (N = 240) |
Percentage |
The Type of Validity |
|
|
Content Validity |
55 |
22.9% |
Face Validity |
56 |
23.3% |
Concurrent Validity |
37 |
15.4% |
Predictive Validity |
46 |
19.2% |
Construct Validity |
102 |
42.5% |
Convergent Validity |
84 |
35% |
Discriminant Validity |
86 |
35.8% |
The Type of Reliability |
|
|
Test-Retest Reliability |
9 |
3.8% |
Internal Consistency |
216 |
90% |
Alternative Form Reliability |
2 |
0.8% |
analyses (EFA and CFA), with significant growth in these methods since the 1990s. However, the use of principal component analysis (PCA) and factor analysis (FA) remained consistent over time. About 79.1% of studies used EFA, but only 26.7% tested the factorability of the correlation matrix before conducting EFA. Various extraction and rotation methods were employed, with PCA being the most common extraction method and varimax rotation the most common rotation method. Moreover, 38.8% of studies used structural equation modeling (SEM)-based CFA. Different approaches were used to assess the overall model fit, but the most frequently reported fit statistics were RMSEA and CFI. Validity measures such as construct, discriminant, and convergent validity were commonly reported, and internal consistency was the most frequently used reliability measure.
5. Discussion and Conclusion
In the current study, we scrutinized the methodologies prevalent in scale development in the advertising, communication, and public relations literatures. Anchored by the five stages recommended by scale-development authorities, our focus was on the application of EFA and CFA techniques.
The significance of rigorous scale development for advancing academic inquiry is undeniable. We observed a notable intensification in the attention directed toward scale development since the 1990s, marked by the adoption of diverse and sophisticated methodologies. In particular, the last decade has seen a shift toward more refined scale-development techniques in advertising, communication, and public relations studies, including a move away from PCA and toward SEM for confirmatory analysis. Moreover, there has been a considerable increase in the simultaneous use of EFA and CFA, indicating a maturation in the field’s methodological approaches.
In addition to the advancements identified, our analysis revealed areas that are ripe for improvement. Notably, while many studies provided clear construct definitions, a significant portion either neglected this crucial step or addressed it inadequately. This lack of precision can lead to ambiguity in the interpretation of results and undermine the reliability of the scales. Future researchers should emphasize more detailed construct definitions at the outset to ensure consistent measurement across studies. This has broader implications for enhancing the replicability and comparability of research findings in advertising, communication, and public relations.
The gaps in the literature that were highlighted in this study were discrepancies in the application of EFA and CFA, unclear justifications for rotation choices, and inadequate reporting on item retention decisions. The underutilization of CFA, despite its value in refining measurement models after EFA, suggests an area in which methodological strengthening is desirable, as highlighted by Brown (2006). The literature often lacks transparency in EFA processes, overlooking the necessity for a methodological rationale; this critique has been echoed by Pett, Lackey, and Sullivan (2003), who also emphasized the importance of pre-EFA data factorability checks.
The choice between orthogonal and oblique rotations—often made without clear justification—also deserves attention. Especially given the frequent intercorrelations among factors within communication studies, oblique rotation, which allows for inter-factor correlations, appears most suitable for the fields of advertising, communication, and public relations. This would ensure more accurate factor structures and lead to better model fit.
Another significant concern is the insufficient discussion on validity and reliability in the scale development literature. A relatively small proportion of the studies (24%) that we reviewed addressed construct or face validity, underscoring a need for a broader, more comprehensive approach to validating scales beyond quantitative assessments alone. Pett et al. (2003) contended that validity should not be assessed based only on quantitative outcomes. It is essential that items earmarked for measuring a construct be both representative and pertinent. In this vein, content-related validity is a pivotal facet of the broader concept of validity (Cronbach, 1971).
The current findings reveal that only 53.8% of the studies included a pilot test, which leaves room for improvement. Pilot testing is crucial for refining scale items and identifying potential issues before larger-scale data collection. Encouraging researchers to conduct more thorough pilot tests could help avoid premature data collection and ensure the robustness of the scales used.
While this study provides valuable insights into scale development processes and identifies methodological enhancements, it should be acknowledged that its focus is limited to a select set of journals, so there is the potential that emerging insights or alternative perspectives may be omitted. Despite these constraints, this study contributes to the discussion of how rigor and quality are measured in the advertising, communication, and public relations research, and we continue to advocate for a more methodical and transparent approach to scale development.
Although the field has matured, with an increase in the use of EFA and CFA techniques, there is room for further advancement by adopting newer, more sophisticated methods such as parallel analysis or machine learning-based factor extraction. These methods could offer more accurate results and contribute to greater methodological rigor.