Validation of the Questionnaire of Students ’ Attitudes toward STEM-PBL : Can Students ’ Attitude toward STEM-PBL Predict their Academic Achievement ?

The interdisciplinary curriculum of science, technology, engineering, math, and project-based learning (STEM-PBL) has currently received deep contemplation for its diverse benefits. This major attention has been focused on optimizing the curriculum by developing tools for assessing the diverse aspects of STEM in relation to teaching methods (e.g., PBL, teacher-centered learning, and game-based teaching), and students’ attitude. Following the principle of fallibility, the current research examined psychometric properties of the scale of students’ attitudes toward STEM-PBL. Also, the power of predictability of academic achievement by means of students’ attitude, and the mediating role of demographic factors were aimed to explore. The scale was completed by 337 students who were selected randomly. Using confirmatory factor analysis, Graded Response Modeling (GRM) based on Item Response Theory (IRT) provided technical informative details about each item of scale. Examining the items of scale displays reasonable psychometric properties including the validity of Student Attitude toward STEM-PBL Questionnaire. However, the analysis revealed that the removal of one of the items (item 22) may result in a better fit of the questionnaire. The results also showed that attitude toward STEM-PBL can predict students’ academic achievement using grade point average (GPA). Exploring the role of some mediating and demographic factors in this prediction revealed that self-learning had a high significant unique contribution in predicting GPA among respondents. The results displayed the significant association between GPA and students’ father education level. Also, students whose GPA and their fathers’ educational levels were high showed more positive attitude toward self-learning. Furthermore, How to cite this paper: Shojaee, M., Cui, Y., Shahidi, M., & Zhang, X. X. (2019). Validation of the Questionnaire of Students’ Attitudes toward STEM-PBL: Can Students’ Attitude toward STEM-PBL Predict their Academic Achievement? Psychology, 10, 213-234. https://doi.org/10.4236/psych.2019.102017 Received: December 19, 2018 Accepted: February 9, 2019 Published: February 12, 2019 Copyright © 2019 by author(s) and Scientific Research Publishing Inc. This work is licensed under the Creative Commons Attribution International License (CC BY 4.0). http://creativecommons.org/licenses/by/4.0/ Open Access

1) Diverse Teaching Methods: Regarding the first research trend, different teaching strategies for STEM education such as teacher-centered learning (TCL), game-based teaching or learning (GBL, Hoge, 2013), problem-based learning, and project-based learning (PBL) have been studied.Of these teaching strategies, project-based learning (PBL) has been recommended for STEM by the reason of being authentic, cooperative, feasible, and motivational (Wallace & Webb, 2016;Shahin & Top, 2015).Thus, the combination of STEM-PBL refers to a type of science, technology, engineering, and mathematics education in which PBL has been used as a teaching strategy.STEM-PBL is considered as an interdisciplinary education in which students are encouraged to complete some projects with respect to integrating knowledge, experiences, and problem-solving methods in a real world through which they enhance student involvement, engagement, cooperation, and motivation (Shahin & Top, 2015).PBL, similar to problem-based learning, is sometimes considered M. Shojaee et al. DOI: 10.4236/psych.2019.102017215 Psychology as a learner-centred approach and is defined as an instructional method that focuses on higher-order learning in which students endeavor to resolve ill-structured or ill-defined projects by combining knowledge and authentic experiences in a real-world setting (Capraro & Corlu, 2013;Capraro & Jones, 2013;Tawfik & Trueman, 2015).Capraro et al. (2013) argued that PBL is a model for class activity in which teachers are just facilitators and practices are rooted in a real-world context and designed in the frame of projects for long-term learning.
The application of PBL for STEM programs reveals that PBL's common features such as higher level of instructional feedback, inquiring strategies, integrated subject areas, students' discussions, and peer and self-assessment lead students to acquire the higher level of learning and positive attitude toward subjects (Hall & Miro, 2016).Tseng and his colleagues (2013) found that STEM-PBL can affect students' academic performance positively.Using a pre and post-test method (Tseng et al., 2013), they concluded that using PBL for STEM education not only increased meaningful learning, it also enhanced students' tendency for future careers since STEM courses integrated knowledge, concepts, and skills systematically.One of the important findings of their study was that PBL was more effective in changing students' attitude toward engineering rather than mathematics.This reveals that students who participate in STEM-PBL education may have the propensity to choose science and mathematics courses in their final years of high school.
Other findings indicated that students learn better when they are authentically involved in meaningful activities and are engaged in a real-world problem, which makes the learning context more relevant for them (Stearns, Morgan, Capraro, & Capraro, 2012).In their study, Karaman andCelik (2008 cited in Tseng, Chang, Lou, &Chen, 2013), demonstrated that those students participated in PBL classes performed better in skill development, knowledge retention and general abilities comparing with those were not in PBL classes.Students also acquire different skills such as collaborative, technical, problem-solving, and leadership skills that enhance their positive attitude toward STEM.These characteristics are more effective when students are provided with diverse, ill-defined, nonroutine, and authentic tasks in the frame of projects (Han, 2017).
However, diverse teaching strategies for STEM education have been used in educational curriculum and PBL as a teaching strategy in STEM education is Psychology seemingly novel (Asghar, Ellington, Rice, Johnson, & Prime, 2012;Duke, Halvorsen, & Strachan, 2016).Thus, its characteristics and applicability through assessing students' perspectives (attitude) toward STEM need to be more researched.
2) Students' Attitude toward STEM-PBL: Students' attitude toward STEM-PBL refers to the way students think and feel about the STEM and the way they involved themselves in the process of learning through PBL.This phenomenon is characterized by three major cognitive, affective and behavioral components that are organized toward STEM.Attitude in this regard reflects how students evaluate learning subjects and environments, and how they react affectively to such learning (Han et al., 2014).Generally, attitude acts through the ABC model (Veresova & Dana Mala, 2016); that is, when attitude is positive, a) students have more positive thoughts and opinions about the subjects (cognitive element), b) they feel about the subjects enthusiastically (affective element), and c) show inner tendency to be involved and engaged fruitfully (behavioral element).It is assumed that positive attitudes increase students' academic achievement.
Studying the impact of positive attitude on different types of phenomena such as school connectedness, school environment, and academic success resulted in different range of findings (Henning, 2013;Veresova & Dana Mala, 2016).
However, there are few studies regarding the impact of students' attitude on STEM and its relationship with academic achievement (Han et al., 2014, Han, 2017).Han (2017) demonstrated that students who had positive attitude toward STEM-PBL components showed increased tendency to pursue STEM-PBL.Such findings were reported by previous studies (Han et al., 2014;Sahin, 2013).In Turky, researchers demonstrated that STEM-PBL has effective capacities to be an integrated model to promote innovations (Corlu, Capraro, & Capraro, 2014).
The major factor in reliable and valid assessing students' attitude toward STEM-PBL is using an appropriate tool since there have been different scales and questionnaires designed to assess students' attitude toward education, school, learning, or science such as the Questionnaire of Attitude Toward School and Learning (Veresova & Dana Mala, 2016), Relevance of Science Education Questionnaire (ROSEQ) (Schreiner & Sjoberg, 2004;Oscarsson, Jidesjo, Karlsson, & Stromdahl, 2009), Student Academic Attitude Survey (SAAS, Henning, 2013) and other scales.
However, the Questionnaire of Students' Attitude toward STEM-PBL (Han et al., 2014;Han, 2017) with remarkable validity and reliability is a scale that designed specifically for assessing students' attitudes toward STEM-PBL.According to Han and Carpenter (2014) 1).
Additionally, it is demonstrated that this scale can reflect the multidisciplinary characteristics of STEM-PBL (Han et al., 2014;Han, 2017).Although it is postulated by Han et al., 2014;Han, 2017 that  Psychology not only for their academic achievement, but also for educational policy makers.
Also, educational literature supported such roles for students' attitude as it may influence their learning style, which is the way they adopt and respond to not only to instructional program, but also to instructional strategies in schools (Han & Carpenter, 2014).
Moreover, studies in Iran revealed that students in traditional classroom in which PBL is not prominent have less desire to study math (Najafi et al., 2012).
This issue may have produced negative attitude that influences their direction of study.Therefore, the current study was designed not only to verify whether the Han et al., 2014;Han, 2017 questionnaire is valid and reliable in Iranian educational context, it also focused on the degree to which students' academic achievement can be predicted through their attitudes toward STEM-PBL.Also, we aimed to examine some major demographic features of the students such as gender, parents' educational levels and others.To reach these purposes the following research questions were designed:

Participants
Three secondary Iranian schools from urban area in Alborz province were randomly selected and 337 middle-grade students responded to the questionnaire.
Of 337 students, 232 were females (68.8%) and 105 were males (31.2%), with the age mean of 14.25 (SD = 1.013).Since the participants of this study included middle-grade students, who are typically about 13 -14 years of age and below the age of consent, parental or guardian consent was also needed before students could participate in the study.One week before the study was administered, parental/guardian consent forms were sent home with students so that approval could be received in time for students to participate in the study.The authors explained for students that participating in the study is completely voluntary.All these students have been working on different projects collaboratively or individually in their classes and school labs in which they have integrated their knowledge of science, technology, engineering, and mathematics.

Ethics
Ethical protocols were undertaken for the school board identified by the author as potential candidates for inclusion in the research.

Measures
The questionnaire of students' attitudes toward STEM-PBL was a 25 Likert-formatted items to measure student attitude toward varied aspects of STEM-PBL.The scale has five sections including student self-regulated learning, collaborative learning environment, interdisciplinary learning environment, technology-based learning and hands-on activity (Han & Carpenter, 2014;Najafi et al., 2012).Figure 1 illustrates the features of the STEM-PBL as were explained in the previous section.
To prepare the scale for this study, translation and back translation was independently conducted by two expert reviewers; both were fluently bilingual in English and Persian language.Following confirmation of translation accuracy, the scale was administrated, and participants were asked to choose one of the options in 5-point scale, ranging from strongly disagree to strongly agree, to show their attitude for each item.Most of the participants were able to complete the questionnaire within 15 minutes.
Besides the above-mentioned instrument, a demographic survey (See appendix)was administered that had ten items, regarding their gender, age, GPA, parents' education, and also some information about their interests in technology and the amount of time that they usually spend in using different types of technology.

Procedure
The collected data were explored for the basic statistical assumptions such as In addition, to validate the STEM-PBL questionnaire the researchers focused on item analysis using R software version 3.4.1.The multidimensional form of Graded Response Modeling (GRM; Samejima, 1969;Ackerman et al., 2003;Bartolucci, Bacci, & Gnaldi, 2012) was utilized in order to find the relationship between item responses and the latent trait.GRM is one of well-developed IRT models designed for polytomous graded responses (Jansen & Roskam, 1986 as cited in Rubio et al., 2007;Muraki & Carlson, 1993), and it also works well when the items in a test have different levels of discrimination.The goal of the GRM is to model the relationship between item responses and the latent trait when the ratings include two or more ordered categories (Muraki & Carlson, 1993).
Cronbach Alpha was used for internal consistency of all questions.Also, the Psychology bivariate correlation was used to determine the degree of association between students' attitude toward STEM-PBL and their key demographic variables.
Through two-steps sequential multiple regression, the degree of predicting role of students' attitude was also examined.In the first step, demographic variables were included in the model to predict GPA and in the second step attitudes variables were added to see whether students' attitudes toward STEM-PBL can predict above and beyond demographics,

Results
Descriptive information for the demographic variables is shown in

1) Confirmatory Factor Analysis of STEM-PBL Questionnaire
To answer the first research question regarding the underlying factors in the questionnaire, a confirmatory factor analysis was employed by using AMOS 18.0.0.The results from AMOS showed that the five factors of STEM-PBL questionnaire fit well with our data.However the analyses revealed that the removal of one of the items (item 22) may result in a better fit of the questionnaire (see Psychology 200 this index is reliable to use and upper than this range, because of impact of the sample size and also the amount of exitance correlations in the model, the chi square will be significant anyway.Thus, considering other indices would be more reliable to check the model fit examination. As Table 2 shows, each of the variables or components has a significant correlation with the total of the scale showing an acceptable internal reliability, range from 0.606 to 0.730 at 0.001 significant level.The Cronbach's Alpha as the reliability of the scale for 25 items was high (α = 0.803).The correlations among factors together were significant but not very high ranging from 0.145 to 0.443.
This result suggested that the factors are correlated in general, but each factor is associated with some uniqueness.
2) Item Analysis Dimensionality of STEM-PBL Questionnaire: Since factor analysis indicated 5 dimensions, for this instrument, we conducted a multidimensional IRT analysis for this data.The analysis was completed with MIRT package (Chalmers, Pritikin, Robitzsch, & Zoltak, 2015) in R. Results indicated that the proportion variance explained by these five factors was 30% of the total variability.Also, the correlations between five factors were ranged from 0.145 to 0.730 indicating a reasonable and significant association between the factors (Tabachnick & Fidell, 2007).
Parameterization: Item parameters of the STEM-PBL questionnaire were shown in Table 3. Regarding item discriminations (a-values), all the items displayed a very rational level of discriminant ranging from 0.471 (for item 6) to 1.79 (for item 9).
Regarding the item difficulties (b-values), there are four thresholds or boundaries (b-values) in IRT for each item with 5-point Likert format (see Figure 3 for item 25).Also, in the psychological instruments, the extreme bi1 shows the least amount of the trait needed for choosing first option, and bi4 indicates the maximum level of the trait needed to move from pervious category to last category (Shojaee, Bulut, & Shahidi, 2016).For example, for question 25, to choose the category 1 a maximum trait level (5.205) with the probability of 0.5 is required; whereas, approximately a low trait level (−0.598) with the probability of 0.5 is needed to choose the category 4, the lowest level of the trait.
Model fit: Inspecting whether GRM is an appropriate model for this type of  data, accuracy of person parameters, expected total score diagram and the standard error of measurement were examined.In regard to person parameter (ability for participants), theta θ for all 337 participants along with their standard measurement error was computed.The theta values ranged from −2.91 to 2.21 with the standard error of 0.31 for the lowest and 0.72 for the highest theta.Statistically, this range represents a normal distribution of the students' attitude.
Thus, the person parameters supported this idea that most of the participants are in the mid-level of the trait distribution, also the level of the standard errors were reasonable.Additionally, since measuring factors are correlated to each other, this multidimensional data is categorized in compensatory model.The model indicates that if an examinee has high ability in one dimension, this high ability may compensate his or her low ability in other dimensions.Supporting this idea, the "expected total score diagram" was depicted, and it displayed that people with low level of one factor tend to choose the options (categories) with the low loading of other factors (see Figure 4).
The diagram of "Test Standard Error" displayed that when there are not enough responses in a specific level of theta, the level of error goes up.That is, there are very small amount of error in estimating the low and moderate level of positive attitude, and there is considerable amount of error to estimate the higher degree of the trait.Therefore, the precision of estimation tends to be low (De Champlain, 2010).Moreover, comparing the plot of Test Information and simple SEM plot supports the abovementioned idea.This plot showed that most gained information come from about middle level of the trait, which is between −3 and +3, and the lowest level of measurement error occur in this interval (see Figure 5).

Predicting GPA through Students' Attitude toward STEM-PBL and Demographic Variables
To examine the degree of GPA prediction via the students' attitude toward STEM-PBL, a two-step sequential linear regression analysis was run.In the first step, demographic variables were included in the model to predict GPA and in the second step students' attitudes variables were added to see whether attitudes can predict above and beyond demographics variables (see Table 4).

Investigation of Demographic Factors
The correlation coefficients between 5 factors of the STEM-PBL and demographic variables indicated that there are some relationships between these variables (see Table 5).Self-Learning as an important variable in this study has a direct correlation with students' GPA and students' father education.That is, the students who have a higher GPA and their father have a higher educational level, their attitude regarding self-learning is more positive.The plot of students' means of self-learning confirmed the results (Figure 6 and Figure 7).This factor, "self-Learning", has indirect correlation with the students age, and the number of hours that they spend using technology.This study also showed that the girls have more positive attitude about self-learning and older students have lower self-learning mark.The last thing that worth to mention about this factor is that students with the higher hours spending for technology have less tendency regarding the self-learning (Figure 8).
In addition, students' attitude towards "Hands-on Activity" has indirect significant correlation with number of hours that they spend on technology (Figure 9).That is, as number of technology hours increases, their attitudes towards hands-on activity decrease.Other components did not show any significant relationships with different demographic variables.

Discussion
This study is the first study in Iran to examine the validity and psychometric    were matched with the new sample, the findings of this replication was in the line of other previous studies (Han et al., 2016;Han et al., 2014;Shahin & Top, 2015;Lim, Cha, & Noh, 2001).However, the result of confirmatory factor analysis through using AMOS showed that item 22 of the questionnaire should be removed or reconsidered from the list of questions (Table 3 & Figure 2).Moreover, this study reveals the higher reliability for all components of Student Attitude toward STEM-PBL Questionnaire, which aligns with other previous studies (Han et al., 2016;Han et al., 2014;Han, 2017;Shahin & Top, 2015).Using IRT-GRM to analyze data provided the researchers with more technical information regarding the psychometric properties of each item.All questionnaire items display reasonable psychometric properties, which further support the use of Student Attitude toward STEM-PBL Questionnaire.
In this study, the second research question was focused on the prediction of students' GPA by means of the components of students' attitude toward STEM-PBL.GPA represents the degree of academic performance to which a student has accomplished specific goals of learning such as the acquisition of knowledge and understanding in a specific intellectual domain which generally called discipline (Steinmayr, Meissner, Weidinger, & Wirthwein, 2015).That also refers to the degree of learning or a specific level of expertise in different areas of inquiry or research that was accomplished through formal education at schools, colleges, or universities (Corsini, 2002;Cigdem, 2015;Hamaideh & Hamdan-Mansour, 2014).The degree of prediction of GPA through using student's attitude towards STEM-PBL can reveal the effectiveness of STEM-PBL in students' academic achievements.Thus, considering GPA as an index of academic progression and learning, the current study also showed that attitude toward STEM-PBL can predict students' academic achievement using GPA.This result was derived from a regression analysis, and the finding aligns with Han's (2017) study in which students that had positive attitude toward STEM-PBL components showed increased tendency to pursue their learning and also, with Tseng et al.'s, (2013) study in which more academic progression in STEM-PBL was reported.However, the role of demographic variables in predicting GPA was prominent too.
The third question was focused on the relationships among five factors of the STEM-PBL and student demographic variables including age, GPA, mother's education, father's education, and the hours using technology.Of all five components of STEM-PBL, self-Learning has a direct association with students' GPA and students' father education.It means that the students who have a higher GPA and their father have a higher educational level, their attitude regarding self-learning is more positive.This factor, self-Learning, has indirect correlation with students age, and the number of hours that they spend using technology (Figure 8).One of the important findings of this research is about the relationship between the time of using technology and hands-on activity as a component of the students' attitude toward STEM-PBL.The result showed that this rela-Psychology tionship is indirect.That is, the more time in using technology such as computer, cell phone or others, the more negative attitude toward hands-on activity in STEM-PBL (Figure 9).

Conclusion
Generally, the study not only corroborated the beneficial role of STEM-PBL as an interdisciplinary curriculum, but also revealed that the Questionnaire of the Student Attitude toward STEM-PBL has reasonable reliability to use in Iranian samples.However, the question 22 in this questionnaire should be revised and refined to fit the scale.Also, the power of this questionnaire to predict students' academic achievement increased the applicability of the scale in different learning settings; hence, researchers can confidently utilize the questionnaire for assessing students' academic progression.The study displayed that fathers' educational levels have direct impact on students' academic performance.Unlike previous studies, the current research revealed that spending more time in using technology such as cell phone will increase more negative attitude toward applied activities in STEM-PBL.

Implications
The findings of this research provide evidence of using a reliable and valid scale for assessing students' attitude toward STEM-PBL in Iranian student population.
Additionally, the results in this paper can be used in educational settings to monitor students' attitude and offer proper interventions to enhance academic achievement.
Regarding our finding related to indirect relationship between students' attitude towards hands on activity and the number of hours using technology, it is recommended that educators, educational policy makers, and even families encourage students to balance technology use and activities that promote students to be active and involve.
a) What are the validity evidences of the Questionnaire of Students' Attitudes toward STEM-PBL (Han & Carpenter, 2014)?b) To what degree students' attitudes toward the five components of STEM-PBL can predict their academic achievement measured by grade point average (GPA)?c) Are there any relationships of the five components of students' attitudes toward of STEM-PBL and with student age, parents' education, and amount of time that they spend on technology?
normality.Then, to acquire the confirmation regarding the factors, which Han and Carpenter had found in 2014 and the position of each item in relation to above noted factors in the questionnaire, a confirmatory factor analysis using the Structural Equation Modeling through AMOS software 18.0.0version used.

Figure 2 )
Figure2).Since discrimination of the item 22 for factor 5 is negative, −0.563, and also, difficulty level of this item for all four choices are too low, 0.955, −0.239, −0.959, −2.071, indicated that this item does not contribute enough information to the scale.Several indicators of fit were used to test the goodness of fit of the model including Tucker-Lewis index (TLI = 0.888), comparative fit index (CFI = 0.911), root mean square error of approximate (RMSEA = 0.046), goodness of fit index (GFI = 0.911).Considering these fit indices were all within acceptable range, we concluded the model is fit for this data set(Ghasemi, 2011).Even though chi-square (χ 2 = 410.289,p > 0.000, df = 238) is significant and not in the acceptable range, according toHooman (2012) when sample size is between 75 to

Figure 3 .
Figure 3. Item Response Category Characteristic Curves of Item 25 in the STEM-PBL Questionnaire: The curves in black, red, green, blue, and aqua blue are item difficulties for boundaries of choices from one to five respectively.

Figure 4 .
Figure 4. Expected total score for all participants.

Figure 5 .
Figure 5. Test information function (the blue line) and the standard error of measurement (the red line) in KADS-11.

Figure 6 .
Figure 6.Relationship between GPA and self-learning as a component of students' attitude toward STEM-PBL.

Figure 7 .
Figure 7. Relationship between fathers' education and self-learning as a component of students' attitude toward STEM-PBL.

Figure 9 .
Figure 9. Relationship between use of technology and hands on activity as a component of students' attitude toward STEM-PBL.
students from schools within that district were invited to participate.The ethical protocols of particular interest to school district ethic board were the following: informed consent letters for both students and their parents or guardians to indicate voluntary involvement in the research.
By the time data collection was scheduled to start, school board had approved this study.Hence principals, M. Shojaee et al.DOI: 10.4236/psych.2019.102017219 Psychology teachers, and

Table 1 .
Female students who participated in this study were 232 (68%), whereas males were 105 (32%).Students' GPA were categorized into five categories from very low to very high, which more than half of the students had high and very high GPA.Parents' education was categorized into four categories, including under diploma, diploma, bachelor, and above bachelor.Close to 50% of parents' education was at diploma level.Finally, the amount of time that students spend on using technology during a day were categorized into three categories, one hour or less, two to three hours, and 4 hours or more.

Table 2 .
Means, standard deviations, correlations, and reliabilities of study variables.

Table 3 .
Estimated item parameters of STEM-PBL questionnaire.