The Rasch Model Analysis for Understanding Mathematics Proficiency—a Case Study: Senior High School Sardinian Students

Many students enrolled in Italian Universities don't have regular careers in their first level of university education mainly because of the obstacles encountered in their studies. As far as Mathematics proficiency assessment there are national and international important systematic studies which give evidence for positive relationships between achievement and varied classroom settings and provide a larger context for better understanding regional performance, extending and enriching the local picture. This paper presents some preliminary results aiming to evaluate specific Mathematics abilities in Senior High School Sardin-ian Students approaching university studies. For this purpose the Rasch model was applied. The information obtained by the application of this measurement approach provides clear indication for further analysis to ascertain the causes that influence Mathematics proficiency. The Rasch model was performed on 888 students coming from 28 High Schools located in the central-northern part of the Sardinia region using a questionnaire to evaluate the level of ability in procedural fluency and a second questionnaire to evaluate strategic competence. The study provides more evidence in favor of Rasch Model as an appropriate way for teachers and researchers to obtain richer interpretations on the relationship between stu-dents' proficiency and test items. Based on Infit and Outfit MNSQ, all items are within acceptable range between 0.7-1.3. In light of preliminary results there is a need for local schools and universities to become attuned to the full extent of the Mathematics problem as it affects Senior High School Sardinian Students.


Introduction
This work is part of the wider project STUD.I.O.: "Students in Orientation", funded by the Sardinia Region with participation of the European Social Fund and the Italian Ministry of Labor and Social Policies.One of the main goals of the project is to promote a realignment of general skills (Reading/Writing, Mathematics and Science), aimed at improving the preparation of students attending the last year of High School (19 years old).There is evidence of a serious decline in students' mastery of basic mathematical skills and level of preparation for mathematics-based degree courses since the late nineties (Hawkes & Savage, 2000).This decline is well established and affects students at all levels.Students' difficulties in accessing universities are increasing and it is common to doubt if High Schools are adequately preparing students for university entrance.The admission test scores are becoming alarmingly low, anticipating a trend where students will not be regular with their course studies.This represents a serious problem in the Italian University system especially after the University reform of 2001 which introduced the "3 + 2" system for most of the curricula.The system is articulated in two parts: a first three year course degree corresponding to a first level of university education in a specific field of interest, followed by a two year specialized course degree in the same field.This system which was supposed to cope with the gap in students' preparation entering university education has generated instead a great level of distress in students who try to adapt effectively to the standards required by university studies and prompted a central role for universities to promote specific support to students in difficulty, creating instruments and opportunities to enable them to finish their courses within the allotted time-span.(Chiandotto et al., 2005;Cingano & Cipollone, 2007;Delvecchio & D'Ovidio, 2002;Solinas et al., 2012).Many students enrolled in Italian universities don't have regular careers in their first level of university education, mainly because of the obstacles encountered in their studies due to lack of preparation in general skills, or because of a wrong choice.The withdrawals from university generally occur in the first year of study, representing about 18% of the total students enrolled (MIUR/CNVSU, 2011).Reliable measures of students' performance are needed in order to identify risk factors and predictors of success, also to evaluate the quality and efficiency of curricula.The typical approach in analyzing students' career, centered on the grades and credits acquired is affected by the inherent characteristics of any higher education system setting (Mouw & Khanna, 1983;Rohde & Thompson, 2007).Therefore, it is necessary to provide standardized indicators to make instructional decisions and to evaluate individual student achievement in order to develop strategies to better understand causes and find solutions.As far as Mathematics proficiency assessment, there are national and international important systematic studies which give evidence for positive relationships between achievement and varied classroom settings (Dunne et al., 2012;Mossi et al., 2012) and provide a larger context for better understanding regional performance, extending and enriching the local picture (Samuelsson, 2010;Teddlie & Reynolds, 2000).Among the most important cognitive tests, we would like to mention the international one conducted by the Organization for Economic Cooperation and Development (OECD), started in 1997, known as Programme for International Student Assessment (PISA), and the national one conducted by the Italian Institute for the Educational Evaluation (INVALSI), started in 2007.Both studies generate theoretical reflections and empirical studies related to issues involved in the evaluation of educational programs, educational institutions, educational personnel and student assessment.They involve groups of students enrolled in the second and fifth year of primary school, in the first and third year of Junior High School and in the second year of High School.The statistics are available worldwide and allow different kinds of evaluations to identify and test promising approaches and curricula able to improve student outcomes.Recent indications show that the average level of Mathematic proficiency in a fifteen year old Italian student is lower (score 483) than the OECD (score 496) with a statistically significant difference (PISA, 2009).Based on these premises and under the circumstances of the project STUD.I.O., we decided to extend the study to Senior High School Students (19 years old), who voluntarily participated to the following two evaluations: the first one aimed at measuring if the average level in procedural fluency established by PISA regarding fifteen year old students, was well achieved by the sample under study.In order to do so, we administered a questionnaire composed of specific items similar in difficulty to the one administered by PISA surveys.The second one aimed at measuring the level of strategic competence required to freshman entering any Italian university scientific degree program by CINECA, a non-profit consortium established in 1969, composed of 54 Italian universities, two National Research Centers, and the Ministry of University and Research (MIUR).
The questionnaire administered composed of specific items was constructed in a similar way to the one used in the last past years.In this study, the choice of two questionnaires provide evidence to measure two distinct aspects of Mathematics proficiency in the same group of nineteen year old students graduating from High School, who aim to pursue scientific academic studies.For this purpose the Rasch model was applied.The information obtained by the application of this measurement approach provides clear indication for further analysis to ascertain the causes that influence Mathematics proficiency.

Methodology Data Collection
The sample consists of 1233 Senior High School Students coming from 28 High Schools located in the central-northern part of the Sardinia region (Figure 1), representing about 10% of the entire population of regional nineteen year old High School Students.

Test Instruments
The first test instrument used in this study was a self-developed 20 item questionnaire, constructed according to guidelines provided by OECD curriculum specifications, aiming to evaluate the level of ability in procedural fluency, that is performing mathematical procedures appropriately and efficiently, including those that require decisions in sequence and interpreting and using representations based on different information sources (Kilpatrick et al., 2001).The second test instrument is a self-developed 25 item questionnaire, constructed according to guidelines provided by CINECA, aiming to evaluate the level of strategic competence, connected to operational skills and problem solving strategies.In particular, the ability to recognize and set a problem properly, to select appropriate information, to identify and organize the most appropriate tools to be used and to represent data and situations in an appropriate manner (Kilpatrick et al., 2001).The questionnaires were administered online through the Moodle platform.The first questionnaire was administered in February 2013, the second in April during scholastic activity.

Methods of Statistical Analysis
In this paper instead of Classical Test Theory methods, we used the probabilistic approach of Item Response Theory (IRT) named Rasch Model (Rasch, 1960).We defined Mathematics proficiency as a latent trait yielding a score that locates student ability and question difficulty on a common scale, measured in logits (Fisher, 1995).In this model the probability P i of answering correctly to the i-th item is expressed in function of the difference between the respondent proficiency level θ and the item difficulty level b i as follows: This brings to the definition of logit: Values are placed along a continuum enabling the item difficulty level to be compared with the respondent proficiency level.The Rasch model allows the estimation of these two parameters, with the advantage that despite different test items with different data collection periods, students' progress is estimated and reported on a common scale regardless of fluctuation in test difficulty.The model was implemented in the Acer ConQuest software, version 3.1 (Adams et al., 2012).The Marginal Maximum Likelihood Estimation method was considered in order to estimate the parameters above, through iterative calibration of both person and item (Bock, 1970).Information about the model quality was gained from the fit statistics on item level.These statistics show whether single test items correspond with the model assumptions.Conquest software generates Infit Mean Square (MNSQ) and Outfit MNSQ statistics, which provide indications on the differences between the data and model's expectations (Bond & Fox, 2007;Wright & Masters, 1982).In this study we adopted the range of acceptable fit between 0.7 and 1.3, for both fit indices (Bond & Fox, 2007), because Infit values greater than 1.30 and smaller than 0.70 are labeled problematic (Adams, 2002).Reliability and validity of measures meaning and interpretation are provided within the framework of Rash measurement model.In addition, a Wright Map (Boone & Scantlebury, 2006) is used to visually display the simultaneous distributions (or "performances") of items and respondents related to the instruments used and the sample considered.Redundant items appear on the same or nearly the same point on the Wright map scale.The Wright map also illustrates regions of the scale in which items are absent, identifying where new items are needed.

Results
The 72% of the total sample (n = 1233) matched the specifications required to be part of the statistical analysis: respect of time administration and answer to all of the items of the questionnaire.Table 1, displays the main characteristics of the study sample: 55% of the students come from Lyceum, 28% Technical and 17% Vocational Schools.
As presented in Table 2, the Rasch Analysis applied to the first instrument finds both means of Infit MNSQ and Outfit MNSQ close to the expected value of 1.00.
Inspection with individual items shows Infit MNSQ values ranging from 0.82 to 1.21 while Outfit MNSQ values range from 0.93 to 1.15.Evaluation of the Infit statistics reveals that all cut off points of the items are compliant with the model.The unidimensionality of the construct was assessed by using prin- cipal components analysis of Rasch residuals and item fit statistics (Brentari & Golia, 2010).Because the scores demonstrate little variation from model expectation there is evidence of consistency between students' responses and items on the scale and the model expectations.
Reliability of item difficulty measures is 0.99 suggesting that the ordering of item difficulty is replicable with other comparable sample of students.Consistency of student measures (KR20), equivalent to Cronbach's alpha (Cronbach, 1951), is 0.72 indicating that the ordering of the student proficiency can be likely replicated since most of the variance is attributed to the true variance of the Mathematics proficiency construct.
Based on the following Figure 2, where the student ability (to the left) and the item difficulty (to the right) are displayed graphically with the most able students and most difficult items at the top, it is possible to note that the majority of the students (each x represent 1.3 students) are represented in the center of the graph (the area included between −1 and +1 logits) as most of the items (represented by their labels).The items and the students are quite well approximated by normal distribution.Therefore, the difficulty of the questionnaire is aligned with the average level of student ability, with Item 7 the most difficult and Item 18 the easiest (see Appendix 1).At the bottom and at the top of the chart there are two small groups of students, for which there are no items able to measure their ability.Alarmingly, it seems that the average level of Mathematics proficiency shown by our sample of Senior High School Students is comparable with that of fifteen-year-old students.Let's now proceed to the analysis of the data collected with the second questionnaire which was developed to measure in the sample of Senior High School Students the presence of the strategic competence level required to enter any Italian university scientific degree program.As presented in the following Table 3, the Rasch Analysis finds both means of Infit MNSQ and Outfit MNSQ close to the expected value of 1.00.Inspection with individual items shows Infit MNSQ values ranging from 0.92 to 1.11 while Outfit MNSQ values range from 0.94 to 1.02.As before having assessed the unidimensionality assumption of the construct and the acceptable variation of the scores from model expectation, there is evidence of consistency between students' responses and items on the scale and the model expectations.Reliability of item difficulty measures is 0.82 suggesting that the ordering of item difficulty is quite replicable with other comparable sample of students.Consistency of student measures is moderate (KR20 = 0.51) and mainly this questionnaire resulted for the students' sample composed of items whose

Conclusion
This paper presents some preliminary results of the project STUD.I.O., aiming to evaluate specific Mathematics abilities in Senior High School Sardinian Students approaching university studies.The study provides more evidence in favor of Rasch Model as an appropriate way for teachers and researchers to obtain richer interpretations on the relationship between students' proficiency and test items.Based on Infit and Outfit MNSQ, all items are within acceptable range between 0.7 -1.3.
Both statistics show that there is enough evidence that the data obtained fits the model expectations.With regard to students' procedural fluency investigated with the first instrument most of the students show responses that are within the expectation of the model.The results give suggestions that the items' questionnaire can discriminate students with different Mathematics proficiency levels.Since the measures are in interval scale, one important observation is that the most difficult item of the first questionnaire, Item 7 (1.142logits), was higher in difficulty as compared to Item 18 that was almost twice as easy (−2.266 logits).For the second questionnaire, it is difficult to define the students' strategic competence because of large gaps of the scale in which items are absent.These results provide evidence for construct validity of the second questionnaire.In particular, the item difficulty measures show that this questionnaire is composed of items whose level of difficulty did not correspond to the level of Mathematics proficiency of the students' sample.Therefore, in light of these preliminary results, there is a need for local schools and universities to become attuned to the full extent of the Mathematics problem as it affects Senior High School Sardinian Students.The diagnostic testing of new undergraduates is recommended as an effective means of investigating levels of Mathematics proficiency and identifying actions to improve quality, equity and efficiency of Sardinian educational systems.

Figure 1 .
Figure 1.Map of the Sardinian High Schools participating in the study.

Figure 2 .
Figure 2. Wright map of the first instrument.

Figure 3 .
Figure 3. Wright map of the second instrument.

Table 1 .
Sample characteristics and school typology.