The Effect of a New Placement Process on Student Success in First Semester Calculus

In this work we describe the placement testing and student intake process at a large, land-grant university and show that long-standing general assessment exams administered during secondary schooling are not accurate predictors of success in university level mathematics courses. We then present results indicating that a placement exam administered during arrival as a first-year student provides a strong indicator of potential for student success in first-semester calculus.


Introduction
The student experience in the first year of enrollment at a major university depends a great deal on the quality of the mathematics classes offered.In order to improve retention in first year courses, the Department of Mathematics at West Virginia University has implemented a placement testing process that attempts to diagnose deficits in the mathematics knowledge that students possess as they enter their first mathematics course.Researchers (Rubino, 1998;Pederson, 2004;Parker, 2005) have shown that student performance on mathematics placement examinations in most cases will give good indicators of student retention rates.Others have shown that mathematics interventions (Lesik, 2007) for students who lack in their mathematics preparation will improve the rate at which students return and ultimately graduate.
In 2008 the Department implemented a placement-testing program that uses the MapleTA Placement Testing Suite based on the collection of Basic Algebra and Calculus Readiness tests authored by the Mathematical Association of America (MAA).The highest course a student may place into through the placement test is first semester calculus, Calculus I.A student may also place into college algebra, applied calculus, and a twosemester Calculus I equivalent that incorporate additional precalculus level remediation.

New Student Orientation
Each academic year, West Virginia University admits approximately 5000 new incoming students.These students must then attend a one-day New Student Orientation (NSO) session during the summer before they start attending classes.During NSO, they spend the day familiarizing themselves with the university campus, policies and procedures, and must also register for classes.As a part of this process, they must determine what mathematics course they will need for their intended major and whether or not they have met the prerequisites to enter that course.Prior to the Fall of 2008, students were placed into a WVU Mathematics course based solely on their score on the mathematics portion of either the Scholastic Assessment Test (SAT) or American College Testing (ACT) standardized tests.The SAT and the ACT have long been thought to be valid predictors of student success in college.As such, institutions of higher education in the United States often use students' scores on these tests for admission purposes.However, as early as the 1970s, researchers began to show the predictive ability of SAT scores for determining college grades to be both low and on the decline (Baron & Norman, 1992;Dalton, 1976;Rothstein, 2004).At WVU the success rates of students placed into a mathematics course via SAT or ACT scores prior to 2008 were mixed; approximately 50% of students placed by SAT Mathematics score and 65% of students placed by ACT Mathematics score were successful (with success defined as either an A, B or C in the course).Further, researchers have found other tests can better predict student performance for college level mathematics (Bridgeman, 1982;Kohler, 1973).In fact, the Mathematical Association of America (MAA) first developed its Placement Test Program (PTP) in 1977 in response to growing concerns with mathematics placement problems in colleges and universities (MAA, 2010).In response to issues of student success, placement procedures at WVU changed to include a new testing process in 2008 based on the MAA placement tests.As shown in (Fuller, Deshler, Kuhn, & Squire, 2013), this process has had significant impacts on student success in lower level courses.

The MAA Placement Testing Suite
In the summer of 2008, a placement process was implemented at WVU derived from a set of exams created by the MAA.This placement exam is implemented online via Maplesoft's MapleTA system in two parts; Basic Algebra (BA) and Calculus Readiness (CR).Each part has 25 multiple-choice questions and students are allowed 30 minutes for completion.To enter our traditional Calculus I coursea student must get at least 17 questions correct on the BA portion, and a total of at least 33 total questions correct.Questions range from computational exercises that test a student's algebra skills as shown in Figure 1 to questions that are more conceptual and test a student's ability to reason from a definition as shown in Figure 2.
Students may take the exam a maximum of two times and their highest scores are used.Students may also be placed into a remedial pre-college algebra workshop if their scores are extremely low.

Students Entering Calculus
As indicated by Table 1, approximately 13% of students entering a mathematics course at WVU each year enter Calculus I. Another 17% enter applied calculus or our two-semester calculus course, and the majority, about 40%, enter a college algebra.Another 30% or so enter a remedial pre-college algebra workshop course.

Entry Paths
Students may qualify for Calculus I in several ways.Since 2008, 43.4% of our students who enter this course do so by the placement test.Students may also qualify based on successful completion of college algebra and trigonometry at WVU (WVU Course), successful completion of college algebra and trigonometry at another institution (Transfer Course), using advance placement scores (AP), or via some other process (Other, e.g.instructor approval, etc.).
Prior to 2008, high enough scores on either the SAT (SAT Math) or ACT (ACT Math) Mathematics portions were used to place students into Calculus I.For the ACT Mathematics portion, a score of 25 or higher was needed.For the SAT Mathematics portion, a score of 600 or higher was required.In 2008 these SAT/ACT placement cutoffs were suspended but existing WVU students satisfying these requirements were allowed to enter Calculus I using the prior SAT/ACT requirements.Students using the SAT/ACT Mathematics placement criteria are shown in the pre-2008 and post-2008 results in Tables 2 and 3

Student Performance
Student performance is measured in this work as success, failure or withdrawal.We define success to be an assigned letter grade of either A, B or C. Failure would be an assigned letter grade of D or F, as well as an incomplete or other indication of non-completion of the course without withdrawal.Students withdrawing from the course before a certain date and not receiving either credit or a penalty in their GPA are assigned a W.

Pre-2008 vs. Post 2008
As mentioned previously, placement testing was implemented in the Fall of 2008.Student success rates in calculus prior to Fall of 2008 varied but overall were declining.As shown in Table 2, success rates averaged around 49.9% over the period from 2002-2008.Comparatively, the student performance after the Fall of 2008 is remarkably different.As shown by Table 3, success rates have increased to 61.3%.It is reasonable then to attempt to ascertain underlying correlations that might suggest influences that led to this increase.

Success by Entry Path
Tables 5 and 6  Students who enter Calculus I via the Placement testing process succeed at a rate of 76.2%.This is 15 percentage points higher than the overall current success rate of 61.3%, and 26 percentage points higher than the 49.9% success rate prior to 2008.
After 2008, a number of students using the SAT/ACT Mathematics placement criteria dropped significantly, but there were enough students to provide a valid comparison with both the prior placement process and with the success rates of the new placement tested students.If we recode student success (A, B or C) as a 1, failure (D or F) as a −1 and withdrawal (W) as a 0, we have the correlation matrix in Table 7.
The underlying correlations of success with any of the three placement scores are low, but we see that the new placement test score (.319) is higher than both ACT Mathematics (.279) and SAT Mathematics (.254) as an indicator of success in the course.We see that much higher average scores are observed in both cases for successful students than the previous cutoffs that were used prior to 2008.

Conclusion
We anticipate the current placement process to provide better placement for all students placing into a WVU undergraduate mathematics courses, and believe the data indicates somewhat conclusively that this is the case for our mainstream Calculus I course.However, appropriate placement is only one part of a more complete approach to ensuring student success that includes implementing best teaching practices, appropriate assessment and feedback techniques and teaching for understanding.In particular, assessment of student performance in prerequisite courses such as college algebra and the subsequent performance of these students in calculus, as well as the performance of student in successor courses such as Calculus II, Calculus III and other upper division courses are also essential elements of a properly functioning placement process.We believe the change in placement at our institution is the first step to encourage as many of our 29,000 students as possible to succeed in mathematics and persist in the study of mathematics and science.Further work will be described in subsequent publications relating to student performance in precursor and successor courses, as well as more refined analyses of student performance for demographic subsets.

SAT and ACT Math Scores of Successful Students
Given the success rates observed for prior SAT and ACT math cutoffs, it is reasonable then to ask what scores do correlate with success in calculus for these tests.Table 8 shows the average ACT mathematics score and average SAT mathematics score for students who in the first row obtained an A in the Calculus I.In the second row, the average scores for students obtaining an A, B or C are shown.We further break these down by entry via placement or using credit for college algebra as a previous course.
In this data we see that the average score of successful students overall is 1.47 points higher than the original ACT cutoff and almost exactly the same for the SAT cutoff.After subsetting by entry path, we see, however, that students entering via prior coursework skew this average downwards, and that stu-dents entering via placement show ACT math averages a full 3.62 points higher and SAT math averages 34.67 points higher.Both of these are close to a standard deviation above the prior cutoffs, suggesting that these earlier placement criteria were too low for students without prior coursework, and that the new placement process is more effective.Further work needs to be done to see if alternative cutoffs for SAT and ACT scores could be used.
. The percentages of students entering Calculus I via different paths from 2008-2012 are shown in Table4.
show student outcomes by entry path from Fall 2002 to Spring 2008 and for Fall 2008 to Fall 2012, respectively.Note that students entering Calculus I via the SAT Mathe-matics and ACT Mathematics placement requirements had mixed results prior to 2008.In particular, the overall success rate of students in this time period placed via ACT Mathematics scores, 56.2%, and the rate for students placed via SAT Mathematics scores, 48.6%, are shown to be close to the overall success rate of the course, 49.9%.

Table 7 .
ACT Math, SAT Math and Placement Score Correlations

Table 8 .
SAT and ACT math score averages by grade in calculus I.