Efficacy of an Online Reading Intervention Program “Literacy Pro” in Classifying Students as “At Risk” and “Not at Risk” among ESL School Students—A Logistic Regression Analysis ()
1. Introduction
Reading Literacy is a skill, which is the foundation of almost all processes of learning and is necessary for students not only to acquire languages and study literature, but also to learn other subjects (Geske & Ozola, 2009). Mullis, Kennedy, Martin and Sainsbury (2004) have defined “Reading Literacy” as “the ability to understand and use those written language forms required by society and/or valued by the individuals” (p. 3). It has been surmised in a study by Lea and Jones (2011) that, reading has an integral role with respect to the choices that students make with the textual resources available to them. This idea is further reinforced by Calhoon (2005) in his study; “Reading ability is a fundamental skill on which academic success, secure employment and personal autonomy depend” (p. 424). Reutzel and Cooter (2004) too contend that the primary goal of any comprehensive reading program ought to be to transform students into independent and fluent readers who continue to fine-tune their literacy skills throughout their schooling. Research by Kern (1989) says that reading in any language is cognitively demanding and reading in a second language tends to put greater stress on the reader. It has been surmised that students who are exposed to a variety of reading texts are seen to develop critical reading skills; eventually, they also develop “independent thinking and skills in analysis and judgement” (White, 2004: p. 42).
2. Online Reading Literacy Program
In the words of Taylor and Ward (1998) technology has created a new “educational space”. The reading path provided by web-based programs is quite different from the traditional single channel; it is now a self-designed, non-linear trajectory (Al-Shehri & Gitsaki, 2010). A web-based reading program is the facility to provide specific content and a customized program that focuses on both instruction and assessment through real time reporting; it has the facility to provide instant feedback and remediation. Thus, a “Web-based instruction provides an active learning environment that epitomizes learning that is student-centered, interactive, exploratory, contextualized, intentional, reflective and collaborative” (Cole & Hilliard, 2006: p. 365).
Reading on the web is a dynamic exercise as readers may be directed to multiple reading paths and may be offered many interesting choices of activities to enhance their reading experience through links and hyper-links (Schmar-Dobler, 2003). The visual and audio multimedia elements integrated into the text also add to the appeal of such programs (Coiro, 2003). Park and Kim (2011) have also commented on the vivid experience provided by the multi-media elements in web-based reading programs. The richness of such resources provided online has been noted by Massey (2014). The interactive feature of web-based programs enables to stimulate and sustain motivation which is inherent to the learning process (Palmer, 2006). Research by Goodfellow and Lea (2005) also stresses on the role of web-based programs in enhancing motivation among learners. This view is also supported by Eilon and Kliachko (2004) who have recorded that technology driven learning environment sustains motivation of a learner by constantly challenging their thinking through tasks suitable to their skill level. A computer mediated environment keeps up the motivation level in learners. Thus, researchers like Harrison (2009) believe that technology provides a learning system that embeds reading strategies that enable learners to improve their reading literacy skills.
Sadik (2008) articulates that through online technology integration, learning becomes a more pleasurable experience as student learn within a social context as well as provided the opportunity to create knowledge as they go along. This idea has been well summarized by Jones (2001), when he says that acquisition of literacy skills is “a fluid process”, one that takes places seamlessly in a social context where students are not passive learners, but instead active participants in the construction of knowledge. Hence, online reading literacy programs empower students as they take control of their own learning and makes them more accountable as well. This accountability also manifests as intrinsic motivation which fuels greater task involvement and eventually learning achievement (Chun-Min & Thomas, 2007). Similar ideas are echoed in a study by Palmer (2006) who says that learning becomes meaningful when learners are actively involved in the process. Similarly, Perlman, Weston and Gisel (2010) too have found that an increased sense of ownership and responsibility is observed among learners engaged in web-based learning. Lamb and Johnson (2010) also assert that student skills in a variety of curricula areas would show improvement through interactive learning environments provided on the web.
It has also been observed by many researchers that online reading literacy programs facilitate effective intervention for struggling readers in the form of personalized instruction (Englert, Manalo, & Zhao, 2004). By virtue of their ability to provide high engagement levels, web-based programs are deemed an effective tool for providing additional reading practice to students identified as at risk of reading failure (Smith & Throne, 2007). Besides, as mentioned by Littleton, Wood and Chera (2006), online reading programs not only allow students the liberty to work at their own pace but also provide them with instant feedback. Fasting and Lyster (2005) too have found benefits for struggling readers on technology driven reading platforms.
Sternberg, Kaplan and Borck (2007) have emphasized the importance of professional development for teachers to establish a culture of literacy within an institution. A similar view was shared by Topping (1999) regarding the crucial role played by teachers in integrating technology into the curriculum to enhance learning. Research conducted by Bishop and Santoro (2006) concludes that the media capabilities of modern technology aid in building vocabulary and enhancing vocabulary skills. Cole and Hilliard (2006) also expound the role of web-based learning in enhancing basic reading skills among students. There is ample research that vouches for the effectiveness of web-based programs if implemented appropriately (Englert, Zhao, Collings, & Romig, 2005; Macaruso & Walker, 2008). The extent of one’s comprehension to a large extent depends on one’s ability to decipher vocabulary in a given text (David, 2010). Thus, as emphasized by Harmon, Wood, Hedrick, Vintinner and Willeford (2009), an effective literacy program ought to devote great attention to the learning of vocabulary.
There have also been studies that acknowledge the negative aspects of online literacy. For instance, Herold (2014) surmises that onscreen reading prompts readers to go no further than skimming the text; thus, resulting in reduced comprehension. However, researchers like Yagci (2014) contend that digital native learners who have grown up in the digital age, best respond to the digital medium and process data differently as well. Inceçay (2013) speaks on similar lines and says that digital native readers employ a new set of cognitive and meta-cognitive strategies to comprehend reading material.
The focus of web-based programs must focus on the five core areas of literacy learning: Phonemic awareness, phonics, fluency, vocabulary and comprehension (Eunice Kennedy Shriver National Institute of Child Health and Human Development, 2001). In their research Biancarosa and Snow (2004) acknowledge that technology can certainly enhance traditional modes of instruction; however, a careful needs analysis is to be conducted to align student skill to the capabilities provided by technology applications. Researchers like Carney (2010) have also pointed out that web-based programs ought not to attempt to oust the teacher from her role, must rather enhance the teacher’s capability in individualizing learning for a learner. Sherman, Kleiman and Peterson (2004) too support the above view that technology supplements the role of a teacher.
3. Reading Skill Intervention Program—“Literacy Pro”
Scholastic Publishing Company is a forerunner in educational publications providing schools and families with both top-quality print and digital learning programs. The company provides reading material to nearly 50,000 schools and libraries and has distributors in more than 20 countries across the globe. Today, they are harnessing the power of adaptive technology like never before to help students improve their reading skills. “Literacy Pro” in particular aims to connect children’s independent reading to the schools’ literacy goals.
“Literacy Pro” facilitates ongoing assessments of reading comprehension that provide a system to track students’ reading skills, monitor their progress, allow for appropriate intervention and help them to attain realistic goals. The program matches readers to leveled texts and provides them with personalized reading material to improve their Lexile scores. The program boasts of a computer-adaptive reading assessment that identifies student reading levels, recorded in Lexile measure, strongly supported by a technology-based program that promotes independent reading among students. The tests are created with the ability to adjust item difficulty to suit student responses. The difficulty level of questions is subject to change based on the responses given by students as they progress through the test. It is recommended that students enrolled on the reading program take 3 or 4 well-spaced out tests in a school year to give students sufficient time for progress and teachers sufficient data for appropriate intervention.
The online reading program Literacy Pro starts off by ascertaining the Lexile level for each individual student and offers him/her a personalized reading path. A Lexile measure is the numeric representation of an individual’s ability to read or a text’s difficulty level. Hence, one may speak about a Lexile reader measure and a Lexile text measure. If used effectively, these measures can help match books according to the reading ability of individual readers. An individual’s Lexile level may range from 0 L to 2000 L.
A Literacy Pro test consists of passages followed by questions that measure the reading ability of students by focusing on reading skills like identifying details, comparing details, drawing conclusions and making inferences. Each test question is presented in the form of a statement with four choices. The Literacy Pro test uses a computer algorithm that utilizes a statistical procedure that estimates each student’s ability to comprehend texts and represents it as a Lexile score. Prior information about each student’s ability to read is used to control the selection of questions and the calculation of the Lexile score. Literacy Pro also makes results and a variety of reports available to educators to make data-driven decision regarding student progress and attainment in reading.
4. Objective of the Study
The primary objective of the current study is to analyze the impact of the online reading program (Literacy Pro) to improve reading comprehension skill of students who are studying at school level and English is taught as a Second Language (ESL). The primary task of this Reading Intervention Tool is classifying the students into two categories, that is, “at risk” and “not at risk” and the current study evaluates the classification accuracy of the tool. The students undertake different online reading exercises provided by “Literacy Pro” once they are classified into the above two categories.
5. Research Hypotheses
A focus on identifying students in need of support and providing targeted, data driven intervention, such as Response to Intervention (RtI), provides a systematic framework designed to change the trajectory of reading outcomes for struggling readers at all levels (Johnson, Mellard, Fuchs, & McKnight, 2006). The foundation of a successful implementation of RtI for ameliorating and addressing reading difficulties is the accurate and timely identification of students with or at-risk for reading difficulties so that additional instruction/intervention can be provided (Glover & Albers, 2007). From the measurement perspective, effective screening tools demonstrate high levels of sensitivity in accurately identifying those students who will actually encounter difficulties, as well as high level of specificity in the accurate identification of those who are not likely to demonstrate reading difficulties (Zhou, Obuchowski, & McClish, 2002). Thus, the goal is to maximize Classification Accuracy (CA), a summative measure of overall proportion of students who were correctly identified as at risk or not-at risk on a screening measure (Kent, Wanzek, & Yun, 2019). Hence, the two hypotheses of the study:
H01: The Classification Accuracy (CA) of the online reading program “Literacy Pro” does not demonstrate a high level of “sensitivity” in correctly identifying those students who will actually encounter reading difficulties.
H02: The Classification Accuracy (CA) of the online reading program “Literacy Pro” does not demonstrate a high level of “specificity” in the accurate identification of those students who are not likely to demonstrate reading difficulties.
6. Conceptual Framework
The basic constructs of the study are the 1) classification accuracy of the online reading program for classifying the students into two groups, namely, at risk, and not-at risk, and 2) the efficacy of the online reading material provided by “Literacy Pro” for improving English Reading skills and Reading comprehension.
Based on the Lexile Score (Appendix A) the students may be categorised into two groups—“at risk” and “not at risk”. Any student whose Lexile score falls in the range of the below basic scores with respect to his/her Grade expected Lexile score, would be identified as “at risk”. Such students will receive intervention that results from informed instructional decisions made by teachers as well as through instruction, practice and independent targeted reading provided by the online reading program, Literacy Pro.
The Lexile level is impacted by two strong factors that determine the difficulty level of a text and in turn influence comprehension skills: word frequency (semantics) and sentence length (syntax). Lexile text measures range from below 0L to above 1600 L. A Beginning Reader code (BR) is assigned to texts that are below 0 L. Thus the Lexile scale may be compared to a thermometer with measures below zero marking a decrease in text difficulty level and measures above zero marking challenge.
The Lexile framework is designed to match reader’s comprehending ability with texts that pose the appropriate challenge. Thus, the success of this model rests on the perfect match of the reader and the text. It is suggested that the desired growth and improvement in reading is expected when texts are matched to students within the range of 50 L above and 100 L below the students’ Lexile measure. For instance a student of Grade 2 must ideally be reading between 300 L - 600 L, he/she should read books marked between a Lexile measure of 200 L - 650 L. Besides, researchers like Anderson, Wilson and Fielding (1988) surmise that students who read independently have an advantage over their peers who do not and thus outperform them.
To ensure the reliability, stability and accuracy of test results, all results are estimated against a number called the Standard Error of Measurement (SEM) shown in Appendix B. Each student takes a unique test which depends on his/her reading ability; the SEM associated with any student score is also unique. The more number of questions that the student answers, the more his/her SEM will decrease.
The conceptual framework of the study is diagrammatically represented in Figure 1 where the Classification Accuracy in terms of Sensitivity (students at risk) has been taken as independent variable 1 and classification accuracy in terms of Specificity (students not at risk) has been taken as independent variable 2 and the Lexile score secured at the end of the intervention has been taken as the dependent variable.
7. Method
The study was conducted in an Indian International School in Dubai. The online reading literacy program “Literacy Pro” has been implemented in the school since 2019. The study was conducted during the academic year 2021-22 in the first 6 months starting from October through March and these students are in Grades VII and X during the period of study. Thus, the target population was a total of 1519 students spread across Grades VII and X. Grade VII has 23 sections with a strength of 35 to 38 students in each section and Grade X has 20 sections with a strength of 31 to 38 in each section. The sub population in each of the 2 Grades was: Grade VII – 823 and Grade X – 696. The sample size for effective and fair representation of the target population was determined by using the Krejcie and Morgan Table(Appendix C). The respondents are all Indian expat girl students studying in a school in Dubai, UAE that follows the Indian, CBSE curriculum. The medium of instruction is English and all students are second language users of the English language. The number of students from each grade that participated in this study are given in Table1 where the divisions from the two grades are selected by simple random technique.
8. Descriptive Analysis
The reading program, Literacy Pro measures reading comprehension skill in Lexile scores. A Lexile score below or equal to 799 is considered to be “at risk” and a score of 800 and above is considered to be “not at risk”. A frequency distribution of the Lexile score 1 (the beginning classification) and Lexile score 2 (the end classification) are given in Table 2 and Table 3 respectively. The data in these tables show that the reading program identifies students in the “at risk” and “not at risk” categories. It reflects the effectiveness of the reading program as there is a rise in the number of “not at risk” students in both Grades 7 and 10 when Lexile scores 1 and 2 are compared. Similarly, there is a drop in the number of students in the “at risk” category in both Grades when Lexile score 1 and 2 are compared. (Figure 2)
![]()
Table 1. Sample and target population.
![]()
Table 2. Frequency distribution for Lexile 1.
![]()
Table 3. Frequency distribution for Lexile 2.
9. Hypotheses Testing—Sensitivity and Specificity
A Chi-square test was used to test the null hypothesis that there is no significant association between Lexile score 1 and 2. The results of the chi-square test suggested that there is significant association between Lexile score 1 and 2 for both the grades 7 (Chi-square (1) = 179.97, p-value < 0.01) and for grade 10 (Chi-square (1) = 177.43, p-value < 0.01). (Table 4)
Balloon plots are given in Figure 3 and Figure 4 to visually understand the frequency distribution for Lexile score in Grade 7 and 10 respectively.
From the two balloon plots it can be observed that 30% of the students in grade 7 were “not at risk” for both the scores and almost 60% of the students were “at risk” before and after intervention. Whereas for grade 10 students, around 72%were “not at risk” and 22% were “at risk” before and after intervention.
10. Simple Logistic Regression Analysis
Logistic regression analysis was done to classify the Lexile score 2 based on Lexile score 1, thus Lexile score 2 is the dependent variable and Lexile score 1 is the independent variable.
The adequacy of the logistic regression model is given in Table 5. It can be observed that for grade 7 and 10, the model is able to explain 71% and 79% (Nagelkerke R-square) variation respectively in Lexile score 2, which is very satisfactory.
On the basis of the results of the data analysis (Table 6), the null hypothesis H01 is rejected, or in other words, the classification accuracy of the online reading program “Literacy Pro” shows sensitivity at less than one percent level of significance. On the basis of the results of the data analysis the null hypothesis H02 is rejected, or in other words, the classification accuracy of the online reading program “Literacy Pro” shows specificity at less than one percent level of significance.
The results of the logistic regression suggest that Lexile1 is statistically significant and it has been interpreted that “at risk” Lexile 1 score students were 534.4 and 412.5 times more likely to exhibit “at risk” Lexile 2 scores than “not at risk” Lexile1 score students for grade 7 and 10 respectively. Now we obtain the confusion matrix to understand the sensitivity and the specificity using the developed model. Sensitivity and Specificity are also known as true positive rate and true negative rate respectively. (Table 7)
![]()
Table 7. Sensitivity and specificity.
For grade 7 and grade 10, the sensitivity is 75.7% and 93.8% respectively whereas the specificity is 99.4% and 96.5% respectively. Here it can be observed that both the models have a higher probability of classifying the students having at risk Lexile scores. The overall accuracy of the model for grade 7 and 10 is 89.9% and 94.4% respectively.
The ROC (Receiver Operating Characteristic) curve for the model is given in Figure 5 and Figure 6. The area under the curve (AUC) value was obtained as 0.8754 and 0.9512 for grade 7 and 10 respectively. An AUC value of 0.5 suggested no discrimination, and a value greater that 0.7 was considered to be acceptable.
11. Multiple Logistic Regression Analysis
Multiple Logistic regression analysis was done to classify the Lexile score 2 based on Lexile score 1, CAT4 and ASSET scores. These are the other two examinations which the students undergo to test their reading comprehension. These exams are conducted by some other international bodies at the school level. The only significant predictor is Lexile1. It can be observed that for grade 7 and 10, the model is able to explain 76% and 84% variation (Nagelkerke R-square) respectively in Lexile score 2, which is very satisfactory. (Table 8)
The regression coefficients are given in Table 9
The results of the logistic regression (Table 9) suggest that Lexile1 is statistically significant and it can be interpreted as the “at risk” Lexile1 score students are 327.5 and 467.3 times more likely to exhibit “at risk” Lexile 2 scores than “not at risk” Lexile1 score students for grade 7 and 10 respectively’. The other predictors do not have a statistically significant effect.
Thus, on the basis of the result of the Simple Logistic Regression as well as the Multiple Logistic regression, it has been proved that the Classification Accuracy (CA) of the online reading program “Literacy Pro” does demonstrate a “sensitivity” in correctly identifying those students who will actually encounter reading difficulties.
It also may be concluded that The Classification Accuracy (CA) of the online reading program “Literacy Pro” does demonstrate “specificity” in the accurate identification of those students who are not likely to demonstrate reading difficulties.
12. Discussion
From the measurement perspective, effective screening tools demonstrate high level of sensitivity in correctly identifying those students who will actually encounter reading comprehension difficulties, as well as high level of specificity in the accurate identification of those who are not likely to demonstrate reading difficulties (Zhou et al., 2002). Researchers have argued that high levels of sensitivity are necessary for universal screening measures (Compton, Fuchs, Fuchs, & Bryant, 2006; Jenkins, Hudson, & Johnson, 2007). Although consensus has not been reached regarding the optimum level of sensitivity, acceptable sensitivity values noted in the literature range from 0.7 to 0.9 (Catts, Petscher, Schatschneider, Sittner Bridges, & Mendoza, 2009; Compton et al., 2006; Jenkins et al., 2007; Kilgus, Methe, Maggin, & Tomasula, 2014). Relatedly, specificity levels of at least 0.7 are generally considered adequate for screening measure. The results of the current study show a sensitivity level of 0.76 and 0.94 for grade 7 and 10 respectively whereas the corresponding specificity levels are 0.99 and 0.97 respectively. Here it can be observed that both the models have a higher probability of classifying the students having “at risk” Lexile scores. The overall accuracy of the model for grade 7 and 10 is 89.9% and 94.4% respectively in simple logistic regression whereas the corresponding values are 89.9% and 95.6% in multiple logistic regressions. Therefore, it can be concluded that the reading intervention measure of “Literacy Pro” is highly accurate in classifying students “at risk” and “not at risk”. The results of the chi-square test suggested that there is significant association between Lexile score 1 and 2 for both grade 7 (Chi-square (1) = 179.97, p-value < 0.01) and for grade 10 (Chi-square (1) = 177.43, p-value < 0.01). The area under the curve (AUC) value was obtained as 0.8754 and 0.9512 for grade 7 and 10 respectively for the ROC curves of the model. Compton et al. (2006) suggested that AUC values above 0.9 represent excellent diagnostic accuracy, between 0.8 and 0.9 as good, 0.7 to 0.8 as fair, and values below 0.7 are considered as poor. Hence, the classification accuracy of grade X is excellent and that of grade VII is good. The current study leads to the assumption that an online reading program must show high levels of specificity and sensitivity assuring its ability to accurately identify students who are “at risk” and “not at risk”. The impact & effectiveness of the intervention conducted through the online reading program depend predominantly on accurate identification of students in the above stated categories. Subsequently, the diagnostic, formative and summative assessments followed by appropriate reading intervention that is well matched to reading levels of students in terms of their Lexile score will show desired results. Nuances of reading comprehension facilitated by online reading programs in greater depth are interesting areas for research as reading is the foundational skill for all learning.
Appendix
Appendix A. Standard Benchmark Proficiency Bands
Source: https://www.scholastic.com/custsupport/images/rnt/dw/SLZLiteracyProUserGuide_v1.14feb18.pdf.
Appendix B. Standard Error of Measurement Monitoring Chart
Mean SEM on the LitPro Test by Extent of Prior Knowledge
Source: https://www.scholastic.com/custsupport/images/rnt/dw/SLZLiteracyProUserGuide_v1.14feb18.pdf.
Appendix C. Krejcie & Morgan Tablefor Determining Sample Size of a Known Population
Tablefor Determining the Sample Size of an Unknown Population
Note: N is population size; S is sample size Source: Krejcie & Morgan 1970. Source: https://home.kku.ac.th/sompong/guest_speaker/KrejcieandMorgan_article.pdf.