The Usefulness of Global Student Rating Items under End Program Evaluation Surveys in Quality Improvements : An Institutional Experience in Higher Education , Saudi Arabia

Program evaluation survey (PES) by students in higher education is one of the range of evaluations of academic programs. Others are course evaluation, teaching skills evaluation; and surveys of facilities and services. The present study employs the available PES data collected in colleges of Dentistry and Medicine, University of Dammam (UD), Saudi Arabia. Our PES relates to students’ experience at the end of their academic program. The present paper analyses these data and discusses the usefulness of global item results vis-a-vis individual item results in quality improvements of higher education. The respective percentage of participating students was 100 and 65. The PES results revealed that in view of poorly graded global items results, there is need of focus on global item results, leading to continuing improvements in all the areas covered in the questionnaire.


Introduction
It is mandatory for academic institutions in higher education to perform various continuing evaluations of courses offered, the teaching skills of faculty members as well as facilities and services.This is especially the case if the institution is pursuing accreditation for its academic programs, or further improvement in quality, or both.The data generated through these evaluations, if collected accurately, analyzed appropriately and interpreted correctly [1][2][3][4], produce some of the most important inputs required in this regard.Furthermore, the utility of such evidence can be maximized through enhancement of the awareness and knowledge of users and policy planners [2,[4][5].
Primarily to obtain academic accreditation from the National Commission for Academic Accreditation & Assessment (NCAAA), UD focuses on three evaluations -course evaluation survey (CES), student experience survey (SES), and, program evaluation survey (PES).A recent study by Rubaish, Wosornu and Dwivedi [4] used CES data from a nursing program to describe institutional practice related to students' global experience at the end of a course, and its comparative appraisal with students' experience related to various aspects of that course.They also described the utilities of the global item in deriving policy-oriented clues at three upper levels, namely: semester, year and program.
The present article deals with PES data aiming at two-fold objectives.First, it describes university practice related to students' global experience at the end of a program, and its comparative appraisal with students' experience related to various aspects of that academic program.Secondly, it describes its use in deriving policy-oriented clues in different environments.
The observations on students' global experience at the end of a program and its comparative appraisal with students' experience related to various aspects of that academic program might be helpful to policy planners in expediting developmental measures [23][24][25].Also, its comparative use in deriving policy-oriented clues in different environments is expected to be equally useful.Furthermore, from policy point of view, other academic institutions might also find these observations potentially useful in expediting quality measures of their own comparable academic programs.
There are seven remaining sections in the article.The information on data collection and methods used in analysis are described in the Section 2: "Materials and Methods".The Section 3, "Results and Discussion", describes PES results in two colleges as well as comparative results.Fourth section, "Summary and Conclusions" points out the issues related to utilities of global item results vis-a-vis individual items results.The next three sections are related to limitations, future study, and acknowledgements.Finally, references are listed.

Data
The PES data sets were acquired from the two academic programs, namely, a 12-semester program of Bachelor of Dental Surgery (BDS), and another 12-semester program of Bachelor of Medicine & Bachelor of Surgery (MBBS).The data from BDS [3] were collected on 27 October 2010 from students who completed 12 semesters of this program, and registered as interns during the academic year 2010-2011.The same data regarding MBBS were collected during May-June, 2011, from students who completed 12 semesters of this program and joined as interns during the academic year 2010-2011.
Under BDS, the PES questionnaire was given to each of the 21 students and could be retrieved from all of them.However, under MBBS, questionnaire could be given and collected from 65 out of 100 students.Thus, for BDS, the response rate was 100%; for MBBS, it was 65%.Hence, this coverage satisfies a requirement for generalisability of the observed results [26], especially, in respective colleges in UD.The PES questionnaire had a total of 22 items (Appendix 1), 22 nd item being global item.Each item is a "Likert type item".The degree of agreement with a statement was recorded on a five-point ordinal scale [3].

Analytical Methods
For item by item analysis of evaluation data on an ordinal scale [27][28], the appropriate methods are the same as those documented by Rubaish et al. [1] and used by Rubaish [2][3][4].However, to report analytical methods used on PES data, each of the four measures used in item by item analysis and respective performance grading criteria [1]

Results and Discussion
The analytical results related to each item in PES of colleges of dentistry and medicine is listed in Table 1.The successive sections describe the planned observations.

College of Dentistry
When the mean grading criterion was considered (Table 1), "acceptable" rating was observed in majority of the items, 14/21 (67%).Maintaining the consistency [4], the related global item was also rated as "acceptable".Further, the grading of majority of the remaining items was "improvement required".When the median performance grading criterion was considered, it was found that, out of 21 individual items, "high quality", "acceptable" and "improvement required" items converged in 4 (19%), 11 (52%) and 6 (29%) items respectively.Accordingly, the related global item again remained as "acceptable".Instead of earlier target of achieving satisfaction among at least 50% students through consideration of median, one may prefer to increase satisfaction level among students to at least 75% (first quartile).Its related grading criterion lowered the proportion of items with "acceptable" to 29 % (6/21), but it increased those with "improvement required" to 67% (14/21).As a result of considering the performance grading criterion based on further increase in satisfaction level to at least 80%, 19/21 [90%] of the remaining items need further improvements (Table 1).Again, consistent with these results, the global item also changed to "improvement required" in each case.Thus, under such circumstances, it is more meaningful to rely on global item results, leading to the need of corrective measures on each individual item.

College of Medicine
Like the College of Dentistry, considering the mean grading criterion (Table 1), the "acceptable" rating was observed in majority of the items, 17/21 (81%).Maintaining the consistency [4], the related global item was also rated as "acceptable".Under the median performance grading criterion, almost all "acceptable" items remain unchanged.Accordingly the related global item again remained as "acceptable".
Again, instead of earlier target of achieving satisfaction among at least 50% students through consideration of median, one may prefer to increase satisfaction level among students to at least 75% (first quartile).Its related grading criterion increased those with "improvement required" to 48% (10/21).But, it still had higher proportion of items with "acceptable" 52% (11/21).Hence, consistent to this result, global item also remained as "acceptable".
As a result of considering the performance grading criterion based on further raising satisfaction level to at least 80%, 19/21[90%] of the individual items need further improvements (Table 1).Again, consistent to this result, the global item also changed to "improvement required".
In summary, under such circumstances, it is more meaningful to rely on the results in the global item, leading to the need of corrective measures on each individual item.

Comparative Results
In both colleges, reporting on the global item is consistent with that on individual items.Further, under the mean as well as the median grading criteria, both colleges had an almost identical pattern of results.However, a comparatively lower proportion of individual items in the college of medicine had grading "improvement required".As a result, when the threshold of satisfaction among students was raised to at least 75%, a higher proportion of items in college of dentistry reached to grading of "improvement required".Also, the global item grading changed to this level.By contrast, global item grading in the college of medicine remained as "acceptable".However, with further increase in threshold of satisfaction among students to at least 80%, the global item grading in the college of medicine also reached to "improvement required".

Summary and Conclusions
Thus, irrespective of the grading criterion, both colleges need to focus on the global item results, leading to corrective measures related to almost all individual items.However, under changing thresholds of satisfaction among students, both colleges need slightly different corrective measures.Other institutions having similar environments, especially those working for quality and academic accreditation in higher education, might also find these observations useful.

Limitations
This study is limited to only two colleges of this university with their specific environments.Also, one of the considered academic programs involves a comparatively small number of students.To ensure appropriate generalisability of the results, even in similar environments, programs involving larger number of students would be a better choice.Accordingly, one needs to take precaution while generalizing these results.

Future Research
Each program as well as college involve varying environment [2][3].Thus, each college requires such evaluations in relation to each of its academic programs [3].The meaningful clues derived from such evaluations may be helpful to the policy planners in developing and managing sustainable high quality in higher education.The feedback from students regarding an academic program is useful, especially when it is early phase of development.

Appendix 1. Program Evaluation Survey Questionnaire
Items: 1) Adequate academic and career counselling was available for me throughout the program.
2) The instructors were available for consultation and advice when I needed to speak with them.
are again reproduced below:

Table 1 . College specific program evaluation survey results.
Classroom facilities (for lectures, laboratories, tutorials etc) were of good quality.12) Student computing facilities were sufficient for my needs.13) Adequate facilities were available for extracurricular activities (including sporting and recreational activities).14) Adequate facilities were available for religious observances.15) Field experience programs (internship, practicum, cooperative training) were effective in developing my skills.(Omit this item if not applicable to your program).16) As a result of this program I have developed sufficient interest to want to continue to keep up to date with new developments in my field of study.17) The program developed my ability to investigate and solve new problems.18) The program improved my ability to work effectively in groups.19) The program improved my skills in communication.20) I have developed good basic skills in using technology to investigate issues and communicate results.21) I am confident that I have developed the knowledge and skills required for my chosen career.22) Overall, I was satisfied with the quality of my learning experiences at this institution.