A Comparative Appraisal of Timings for Program Evaluation Survey and Related Institutional Results in Saudi Arabia : Quality Management in Higher Education

The periodic evaluation of academic programs is mandatory for quality management in higher education world-wide. This paper reports a unique setting in which each student performed two such evaluations using structured questionnaires, viz.: (a) “Student Experience Survey” (SES) for their learning experience halfway through their academic program, and (b) “Program Evaluation Survey” (PES) at end of the program. A comparative appraisal of these two sets of data from students doing the Bachelor of Dental Surgery, College of Dentistry, University of Dammam, Saudi Arabia, aims to see if it is valid to generalize the observed related differentials in Saudi Arabia. The percentage of students’ participation was 100% in both SES and PES. In the students’ perceived cumulative experience, none of the total 20 items in SES was reported to be of either “high” or “acceptable” quality. By contrast, in the PES, one of the 13 items common to both questionnaires was reported to be of “high quality” (“What I have learnt in this program will be valuable for future”). Again, one of nine additional items in PES (“Developed knowledge & skill for my chosen career”) emerged to be of acceptable quality. In summary, irrespective of timing for PES, the results suggest the need of improvements in relation to almost every item confirming ongoing developmental phase.


Introduction 1.Background
The evidence generated from students' evaluation surveys on specific aspects of the function of Colleges and universities (e.g.course, faculty, program, support services, and the institution in general) remains valuable input in guiding the management of the quality in higher education [1,2].This has led to the continuation of institutional studies involving such evaluations, with resultant increased the flow of literature on these topics.To maximize utility of such evidence, in time knowledge about concerned users and organization of relevant orientation for them are equally important [1,2].
As reported recently by Rubaish [2], the University of Dammam (UoD) is currently involved in a range of evaluations by students.These are required for academic accreditation by the National Commission for Academic Accreditation & Assessment (NCAAA).A unique setting was described in which two students' evaluations deal with the program-viz.: Student Experience Survey (SES) and Program Evaluation Survey (PES).Whereas the SES assesses the experience of students midway through a given academic program, PES does so at the program's end.These two terminologies are rare in the literature [2].
In recent study [2], results under SES of two colleges were used to describe the related institutional practice, and also its policy implications for the purpose of quality management in higher education.This study drew attention to the need for environment specific planning for that purpose.
The present article is an attempt to provide additional clues to policy planners involved in program development.It has a two-fold objective: first, to describe institutional practice related to students' overall experience at the end of an academic program; and, second, to carry out its comparative appraisal with students' overall experience at halfway of this program [2].
The observations on PES results and its comparative appraisal with those on SES results [2] might be helpful to policy planners in under-taking developmental measures for academic programs.Further, from research as well as administrative point of view, other academic institutions might also find these observations equally useful in quality management of their own comparable academic programs.

Content Organisation
The content organisation of the article involves various sections.The next section "2.Materials and Methods" provides information on collection of data, and also methods used in its analysis.The section "3.Results and Discussion" describes PES results as well as comparative PES vs. SES results.Fourth section "Summary and Conclusions" mainly points out the issues related to utilities of PES and SES.The next three sections are related to limitations; future study; and acknowledgements.Finally, in the end, references are also listed.

Data
Both data sets (PES and SES) were acquired from the same academic program, namely, a 12-semester program of Bachelor of Dental Surgery.PES data were collected on 27 October 2010 from students who completed 12 semesters of this program and joined as interns during the academic year 2010-2011.The SES data [2] were collected on 27 February 2010 from students of the 7 th semesters under this program during the academic year 2009-2010.
Under PES, questionnaire was handed over to each of the 21 students and could be collected from each of them.Likewise, under SES, questionnaire was handed over to each of the 20 students and could be collected from each of them.Under each of these two surveys, response rate was 100%.Hence it can satisfy a requirement for generalisability of the observed results [20], especially, in College of Dentistry of the University of Dammam (UoD).However, because of availability of limited sample and also varying environment, this study will be carried out later in an extended manner in other colleges of the UoD.
The PES questionnaires had 22 items (Appendix 1) whereas SES questionnaire had 20 [2].Of these items, 13 are common to both questionnaires (Table 2).Each of the items is a "Likert type item".To be more precise, the degree of agreement with a statement was recorded on a five-point ordinal scale [2].

Analytical Methods
The methods appropriate in item by item analysis of evaluation data on an ordinal scale [21] are the same as those documented by Rubaish et al. [22] and used by Rubaish [2].However, to report analytical methods used on PES data, each of the four measures used in item by item analysis and respective performance grading criteria [2,22] are again reproduced below (see the bottom): Given that the ultimate goal is to achieve agreement for each item by at least 80% of students, for the statistical comparison between PES and SES results (Table 2), the preference is to use the cumulative % of students with rating score 4 or 5, and its 95% confidence interval (95% C.I.) [23].

Pooled Analysis
In the SES data [2], each program at UoD is under developmental phase, especially regarding academic accreditation by NCAAA.Similarly, each of the 22 items in PES data related to this program might be considered equally important.The pooled results are depicted in a diagram, describing the distribution of total items in relation to their performance levels using four measures of agreement: namely: mean, median, first quartile and cumulative % of students with rating score 4 or 5. PES.The similarity of PES items with SES items [2] has also been recorded in second column of this table.In addition, pooled PES results at program level are embodied in Figure 1.For comparison with the SES results, although not listed here, those already reported by Rubaish [2] were used.The successive sections describe the planned observations.

PES Results
When the mean grading criterion was used, it was found that "high quality" perception of students was only 2/22 (9%) items (   The consideration of the median rating of an item implies that at least 50% of the students assigned that rating to the item.Its use yields more clarity in observations and their implications [2,22].Under the related performance grading criterion, apart from 3/22 (14%) items, the earlier observations remain unchanged.The students' ratings now improved from "acceptable" to "high quality" in relation to two items, first-consultation & advice opportunity by instructors; and second-improved ability to work in groups."In contrast, in relation to third item, students' perception of adequate facilities for religious observances, declined from "acceptable" to "improvement required".Out of 22 items, "high quality", "acceptable" and "improvement required" items converged in 4 (18%), 12 (55%) and 6 (27%) items respectively (Figure 1).
Instead of earlier target of achieving satisfaction among at least 50% students, an increase in satisfaction level to at least 75% (first quartile), its related grading criterion lowered proportion of items with "high quality" to 5 % (1/22) and "acceptable" to 32% (7/22).As a result, as evident from Figure 1, it increased those with "improvement required" to 63% (14/22).The only item rated with "high quality" was "valuable knowledge & skill for future career." The performance grading criterion based on further increase in satisfaction level to at least 80% failed to alter earlier ratings of item with "high quality".However, it pushed down the satisfaction level in majority (6/7) of the items with "acceptable" rating earlier.As a result, apart from one item (5%) with "high quality" and another one (5%) with "acceptable" ratings, 90% of the remaining items (20/22) need further improvements (Figure 1).

Comparative PES vs. SES Results
The comparative observations between PES results just described in the previous section and SES results recently reported by Rubaish [2] are presented next.To do this, use of 95% C.I. (Table 2) of the cumulative % of students with their reported ratings 4 or 5 was made.For this, comparison between PES and SES questionnaires revealed that PES has 60% (13/22) common questions with those in SES (Table 2).Among them, results of 70% (10/13) of the items were in comparable range.These items were: adequate classroom facilities; sufficient computing facilities; sufficient library resources; adequate facilities for extracurricular activities; adequate facilities for religious observances; faculty interest in students' progress; developed skills to investigate issues & communicate results; increased ability to investigate and solve new problems; improved communication skills; and improved ability to work in groups.In relation to remaining three items, the observed proportions of satisfied students were significantly higher under PES in comparison to those under SES.These items were: consultation and advise opportunity by instructors; stimulat-ing interest in further learning; and knowledge & skill for future career.
There were nine additional items (Table 2) in PES.Students had agreement with each of them.It ranged from 19% to 62%.These items were: adequate academic & career counseling; inspiration to do best by instructor; helpful feedback from instructors; instructors' thorough knowledgeable of subject areas; enthusiastic instructors; up-to-date and useful study material; effective field experience programs; developed knowledge and skill for chosen career; and also the overall satisfaction with the quality of learning experiences at the University of Dammam.
Out of the additional seven items under SES (Table 2), apart from two items (namely helpful orientation week; and helpful library staff), students failed to agree on the remaining items.These items were: easy to find information about University; simple and efficient procedure for enrolling in courses; convenient opening timing of library; faculty are fair in their treatment of students; and, also the overall satisfaction as a student at the University of Dammam.

Summary and Conclusions
PES & SES are both program evaluation surveys.On one hand, once the students complete an academic program, PES tries to capture their experiences on specific items related to that program.On the other hand, SES tries to capture experiences of students on specific items related to an academic program, immediately after they complete half-period of that program [2].Apart from varying timings of surveys and number of involved items, they were both, essentially, program evaluations.But, for the sake of easier differentiation and clarity in management, they have been named differently.
As described earlier, this university has a unique setting of presently using both program evaluations.The two pertinent questionnaires have 13 items in common.Further, PES has nine additional items whereas SES has seven.The additional PES items (Table 2) mainly relate to various academic attributes in faculty; field experience (internship, practicum, cooperative training); knowledge and skills for chosen career; and also overall satisfaction with the quality of learning experiences at the UoD.On the other hand, the additional SES items (Table 2) relate to mainly initial stage of program development.They are related to admission, orientation, course enrollment; library staff and its opening time; treatment of students by faculty; and also overall satisfaction as a student at the UoD.
Although it may be useful to have more information, especially during developing phase of a program, keep-ing in view of the almost comparable results related to 13 common items in PES & SES, one may argue against repeated use of both program evaluation tools.This view is more applicable to programs which become fully established.To support it further, the results on additional SES items clearly relate to initial stages of the program development.There may not be much utility in repeated surveys to capture data on these aspects.Most of these items need to be managed at the level of academic program developers.But, most of the additional PES items need occasional follows up so that related process can be meaningfully monitored, especially among faculty for further improvements.
The observations made in the present study might also be useful to other institutions having similar environments, especially those working for quality and academic accreditation in higher education.

Limitations
This study is limited to only one college of this university with its specific environment.Also, the considered academic program involves comparatively a small number of students.To ensure appropriate generalizability of the results even in similar environment, a program involving larger number of students would be a better choice.Hence, one needs to be careful while generalizing these results.

Future Research
Each college as well as program involves varying environment [2].Thus, each college requires such evaluations in relation to each of its academic programs.The feedback from students regarding an academic program is unavoidable especially when it is early phase of developments.The meaningful clues derived from such evaluations may be helpful to the policy planners in managing sustainable high quality in higher education.

Acknowledgements
The author is thankful to Professor Lade Wosornu and Dr Sada Nand Dwivedi, Deanship of Quality & Academic Accreditation (DQAA), University of Dammam [UoD], for their help in completion of this article.He is equally thankful to Dr Fahad A. Al-Harbi, Dean, College of Dentistry; and Dr Sarfaraj Akhtar, Quality Management Officer, Q & P Unit, College of Dentistry, for their help and cooperation in data collection.Also, helps from Mr. R. Somasundaram, Mr Arun Vijay, & Mr C. C. L. Raymond, DQAA, UoD, in this regard are duly acknowledged.He is equally thankful to Mr Royes Joseph, DQAA, UoD, for his help in analysis.Finally, he also thanks to all students for their mature, balanced and objective response.

Appendix 1: Program Evaluation Survey Questionnaire
Items: 1) Adequate academic and career counselling was available for me throughout the program.
2) The instructors were available for consultation and advice when I needed to speak with them.
3) The instructors in the program inspired me to do my best.4) The instructors in the program gave me helpful feedback on my work.5) The instructors in the program had thorough knowledge of the content of the courses they taught.6) The instructors were enthusiastic about the program.7) The instructors cared about the progress of their students.8) What I have learned in this program will be valuable for my future.9) Study materials in courses were up-to-date and useful.10) Library resources were adequate and available when I needed them.11) Classroom facilities (for lectures, laboratories, tutorials etc) were of good quality.12) Student computing facilities were sufficient for my needs.13) Adequate facilities were available for extracurricular activities (including sporting and recreational activities).

Table 1
lists the analytical results related to each item in

Table
14) Adequate facilities were available for religious observances.15) Field experience programs (internship, practicum, cooperative training) were effective in developing my skills.(Omit this item if not applicable to your program).16) As a result of this program I have developed sufficient interest to want to continue to keep up to date with new developments in my field of study.17) The program developed my ability to investigate and solve new problems.18) The program improved my ability to work effectively in groups.19) The program improved my skills in communication.20) I have developed good basic skills in using technology to investigate issues and communicate results.21) I am confident that I have developed the knowledge and skills required for my chosen career.22) Overall, I was satisfied with the quality of my learning experiences at this institution.