Academic Program Assessment: a Case Study of a Pragmatic Approach

Accreditation criteria of Information Technology programs require effective learning outcomes assessment and evaluation with rigorous processes, well documented results, broad faculty participation, and complete coverage of the assessment and evaluation cycle. This paper describes a model that the College of Information Technology at Ajman University of Science and Technology uses to implement a complete outcome-based assessment and evaluation plan of its programs. The plan contains detailed accounts of procedures and tools used to measure the achievements of program learning outcomes. Information which is gathered from exam results, faculty, students, alumni, internship, and employers are used to measure the level of achievement of each learning outcome from a different perspective. A final decision is made with respect to each learning outcome. This decision is based on combining the results of the various relevant measurement tools for that outcome. The assessment model described in this paper was used for the successful accreditation of all programs offered by the college of Information Technology and adopted by other colleges at the University.


Introduction
Academic programs assessment and evaluation is becoming an important process in providing improved education to students through modified curriculum and instruction.The Commission for Academic Accreditation (CAA) of the Ministry of Higher Education and Scientific Research (MOHESR) accredits each of the programs offered by the College of Information Technology (CIT) at Ajman University of Science & Technology (AUST).The CAA implements standards and procedures that require an academic program to provide an evaluation of program quality and effectiveness as part of a self-study report.Assessment has also become a tool of accountability in education by providing evidence on how effective the teaching is (ABET, 2010).An assessment plan will determine how well students are benefiting from a learning experience offered by a program of study.
Assessment activity at AUST started in 2001 with two online forms filled by students.These are The Student Course Evaluation Form and The Academic Advisory Evaluation Form.The first form collects students' feedback with regard to each course taken during the semester.The first set of questions evaluates the course, textbooks, and laboratory work.The second set of questions evaluates the performance of the instructor from a student's point of view.The third and the fourth group of questions relates to examinations and information resources respectively.The second form contains a set of questions that evaluate the performance of the academic advisor again from a student's point of view.These evaluations which are useful in identifying persistent problems in some courses or with instructors aretaken into account in the annual evaluation of faculty members.
The next major advancement in assessment at AUST oc-curred in the second semester of 2004/2005 when the Department of Computer Science established procedures, and tools for assessing and evaluating the learning outcomes of the Computer Science program as part of a pilot study conducted by AUST.This paper describes the assessment model as it is applied to the Computer Science Program.The model adopted the definitions of program educational objectives and program learning outcomes provided by ABET (ABET, 2010).The model has been applied by the College of Information Technology for the accreditation of its current programs.A similar approach was used by other colleges at AUST and will be used for the upcoming re-accreditation of the IT programs.It is based on a strong course committee structure, and a detailed mapping of program learning outcomes to course learning outcomes.This model allows assessment to be done effectively at the courses level so that each course can use the most appropriate assessment tools.The approach is vigorous in completing the entire assessment cycle, and it enhances faculty participation.Hereafter, the term "program" in this study refers to the Computer Science program unless stated otherwise.The Computer Science, Computer Engineering, Information Systems, and Multimedia programs of the College of Information Technology have long been accredited by the CAA.A new Information Technology Program was also accredited recently by CAA based on the assessment model described in this work.

Literature Review
Assessment of students' learning outcomes plays an important role in educational effectiveness, improvement, and sustainability that is increasingly being recognized and required by accrediting bodies (Buzzetto-More & Alade, 2006).Assessment is an integral part of assuring that an educational institution meets necessary standards, as well as crucial means of providing evidence necessary for seeking and maintaining accreditation (Love & Cooper, 2004).Kellough identified seven purposes of assessment (Kellough, R. D. & Kellough, N. G., 1999): 1) Improve student learning.
3) Review, assess, and improve the effectiveness of different content delivery strategies.
4) Review, assess, and improve the effectiveness of program curriculum.
5) Improve teaching effectiveness.6) Provide useful administrative data that will expedite decision making.
7) Communicate with stakeholder.Thomas, T. A. (1995) stated that "Assessment is an on-going process aimed at understanding and improving student learning.It involves making our expectations explicit and public; setting appropriate criteria and high standards for learning quality; systematically gathering, analysing, and interpreting evidence to determine how well performance matches those expectations and standards; and using the resulting information to document, explain, and improve performance.When it is embedded effectively within larger institutional systems, assessment can help us focus our collective attention, examine our assumptions, and create a shared academic culture dedicated to assuring and improving the quality of higher education".
Although there are differences in detail between different accreditation agencies, most accreditation criteria are structured in a similar manner.The criteria typically require programs or institutions to:  Specify in clear terms the skills, including cognitive skills (i.e.knowledge) that they expect students to achieve by the time they graduate (sometimes referred to as "program learning outcomes"). Set up an assessment process to determine the extent to which the program or institution is successful in enabling students to achieve these learning outcomes. Establish system, which implements program improvements, by using data collected through the assessment process.Most of the other criteria are driven by the explicitly formulated program learning outcomes.Thus, a typical curriculum criterion will state that the curriculum is designed in such a way that successful completion allows students to achieve the specified skills by the time of graduation.Atypical faculty criterion will state that the faculty must be qualified to deliver the curriculum and to revise the program in light of the data collected in the assessment process.A typical facilities criterion will state that the physical and library facilities are adequate to allow students to achieve the specified skills etc. (Gowan, MacDonald, & Reichgelt, 2006).
In the last few years, learning outcomes have achieved a widespread importance in conferences and the literature as a model of assessing the knowledge and skills obtained from a learning experience.Learning outcomes have applications at three distinct levels: 1) the local level of the individual higher educational institution for course units/modules, programs of study and qualifications; 2) the national level for qualifications frameworks and quality assurance systems; and, 3) internationally for wider recognition and transparency purposes (Ashiem, Gowan, & Reichgelt, 2007;Ashiem et al., 2007).
Learning outcomes focus on measurable cognitive, behavioural, and attitudinal development of students as they interact with a learning activity.They are what students are expected to demonstrate in terms of knowledge, skills, and attitudes upon completion of a learning experience (Adam, 2004;Ashiem, Gowan, & Reichgelt, 2007).
Learning outcomes and outcome-based approaches have implications for curriculum design, teaching, learning and assessment, as well as quality assurance.They are likely to form an important part of the twenty-first century approaches to higher education and reconsideration of such vital questions as to what, who, how, where and when we teach and assess (Ashiem, Gowan, & Reichgelt, 2007).In terms of curriculum design and development, learning outcomes are at the forefront of educational change.They represent a change in emphasis from teaching to learning that characterize what is known as the adoption of a student-centred approach in contrast to traditional teacher-centred viewpoint.Student-centred learning produces a focus on the teaching-learning-assessment relationship and the fundamental links between the design, delivery and measurement of learning (Adam, 2004).
To implement a learning outcomes approach, program must first formulate the program educational objectives (broad goals) that address the institutional and the program's mission statements.The program's mission is responsive to the expressed interests of various program stakeholders.Then, the program must formulate a set of program learning outcomes (knowledge, skills, and attitudes) the programs' graduates should have.Program learning outcomes must directly address the educational objectives andmay encompass certain specified learning outcomes specified by appropriate bodies as in the case of ABET for engineering and information technology programs.These program educational objectives and learning outcomes must be specified in a self-study report.The next step is to formulate a set of measurable learning outcomes for each course in the curriculum.Based on these courses' learning outcomes, a mapping is constructed between the program learning outcomes and courses' learning outcomes.This mapping will be used as part of a system to provide a quantitative measurement of the level of attainment of each program learning outcome.This system is based on the degree to which the learning outcomes of the corresponding mapped courses have been achieved.
Program learning outcomes are also assessed by using other complementary assessment tools.Sanders and McCartney (2003) reported a survey on twelve assessment tools used in computer science accreditation.These tools include, among others, senior exit surveys, alumni survey, written and oral exit examinations, portfolio, and external advisory panel.Each tool, by itself, has its own set of limitations and none of them is course-based.Blanford and Hwang (2003) suggested five assessment methods including class assessment, faculty interview, and a student focus group.Class assessment is a course-based assessment tool in which an instructor writes an assessment of course being taught.Course assessment processes, among other things, enablea program to demonstrate precisely how specific program learning outcomes are addressed in the curriculum.If course learning outcomes are then assessed continuously and the results are used to improve instruction that address them, the degree to which the program meets its self-selected goals must inevitably improve.In this paper, program educational objectives are measured indirectly by mapping them to program learning outcomes.
Course assessment can be time consuming; consequently, minimizing faculty time is of key importance.For closing the loop, Crouch and Schwartzman (2003) recommended forming a departmental steering committee of senior faculty members to consolidate all course learning outcomes into a final set of learning outcomes.As a result, not all faculty members will then need to participate in this step.Blanford and Hwang (2003) suggested an assessment day as an effective way for faculty to meet, evaluate assessment results, and provide improvement recommendations.

Program Assessment Methodology
The Department of Computer Science hasimplemented the following sequence of steps that are used to conduct assessment and evaluation of program learning outcomes.
1) Establish program educational objectives that are consistent with the program mission.
2) Develop measurable program learning outcomes.
3) Map program learning outcomes to program educational objectives.
4) Design the curriculum to reflect program educational objectives and program learning outcomes and adhere time to international standards of professional bodies at the same time.
5) Develop student learning outcomes for each of the courses in the curriculum.
6) Create a mapping matrix between courses and program learning outcomes, indicating which courses contribute to which program learning outcomes.
7) Determine a reasonable number of measurable performance criteria for each program learning outcome.
8) Identify measurement (assessment) tools used to assess the extent to which a student achieves the learning outcomes for each course.9) Choose program learning outcome assessment and evaluation tools.
10) Develop a plan for data collection.For each program learning outcome, identify what data is to be collected, when to be collected, and who is responsible for collecting it.
11) Measure program learning outcomes.12) Measure program educational objectives using the mapping developed in step 3 above.
13) Determine how and when the assessment information will be reviewed and evaluated to make program improvements (Closing the loop).
The College Assessment Committee (CAC) conducts the assessment on regular and timely bases.The CAC consists of the dean and heads of departments or their representatives.The committee meets between semesters or upon request from the dean.The committee has the following functions.
1) Analysis of data gathered from individual course assessment tools.
2) Development and modification of assessment tools.

Program Educational Objectives and Learning Outcomes
The Curriculum Development Committee in the Department of Computer Science has formulated both educational objectives and program learning outcomes consistent with the department mission.The curriculum was designed specifically to allow students to achieve the intended program learning outcomes.

Program Educational Objectives
Program educational objectives are broad statements that describe the career and professional accomplishments that the program is preparing graduates to achieve (ABET, 2010).For the Computer Science program offered at AUST, they can be stated as: 1) Provide students with current core knowledge of computer science that allows them to investigate and provide solutions to computer science related problems.
2) Prepare students for a professional career in computer science and related areas.
3) Offer broad and in-depth curriculum that prepare students to pursue graduate studies or engage in life-long learning in computer science and related disciplines.

Program Learning Outcomes
The curriculum design of the Computer Science Program at AUST is influenced by three considerations.First, the curriculum structure and areas of knowledge are based on the recommendations of the Association for Computing Machinery ACM and the IEEE Computer Society standard in order for the curriculum to be of acceptable international standards (Association of Computing Machinery & IEEE Computer Society, 2008).The second consideration is the guidelines of the MOHESR which stipulate that students must attain knowledge and competency equivalent to completing one or more university-level courses in each of the following area: 1) English, Arabic or other languages; 2) the humanities or arts; 3) the natural sci- ences; 4) information technology or mathematics; and 5) the social or behavioural sciences.The final consideration is to accommodate local and regional market needs with regard to specific knowledge and skills required through appropriate courses such as providing optional courses in areas relating to networking and databases.
The learning outcomes of the Computer Science program and their mapping to program educational objectives are shown in Table 1.

Mapping Courses to Program Learning Outcomes
Courses were mapped to program learning outcomes by examining the individual learning outcomes of each course.When one or more course learning outcome contributes to a particular program learning outcome, then that course is linked to the particular program outcome.Table 2 shows the mapping of courses to program learning outcomes.

Assessment Tools, Measurement Tools, and Success Criteria
To measure the achievements of each of the learning outcomes, appropriate assessment tools, measurement tools, and corresponding success criteria were developed as described in Table 3.Each program learning outcome is assessed using one or more of the assessment tools described in Table 3 as indicated by Table 4.

Course Assessment Measurement Tools
The following tools which are used to measure the achievements of course learning outcomes are given in Table 5.Table 6 facilitates the measurement of the level of achievement of each learning outcome of a particular course.For each course learning outcome I and tool J, the maximum students' grade and theiraverage scored gradeare entered.Learning outcome I is achieved if the overall score is ≥70.Overall score is calculated as total scored/total_max where,  total_scored is the sum of average grades obtained by students from all tools J for learning outcome I, and  total_max is the sum of the maximum grade of all tools J for the learning outcome I.

Assessment Time Frame
The assessment and evaluation activities are conducted according to the time frame shown in Table 7.

Data Collection
The following forms were designed to collect the required assessment data: 1) Course Learning Outcomes Achievements Form: this form is filled by course lecturer.The form contains information based on calculations done in Table 6 showing the overall score for each learning outcome of a particular course.Depending on the number of course learning outcomes which has been achieved, a decision is made on whether the whole course learning outcomes were achieved or not.
2) Exit Survey Form: this survey form contains questions relating directly to the program learning outcomes.
3) Alumni Survey Form: this survey form contains questions relating directly to the program learning outcomes.
4) Employer Survey Form: this survey form contains questions about program learning outcomes that can be evaluated from an employer's point of view.
5) Internship Survey Form: this survey form contains questions about internship learning outcomes which are directly related to some of the program learning outcomes that can be evaluated by the field supervisor of the trainee student.

Analysis of Program Learning Outcomes
The analysis starts by investigating the achievements of the learning outcomes of each course mapped to program learning outcomes.Then, each program learning outcome will be analysed individually in terms of the tools used to measure the achievements of that learning outcome stated in Table 4.

Analysis of course learning outcomes
Based on the computations described in Table 6, a decision has been reached regarding whether the course learning outcomes have been achieved for each course as shown in Table 8.
The next step uses mapped courses to decide whether each program learning outcome has been achieved.The percentages of courses which have successfully contributed to the achievement of each program learning outcomes given in Table 9.It is

Objective
After graduation, students should have: 1.An understanding of the theoretical foundations of computer science.
2. Analytical and critical thinking ability for problem solving.
3. An understanding of the principles of efficient program design techniques and strategies.Objective#1 4. Knowledge, skills and use of a variety of systems and application software; hardware; computational algorithms; programming languages; and human computer interfaces techniques.
5. Knowledge and skills to store, retrieve, and manipulate information.
6.The ability to analyze, design, implement, test, and evaluate a computer-based system.
7. The ability to work both independently and as team members.
8. The ability to communicate effectively orally and in writing.

Objective#2
9. An awareness of the ethical issues affecting computer science and the impact of computers on society.
Objective#3 10.The ability to pursue postgraduate study and research.
clear from Table 9 that program learning outcomes 1 and 2 were not achieved according to the criteria specified in Table 3.

Analysis of Exit and Alumni Survey Forms
Analysis of data collected from Exit and Alumni Survey Forms is presented in Table 10.Results show that graduates are satisfied with all program learning outcomes.The average rating score is the average of all the scores of the learning out-comes based on all exit and alumni forms.The numbers in bracket are the standard deviations.

Analysis of Employer Survey Forms
Data obtained from collected Employer Survey Forms were analysed, and summary of the statistics is presented in Table 11.
The results indicate that employers are satisfied with the standard of the graduates, yet they believe that more efforts are

2.
Alumni Survey Forms: These forms ask the graduate to rate the achievement of a given learning outcome on a scale of 1 to 5:1 being poor and 5 being outstanding.
Average rating obtained for the learning outcome.
The average score for the learning outcome from all Alumni Survey forms is ≥3.5

3.
Exit Survey Forms: These forms ask the graduate to rate the achievement of a given learning outcome on a scale of 1 to 5:1 being poor and 5 being outstanding.
The average score for the learning outcome from all Exit Survey forms is ≥3.5

4.
Employer survey forms: These forms ask the graduate to rate the achievement of a given learning outcome on a scale of 1 to 5:1 being poor and 5 being outstanding.
The average score obtained from all Employer Survey forms is ≥3.5

5.
Internship survey forms.This form asks the field supervisor to rate the achievement of specific learning outcomes on a scale of 1 to 5:1 being poor and 5 being outstanding.

Capstone project
Percentage of students who score grade B or above.
Percentage of students who score grade B or above in capstone project is ≥70%.

Analysis of Internship Survey Forms
Internship Survey Forms collected were analysed, and a summary of the statistics is given in Table 12.The number in bracket is the standard deviation.

Achievement of Program Learning Outcomes
A decision on each program learning outcome is made by combining results from different tools for that outcome.A program learning outcome must pass all tools to be declared achieved.Table 13 presents the final decision on the achievement of each program learning outcome.

Program Evaluation-Closing the Loop
Table 13 shows that the Computer Science Program on which the above analysis was based needs to address each underachieved or marginally achieved program learning outcomes, and consider one or more of the remedial actions for each of these program learning outcome.Program learning outcomes 1 and 2 must be readdressed, and measures need to be considered to ensure that these learning outcomes are achieved satisfactorily in the future.
The College Assessment Committee (CAC) has developed a set of remedial actions to be considered in improving the level of attainment of unsatisfactorily achieved program learning outcomes.The remedial actions to be taken for unsatisfactory program outcome depend on the assessment tool which has caused that the program outcome to fail.The set of remedial actions is listed below.Other appropriate actions may be considered as well.
1) Adding new knowledge units to a course.
2) Refining or deleting certain course knowledge units.
3) Changing prerequisite courses.4) Introducing appropriate elective courses.5) Increasing the number of or changing the nature of course    filled in by each faculty member at the end of the semester for each course taught by the faculty member in that semester.

Conclusion and Future Work
This paper describes a model that the College of Information Technology at Ajman University of Science and Technology uses to implement a complete outcome-based assessment, and evaluation process for the accreditation of its programs.The procedures and tools described were capable of showing the degree to which each program learning outcome has been achieved.The model is generic in the sense that it can be applied to any computing program with measurable learning outcomes.Each academic program has to identify its own set of actions to deal with each underachieved program learning outcome.Implementing a systematic assessment and quality assurance process model will help academic institutions to identify problematic areas and take the appropriate remedial actions.The assessment scheme described in this document is continuously updated as new measurement tools are developed or program learning outcomes revised.The model described assumes that courses contributing to a program learning outcome have an equal weight on that learning outcome; however, this may not be accurate, and different weights may have to be used to reflect the contribution of each course to a particular program learning outcome.Different thresholds to measure the achievements of course learning outcomes might have to be used for different courses.For example, courses that are used to filter students, such as mathematics and programming, early in the program may require lower thresholds than other courses.In this study, the same threshold (70% of students achieve grade C or above) was used for all courses.This point will be given due consideration in the incoming assessment activity.Currently, data collection and analysis are donemanually.Faculty mem-bers have to devote a considerable amount of time to these tasks.Computerizing the process will reduce much of the burden currently endured by faculty members.
3) Report changes according to ACM/IEEE Curriculum 4) Report to the college council the suggested changes in the program or course learning outcomes.5) Review and provide recommendations to the college council for mapping course to program learning outcomes 6) Report to college council any shortcomings of meeting program learning outcomes.The overall assessment plan for the Computer Science program is depicted in Figure 1.Assessments of program learning outcomes are detailed in the following subsections.

Figure 1 .
Figure 1.Assessment and evaluation information flow plan.
Tool ID: Tool ID: Tool ID: Tool ID: Tool ID: Tool ID: Tool ID: Tool ID: Tool ID: Tool ID: Total Overall score ) Recommending additional research, assessment, and evaluation in case of unclear evidence Figure2describes how the results of the assessment process are used in improving the curriculum within the guidelines of the Ministry of Higher Education-Commission for Academic Accreditation standards and the ACM curriculum.Information regarding lecturer course feedback and student course feedback will be obtained from two survey forms designed for this purpose.Students fillin Course Evaluation Form online each semester and data is accessed by members of the College Assessment Committee.The Lecturer Course Assessment form is

Table 1 .
Mapping program educational objectives to program learning outcomes.

Table 2 .
Mapping courses to program learning outcomes.

Table 3 .
Assessment tools, measurement tools, and success criteria.

Table 4 .
Assessment tools used for each program learning outcome.

Table 5 .
Assessment tools used for each program learning outcome.

Table 6 .
Calculating the achievements of course learning outcomes.

Table 7 .
Assessment time frame.

Table 8 .
Achievement oflearning outcomes for each individual course.

Table 9 .
Percentage of courses that contributed to each program learning outcome.

Table 10 .
Analysis of exist and alumni survey forms.

Table 11 .
Scores obtained for program learning outcomes from employer survey forms.

Table 12 .
Scores obtained for program learning outcomes from internship survey forms.

Table 13 .
Program learning outcomes achievements.