Critical Review of Literature on Radiologic Technology Education Program Evaluation ()
1. Introduction
Radiologic technology educational programs in the United Sates accredited by the Joint Review Committee on Education in Radiologic Technology (JRCERT) are required to perform and submit for peer-review a self-study program evaluation [1] .
Radiologic technology educational programs in the United States exist to prepare students for the workplace as well as strengthen the profession. The profession is based on competency and requires the training and education of individuals using a set of standards required by regulatory or professional bodies in a progressive and successful problem-based learning model [2] . Reference given to any international standardization of educational programs will be limited as this review is to look at the best practice of using program evaluation in educational programs accredited by the Joint Review Committee on Education in Radiologic Technology (JRCERT) in the United States.
2. Methods
2.1. Methodological Approach
The literature review was conducted to gather information that is focused on best practice or application of programmatic evaluations in radiologic technology programs accredited by JRCERT for use in a doctoral study. This literature review will be used to support research in the areas of the integration of eLearning technologies (for example, webinars) in the curriculum. The integration and use of eLearning technology is garnering attention in higher education trends and in professional development training on a national and international level in medical imaging. As such, the review purposed in content, will focus on using the educational frameworks employed by programs in the United States to evaluate program effectiveness in the transfer of information on a global learning platform as part of a doctoral research project.
The intention and focus of this literature review is to demonstrate the author’s knowledge of curriculum development, integration of learning resources, and implementation of technological advances in the education and training of medical imaging professionals. The goal is to then use this review to support the pending research outcomes, research models, proposed theories, practices, and/or applications for professional development training programs that are geared at looking at the best practices in higher education used for assessing radiologic technology educational/training programs as they relate to the use of eLearning technology in under resourced geographical areas [3] .
2.2. Search Strategy
Additionally, this purposed search strategy allowed for the literature review to be organized, and the topics categorized based on depth, relevance, and methodology for the doctoral study [3] . The secondary review then concentrated on finding literary themes within the following areas to be used for classification/categorization:
・ Formative evaluation: the process for performing formative evaluations in radiologic technology education programs and
・ Professional development programs assessment and evaluation processes and implementation reviews.
・ Program evaluation best practices in accrediting of educational programs in radiologic technology.
・ Needs Assessment, resources, and reporting of program assessment as it pertains to programmatic evaluation.
2.3. Inclusion and Exclusion Criteria
The rationale behind the inclusion and exclusion criteria used for the review is reflec- tive of the focus needed for adiologic technology, medical imaging, radiation sciences program to have a mechanism in place for professional development programs and their use in training imaging professionals globally. Table 1 demonstrates the criteria used for inclusion and exclusion in the literature search.
3. Results
To further substantiate the results of the literature review, key concepts were will be described in further detail. These areas will provide the basis for future research.
The results of the literature review search are outlined in Table 2.
4. Discussion
4.1. Radiologic Technology Education Program Evaluation
Radiologic technology educational programs in the United States accredited by the
![]()
Table 1. Inclusion and exclusion criteria
Joint Review Committee on Education in Radiologic Technology (JRCERT), are required to perform and submit for peer-review self-study program evaluations [1] . For programs to receive recognition as a JRCERT accredited program, program directors not only have to undertake the task of performing an extensive self-study, but they also have to participate in a site visit by their peers. The peer reviewers review documents; assess program operations, procedures, and processes; assess staff accountability measures; and provide feedback on the documents the program directors provide. During the site visit, the peer reviewers also inspect the learning environment, and interview students, faculty, administrators, and community members to evaluate holistically the program, according to the established standards. Participation in this process is not mandatory and programs may elect to receive JRCERT accreditation [1] . However, program directors in the United States that do seek JRCERT accreditation are required to use the standards that JRCERT has established to maintain the program curriculum.
Radiologic science programs directors are not required to make public their evaluation documents or the self-study program evaluation. However, directors of JRCERT and JRCERT accredited programs are required to monitor their performance measures and to make public the results of such measures. Data required included ocumentation of board pass rate, employment rates, attrition rates, graduation rates, and availability of resources [1] . Publication of the performance measures and results serves as a way of
![]()
Table 2. Literature review search results.
not only being transparent but at the same time promoting good will in the community [1] .
While it would be ideal to include a program evaluation in the literature review process, program evaluation documents are confidential and are not available for public inspection or review. However, the process and standards by which program directors perform their self-study program evaluations is open for public inspection and review. This aspect of the evaluation process will be discussed briefly in the areas of program evaluation in the field of medical imaging. Programs awarded accreditation have met or exceeded established standards, and were found to promote excellence in education [1] .
The JRCERT established standards to assist program directors with maintaining and meeting the mission and goals of the program. Six standards address the following objectives: 1) integrity, 2) resources, 3) curriculum and academic practices, 4) health and safety, 5) assessment, and 6) institutional/programmatic data. These standards require program directors to articulate the purpose of the program and demonstrate that the program has students and the faculty, physical, and financial resources to support it. The program directors also use the standards to document the effectiveness of the program and to demonstrate that the program can continue to meet accreditation standards via a self-study and peer site visit evaluations [1] . Table 3 identifies the Standards in Radiography effective January 2014.
![]()
Table 3. Joint review committee on education in radiologic technology standards, radiography 2014.
4.2. The Accreditation Process
Accreditation is a voluntary process. Applicants submit a request for a peer site visit at least 1 year prior to the scheduled date. Within 6 months of the application, the program director submits an external peer review of the self-study program. The self-study report includes established standards as a measure to guide the review and to serve as a protocol for program staff to reflect upon the curriculum of the program and student outcomes. After the site visit reports are written and disseminated to the applicant programs outlining strengths and weaknesses. Recommendations are made based on the review findings. The governing body will then make the decision and inform the program of its accreditation status [1] .
The JRCERT has a well-established accreditation process and cycle. The process with which programs in the United States subscribe to. This cycle is in alignment with the committee’s efforts to provide programs with information, resources, and assessment practices that are supportive of student learning outcomes and assessment/evaluation best practices [1] . The accreditation cycle requires programs to submit an initial application for review followed by submission of self-study materials that address and answer if the program is in compliance with the Standards as defined by JRCERT in its accreditation documents. The submission of the self-study materials is required at minimum 1 year prior to the scheduling of a site visit by JRCERT accreditation staff. Accreditation staff members will review all materials prior to the site visit and during the site visit. A program is not granted approval at upon completion of the site visit until the board has convened and reviewed the materials and recommendations from the site visitor report. This process is followed for all programs initiating accreditation and is imperative in providing empirical qualitative and quantative data to analyze the effectiveness of a program to meet educational goals [1] .
This process of programmatic evaluation are both self and peer guided enabling medical imaging program coordinators follow the same set of standards and guidelines set by JRCERT. In addition, program evaluation is only one element that defines the accreditation process. When a program director makes the decision to seek accreditation, the director makes the decision to undertake the task of exposing documents, processes, and procedures of the program to scrutiny in an effort to improve the effectiveness and outcomes of the program [1] . The use of the JRCERT standards to guide the evaluation process is a required part of the evaluation process. These standards serve as the preeminent guide for documenting and articulating the overall strengths, weaknesses, opportunities, and threats that a program may encounter within a specific time period [1] . Thus, a program director exposes a program to critical analytical review as well as approval and disparagement during the accreditation process.
As such, program evaluation and the accreditation process can be viewed as a similar process. However, the difference is that; whereas, an accreditation process is more comprehensive, covering all aspects of the institution in which the program is located, the program evaluation is focused on one program within the institution [1] [4] . Its purpose then provides the foundation for distinguishing the effective elements within the educational paradigm. The evaluation should be conducted to examine the effective elements that currently exist within the field and scope of radiologic sciences education. Determining the strengths and weaknesses of the program as it stands will form the basis for the establishment of sound educational programs in the field of medical imaging. By performing and conducting research in the field of program evaluation, where established industry standards and external peer site visit evaluations serve as best practices, the possibility of improving the educational experiences of the learners can be explored and defined. Further research within this area can only enhance the future of the field and continue to demonstrate relevance.
4.3. Needs Assessment in Educational and Training Program Evaluation
According to McKenzie, Neiger, and Thackeray [5] , the need for knowledge or the need to possess a certain skill should be identified and prioritized for a population. This process is known as a needs assessment, and it is the most critical step to perform in planning, implementing, and evaluating a program [6] [7] . A needs assessment provides objective data required to establish priorities of a program as well as provides a baseline with which an evaluation of the impact of a program can be performed [6] . The performance of a needs assessment serves as the roadmap for the direction in which a program should be designed and implemented [7] . Literature that reflects the importance of performing a needs assessment in training programs is included in this review.
The needs of trainees prior to implementing any type of educational or professional development program is necessary. A needs assessment prior to the establishment of a training program will most often reveal what type of program design is necessary [6] . In addition, the process of collecting data to determine what type of training is needed could help the organization to establish and deliver the proper courses [6] . Conducting a needs assessment could ensure that the training program offered is not over- or under-developed and that key concepts are the focus of the learning experience. It is understood that not all problems can be solved within any one training program, but that via the implementation of training towards improving the level of knowledge that exists, progress toward a goal can be assured [6] [8] .
When thinking of how needs assessment is essential in diagnostic imaging training, it is necessary to explain the purpose of diagnostic imaging to establish an understanding of the focus of the evaluation. Diagnostic imaging plays an important role not only in identifying pathology and tracking the progression of a disease, but it also aids in the prevention of disease by its use as a screening tool [9] [10] [11] .
A report from the WHO [12] reflected a lack of access to medical imaging technolo- gy and the lack of access to critical talent. Understanding the purposes of a program, with regards to results from needs assessment, is helpful in guiding the evaluation process. The program evaluation allows for construction and definition of the goals and direction of a program. According to Fyre and Hemmer [13] , in any medical educational program, the evaluation retrieves data based on trainees or participants’ assessment so as to make sound and evidentiary decisions regarding the content, delivery, and intent of the program. The data collected from participants then contributes to part of the overall review, analysis, and judgment of the program evaluation for the purpose of monitoring and improving the quality and effectiveness of the program [13] . Essentially, what program evaluation attempts to identify with regards to interest or needs, are its sources of variation or outcomes that are desirable and undesirable.
An educational program is not static, but rather dynamic, and the evaluation process should be reflective of this concept [13] . A needs assessment indicates what program planners can expect. A needs assessment is a systematic, planned collection of knowledge that provides valuable information regarding the interest, attitude, perceptions, and motivations of individuals or groups within a given socioeconomic environment [5] . In other words, a needs assessment can be associated with what interests or motivates an individual or groups to seek training in a particular subject. It can also pinpoint areas of a program that needs strengthening, continuing, or eliminating. With this type of acquired knowledge, effective program planning, implementation, and evaluation can be performed.
In formative evaluations, a needs assessment can aid an instructor in determining if students are acquiring knowledge and if the intended program course goals are being met [5] . The same needs assessment can help program developers identify standards that are exceeded, met, or need improvement. Needs assessment can identify disconnections between what is taught and what is tested, omissions in course content, instructors’ preparation for teaching specific concepts and skills, or instructional resources and learning materials in the area of learning in which the needs assessment is conducted.
4.4. Resources
RAD-AID [15] , an international organization, indicated that a focal point of health disparity that can break the chain of health care can occur when there is a gap in radiology resources. In addition, RAD-AID reported that radiology resources involve several aspects, including human resources, examinations, and equipment to name a few [16] [27] . For the purposes of this literature review, the focus of resources will be limited to human resources and what is required to deliver the webinar courses as far as technology and educational materials are concerned. This will allow for expansion in this area of study at a later date. For example, a research study that focused on the supply and demand of radiographers in Lithuania was conducted in 2012 [17] . The aim of this study was to analyze the need and demand for radiographers and to provide a prognosis for the time period of 2012-2030 [17] .
In addition, this study revealed a gap forming between the need for human resources in relationship to the need for services, equipment, and examinations. Findings from the study predicted that there will be a shortfall of radiographers during the 18 years analyzed, with a significant expected shortfall to be reached by 2030 [17] .
The researchers looked at several factors in making these determinations. Factors included, for example, student acceptance, entrance, retention, and attrition rates, annual mortality rates, retirement rates, population demands by age and gender, and needs for outpatient services. These needs included computed tomography and magnetic resonance imaging. What this study supplied is tantamount in the form of data, supports the increasing demand to supply resources for the future work force. Areas identified as viable and plausible solutions to address the impending gap were identified in education as tantamount [17] .
4.5. e-Learning Technology Integration
In medical education, the challenge has increased to develop, implement, and evaluate strategies for incorporating the use of e-technologies and e-learning into its curriculum. E-learning refers to the use of internet technologies to enhance knowledge and performance. The use of e-learning in medical education has increased; therefore, research in this area since 2000 has focused on the efficacy and effectiveness of this educational intervention [18] . When looking at human resources in developing countries regarding medical imaging the focused literature is limited. More attention has been given recently to the curriculum development for medical schools regarding radiology and medical imaging.
Webb, Naeger, McNulty, and Straus [19] conducted a needs assessment study that focused on standardized medical imaging curriculum. Medical school deans and chairs reported that there is a need for more overall radiology content. The results of the study indicated that there is only one single published medical school radiology curriculum available via the Alliance of Medical Educators in Radiology. This study concluded that there is a need for more content, and more instructional materials, but there are very few resources available to guide educators in content delivery. However, what Webb et al. [19] identified as an area for consideration for improvement is the establishment of a standard curriculum for instruction in radiology to combat the lack of current available resources. In 2015, the United Nations Educational, Scientific, and Cultural Organization (UNESCO) identified the need to develop a skilled workforce in low resource countries in the areas of technical and vocational education and training [20] .
The UNESCO outlined a method for transforming an unskilled workforce into a virtuous skilled workforce. In addition to identifying the need to train and educate, UNESCO also noted that technology advances play an integral role in the sustainability and economic growth of these under developed, low resource countries [20] . UNESCO recognized that education and the creation of a highly skilled workforce will lead to the development of a sustainable community. In order to do this, education must include critical thinking, problem solving, and decision making attributes that can only be acquired through participation in one’s education and the employing and integration of technology in the learning process [20] [21] . Given the complex nature of radiologic technology and the need to develop and maintain a skilled and competent workforce, the sustainability of an educational and training program in the field of medical imaging is demonstrative of its importance [15] [21] [28] .
4.6. Program Evaluation
The program evaluation process should provide data that covers the topics of interest, needs, and resources so that evaluation of the content is attainable and deliverable [13] . In medical education, program evaluation is essential. Understanding theoretical and conceptual models as they pertain to and relate to common evaluation models is essential to informed evaluation choices in any medical education program [13] . The primary purpose for performing a formative program evaluation is to look for potential changes that can enhance the overall effectiveness of the program. According to Frye and Hemmer [13] , the educational aim that dictates a program evaluation should include both intended and unintended changes associated with the program. An educational program itself is rarely static, so an evaluation plan must be designed to feed in-formation back to guide the program’s continuing development [13] . Thus, the program evaluation becomes an integral part of the educational change process [13] .
Formative/Process evaluation can be used to explain why programs succeed and fail, and whether there are characteristics or mechanisms involved in the program’s implementation that potentially mediate or moderate outcome [22] [23] . It is important to evaluate the implementation process in a training program. Formative evaluations can provide data as part of on-going monitoring and quality assessment to maximize the performance of a program. An open-minded approach to program evaluation is required in order to foster and develop the concepts association with educational change and programmatic development. This open-minded approach is quintessential to the improvement of medical educational programs currently in place [13] .
In a review of traditional and new thinking approaches to implementation research, Stetler et al. [24] described how the Department of Veteran Affairs integrated formative evaluation into its implementation program design. Evaluations on implementation in training programs serve as a means to ensure that the originally designed intervention is being conducted in a manner that is consistent with the intended goals and plan of the program. In action oriented improvement programs, for instance, summative data is important, but may not be sufficient when analyzing data to determine if a chosen strategy worked within the scope of a programmatic change. Despite the importance of performing formative/process evaluations in a timely manner, outcome analyses frequently are conducted without an assessment of program implementation [24] . According to Wilson et al. [22] , this occurrence is often referred to as the black box approach to evaluation. This means that the outcomes of a program were examined without examining its internal operation. When this type of evaluation practice occurs a sense of ambiguity persists regarding the meaning and scope of the process. For example, even reference to the naming of the type of evaluation, including process evaluation, formative evaluation, and formative research occurs.
5. Reporting
The final stage of an evaluation process is to report the results and findings [23] . When conducting a formative evaluation for possible program improvement, a written report may not always be generated. The results may be reported orally to the organization. The results may even be truncated, given in an outline form or even presented as an executive summary, due to cost constraints. In addition, the manner in which the results are reported can even be due to the personal nature of the relationship of the evaluator and the organization [25] . In whatever manner the results are reported, they will most likely be reflective of the nature of the relationship between the evaluator and the subject audience and the results of analysis of the data that has been collected [25] .
Program evaluation must cover topics that are relevant and are related to resources, technology, and evaluation methods. The researcher’s ultimate goal should be reporting the results to the communities of interest and stake holders. According to Dal Poz et al. [26] , there is a growing demand for program developers in developing countries to construct and implement programs for the management and planning of human resources in health (HRH). This study demonstrates the importance of reporting evaluation results. To begin with, the study was prompted by the identification of the crisis in the global health workforce. This crisis is characterized by a shortage of professionals, an inadequate skill mix, and an unequal distribution of professionals. The Pan American Health Organization (PAHO) and the Institute of Social Medicine, State University of Rio de Janeiro conducted a programmatic evaluation of programs in 15 Latin American and Caribbean countries. These countries included Belize, Costa Rica, El Salvador, Guatemala, Honduras, Nicaragua, and Panama in the Central America, Dominican Republic in the Caribbean, Chile, Colombia, Ecuador and Peru in the Andean sub region, and Argentina, Paraguay, and Uruguay in the South Cone.
The purpose of developing collaborative strategies in the Americas included (a) gathering information about the development of human resources for health programs; (b) supporting decision making in the formulation, implementation, or modification of health policies; and (c) expanding and maintaining a workforce able to support primary health care [26] . The evaluation reflected an effort of both academic research and the development/application of an advocacy tool. The evaluation process itself represented the challenges the two organizations faced, as well as allowed for the exchange and dissemination of practices, interventions, and programs currently available in the region [26] . The program evaluation that was produced and ultimately reported to the public provided insight as a shared lesson reflective of the importance of careful planning of the implementation of programs and interventions [26] . Program evaluations performed at this level demonstrate the importance of being able to not only assess the resources, but also the importance of identifying resources and reporting evaluation methods.
6. Conclusions
Medical imaging is one of the many allied health professions that require competency in the performance of profession-specific tasks. As such, it is important to be able to not only determine if a medical imager has attained the necessary skill level with which to perform said task, but also be able to continue to develop and improve upon the particular skill set over time. For a potential formative program evaluation study to be useful, it must close the gap between the delivery of courses and the desired outcomes by focusing on specific elements, including justification, evidence, resources, participant satisfaction, and accountability, during the evaluation process. Using formative evaluation assessment could provide stakeholders with the data needed to close the gap between the participants’ knowledge/performance levels and the desired outcomes of the program [29] . A formative evaluation, therefore, will help illuminate the potential and actual implementation process and possibly identify problems, if any, in areas in the professional development training and process planning stage [30] .
The program evaluation is the best practice in determining if an educational program in radiologic technology/radiologic sciences is using established standards that focus on integrity, resources, student success. The ability to identify gaps in educational practices and using evidenced based data to correct and improve programs is the overall goal of program evaluation. Evaluation uses procedures that are feasible, ethical and accurate. It is a non-prescriptive tool that summarizes and organizes the steps and standards that ground effective programmatic evaluation [31] . This literature review emphasizes many of the critical points essential to performance of an evaluation. It provides a basis for demonstrating the importance of assessment and evaluation and the need for further research in this area that is fluid and on-going as the educational needs of medical imaging professional changes due to technological advances and educational trends.
Acknowledgements
This review is being conducted in partial fulfillment by the author for completion of a doctoral degree in education in adult education.