Research and Practice of Training Effect Evaluation System Based on Kirkpatrick Model—Taking Teacher Training System of Zhejiang Open University as an Example ()
1. Introduction
In the 21st century, information and knowledge are the driving force of all industries. In the human resource management of adult colleges and universities, teacher training plays a very important role and has become a strategic requirement for the development of adult colleges and universities. Training is an important way to provide information, knowledge and relevant skills. At present, the evaluation of the training effect is not perfect, which affects the teacher training to some extent. Most adult colleges and universities in China often encounter such questions as: “What are the improvement of teachers’ abilities after the training? After the implementation of the training, which teachers will transform the training content into the content required for teaching? Has the satisfaction of teachers and students improved after the training? Is the training effective and to what extent?” Only by clarifying these problems can we know whether the training is significant to the increment of human resources. And what kind of evaluation model can be used to test the effectiveness and profitability of the training effect and the satisfaction of teachers and students is worth studying ( Shi, 2007 ).
Training effect evaluation is a process to evaluate the degree of training goals to achieve, therefore in the process of training effectiveness evaluation to systematically collect information of training program, used to analyze the factors that influence the training effectiveness of training teachers in training to learn the use of the knowledge, skills and the impact on the organization, thus to judge the effectiveness of training activities, and provide reference for improving the effect of the future training activities.
Zhejiang Open University (formerly Zhejiang Radio and Television University) is a modern open and distance university with distance and open education as the main body, diversified education and coordinated development of various types of education. The school network consists of ZJOU, 10 municipal-level TV universities (except Ningbo), 59 county-level TV universities, and 11 directly affiliated colleges (branch schools and teaching sites). Open University to carry out the teaching center of Zhejiang Open University teacher’s ethics training, training of education theory, education teaching ability training, academic standard training, physical and mental health promotion work each teacher’s teaching development as the center, through the system effective teacher training, teachers plan to gradually increase the Open University, Zhejiang system of distance education teaching ability, humanities accomplishment, information literacy, management, coordination, etc., to the overall quality of the faculty of the Open University, Zhejiang system to a new level.
From 2018 to 2020, ZJOU has carried out 40 teacher trainings covering the whole province, and held 18 teaching competitions of various kinds for a total of 4076 person-times and 24,934 class hours in three years. Such a huge training scale, in the history of Zhejiang Open University system teacher training set a new high. However, how to play the role of training, whether to achieve the purpose, and what form of training can produce the best effect on the school’s teaching and scientific research, these are the problems of concern to the teacher managers of Zhejiang Open University system. The author understands, the Open University in Zhejiang province at present set up perfect training effect evaluation system, the effect after training evaluation work is less, the problems existing in the training effect evaluation activity at present are: first, the design of the feedback form is relatively flat and thin, not according to the specific training objects and training hierarchical comprehensive investigation. Second, the follow-up and feedback after the event did not form a system, but a random and individualized behavior. Most training evaluations only assessed the knowledge and skills granted in the training course. Third, there is no complete training effect evaluation system, only some guiding opinions, lack of operability, there is no clear evaluation purpose, scientific evaluation standards and standardized evaluation process in the actual implementation process, it is difficult to ensure the pertinence and practicability of the evaluation. Fourth, lack of professional evaluators. Fifth, the evaluation method is single. Sixth, not fully analyze and use the evaluation results, resulting in the inefficient use of training effect evaluation results ( Liu, 2006 ).
The reasons for the above problems are as follows: First, the positioning of the training work leads to the lack of mechanism. The training work still stays on the training management, and there is no corresponding mechanism for the evaluation of the training effect. Second, estimates are abandoned largely because they are difficult or impossible to quantify. Thirdly, there is a disconnect in the operation of several links of human resources. The evaluation results of training effect are rarely used effectively in other aspects of human resources, and the evaluation work is more of a formality, which is only evaluated for the sake of evaluation ( Zhao & Hao, 2008 ).
In order to measure and evaluate the efficacy of the Open University in Zhejiang province teacher training, this article selects Kirkpatrick Model of four hierarchical Model, mainly USES the research method of case analysis, the Open University in Zhejiang nearly three years of teacher training project as the research object, the selection of classic Kirkpatrick Model as assessment tool, combined with the literature method, qualitative analysis and quantitative analysis, reasoning method, comparison method, five kinds of methods are studied.
According to the depth and difficulty of the evaluation, the training effect is divided into four progressive levels—response, learning, behavior, and result. Thus, a four-dimensional evaluation standard framework is proposed. The first and second dimensional criteria (response and learning) are collected before the trainees return to work. The third and fourth dimensions (behavior and results) measure the degree to which a trainee applies the training in his or her work. On the basis of extensive literature collection and analysis, the training effect evaluation model and training effect evaluation system, which are in accordance with scientific principles and the actual situation of Open University, are deduced by combining system with emphasis, case with theory, model display with logical reasoning. Considering the large number of teachers in the system of Zhejiang Open University, and the uneven age span, educational background, knowledge background and other issues, the content of teacher training is diversified. In order to facilitate the evaluation, the training is divided into four categories: concept, knowledge, technical and business skills and management skills. In the evaluation, the four-level classification method of Kirkpatrick Model is mainly used for reference, and the evaluation is carried out at four levels: reaction, learning, behavior and result ( Yan & Cao, 2006 ).
Any training has its value, want to maximize the value of the training, should not only focus on the training process, and track the training results, trained to understand the teacher on the degree of satisfaction, mastery learning content, to help teachers to create more beneficial achievements transformation condition, also should establish a corresponding system, measures to ensure that trained teachers to apply what they have learned in the work, and enhance the level of performance. The four-level teacher training evaluation system of Zhejiang Open University based on Kirkpatrick evaluation model lays a theoretical foundation for the evaluation of teacher training of Open University, and also provides a practical basis, thus effectively improving the scientific and normative nature of the evaluation of teacher training of Open University.
2. Study on Selection of Evaluation Model of Training Effect
In recent years, US statistics show that the income gap caused by human capital and physical capital investment has been further widened. According to the actual sampling data, American scholars calculated that the return of physical capital investment was 1:3.5, while the return of intellectual investment was 1:17.5, 4 times of the return of physical capital investment. The United Nations Economic and Development Organization recognizes and affirms the convergence that exists between economic and human resources training by emphasizing: 1) the human factor, as a factor of production, is assuming a prominent position and human resources are viewed as a key factor in school and economic development; 2) The development, application and popularization of advanced technology are the necessary conditions for economic success in a technologically advanced society; 3) Education is no longer a synonym for school education. Education and training provided by the labor market, schools and various organizations are playing an increasingly important role in the process of skill formation. All these show that the improvement of the quality of talents brought by training will provide a broader space for the economic growth of the school ( Nooyi, 2007 ).
As a result, many adult colleges and universities spend a lot of money on teacher training in order to gain a competitive edge. How does the training play its role, whether it achieves its purpose, and what kind of training can produce the best effect on the development of teachers, which are the issues of management concern. To understand this, the means of training must be used in the training process. However, many adult colleges seldom evaluate the training when they train teachers. In view of this, here a brief introduction to the effective and widely used training evaluation method—Kirkpatrick 4D training effect evaluation Model framework, Kirkpatrick Model by international famous scholar Wisconsin University professor Donald. L. Kirkpatrick is put forward in 1959, is one of the world’s most widely used training assessment tools, in the field of training evaluation is difficult to shake the status, it is used to help managers and training people awareness, understanding and application of training evaluation.
2.1. Kirkpatrick Four-Dimensional Training Effectiveness Evaluation Framework
Kirkpatrick Model divides the training results into four dimensions, thus putting forward the four-dimensional evaluation standard framework, as shown in the attached table. The first and second criteria (response and learning) are collected before the trainee returns to work. The third and fourth dimensions (behavior and results) measure the degree to which a trainee applies the training in his or her work.
Among all training effect evaluation models, the four-level Model of Kirkpatrick Model has been widely recognized and applied in practice. Many academic studies and enterprise practices have proved that Kirkpatrick Model’s training effect evaluation Model has high reliability and validity.
Kirkpatrick Model divides training effectiveness into four progressive levels—response, learning, behavior, and outcome—in terms of depth and difficulty of assessment. This conceptual model helps determine the kind of data to collect and answers four very important questions.
Kirkpatrick Model’s four-dimensional training effect evaluation framework has been widely accepted by many international schools and has great influence. However, since 1970, a large number of surveys have shown that many industrial schools only use response criteria, seldom use learning criteria, and basically do not use behavior and outcome criteria to evaluate training. For more than 30 years, many scholars have tried to prove the effectiveness of the four-dimensional training effectiveness evaluation framework using Kirkpatrick Model in training. However, the relationship between the described training results and the training itself is subjective and weak. Researchers have tried unsuccessfully to isolate and ignore the effects of environment in training. In fact, Kirkpatrick Model has observed that the evidence provided by training effectiveness assessments has not been successfully validated in the changing workplace. The first dimension of the Kirkpatrick Model (response) study suggests that it is insufficient to test the benefits of training assessment by response alone. Because the trainee’s response is a trainee’s preference, there is a risk that this behavior tends to produce better considerations than other aspects, such as the actual learning, the minimum cost of completing the learning, etc. Love in the learning process can improve the efficiency of learning, but it is not the final result of training. Even a good response does not ensure that learning takes place. If given a test, trainee teachers also respond by self-assessing what and how much they need to learn ( Shi, 2008 ).
2.2. Development of the Kirkpatrick Model Four-Dimensional Training Effectiveness Evaluation Framework
Based on the four-dimensional training effect evaluation framework of Kirkpatrick Model, scholars such as Kraiger, Phillips and Alliger classified training and evaluation into five categories: cognitive achievement, skill achievement, emotional achievement, performance achievement and return on investment. Are used to measure cognitive achievement trainees in this paper, the principle of the training program emphasizes the fact that familiarity, degree of technology, or process, it includes the Kirkpatrick Model technology or using skills and behavior level, it includes skills to learn (learning) and skills in the job application two aspects, it has to do with Kirkpatrick Model of the second (learning) and the third dimension (behavior). Emotional outcomes are used to measure attitudes and motivation towards training programs. Performance results are used to determine how much the department pays for training programs. These measures include cost reductions due to reductions in employee turnover or accident rates, increases in output, and improvements in product quality or customer service. Return on investment measure training costs and yields the ratio of the size, training costs include direct and indirect costs, direct costs, including participation in training for all employees (after training, training teachers, etc.), consultants and project design staff salaries and benefits, training, and facility costs of materials, equipment or to rent or purchase cost, transportation cost of the classroom. Indirect costs are the costs not completely directly related to the design, development or teaching of training. They mainly include office supplies, facilities and related costs. Benefits refer to the value that an enterprise obtains from training ( Li & Liu, 2006 ).
Subsequently, Fredman and Yarbrough put forward the training evaluation model, and the center of the evaluation model includes many training methods and their combinations, which are adopted according to the training objectives of different organizations. In addition, the evaluation model also includes the evaluation of multiple training outcomes: 1) the response of the training participants (trainees and trainers) (training desire and motivation); 2) the change of the attitude, knowledge and skills of the training participants; 3) tangible measurement, such as product sales, the number of lost customers, etc.; 4) Other evaluation tools; 5) Implementation of management. As can be seen from the above analysis of Kirkpatrick Model, many researchers have studied and expanded the Kirkpatrick Model four-dimensional training effect evaluation framework, but these studies are always limited to the training activities themselves, and do not consider and study the issues of enterprise training and evaluation from the perspective of corporate strategy ( Liu, 2006 ).
Four-dimensional training effect evaluation framework has been greatly improved. It can help managers clarify the many evaluation methods in training evaluation and the role of multiple training objectives in evaluating training outcomes. In addition, it compares various training methods and evaluates their influence on training results during the training process. The implementation of management is also introduced into the evaluation, that is, to evaluate the impact of management implementation on the training effect. However, participants’ self-evaluation reports, observations, comments and simple meetings were used to evaluate the responses of the training participants, so the accuracy of the evaluation was greatly influenced by individuals and the training instructors ( Liu, 2006 ).
2.3. Kirkpatrick Model Application Analysis
The Kirkpatrick Model’s Model helps determine the kind of data to collect, and it also answers four important questions. Because this model is simple and easy to operate, it has been widely used in the business world. For example, AT&T, Big Blue IBM, Lenovo Group and so on have developed similar four-level evaluation model ( Yang, 2007 ).
Lenovo Group, a famous company in China, has established a four-level training evaluation system: response evaluation, learning evaluation, behavior evaluation and benefit evaluation. Lenovo has been bold in the first three levels of evaluation and has built up some experience, especially in the behavioral evaluation, because any training can only be effective if it is implemented on the behavior of the trainees. After the management training and core values training in the president’s office, the enterprise will require each trainee to develop their own action plan and check the implementation in the next training. After the training of new managers’ management skills, the enterprise will require the supervisor and supervisor of the trainees and the trainees themselves to evaluate the behavior improvement of the trainees after the training, so as to examine the training effect of the trainees from different perspectives. After the sales skills training of key customers, the company will teach the trainees to supervise some sales coaching tools, so as to give targeted guidance to the trainees who have learned sales skills training ( Yang, 2007 ).
The Kirkpatrick Model is also widely used in domestic colleges and universities. Some scholars put forward that Kirkpatrick Model is a solution for enterprise training evaluation, and the similarities and differences between training and education must be faced up to in order to apply Kirkpatrick Model to classroom teaching quality evaluation in colleges and universities. Although both are improve people’s knowledge, skills and attitude for the purpose, but in most of the academic course, college students’ learning is a basic principle and the theory of abstract knowledge, university teachers care about is to let students not only understand their knowledge in historical background, but also let them know about these deeper theoretical knowledge. It can be seen that education is often longer than training, focusing on the cultivation of people’s comprehensive quality and the impact of physical and mental development ( Ma, 2018 ).
A large number of literatures show that the domestic teacher training evaluation system in higher vocational colleges based on Kirkpatrick Model consists of 3 stages, 7 indicators and 24 main observation points. The evaluation system can be used to make qualitative and quantitative analysis on the training effect of higher vocational teachers, so as to evaluate the training of higher vocational teachers accurately.
The training effect evaluation of the above schools and universities has common characteristics. First, the most commonly used evaluation is based on the four-level Model of Kirkpatrick Model. The second is to scientifically determine the evaluation level of different training projects. Third, pay attention to the selection of appropriate assessment time, that is, how long after the end of the training to evaluate: the interval time is too short, trainee teachers may not be skilled, it is difficult to reflect the long-term effect of training; The time interval is too long, and the effect of multiple causes and effects is enhanced, which is difficult to evaluate. Fourth, strengthen cooperation with competent authorities ( Ma, 2018 ).
3. Design of Evaluation System for Teacher Training Effect of Zhejiang Open University System
When designing the training effect evaluation system, this paper fully considers the large number of teachers in the system of Zhejiang Open University, the uneven age span, educational background and other knowledge, and the complexity of human resources, complex structure and many levels, which leads to the diversity of training content. In order to facilitate the evaluation, the training contents are classified according to the training objectives. The evaluation methods for the same kind of training are the same, and different levels of training can be carried out according to the different training objects and training objectives. In this paper, the four-level classification method of Kirkpatrick Model will be used for reference, and the four-level assessment of response, learning, behavior and outcome will be carried out. Combined with the existing problems and their causes, the paper designs the training effect evaluation system and constructs the model.
3.1. Training Effect Evaluation Objectives and Principles
The different objectives of training effect evaluation require different evaluation means and methods. Based on the theoretical discussion on the evaluation objectives of the training effect and the existing problems in the evaluation of the school’s training effect, the objective of the evaluation of the teacher training effect of Zhejiang Open University system is as follows:
First, improve the training plan, improve the quality of training. Establish the province level, at the two reasonable division of labor and coordinated training management system, in which the branches of the personnel department every year at the end of next year’s report to provincial school training needs, provincial school teaching center staff to these training needs analysis summary, after next year’s training program, including select training experts, organizers and the training object, arrange training time, place, content and way, to ensure the scientific nature and feasibility of these decisions must be clear advantages and disadvantages of previous training programs, what factors led to the differences in training effect and the anticipated goal, is an expert in training the level of the problem? Or is the quality and learning enthusiasm of the trained teachers insufficient? Is the training content lack of targeted? Or does the training style affect the learning effect? And so on. Because there is no systematic training effect evaluation system at present, the answers to the above questions can only be judged by the experience of training managers, which means that the training work lacks sufficient scientific basis from the beginning of the analysis of training needs and the formulation of training plans ( Fan, Jin, & Li, 2019 ).
Second, enhance the learning effect, promote the province’s teachers in the actual work of the use of training learned knowledge and skills. Before the implementation of a training project, the provincial school should clearly carry out the evaluation of the training effect, so as to arouse the attention of all participants in the training. The training experts will try their best to improve the teaching professional level and teaching skills, and carefully choose the training materials and teaching methods. Trainee teachers will cherish the training opportunities more, enhance the enthusiasm and consciousness of learning; the organizer and manager of the training will try their best to provide perfect auxiliary facilities. All these factors are important to ensure the effectiveness of the training.
Third, provide feedback to the training designers, managers, training experts and trainee teachers. The evaluation of the training effect can provide information feedback to the training decision-makers, the training managers and the training teachers. The training decision-makers must know whether the training plan is reasonable, whether the training project has achieved the expected goal, and whether the training can create high-quality and efficient teachers. The manager of the training project must know how the training is organized and implemented. Training experts must know how to choose training materials and arrange teaching schedule for specific trainees. Trainee teachers must know whether their knowledge and skills can promote the improvement of job performance. If there is comprehensive and objective training effect evaluation information to provide feedback to relevant personnel, it will improve the scientific and targeted training decision-making, the order of training management and the enthusiasm of the training teachers to learn.
The principles of training effect evaluation are as follows:
The scientific nature. The selection of training effect evaluation level, evaluation index and evaluation method should be based on the operation mechanism of the training system, the training effect evaluation principle, as well as the specific training content and training objectives. Qualitative and quantitative evaluation methods should be adopted comprehensively to carry out multi-angle evaluation.
Normative. The evaluation level, evaluation method and evaluation process should be carried out in accordance with the steps of the evaluation program, and the principles and instructions must be observed. The evaluation results of each training project should be fed back to the designated parties concerned, and the links that directly affect the achievement of the evaluation objectives should not be deleted at will.
High efficiency. Evaluation scheme for the training content and to assess the level of the scheme evaluation index must be aimed at training target to design, the choice of evaluation methods for the corresponding training content and evaluation level, the results of the assessment analysis must focus on the expected purpose of the training program and effect assessment target, so as to ensure the realization of the aim of training effect evaluation.
Practicality. According to the difficulty and cost of the assessment, combining the existing technical resources of the school and the number, quality and structure of the assessment personnel, one or more assessment methods recommended in each assessment plan can be selected. On the premise that necessary information can be collected, the assessment method with relatively low cost and difficulty of operation can be selected ( Lan, 2015 ).
3.2. Training Effect Evaluation Process Design
A process is a series of activities that are logically related and constantly changing to achieve a goal. The principle of efficiency should be followed first in the process of evaluating the effectiveness of teacher training in Zhejiang Open University. The evaluation activity is a value-added activity, and the selected evaluation program can be successfully carried out without wasting resources. The second is the principle of feasibility. That is to say, the objective to be achieved by the evaluation scheme is logically possible; the required data can be obtained, personnel, equipment, funds can be implemented. Third, the integrity principle. The evaluation process should be a series of evaluation activities logically connected circular chain, this chain is a closed model, but also does not exclude the absorption of external useful information. From the main body of training evaluation, evaluation object and content analysis, evaluation method, training evaluation results feedback and other aspects should be carefully considered, and fully consider the reasonable allocation of resources, with more economic input, obtain reliable and rich output. The process of training effect evaluation is shown in Figure 1.
3.3. Training Evaluation Index System Design
Kirkpatrick Model divides the training results into four dimensions and proposes a four-dimensional evaluation standard framework. The first and second criteria (response and learning) are collected before the trainee returns to work. The third and fourth dimensions (behavior and results) measure the degree to which a trainee applies the training in his or her work.
Kirkpatrick Model divides training effectiveness into four progressive levels—response, learning, behavior, and outcome—in terms of depth and difficulty of assessment. This conceptual model helps to determine the kind of data to be collected and, at the same time, answers four very important questions ( Lin & Feng, 2015 ) as shown in Table 1.
Level one, response assessment, that is, level one assessment.
Response assessment is the lowest level of training effect assessment, which is used to evaluate the trainee teacher’s response to the whole training project,
![]()
Figure 1. Training effect evaluation flow chart of ZJOU.
![]()
Table 1. Kirkpatrick Model four-level evaluation model.
including the views on the training materials, trainers, facilities, methods, content, etc. At this level, the assessment usually takes the form of questionnaire to obtain the required information. On this basis, further quantitative analysis can be conducted, such as measuring the trainees’ satisfaction degree (or liking degree) to the training materials, trainers, training contents and methods, and facilities, and then calculating various satisfaction rate indicators.
Level two, learning assessment, that is, level two assessment.
In the assessment of the learning level, it is necessary to assess what the trainee teachers have learned from the training program and how much they have absorbed. For example, what knowledge has been learned, what skills have been learned or improved, and what attitudes have changed.
Evaluation of learning, though at a low level, is also very important and essential, because without the acquisition and change of knowledge, skills or attitudes, it is difficult to lead to changes in behavior and results. The methods of written test, operation test, grade scenario simulation test, or before and after comparison, setting control group and so on were used to evaluate the learning effect of the training.
Level three, behavioral assessment, that is, level three assessment.
Behavioral assessment focuses more on the changes in the working behavior and performance of the trained teachers after receiving the training. In fact, it evaluates whether the attitude, knowledge and skills of the trainees have been improved after they return to work and to what extent? It is more complex and more difficult to operate than the assessment at the response and learning levels described above.
The fourth level, the outcome evaluation, namely the fourth level evaluation.
The Kirkpatrick Model is one of the most important and most difficult part of the operation, it is used to evaluate “the (reaction), learning, behavior changes visible on organizational development and positive role”, as a result, four evaluation should be in a good one, two, three evaluation, based on development projects that can assess trained teachers’ individual performance and the degree of organization performance improvement (the specific results due to behavior change).
According to the four-level evaluation Model of Kirkpatrick Model, the evaluation of teacher training effectiveness of Zhejiang Open University system should also include the evaluation of response layer, learning layer, behavior layer and result layer. In the index system design, the evaluation results of response layer and learning layer are mainly related to the organization and implementation of training. The behavioral layer evaluates the work situation after training, and the outcome layer evaluates the changes to the organizational performance. Therefore, the evaluation indicators of training effect are divided into three parts (first-level indicators), which are the organization and implementation of training, the work situation after training, and the change of organizational performance. According to the main components of the first-level indicators, a number of second-level and third-level indicators have been established under the three first-level indicators. The composition of all levels of indicators uses quantitative indicators as the main, qualitative indicators as the auxiliary, the combination of qualitative and quantitative methods, and fully consider the scientific nature, feasibility, simplicity of the principle of combination ( Lin & Feng, 2015 ).
A training project may contain concepts, knowledge, skills and other aspects at the same time. When selecting the evaluation level, the training effect evaluation professionals shall make the choice according to the main objectives of the training, the importance of the training, the resources required for the evaluation and the difficulty. Therefore, we should choose the evaluation level pertinently, so as to design the specific indicators of training efficiency evaluation of Zhejiang Open University, as shown in Table 2 and Table 3.
![]()
Table 2. Evaluation index of training effect of ZJOU.
![]()
Table 3. Evaluation index system of training effect of ZJOU.
3.4. Evaluation Methods
1) Evaluation methods during the training. In order to find out whether the expected behavior changes of the trainees occurred during the training period, we used tests, interviews and questionnaires to review and improve the training work. When these methods are used, it is necessary to give the whole course high marks for liking the teacher. Or because of a certain element of dissatisfaction and the overall denial of the course, in the assessment of the following solutions. For example: emphasize the purpose of evaluation, request everyone to cooperate; Encourage everyone to write opinions and suggestions; Comparison with historical data or other school data; Test a course before rolling it out; Combine the use of questionnaire, interview, discussion and other ways; When classes on different topics are held together, feedback should be given in time, and questionnaires should be filled out immediately.
2) Evaluation method for the completion of training. By means of test, skill verification, simulation exercise, role play, questionnaire survey, symposium and so on, we can understand the learning situation of trainees and evaluate the achievement of training objectives and the actual results.
3) Evaluation methods of work performance. It takes time to transform knowledge into ability. Therefore, the evaluation of job performance Trainee teachers undergo a number of evaluations to determine their pre-training level of competence before they are to be trained. After receiving the training, work for a period of time, and reassess the ability. Through the comparative analysis of the two evaluation results, the influence and effect of training on the ability development of trained teachers are determined. Specific methods such as questionnaire survey, interviews and talks can be used to obtain the relevant information and opinions of the graduated trainees and their supervisors, and then collate, analyze and study, and evaluate them.
4) Evaluation method of the result layer. The utility formula (utility = YD × NT × PD × V-NT × C) is adopted, which is a qualitative and quantitative method.
YD—The number of years that training has had an impact on the job.
Nt—Number of people trained.
PD—Differences in job performance between trained and untrained persons.
V—Value, a monetary calculation of performance.
C—The cost of training facilities for each member.
On the number of training to influence work in formula (YD), trainers and did not receive trainers difference (PD) on the job and the work performance of currency terms (V) on the confirmation of using the qualitative interviews and observation of trainers I in-depth interviews and group interview and supervisor to attend trainer interview leaders, colleagues and subordinates the same; At the same time, combined with on-the-spot sampling observation, the improvement of the work of some trainers after training spot check; As well as literature and practical data analysis for confirmation ( Liu, 2006 ).
3.5. Data Analysis Method
1) Grading proves a law. All the feedback questionnaires from the trained teachers were collected together to form a document to test the training effect.
2) Score and text analysis. In addition to data analysis, refer those textual statements and explanations to him.
What have we learned in the training, what will be the effect of these gains, combined to judge the training effect.
3) Frequency distribution analysis. By analyzing the distribution of test scores, we can see where the scores are concentrated. It mainly includes simple mean, median, mode and so on.
4) Departure analysis. There are three commonly used in the evaluation: range, standard deviation and four-point difference. Range is the most commonly used indicator in evaluation practice. This indicator shows the maximum and minimum extremes of a set of scores. If the range is large, it is time to rethink the process of grading the test or the training program itself.
4. A Case Study on Teacher Training System of Zhejiang Open University
The contents of teacher training in Zhejiang Open University system include teacher ethics training, education theory training, education and teaching ability training, academic norms training, physical and mental health improvement training, etc. From 2018 to 2020, the following contents have been carried out in the teacher training system of ZJOU: teaching design training, such as the teacher training of “Application of China Open Learning Network”; Humanities training such as teacher etiquette training, new teacher training, etc.; New technology training includes teacher training for digital learning and new media application, teacher training for distance education in the era of Internet+, teacher training for information technology ability of TV University teachers in the province, and teacher training for e-commerce held in cooperation with Alibaba Business School. In addition, there is an annual professional (a total of nine major categories) teaching and research activities.
4.1. Present Situation of Teacher Training Effect Evaluation in the Whole Province
First, the design of the feedback form is relatively flat and thin, and there is no comprehensive survey conducted at different levels according to specific training objects and training courses. When using the standard feedback form, there is no good summary after the feedback, and it cannot really reflect the real opinions and ideas.
Second, tracking and feedback after the fact does not form a system, but is a random, individual-to-person behavior. Some measures, actions and planned goals required in the training class have no clear channels to implement and track after the training, and the responsibilities are unclear. After the training, no one pays attention to them, which is just a formality.
Third, the purpose of assessment is not clear. Most of the training staff and teachers regard the effect assessment as the means to test the learning effect, but ignore the effect of the knowledge and skills in the work application.
Fourth, lack of professional evaluators. It is basically the work of ordinary management personnel replacing professional evaluators. There is a large gap of evaluators, which cannot meet the requirements of a large number of training of system teachers ( Cao & Zhang, 2019 ).
Fifth, the evaluation method is single. The evaluation method of training evaluation is simple, deep and difficult. There are qualitative and quantitative evaluation methods, and there are many specific methods, such as ex ante and ex post test method, cost and benefit method, control experiment method and so on. However, the current method is single, only in the form of filling in the satisfaction questionnaire and examination. The evaluation work is only at the primary level, less in-depth training teachers’ work behavior, the change of attitude, the improvement of work performance, the improvement of ability and bring social benefits for the school.
Sixth, lack of tracking evaluation. Training effect evaluation is limited to training satisfaction survey, or write about the learning, but also for these also there is no scientific evidence for the selection of evaluation methods, evaluation is emphasized at the end of the training is to master the knowledge and skills, without pay attention to training teachers in the practical work in the future effect for the application of new knowledge, new skills, whether can improve individual performance and organizational performance.
Seventh, failure to fully analyze and use the evaluation results. There is no in-depth and systematic analysis of the information collected by satisfaction survey and examination, and there is no sufficient communication and feedback with the training teachers, the training organization management personnel and the school decision-making level, thus resulting in the inefficient use of the training effect evaluation results ( Cao & Zhang, 2019 ).
4.2. Teachers’ Feedback on Training in the Whole Province
Through the training summary and survey of 2018-2020, the question “Do you think it is necessary to participate in the training? This question, a total of 138 respondents made an effective answer, accounting for 99.0% of the total number of respondents (140). The overall mean was 1.31 (responses ranged from 1 to 5, with L being very necessary and 5 not necessary at all) and the standard deviation was 0.49. Explanation: Respondents generally believe that training is very necessary and have a relatively full understanding of the necessity of training.
In the questionnaire answered by the respondents, 47.38% of them recognized the point of “improved working ability and personal quality”; the second is “broaden the scope of knowledge, increase the amount of information”, accounting for 10.85%; “Broaden horizons, update ideas, change the way of thinking” accounted for 32.24%. Compared with the trainee teachers’ expected goal, it shows that the training basically realizes the trainee teachers’ goal of increasing knowledge, improving skills and updating ideas, and also reflects the high effectiveness of training in these aspects from one side.
In terms of the training system, the trainee teachers hope to improve it further. In terms of the training form, the trainees agree with the current training form of the school, and hope to further combine the characteristics of the training objects to choose the training method suitable for the training content. In terms of training time, trainees basically believe that the school can consider the characteristics and requirements of trainees in the selection of training objects and the arrangement of training time, but sometimes some conflicts exist.
After the training, according to the existing conditions, we used questionnaire survey, in-depth interview, monitoring the teaching effect and other forms to evaluate the training effect. Through the collection and sorting of data, we can conclude that the training has achieved a good effect, and suggest the leadership to continue to strengthen the training of teachers.
4.3. A Provincial Teacher Training Evaluation Process Based on Kirkpatrick Model Evaluation Model
1) Define the purpose of evaluation. Through the analysis of the implementation process and results of the training project, this paper discusses the factors that affect the training effect, finds out the problems existing in the training, and puts forward some suggestions to improve the training work.
2) Determine the assessment subject. Including training department, personnel department.
3) Determine the assessment work plan. January 2021 Questionnaire design, interview outline design, work plan of the evaluation project, preparation of relevant technical tools; February 2021 survey study; Statistical analysis of survey data in March 2008 to make an evaluation report.
4) Determine the assessment level. This evaluation mainly uses Kirkpatrick’s four-level evaluation model to evaluate the response layer, learning layer and behavior layer of the trained teachers. Considering the application and benefit principle of evaluation, the result layer is not evaluated this time.
5) Evaluation methods
Questionnaire survey: 200 copies of questionnaire A and B were sent out, and 44 incomplete and invalid questionnaires were eliminated after collection and sorting. 256 valid copies of questionnaire A and 160 valid copies of questionnaire B were obtained. The effective recovery of volume A was 85.35%, and that of volume B was 53.3%. Study layer uses written examination form. The online examination was conducted for 200 trained teachers, and 180 papers were recovered with an effective recovery rate of 90%.
Knowledge test has the training implementation department, according to the training content to formulate the knowledge points of the test paper, for all the training teachers in the training before and after the training were tested respectively.
Expert consultation. For the training content and the establishment of the evaluation system, we consulted the experts from the education authorities in the form of questionnaires, as well as the experts who have been engaged in staff training for many years.
Qualitative interviews and observations. In addition to the interview and observation of the trainer himself, the trainer’s superiors, colleagues and subordinates are also interviewed ( Song et al., 2019 ).
4.4. Evaluation and Analysis of Provincial Teacher Training Effect
1) Reaction layer evaluation
Through the analysis of the evaluation data, the average value of trainee teachers’ satisfaction with various factors of training implementation, implementation and learning outcomes is 4.101, and the value of the lowest satisfaction is also 3.714, indicating that trainee teachers have high satisfaction with various measures of training implementation and implementation. According to the data analysis, the items that the trainees have a high degree of satisfaction with are the training hardware facilities, the overall environment and the cleanliness of the environment, etc., and they also agree that they can learn new knowledge concepts and skills during the training, indicating that the main purpose of the training can be achieved. Trainee teachers’ sense of harvest has two of the five lowest items of measuring project satisfaction of the implementation and implementation of the training, indicating that the trainees have a low degree of satisfaction with the measuring project.
2) Evaluation of learning level
The excellent rate of examination results in learning level is 58.6%; the good rate was 40%; Qualified is 1.4%. According to the data analysis, during the training period, trainee teachers can take training opportunities seriously and master the training content well in a short time. Specific data are shown in Table 4.
3) Evaluation of behavior layer
The average value of the top five items of satisfaction was 4.112, which indicated that the main objectives of the training, such as improving the working methods of the trainees and enhancing their professional knowledge and skills, could be achieved.
4) Statistical analysis of evaluation
Using the concise statistical analysis software, the top two items and the bottom two items in terms of the satisfaction of the trainee teachers in terms of the implementation and implementation of the training are statistically analyzed, as shown in Table 5.
Using the concise statistical analysis software, the statistical analysis of the top two items and the bottom two items for the satisfaction of the change factors in the work behavior of the trainees by the training supervisor is shown in Table 6.
4.5. Trainee Teachers’ Feedback on the Evaluation of the Training Effect
In the 2018-2020 trainee teacher training survey on how to evaluate the training effect, 42.04% suggested using the practicability of the knowledge and skills learned in the training to measure the training effect, and 37.08% suggested using the follow-up survey after the training, especially the performance changes of the trainee teachers and their departments. This indicates that the trained teachers
![]()
Table 5. Statistical analysis table of the top two and the bottom two students’ satisfaction with each project of training implementation.
![]()
Table 6. Statistical analysis table of the top two and the bottom two training supervisors’ satisfaction with the factors of work behavior change of trainees.
focus on the practicability of the knowledge, skills, thinking mode and the improvement degree of work performance for the training effect.
Among the suggestions on improving training, 43.18% involved training organizations, including expanding the scope of trainees, carrying out various forms of training, and improving the level of training teachers. Among the factors that affect the training effect answered by the respondents, the organizational aspects of the training include whether the training content is reasonable (ranking first); The level of teachers trained (second); Timing of training (third); Training methods (4th); Teaching environment (5th). Therefore, there is a big gap between the training organization and the expectation of the trainees. It is necessary to focus on the practicality and pertinence of the training contents, improve the professional level of the teachers, improve the rationality of the time arrangement, and carry out various forms of training courses.
According to the evaluation of the training effect by the trained teachers, among the reasons that the training effect is good, the first reason is that the training is targeted and meets the practical needs. Among the reasons that the training effect is not good, “training is not targeted, disconnected from reality, and ineffective” is the first. Therefore, trainee teachers are extremely concerned about whether the training can be targeted and whether it can meet the needs of trainee teachers for improving skills, updating knowledge and broadening horizons.
As for how to measure the effect of training, trainees of different ages, seniority and educational levels tend to measure the practicability of training and the improvement degree of performance of trainees and their departments after training, so do managers and internal staff in this aspect.
4.6. Evaluation Analysis and Application of Training Effect
1) Reaction layer evaluation
Through the analysis of the evaluation data, the average value of trainee teachers’ satisfaction with various factors of training implementation, implementation and learning outcomes is 4.101, and the value of the lowest satisfaction is also 3.714, indicating that trainee teachers have high satisfaction with various measures of training implementation and implementation.
According to the data analysis, the items that the trainees have a high degree of satisfaction with are the training hardware facilities, the overall environment and the cleanliness of the environment, etc., and they also agree that they can learn new knowledge concepts and skills during the training, indicating that the main purpose of the training can be achieved. Trainee teachers’ sense of harvest has two of the five lowest items of measuring project satisfaction of the implementation and implementation of the training, indicating that the trainees have a low degree of satisfaction with the measuring project.
2) Evaluation of learning level
The excellent rate of examination results in learning level is 58.6%; The good rate was 40%; Qualified is 1.4%. According to the data analysis, during the training period, trainee teachers can take training opportunities seriously and master the training content well in a short time.
3) Evaluation of behavior layer
The average value of the first five items of satisfaction was 4.112.
The data analysis shows that the main objectives of the training, such as improving the working methods of the trainees and enhancing the professional knowledge and skills of the trainees, can be achieved. Trained teacher training after work behavior dimensions change factors, satisfaction sorting last five project training is training teachers leave situation reduce, teachers will be active to find things to do after training, training makes teachers have confidence in the prospect of the school, belongs to the trained teachers personal work willingness, attitude, work spirit and mental problems.
5. Conclusion
Based on extensive literature collection and analysis, on the basis of the existing training effect evaluation model of a mature at home and abroad has carried on the profound analysis, explore the Zhejiang Open University teacher training system, the feasibility of application of existing training effect evaluation theory, and the problems existing in the training effect evaluation activities carried out investigation and study, and analyzes the reasons of these problems. Through the construction of effective training effect evaluation system and the corresponding evaluation model to solve these problems.
At present, adult colleges and universities generally exist problems in the training effect evaluation activities mainly for the following three reasons: First, the positioning of the training work leads to the lack of mechanism, training work still stay in the training management, training effect evaluation work has not established the corresponding mechanism. Second, estimates are abandoned largely because they are difficult or impossible to quantify. Thirdly, there is a disconnect in the operation of several links of human resources. The evaluation results of training effect are rarely used effectively in other aspects of human resources, and the evaluation work is more of a formality, which is only evaluated for the sake of evaluation.
Referring to the academic achievements of the evaluation model and the application examples from all walks of life, this paper constructs a training effect evaluation system for the teacher training of Zhejiang Open University system from the perspectives of evaluation objective, evaluation model, main body method and the process of training effect evaluation. The corresponding implementation steps of the training effect evaluation system designed in this paper are also given, and the relevant data of Zhejiang Open University are also evaluated and analyzed. In the evaluation model, a multi-dimensional, multi-level and multi-method evaluation model (DLM) is designed for schools.
Through the above research process, the following conclusions are drawn:
First of all, DLM training effect evaluation method is not affected by school size or development stage. It can be applied to both the mature provincial adult colleges and universities and a new branch of adult colleges and universities. According to the specific training activities and courses carried out, the targeted methods in the DLM training effect evaluation model were selected to effectively evaluate the training effect, and the training effect evaluation work was incorporated into the daily work.
Secondly, “training” and “training evaluation” are all schools and teachers growth, development and expansion of necessary means and ways, in order to make the training more effective transformation, the training of learning as a long-term system of school work to do, both training and training effect evaluation is not a one-off, but a systems engineering, schools and teachers to develop must continually accept the new knowledge, new skills, therefore, with the continuous development of training, and constantly changing, continuous innovation, we conducted the training effect evaluation of innovation.
Thirdly, training evaluation is the guiding ideology and basic requirement to guide and develop follow-up training. Training evaluation is a highly technical work, and whether the evaluation can be organized scientifically has an important impact on the quality of training and the reliability and effectiveness of the results.
Finally, the ultimate goal of training effect evaluation is to training to maximize the effect of myself into training teachers, into the reality of labor, to create value, improve efficiency, therefore, the training effect evaluation smoothly to the school each related department’s support, also needs to have the relevant supervision and guarantee system, combining training and training effect evaluation is very good, can make both promote each other.
The shortcomings of this paper are as follows: 1) There are still some variables in the evaluation method of the result layer, and the data have not been obtained yet, which need to be further analyzed in the future. 2) In the design of the evaluation index system, it fails to be well combined with the model. It is necessary to further establish the evaluation index that is in line with the reality and easy to operate. 3) In the design of evaluation method, the existence of non-training factors should be fully considered to find an economic and effective evaluation method to improve the accuracy of evaluation of training effect.
Fund Project
This paper is the research result of Zhejiang Radio and TV University 2019.
Yanfang Yu Education Technology Master Studio Project; research results of Zhejiang Distance Education Association 2016 scientific research project NO.: DES-16Y02.
Acknowledgements
In the process of writing this paper, I have received the great support and help from Professor Weifu Wu, Professor Yanfang Yu and Associate Editor Yuan Chen of Zhejiang Open University, and I would like to express our sincere thanks for the successful publication.