Development and Validation of a Problem Solving Skill Test in Robot Programming Using Scaffolding Tools

Problem solving is a crucial skill for students who experience learning and living in the 21st century. To enhance this skill, students need to face a situation setting problem, then students solve the problem. After students overcome the obstacles, feelings of pride and success grow in students’ hearts. Successful minds of students will lead the students to become problem solvers and will be embedded into their thought process. It is quite hard to find the right way to establish the problem solving skill. Robot programming is a selective course for the secondary level of education for Thai schools. The activities in this program provide students a chance to identify the problem, identify and analyze the cause of the problem, propose a problem solving method and examine the problem solving result. The problem solving skill test (PSST) included 4 levels of problem solving skills as well. PSST consisted of 57 multiple-choice items. The test can bridge the content of robot programming and the problem solving skills. It is useful for evaluating the skill progress in secondary schools in Thailand.


Introduction
Problem-solving skills are an important factor in life. Helping students learn how to be problem solvers is a critical skill. Students have to be able to solve problems. Problems pop up every day. Sometimes they are small and sometimes large. Fostering problem solving skills through the robot programming course is an alternative way for many teachers to invent the appropriate methods for the teenager students. This course provided many chances for students to build up their problem solving abilities with scaffolding tools which teacher prepared well in class. These skills are increased by technology-enhanced scaffolding in the field of computer-based learning environments [1], teacher-enhanced scaffolding [2], and teachers' roles, which become very crucial to help students in reaching the goal and interaction between scaffolding and students' characteristics. This paper aimed to develop and validate the Problem Solving Skill Test (PSST) in the Robot Programming Course. The test would be an assessment tool for teachers in Thailand who teach robot programming and would be a great way to measure the problem solving skill of students who engage in the scaffolding procedure.

Problem Solving
Problem solving has long been a goal in education; researchers and theorists have advanced markedly different conceptions and method of study [3] Numerous problem-solving phases and associated learning activities have been proposed, reflecting diverse theoretical orientations such as information processing [4], cognitive science [5] and constructivism [6]. Reference [7] proposed four problem-solving steps: understanding the problem, devising a plan, carrying out the plan, and looking back at work. These activities are often combined with heuristics. Extending [7]'s approach, Reference [8] developed a 5-stage problem-solving model that includes identifying problems and opportunities, defining goals, exploring possible strategies, anticipating outcomes and acting, and looking back and learning from them. However, Reference [9] introduced the 4 problem solving levels: identifying the problem level, analyzing and identifying the cause of the problem level, proposing the problem solving method level and examining the problem solving result level. The scaffolding tools and scaffolding environment in class will lead students to obtain the ability of problem solving. The PSST provided various situations to promote students' problem abilities with 4 questions in each situation, such as: like what is the problem in this situation? What is the cause of this problem? What should you do to solve this problem? And what is the result after the problem was solved?

Scaffolding in Instruction
Scaffolding refers to a "process that enables a child or novice to solve a problem, carry out a task, or achieve a goal which would be beyond his unassisted efforts" [10]. They are "forms of support provided by the teacher or another student to help students bridge the gap between their current abilities and the intended goals" [11]. The notion of scaffolding instruction was introduced in Vygotsky's Social Development Theory [12] which held that learning and development are interrelated in students' everyday lives. Learning should be matched in some manner with the students' development level. The relationship between learning and development was explained in terms of the zone of proximal development (ZPD), which refers to "the distance between the actual developmental level as determined by independent problem-solving and the level of potential development as determined through problem-solving under adult guidance or in collaboration with more capable peers" [12]. He also pointed out the scaffolding should be provided only within the ZPD. Learning activities that are oriented toward development levels that have already been reached are ineffective and learning activities that are oriented toward development levels that are too far advanced for the learners' potential ability are also ineffective. When the learner interacts with an adult or a more skilled peer within the ZPD, he or she is guided and supported to a greater competence and becomes capable of performing at a higher cognitive level independently once the guidance and support are internalized [13]. The scaffolding internalization process enables learners to achieve the tasks without guidance and support from social interaction. This is a critical process for students' development.
Many researchers have investigated varieties to design and use technologies to scaffold learning. Reference [14] classified technology-enhanced scaffolds for open-ended, student-centered learning into four types: conceptual, metacognitive, procedural, and strategic. Reference [15] proposed implicit and explicit scaffolds, while [16] distinguished between hard (fixed, stable, pre-set) and soft (dynamic, flexible, adaptive) scaffolds. Reference [17] indicated that such adaptive scaffolding enhances student problem solving by fostering.
The scaffolding strategies in robot programming were the covered area of the introduction to robot, robot assembling, movement of robots, robot touch sensor, data transferring of robots and infrared sensors of robot.
Moreover, computer-enhanced scaffolding can assist students in structuring complex tasks by problematizing content knowledge [18]. Scaffolding has been applied to help articulate and act upon problem-solving process and learning activities.
The instruction of robot programming increases the problem solving ability of students. The multiple pre-set questions, tasks and situations provide a variety of ways for students to identify the problem level, identify and analyze the cause of the problem, propose the problem solving method and examine the problem solving result. Students need to process whole knowledge and skills from the scaffolding tools such as teachers, peers, flow charts, computer enhancement. The process of learning gears them to gain problem solving skills.
This paper shows the validation of the PSST. This test evaluated the problem solving ability enhancement of students after integrated the scaffolding tools in the class.

Methodology
The PSST was designed based on Thai Basic Education Curriculum and focused on the occupation and technology curriculum for secondary education. The items of the PSST were validated by occupation and technology experts in content validation and language use appropriateness. The PSST included 14 situations from 7 topic. The topic, objective and number of situation are shown in Table 1.
Students gave the correct answer after reading each situation. The PSST construction was based on [9]. It included identifying the problem, identifying and analyzing the cause of the problem, proposing the problem solving method and examining the problem solving result. The whole test consisted of 57 multiple-choice items. The PSST was validated by 3 experts in the learning areas of occupations and technology from the basic education schools. The test was to be suitable for Thai students who registered for the robot programming course. The test lasted 60 minutes. The numbers of item for each component of PSST are listed in Table 2. The data were calculated by the Test Analysis Program (TAP).
The preliminary version of the PSST was pilot tested on 20 students of grade 7 in secondary schools in Thailand. These students had taken the robot programming course in 2012. The students were given 60 minutes to complete the test. Data collected was also used to investigate reliability and validity of the test. The reliability of the test was provided for by the use of index of the Kuder-Richardson Formula 20 (KR20). Item indices were examined for the purpose of item revision. Item analysis was performed in order to determine item difficulty and discrimination.

Results
The mean and standard deviation of the PSST on 57 items were 36.6 and 5.63 respectively. Score ranged from 27 to 45. Reliability of the test was measured at 0.68. Indices of item difficulty and item discrimination were analyzed as shown in Table 3.
An analysis of the options chosen by students is shown in Table 4.  The distracters "D" in item 13 has attracted the response from the students at the same number of correct answer. This item needs to be review and rewritten for its intended purpose.

Conclusion
The intent of this paper is to develop an instrument to measure problem solving skills with reference to the Thai secondary school occupation and technology curriculum. Findings from analysis showed that improvement is still needed for 23 of the PSST item to ensure instrument is reliable and useful. The qualified 44 items will be selected to measure the progression of problem solving skill. The development and validation of the PSST will be beneficial for the further study in the area of robot programming and problem solving. This instrument may be limited in its usage as it is specific to the Thai secondary school level. The findings also reflect the students' problem solving skill acquisition in the content area of occupation and technology curriculum. The examples of the PSST are shown as follows: