Automatic Identification for Laparoscopic Surgical Procedure for Ligation and Online Distinction of Abnormal Manipulation for Thread Knotting

Laparoscopic surgery is a difficult surgical procedure compared with laparotomy. In particular, considerable skills and care are required for thread knotting in laparoscopic surgery. In this paper, a method for automatic identification of a laparoscopic surgical procedure for ligation and online distinction of an abnormality, defined as any unusual manipulation, in the identified surgical procedure is proposed. Ligation is divided into several individual surgical procedures, and on the basis of the threshold criteria, each surgical procedure is identified. Next, the identified surgical procedure, thread knotting, is classified as either normal or abnormal using a self-organizing map. Finally, to reduce surgical error, an abnormality warning system which warns detection of an unusual manipulation in the surgical procedure to the operator is constructed.

open procedures.To perform laparoscopic surgery, a surgeon must have considerable skills.A less-experienced surgeon has tendency to perform unusual manipulation in surgical procedure as compared with a skilled surgeon.Such surgeries may result in serious surgical error.Therefore, to reduce surgical error, development of a system that alerts less-experienced surgeons to the unusual manipulation is in demand.
To the best of our knowledge, a system that identifies any abnormality in a surgical procedure, defined as an unusual manipulation usually caused by the inexpertness of a less-experienced surgeon or careless mistake of a skilled surgeon, has not yet been considered.It is a great challenge to identify any abnormality in a surgical procedure and to alert the surgeon to the detection of the unusual manipulation during the surgery.Objectives of this study are to develop a method that identifies laparoscopic surgical procedure and distinguishes any abnormality for the identified surgical procedure automatically during the laparoscopic surgical procedure, and to construct an abnormality warning system which warns detection of the abnormality in the surgical procedure to the operator.To this end, haptic device "PHANTOM Omni", strain gauge, and surface electromyography (SEMG) are employed to detect any abnormality in the surgical procedure.
In the case where the sensor like a motion capturing system is used, only an appearance of the motion can be measured.On the other hand, by using the SEMG, a muscular activity of surgeon is measured.Therefore, the features such as straining of the muscle can be obtained, which cannot be measured by the motion capturing system.Use of SEMG has attracted the attention of researchers as a method of interaction between humans and machines.The amplitude property of a waveform and the power spectrum based on a frequency analysis are typical information that can be extracted from the SEMG signal.Recognition of 25 kinds of hand gesture, consisting of various motions of wrist and fingers, has been performed in [1] using only two electrodes, and a high recognition rate is successfully obtained.In a separate study, a hand-gesture identification method and a distinction method of any abnormality in the identified hand gesture on the basis of the SEMG measurements are proposed in [2].
As for surgical procedures, it is reported in [3] that a difference arises between a skilled and an unskilled surgeon in the magnitude and direction of the handling force of the object, the manner of holding surgical instruments, and the surgeon's posture.In [4], whether the level of the trainee's surgical skills can be classified as experts or novice level is investigated based on force data during the suture task.In [5], for the surgical procedures, an automatic classification method of four basic surgical procedures using sensing forceps, consisting of forceps and strain gauges, is proposed.In addition, skill evaluation is also performed.
In this paper, a novel method for an automatic identification of a surgical procedure for ligation and an online distinction of any abnormality in the identified surgical procedure on the basis of SEMG measurements of an operator and movements of the forceps is proposed.Ligation is divided into seven distinct procedures.The features of the procedure are extracted from the measurements of the movement of the forceps, and then, on the basis of the threshold criteria for the seven procedures, the surgical procedure is identified as one of the seven procedures.Next, the characteristics of any abnormality in the surgical procedure are extracted from the measurement of the movement of the forceps and operator's SEMG signals, and the identified surgical procedure is classified as either normal manipulation or abnormal manipulation using a self-organizing map [6] (SOM).
Using a simulation box built for laparoscopic surgery incorporating two forceps, the identification of each surgical procedure in ligation and distinction of the abnormality of the identified surgical procedure were carried out for a specific surgical procedure; namely, thread knotting by pulling the suture during ligation.Each surgical procedure in ligation could be identified with greater than 88% accuracy on average, and the abnormality of the surgical procedure of thread knotting could be distinguished with greater than 97% accuracy on average.Thus, the experimental results showed the effectiveness of the proposed method.Therefore, prototype of a novel system to warn detection of the abnormal manipulation to the operator was constructed in application with the proposed method for distinction of the abnormality of the surgical procedure.By using this system, it is considered that serious surgical error can be reduced, since the surgeon can notice unusual manipulation during the surgery.
A preliminary version of this paper is presented in [7].Significant improvements are as follows.In [7], experiments were carried out for only one subject.In this paper, subjects for the experiments were increased.A skilled surgeon was added as a subject for the experiments on automatic identification of surgical procedure for ligation and abnormality distinction for the thread knotting.In addition, two subjects were added for the experiment on automatic identification of surgical procedure for ligation.Besides, a novel system to warn detection of the abnormal manipulation to a surgeon was constructed to reduce surgical error.

Simulation Box
The simulation box for laparoscopic surgery is shown in Figure 1.Inside of the mannequin, a rubber sheet of 1 mm thickness is installed.The image inside the simulation box taken by a web camera is projected on a central monitor.Looking at the monitor, an operator performs a surgical procedure using the two forceps, a needle driver (right hand side), and assistant needle driver (left-hand side) inserted into the inside of the mannequin through the trocar.In this study, an operator simulates the ligation performed in laparoscopic surgery using the simulation box.

Feature of the Operation of Forceps
In order to identify operating characteristics of the forceps, as shown in Figure 2, strain gauges were pasted on three places of each forceps, at the top and the right hand side of the shaft and on the handle parts of both forceps.In addition, as shown in Figure 1, each forceps was connected to the haptics device, PHANTOM Omni, and the rotation angle and the velocity at the attached position of the forceps on the haptic device were measured, respectively.
The operating characteristics of the forceps as a mean absolute value are defined as follows:  ( ) ( ) where Strain ch(n) ( ) is the measured value from each strain gauge, Vphantom ch(n) (ch = 1,2) is the velocity at the attached position of the forceps measured from the haptic device, and n represents the number of the sampled signals for a sample size of N = 512.The values Strain ch(n) and Vphantom ch(n) are measured by sampling frequency, Fs = 1 kHz.

EMG Characteristics
In order to distinguish an abnormality caused by straining of the muscle of the operator, the operator's SEMG signals were also assessed.The SEMG signals were measured using six electrodes pasted on each forearm of the operator, as shown in Figure 3.The electrodes ch1 and ch4 were pasted on the musculus flexor carpi radialis, ch2 and ch5 were pasted on the musculus extensor carpi ulnaris, and ch3 and ch6 were pasted on the musculus extensor carpi radialis longus, while the earth electrode was pasted on the wrist.
The SEMG signals were measured using a sampling frequency Fs = 1 kHz, and fast Fourier transform (FFT) was performed to each SEMG signal for every N = 512 sampled data, which is equivalent to performing FFT every 0.512 s.After filtering the SEMG signals using the fourth-order Butterworth-type high-pass filter from 1 Hz to 1 kHz range, a full wave rectification was carried out.In addition, for normalization, the measured SEMG signal of each electrode was divided by the maximum value of the premeasured SEMG for each procedure.The characteristics of the SEMG are defined as follows: 1) Mean absolute value: In order to perform pattern recognition, the mean absolute value of each electrode is typically used, which is given as follows: , where EMG ch(n) ( ) is the absolute value of the SEMG signal of each electrode, and n represents the number of sampled signals.
2) Center-of-gravity: In the case where an unusual manipulation caused by straining of the operator is performed, it is expected that a change in the waveform can be observed in the SEMG signal.Therefore, the value of the center-of-gravity is used to represent the change of the waveform of the SEMG signal, which is defined as follows: 3) Spectrum ratio: In the case where an unusual manipulation caused by straining of the operator is performed, it is also expected that a change of distribution of the power spectrum can be observed in the SEMG signal.Therefore, the ratio of distribution of the power spectrum of the SEMG signal was also employed.It is well known that the SEMG signal is distributed in the frequency band between 5 and 500 Hz.Therefore, to record the ratio of the spectrum, the frequency band is divided into two bands; from 5 to 250 Hz and from 250 to 500 Hz.Thus, the value of the spectrum ratio is defined as follows: where and  is a frequency spectrum at frequency kf for each electrode.

Classification of Surgical Procedure
Now, consider to divide a series of ligation procedures into several procedures from a viewpoint of the forceps manipulation.With taking a significance of the manipulation into consideration, the surgical procedure for ligation is divided into seven procedures, as shown in

Method of Automatic Identification of Surgical Procedure
In order to identify the surgical procedure for ligation, the following new features are defined using the features defined by ( 1) and ( 2).
( ) Then, on the basis of the threshold criteria, the surgical procedure for ligation is identified as one of the seven procedures.The threshold value TH i (I = 1, 2, 3, 4, 5) for each new feature was determined by trial and error through repetition of the experiments.The experiments were performed by an intermediate-level subject who is not engaged in medical treatment but can perform the surgical procedure for ligation smoothly through training.To determine threshold values which are applicable to persons with various skill levels, the intermediate-level subject was chosen.The determined threshold values are shown in Table 1.
( ) Thus, the surgical procedure for ligation is identified according to the logical criteria for the seven procedures shown in Table 2, where Table 2 was determined by watching a video of actual surgical procedure for ligation.

Self-Organizing Maps
A SOM is a kind of artificial neural network that can be trained without using teacher signals.Using the SOM, high-order input data can be classified visually on the map.SOM is suitable for this study because it is easy to change the number of classifications without teacher signals and the learning time is comparatively short as compared with other traditional methods.

Abnormality of Surgical Procedure
After the surgical procedure was identified through automatic identification, online distinction of the abnormality of the identified surgical procedure is performed.In this paper, taking the manipulations which a less-experienced surgeon tends to do into consideration, a normal manipulation and abnormal manipulations are defined as follows: 1) Normal: The surgical procedure that is performed in normal manner; 2) Abnormal: The surgical procedures that are assumed to be performed in the following unusual manner: a) Posture: The surgical procedure that is performed in a posture in which the operator's elbow is raised; b) Grasp: The surgical procedure that is performed in a state in which the operator grasps the forceps too Firmly; c) Pull: The surgical procedure that is performed in a state in which the operator pulls the suture too forcefully; d) Sudden: A rough surgical procedure that is performed suddenly by the operator; e) Straining: The surgical procedure that is performed in a state in which the operator is straining.These different abnormal scenarios are illustrated in Figure 5.The validity of the abnormal scenarios has also been approved by some skilled surgeons of laparoscopic surgery.According to the surgeon's comment, these abnormal scenarios are applicable to not only ligation but also other surgical procedures.
A questionnaire about the degree of danger for each state of abnormal manipulation was conducted to three skilled surgeons of laparoscopic surgery.The results are shown in Table 3.The degree of danger was assessed by five-point system, namely it means that the risk becomes high as the value becomes large.
From these results, the Grasp, Pull and Sudden states are regarded as states which are easy to lead to danger compared with other states of abnormal manipulation.Therefore, these three states are categorized as high-risk states, since the strong load imposed on the suture may lead to serious surgical error.

Method of Online Distinction of Abnormality
The identified surgical procedure through automatic identification is classified either as normal manipulation or abnormal manipulation using two kinds of SOM.Although this method is applicable to any identified surgical procedure, to save the space of the paper, in this study, the thread knotting of ligation was chosen for the distinction of a possible abnormality.This is because thread knotting is the procedure where the surgeon must perform most carefully, according to the surgeon's opinion in the questionnaire.Therefore, hereafter the thread knotting is classified either as normal or abnormal using two kinds of SOM.Online distinction of the abnormality is  performed in the following two-phase process.First, states with greater risk to the patient must be distinguished.Therefore, the distinction of five states including three high-risk states, namely Normal, Posture, Grasp, Pull and Sudden, is executed based on only the feature of the operation of the forceps using SOM1.Next, for a state of the procedure which has been recognized as Normal by SOM1, the distinction of two states, namely Normal and Straining, is executed based on the feature of operator's SEMG using SOM2.
Then, SOM1 and SOM2 are built for the left hand side and right hand side, respectively.Therefore, four SOMs are built in total to distinguish the abnormal manipulation.Thus, an accurate distinction of the abnormality is expected by hierarchy classification on the basis of the forceps operation and biological signals.The feature vector for each SOM, which is also an input of the SOM, is defined as follows: 1) SOM1: When configuring the feature vectors, it is desirable to normalize each component.To make the component of the feature vector fall within the range 0 -1, the following new variables are defined.

Experiments
22 -27 year-old's six subjects who are not engaged in medical treatment but fully trained the surgical procedure for ligation, and one skilled surgeon who is an authorized surgeon for assessment of skill in laparoscopic surgery were chosen as the operator for the surgical procedure for ligation, and automatic identification of the surgical procedure and online distinction of the abnormality of the surgical procedure identified as thread knotting were performed.

Automatic Identification of Surgical Procedure
1) Experimental method: In accordance with the algorithm shown in Table 2, a program for automatically identifying the surgical procedure for ligation as one of the seven procedures, was built using MATLAB/Simulink software.Using the simulation box, the six subjects and the skilled surgeon performed a series of surgical procedures for ligation 15 times and 13 times in the order 1→2→3→4 respectively, and whether each procedure was identified correctly was evaluated.Since the procedures in the order 5→6→3→4 are symmetric procedures of the procedures in the order 1→2→3→4, these were omitted here to save the space of the paper.
2) Experimental results: The identification accuracy for each individual procedure was calculated.The results are shown in Table 4.In this experiment, each surgical procedure in ligation could be identified with greater than 88% accuracy on average.

Online Distinction of Abnormality
1) Experimental method: Using the simulation box, the thread knotting in the surgical procedure for ligation was performed for the rubber sheet in the mannequin under the six states, namely Normal, Posture, Grasp, Pull, Sudden and Straining, repeatedly.Then, feature vectors for SOM1 and SOM2 were constituted, respectively.The SOM toolbox was used to build SOMs, and the size of the SOM was determined as 10 × 10 with hexagon lattice.The SOM1 and SOM2 were constructed by batch learning using the premeasured 20 feature vectors for each state (a total of 120 feature vectors).
In addition, the k-means method was employed to divide the domain of SOM1 into five fields and domain of SOM2 into two fields.The thread knotting was performed 60 times for each state (360 times in total).A feature vector extracted from the online surgical procedure was mapped on the map of the learned SOMs, and an abnormal manipulation was distinguished by viewing the distribution on the map.
For distinction of the abnormality, common SOM1 was used for multiple subjects, but SOM2 was built for each subject and the distinction was conducted individually.This is because SOM2 uses the SEMG as the features and there exists individual difference in the SEMG.It should be described that authors have tried to combine SOM1 and SOM2 and use solo SOM trained with multiple subjects.However, the trained SOM did not work well at all.
2) Experimental results: Figure 6 (left) shows the constructed SOM1 with distribution of the mapping of the feature vector based on right hand operation of the forceps by the subject A. Figure 6 (right) shows the constructed SOM2 with distribution of the mapping of the feature vector based on right hand SEMG of the subject A. When the thread knotting was performed, the mapping of the feature vector was outputted as a plot of hexagons on the SOMs, and the plotted domain was recognized as the state of the procedure.Then, the distinction accuracy was calculated as the number of times in which the feature vector is mapped into correct domain divided by the number of times of the trial.
The distinction accuracy using SOM1 and SOM2 based on two-handed manipulation of the forceps and SEMG of both hands for the Subject A-D and the skilled surgeon is shown in Table 5.The results of the abnormality distinction and whole distinction accuracy for five subjects are shown in Table 6.In Table 5 and Table 6, the term "Abnormal" represents all abnormal manipulations, without distinguishing the state in each abnormal manipulation.
From Table 6, the abnormality could be distinguished with greater than 97% accuracy on average.In addition, each state of the abnormal manipulation could be distinguished with greater than 90% accuracy on average.
3) Consideration: In the states Grasp, Pull and Sudden, an incorrect distinction can be seen mutually.This is because the characteristic of the feature vector in these states is similar, and the difference does not appear clearly in the feature vector.However, this is not a serious problem in terms of an abnormality warning.Because, for the states Grasp, Pull and Sudden, a common acoustic warning is executed in an abnormality warning system described in the next section.

Abnormality Warning System
An abnormality warning system to reduce medical errors is constructed in application with the above results, in which a series of surgical procedures for ligation is automatically identified as one of the seven procedures.When the procedure is identified as thread knotting, distinction of the abnormality is performed.In addition, when the procedure is identified as an abnormal manipulation, the detection of the abnormal manipulation is communicated to the operator as a warning.The outline of the system is shown in Figure 7.When the surgical procedure was classified as thread knotting, distinction of five states, including three high-risk states, namely Normal, Posture, Grasp, Pull and Sudden, was first executed using SOM1.Next, only when the state of the procedure was identified as Normal by SOM1, distinction of two states, namely Normal and Straining, was executed using SOM2 on the basis of operator's SEMG.The reason that the abnormality distinction by SOM1 comes first is because SOM1 distinguishes abnormal manipulations with greater risk to the patient compared with SOM2.
The abnormal manipulations are classified into Posture and Straining as comparatively low-risk states, and Grasp, Pull and Sudden as high-risk states.For the former, the operator is warned of the detection of the abnormal manipulation by a change in the color of the box at the bottom of the monitor (Figure 8) and acoustically by a chime sound.For the latter, in addition to a visual and acoustic warning, a resistance force of the degree which does not influence the manipulation of the forceps was provided to the operator by the PHANTOM Omni.
Using the proposed system, the thread knotting was performed online for the simulation box.The system worked well as expected.In online distinction of the abnormality, approximately 0.2 s of time delay occurred due to the signal processing.However, it should be mentioned that this was not a cause for concern when performing a surgical procedure.

Conclusions
In this paper, a novel method for the automatic identification of a surgical procedure and online distinction of the abnormality of the identified surgical procedure was proposed.The surgical procedure for ligation was divided into seven procedures.First, on the basis of the threshold criteria, a surgical procedure was identified as one of the seven procedures automatically.Next, when the surgical procedure was identified as thread knotting, the procedure was automatically classified as either normal manipulation or abnormal manipulation using SOMs.
In order to evaluate the proposed method, using the simulation box built for laparoscopic surgery, the experimental works were executed.In the experiments, for both automatic identification of the surgical procedure and online distinction of abnormality for the identified surgical procedure, high recognition accuracy was obtained.Thus, it was concluded that the effectiveness of the proposed method was demonstrated.
In addition, an abnormality warning system to reduce medical errors was constructed.In this system, for comparatively low-risk states, the operator was warned of an abnormal manipulation by visually changing the color of the box at the bottom of the monitor and acoustically by a chime sound.For high-risk states, in addition to the visual and acoustic warnings, a resistance force was provided to the operator by the PHANTOM Omni.The validity of the system was also verified through a series of experiments for the simulation box.

Figure 4 .
The validity of this classification has been approved by some skilled surgeons of laparoscopic surgery.Each procedure for ligation is described as follows:1) Grasping (left): Grasping the suture with the left forceps; 2) Overwrapping (left) + Translation: Twisting around the suture held by left forceps on the axis of the right forceps, and moving the right forceps closer to the opposite side of the suture; 3) Grasping (left + right): Grasping the suture with both forceps; 4) Thread knotting: Thread knotting by pulling the suture; 5) Grasping (right): Grasping the suture with the right forceps; 6) Overwrapping (right) + Translation: Twisting around the suture held by right forceps on the axis of the left forceps, and moving the left forceps closer to the opposite side of the suture; 7) Neutral: The state where nothing is operating.

Table 3 .
Degree of danger assessed by skilled surgeons.

Figure 6 .
Figure 6.Distribution on SOM1 and SOM2 based on right hand operation of forceps by subject A.

Table 2 .
Logical definition of forceps operation.

Table 4 .
Identification accuracy rate for ligation.

Table 5 .
Distinction accuracy rate for abnormality.

Table 6 .
Results of abnormality distinction.