Pruned fuzzy K-nearest neighbor classifier for beat classification

Arrhythmia beat classification is an active area of research in ECG based clinical decision support systems. In this paper, Pruned Fuzzy K-nearest neighbor (PFKNN) classifier is proposed to classify six types of beats present in the MIT-BIH Arrhythmia database. We have tested our classifier on ~ 103100 beats for six beat types present in the database. Fuzzy KNN (FKNN) can be implemented very easily but large number of training examples used for classification can be very time consuming and requires large storage space. Hence, we have proposed a time efficient Arif-Fayyaz pruning algorithm especially suitable for FKNN which can maintain good classification accuracy with appropriate retained ratio of training data. By using Arif-Fayyaz pruning algorithm with Fuzzy KNN, we have achieved a beat classification accuracy of 97% and geometric mean of sensitivity of 94.5% with only 19% of the total training examples. The accuracy and sensitivity is comparable to FKNN when all the training data is used. Principal Component Analysis is used to further reduce the dimension of feature space from eleven to six without compromising the accuracy and sensitivity. PFKNN was found to robust against noise present in the ECG data.


INTRODUCTION
Arrhythmias result due to improper pacing of the cardiac muscle or any discrepancy in the electrical conduction network of the heart [1].Detection of these pathologically significant arrhythmias is an imperative task in the diagnosis of cardiac diseases.Electro-cardiograph (ECG) can be used as a non-invasive diagnostic tool for the detection of these disorders.With the development in computing and sensor technology, standalone automated ECG based decision support systems are an active area of research.A clinical decision support system includes ECG acquisition, pre-processing and noise removal (baseline variation, electronic and electromyographic noise etc.), ECG Delineation (for detection and delineation of P, QRS and T waves of ECG), feature extraction and beat classification.
A variety of methods exist in the literature for QRS delineation [2] which rely upon derivative based methods, use of digital filters and filter-banks etc.One of the most promising approaches for QRS detection and delineation has been proposed by Martínez et al. [3] as it offers very high detection and delineation accuracy.It uses wavelet domain analysis for performing QRS detection and delineation which is particularly suited to the ECG signal due to the non-stationary nature of the signal.
ECG beat classification, being an integral part of any ECG based automatic decision support system, has been studied by a number of researchers.Different feature extraction methods for beat classification include use of Fourier Transform [4], multi-resolution analysis [5], wavelet transform [6][7][8][9], independent component analysis [10], morphological analysis [11] etc.For the purpose of beat classification, literature reports a variety of classifiers such as Backpropagation Neural Networks [8], Learning Vector Quantization and Probabilistic Neural Networks [6], Fuzzy Inference Systems [12], Nearest Neighbor classifiers [13] etc.
In our previous work [9], we have used features extracted from two-level wavelet decomposition of an ECG signal.The wavelet decomposition was performed through algorithm a'trous using the wavelet proposed by Martínez et al. [3] for QRS delineation.This wavelet offers inherent noise suppression and eliminates the need of re-evaluation of wavelet coefficients for beat classifi-cation as these are already obtained during QRS detection and delineation.A simple K-nearest neighbor (SKNN) classifier has been employed for the classification of 6 types of beats (Paced Beats (PB), Atrial Premature Beat (APB), Premature Ventricular Contraction (PVC), Normal (N), Left and Right Bundle Branch Blocks (LBBB & RBBB)) to give an accuracy of ~ 99.5% over selected records (23,200 beats only) from the MIT-BIH Arrhythmia database [14] with high noise tolerance and robustness against decrease in the size of the training data set.
Simple K-Nearest Neighbor (SKNN) classifier used in our previous work offers many advantages over other classifiers including simplicity and ease of parallel implementation, adaptability and online learning [15,9].Moreover, we have demonstrated its high accuracy for beat classification in comparison to other existing approaches.SKNN classifier assigns equal weights to all of the K-nearest neighbors regardless of their distances from the query point.An improvement over the SKNN classifier is the Fuzzy K-Nearest Neighbor classifier (FKNN) [16] which uses concepts from fuzzy logic to assign degree of membership of the given query point to different classes while considering the distance of its K-nearest neighbors.Points closer to the query point contribute a larger value to be assigned to the membership function of their corresponding class in comparison to far away neighbors.Class with the highest membership function value is taken as the winner.Fuzzy KNN gives class memberships for a beat to be classified as compared to true or false decision by SKNN.In case of comparable class memberships for winner and runner up classes, a confidence metric can be used on the decision.
Inherent variability of the ECG signal for different individuals, variability over age, amongst different beat classes and within each beat class itself [1] requires large amount of training data for effective training of an ECG beat classification system.Therefore, an instance based classifier like SKNN or FKNN can only be efficient in terms of both time and space complexities while offering high classification accuracy when number of training examples is very large.In such a case, amount of memory required to store the training prototype set and time required for finding the distance or the nearest neighbors of the query point can be tremendous.Solution to this issue is to use a pruning algorithm on the training data removing some data points from the training dataset without greatly affecting classification accuracy.A variety of pruning algorithms exist in the literature.A very good introduction to pruning techniques for instance-based learning algorithms is given in [17].In this paper, we have proposed a new pruning algorithm that can be integrated in FKNN and pruning time is very small as compared to other pruning methods.
In this paper, we present a beat classification algorithm which inherits its noise robustness from the use of ten wavelet domain features and the instanttaneous RR interval as used in [9].Use of the same wavelet transform for delineation of the QRS complex through [3] eliminates the need of re-evaluation of wavelet coefficients for beat classification, thus reducing over all time complexity of the system.To reduce the space complexity the dimensionality of the feature space can be reduced using Principal Component Analysis (PCA).A Pruned Fuzzy K-nearest neighbor (PFKNN) classifier is proposed for beat classification.Training data is pruned first and fuzzy decision of this classifier is weighted with respect to a priori probabilities of different classes to handle the class imbalance problem.To further reduce time complexity of PWFKNN, an efficient nearest neighbor search called ATRIA [18] has been used.

ARRHYTHMIA BEAT CLASSIFICATION
Architecture of an ECG based clinical decision support system is shown in Figure 1.Beat classification module of the system includes feature extraction, normalization of features, dimensionality reduction if applicable and Classification.Details of each of these components are given in detail as under.

Feature Extraction
For feature extraction, we have used the same wavelet as in [3] with wavelet transform implemented through Algortihm a' trous.The wavelet is taken to be the derivative of a low pass filter which offers inherent noise suppression.This wavelet is given by, ( ) From the implementation viewpoint, it can be implemented through FIR low pass (H) & high pass (G) filters whose frequency responses are given by: ( )  For details, please refer to [9].The same wavelet transform can be used for detection and delineation of the QRS complex.For the purpose of beat classification, we have used wavelet coefficients of a 64 point window centered at the QRS fiducial point, only up to scale 2 2 .Following eleven features are extracted from the ECG signal: 1) Variance of the original QRS complex signal denoted by 2  σ S 2) Variance in each sub-band denoted by 2 3) Variance of the autocorrelation function of wavelet coefficients in each sub-band denoted by ,,,,,,,, ,, σσσσσσσ SDRDDDRDDARAA rrrRR for a single beat.

Normalization
A normalization process is necessary to standardize all features to the same level.Tangent sigmoid function is used for the normalization as given below, where j x and σ

Dimensionality Reduction
Features extracted from a beat except the RR interval are subjected to Principal Component Analysis (PCA) for dimension reduction.RR interval is treated separately because of its temporal nature.A covariance matrix is formed on the basis of the first ten features and its eigen-values and eigen-vectors are computed.Five of the ten eigen-vectors corresponding to the highest eigenvalues are retained as they capture about 98% of energy in the features [9].Input data is then projected onto these bases and normalized RR interval values are appended to the projected feature set that result in a 6 dimensional feature space.

Pruned Fuzzy K-Nearest Neighbor Classifier (PFKNN)
Consider a training set T and class label of a point x in the training set is denoted by c(x).Fuzzy K-Nearest Neighbor search is used in training and classification of PFKNN.It is explained as follows.

Fuzzy K-Nearest Neighbor Search
Fuzzy KNN search is similar to simple KNN search.In simple KNN, every data point can belong to only one class which is the majority class in the K-nearest neighbor search.Whereas in fuzzy KNN, a data point can belong to multiple classes with different membership functions associated to these classes.Fuzzy KNN is described as follows, Step 1: Find K nearest neighbor x j , j = 1…K of the given query point x using Euclidean distance from a set of stored data points using Fast nearest neighbor search through ATRIA [18].
Step 2: Evaluate the membership function value of each of the N c classes (c i , i = 1… N c ) using the following relation.
where k i is the number of points from the original training set among the K nearest neighbors of x p that belong to the same class as x p itself.
The parameter m is used to control the effective magnitude of distance of the prototype neighbors from the query point and it can be selected through cross validation along with K.If m is taken to be infinity then the classifier reduces to a SKNN classifier.
Step 3: The class label of the query point x, c (x), is chosen as follows:

Proposed Pruning Method
It involves pruning of the training data set T to obtain the prototype set P. Following steps explain our proposed Arif-Fayyaz Pruning Algorithm.
Step 1: Start with an empty prototype set, φ = P and training set T.
Step 2: Find K-Nearest Neighbors, x j , j = 1…K, of each training point, x, such that ( ) ( ) ≠ j cxcx and add them to the prototype set P. This gives us the border points of different clusters in the data.
Step 3: Classify each training point using the prototype set P through FKNN explained in Subsection 3.1.If the training point is misclassified, add it to the prototype set P and re-evaluate class weights and membership values of prototype set P. This is done in order to accommodate any clusters which may have been missed in Step 2.
Step 4: For each i th class in the training set, Initialize a set Pruned , where N c Pruned is the number of prototypes in P for i th class.For each training point in the class, find the winner from the pruned set of same class.After all the training points are finished, remove the entire prototype from set P whose W (i) is an empty set.
The prototype set P obtained after Step 4 will be a pruned set of prototypes obtained from the training set T. In the next step, class weights are calculated to deal with the data imbalance problem.

Incremental Policy in the Pruned Set using Arif-Fayyaz Pruning Method
Once pruned prototype is set for a certain training data points, it is very easy and efficient to include any new data points available in the later time without effecting already pruned prototype set.New training data point can become member of pruned prototype set if it is misclassified with the existing pruned prototype set.

Classification Using PFKNN
Classification involves calculation of the membership function values of an unknown query point using FKNN from the stored prototype training data set P obtained after pruning and assigning its class label.

DESCRIPTION OF DATABASE
The MIT-BIH Arrhythmia Database is used in this paper for beat classification using PFKNN.It contains twochannel ambulatory ECG recordings from 47 subjects studied by the BIH Arrhythmia Laboratory between 1975 and 1979.ECG recordings were digitized at the sampling rate of 360 Hz with 11-bit resolution.We have used the annotations of the cardiologist originally provided by the database.We have used six types of beats (Paced Beats (PB), Atrial Premature Beat (APB), Premature Ventricular Contraction (PVC), Normal (N), Left and Right Bundle Branch Blocks (LBBB & RBBB)) from the MIT-BIH Arrhythmia database.Number of beats is plotted in Figure 2 for all six beat types.It can be seen in the figure that Normal beats dominate the database and rest of the beats are also not equally represented.

Results & Discussion
In this section, results of classification of the six types of cardiac rhythms are presented.where N error is the number of misclassifications and N test is the total number of testing beats for all classes.4) Mean & Standard Deviation of the Geometric Mean of Sensitivity values (G) over five runs with random selection of training and testing data.G is given by,

Performance Metrics
This measure is used to assess the performance of different classifiers while dealing with the class imbalance problem.
Performance measure for analyzing the performance of pruning algorithms is defined as retained ratio, R, given as below, No. of points in the prototype set (after pruning) No. of points in the training set (before pruning) = R (13) We have also tested noise robustness of the classification by adding uniform Gaussian noise of different intensity levels to the ECG signals and analyzed the results.The level of noise in the signal is quantized through the Signal to Noise ratio given by,

Comparison of Different Pruning Algorithms
In ECG based beat classification problem, number of training beats can grow very large and an efficient pruning method is required.Pruning of data for different classes is a difficult task when the separation boundary is nonlinear and complex and representation of classes is sparsely clustered.Checkerboard data is an example of multi-modal two class problem as shown in Figure 3(a) and Spiral data corresponds to two class separable problem having complex separation boundary as shown in Figure 3(b).In this section, our proposed Arif-Fayyaz pruning algorithm is compared with the other pruning algorithms.We have used the implementation of Wilson et al. [17] of different pruning algorithms.For the comparison 10 fold cross validation results are presented with K = 1 and m = ∞ (FWKNN being used as SKNN) over two different data sets.
In Table 1, different pruning methods are compared with our proposed Arif-Fayyaz pruning method.The accuracy of the proposed pruning approach is comparable to other approaches over these data sets.We have compared different algorithms in terms of Pruning Time, Accuracy after pruning and Retained fraction R. All algorithms were run on PIV 2.26 GHz PC with 512 Mb RAM.
In the first row of   Arif-Fayyaz pruning method in terms of accuracy but the retained ratio of ENN is very high (more than 98% in both cases).Explore method offers least retained ratio but its classification accuracy is very poor.Incremental policy of Arif-Fayyaz pruning algorithm is very simple and straightforward.These results clearly indicate the advantage in terms of computational complexity of using Arif-Fayyaz pruning method.
Our proposed approach offers a suitable and manageable reduction factor and faster pruning.Our proposed Arif-Fayyaz pruning method is used in the rest of paper for pruning.

Classification Results of PFKNN
Firstly, we present the result of fuzzy KNN with parameter values of k = 5 and m = 1.5.No pruning or class weights are used.We have divided total number of beats into two sets; training set include 50% of total number of beats (51600 beats) and 50% testing beats (51599 beats).Beats are selected randomly for training and testing sets.
Classification results for FKNN based classification are given in Table 2 with 11 features as explained in Subsection 2.1.Effect of noise on classification accuracy in terms of SNR as explained in Subsection 3.1 is also illustrated in Table 2. Overall accuracy is dropped by only 1% for SNR equals to 20 dB.This shows good robustness of FKNN against Gaussian noise.
In PFKNN, our proposed Arif-Fayyaz pruning method is used to prune the training data set of 51600 beats and pruned feature set is used to classify the testing set of 51599 beats of six classes.Results of classification with and without noise are illustrated in Table 3. Overall accuracy and geometric mean of sensitivity G are given in the 9 th column as A (G) and retained ration R is given in the last column of the table.It can be observed from Table 3 that with only 19% retained ratio of the training feature set, overall accuracy is dropped by only 0.3% and geometric mean of sensitivity G is dropped by only 0.2%.Hence our proposed PFKNN method offers advantage in terms of space and computational complexity.
Principal Component Analysis (PCA) is used to reduce the number of features.Considering the eigen-values of the resultant principal components, six principal components are selected and 11-dimensional feature space is projected on six-dimensional space.Table 4 shows overall accuracy, geometric mean of sensitivity and retained ratio considering no noise and different level of noise.It can be observed from the Table 4 that with the same overall accuracy and geometric mean of sensitivity it can further reduce the computational complexity of the classifier.

Analysis of Results
Main advantage of using fuzzy KNN over simple KNN is the membership value of each class for a particular query beat.Hence it gives a much more informed decision by inclusion of a higher level decision process.To accomplish this, we can first normalize the firing strengths of each class and then use the following classification rule: If the normalized firing strength of the runner-up class lies within a certain threshold of that of the winner class then we can assume the winning class label to be doubtful and take the runner-up class label in to consideration as well.Mathematically the winner class label is considered to be doubtful if, where θ is the threshold value and it is user specific.If we take winner and runner-up classes into consideration when inequality Eq.15 is satisfied, we can consider the beat classification as correct if winner or runner-up belongs to correct class.Following this strategy, overall accuracy versus threshold value is plotted in Figure 4.
We can observe from the figure that accuracy increases with the relaxation of threshold value.Many researchers have compared their methods with the other existing methods present in the literature.Table 6 shows the comparison of the proposed approach with other methods in the literature.In our view, a fair comparison of methods is not possible as most of the researchers have focused on different set of beat types and tested their methods on selected number of records present in the MIT-BIH database.From the table, we can observe that accuracy of our method is comparable to other methods although we have tested our method on whole set of six beat types present in the MIT-BIH database.Moreover, our feature set is very small (only six) and exhibits good time and space complexity.While using a limited set of 23200 beats, our accuracy is dropped by only 0.5% as compared to [30].

CONCLUSIONS
Pruned fuzzy KNN (PFKNN) classifier is proposed for arrhythmia beat classification that can offer reduced computational complexity and simple incremental policy.
A new pruning algorithm is proposed especially suited for KNN based classifiers that can prune the data efficiently in less computational time without compromising the accuracy of the classifier.PCA is used to further reduce the feature set to only six features per beat.Results have proved that PFKNN can offer better computational complexity than FKNN without compromising the classification accuracy.Hence PFKNN is a suitable option for online implementation of such clinical decision support system for Arrhythmia beat classification that demands less space and time complexity.Because of simplicity of the proposed classifier it is very easy to increment further training data of similar beat types or new beat types.

Figure 1 .
Figure 1.Architecture of a ECG based clinical decision support system.

σ RD 4 )
Ratio of minimum to maximum wavelet coefficient in each sub-band denoted by These features are combined with the instantaneous RR interval to produce a feature set given by and the variance of the j th component of the feature vector.This function will normalize the range of features to [-1,1].The normalized feature set for the k th beat is denoted by 1 k F .
value of the point j x for class c i .These membership values are calculated from the stored data points.For each point, x p, in the training set membership values for each class are as follows,

σ s and 2 σ
e are the power of the signal and noise respectively.

Figure 4 .
Figure 4. Plot of overall accuracy a versus threshold value.

Table 1 ,
result of simple KNN is presented without any pruning.The maximum accuracy achieved in two data sets is 98.33% and 99.97% for checkerboard and spiral data respectively by retaining all the training examples.Our proposed Arif-Fayyaz pruning method has achieved the accuracy of 97.54% and 99.83% respectively while retaining only 19.86% and 7.72% training examples.Other pruning methods are

Table 1 .
Comparison of different pruning algorithm for checkerboard and spiral data.

Table 4 .
Beat classification using pruned fuzzy KNN with reduced feature space using PCA.

Table 5
illustrates the time and space complexity of standard FKNN algorithm and our proposed PFKNN algorithm with fast nearest neighbor search using ATRIA.It can be seen from the table that PFKNN is very time efficient with small retained ratio.Overall accuracy and geometric mean of sensitivity of standard FKNN and PFKNN are also comparable.Hence, it highlights the effectiveness of using PFKNN. θA

Table 5 .
Time and space complexity for standard FKNN and PFKNN.

Table 6 .
Comparison of beat classification techniques.