^{1}

^{1}

^{*}

Semi-supervised Support Vector Machines is an appealing method for using unlabeled data in classification. Smoothing homotopy method is one of feasible method for solving semi-supervised support vector machines. In this paper, an inexact implementation of the smoothing homotopy method is considered. The numerical implementation is based on a truncated smoothing technique. By the new technique, many “non-active” data can be filtered during the computation of every iteration so that the computation cost is reduced greatly. Besides this, the global convergence can make better local minima and then result in lower test errors. Final numerical results verify the efficiency of the method.

In the field of machine learning, it’s essential to collect a large amounts of labeled data for the purpose of training learning algorithms. However, for many applications, huge number of data can be cheaply collected, but manual labeling of them is often a slow, expensive and error-prone process. It’s desirable to utilize the unlabeled data points for the implementation of the learning task. The goal of semi-supervised classification is to employ the large collection of unlabeled data jointly with a few labeled data to finish the task of classification and prediction [11,18].

Semi-supervised support vector machines (S^{3}VMs) is an appealing method for the semi-supervised classification. In [^{3}VMs, such as convex-concave procedures [

As pointed out in [^{3}VMs is that the resulting optimization problem is non-convex that generates local minima. Hence, it’s necessary to find better local minima because better local minima tend to lead to higher prediction accuracy. In [

The focus of this paper is giving a new efficient implementation of the smoothing homotopy method for the S^{3}VMs. In Section 2, we first introduce the new S^{3}VMs model used in [^{3}VMs is recalled. In Section 3, the new truncated smoothing technique is established to give a more efficient pathfollowing implementation of the smoothing homotopy method. The new technique is based on a fact that, some “non-active” data do little effect on the value of the smooth approximation functions with their gradients and Hessian during the computation, as a result, these “non-active” data can be filtered by the new truncated technique to save the computation cost. With the inexact computation technique, only a part of original data is necessary during the computation of every iteration. In the last section, Two artificial data sets with six standard test data from [

A word about the notations in this paper. All vectors will be column vectors unless transposed to a row vector by a prime superscript T. The scalar (inner) product of two vectors x and y in the n-dimensional real space will be denoted by. For a matrix, will denote the th row of. For a real number, denotes its absolute value. For a vector,

denotes its 1-norm, i.e., , denotes its infty-norm, i.e.,. For an index set, denotes the element number of it. For a given function, if is smooth, its gradient is denoted by, if is nondifferential, denote its subdifferential as.

There lies several formulations for S^{3}VMs such as the mixed integer programming model by K.P. Bennett et al. [

Given a dataset consisting of m labeled points and p unlabeled points all in, where the labeled points are represented by the matrix, p unlabeled points are represented by the matrix and the labels for are given by diagonal matrix of. The linear S^{3}VMs to find a hyperplane far away from both the labelled and unlabeled points can be formulated as follows:

where

and are loss functions corresponding to the labeled data and unlabeled data respectively and defined as follows,

where denotes the absolute value of. The constraint is called balanced constraint that is used to avoid unbalanced solutions which classify all the unlabeled points in the same class.

For arbitrary vector, there lies an equivalent relation between its 1-norm and inf-norm in the sense that, then the sum term of model (1)

can be substituted by the inf-norm form and model (1) can be reformulated as follows,

where

.

We rewrite the constraint into the objective as a barrier term and reformulate into its equivalent formulation, and then obtain the following formulation that is our goal in the paper.

where

is a barrier parameter.

If the dataset is nonlinear separable, we need construct a surface separation based on some kernel trick (detailed discussion of it can be seen from [

is the kernel parameter,. To find the nonlinear decision surface, we need to solve the following problem:

where

Aggregate function is an attractive smooth approximate function. It has been used extensively for the the nonsmooth min-max problem [

In short, let (or,) and denote model (3) or (4) as the following unified formulation:

where

and are defined as (3) or (4).

Denote, based on the optimality condition of non-smooth optimization theory in [

where

moreover, a point can be called a stationary point or a solution point of (5) if satisfying.

To solve model (5) by an aggregate homotopy method, we first introduce the following two smoothing functions,

where is defined as (6) and

.

We call and as aggregate function and twice aggregate function respectively. The two smoothing functions have many good properties such as uniform approximation and etc. More details can be seen from [

Using above two uniformly approximations functions in (8), we define the following homotopy mapping:

where is an arbitrarily initial point and

.

We call Equation (9) as an aggregate homotopy equation. It contains two limiting problems. On the one hand as, it has a unique solution. On the other hand, as, the solution of it approaches to a stationary point of (5), i.e., a solution satisfying.

For a given initial point, we denote the zeros point set of the aggregate homotopy mapping as

It can be proved that includes a smooth path with no bifurcation points, starting from and approaching to the hyperplane that leads to a solution of the original problem [

The path-following of the homotopy path can be implemented by some predictor-corrector procedure. Some detailed discussion on the predictor-corrector algorithm with the convergence can be seen from [3,8] and etc. In the following, we first give the framework of the predictor-corrector procedure in this paper, and then discuss how to make an inexact predictor-corrector implementation.

Parameters. initial stepsize, maximum stepsize, minimum stepsize, stop criteria for procedure terminated, stop criteria for Newton corrector stopped, criteria for judging corrector plane, counter.

Data.Step 0., ,.

Step 1. Compute a predictor point

1) Solve the following linear equation

to obtain a unit tangent vector;

2);;

3) If or, , return to 2); else, go to Step 2;

Step 2. Compute a corrector point

1) If, take and; else, take. Go to 2);

2) Solve equation

by Newton method with the stopping criteria and go to 3);

3) If Newton corrector fail, , go to 2); else, denote the solution as, go to 4);

4) If or, , go to 2); else, go to 5);

5) If, stop ; else, k = k + 1, go to 6);

6). If and the iteration number of Newton corrector is less than 3, go to 7); else , go to 1);

7),; go to 1).

Notice that, the main computation cost of Algorithm 3.1 lies in the equations solver in Step 1) and 2), some inexact computation technique can be introduce to save the computation cost of them. we take the following approximate homotopy equation with its Jacobian in place of the original with during the computation of step 1) and 2).

Given parameters, , denote, ,

where

It’s proven in [

Denote

and, is the unit tangent vector obtained by the approximate computation, is the tangent vector by exact Euler predictor, we have the following lemma to guarantee the efficiency of the approximate tangent vector.

For a given, if is small enough and satisfies

The approximate unit predictor tangent vector is effective, i.e., still makes a direction with arclength increased.

During the correction process, the following equation must be solved

where is an appropriate predictor point obtained from the former predictor step. We adopt the following approximate Newton method to solve (11),

where

From 0 is a regular value of and is a unit tangent vector induced by, we know, if the step is chosen appropriately, the equation (11) has a solution and the approximate Newton iteration (12) is effective.

Suppose that have solution with nonsingular, there exists a neighborhood and, for any, if for each, and satisfying

.

Then the approximate Newton iteration point sequence from (12) is well defined and converges to.

In this section, some numerical examples and comparisons are given to illustrate the efficiency of our method. Two artificial datasets are generated first. The first one consists of 34 points generated by “rand” function, 14 of them are labeled and the remaining 30 are seen as unlabeled data. The second one consists of 242 points taken from two nonlinear bihelix curves that are generated by, where one is obtained by taking, the other is by taking,. We take randomly of them as labeled and the remaining as unlabeled. The comparisons of our method with the LSVM method from [

To reveal the efficiency of our algorithm for S^{3}VMs, comparisons of our algorithm (InSH) with some other existing algorithms for S^{3}VMs such as the convexconcave procedure in [^{3}VM), the continuation method in [^{3}VM) and the gradient descent method in [^{3}VM), are given for some standard test data from [

-: denotes the method fails for the dataset.

All the computations are performed on a computer running the software Matlab 7.0 on Microsoft Windows Vista with Intel(R) Core(TM)2 Duo CPU 1.83 GHz processor and 1789 megabytes of memory. During the computation, we take, , , , , , , are taken as the one for the least test error. If necessary, the kernel parameter is taken the best leads to the least test error among. The parameters for determining the inexact index set are taken as

where, , and. has the same expression as where and. and are taken as that are given to bound the error of and.

The research was supported by the National Nature Science Foundation of China (No. 11001092) and the TianYuan Special Funds of the National Natural Science Foundation of China (Grant No. 11226304).