J. Software Engineering & Applications, 2010, 3, 704-708
doi:10.4236/jsea.2010.37080 Published Online July 2010 (http://www.SciRP.org/journal/jsea)
Copyright © 2010 SciRes. JSEA
Design of Radial Basis Function Network Using
Adaptive Particle Swarm Optimization and
Orthogonal Least Squares
Majid Moradi Zirkohi, Mohammad Mehdi Fateh, Ali Akbarzade
Department of Electrical and Robotic Engineering, Shahrood University of Technology, Shahrood, Iran.
Email: m.moradi@ieee.org
Received May 12th, 2010; revised June 13th, 2010; accepted June 15th, 2010.
ABSTRACT
This paper presents a two-level learning method for designing an optimal Radial Basis Function Network (RBFN) using
Adaptive Velocity Update Relaxation Particle Swarm Optimization algorithm (AVURPSO) and Orthogonal Least
Squares algorithm (OLS) called as OLS-AVURPSO method. The novelty is to develop an AVURPSO algorithm to form
the hybrid OLS-AVURPSO method for designing an optimal RBFN. The proposed method at the upper level finds the
global optimum of the spread factor parameter using AVURPSO while at the lower level automatically constructs the
RBFN using OLS algorithm. Simulation results confirm that the RBFN is superior to Multilayered Perceptron Network
(MLPN) in terms of network size and computing time. To demonstrate the effectiveness of proposed OLS-AVURPSO in
the design of RBFN, the Mackey-Glass Chaotic Time-Series as an example is modeled by both MLPN and RBFN.
Keywords: Radial Basis Function Network, Orthogonal Least Squares Algorithm, Particle Swarm Optimization,
Mackey-Glass Chaotic Time-Series
1. Introduction
The radial basis function network (RBFN) as an alte-
native to the multilayered perceptron neural network
(MLPN) has been studied intensively [1]. The RBFN
has the universal approximation ability; therefore, the
RBF neural network can be used for the interpolation
problem. A Gaussian radial basis function is highly
nonlinear, and provides some good characteristics for
incremental learning with many well-defined mathe-
matical features [2]. It is a powerful scheme for learn-
ing, identification, equalization, and control of nonlin-
ear dynamic systems.
The training of feed forward ANN is based on non-
linear optimization technique; however, it may get
trapped at a local minimum during the learning proce-
dure using the gradient descent algorithm. The RBFN
is an alternative method for aforementioned method.
The performance of RBFN critically depends upon
the chosen RBF centers [3]. A new distance measure,
which is superior to the Euclidean distance, was ap-
plied for selecting the centers from highly correlated
input vector [4]. Another approach was proposed to
determine the centers of RBF networks based on sensi-
tivity analysis [5]. However, this approach has not
considered the role of spread factor that is a significant
factor to increase the accuracy of results. In contrast,
Orthogonal Least Squares (OLS) algorithm [6] selects
required number of RBF centers depending on the val-
ue of spread factor. The OLS employs the forward re-
gression procedure to reduce the size of RBFN result-
ing in an adequate and parsimonious RBFN.
The OLS algorithm has solved a crucial problem of
how to select RBFN centers very well; however, it
doesn’t give a method for selecting the spread factor of
an RBF [6].
The PSO was first introduced by Kennedy and
Eberhart in 1995 [7]. Through the simulation of a sim-
plified social system, the behavior of PSO can be
treated as an optimization process. As compared with
other optimization algorithms, the PSO requires less
computational time. Therefore, it has successfully been
applied to solve many problems [8-10].
This paper proposes a novel adaptive version of a
two-level learning method for constructing a RBFN
using a Velocity Update Relaxation Particle Swarm
Optimization (VURPSO) presented in [11]. The nov-
elty is to find the global optimum of spread factor pa-
Design of Radial Basis Function Network Using Adaptive Particle Swarm Optimization and Orthogonal Least Squares
Copyright © 2010 SciRes. JSEA
705
rameter at the upper level using adaptive velocity up-
date PSO namely AVURPSO that has more conver-
gence speed and accurate response than VURPSO. At
the lower level, it constructs a parsimonious RBFN
using the OLS algorithm.
This paper is organized as follows: Section 2 de-
scribes the RBFN. Section 3 formulates the PSO algo-
rithm and develops the AVURPSO algorithm. Section
4 introduces Mackey-Glass chaotic time-series. Section
5 presents simulation results and finally Section 6 con-
cludes the paper.
2. RBFN
An RBFN has a feed forward structure consisting of a
single hidden layer of locally tuned units which are fully
interconnected to an output layer of linear units, as
shown in Figure 1. All hidden units simultaneously re-
ceive the p-dimensional real valued input vector. The
input vector to the network is passed to the hidden layer
nodes via unit connection weights. The hidden layer con-
sists of a set of radial basis functions. The hidden layer
node calculates the Euclidean distance between the cen-
ter and the Network input vector and then passes the re-
sult to the radial basis function. All the radial basis func-
tions are, usually, of the same type. Thus the hidden layer
performs a fixed nonlinear transformation and it maps
the input space onto a new space. The output layer, then,
implements a linear combiner on this new space and the
only adjustable parameters are the weights of this linear
combiner. These parameters can be determined using the
linear least Squares method, which is an important ad-
vantage of this method. An RBFN is designed to perform
a nonlinear mapping from the input space to the hidden
space, followed by a linear mapping from the hidden
space to the output space. Thus, the network represents a
map from the p-dimensional input space to m-dimen-
sional output space, according to:
0
1
(,)
h
ii jijj
j
dww xc
 
(1)
Figure 1. RBF Structure
where for1, 2,...,im
,1, 2,...,jh
,
x
is an input vec-
tor. n,mand hare the number of input nodes, output
nodes, and hidden units respectively.
j
cis the j-th center
node in the hidden layer, j
x
c denotes Euclidean
distance, (.)
is a nonlinear transfer function called as
RBF, ji
wis the weighting value between the i-th center
and the j-th output node,
is the real constant known as
spread factor. Equation (1) reveals that the output of net-
work is computed as a weighted sum of the hidden layer
outputs. The nonlinear output of the hidden layer is radi-
cally symmetrical. In this paper, the most widely used
Gaussian function for the j-th hidden unit is chosen as
follows:
2
2
(,)exp( )
2
j
ji
j
xc
xc


 (2)
The accuracy is controlled by three parameters: the
number of radial basis functions or hidden units, centers
of the hidden units, and the spread factor.
A common learning strategy for an RBF network is to
randomly select some input data sets as the RBF centers
in the hidden layer. The weights between hidden and
output layer can then be estimated by using the stochastic
gradient approach. The main disadvantage of this method
is that it is very difficult to quantify how many numbers
of center should be adequate to cover the input vector
space. Furthermore, the training algorithm is possibly
getting stuck into local minimum. To overcome these
shortages, this paper develops the OLS-AVUURPSO to
construct the RBFN.
3. AVURPSO Algorithm
The PSO algorithm is performed as follows: the un-
known parameters are called the particles. Starting with a
randomly initialization, the particles will move in a
searching space to minimize an objective function. The
parameters are estimated through minimizing the objec-
tive function. The fitness of each particle is evaluated
according to the objective function for updating the best
position of particle and the best position among all parti-
cles as two goals in each step of computing. Each article
is directed to its previous best position and the global
best position among particles. Consequently, the particles
tend to fly towards the better searching areas over the
searching space. The velocity of i-th particle i
v will be
calculated as follows [7]:
11
22
(1) .()(()())
(()())
ii ii
i
vkwvk crpbestkxk
cr gbestkxk
 
 (3)
where in the k-th iteration, i
is the position of parti-
Design of Radial Basis Function Network Using Adaptive Particle Swarm Optimization and Orthogonal Least Squares
Copyright © 2010 SciRes. JSEA
706
cle, i
pbest is the previous best position of particle,
g
best is the previous global best position of particles,
wis the inertia weight, 1
c and 2
c are the acceleration
coefficients namely the cognitive and social scaling pa-
rameters,1
r and 2
r are two random numbers in the
range of [0 1]. It is worthy to note that the inertia weight
has not been in the first version of PSO [7]. If the inertia
weight in (3) is set to 1, the first version of PSO is ob-
tained.
The new position of i-th particle is then calculated as
(1)() (1)
iii
xkxk vk (4)
The PSO algorithm performs repeatedly until the goal
is achieved. Number of iterations can be set to a specific
value as a goal of optimization.
The first version of PSO has been improved in terms
of convergence and accuracy, so far. To control the ve-
locity, if the velocity i
v exceeds a maximum value of
max
v, then i
v is set to max
v. In many applications,
max
vhas been set tomax
x. In addition, the inertia weight
was proposed to control the velocity [12] as
max
12 2
max
()
kk
www w
k

 


(5)
where wdecreases from a higher value 1
w to a lower
value 2
w, and max
k is the maximum number of itera-
tion. Moreover, the velocity was modified to improve
the convergence [12] as
11
22
(1)
()(() ())
(()())
i
iii
i
vk
v kcrpbestkx k
crgbestkx k








(6)
where for 12
cc
 and 2
,
is given by
2
2
44

 (7)
Adopting low values for 1
c and 2
c allows the parti-
cle to roam far from the target regions before being
tugged back. On the other hand, adopting high values
results in abrupt movement toward or passes the target
regions. Therefore, 1
c and 2
c were introduced [12] as
max
111 1
max
max
222 2
max
()
()
if f
if f
kk
cccc
k
kk
ccc c
k

 



 


(8)
where 1i
cand 2i
c are the initial values of 1
c and 2
c,
and 1
f
cand 2
f
care the final values of 1
cand 2
c, re-
spectively. Actually, the best solutions were determined
over the full range of search for changing 1
c from 2.5 to
0.5 and 2
c from 0.5 to 2.5. With a large value of 1
c
and a small value of 2
c at the beginning, particles are
allowed to move around the search space instead of
moving toward thei
pbest . A small value of 1
c and a
large value of 2
c allow the particles converge to the
i
g
best in the latter part of the optimization.
In traditional PSO, the velocities of the particles are
limited in the range of [vmin; vmax]. Usually vmin and vma x
are set to xmin and xmax, respectively. The positions of the
particles are given in the range of [xmin; xmax]. Thus,
evaluating the obtained results according to the limits for
confining or rejecting the results takes extra computa-
tional burden. Velocity update relaxation particle swarm
optimization (VURPSO) postulates the boundary veloc-
ity validity checking without checking the validity of
positions in every iteration cycle.
In traditional PSO algorithm, the velocity is updated at
every iteration cycle. In contrast, in velocity-updating
relaxation [11], the velocity of each particle kept un-
changed if its fitness at current iteration is better than one
at preceding iteration; otherwise the particles’ velocity is
updated as stated by (3). As a result, the computational
efficiency is enhanced. The new position of particle is
then calculated as:
11
(1)( )
kkk
iii
x
mfxmf v
  (9)
where m
f
is called momentum factor given in the range
of 01mf
because the new position vector is a
point on the line between the former position vector, k
i
,
and the new velocity vector, 1k
i
v. In many applica-
tions, m
f
was given a constant. VURPSO exhibits to
have strong global search ability at the beginning of the
run and strong local search near the end of the run. The
use of velocity update relaxation in traditional PSO helps
to reduce the computational efforts.
In order to speed up the convergence speed, we propose
a novel adaptive VURPSO strategy named AVURPSO. In
this new strategy we change the momentum factor adap-
tively as follow:
max
121
max
()
kk
mf mfmfmfk

 

(10)
where m
f
decreases from a higher value 1
mf to a
lower value 2
mf . Moreover, we use (6) instead of (3).
4. Mackey-Glass Chaotic Time-Series
The Mackey-Glass Chaotic Time-Series [13] is stated as
Design of Radial Basis Function Network Using Adaptive Particle Swarm Optimization and Orthogonal Least Squares
Copyright © 2010 SciRes. JSEA
707
10
0.2 ()
()0.1 ()1()
xt
xtxt xt
 
(11)
where we set 17
and ()1.2xt
 for0t
.
The Mackey-Glass Chaotic Time-Series is modeled by a
RBFN.
This time series is chaotic, and so there is no clearly
defined period. The series is not converged or diverged
and the trajectory is highly sensitive to initial condition.
The input training data for RBF predictor is a four-di-
mensional vector in the following form of
()[(18) (12) (6) ()]wt xtxtxtxt   (12)
The output training data corresponds to the trajectory
prediction.
()( 6)yt xt (13)
A set of data with 1000 samples is obtained. We use
the first 500 samples for training and the second 500
samples for validation (Test Data). The data is shown in
Figure 2.
5. Simulation Results
To verify the performance of proposed method we pre-
sent two comparisons. First of all, AVURPSO and
VURPSO are compared in the Design of RBFN. Then,
the RBFN and the MLPN are compared in Modeling of
the Time Series.
5.1 Comparing AVURPSO and VURPSO in the
Design of RBFN
As mentioned in previous section, the input of RBFN is
the train data. For a given value to the spread factor, the
OLS algorithm provides an optimum number of centers
(NC) in RBFN from the training patterns. Next, it esti-
mates the bias vector and weighting matrix using least
square error technique for the prescribed sum of squared
errors (SSE).
The RBFN is trained using OLS algorithm while we
use the training patterns, the values of
given in Ta-
ble 1, and 0.01SSE . We should find the optimum
0200 400 600 8001000
0.4
0.5
0.6
0.7
0.8
0.9
1
1.1
1.2
1.3
1.4
Tim e (Se c)
Mackey-Glass Chaotic Time Series
Figure 2. Chaotic time-series behavior
value of spread factor to improve the results since
significantly affects the NC as confirmed by Table 1.
The Fitness function is defined in order to optimize the
value of spread factor as follow:
2
1
1()
Q
real net
i
F
itnessy yNC
Q

(14)
where Qdenotes the number of samples, while real
y
is the real output, and net
y is the desired network out-
put. In this approach, to escape from the local minima the
fitness function is changed to
6
2
1
1()
Q
real net
i
F
itnessy yNC
Q





(15)
The number of particles and the maximum value of it-
erations are selected 12 and 50, respectively. And, m
f
is
varied from 0.5 to 0.3. Table 2 presents the obtained
optimal value of spread factor, and the NC from these
two methods.
The AVURPSO has obtained 0.649
resulting in
a less NC and a less MSE in both sets of train data and
test data. Moreover, the AVURPSO has a higher speed of
convergence as shown in Figure 3.
Table 1. The role of spread factor in determining the RBFN
centers and MSE
0.01
0.1
0.8
1
NC 490 162 42 499
MSE
(Train) 0.007 0.0096 0.0092 0.0227
MSE (Test)87.94 0.012 0.0086 0.0022
Table 2. Optimal value of spread factor and the NC obtain-
ed from the VURPSO and AVURPSO methods
method
NC MSE
(Train)
MSE
(Test)
VURPSO 0.66 42 0.0088 0.0085
AVURPSO 0.649 36 0.0083 0.0079
010 20 30 40 50
36
36.5
37
37.5
38
38.5
39
39.5
iteration
Cost Function
AVURPSO
VURPSO
Figure 3. Comparing the convergence speed
Design of Radial Basis Function Network Using Adaptive Particle Swarm Optimization and Orthogonal Least Squares
Copyright © 2010 SciRes. JSEA
708
Table 3. A comparison on computing time
MLPN RBFN algorithm
21.63
3.81
CPU Time
5.2 Comparing RBFN and MLPN in Modeling
of the Time Series
First a MLPN with optimum topology is designed. Then,
we compare the RBFN with the optimum MLPN in mod-
eling of the time series algorithm. The architecture of
MLPN consists of an input layer, one or more hidden
layers, and an output layer. The MLPN is trained with
Levenberg-Marquardt back propagation. To obtain the
MLPN with optimal topology, different topologies are
tested by AVRPSO algorithm as
1) One hidden layer with 7 neurons.
2) Two hidden layers with variant neurons between 1
to 20 at each.
The optimum topology is obtained (8, 5, 1) from op-
tion 2.
Now, we compare RBFN and the MLPN in modeling
the time series on the computing time as shown in Table
3.
The RBFN possibility needs more neurons than MLPN;
however, RBFN often can be designed in a fraction of
time that it takes to train MLPN.
6. Conclusions
A two-level learning method has been presented for de-
signing the RBFN using OLS-AVURPSO method. The
proposed method at the upper level finds the global op-
timum of the spread factor parameter using the
AVURPSO algorithm while at the lower level automati-
cally constructs the RBFN using the OLS algorithm. To
verify the performance of proposed method, two com-
parisons have been presented. First, the AVURPSO algo-
rithm and the VURPSO algorithm are compared in the
design of RBFN. Second, the RBFN and the MLPN are
compared in the modeling of the time-series. The superi-
ority of the AVURPSO algorithm to the VURPSO algo-
rithm is verified due to obtaining a less NC, a less MSE
and a higher speed of convergence. In the modeling of
the Mackey-Glass time-series, simulation results confirm
that the RBFN is superior to MLPN in terms of the net-
work size and computing time.
REFERENCES
1 S. Chen, S. A. Billings, C. F. N. Cowan and P. M. Grant,
“Non-Linear Systems Identification Using Radial Basis
Functions,” International Journal of Systems Science,
Vol. 21, No. 12, 1990, pp. 2513-2539.
[2] M. M. Gupta and L. Jin, “Static and Dynamic Neural
Networks,” John Wiley, 2003.
[3] R. Segal and M. L. Kothari, “Radial Basis Function (RBF)
Network Adaptive Power System Stabilizer,” IEEE Tran-
sactions on Power Systems, Vol. 15, No. 2, 2000, pp.
722-727.
[4] S. A. Billings and X. Hong. “Dual Orthogonal Radial Basis
Function Networks for Nonlinear Time Series Prediction,”
Neural Networks, Vol. 11, No. 3, 1998, pp. 479-493.
[5] D. Shi, D. S. Yeung and J. Gao. “Sensitivity Analysis
Applied to the Construction of Radial Basis Function
Networks,” Neural Networks, Vol. 18, No. 7, 2005, pp.
951-957.
[6] S. Chen, C. F. N. Cowan and P. M. Grant, “Orthogonal
Least Squares Learning Algorithm for Radial Basis Func-
tion Networks,” IEEE Transactions on Neural Networks,
Vol. 2, No. 2, March 1991, pp. 302-309.
[7] J. Kennedy and R. Eberhart, “Particle Swarm Optimiza-
tion,” Proceedings of IEEE International Conference on
Neural Networks, Vol. 4, 1995, pp. 1942-1948.
[8] S. Naka, T. Genji, T. Yura and Y. Fukuyama, “A Hybrid
Particle Swarm Optimization for Distribution State Esti-
mation,” IEEE Transactions on Power Systems, Vol. 18,
No. 1, 2003, pp. 60-68.
[9] M. Clerc, “The Swarm and the Queen: Towards the De-
terministic and Adaptive Particle Swarm Optimization,”
Proceedings of the Congress on Evolutionary Computa-
tion, Washington, DC, 1999, pp. 1951-1957.
[10] A. Alfi and M. M. Fateh, “Parameter Identification Based
on a Modified PSO Applied to Suspension System,”
Journal of Software Engineering & Applications, Vol. 3,
2010, pp. 221-229.
[11] A. Chatterjee, “Velocity Relaxed and Craziness-Based
Swarm Optimized Intelligent PID and PSS Controlled
AVR System,” Electrical Power and Energy Systems,
Vol. 31, No. 7-8, 2009, pp. 323-333.
[12] A. Ratnaweera and S. K. Halgamuge, “Self Organizing
Hierarchical Particle Swarm Optimizer with Time-Varying
Acceleration Coefficient,” IEEE Transactions on Evolu-
tionary Computation, Vol. 8, No. 3, 2004, pp. 240-255.
[13] L. Wang, “A Course in Fuzzy Systems and Control,”
Prentice-Hall International, 1997.