Optics and Photonics Journal, 2013, 3, 76-78
doi:10.4236/opj.2013.32B019 Published Online June 2013 (http://www.scirp.org/journal/opj)
Copyright © 2013 S ciRes. OPJ
Image Fusion Real-time System Based on
FPGA and Multi-DSP*
Feng Qu, Bochao Liu, Jian Zhao, Qiang Sun
Changchun Institute of Optics, Fine Mechanics and Physics Chinese Academy of Sciences, Changchun, China
Received 2013
ABSTRACT
In order to solve complex algorithm that is difficult to achieve real-time processing of Multiband image fusion within
large amount of data, a real-time image fusion system based on FPGA and multi-DSP is designed. Five-band image
acquisition, ima ge regi stra tio n, image f usi on and disp la y outp ut ca n be d one within t he sys te m whic h uses FPG A as the
main processor and the other three DSP as an algorithm processor. Makin g ful l use of Flexible and hig h-speed characte-
ristics of FPG A, while an image fusion algorithm based on multi-wavelet trans for m is opti mized a nd ap plied to the s ys-
tem. The final experimental results show that the frame rate of 15 Hz, with a resolution of 1392 × 1040 of the five-band
image can be used by the system to complete processing within 41ms.
Keywords: Multi-band; Real-time Image Fusio n; Multi-wav e le t Transform; Image Registration
1. Introduction
Image fusion which is a branch of the data fusion con-
solidates two or more source images come from comple-
mentary band sensors, and is an image processing me-
thod which makes the useful information integrated into
a unified image for observation or further more accu-
rately full recognition, analysis and judgment to the tar-
get or scene through some fusion system [1-3]. Due of
very limited data infor matio n conve yed of a s ingle band ,
which is often difficult to meet the demand, Multisensor
image fusion has the information that single source im-
age can’t be captured, that is Co mplementary betwe en o f
multisource image. So multi-source image fusion can
obtain information from multiple viewpoints of expand
of se nsing range of time and space, to improve the accu-
racy and robustness of observation.
Seen from the definition of image fusion and co mplete
fusion mainly includes two parts, namely the image
source acq uisition and image fusion system, the core task
of image fusion system is the implementation of image
fusion algorithm. With the fusion algorithm improve-
ments, as well as the amount of data increases, the image
fusion system developed becomes with considerable dif-
ficulty. Therefore, how to design a reasonable real-time
image fusion hardware system is a prerequisite, and a
suitable image fusion algorithm image fusion system is
critical.
In this paper, in accordance with the requirements of
the actual project, an image fusion system based on FPGA
and multi-DSP is designed, and an image fusion algo-
rithm based on multi-wavelet transform is optimized, and
the frame rate of 15 Hz, with a resolution of 1392 × 1040
of the fi v e-ba nd image is processed in real-time at last.
2. Structure of System
Image fusion system hardware design is mainly three
types: single DSP or DSP parallel processing program;
FPGA + DSP processing program; a large FPGA proc-
essing program [4-6]. With the combination of these
strengths and weaknesses of the program, we have
adopted the FPGA + multi-DSP processing program as
our i mage fusio n s yste m. T his way by t he F PGA to co m-
plete the pre -processing part and the corresponding logic
control, complete the core algorithm in the DSP part, has
the advantage of more flexible and can be formulated to
give full pla y to the FPG A a nd D SP r espe ctive st rengt hs.
also with the difficulty is how to coordinate communica-
tion and data transfer between the FPGA and DSP, to
ensure that the in entire treatmen t can be quickly and
efficiently. Therefore, we designed the system structure
shown in Figure 1, Which FPGA chosen Alter Corpora-
tion Cyclong II series EP2C70F896, and DSP chosen
TMS320C6416 produced by TI.
In this system, three-way camera data is collected di-
rectly by the FPGA, and after preprocessing, the data is
*Supported by the program of academy-locality cooperation of
the
Chinese Academy of Sciences (2011CJT0004) , the Jilin province
science and technology development plan item (2 0090557 and

F. QU ET AL.
Copyright © 2013 S ciRes. OPJ
77
FP GA
DSP A
DSP B
DSP C
Visible light
camera
Near Infrared
Camera
UV camera
R
G
B
FIFO
F
I
F
O
FIFO
F
I
F
O
Figure 1 . Schematic diagram of the image fusion s yst em based on FP G A and DSP.
transferred to the DSP A and DSP B, with image regis-
tration work in them. Visible image is chosen as refer-
ence image because of more detail texture characteristics,
and near-infrared image is registered in DSP A, while
UV i mage is do ne in D SP B , then five-band image fusion
work is completed by DSP C. Finally, the fused image is
transferred to FPGA and output by FPG A.
In order to improve the efficiency of the transmission,
as well as to coord inate the timing of the various parts of
the asynchronous, we ta ke advantage of the FPGA inter-
nal structure generated FIFO for data exchange between
camera data acquisition and FPGA and DSP. The entire
process of using the pipeline work, as long as the time
occupied by the image registration and image fusion
processing must not exceed 66.7 ms to complete
real-time processing of multi-band image.
3. Multi -w avelet Transform Algorithm
Multi-wavelet is the development of wavelet theory, and
which is the wavelet generated by two or more scaling
function. Multi-wavelet can have many good properties,
such as symmetry, short orthogonality and higher order
vanishing moments, so multi-wavelet have more advan-
ages than a single wavelet [7].
3.1. Definition of Continuous Mul ti -wavelet
Transform
Assuming multi-wavelet as Ψ = （ψ1，…，ψr）T ∈
L2(R)r, and for a, b ∈ R, a ≠ 0, and by dilation and
translation, the orthonormal basis on L2(R)r is generated
as below equation (1):
1
2
,() ||() ab xb
xa a ψ
−−
= Ψ
(1)
For any f = (f1，…，fr) ∈L2(R)r, f in Ψ on t he con-
tinuous multi-wavelet transform is
, ,,,
1
( ),()()
r
wav
ab ababi
i
Tfff xxdx ψ
=

=Ψ=


∑
∫
(2)
3.2. Mul ti -wavelet Decomposition and
Reconstruction Algorithm
Let f ∈VJ , then we have equation (3):
00 0
,,
1
, ,,,
11
() ()
()()
rll
JkJk
l kZ
rl lll
J kJkjkjk
lkZlJ jJkZ
fxC x
C xdx
ϕ
ϕψ
= ∈
= ∈=≤≤∈
=
= +
∑∑
∑∑∑ ∑∑
(3)
where J < J0, and:
,,
() ()
ll
jk jk
Cfxx dx ϕ =
∫
(4)
,,
() ()
ll
jk jk
dfxx dx ψ =
∫
(5)
11
,,,,, ,
( ,),(,),
rl rl
jkjk jk jkjkjk
Cc cDdd= =
following de-
composition formula is:
1, ,2
1, ,2
jknjkn
n
jknjkn
n
C HC
D GC
++
++
= 

=


∑
∑
(6)
And the reconstruction formula is :
,1,2 1,2
()
jnkj knkj kn
n
C HCGD
++ ++
= +
∑
(7)
Can be seen from Equation 6 and 7, just make sur e C_
(j, k), multi-wavelet decomposition and reconstruction
can be done by multiple Mallat algorithm. When the
multi-wavelet signal is processed, the fast Mallat de-
composition and reconstruction algorithm based on
wavelet coefficients also is availab le.
3.3. Mul ti -wavelet Transform of the Image
When multi-wavelet transform two-dimensional image,
the followin g steps should be.
a) The first preprocessing all rows, and then the data
after the pre-processing of all columns (rows). If the crit-
ical sampling method is used, the amount of data is un-
changed; while the repeated-row prep ro cessi ng metho d is
used, The data quantity is 4 times of the original.

F. QU ET AL.
Copyright © 2013 S ciRes. OPJ
78
(a) (b) (c) (d)
(e) (f) (g) (h)
Figure 2. Ex periment results: (a) Or iginal U V image; (b) Orig inal near infra red image; (c) Original R image; (d) Ori ginal G
i ma g e ; (e) Original B image; (f) UV image of r e gistration; (g) Near infrared image of registration; (h) Image of f usion.
b) When after pretreatment completed, 2D multi-wavelet-
transform is applied. In the computation, the first line of
multi-wavelet transform, then the column multiple wave-
let transform.
c) At last, post -processing is used to complete image
fusion, which is the inverse transform of the pretreat-
ment.
4. Conclusions
Using FPGA as the main processor, and the DSP pro-
cessor as arithmetic operations, a real-time image fusion
system based on FPGA and multi-DSP is designed,
which can effectively utilize the FPGA flexible high-
speed characteristics, and take full advantage of the DSP
powerful computing function. In order to coordinate
the asynchronous timing problems between the various
modules, and improve the efficiency of data exchange,
FIFO generated in FPGA is used to complete a five-
band i ma ge d at a a cq ui si tion. T he n the image fus io n a l go-
rithm based on multi-wavelet tr ansform is optimized and
transp lant ed. I n the final e xperiment, the image fusion of
the five-band image with a frame rate of 15Hz, and a
resolution of 1392 × 1040, is successfully completed in
41ms by this system. The experimental results are shown
in Figure 2.
5. Acknowledgements
F. J. Q would li ke to thank Dr. Jian Wang at our depart-
ment for many helpful suggestions and discussions; this
work is partly supported by the program of academy-
locality cooperation of the Chinese Academy of Sicences
and the Jilin province science and technology develop-
ment plan item.
REFERENCES
[1] X. Suxia, “Real-Time Fusion System Hardware Design of
Visual/IR Image Based on DSP and FPGA,” Journal of
Beijing Technology and Business University (Natural
Science Edition), Vol. 26, No. 6, 2008, pp. 44-47.
[2] Z. Jianbin, H. Bo, Y. Rong and Z. Yong, “A Improved
Pyramidal Decompositions Algorithm for Image Fusion
and the Implementation on DSP,” Journal of Wuhan
University of Science and Engineering, Vol. 22, No. 1,
2009. pp. 17-20.
[3] Q. Feng, L. Ying, W. Jian, D. Keyan, et al., “Infrared
dual-Band Real -Time Image Fusion System,” Optics and
Precision Engineering, Vol. 18, No. 7, 2010, pp.1684-16
90.
[4] H. Keming, W. Guocheng and W. Yang, “Multi-Source
Image Fusion System Based on D SP,” Ordnance Industry
Automation, Vol. 31 , No. 2, 2012, pp. 61-63.
[5] S. Yajun, Z. Zhenfu and L. Zhongling, “Analysis and
Application Prospect of Image Fusion Technology Based
on Multicolor FPA,” Infrared and Laser Engineering,
Vol. 3 8, No. 4, 2009, pp. 725-730.
[6] G. Zhiqiang and L. Y i, “ Design of An Image Fusion Sys-
tem Based on DSP Parallel Processing,” Journal of WUT
(Information & Management Engineering), Vol. 28 No.
11, 2006, pp. 107-109.
[7] C. K. Chui and J. A. Lian, “A Study of Orthonormal Mul-
tiwavelets,” Applied Numerical Mathematics, Vol. 20,
1996, pp. 273-298.