Image Fusion Real-time System Based on FPGA and Multi-DSP *

In order to solve complex algorithm that is difficult to achieve real-time processing of Multiband image fusion within large amount of data, a real-time image fusion system based on FPGA and multi-DSP is designed. Five-band image acquisition, image registration, image fusion and display output can be done within the system which uses FPGA as the main processor and the other three DSP as an algorithm processor. Making full use of Flexible and high-speed characteristics of FPGA, while an image fusion algorithm based on multi-wavelet transform is optimized and applied to the sys-tem. The final experimental results show that the frame rate of 15 Hz, with a resolution of 1392 × 1040 of the five-band image can be used by the system to complete processing within 41ms.


Introduction
Image fusion which is a branch of the data fusion consolidates two or more source images come from complementary band sensors, and is an image processing method which makes the useful information integrated into a unified image for observation or further more accurately full recognition, analysis and judgment to the target or scene through some fusion system [1][2][3].Due of very limited data information conveyed of a single band, which is often difficult to meet the demand, Multisensor image fusion has the information that single source image can't be captured, that is Complementary between of multisource image.So multi-source image fusion can obtain information from multiple viewpoints of expand of sensing range of time and space, to improve the accuracy and robustness of observation.
Seen from the definition of image fusion and complete fusion mainly includes two parts, namely the image source acquisition and image fusion system, the core task of image fusion system is the implementation of image fusion algorithm.With the fusion algorithm improvements, as well as the amount of data increases, the image fusion system developed becomes with considerable difficulty.Therefore, how to design a reasonable real-time image fusion hardware system is a prerequisite, and a suitable image fusion algorithm image fusion system is critical.
In this paper, in accordance with the requirements of the actual project, an image fusion system based on FPGA and multi-DSP is designed, and an image fusion algorithm based on multi-wavelet transform is optimized, and the frame rate of 15 Hz, with a resolution of 1392 × 1040 of the five-band image is processed in real-time at last.

Structure of System
Image fusion system hardware design is mainly three types: single DSP or DSP parallel processing program; FPGA + DSP processing program; a large FPGA processing program [4][5][6].With the combination of these strengths and weaknesses of the program, we have adopted the FPGA + multi-DSP processing program as our image fusion system.This way by the FPGA to complete the pre-processing part and the corresponding logic control, complete the core algorithm in the DSP part, has the advantage of more flexible and can be formulated to give full play to the FPGA and DSP respective strengths.also with the difficulty is how to coordinate communication and data transfer between the FPGA and DSP, to ensure that the in entire treatment can be quickly and efficiently.Therefore, we designed the system structure shown in Figure 1, Which FPGA chosen Alter Corporation Cyclong II series EP2C70F896, and DSP chosen TMS320C6416 produced by TI.
In this system, three-way camera data is collected directly by the FPGA, and after preprocessing, the data is In order to improve the efficiency of the transmission, as well as to coordinate the timing of the various parts of the asynchronous, we take advantage of the FPGA internal structure generated FIFO for data exchange between camera data acquisition and FPGA and DSP.The entire process of using the pipeline work, as long as the time occupied by the image registration and image fusion processing must not exceed 66.7 ms to complete real-time processing of multi-band image.

Multi-wavelet Transform Algorithm
Multi-wavelet is the development of wavelet theory, and which is the wavelet generated by two or more scaling function.Multi-wavelet can have many good properties, such as symmetry, short orthogonality and higher order vanishing moments, so multi-wavelet have more advanages than a single wavelet [7].

Definition of Continuous Multi-wavelet Transform
Assuming multi-wavelet as Ψ = (ψ1，…，ψr)T ∈ L2(R)r, and for a, b ∈ R, a ≠ 0, and by dilation and translation, the orthonormal basis on L 2 (R) r is generated as below equation (1): For

Multi-wavelet Decomposition and Reconstruction Algorithm
Let f ∈VJ, then we have equation ( 3): where J < J 0 , and: , , ( ) ( ) , And the reconstruction formula is: , Can be seen from Equation 6and 7, just make sure C_ (j, k), multi-wavelet decomposition and reconstruction can be done by multiple Mallat algorithm.When the multi-wavelet signal is processed, the fast Mallat decomposition and reconstruction algorithm based on wavelet coefficients also is available.

Multi-wavelet Transform of the Image
When multi-wavelet transform two-dimensional image, the following steps should be.
a) The first preprocessing all rows, and then the data after the pre-processing of all columns (rows).If the critical sampling method is used, the amount of data is unchanged; while the repeated-row preprocessing method is used, The data quantity is 4 times of the original.b) When after pretreatment completed, 2D multi-wavelettransform is applied.In the computation, the first line of multi-wavelet transform, then the column multiple wavelet transform.c) At last, post-processing is used to complete image fusion, which is the inverse transform of the pretreatment.

Conclusions
Using FPGA as the main processor, and the DSP processor as arithmetic operations, a real-time image fusion system based on FPGA and multi-DSP is designed, which can effectively utilize the FPGA flexible highspeed characteristics, and take full advantage of the DSP powerful computing function.In order to coordinate the asynchronous timing problems between the various modules, and improve the efficiency of data exchange, FIFO generated in FPGA is used to complete a fiveband image data acquisition.Then the image fusion algorithm based on multi-wavelet transform is optimized and transplanted.In the final experiment, the image fusion of the five-band image with a frame rate of 15Hz, and a resolution of 1392 × 1040, is successfully completed in 41ms by this system.The experimental results are shown in Figure 2.

Figure 1 .
Figure 1.Schematic diagram of the image fusion system based on FPGA and DSP.transferred to the DSP A and DSP B, with image registration work in them.Visible image is chosen as reference image because of more detail texture characteristics, and near-infrared image is registered in DSP A, while UV image is done in DSP B, then five-band image fusion work is completed by DSP C. Finally, the fused image is transferred to FPGA and output by FPGA.In order to improve the efficiency of the transmission, as well as to coordinate the timing of the various parts of the asynchronous, we take advantage of the FPGA internal structure generated FIFO for data exchange between camera data acquisition and FPGA and DSP.The entire process of using the pipeline work, as long as the time occupied by the image registration and image fusion processing must not exceed 66.7 ms to complete real-time processing of multi-band image.

Figure 2 .
Figure 2. Experiment results: (a) Original UV image; (b) Original near infrared image; (c) Original R image; (d) Original G image; (e) Original B image; (f) UV image of registration; (g) Near infrared image of registration; (h) Image of fusion.