Image Motion Deblurring Based on Salient Structure Selection and L0−2 Norm Kernel Estimation

Single image motion deblurring has been a very challenging problem in the field of image processing. Although there are many researches had been proposed to solve this problem, it still has problems on kernel accuracy. In order to improve the kernel accuracy, an effective structure selection method was used to select the salient structure of the blur image. Then a novel kernel estimation method based on 0 2 L − norm was proposed. To guarantee the sparse kernel and eliminate the negative influence of details 0 L -norm was used. And 2 L -norm was used to ensure the continuity of kernel. Many experiments were done to compare proposed method and state-of-the-art methods. The results show that our method can estimate a better kernel and use less time than previous work, especially when the size of blur kernel is large.


Introduction
Motion blur is caused by the relative displacement between the camera and the target due to the camera or a hand shake. With the popularity of the camera in recent years, the blur is very common in life, so it's important for the image motion deblurring. However, motion deblurring is a completely challenging problem since it's an ill-posed problem. Ordinarily, if the motion blur is shift-invariant, the image blur process can be modeled as: where g is the known blur image, f is the latent image, k is a motion blur kernelor a point spread function(PSF), n is noise produced during image ac-quisition, ⊗ is the convolution operator. In general, the image deblurring could be divided into two types-non-blind: deconvolution and blind deconvolution. In non-blind image deblurring method, the blur kernel and blurred image are known, and the major problem is the deconvolution. In the blind image method, the kernel is usually unknown when the blurred image is acquired in life.
In this paper, we are more concerned about the blind image. One of the most difficult problem of the blind image deblurring method is the available information is too little, only one blurred image, we need to estimate the blur kernel from the blurred image, and then restore the latent image by the deconvolution operation of the kernel and the blurred image.
The kernel estimation is an important step in the deblurring process, since the kernel will impact directly on the restoration of latent image. The awful or inaccurate kernel will produce ringing artifact in the latent image. At the beginning, the researchers used some regularization to estimate the accurate kernel [5] [9], but this approach is very costly. Cho [2] used bilateral filtering and shock filtering to predict the image edges for the kernel estimation, but this method is too simple.
Now, we use a more effective image structure extraction algorithm. It avoids the negative effect of the details on the kernel estimation. Meanwhile, we propose a novel method of kernel estimation, it can not only estimate the kernel much faster but also suppress the noise, and guarantee the continuity of kernel.
The coarse-to-fine iterative process is taken. The image structure is selected from the blurred image. Then, the kernel can be obtained from the image structure. The latent image, which be got from the estimated kernel and the given blurred image, is prepared for the next finer level.

Related Work
Image motion deblurring began to be studied in the last century 60's, in 1967, Helstron et al. proposed the classical Wiener filter. In 70's, Richardson and Lucy proposed the RL algorithm based on Bayesian theory, but this method couldn't get the satisfying result since they didn't take into account the effects of noise and the algorithm model was too simple. Since twenty-first Century, researchers have made great progress in the field of image motion deblurring, they utilized the priors of the image and kernel to restore the latent image. Fergus et al. [3] proposed the image gradient probability distribution has a heavy-tailed distribution, and used this prior of image as the basis for the restoration of the latent image with the RL algorithm. Shan et al. [8] utilized subsection to model the heavy-tailed distribution, and used the image intensity, first and second derivative to restored the image, but it cost a lot of time by reason of too much data to be used. Cho and Lee [2] proposed an algorithm based on edge prediction, employing a bilateral filter and shock filter to denoise and preserve edge. Then they used the edge to estimate the kernel and updated the intermediate latent image.
Finally, they alternately iterated this process to get the final clear image. Comparing with Shan et al. [8], as it turned out, this method obviously improved the deblurring speed, and had a wonderful result. Xu & Jia [9] discovered that these details had a negative impact on the kernel estimation, so they used a model based on image gradient to eliminate the details, and ISD was employed to refine the kernel. Pan et al. [7] proposed an algorithm which utilized the salient structure to estimate the kernel, it could get better results when the kernel size is large and the blurred image had complex structure, but it need a lot of time when the kernel size is large. Long et al. [6] proposed a kernel fusion method which used several state-of-the-art motion deblurring methods to estimate the kernel. They trained a more precise kernel via the several kernel results. However this approach was dependent on these state-of-the-art methods.

Our Method
Our method focuses on the salient structure selection and the kernel estimation.
An effective salient structure extraction method is critical to the kernel estimation, and the accurate kernel is indispensable to the deconvolution. Figure 1 shows the image structure and the kernel in the reconstruction process. Both image structure selection and kernel estimation are under the multi-scale iteration.

Effective Salient Structure Selection
We find that the effective image structure facilitates the kernel estimation. In previous researches, bilateral filtering and shock filtering are used to predict the image edges [2]. In this paper, we select an image smoothing algorithm based on 0 L norm [10] to extract the outstanding structure. 0 L -norm can smooth the region which gradient is small. The structure extraction model is defined as: where S is the structure image, I is the blurred image, p is the coordinate of the image pixel, ( ) C S which can be written as is the 0 L -norm of S . It denotes the number of non-zero pixels, | Formula (3) is first used in [9], I ∇ is the gradient of the blurred image, | | ⋅ is the absolute value, ( ) h x is a h h × window centered at the pixel of x . We assume that x r and y r respectively express Formula (3) (4) and we call it model (4) in the following paragraphs, The solution of model (4) can be decomposed into two sub problems-S and ( , ) h v : The sub problem about S : We can use the Fast Fourier Transforms (FFT) to compute S . Based on Paseval theorem, the solving equation can be expressed as: where ( ) F ⋅ and 1 ( ) F − ⋅ represent the Fast Fourier Transforms and its inverse operation respectively, x ∂ and y ∂ denote the gradient filter in x direction and y direction, * is the element-wise multiplication operator, ( ) ⋅ is the conjugate operator.
The sub problem about ( , ) h v : We add x r and y r to Formula (7) and use x D and y D to replace x S ∂ and y S ∂ respectively. As Formula (8) shows. * According to [10], we can get the result of Formula (8) as following x y Proof details can be found in [10]. After obtaining the salient structure of image, we use the shock filter to further enhance the structural information. Figure   2 shows the different results between the model with filter ( ) r x and the model without filter ( ) r x . We can see that the kernel and latent image with ( ) r x used are more satisfying than without ( ) r x used.

The Kernel Estimation
In the last few years, a variety of regularizations were used to estimate the kernel, the most commonly used were 2 where the first submodel is data fidelity term, 0.5 α = and B ∇ is the gradient of the blurred image, S ∇ is the gradient of the structure image, γ and µ are the weight value. Formula (10) is a non-convex function, and it is difficult to be minimized. Pan minimized this function by using iterative reweighed least square (IRLS). IRLS requires a lot of time because of the complexity of computation. Now, we propose a new kernel estimation method based on 0 2 L − norm. In fact, the blur kernel is the path of camera movement, so the kernel is sparse. For the sparsity of the kernel, we utilize 0 L norm ensure it. But 0 L norm can't preserve the continuity in general. So 2 L norm is used to guarantee the continuity. And our model can be written as Formula (11) and we call it model (11) in the following paragraphs.
In order to solve model (11), we add two auxiliary variables ( , ) h v into model (11), model (11) can be rewritten as: We call it model (12) and the solving of it is same to Section 3.2.
To remove noise, we set the kernel elements with the value smaller than 0.075 of the biggest one to zero. Then, the remaining non-zero values are normalized so that their sum becomes one. The experimental results will be compared in Section 4.

The Latent Image Reconstruction
The latent image can be restored via the deconvolution operation of the kernel and blurred image. In order to ensure the details and smoothness of the latent image, as well as suppress ringing artifacts produced in process of deconvolution, we use 1 L norm as regularization to restore the latent image, the model can be expressed as Formula (13).

Analysis
From Section 3, we can see that our method mainly takes advantage of 0 L regularization. 0 L norm has the excellent effect on ensuring image sparsity on the blur kernel, because kernel is the displacement path generated by the camera shake. This path is sparse in the image. On the blurred image, the degeneration of image details is more serious than the image outline. So it is more effective to using the image structure to estimate the kernel. The image structure is also sparse.
For the image structure extraction and the kernel estimation, we both execute in the gradient image. Because of the pixel value of gradient image are all zero in addition to the outline section. It can improve execution efficiency.
Because of Pan use the IRLS (iterative reweighed least square) to solve Formula (10) at the stage of kernel estimation, it is very time-consuming. We use the Fast Fourier Transforms to solve our models, so our algorithm is more efficient. Table 1 shows the time contrast results between Pan [7] and our algorithm. Figure   3 shows the pictures we used and comparison results among Xu & Jia [9], Pan [7] and our algorithm.
The complex building images are challenging for all methods, because the building images contain too many edges, the invalid edges will restrict the image restoration. Figure 4 shows that, our method can result the commendable latent images.  We evaluate our results on the dataset in [5] by comparing the Peak Signal to Noise Ratio (PSNR) and Structural Similarity (SSIM) with Xu & Jia [9] and Pan [7]. In Figure 5, PSNR is employed to compare the reconstruction accuracy for the latent image. And Table 2 shows the SSIM results of Xu & Jia [9], Pan [7] and our algorithm. We can see that our accuracy is higher than other methods in most cases.

Conclusion
In this paper, an effective image structure extraction algorithm was used to select the salient structure. It eliminated the details of image to avoid the negative effects for the kernel estimation. Meanwhile, a novel kernel estimation method was proposed. It can not only ensure the sparsity of kernel, but also guarantee  show that our algorithm can provide reliable kernel and image. Next stage, we will extend our method to handle non-uniform deblurring.