Corner-Based Image Alignment using Pyramid Structure with Gradient Vector Similarity

This paper presents a corner-based image alignment algorithm based on the procedures of corner-based template matching and geometric parameter estimation. This algorithm consists of two stages: 1) training phase, and 2) matching phase. In the training phase, a corner detection algorithm is used to extract the corners. These corners are then used to build the pyramid images. In the matching phase, the corners are obtained using the same corner detection algorithm. The similarity measure is then determined by the differences of gradient vector between the corners obtained in the template image and the inspection image, respectively. A parabolic function is further applied to evaluate the geometric relationship between the template and the inspection images. Results show that the corner-based template matching outperforms the original edge-based template matching in efficiency, and both of them are robust against non-liner light changes. The accuracy and precision of the corner-based image alignment are competitive to that of edge-based image alignment under the same environment. In practice, the proposed algorithm demonstrates its precision, efficiency and robustness in image alignment for real world applications.


Introduction
In automatic optical inspection (AOI) systems, speed, precision, and robustness are the key requirements in real world applications.The technique of image alignment can be used to achieve these requirements.The common image alignment technology can be categorized into two kinds [1]: (1) area-based matching, and (2) feature-based matching.In the area-based template matching, normalized cross correlation (NCC) method is a popular one that can be used to evaluate the degree of similarity between template and scene images.However, it is not robust against non-liner light changes and occluded objects [2].
The main advantage of the feature-based image alignment is its efficiency.Chen et al. [1,3] proposed an image alignment algorithm based on Fourier descriptor.This is a kind of feature-based alignment algorithm, so-called contour-based image alignment.Here, the Fourier descriptor is used to describe the contour information of an object.The contour information is extracted using the maximally stable extremal regions (MSER) method.The MSER method can be used to effectively extract the contour information in an unstable environment.However, the contour-based alignment algorithm is based on the contour information that has been strongly influenced by the results of segmentation.Thus, this algorithm is not suitable for occluded object recognition.
To adapt the complicated environment for template matching, Steger [4] presented a similarity measure based on the difference of gradient direction in edges that is robust against non-linear illumination changes, and it can also be utilized to recognize occluded objects.However, the computation of this similarity measure is not efficient enough for real-time applications.
To speed up the computation of the edge-based similarity measure, we have presented a corner-based similarity measure that is based on the difference of gradient direction in corners instead of in edges.This cornerbased similarity measure has the same characteristic with the edge-based similarity measure as mentioned before.Additionally, we have further developed a corner-based template matching algorithm based on the presented similarity measure.A corner-based image alignment combines two procedures of the corner-based template matching and geometric parameter estimation.
In the presented corner-based template matching, the stabilities of corners have been influenced by the process of corner detection.To address the instabilities of corners, Moravec [5] detected corners using a rectangular window that these corners are determined according to the variation of intensity.Harris and Stephens [6] improved the algorithm of Moravec to reduce the noise of the image.Smith [7] further used a circular window instead of a rectangular window to detect corners.Tran [8] considered the geometric relationship of corners with their gray values.Tran's method has been adopted in the cornerbased template matching because of its efficiency and robustness for corner detection.

Architecture of the proposed image alignment algorithm
The flow chart of this algorithm is shown in Figure 1.
The algorithm consists of two phases, including a training phase and a matching phase.
The corner points are obtained by the intuitive corner detection, and the gradient of corner points are determined using Sobel operator.The corner-based pyramid image technique is used to speed up the computation in the matching process.Here, the corner-based pyramid image is constructed by using the geometric relationship between each corner point.The similarity measure used here is based on the difference of gradient direction of corner points instead of that of edges points.It can therefore decrease the complexity of computation in the matching process.

Corner-Based Image Alignment Algorithm
There are four parts in the corner-based image alignment algorithm, including: 1) intuitive corner detection; 2) corner-based pyramid image; 3) similarity measure and search strategy; 4) refinement.The detail of each part is described in the following subsections.

Intuitive Corner Detection
Chen [9] proposed an intuitive corner detection, and it is much faster than Harris corner detector.Hence, the intuitive corner point extraction algorithm has been adopted here.Figure 2 shows the flow chart of the intuitive corner point extraction algorithm.This method makes a circle around and detects all the candidate points.There are two criterias to check each interesting point whether it is a corner or not.

p p
The two criteria are defined as: Criterion 1 Criterion 1 is a first step to detect whether the candidate point is corner or not.If the two diametrically opposed pixels have approximately the same intensity (smaller than a threshold 1  ), we will decide that this point is not a corner.Criterion 2 of the modified intuitive corner detection will remove the possibility of skew edge by checking the neighborhood of the relative pixels.

I p I p dR and I p I p dR
Then, an incremental step is added to  and Criterion and 2 should be checked again.When the angle  scans all the range between [0,  ] and Criterion 1 and are not satisfied, the point p will be a corner.

Corner-Based Pyramid Image
The image pyramid technique represents the original image in the multiple resolutions using a factor of k in each level.The image pyramid architecture is shown in Figure 3.In general, the basic image pyramid is an image array.The width and the height of each level are reduced using a factor, k, compared with the previous level.
The pixel value at level l in the image pyramid is constructed from the previous level.To reduce the computational complexity, we use the mean operator to obtain the pixel values at level l.Based on the image pyramid technique; the appropriate number of levels is mainly defined by the size of the template.On the top pyramid level, the relevant features of the template should be distinguished.Figure 3 shows the MCU image using the image pyramid technique, and the number of pyramid levels is equal to 3. In the level 3 of the image pyramid, the features of the template is difficult to be recognized.Consequently, we can only set the top level up to 2 in the image pyramid.
) , ( j i f l The feature points, so-called intuitive corner points, can be used to evaluate the similarity measure between the template and the inspection images.As mentioned in Section 3.1, this corner detector usually obtains several positive responses that simultaneously satisfy Criterion 1 and Criterion 2 around the corner locations.We adopted a score computed in each response that retains the local optimum locations as corner.The Laplacian of Gaussian (LOG) is a commonly used score that has been proven to obtain stable corners.It could be approximated up to a scale factor by: According to image pyramid technique, the higher level images are blurred by using mean operator.Thus, the locations of the feature points are unstable.
To improve the robustness of the feature points, we have proposed a new criterion, which is based on the geometric relationship for constructing the image pyramid structure.This criterion uses the feature points to build the image pyramid structure.In the corner pyramid image (CPI), the width and the height of each level are reduced using factor kc.At level l, the pixel value is obtained from the previous level, i.e., ( , ) where is obtained using a maximum LOG value in each specific region.The size of the region is , and i and represent the location of the specific region at the previous level l-1.To make a comparison of the conventional image pyramid and the CPI, the results of intuitive corner detection are shown in Figure 5.As can be seen, the CPI preserves the main feature points in level 1 while the conventional method abandons damages the original structure.The CPI is more robust and stable to the conventional one with the image pyramid technique.
These feature points will be considered in the determination of the similarity measure between the template and the inspection image.The detail of the similarity measure is described in next section.

Similarity Measure and Search Strategy
Similarity measure is used to recognize the template existed within the scene image.The use of the difference of gradient direction shows its robustness to light changes.The gradient of the selected corner points with the coordinate information are stored as the template model which contains a set of points .These points are relative to the center of gravity, and the feature points of the template are represented using gradients , .The points and their corresponding gradients are respectively denoted as and , , , , in the candidate objects.If there is not rotation angle between the template and the inspection image, the similarity measure is defined as follows ( , ) where n is the amount of the feature point.In contrast, if there is a rotation angle between the template and the search image, the similarity measure should be modified by , , , , 1 ( , , ) where  is a rotation angle; the is the gradient vectors after rotation.( , ) T i i

  t u
Perfect match corresponds to 1 in similarity measure.The search strategy will be the subject of a separate paper.

Refinement
The rotation angle  is obtained using the neighborhood of  that notes . The optimal rotation angle is defined as where is the maximum distance between the corner point and the center, is the pixel displacement.The d is equal to 1 here.The above five rotation angles and their corresponding similarity coefficients are fed into the fitting model to obtain the refinement angle.The computation of the best-fitting parabola is achieved by solving a system equation, where each rotation angle provides one equation of the form The whole system can be written as The three unknown parameters  ,  and  , rep- resented by vector in the Eq. 7, are calculated from the known variables X   i   and i  (matrix  and vector  in the Eq. 7, respectively).Taking the values of i  for the five nodes described above, the matrix  and the vector  are filled with these five nodes as: The following equation yields the three unknown parameters  ,  and  as: After the determination of the parabola parameters  ,  and  , the optimal refinement angle *  can be cal-

Results and Discussion
The experiments are divided into three cases as (1) rotation estimation, (2) computation cost and (3) robustness for comparison.Figure 6 shows the two cases of the inspection images and their corresponding template images.The sizes of two template images are 160  120 and 120  110, respectively, and the inspection images are 640  480 and 512  512, respectively.These experiments were performed with Visual C++ on Intel core i3 530 2.93 GHz with 4GB of memory.

Rotation Estimation
In this case, the inspection images were rotated from -20 to 20 for performance evaluation.This interprets that the corners obtained from the templates of Case 1 and Case 2 are more distinctive than the edges.Although our algorithm demonstrates that it is more accurate and precise than the edge-based one for rotation estimation in these two cases, it would be failure when none or only fewer corners are obtained from the images.This means that we have to select the template image carefully in corner-based image alignment.

Computation Cost
Figure 8 shows the computation cost of the two compared alignment algorithms.These figures clearly show that our algorithm is more efficient than edge-based one in these two cases.The average computation times of the proposed algorithm in Case 1 and Case 2 are 43.3 ms and 37.25 ms, respectively.The computation cost of the corner-based and the edged-based algorithms are both influenced by the amount of feature points.Hence, the corner points are used in similarity measure calculation and pyramid image reconstruction instead of using edge points to speed up the computation.Results interpret that this is an effective way to simultaneously reduce the computational complexity and remain the accuracy for image alignment.

Robustness
To evaluate the robustness, the proposed algorithm was utilized under the illumination changes.We used the commercial software PhotoCap to simulate the illuminant variation.The simulated light directions were divided into 0 degree, 90 degree, 180 degree and 270 degree.The exposure value was set from 1 to 2 for each direction.Figure 9 shows the illumination direction with 180 and 0 degrees.In Table 2, the proposed algorithm shows that it is comparative to the edge-based one in robustness evaluation.Both of the corners and the edges were not influenced by the illumination variation in this experiment.

Conclusions
The main contribution of the corner-based image alignment algorithm is to introduce the corner-based pyramid image (CPI) and the corner-based similarity measure.
The CPI architecture and the intuitive corner detection are integrated to improve the efficiency and robustness in the matching process.The gradient vectors of corner points are considered in the corner-based similarity measure.Results show that the proposed corner-based algorithm outperforms the edge-based one in accuracy and efficiency estimation for the two cases.The robustness of the proposed algorithm is also competitive to the edge-based one.This algorithm especially suits to the template image which contains enough distinctive corners.In the nearly future, the search strategy and the full comparison of NCC-based, edge-based and corner-based similarity measures for image alignment will be presented in a separate paper.

Figure 4 1 ,Figure 3 .
Figure 4 illustrates a simple example of the CPI structure which has 3 levels and the specific region is a 3 3  mask.There are 9 regions in level 0, where 3 regions

Figure 4 .
Figure 4.The CPI architecture, the levels of this case is equal to 2.

Figure 5 .
Figure 5.The results of Intuitive corner detection on conventional image pyramid and CPI.(a) The original image, (b) The CPI at level 1 (N = 3), (c) The conventional image pyramid at level 1.

Figure 8 .
Figure 8. Executing time of case 1 and case 2 with the competing method.