Variance Window Based Car License Plate Localization

In this paper, a new method for discovering the candidate car license plate locations is presented. First, the image is decomposed using a Haar wavelet to get the HL band with vertical edges. Then, the HL band image is binarized using an Otsu threshold. Next a black top-hat algorithm is applied to reduce the effects of interfering large continuous features other than the license plate. At this time, a moving window based modified variance score calculation is made for areas with white pixels. This work found that the top 3 detected rectangle windows correctly locate the license plate regions with a success rate of about 98.2%. Moreover, the proposed method is robust enough to locate the plates in cases where the rough vehicle position has not been previously discovered and the cars are not centered in the image.


Introduction
There is growing interest in license plate detection, segmentation and recognition among researchers due to wide applications in the fields of computer vision, vehicle toll collection, vehicle traffic monitoring, security, control and law enforcement.
Many researchers have tried to detect license plates using color information, plate features, edge detection, artificial neural networks and others [1]- [6].Moreover, most reports for license character segmentation have used license plate character features such as character projection methods [7] [8], and classifier methods [9].As for license character recognition, a lot of research efforts have focused on feature based methods, learning based methods [10], and template matching methods [11] [12].
Usually, color based methods are memory-consuming and unsuitable for real time applications due to large data processing, and sensitivity to illumination variations [13].
Learning based methods need different classifiers for license plate detection, segmentation and recognition [9] [14] [15].Intelligent architectures such as neural networks are implemented for such tasks [16].However, these methods require many training samples with great processing time and design complexity.
Feature-based methods depend on the quality of the license plate image including the edge, symmetry, texture and so on [16] [17].The grayscale gradient based edge detection methods are good candidates for real-time applications due to lower computational complexity [18].
General template based character recognition is to find a template image which matches a car license plate character image.Usually templates are designed according to prior knowledge about the license plate.After the car character is normalized to the template size, the most similar template will be considered as the match.The success rate of this method strongly depends on the high similarity between the template image and the original image [19].However, different image capturing angles, lighting, plate sizes, and so on can cause errors.
One of the most challenging aspects of license plate identification, however, is establishing an approximate target location of the plate in a raw image.Raw images that have not been manually cropped to an area of interest can contain many competing elements that will interfere with detection.Firstly, a natural scene contains many high frequency elements like leaves, reflections, and lettering on signs and roads.Even in a traffic video that has a relatively uninteresting background, there are many features on a vehicle that provide sources of false positives.Trucks frequently have advertisements, bumper stickers frequently have text, and since the vehicles are moving through the video frame, it is not possible to simply guess at a target rectangle in the frame based on assumption that a car would be in the center frame, or that the plate would be in the bottom half of the image.
The remains of this paper will discuss in detail the proposed new method for locating the license plate.In Section 2, the image pre-processing techniques using conventional transforms and filters are discussed.Section 3 presents the window based modified variance technique for locating the license plate.Section 4 shows design results and Section 5 draws conclusions.

Plate Localization
The objective of this work was to use a published dataset taken from a video using a fixed camera positioned near or above a roadway.Because the technique should be fully automated, no manual pre-cropping to a region of interest (ROI) was permitted.The 1290 un-cropped pictures from the gilman frames folder of the published dataset [20] were used in this design work.

Plate Localization Challenges
If the image is cropped to the back of the car and near the license plate, many existing techniques are very effective at using the character textures in order to precisely identify the outline of the characters of the license plate in the image.For example, the wavelet based texture identification is excellent at defining the plate region, but it does not work well if the image is not first cropped to the approximate area of the plate.
There are several challenges in locating the license plate with the published gilman dataset.Firstly, the license plate can appear almost anywhere in the picture, and can't be easily narrowed down by techniques such as taking the bottom half of the image.Secondly, the pictures are expansive enough so that there are plenty of edges from the background scene that are not related to the license plate itself.Also the view of the car with the plate may be obstructed by another car behind it where another plate is visible.This prevents the use of shadows under the car as a method for isolating the approximate location of the car in the image.Finally, the images in the gilman dataset are low resolution to the point where many letters on license plates cannot even be read by human eyes.This makes their signature much less obvious to globally applied texture algorithms.This work tested a technique that used the shadows to identify the bottom of the car, and it worked on less than 60% of the pictures in the gilman dataset because the rest of the time the region below the car was cropped or obstructed.
Figure 1 shows new strategy used by this work.

Crop and Grayscale
The image in Figure 2 shows the text areas that are added by the camera to the top and bottom of the image.These should be removed.So the first step is to remove any annotation added by the camera through cropping the top and bottom of the image in case annotation is added to the image.Cropping is not needed for images without camera annotation.
The Figure 2 image is also in color.While colors can be useful during image segmentation, they are inconsistent.Most of the colors found in the plate are made up of dark and light shades of gray, which are ubiquitous  in the image, especially in the shadows and roads.In addition, many camera models may only have grayscale.
In this work, the RGB color car image is converted into a grayscale image using Equation (1) shown below.
F 0.299 *R + 0.587 * G + 0.114 * B = . ( where F is the luminance value, R is read color value, G is green color value, and B is blue color value.The result of the converted grayscale image is shown in Figure 3.

Haar Wavelet
In order to locate the plate, it is necessary to inspect the car vertical edges.Many techniques are available, including Canny and Sobel edge detection algorithms, as well as morphological methods including the Black-hat transform.However, the wavelet technique [21] [22] of using the Haar wavelet has many advantages, such as high speed, a smaller output image and image components that separate horizontal from vertical edges.The decomposition of the image using a Haar wavelet is shown in Figure 4.
In Figure 4, there are four wavelet sub-bands: LL, HL, LH, and HH.The LL represents the low frequency band.The HL, LH and HH represent the high frequency bands.The HL band shows the vertical car edges, the LH band shows the horizontal car edges, and the HH band shows the diagonal car edges.Especially, the HL band shown in Figure 5 is of principal interest in this design.
While the plate clearly stands out in Figure 5, a typical approach at this point is to apply a row-sum technique with a threshold in order to isolate the plate.The result is shown in Figure 6.

Binarization Using Otsu's Thresholding
Since the variation in the plate region is consistent across a large windowed area, a new window based variance method shown in Figure 1 is proposed to locate plate.
The variance values from the other vertical edges of the car were so much more prominent that they tended to produce false positives, so the image binarization using Otsu's thresholding method in Figure 7 is applied to minimize the intra-class variance.Figure 7 also shows that a row-sum calculation after binarization still appears     about the same as Figure 6.However, the license plate area is filled with frequent jumps from white to black, while the other edges tend to experience less variation.

Black Top-Hat
The black top-hat method is used to reduce the impact of large, low-variation elements in the image in this work.Although the plate has a lot of high frequency variation that would likely be preserved by a top-hat filter, many of the other features like the tail lights, car edges and others would likely be removed.This step helps to reduce the chance for false positives, but also reduces the amount of white pixels in the image.

Conditional Windowed Variance
For the dataset tested in this work, the size of the letter containing region of the plate is approximately four times as wide as it is tall in the original image, and approximately 40 by 10 pixels in the Haar wavelet HL sub-band image.So a moving window of 40 × 10 pixels can be used based on a rule of thumb for this dataset to search for the maximum variance.
Usually, even though moving windows are extremely effective, they are quite slow.Typically the number of evaluations is equal to the window area in pixels multiplied by the total number of pixels in the image.
However, in the sample image shown in Figure 8, only 412 pixels out of 60192 are actually white.So it is reasonable to make the generalization that only if the pixel in the center of the searching window is white, then variance inside the window needs to be calculated.Otherwise no variance calculation is needed within the window and the window will be shifted to the next position.
For larger resolution images, it is obviously possible to increase the window size, but it may be more efficient to use the Haar wavelet to down-sample the image until the size of the plate in the image is approximately scaled to the 40 × 10 window size.If window size is smaller, the license plate characters won't be recognized.

Border Subtraction
While the windowed variance is a good indication, many textures besides a license plate may also have high frequency variation that could fill a moving window.To solve this problem, consider the neighboring regions outside the moving window.In the license plate region, there is high variance within the moving window, and low variance outside the window.In other non-plate regions, the outside of the moving window may still have high frequency components due to their long vertical edges.In this work, a two pixel thick border marked in red in Figure 9 is considered.
In Figure 9, the outside border marked in red is offset by 5 pixels in all directions from the moving variance window marked in green.It allows checking for approximately (50 × 4 + 20 × 4) pixels.Mirror symmetry is used when the bounds extend beyond the image outmost borders.A parameter called F score is defined below.
where w represents pixel values in the window region, and b represents the pixel values in the border region.

Overlap Prevention
The result of the conditionally applied windowed variance calculation will frequently produce several rectangles with high F score values all approximately in the region of the plate.This doesn't matter if the rectangular region with the maximum variance shows the location of the plate.However, in the gilman image dataset, there are several images with multiple plates, and a few cases where there is a false positive that actually scores higher than the actual license plate window.In these cases it is beneficial to evaluate more than one rectangle region result.In this design work, the top ten detected rectangles with high F scores are preserved.But if the results overlap, it will not be helpful.
The rule is that if any of the existing overlapping windows in the top 10 list has a higher F score than the new window, discard the new window.Otherwise the new window will replace all of the existing overlapping windows currently in the top 10.This guarantees that the list of rectangle windows in the top 10 will have the highest F scores, but not intersect each other.The number of 10 is chosen arbitrarily for testing purpose.As seen from the results, in fact the top three windows are enough to identify the plate 98% of the time.

Results and Analysis
As explained above, the detected top 10 rectangle windows with highest F scores for Figure 9 are ranked according to their F scores shown in Table 1.
In Table 1, the first window has much higher F score than the other windows.Moreover, the license plate coincided with window 1.
For this work, 1290 total pictures were tested.Of these, 18 pictures had no plates at all in them, so effectively there were 1272 pictures with possible plates.This design work only allowed for one "correct" plate answer for each image and there were never three or more plates in the tested images.Table 2 shows the number of times that the actual license plate coincided with a rectangle moving window.The rectangle moving window number is ranked based on its F score.
The major observations from Table 2 are listed below.1) For 1250 of those 1272 images, or a little better than 98% of the time, the actual plate rectangle coincided with one of the top 10 detected rectangles.
2) In 1135 cases, or about 89% of the time, the rectangle with the highest F score was actually coincided with the plate region.
3) In 75 cases, the license plates coincided with the rectangle having the second highest F score due to interference from car mirrors or additional license plates existing in some images.4) In 19 cases, the license plates coincided with the rectangle having the third highest F score due to interference from car mirrors or additional license plates existing in some images.
5) The top three windows with the highest F scores are enough to identify the plate 98% of the time.Moreover, two more important results were observed through experiments with 1250 pictures shown in Table 2.
6) In about 79% cases, all the license plate letters were completely contained by the rectangle moving window without requiring additional changes of the boundaries.
7) In about 21% cases, the rectangle only partially contained the license plate characters.This is because the actual size of the window will not match the size of the letters at different distances from the camera.
In conclusion, the top 3 detected rectangle windows with highest F scores provide an excellent approximation for where the license plate should be.

Conclusion
In this paper, a new method for license plate detection that is especially effective at identifying the general location of a license plate in un-cropped traffic video was introduced.The top 3 detected window results accurately find the plate more than 98% of the time, and the detected highest scoring window finds the plate 90% of the time or better.All the letters in the plate were precisely contained without requiring additional higher precision methods 78% of the time.The results were scored so that candidates that are most likely to contain a plate can be inspected first.

Figure 1 .
Figure 1.Car license plate localization flow chart.

Figure 6 .
Figure 6.Grayscale HL sub-band car image with row-sum distribution and identified plate candidate regions.

Figure 8 .
Figure 8. Car image after black top-hat processing.

Table 1 .
Window ranking with different F scores.

Table 2 .
The number of the actual license plates coinciding with the top scoring rectangles in 1250 pictures.