Robust Lane Detection in Shadows and Low Illumination Conditions using Local Gradient Features

This paper presents a method for lane boundaries detection which is not affected by the shadows, illumination and uneven road conditions. This method is based upon processing grayscale images using local gradient features, characteristic spectrum of lanes, and linear prediction. Firstly, points on the adjacent right and left lane are recognized using the local gradient descriptors. A simple linear prediction model is deployed to predict the direction of lane markers. The contribution of this paper is the use of vertical gradient image without converting into binary image(using suitable threshold), and introduction of characteristic lane gradient spectrum within the local window to locate the preciselane marking points along the horizontal scan line over the image. Experimental results show that this method has greater tolerance to shadows and low illumination conditions. A comparison is drawn between this method and recent methods reported in the literature.


Introduction
Lane detection has long been an important area of research for people in autonomous vehicle systems.There are a large number of lane detection algorithms that are available in the literature during recent years [1][2][3][4][5][6][7][8].The lane detection algorithm can be classified in two broad categories, namely model based [8,[13][14][15][16][17][18] and feature based [1,7,11,12].The Model-based lane detection system begins with selecting a model/template for lane, extracting parameters to represent lanes, and fitting the model to the extracted parameters and processing.Applications of these template based methods (linear, spline, parabola) are limited to some road environments and cannot perform well in the presence of shadows and low illumination condition.The algorithm proposed by Yue et al. [15] used the cubic B-snake to model the lane boundaries.The use of Canny Hough Estimation of Vanishing Points (CHEVP) algorithm for locating the initial points is sensitive to choice of threshold.Moreover, this method is vulnerable to presence of shadows and illumination condition.
The most common and simple technique for detecting road markings is based on Image gradient or edges (feature based).McDonald and Palmer et al. presented methods based on detecting edges and applying Hough Transform in [11], [12] respectively.However these edge methods are sensitive to illumination, heavily affected by shadows, environmental conditions.Additionally, these methods have experienced difficulties when choosing a suitable intensity threshold to remove the unwanted edges.In [6], a layered approach is presented for robust lane detection at night which relies on temporal frame averaging, edge detection and Hough transform.The algorithm is very fast, robust, and can be used in real time applications.However, it is susceptible to lighting, shadows, and environment conditions and focuses only on straight road segments.
The authors in [3] presented a method which combines input from multiple cameras (data fusion) and line models are formed from feature points using a RANSAC algorithm.The approach is fast and robust; however, it can support real time operation only at low speeds, and is heavily affected by the presence of cars and other obstacles on the road, resulting in generation of false feature points.There have been efforts by C. Rasmussen in [4] to use Principal Component Analysis (PCA) for lane detection where they use gradient within a local window to obtain the vanishing point, but the directions obtained from gradients of PCA may not converge all the time.The algorithm in [19,22] presented the Likelihood of Image Shape (LOIS) lane detection system that applies likelihood function into lane marker detections.Lane finding in ANother domAin (LANA) [20] uses sets of frequency domain features to capture information concerning the strength and orientation of spatial edges.LANA is similar to LOIS in detection stages but differs only in the use of frequency features of lanes while LOIS uses spatial edge features.The algorithm presented herein, also uses the frequency features of the lane markings that are a characteristic representation of the lane.
Sivaraman et al. presented a method in [15] for lane detection and tracking in high density traffic scenes using vehicle localization where lane detection and tracking is performed by using steerable filters and Kalman filters.The algorithm in [15]performs well only in high density traffic scenes but its performance degrades highly in an empty road.Also good performance of this method in presence of shadows is doubtable.The research presented in [10] also fails to perform well in presence of shadows.
A survey of the recent lane detection methods are presented in [9,10].There has been progress in detection of lanes of various shapes but there is a need for robust detection of lanes in presence of shadows and other image artifacts.The work presented in [7] performs well in presence of shadows but cannot track the curved sections of the road in the far field of the image.The work presented here is compared with the work [7] using the same datasets [23].The remaining sections of this paper are organized as follows.Section 2 contains the description of method and path chosen.Section 3 presents the experimental results.The conclusion and suggestions for future work are presented in section 4.

Description of the method
The proposed method begins with extracting the individual frames from the video and processing each frame to detect and track the road lane markers.Then the vertical gradient of the image is calculated and processing is done in this vertical gradient image.The vertical gradient is selected to remove the effects of shadows along the road which are usually horizontal.Additionally, this method does not require threshold to convert the gradient image into binary image and is thus unaffected by illumination conditions and shadows.

Feature extraction
The vertical gradient of an image [21]is obtained by 2D discrete convolution of an image I (m, n) with h (m, n) where h (m, n) is a mask given by ( ) In the spatial domain, a window of size 15x15 is selected and placed at randomly selected points s1, s2, s3, s4, and s5.All the points within these five local windows are placed in five different matrices namely w1,w2, w3, w4 and w5.Then the magnitude of FT (Fourier Transform) of these five matrices is computed and their average is calculated.This average value of the magnitude of FT is the characteristic spectra for the lane markings and will be represented as Lspectra.The variable Lspectra used further in this discussion will refer to the characteristic spectra of the lane markers.This process of obtaining characteristic spectra for a point 's1' is shown in Fig. 2.  Analysis of result for different window dimension indicated the 15x15 size window produced most consistent and accurate representation of distinguishing road lane markers from background settings.The spectra for the lane markings are characteristic to the respective lane and invariant to the selection of location on a given lane marking (independent of translation, rotation and scal-ing).The characteristic spectrum is obtained from the first frame of the video file from the image database [23].This spectrum (Lspectra) is consistent for all the images of the video file from the database.Once the characteristic spectrum of lane marking is obtained from the first frame of the, we do not require to obtain the characteristic spectra from all the successive frames for a period of time.

Gradient spectrum match function
The gradient spectrum match function is used to locate the precise points on the adjacent lanes in which the vehicle is moving.The 15x15 local window is moved along the horizontal scan line (see Figure 3) and magnitude of FT over the local window is calculated as represented in Equations 2-4.Let , I n n be the gradient image (see Figure 3) with the following dimensions.
( ) , n n are discrete variables in the range The sliding window is placed at each point along the horizontal scan line (green colored line in Figure 3)and corresponding gradient values of image I ( ) , n n ₁₂ is correlated with the characteristic spec- trum Lspectra and the maximum value is chosen.This maximum value is R (0, 0) and can be obtained by element wise multiplication of Lspectra and P as shown in Equation 5.The maximum correlation is calculated at each point along the horizontal scan line (refer to Figure 3) and stored in a variable S m ( n₂ ) where n₂ = 1, 2,…..C.The normalized plot of S m for all the values of along the horizontal line is called Spectrum matching plot (Figure 3).R(0, 0) = ( ) ( ) (5) Sm(n2) =R (0, 0), where n₂ = 1, 2, 3 …C (6)

Processing of spectrum match (S m ) plot
The spectrum matching plot (S m ) is smoothed with a Moving Average (MA) filter of order 20, and the maxima points on the plot are found using the algorithm in [24].To remove the unnecessary processing of maxima points, the S m plot is threshold by a suitable value of 0.3(if S m (i) < 0.3, thenS m (i) = 0).This choice of this threshold value is not critical and helps only in the removal of unnecessary maxima points.

Local Maxima Discrimination Via Bayesian Classifier
The local maxima points obtained in the S m plot is decided as either belonging to the left or right.Bayesian classifier is devised to discriminate the maxima points into one of the two designated classes, namely, left line class(C L ), and right line class(C R ) [22].The maxima points which lies on the left of the median line (line passing through the median of column number) is decided as belonging to left lane class C L , and the maxima points to the right of median line as designated as right lane class C R .Let X be the vector containing all the column number of maxima points and =median(C), where C is the Column of the image shown in Figure 3.
where i =1, 2 …N; N=total number of local maxima.
The Figure 3 shows the first horizontal scan line which is set to start from at 3/4 th of the length of the image (0.75R).The second scan line will start at 5 pixel points above the first line and so on.The first scan line has only one maxima points at x= 60 (N=1).This point is classified as C L , because Equation.7 is satisfied.The median for all the images obtained from the database [23] is constant ( =256/2 + 1 = 129).Figure 4 shows an instance in which three scan line, separated by 50 pixels, are produced and the spectrum matching plot S m is generated for each of the three horizontal lines.The S m plot from scan line 1 has only one maxima point that belongs to left class (C L ) as that points lies to the left of medina line.The scan line 2 intersects with both the adjacent left lane and right lane markers and it is reflected in the corresponding S m plot with two maxima points on the either side of median line.The S m plot for the scan line 3 has four maxima points with the left one belonging to the class C L and the other three belongs to class C R .In the near field of the image, the Sm plot along the scan line produces less maxima points.As the scan line is mover further into the far field the number of maxima points increase.The maxima points are classified into one of the two classes.
For all the scan lines separated by a distance of 5 pixels, the left and right lane points are selected as maximum values in respective class; i.e, RT=max (C R ) (10) Due to the effect of filtering, the maxima points are shifted away from their true location in the S m plot.Therefore the precise locations of the lane markers are found by searching the pixel with maximum intensity in the neighborhood of the points obtained from Equation 9and Equation 10 along the respective horizontal line.It was found that searching for the maximum intensity in the spatial domain of the gray scale image produced better results than searching in the vertical gradient of the image.Mathematically, it is represented in Equation 11and 12.
LT= max{img(row,LT-20:LT+20)} ( 11) where row is the row number for a given scan line, and img (row, column) is the original gray scale image.
After the first five left and right lane points are determined, next left and right lane points are predicted based on the last five left and right points, respectively.The prediction scheme is described in the next section and is useful in removing the false lane points obtained from the maxima points of S m plot.

Linear prediction of lane points
After the first five horizontal scan lines are drawn, the left and right lane points on the next scan line are estimated using a simple linear prediction model.Since the distance between the scan lines are predetermined and fixed in advance, the row number for each horizontal line is known.Let y1, y2, y3, y4, and y5 be the column number of five lane marking points.The 1D displacement ( ) d i , between the five points is calculated and their mean value is computed.The next predicted point is then calculated by adding the mean displacement of last five to the most recent point y5.The process described here is presented in Equations 13 -16.

Error Correction
The estimated points ( ) yp i for the next scan line are compared with the calculated lane points, ( ) y i of the next scan line for both left and right lane points.If the computed point ( ) y i deviates very far away from the predicted point ( ) yp i an error has occurred in the computation of the actual lane marking points.In such cases the predicted point is selected as correct point.All these steps are shown in Equation 17.
If abs {}> threshold, then ( ) ( ) where the empirically determined threshold is set to 20.
Figure 5 shows the calculated points and their predicted values.The linearly predicted points after error correction yields highly accurate representation of the precise location of the lane markings.
A 2D graph showing the comparison of computed lane points with the predicted and corrected points is shown in Figure 6.The processing steps outlined through sections 2.2 through 2.6 are performed for an image until both the left and right lane marking point falls to the same class.That is, algorithm stops when both the lane points lies on the same side of the median line(same class).

Results
This section, presents the results obtained from the process described in section 2. The various stages of feature extraction, generation of spectrum matching plot, discrimination via Bayesian learning, and linear prediction of lane points are implemented using a laptop computer with a clock rate of 2.24 GHz executing Matlab software version R2008a.The data sets of images used for the testing and validation of the proposed method is same as that has been used in [cmu].(http://vasc.ri.cmu.edu/idb/html/road/may30_90/).The 157 image sequence was tested with our algorithm and it was found that 150 images were successfully detected with 7 false detections.The succesfully detection rate was found to be 95.54 % which are better than the detection rate achieved in [7,15].Some of the results of lane marker detection in presence of shadows are presented in Figure 7-8.The presence of error correction scheme in the algorithm rectifies the large deviations from the lane markers position as the lane markers do not change its course abruptly.An important advantage/contribution of this method is to track roads lane markers of various shape (curved or straight) and locate precise lane marking points on each horizontal scan line which is not affected by presence of shadows and other low illumination condition.This method has a disadvantage in that the first five scan line needs to locate almost precise points because the error in the first five scan lines cannot be corrected.It was found that any high dynamic range portion of image leading to errors in the first five scan lines cannot be rectified.

Conclusion
This paper has introduced an approach for the detection of road lane markers based on local gradient features, characteristic lane spectrum, and spectrum match plot (S m ).Further the linear prediction and error correction schemes leads to robust detection of lane marking points along the scan line.The experimental result suggest that this method is robust for lane detection in presence of shadows and other vehicles.
The future extension of this work can be information fusion of this method and other methods (e.g.Hough transform)for higher detection rate in other inclement weather conditions such as rain, snow, fog.Nevertheless, there is always room for improvement.

Figure 2 :
Figure 2: Graphical representation for the process of obtaining characteristic spectra.
n n ₁₂ within the window can be represented by variable p as in the Equation 3. Along the horizontal line the row component of the image i.e.

Figure 3 :
Figure 3: Spectrum matching (Sm) plot generated along the horizontal scan line on the gradient image.Filtered version of spectrum matching plot with green points representing maxima points in extreme right.

Figure 4 :
Figure 4: Image gradient with three scan lines and their respective Sm plots with maxima points.The maxima points are represented by green dots on the lower portion of Figure and the median line by yellow and red line.The distance between the scan lines is set to 50 pixels in this Figure for the purpose of simplifying the explanation of process.The distance between the scan lines in the actual algorithm is set to 5. Red line is the median line.

Figure 5 .
Figure 5. Computed lane markings(blue colored box), and predicted markings(red color dots) plot on the vertical gradient image and RGB image.Note the presence of shadows.

Figure 6 :
Figure 6: Comparison of the computed and predicted & corrected lane marking points as a function of row and column number.

Figure 7 :
Figure 7: Experimental results for some of the extreme cases of illumination and road conditions.Note that even thepresence of shadows and vehicle on the lane markers has little to no affect the detection results.