Abstract: License plate location is a critical part in the license plate detection and recognition system. It is sensitive to weather, illumination and the size of license plate. A variety of license plate location algorithms is proposed, which have limited adaptability. In order to resolve the above mentioned problem, this study improves the image preprocessing, image filtering and discrimination of pseudo-regions in 6 different algorithms based on the corner detection, on wavelet transform, on HSV color space, on RGB color space, on license plate texture and on the integrated features of license plate respectively. The experiments show that the improved algorithms have strong robust.
INTRODUCTION
License plate location and recognition system has numerous applications such as security control of restricted areas, traffic law enforcement, congestion pricing and automatic toll collection (Sheng et al., 2009). As the first step of the license plate detection and recognition system, license plate location plays an important role in the system. If a license plate is located incorrectly, it will invalidate the subsequent steps such as character segmentation and character recognition. The difficulty of license plate location lies in environmental complexity, a variety of interference regions, variation of illumination and the uncertain plate size (Qi-Chang and Na, 2010). Popular algorithms for this purpose can be based on corner detection (Huang et al., 2008; Ma and Zhao, 2011), on wavelet transform (Wen et al., 2009; Rong and Yanping, 2011; Mao et al., 2010), on color features (Xie and Wang, 2010; Zheng, 2010; Shen and Zhang, 2008), on texture of license plates (Liu et al., 2010; Zhang et al., 2010) and on integrated features (Wang et al., 2010; Wang and Yuan, 2010). However, these algorithms have in some way restricted their working conditions, such as limiting them to indoor scenes, stationary backgrounds, fixed illumination, prescribed driveways, limited vehicle speeds or designated ranges of the distance between camera and vehicle (Chang et al., 2004). In addition, algorithms, which are based on neural network (Parisi et al., 1998; Jalil et al., 2003) or on genetic algorithm (Jun et al., 2004), are also implemented for the task of license plate extraction. Algorithms based on neural network are applicable for the particular environment where the neural network is trained and algorithms based genetic algorithm has limited adaptability (Anagnostopoulos et al., 2006). In order to improve the performance of popular algorithms, the rest of the paper improves and implements some of the representative algorithms. The location of license plates is also analyzed, which are color-distorted or covered by shadow.
IMPLEMENTATION AND ANALYSIS OF SOME EXISTING ALGORITHMS
Algorithm 1: License plate location based on corner detection: Corner detection is a texture detection method. As the license plate region has rich corners, corner detection can be used to locate plates. The main steps of algorithm 1 are given as follows:
| Step 1: | Convert the color space of image from RGB to HSV. Because the difference between the saturation components of characters and background in license plates is large, it chooses the S component as the grayscale image to be detected |
| Step 2: | Filter the S component with a median filter (Ge and Song, 2011) and detect the corners with the Harris corner detection algorithm (Harris and Stephens, 1988). It obtains a corner metric matrix C from this step. The corner detector is given by: |
| (1) |
Where:
| (2) |
| (3) |
| with I being the intensity of a grayscale image and * the convolution operator, O(N) denotes the order operator and wu,y is a Gaussian window |
| Step 3: | Find the local maxima in C. Larger values in C correspond to pixels with higher likelihood of being a corner feature |
| Step 4: | Use the H component of HSV to remove the corners that do not meet the color features of license plate |
| Step 5: | Use morphological dilation and hole-filling algorithm to obtain candidate regions of license plates |
| Step 6: | Use morphological opening algorithm to remove candidate regions whose size do not meet the requirements |
| Step 7: | Convert the candidate regions to binary images with Otsu algorithm |
| Step 8: | Discriminate the real license plate from candidate regions according to the white-and-black transitions, aspect ratio and the vertical projection result |
The steps 2-4 in algorithm 1 refer to the method proposed by Huang et al. (2008). Choosing the S component as the grayscale image in step 1 can enhance the edge of characters and background in license plates. Morphological operations in steps 5-6 can form candidate regions and remove candidate regions whose size do not meet the requirements. In the license plate region there are 7 characters and the number of white-and-black transitions is more than 14, so step 8 eliminates the pseudo-regions effectively.
The results of corner detection are shown in Fig. 1. The red points in Fig. 1a represent the detected corners. It can be seen from Fig. 1a that corners are mainly distributed in the license plate region and vehicle’s headlights. The connected components in Fig. 1b are the results of morphological processing. The red rectangle regions in Fig. 1c are the candidate regions of license plate.
Algorithm 1 is tested with a large number of images. For images under appropriate illumination, it is able to detect intensive corners in the license plate region. However, it also increases the number of interference regions and cause difficulties in the subsequent elimination of pseudo-regions.
| Fig. 1(a-c): | License plate location based on corner detection, (a) Corner detection, (b) Connected components and (c) Candidate regions of license plate |
For images shaded by shadow, it is unable to detect enough corners for license plate location. In a word, corner detection is sensitive to illumination and leads to missed detection or expansion of a license plate region.
Algorithm 2: License plate location based on wavelet transform: License plates have rich texture features. The region with high-frequency wavelet transform coefficients corresponds to the license plate region. The main steps of algorithm 2 are given as follows:
| Step 1: | Select the R component from RGB color space as the grayscale image and spread the intensity levels |
| Step 2: | Filter the image with a Gaussian filter |
| Step 3: | Perform wavelet transform on the image and extract the sub-band of vertical high frequencies |
| Step 4: | Convert the sub-band image into a binary image with Otsu algorithm |
| Step 5: | Perform the similar operations as the step 5-8 in algorithm 1 |
The steps 2, 3 and 5 in algorithm 2 refer to the method proposed by Wen et al. (2009). Among the RGB components, the difference of R components of the blue and white pixels in a blue-and-white plate is the greatest. The same is true for the difference of R components of the yellow and black pixels in a yellow-and-black plate. Thus using the R component as the grayscale image in step 1 can enhance the license plate.
The results of location are shown in Fig. 2. Figure 2a is the image after spreading of intensity levels and filtered with a Gaussian filter. The sub-band of vertical high frequencies is shown in Fig. 2b. Figure 2c and d show the connected components and the detected candidate regions, respectively.
| Fig. 2(a-d): | License plate location based on wavelet transform, (a) Grayscale image (b) Sub-band of vertical high frequencies (c) Connected components and (d) Candidate regions |
Tests show that algorithm 2 has great robustness and it locates the license plate efficiently. However, for images with complex environments it locates too many candidate regions, hence increasing the computational complexity in the subsequent steps of elimination of pseudo-regions.
Algorithm 3: License plate location based on color edge detection: The majority of license plates of China have white characters with blue background or black characters with yellow background. So in the license plate region there are rich color edges. The main steps of the algorithm 3 are given as follows:
| Step 1: | Convert an image from RGB space into HSV space |
| Step 2: | Move a window with 3x3 neighborhood in an image to detect the color edges. For pixels in the first and third columns, if one column of pixels is blue and another is white, it implies that the second column in the window is the edge of white-and-blue. The same holds for black-and-yellow edges |
| Step 3: | Perform the similar operations as the step 5-8 in algorithm 1 |
The steps 1 and 3 in algorithm 3 refer to the method proposed by Shen and Zhang (2008). Because the characters in license plate region has rich vertical strokes, so the window choosed in step 2 can detect the color edges.
The result of color edge detection is shown in Fig. 3. Figure 3a is the original image in RGB color space. The white-and-blue edges are shown in Fig. 3b. Figure 3c and d are the connected components and candidate regions, respectively.
| Fig. 3(a-d): | License plate location based on HSV color space, (a) The original image, (b) Image after white-and-blue edge detection, (c) Connected components and (d) Candidate regions |
Figure 3 shows that rich color edges exist in the license plate region compared to the interference regions. Color edge detection is widely used to locate license plates. The method locates license plates accurately in an appropriate illumination condition. However, when a license plate is in shadow or shined by powerful beams, its color is distorted and it is difficult to detect the plate. Experiments show that the difficulty of this algorithm lies in color segmentation (Guo-Quan and Zhan-Ming, 2011). White pixels can be defined as H = 0, S = 0, V = 1. In reality, white pixels used in license plates are not truly white pixels. The values of white pixels in HSV space spread very widely and are distributed discontinuously. Another shortcoming of this algorithm is its too high computational complexity.
Algorithm 4: License plate location based on RGB color space: An algorithm based on RGB color space is proposed by Zheng (2010). However, it does not set strict constraints on the three components as HSV color space does in algorithm 3. As this algorithm relaxes the restrictions on color, it is able to locate the color-distorted license plates. The main steps of algorithm 4 are given as follows:
| Step 1: | Extract the color features of a license plate. The extracted blue feature is defined by: |
| (4) |
| if d1(x, y)>0, d2(x, y)>0 and fb(x, y) = 0 otherwise, where, d1(x, y) = b(x, y)-r(x, y), d2(x, y) = b(x, y)-g(x, y), with b(x, y), r(x, y), g(x, y) denoting RGB components, respectively, k1 = 0.33748 and k2 = 0.66252. Notice that algorithm 4 exploits the fact that for the blue pixels in a blue-and-white plate the B component is the largest among the R, G and B components. Yellow can be viewed as the opposite color of blue and thus the extracted yellow feature is defined by: |
| (5) |
| if d1(x, y)<0, d2(x, y)<0 and fy(x, y) = 0 otherwise. Original image is shown in Fig. 4a while Fig. 4b shows the blue feature image |
| Step 2: | Convert the color feature image into a binary image with Otsu algorithm, as shown in Fig. 4c |
| Step 3: | Perform the similar operations as the step 5-8 in algorithm 1 and the candidate regions are obtained, as shown in Fig. 4d |
| Step 4: | Step 2 and 3 in algorithm 4 are improved. Otsu algorithm in step 2 is generally used to separate the background and object |
After a number of experiments, it can be found that algorithm 4 locates the license plate region effectively, even for color-distorted plates. However, it classifies the interference regions with the same color of the license plate into plate regions, leading to the failure of subsequent precise location of license plate. The algorithm is insensitive to illumination and is able to detect the license plate in shadow.
Algorithm 5: License plate location based on the texture features of license plate: License plate region has rich vertical texture features. The main steps of algorithm 5 are given as follows:
| Step 1: | Select the R component from RGB color space as the grayscale image and spread the intensity levels |
| Step 2: | Filter the grayscale image with wavelet transform to reduce the salt-and-pepper noise (Zheng et al., 2005; Raju et al., 2008) |
| Step 3: | Detect the vertical edges of the grayscale image using Sobel kernel |
| Step 4: | Scan black-to-white transitions from left to right in the row direction. If the distance of adjacent transitions is less than T, the two transitions are on the same segment. T is set to license plate height H, which is decided by the distance of adjacent characters in the license plate. When reaching the end of the segment, count the number of transitions for that segment. If the count is greater than 10, the scanning line on the segment is set to the color of the edge |
| Step 5: | Perform the similar operations as the steps 5-8 in algorithm 1 |
| Fig. 4(a-d): | License plate location based on RGB color space, (a) Original image, (b) Image of extracted blue feature, (c) Connected components and (d) Candidate regions |
| Fig. 5(a-d): | License plate location based on texture features of license plate, (a) De-noised image, (b) Image after texture detection, (c) Connected components and (d) Candidate regions |
Algorithm 5 refers to the basic idea proposed by Liu et al. (2010). However, it improves all the steps to make it locate license plate more correctly. The wavelet transform in step 2 is used to reduce the salt-and-pepper noise. The Sobel kernel in step 3 is used to detect the vertical edges.
The result of texture detection is shown in Fig. 5. Figure 5a is the de-noised image. The texture-detected image is shown in Fig. 5b. Figure 5c and d are the connected components and candidate regions, respectively.
After a number of experiments, it can be found that algorithm 5 locates the license plate region effectively. However, it classifies the interference regions with the texture features that are similar to those of license plate into plate regions, leading to the same failure for algorithm 4.
| Fig. 6: | Flowchart of the location algorithm based on integrated features |
Furthermore, algorithm 5 cannot locate license plates in deep shadow. In order to solve the difficulty of location in deep shadow, we enhance the edge detection to highlight the license plate. Compared with corner detection and wavelet transform based algorithms, texture features based algorithms are less sensitive to illumination.
Algorithm 6: License plate location based on the integrated features of license plate: Wang et al. (2010) proposed a license plate location algorithm based on the integrated features of license plate. Algorithm 6 combines the texture features and color features of a license plate. The flowchart is shown in Fig. 6.
In the flowchart, step 2 is implemented by morphological top-hat filtering on the grayscale image, step 3 is performed by Sobel kernel, Otsu algorithm is used to convert the color image into an intensity image, the color space is transformed into HSV space and the blue color or yellow color is segmented according to the H, V, S components.
Step 1, 3, 4, 5, 7, 8, 9 and 12 are improved in algorithm 6. The results of location based on the integrated features are shown in Fig. 7. Figure 7a is the original image in RGB color space. Figure 7b and c are the connected components of texture features and color features. The image of integrated features, which is obtained by intersection of candidate region sets A and B, is shown in Fig. 7d. Figure 7e shows the candidate region.
| Fig. 7(a-e): | License plate location based on integrated features, (a) Original image, (b) Candidate region set A, (c) Candidate region set B, (d) Image after intersection of A and B and (e) Candidate region |
Experiments show that the result of algorithm 6 is desirable with few interference regions when the illumination condition is balanced. License plates in shadow cannot be detected by Sobel kernel and color segmentation based on HSV color space is also unavailable to color-distorted license plates. As a consequence, when the illumination is too strong or too weak, intersection of texture features and color features can hardly be useful.
COMPARISON OF THE LICENSE PLATE ALGORITHMS
This study tests each algorithm with 500 images, which are selected randomly from a library with 6,000 images with license plates. Table 1 shows the accuracy of license plate location algorithms. The second column is the accuracy of improved algorithms and the last columns is the accuracy of original algorithms. The accuracy of algorithm from 1 to 6 is increased by 5.8, 2.6, 7.8, 4.6, 5.2 and 2.6%, respectively.
Tests show that algorithm 4 and algorithm 5 have high performance and can be applied to complex environment. The rest of four algorithms generally are sensitive to illumination, leading to the failure of license plate location.
| Table 1: | Accuracy of license plate location algorithms |
CONCLUSION
This study improves the image preprocessing, image filtering and discrimination of pseudo-regions in 6 different algorithms based on the corner detection, on wavelet transform, on HSV color space, on RGB color space, on license plate texture and on the integrated features of license plate respectively. The experiments show that the performance of each algorithm is increased significantly. Through the comparison of 6 different license plate location algorithms, it also finds out the shortcomings in these algorithms.
ACKNOWLEDGMENTS
This study is supported by Enjoyor Labs, Enjoyor Inc., Zhejiang Provincial Key Science and Technology Innovative Team (2012R10011-05) and Zhejiang Provincial Key Laboratory of Communication Networks and Applications.