• [email protected]
  • +971 507 888 742
Submit Manuscript
SciAlert
  • Home
  • Journals
  • Information
    • For Authors
    • For Referees
    • For Librarian
    • For Societies
  • Contact
  1. Information Technology Journal
  2. Vol 12 (4), 2013
  3. 569-583
  • Issues
    Online First Current Issue All Issues
  • Information About
    Aims and Scope Editorial Board Guide to Authors Article Processing Charges
    Submit a Manuscript

Information Technology Journal

Year: 2013 | Volume: 12 | Issue: 4 | Page No.: 569-583
DOI: 10.3923/itj.2013.569.583
crossmark

Facebook Twitter Reddit Linkedin E-mail
Research Article

Video-based Water Drop Detection and Removal Method for a Moving Vehicle

Hsien-Chou Liao
Department of Computer Science and Information Engineering, Chaoyang University of Technolog, 168, Jifeng E. Rd., Wufeng District, Taichung, 41349 Taiwan, Republic of China

De-Yu Wang
Department of Computer Science and Information Engineering, Chaoyang University of Technolog, 168, Jifeng E. Rd., Wufeng District, Taichung, 41349 Taiwan, Republic of China

Ching-Lin Yang
Department of Computer Science and Information Engineering, Chaoyang University of Technolog, 168, Jifeng E. Rd., Wufeng District, Taichung, 41349 Taiwan, Republic of China

Jungpil Shin
Graduate School of Computer Science and Engineering, The University of Aizu, Aizu-Wakamatsu City, Fukushima-ken, 965-8580, Japan

ABSTRACT


The removal of water drops from video frames recorded from a moving vehicle is a challenging task with respect to the videos from a fixed scene. It would also enhance the performance of image-based driving assistance systems in rainy weather. Therefore, a method for detecting and removing water drops from video frames is proposed herein. The detection of water drops involves several steps, including histogram equalization, frame intersection, binarization and shape checking. Detected drops are then removed according to their location to achieve a desirable result. In the experimental study, the demonstration shows that the water drops in the video frames can be removed clearly. Simulation results also show that the method can achieve a maximum PSNR (peak signal-to-noise ratio) of 33 dB. These results show the effectiveness of the proposed method.
PDF Abstract XML References Citation

Keywords


  • Raindrop detection
  • driver assistance system
  • pattern recognition
  • image recovery
  • image inpainting
  • noise removal

Article History

Received: October 11, 2012;   Accepted: December 07, 2012;   Published: January 22, 2013

How to cite this article

Hsien-Chou Liao, De-Yu Wang, Ching-Lin Yang and Jungpil Shin, 2013. Video-based Water Drop Detection and Removal Method for a Moving Vehicle. Information Technology Journal, 12: 569-583.

DOI: 10.3923/itj.2013.569.583

URL: https://scialert.net/abstract/?doi=itj.2013.569.583

INTRODUCTION


Now-a-days, cameras are popularly installed in vehicles for improving driving safety, such as rear view cameras for parking; front view cameras for lane departure warnings, front collision warnings, traffic light or sign recognition or vehicle Digital Video Recorders (DVRs) and surrounding cameras for video monitoring and recording. For those cameras installed behind the front windshield of a vehicle, the water drops on the window compromise the clarity of the camera image in rainy weather. Of course, the windshield wiper is assumed to be unable to clean the water drops in front of the camera. An example is shown in Fig. 1. The clarity of the camera image is obviously influenced by the water drops and thus limits the performance of the image-based function. For example, a pedestrian is difficult to be detected in the image with water drops (Mazoul et al., 2012). Therefore, the objective of this study is the detection and removal of water drops from videos recorded from a moving vehicle.

WATER DROP PROBLEM

For a camera installed in a moving vehicle, the objects in the video frames are moving radially away from the center, i.e., the vanishing point.

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
Fig. 1: Water drops influencing the clarity of the camera viewed image

Only the water drops exist for a period of time; thus, foreground object detection methods, such as background subtraction, temporal image differencing, optical flow or Gaussian feature extraction, are unable to detect them. A change in brightness or vehicle vibration also influences the detection of water drops in the video frames. The four successive frames shown in Fig. 2 depict the above conditions. Clearly, the removal of these water drops from video frames is a challenging task.

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
Fig. 2(a-d): Four successive video frames with water drops, (a-d) Frame i to frame i+3

To sum up, three key points must be resolved, in turn, in order to remove water drops from video frames:

• Water drop detection: Water drops on a vehicle windshield have three characteristics. The first is that they exist in the video frames for a period of time. The second is that their shape is close to an ellipse. The third is that the condensation effect causes the water drops to have a high degree of brightness. The proposed method was designed according to these characteristics to prevent the misdetection of water drops
• Patch locating: When a water drop area is detected, it must be replaced by the image covered by the water drop. This area is also called a patch. A patch can be located from the previous frame(s); however, it might be covered by another water drop in the previous frame. A patch is also difficult to locate if the water drop is in the central area, i.e., the area around the vanishing point. These factors increase the difficulty of locating a patch
• Water drop replacement: If a patch can be located successfully, the next key point is the replacement of the water drop. This is similar to the stitching of two images. The elimination of the replacement trace was the main concern here. On the other hand, if there is no patch available, some way must be designed to remove the water drops as much as possible

Many studies have focused on solving the clarity problem in camera images due to bad weather, such as rain, fog and snow. Image restoration is the main goal, as in the removal of rain (Garg and Nayar, 2004, 2005; Zhang et al., 2006), snow (Barnum et al., 2007; Sun et al., 2010) or fog (Murase et al., 2006; Hautiere et al., 2008). Several methods, including temporal and chromatic properties, Kalman filter and Fourier transform, have been used. However, the removal of raindrops or water drops on the windshield of a moving vehicle presents greater difficulty compared to the above removal methods for a fixed scene. Several methods have been proposed. For example, different methods to restore images stained with water drops by Yamashita et al. (2005a, b, 2008), including using a stereo camera or a PTZ (pan-tilt-zoom) camera. Two raindrop detection and restoration methods were proposed by Halimeh and Roser (2009), Roser and Geiger (2009). A photometric raindrop model was defined for detecting the raindrops; then, a multi-band blending method was used to reconstruct the image areas covered by the raindrops. Some image recovery techniques have restored the area covered by a water drop by using its surrounding pixels (Bertalmio et al., 2003; Chan and Shen, 2005) or texture (Criminisi et al., 2004). Such techniques may be useful when the water drops are in a simple background but not in a complex background. In the above related works, only the two studies presented by Halimeh and Roser (2009) and Roser and Geiger (2009) addressed the problem under discussion in the present study. However, raindrops were detected only when it was clear and met the photometric raindrop model; the water drops were removed using the intensity from neighboring image frames. Such a method would be problematic when removing water drops from images with a complex background, such as those shown in Fig. 2.

MATERIALS AND METHODS


The process which has been proposed for overcoming the key points listed in the first section consists of two phases: water drop detection and water drop removal. Each phase consists of several steps designed from the empirical study. The process is shown in Fig. 3.

The video stream is input in the first phase: water drop detection. In this phase, the frames are separated into two areas: roadway and building, based on traffic lane detection. The roadway area is simpler than the building area. The water drops detected in the two areas are removed by different steps; the details are presented in the following subsections.

Water drop detection phase: In this phase, the steps were designed according to the following water drop characteristics:

• A water drop usually appears in the same location in a series of video frames
• The illumination of a water drop is higher than that of the surrounding area
• The shape of a water drop is close to that of an ellipse
• When a water drop is more obvious, the change of the gradient around the boundary is more apparent

Lane detection: In this step, every video frame was divided into roadway and building areas according to the result of lane detection. Initially, there was no such division but it is found from the empirical study that the removal of water drops within the two different areas could not be done in a single process. The background of the water drops in the building area was more complicated than those in the roadway area, so lane detection was performed at the beginning of this phase.

In order to detect the leftmost and rightmost traffic lanes, the bottom half of the frame was processed by dilation and subtracted from the original frame (Gonzalez and Woods, 2002). Then, an optimal binarization method was used to get a clear image of the traffic lanes. A Hough transform was also used to estimate the leftmost lane by finding the desired traffic line with a strength larger than δ (Duda and Hart, 1972); δ was set to 0.3 to ensure the strength of the straight line.

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
Fig. 3: The process of the proposed method

The same step was performed to find the rightmost lane. The Hough transform algorithm is detailed below:

Step 1: Obtain a binary mask of the image by using optimal binarization. The white pixels are deemed as the feature points for the Hough transform
Step 2: Perform the following sub-steps for every feature point:
  • A feature point (xi, yi) is calculated by using Eq. 1 in polar coordinates to obtain the straight line (γ', θ') passing (xi, yi):

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
(1)

  • The accumulator of the line (γ', θ') is plus one
Step 3: After all the feature points are processed, the accumulator of the line with the maximum value is the traffic lane

According to the above steps, the leftmost and rightmost traffic lanes could be detected. An example is shown in Fig. 4.

Image pre-processing: In the building area, the brightness of the water drops might be less obvious but it still influences the clarity of frames for visual perception. Therefore, a histogram equalization method was used to increase the contrast between the water drops and the gray background. The method is as follows:

For the value x of every pixel, 0≤x≤L-1, its probability density function, p(x), is computed by using Eq. 2:

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
(2)

where, nx is the count of the pixels with value x in L (L equals to 256 for a gray scale image) and N is the total number of pixels. Cumulative distribution function c(x) can be mapped from p(x) by using Eq. 3:

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
(3)

Finally, pixel value x is reallocated as I(x) by using Eq. 4:

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
(4)

In roadway areas the detection of water drops was much easier. A Sobel edge operator was used to locate water drop areas with a large gradient change (Gonzalez and Woods, 2002). An example is shown in Fig. 5. Figure 5a is the original video frame. The result of the Sobel edge operator is shown in Fig. 5b. Some objects were detected by the edge operator, such as the traffic light and the vehicle in front and were filtered out by the following steps.

Intersection: According to the characteristics of a water drop appearing in the same location of a series of frames, N successive frames are intersected to preserve the possible water drops in this area.

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
Fig. 4(a-c): Example of traffic lane detection, (a) Original frame, (b) Detection result and (c) The separation of roadway and building areas

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
Fig. 5(a-g): Water drop detection steps (yellow circle: water drop detected by using optimal binarization, red circle: water drop detected by using adaptive binarization), (a) Original frame, (b) Pre-processing, (c) Intersection, (d) Removal, (e) Optimal binarization, (f) Adaptive binarization and (g) OR operation

An example of the intersection result is shown in Fig. 5c for N = 20. Except in a sky area with high brightness, the water drops were clearly visible. The more frames that are intersected, the clearer the water drops; however, each new water drop takes time to be detected. Here, N set to 20 was about 4 sec at 5 FPS (frames per sec).

Binarization: In this step, binarization was used to extract the possible water drop areas. Two binarization methods were used: simple image statistics (SISB) (Gasparini and Schettini, 2004) and adaptive (AB) (Bradley and Roth, 2007; Wellner, 1993). A description of each follows:

SISB method: Assume the size of a possible water drop area is n by m pixels. The value of a pixel at (x, y) is I(x, y).The steps of the SISB method are:

Step 1: Two gradients values, denoted as Ex and Ey, are calculated for every pixel by using Eq. 5 and 6. The maximum value of Ex(x, y) and Ey(x, y) is denoted as MaxG(x, y):

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
(5)

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
(6)

Step 2: For all the pixels, their MaxG(x, y) are summarized and denoted as Total MG:

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
(7)

Step 3: MaxG(x, y) is then multiplied with I(x, y). The sum of the product of all the pixels is calculated and denoted as Total MGI:

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
(8)

The result threshold for binarization, denoted as TSISB, equals the Total MGI divided by Total MG.

AB method: Every pixel is thresholded using the following steps:

Step 1: The intensity, i.e., value, of a pixel at (x, y) is denoted as f(x, y). The sum of all f(x, y) terms to the left and above the pixel (x, y) is denoted as I(x, y). I(x, y) is computed using Eq. 9:

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
(9)

According to the region setting of Pr, the surrounding rectangle D of a pixel (x, y) is determined. Assume the upper left corner is (x1, y1) and the lower right corner is (x2, y2). The sum of f(x, y) over D, denoted as sum(x, y), is computed using Eq. 10:

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
(10)

The thresholding result of the pixel (x, y) is denoted as T(x, y). It equals one if f(x, y) is smaller or equal to the ratio of sum(x, y) determined by Pb as shown in Eq. 11:

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
(11)

The above steps are depicted in the example shown in Fig. 6. The input pixel values are shown in Fig. 6a, with the pixels with higher values marked in bold. The results of I(x, y) and sum(x, y) computed using Eq. 9 and 10 are shown in Fig. 5b, c, respectively. Finally, the thresholding results of the pixels using Eq. 11 are shown in Fig. 6d. The pixels with higher values were extracted successfully.

For the example shown in Fig. 5c, the binarization results of SISB and AB are shown in Fig. 6e, f, respectively. The brightness difference parameter Pb and the region parameter Pr for AB were set to 0.005 and 51, respectively.

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
Fig. 6(a-d): Example of adaptive binarization (AB) method (Pr = 5 and Pb = 0.5), (a) Input, (b) Integral image, (c) Sum(x, y) and (d) Result

The results show that the water drops in the SISB results were clearer than those in the AB results, as with the drops marked by the yellow dashed circle because when water drops fell near the border of the frame or when the water content was not enough to reflect the light; these water drops, missed in the results of SISB, were available in those of AB, such as those marked by a red dashed circle.

In order to get a clear result of SISB and include the missing drops of AB, every blob (binary large object) found in SISB has the OR operation performed with the corresponding blob in AB. Then, the unmatched blobs in AB also have the OR operation performed with the corresponding area in SISB. The steps of the SISB and AB methods were as described and the result is shown in Fig. 5g.

Shape/mean value checking: The detected blobs in the previous step may contain some misdetected ones. They are eliminated by checking whether their shape and pixel values satisfy the characteristics of a water drop. Firstly, the shape of a water drop is usually close to an ellipse. In order to check the shape of a blob, an ellipse mask with the same size of the blob is generated. Then, the blob is differenced with the mask. A ratio (η) is computed by dividing the number of white pixels of the difference result with the total number of pixels. The blob passes the shape checking if its ratio, η, is less than a pre-defined threshold, ε = 0.3. Two examples of shape checking are shown in Fig. 7. There are three images in Fig. 7a; the water drop, ellipse mask and difference result from left to right. The ratio of the difference result was 0.18. Conversely, the ratio of the difference result in Fig. 7b was 0.45, so it did not pass the shape checking.

If a blob passes the shape checking, the next checking is the mean pixel value. The mean pixel value of a blob is computed and denoted as μ. Then, the mean pixel value of its surrounding eight-connection areas with the same size of the blob is computed and denoted as M.

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
Fig. 7(a-d): Example of shape/mean value checking, (a-b) Shape checking (blob, ellipse mask, difference result) and (c-d) Two eight-connection examples

A blob passes the mean pixel value checking if its μ is larger than M. Two examples are shown in Fig. 7c-d.

WATER DROP REMOVAL PHASE

The water drop areas which have been detected are removed in this phase. The removal consists of two main steps: one is the patch location for every water drop in the previous frame and the other is the replacement of every water drop area. The vehicle is assumed to be driving straight on; therefore, the buildings or roads in the video frames move radially away from the vanishing point, so that the greater the distance between the object and the vanishing point, the larger the displacement between two successive frames.

The following steps were designed to achieve the abovementioned removal. Template matching was used to locate patches in the building area and morphological operation and image inpainting methods were used to remove water drops in the roadway area.

Template matching: Several methods were tried in order to locate the patch of a water drop in the previous frame, such as optical flow or Gaussian feature matching. However, these failed for various reasons, such as video shaking or brightness/size change. A template matching method was used here to locate the desired patch (Lewis, 1995). The template matching method requires a template and a Region of Interest (ROI) for matching. For every detected water drop area, the template was an eight-connection area, as shown in Fig. 7d. The size of the ROI in the previous frame was set to 80x150 pixels. If the water drop area was on the right side of the building area, the area to be located was on the left side of the template. The right margin of the ROI was aligned along the left margin of the template. After the template and ROI were determined, they were resized to one quarter of the original size to shorten the matching time. A popular method, called the sum of absolute difference (SAD) (Vanne et al., 2006), was used here to compute the similarity of any possible areas within the ROI. The area with the highest matching similarity was deemed as the patch and used to replace the water drop area in the current frame.

An example of template matching and replacement is shown in Fig. 8. The original video frame is shown on the left and the replacement result is on the right. In Fig. 8a, the water drops have not yet been detected; therefore, the original and replacement frames are the same. After the template matching and replacement were performed, the replacement result was obviously better than the original frame.

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
Fig. 8(a-c): Example of template matching and replacement, (a) Frame 1, (b) Frame 5 and (c) Frame 9

Morphological operation and inpainting: In the roadway area, the template matching method was not effective since the roadway area is quite similar. Therefore, a morphological operation and image inpainting were used in turn to remove the water drops. The difference between the morphological operation and image inpainting is shown in Fig. 9. The original water drop image is shown in Fig. 9a. The water drop area detected in the previous phase is shown in Fig. 9b. As can be observed in the two examples in Fig. 9c-d, the result of the morphological operation was better than that of image inpainting. However, the morphological operation was not suitable for all the cases in the roadway area. The removal of a water drop depends on whether the drop is covering the traffic lane. For a water drop that did not cover the traffic lane, such as the examples in Fig. 9, the removal steps for every water drop are as follows:

Step 1: The area to be processed was set as 1.5 times the detected area, as marked by the red dashed rectangle in Fig. 9a
Step 2: Two dilations and five erosions were performed in the marked area
Step 3: Image inpainting was performed on the above result to make the water drop area close to the boundary of the marked area

A water drop which covered the traffic lane could be detected by intersecting the water drop with the traffic lane in the previous step, as shown in Fig. 4b. The water drop was removed by using the image inpainting method, as in the example shown in Fig. 10. Two water drops, as marked by blue dashed circles, were clearly removed.

Some special cases of water drops were also handled here:

• If template matching failed because the water drop was within a large similar area, e.g., a wall or a row of identical houses, the image inpainting method was used to remove the water drop. An example is shown in Fig. 11a
• When a water drop was close to the vanishing point, the area it covered showed almost no change in two successive frames. Neither template matching nor a morphological operation could be used to remove the water drop. Here, image inpainting was used to reduce the masking effect of the water drop. An example is shown in Fig. 11b

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
Fig. 9(a-d): Difference between morphological operation and image inpainting (red square boxes: the areas processed separately by the inpainting and morphological operations), (a) Original, (b) Mask, (c) Inpainting result and (d) Morphological operation result

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
Fig. 10(a-b): Inpainting replacement of water drop (a) Before and (b) After covering traffic lane (blue circles: the areas processed by the inpainting operation)

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
Fig. 11(a-b): Two special cases using image inpainting directly, (a) Template matching failure and (b) Water drops close to the vanishing point

EXPERIMENTAL STUDY

A tool was implemented using C# and AForge.NET in order to evaluate the performance of the proposed method. The evaluation consisted of three parts. Firstly, three test videos were used to demonstrate the performance of the proposed method. Separate recordings were made of small, medium and large numbers of water drops. Secondly, the Peak Signal-to-noise Ratio (PSNR) was used to evaluate the quality of the image after water drop removal. Thirdly, the processing time of the proposed method was measured. A description of each of the three parts of the evaluation follows.

Demonstration: The results of the three test videos are shown in Fig. 12-14, respectively. By comparing the original frame and the result frame, it is apparent that the water drops have been removed. The clarity of the frame with the large number of water drops was significantly improved. The removal traces were also increased with the increase in water drops. The images in the test video with a large number of water drops, as shown in Fig. 15, were taken every second. The water drop detection phase lasted 4 sec (N = 20). In Fig. 15a-d the water drops have not yet been removed; in Fig. 15e-i, the water drops have been removed and the clarity has obviously improved. The results demonstrate that the proposed method achieved its purpose of removing water drops in videos recorded from a moving vehicle.

PSNR evaluation: PSNR was used to evaluate the image quality after removal of the water drops. However, video frames without water drops were needed to compute PSNR. Several attempts were made but found that it was difficult to record videos with and without water drops at the same time. Therefore, water drops were manually placed in the video frames using an image editing tool (Photoshop) in order to generate video frames with water drops. The number of water drops for small, medium and large conditions were 15, 45 and 75, respectively. The results of the simulated frames are shown in Fig. 16. Figure 16a is one of the original video frames without water drops. A small, medium or large number of water drops was added to the original frame separately. The results are shown on the left side of Fig. 16b-d. The corresponding results of the water drop removal are shown on the right side. The results clearly show that the small number of water drops was removed.

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
Fig. 12(a-d): Example of removal of a small number of water drops, (a) Original, (b) Intersection, (c) Detected water drops and (d) Result

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
Fig. 13(a-d): Example of removal of a medium number of water drops, (a) Original, (b) Intersection, (c) Detected water drops and (d) Result

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
Fig. 14(a-d): Example of removal of a large number of water drops, (a) Original, (b) Intersection, (c) Detected water drops, (d) Result

As the number of water drops increased, it was clear that partial drops were not removed. A comparison of the simulation results with the demonstrations, shown in Fig. 12-14, show that the demonstrations were better than the simulation results. The main reason was that the proposed method was designed for real water drops. If the simulated water drops were similar to the real ones, the simulation results would be similar to the above demonstrations.

The PSNR of the frames after the removal of the water drops and the corresponding original frames without water drops were computed, as depicted in Fig. 17.

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
Fig. 15(a-l): Results of the test video with a large number of water drops captured every second, (a-l) the processed frames in 12 sec interval

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
Fig. 16(a-d): Results for manually placed water drops, (a) Sample frame and Manual placement of (b) Small, (c) Medium and (d) Large number of water drops and the removal result

Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle
Fig. 17: PSNR results for different numbers of water drops

Table 1: Processing time of the proposed method
Image for - Video-based Water Drop Detection and Removal Method for a Moving Vehicle

There were 90 frames in total. The PSNR of the frames with a small number of water drops was higher than for those with a medium or large number of water drops. The average PSNR was 33.0, 29.1 and 26.9 dB for a small, medium or large number of water drops, respectively. In general, a PSNR over 30 dB is considered acceptable for frame quality. Only the PSNRs of the frames with a small number of water drops satisfied this criterion. However, the method and the related parameters were not adjusted for the simulated water drops. It would be expected that the PSNR of frames under real conditions would be higher than the current results.

Processing time: The processing time was measured in order to determine the performance of the proposed method. The implementing tool was executed on a 2.7 GHz PC with 2 GBytes RAM. The average processing times for different numbers of water drops are listed in Table 1. As expected, the processing time of frames with a small number of water drops was shorter than for those with a medium or large number of water drops. The overall time was from 732.1 to 971.8 msec. The detection phase was less influenced by the number of water drops. The processing time of the removal phase increased as the number of water drops increased because the steps, such as template matching or water drop replacement, were related to the number of water drops.

DISCUSSION


Although, the proposed method shows promise, based on the demonstration and simulation results, several issues remain to be resolved, as per the following discussion.

• When a water drop was too large, the area covered by the drop appeared after several frames. This meant that the template matching method may simply have failed to locate the patch from the previous frame. The locating step should be refined to locate the desired patch from the previous N frames
• When a water drop was close to the vanishing point, the area covered by the drop appeared after a long time. Although the image inpainting method was used to solve this problem, the result was less than perfect and the method should be improved further.
• The test videos were recorded by an in-vehicle DVR when the vehicle was driving straight on under rainy conditions. When the vehicle was turning, objects in the video frames moved quickly to the left or right which made it difficult to locate the patch during this period. Fortunately, the vehicle was driving nearly straight on most of the time so the proposed method was effective. On the other hand, when the vehicle was stopped at a traffic light or in a traffic jam, the method failed to detect and remove the water drops. However, the driver was quite safe and the method was again effective after the vehicle was moving again
• Most of the previous studies didn’t handle the water drop removal of the camera image in a moving vehicle. Only one attempt was made to compare the proposed method with a previous study, presented by Halimeh and Roser (2009), by performing the water drop removal function on their test videos. However, it was found that the wiper was turned on and the water drops were cleaned every 5 sec in their test videos, so the proposed method was unable to perform water drop detection since 20 video frames, i.e., 4 sec, were needed in the intersection step of the current prototype which is why the above simulation was designed to evaluate the method using the standard measure of PSNR. Such results will be easy to compare with the further studies in the future

CONCLUSION


In this study, a water drop detection and removal method has been proposed. Firstly, an intersection operation was used to detect all of the possible water drop areas. Then, every water drop area was removed according to its location in either a roadway or building area. In a complex building area, the template matching method was used to locate the patch for water drop replacement. In a simple roadway area, a morphological operation and image inpainting method were used to remove the water drops. Some special cases, such as with an area close to the vanishing point or rain drops on the traffic lane, were also handled to improve the clarity of the video frames.

The proposed method was able to detect not only the obvious water drops but also those water drops around the corners or even their traces. The demonstrations show that the clarity of the video frames was improved significantly for visual inspection. The simulation results also show that the proposed method provided an acceptable image quality for a small number of water drops. Several issues have been discussed previously that are worth further study. In conclusion, the proposed method could be applied to other kinds of vehicles, such as automatic mobile vehicles or unmanned vehicles.

ACKNOWLEDGMENT


This study was partially supported by I-Services project funded by the Ministry of Education, Taiwan.

REFERENCES


  1. Hautiere, N., D. Aubert, E. Dumont and J.P. Tarel, 2008. Experimental validation of dedicated methods to in-vehicle estimation of atmospheric visibility distance. IEEE Trans. Instrum. Meas., 57: 2218-2225.
    CrossRef

  2. Barnum, P., S. Narasimhan and T. Kanade, 2007. Spatio-temporal frequency analysis for removing rain and snow from videos. Proceedings of the 1st International Workshop on Photometric Analysis for Computer Vision, October 14, 2007, Rio de Janeiro, Brazil, pp: 1-8.

  3. Bradley, D. and G. Roth, 2007. Adaptive thresholding using the integral image. J. Graph. GPU Game Tool, 12: 13-21.
    CrossRef

  4. Chan, T.F. and J.J. Shen, 2005. Variational image inpainting. Commun. Pure Applied Math., 58: 579-619.
    CrossRef

  5. Criminisi, A., P. Perez and K. Toyama, 2004. Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process., 13: 1200-1212.
    CrossRef

  6. Garg, K. and S.K. Nayar, 2004. Detection and removal of rain from videos. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Volume 1, June 27-July 2, 2008, Washington, DC., USA., pp: 528-535.
    CrossRef

  7. Garg, K. and S.K. Nayar, 2005. When does a camera see rain?. Proceedings of the 10th International Conference on Computer Vision, Volume 2, October 17-21, 2005, Beijing, China, pp: 1067-1074.
    CrossRef

  8. Gasparini, F. and R. Schettini, 2004. Color balancing of digital photos using simple image statistics. Pattern Recognit., 37: 1201-1217.
    CrossRef

  9. Gonzalez, R.C. and R.E. Woods, 2002. Digital Image Processing. Pearson Education Inc., Upper Saddle River, NJ., USA.

  10. Halimeh, J.C. and M. Roser, 2009. Raindrop detection on car windshields using geometric-photometric environment construction and intensity-based correlation. Proceedings of the IEEE Intelligent Vehicle Symposium, June 3-5, 2009, Xi'an, Shaanxi, China, pp: 610-615.
    CrossRef

  11. Duda, R.O. and P.E. Hart, 1972. Use of the Hough transformation to detect lines and curves in pictures. Commun. ACM, 15: 11-15.
    CrossRef

  12. Lewis, J.P., 1995. Fast template matching. Proceeding of the Fast Template Matching, Vision Interface, May 15-19, 1995, Quebec, Canada, pp: 120-123.

  13. Zhang, X., H. Li, Y. Qi, W.K. Leow and T. Ng, 2006. Rain removal in video by combining temporal and chromatic properties. Proceedings of the IEEE International Conference on Multimedia and Expo, July 9-12, 2006, Toronto, Canada, pp: 461-464.
    CrossRef

  14. Murase, H., I. Ide, K. Mori, T. Kato, T. Miyahara, T. Takahashi and Y. Tamatsu, 2006. Visibility estimation in foggy conditions by in-vehicle camera and radar. Proceedings of the 1st International Conference on Innovative Computing, Information and Control, Volume 2, August 30-September 1, 2006, Beijing, China, pp: 548-551.
    CrossRef

  15. Roser, M. and A. Geiger, 2009. Video-based raindrop detection for improved image registration. Proceedings of the IEEE 12th International Conference on Computer Vision Workshops, September 27-October 4, 2009, Kyoto, Japan, pp: 570-577.
    CrossRef

  16. Bertalmio, M., L. Vese, G. Sapiro and S. Osher, 2003. Simultaneous structure and texture image inpainting. IEEE Trans. Image Process., 12: 882-889.
    CrossRef

  17. Vanne, J., E. Aho, T.D. Hamalainen and K. Kuusilinna, 2006. A high-performance sum of absolute difference implementation for motion estimation. IEEE Trans. Circuits Syst. Video Technol., 16: 876-883.
    CrossRef

  18. Wellner, P.D., 1993. Adaptive thresholding for the digitaldesk. Technical Report EPC-1993-110. http://www.xrce.xerox.com/content/download/18464/133152/file/EPC-1993-110.pdf.

  19. Yamashita, A., K.T. Miura, T. Kaneko and Y. Tanaka, 2005. Restoration of images stained with waterdrops on a protection glass surface by using a stereo image pair. Proceedings of the IAPR Conference on Machine Vision Applications, May 16-18, 2005, Tsukuba Science City, Japan, pp: 152-155.
    Direct Link

  20. Yamashita, A., T. Harada, T. Kaneko and K.T. Miura, 2005. Virtual wiper-removal of adherent noises from images of dynamic scenes by using a pan-tilt camera. Adv. Robot., 19: 295-310.
    CrossRef

  21. Yamashita, A., I. Fukuchi, K.T. Miura and T. Kaneko, 2008. Removal of adherent noises from image sequences by spatio-temporal image processing. Proceedings of the IEEE International Conference on Robotics and Automation, May 19-23, 2008, Pasadena, CA., USA., pp: 2386-2391.
    CrossRef

  22. Sun, Y., X. Duan, H. Zhang and Z. Yu, 2010. A removal algorithm of rain and snow from images based on fuzzy connectedness. Proceedings of the International Conference on Computer Application and System Modeling, Volume 5, October 22-27, 2010, Taiyuan, China, pp: 478-481.
    CrossRef

  23. Mazoul, A., K. Zebbara and M. EI Ansari, 2012. Street crossing pedestrian detection based on edge curves motion. Int. J. Comput. Appl., 41: 20-24.
    CrossRefDirect Link

Leave a Comment


Your email address will not be published. Required fields are marked *

Useful Links

  • Journals
  • For Authors
  • For Referees
  • For Librarian
  • For Socities

Contact Us

Office Number 1128,
Tamani Arts Building,
Business Bay,
Deira, Dubai, UAE

Phone: +971 507 888 742
Email: [email protected]

About Science Alert

Science Alert is a technology platform and service provider for scholarly publishers, helping them to publish and distribute their content online. We provide a range of services, including hosting, design, and digital marketing, as well as analytics and other tools to help publishers understand their audience and optimize their content. Science Alert works with a wide variety of publishers, including academic societies, universities, and commercial publishers.

Follow Us
© Copyright Science Alert. All Rights Reserved