Abstract: In this study, we present a new calibration method for the cooperation of wide-angle and Pan-Tilt-Zoom (PTZ) cameras. A wide-angle camera enables the surveillance of a large area. However, it may be difficult to perceive the detail of an object. On the other hand, a PTZ camera enables the surveillance of the object detail. However, its view is limited when the camera zooms on a specific object. Therefore, the cooperation can complement the disadvantage of two types of cameras. For an object appears in the image of wide-angle camera, the desired pan and tilt angles of a PTZ camera for focusing on the object can be computed from its image coordinate. Earlier methods can be classified into three categories: 3D world coordinate system, scene-dependent, or special technique. They are not simple and instinctive enough. The proposed method is simply based on the 2D image plane of wide-angle camera. A set of calibration points are selected directly on the image. Every point is a pair of pan/tilt angles (p, t) and 2D coordinate (x, y). Then, the calibration parameters are calculated from the set of points. An experiment was designed to evaluate the accuracy of the proposed method at fixed points. The results show that the proposed method is accurate enough for the cooperation of dual cameras. In advance, a prototype system for capturing close-up images of moving objects in an open area was implemented to demonstrate the feasibility of the proposed method.
INTRODUCTION
Pan-Tilt-Zoom (PTZ) cameras are popular used in the surveillance system currently. Its advantages include the capabilities of monitoring a large area, tracking a moving object and providing the detail of the object. However, when a PTZ camera zooms on an object, it is unable to perceive the other objects outside the current view. Therefore, a dual-camera system is proposed in recent years by incorporating a wide-angle camera and PTZ camera for providing a better surveillance capability than a single camera.
When a moving object appears in the image of the wide-angle camera, the desired pan and tilt angles must be computed and sent to the PTZ camera for object capturing or tracking. The cooperation of two types of cameras relies on a calibration method. The method provides the necessary parameters for the computation of desired pan and tilt angles from the object`s coordinate in the image plane. Many calibration methods were proposed in the earlier studies. Some methods are based on the common 3D world coordinate system of two cameras. The transformation from 3D coordinate system to 2D image plane is based on the operations of coordinate translation, scaling and rotation. Others are scene-dependent. They rely on the specific features in the surveillance scene. The method cannot be used in a scene without the features. Others utilize special techniques to establish the calibration parameters of two cameras.
In this study, a new calibration method is proposed. Unlike the previous methods that are based on 3D world coordinate system, scene-dependent, or special techniques, the proposed method simply based on the 2D image plane of the wide-angle camera. A set of calibration points are established and every point is a pair of pan/tilt angles (p, t) and 2D coordinate (x, y). After the calibration parameters are calculated from the set of points, the 2D coordinate of a moving object (x, y) is used to compute the desired pan/tilt angles (p, t) for controlling the PTZ camera to focus on the object. An experiment was designed to evaluate the accuracy of the proposed method. A prototype system for moving object capturing was also implemented for demonstrating the feasibility of the proposed method.
The calibration method of cameras is important for the cooperation of multiple cameras. The methods of the earlier studies can be classified into three categories:
| • | 3D world coordinate system: This category of methods is based on a common 3D world coordinate system for multiple cameras to compute the desired pan and tilt angles of PTZ camera (Shen and Menq, 2001). Bodor et al. (2004) proposed a dual camera system for activity recognition. The calibration is based on the inverse kinematics for computing the pan and tilt angles from the 3D world coordinate. Marchesotti et al. (2002, 2005) also proposed a dual-camera system for face detection. The calibration method is based on Tsai`s method (1987). At least 12 pairs of 2D image plane and 3D world coordinates are needed to achieve precise calibration. Menudet et al. (2008) proposed a similar self-calibration method based on the image plane |
| • | Scene-dependent: This category of methods is depended on some specific features in the scene. For example, Song and Tai (2006) proposed a calibration method for traffic monitoring. It is mainly based on the parallel lane markings and the lane width to compute the camera parameters. Lv et al. (2006) proposed a self-calibration method from vertical line segments of the same height. Chen and Wang (2006) also proposed a novel and efficient method for the calibration of multiple PTZ cameras. The PTZ cameras are mounted on the ceiling of indoor environment. Therefore, the calibration is based on the tilt angle and altitude of PTZ cameras. The image processing technique is also used to infer the relative positioning and orientation among multiple PTZ cameras. However, the calibration method is invalid when PTZ cameras are not mounded on the ceiling |
| • | Special technique: This category of methods utilizes special techniques for camera calibration. For example, Xing et al. (2007) proposed a calibration method based on the genetic algorithm. Tresadern and Reid (2008) proposed a calibration based on the human motion. Davis and Chen (2003) proposed a more complete calibration method for pan-tilt cameras. Researchers use an improved model of camera to provide a large virtual calibration target by tracking the motion of a LED point feature. However, it is troublesome to obtain the calibration data for the variety of cameras. Lee et al. (2005) also proposed a manual calibration method. An image registration technique is used to establish the panning, tilting and zooming tables separately. However, the construction of the pan/tilt/zoom tables is also a problem for variety types of cameras |
Some disadvantages are mentioned in the earlier studies. Therefore, a new calibration method is proposed simply based on 2D image plane without considering the 3D world coordinate system. It is also scene-independent and general for the cooperation of wide-angle and PTZ cameras.
CALIBRATION METHOD
The method is designed to obtain the values of desired parameters for the computation of pan/tilt angles from a coordinate on the 2D image plane. The pan and tilt operations of a PTZ camera is independent. Thus, the calibration of pan and tilt angles can be processed separately. A set of points are needed for the pan or tilt calibration. Every point is a pair of a coordinate (x, y) on the 2D image plane of wide-angle camera and the corresponding pan/tilt angles (p, t) of PTZ camera. The process of the calibration method is shown in Fig. 1.
The calibration and computation of pan angle: Three points are needed for the calibration of pan angle according to the parameter estimation. Assume the three points are denoted as {(xi, yi), (pi, ti)} for 1 ≤ i ≤ 3. The model of the pan angle calibration and computation is shown in Fig. 2. The camera position is marked as O and its sight line of zero pan angle exists α degrees with the y-axis of the 2D image coordinate system as shown in the Fig. 2. Three calibration parameters, x0, y0 and α, are estimated firstly. Then, the desired pan angle p` for a given (x`, y`) can be computed based on the three parameters.
| Fig. 1: | The calibration process |
| Fig. 2: | The model of pan angle calibration and computation |
Firstly, the equation for calculating cot (p1+α) can be expressed in Eq. 1.
| |
(1) |
Similarly, the equations for calculating cot (p2+α) and cot (p3+α) are shown in Eq. 2.
| |
(2) |
The three parameters, x0, y0 and α, can be calculated from the above three equations. After the parameters are estimated, the desired p` for a given (x`, y`) can be computed based on Eq. 3.
| |
(3) |
The calibration and computation of tilt angle: The model of the tilt angle calibration and computation is shown in Fig. 3. Assume the camera position is marked as O, its height is h and its sight line of zero tilt angle exists θ degrees with the horizontal line as shown in the Fig. 3. The parameter h is a hypothetical value for tilt calibration but not the physical height of the camera. Two calibration parameters, h and θ, are estimated firstly. Then, the desired tilt angle t` for a given y` can be computed based on the two parameters.
According to the trigonometric formula, the calculation of the length
from y1 to y2, denoted as
| Fig. 3: | The model of tilt angle calibration and computation |
| |
(4) |
The equation of
| |
(5) |
Let m1 = cot (t1) – cot (t2), n1 = cot (t1)+cot (t2) and k1 = cot (t1)x cot (t2), the Eq. 5 is rewritten as Eq. 6.
| |
(6) |
Similarly, let m2 = cot (t2) – cot (t3),
n2 = cot (t2)+cot (t3) and k2
= cot (t2) x cot (t3), the computation of
| |
(7) |
Then, Eq. 6 is divided by Eq. 7. The result is shown in Eq. 8.
| |
(8) |
Let,
| |
(9) |
Therefore, the solutions of Eq. 9 are shown in Eq. 10.
| |
(10) |
There are two solutions for Eq. 10. However, the negative solution is not considered in the calibration model. The equations of two parameters θ and h are shown in Eq. 11.
| |
(11) |
Then, the desired tilt angle t` for a given y` can be computed according to Eq. 12.
| |
(12) |
EXPERIMENTAL STUDY
A tool was implemented by using Visual Studio 2005 in order to evaluate the accuracy of the proposed calibration method. Two cameras were installed on our building and focus on a square. The model of the wide-angle camera is AXIS-207 and the model of the PTZ camera is AXIS-214 with 18X optical zoom as shown in Fig. 4.
The experiment is separated into three steps. Firstly, the tool is used to setup calibration points and generate the desired parameters for computing the pan/tilt angles. Three different sets of calibration points are chosen according to their distribution. They are used to realize the influence of chosen calibration points to the proposed method. Secondly, 24 testing points covering the whole square are chosen and a mark piece is placed on every point. The coordinate of every testing point is used to compute the pan and tilt angles according to the proposed method. The computed angle is recorded and sent to the PTZ camera to focus on the mark piece.
| Fig. 4: | The cameras used in the experiment |
| Fig. 5: | An example of five calibration points |
Finally, the PTZ camera is adjusted manually to let the mark piece in the center of the image. The correct pan/tilt angles are recorded, too. The difference between the computed angles and correct angles of pan and tilt are calculated separately. The evaluation of the proposed method is not based on the difference in pixels on the camera image since the value is easily influenced by the zoom of the camera.
Five calibration points, denoted as C1~C5, for coordinate transformation are shown in Fig. 5. The image coordinates and corresponding pan and tilt angles of five points are recorded. Three points, C1, C2 and C3, for the tilt angle calibration are chosen about a vertical line. Another three points, C1, C4 and C5, are chosen to form an triangle for the pan angle calibration. Additional two sets of calibration points are also chosen in the experiment. One is more concentrated, the other is more distributed than the above points.
Then, 24 points are chosen for evaluating the accuracy of the proposed method as shown in Fig. 6. A 32x32 cm piece is placed on every point to mark its location. These points are similar to a grid covering the whole square in order to perform a complete testing.
The computation of angle difference for every testing point is shown in Fig. 7.
| Table 1: | The experimental results of fixed testing points |
| Com.: Computed, Cor.: Correct, Diff.: Difference = |Com-Cor|, Con.: Concentrated, Nor.: Normal, Dis.: Distributed | |
| Fig. 6: | An example of the 24 testing points |
When the computed pan and tilt angles are sent to the PTZ camera, the mark piece may be not in the center of the camera image as shown in Fig. 7a. Then, the camera is adjusted manually to enable the piece in the center of the image as shown in Fig. 7b. The corresponding pan and tilt angles are the correct values. The difference between the computed and correct pan/tilt angles are calculated separately to evaluate the accuracy of the proposed method.
The experimental results of normal, concentrated and distributed calibration points are shown in Table 1. The computed and correct values of the pan angle are listed on the left part of the table.
| Fig. 7: | Angle difference computation (a) The image of computed angles and (b) Correct angles |
Only the difference values of the tilt angle are listed on the right part of the table for the page limitation. For the concentrated calibration points, the average difference of pan and tilt angles is 0.63 and 0.72°, respectively. The maximum difference of pan and tilt angles is 1.51 and 2.32°, respectively. The standard deviation (σ) of pan and tilt angles is 0.44 and 0.63, respectively.
For the pan angle, we found that the result of concentrated calibration point are the smallest among the three sets. The results of normal calibration points are the smallest for the tilt angle. It provides an important guideline on choosing calibration points to increase the accuracy of present method.
The experimental results of fixed testing points show that the proposed method is accurate enough for the cooperation of wide-angle and PTZ cameras. However, the location of a moving object is changed continuously in the practical environment.
PROTOTYPE SYSTEM
The experiment is mainly focus on the accuracy of fixed locations. Although the results show that the method is accurate enough, the performance of the method in the practical environment is further evaluated. A prototype system was implemented for capturing moving objects in an open area. A close-up image of objects are captured and saved when they pass through the area. The saved images are useful for the inspection when some special events occur, e.g., suspects identification. In order to evaluate the performance of the prototype, one wide-angle and two PTZ cameras were installed on the forth floor of our building. The distance between the open area and cameras is about 50 m. A moving object in the real-time image of a wide-angle camera is detected by integrating a convenient motion detection algorithm (Kirillov, 2007). The coordinate of the object is predicted in order to let PTZ camera be ready before the object moving through its camera image. A linear prediction method based on the difference of preceding coordinates is used here. Then, the predicted coordinate of an object is used to estimate the corresponding pan/tilt angles according to the proposed method. The angles are sent to PTZ camera for capturing close-up images. The screen shot of the prototype is shown in Fig. 8.
The real-time image of wide-angle camera is displayed at the center of the user interface. The rectangle marked by a white-dashed line is a pre-defined monitoring range. The capturing task is activated when an object appears in the range. The real-time image of PTZ cameras are displayed at the top of images on the right-hand side. The prototype system can support multiple PTZ cameras to dispatch capturing tasks to these cameras. Two PTZ cameras are connected as shown in the above screen shot. Four images marked by a red-dashed rectangle are continuous four shots of a moving object after PTZ camera focusing on it. Such a design takes into account the object moving and the camera rotating. The image is blurred while the PTZ camera is rotating. The four shots enable at least two clear images could be found among them. The actual size of these captured images is 640x480 pixels.
When a capturing task is finished, a checking step is also designed to verify whether an object is included in the captured images or not. The checking step is based on the difference of the last two images. If there is a moving object in the last two images, there exists a specific number of pixels in the difference result of two images. Otherwise, there is no moving object and the capturing task is dispatched again. An example of the checking step is shown in Fig. 9. The last two shots of a moving object is shown in Fig. 9a and b.
| Fig. 8: | The screen shot of the prototype system |
| Fig. 9: | Examples of the checking step (a) and (b) the last two captured images (c) difference (d) Sobel edge detection (e) thresholding (170, pixel count = 3,676) (j) pixel count = 745 and (o) pixel count = 24,223 |
The difference of two images are shown in Fig. 9c. A Sobel edge detection technique is used to enhance the difference and the result is shown in Fig. 9d. Then, the thresholding of the enhanced image is used to convert grayscale image to black and white as shown in Fig. 9e. The count of the white pixels is used to determine the result of the checking step. The checking is passed when the pixel count is within 1,200 and 12,000 which is obtained from the empirical study. Two examples with low and high pixel counts are shown in Fig. 9j and o. Their checking results are both failure. The captured images are discarded and the object will be scheduled to be captured again.
The operation of the prototype system is presented below. Initially, when an object moves into the monitoring range, it is marked by a rectangle with red border. When the capturing task of the object is dispatched to a PTZ camera, its color is changed to blue and the camera number is labeled outside the rectangle. After the continuous four shots is done and pass the checking step, the rectangle color of the moving object is then changed to green. Otherwise, its color is still blue.
The performance of the prototype system is evaluated by its success/failure percentage of object capturing. For a capture task passing the checking step, it is classified into one of the following categories after inspecting the four captured images:
| • | Success: Head is the most important feature of people(s). If the head(s) appears completely in one of four captured images, the capture task is deemed as success |
| • | Portion: A capture task is partially accomplished when the body of people(s) is captured except the head in all the four captured images |
| Fig. 10: | Several examples of capturing tasks (a) Success (b) Portion (c) Failure (pixel count = 5,032) and (d) Failure (pixel count = 3,006) |
| • | Failure: A capture task is failure when there is no moving object in the four images |
Several examples of these categories are shown in Fig. 10. The success and portion examples are shown in Fig. 10a and b, respectively. The distinction of two categories is obvious. Two failure examples are shown in Fig. 10c and d. They are the last two images of the four captured images. The pixel count of Fig. 10c in the checking step is 5,032. It is caused by the complicate ground and a little displacement between two images. Similarly, the right image of Fig. 10d is a bit blur caused by the camera rotation. The checking step is passed because the pixel count is 3,006, but the capturing task is failure.
The evaluation results of the prototype system are shown in Table 2. The time period is 40 min. The total capturing tasks dispatched within 40 min is 743. They consist of 287 tasks that pass the checking step and 456 tasks that do not pass the checking step and are discarded. Among 287 tasks, 87.8% of objects are captured successfully. There are less than nine percentage is portion and less than four percentage is failure. The average capturing times of a moving object equal 743/287 = 2.59.
| Table 2: | The evaluation results of moving objects capturing tasks |
It means that an average extra 1.59 times of capturing tasks is redispatched for a moving object. They are mainly caused by the error of coordinate prediction and timing of PTZ camera control. Although the number of discarded tasks is high, the results show that the prototype system based on the proposed calibration method is still feasible.
CONCLUSION
In this study, a new calibration method is proposed based on the 2D image plane for the cooperation of wide-angle and PTZ cameras. It is easy to deploy the cameras and establish the set of calibration points. It is also scene-independent. The estimation of calibration parameters is easy and the computation of desired pan/tilt angles is very fast. The method is suitable for the real-time application similar to the prototype system of moving object capturing. According to the experimental results of fixed locations, the accuracy of the proposed method is good enough for satisfying the moving object capturing in an open area. In advance, the experimental results of the object capturing application shows that the method is feasible in the practical environment. The only disadvantage of the proposed method is that the calibration points must be re-established manually when one of the cameras is moved.
In the future, the number of discarded tasks will be decreased to increase the efficiency of PTZ camera. The average redispatched times should be decreased to less than one. Besides, a mechanism for establishing of calibration points automatically will be designed to enable fully automatic operation of the system.
ACKNOWLEDGMENT
This research was partially sponsored by the National Science Council with grant No. NSC 96-2815-C-324-011-E. Special thanks to Tsung-Ming Lee and Wei-Chieh Chang for the implementation of the prototype system.