Subscribe Now Subscribe Today
Abstract
Fulltext PDF
References
Research Article
 

Optical Hyperacuity Mechanism by Incorporating Human Eye Microsaccades



Haitian Zhai and Hui Li
 
ABSTRACT

Background: Super resolution is a technique that reconstruct a sequence of low resolution images into a high resolution image with more details. Methodology: In this study a novel method enhancing the image resolution that relates the conventional super resolution with the biological visual hyperacuity is presented. Results: The proposed algorithm is constructed by strictly comply with the principle of biological hyperacuity. The proposed algorithm contains image acquisition part and image reconstruction part. For the image acquisition part, key frames are selected according to the blurring degree and sub-pixel motion parameters. For the reconstruction part, majorization-minimation algorithm is utilized to solve the optimization problem. Conclusion: Finally, an experiment is conducted to evaluate the performance of the proposed mechanism. The results show that robust recovery of high resolution image can be obtained by this system.

Services
Related Articles in ASCI
Similar Articles in this Journal
Search in Google Scholar
View Citation
Report Citation

 
  How to cite this article:

Haitian Zhai and Hui Li, 2016. Optical Hyperacuity Mechanism by Incorporating Human Eye Microsaccades. Journal of Software Engineering, 10: 416-423.

DOI: 10.3923/jse.2016.416.423

URL: https://scialert.net/abstract/?doi=jse.2016.416.423
 
Received: March 15, 2016; Accepted: May 05, 2016; Published: September 15, 2016

INTRODUCTION

Up to now, a large number of Super Resolution (SR) methods1-3 have been developed successfully and widely used in many application fields. However, there is very little research on SR from the perspective of biology. In the field of biology, visual hyperacuity is a similar phenomenon to the technic of SR that the acuity of perceiving stimulus detail is far better than the resolution of photoreceptors in retina4. Although visual hyperacuity and SR both share the same objective, which is to increase the resolution of the real world, there is not much cross application between them, since SR is not originated from biology but from signal processing and is then applied to process the image.

In this study a mechanism of hyperacuity imaging system by incorporating the principle behind the human eye is presented. By analyzing the properties of the involuntary eye-movement, which is a key element in human hyperacuity phenomenon, a system that imitates the working mechanism of the human eye is built. There are mainly 2 steps in our system including raw data acquisition and High Resolution (HR) reconstruction. For the image acquisition part, key frames are selected according to blurring degree and sub-pixel motion parameters. For the latter procedure, a new reconstruction algorithm which incorporates the Majorization Minimization (MM) algorithm is presented. Up till now, there is no similar research that relates SR with biological hyperacuity.

MATERIALS AND METHODS

Visual hyperacuity: The visual acuity of human eyes is limited by the distance between two sense cells on the fovea to about 1 arcmin of visual angle, which means anything smaller than that5 should not be distinguished. However, human eyes are capable of resolving a displacement with less than 10 arcsec that is at least one third of the diameter of a foveal cone. This phenomenon is called hyperacuity and has given rise to a large number of psychophysical studies and several qualitative theories about perception as well as the underlying neuronal properties. The reason that the spatial resolution of the human eye is better than the traditional visual acuity is the different measure methods. Traditional visual acuity is used to describe the ability of the human eye to distinguish two object as separate. Due to the diffraction effect, the optical system of human eye is not able to reproduce the object sufficiently compact. Even for the smallest object, its light will spread on the fovea and occupy more than a dozen sense cells. According to Rayleigh criterion if two objects need to be seperated, these two objects should be apart from each other for at least half the width of the joint light distribution. Fig. 1a illustrates the difference between visual acuity and hyperacuity.

Hyperacuity, however, defines the visual capabilities that the detectable difference in separation has values lower than the resolution limit. No laws of physics are disobeyed in the hyperacuity phenomenon since human eye is not only a spatial system but also a spatiotemporal system.

Fig. 1(a-b):
(a) Illustration for visual acuity and hyperacuity. The relative location of two objects can be detected when the shift is much smaller than the one required to resolve them as separate and (b) Involuntary eye-movements: Tremor (curved lines) and microsaccades (straight lines)

Fig. 2:Hyperacuity mechanism

Researchers maintain that hyperacuity would require eye movements to achieve the high spatial resolution. Eye movements are critical in stitching the visual perception of the world around us to a seemingly large continues scene. Microsaccades is the most important fixational eye movements, which are the unconscious motions of the eye made when fixation is needed after large eye movements. Figure 1b shows the concept of microsaccades. Their peak velocities as well as durations are related to microsaccades amplitudes parametrically. However, there is no correlations between changes in microsaccades amplitude and visual acuity.

Mechanism design: The principle of biological hyperacuity is strictly complied to construct the system. A motor building under the signal acquisition device is used to imitate the effect of microsaccades of the human eye. Figure 2 shows is the scheme of the proposed mechanism. The motor is able to provide horizontal and vertical motions for the sensor. Since the obtained motion is unpredictable, it can be considered as random, which is similar to the involuntary microsaccades of the human eye. The temporal resolution of the human eye lies between 15 and 35 ms and is bad at low and high speed, depending on the experimental conditions. In other words, appropriate velocity of movement is necessary for temporal offsets to be seen as spatial offsets. There is no mechanism for human eyes to control and adjust the speed of the involuntary microsaccades to a specific value. However, it is known that our eyes are continuously reaching the maximum speed several times per second to guarantee enough spatial offsets have been perceived. Figure 2 shows, the velocity of the motor continuously oscillates along x-axis and reach its peak value in a short time.

Human eyes have the ability to skip the visual scene perceived when the velocity exceeds the limiting interval. Thus, our system should have a similar mechanism to constrain the frame obtained by the sensor. The frame that meets the standards of appropriate spatial offsets is referred as the key frame. Here the blur degree of the image is used to determine if one image is obtained when the sensor is moving too fast. First, two blurring operators are defined as follows:

(1)

Then convolutions between the image and operators is calculated by:

(2)

where, f(x, y) represents the image function, fx(x, y) represents the convolution between f(x, y) and Kx, fy(x, y) represents the convolution between f(x, y) and Ky. The blur degree of the image S can be obtaind by:

(3)

where, |·|2 is the l2 norm of a matrix. For all frames satisfying s>σ should be removed, where σ is a predefine blurring threshold. Note that image noise is not used as a factor determining if the frame should be considered as a key frame, since the noise models for all frames are assumed to be same.

The motion parameter among all frames should also satisfy a certain condition. Here feature based motion estimation method is utilized to estimate the motion parameters. Let P = {Pi = (xi, yi), i = 0, …., M-1} where, Pi is the central point of ith image. Since there is no contribution to the super resolution reconstruction procedure if a frame contains integral pixel movement, P should be processed that all the integer parts are removed and keep only the fractional parts results in . Let D = {di,j, i = 0, …, M-1; j = 0, …, M-1; i≠j} represents the distance between any two of the points in P’, where di,j is calculated by Euler distance:

(4)

where, . 0≤di,j≤0.5 reflects the relative distance between two images in sub-pixel level and larger di,j means shorter distance. The di,j = 0.5 means two images are completely overlapped in sub-pixel level. The index of key frame can be obtained by:

(5)

Super resolution reconstruction: Traditionally, regularization has been described from both the algebraic and statistical perspectives. Using regularization techniques, the desired HR image can be solved by minimizing the function:

(6)

where, y is the vectorized version of the LR image and x is the desired HR image and λ is a constant controlling the strength of Γ(x) and Γ(x) denoting the Total Variation (TV) term6 is a variable introduced to regularize the SR problem:

(7)

Then the reconstructed image can be obtained by minimizing L(x):

(8)

In most situations, it is impossible to estimate the high-resolution image by direct minimizing Eq. 8, because the high-frequency errors are enhanced in this estimate. Majorization Minimization (MM) method7 is one of the most popular strategies to solve the nonlinear problem in Eq. 8. The MM algorithm is of the following form:

(9)

where, x(m) is the HR image in mth iteration. For any given x and x(m), G(x|x(m))≥L(x) and only when x = x(m), G(x|x(m)) equals L(x), which means G(x|x(m)) is the upper boundary function of L(x) and touch the upper bound only when x = x(m). With the above assumption, L(x) and G(x|x(m)) satisfy the following relation:

(10)

To apply MM algorithm to our problem, consider the following in Eq. 11:

(11)

where, a≥0, b>0. Let and:

and substitute a, b into Eq. 6:

(12)

The above in Eq. 12 is equivalent to:

(13)

Equation 6-8 can be write as:

(14)

In addition, since is irrelevant with x, can be obtained by:

Fig. 3:Flowchart of MM reconstruction procedure

(15)

Let , is defined as follows:

(16)

So can be expressed as:

(17)

Equation 17 can be written in matrix form as follows:

(18)

Where:

and. can be obtained by replacing Γ(x) with :

(19)

where, C3 is a constant unrelated to x. Since, is a quadratic function, the minimization problem can be converted to the following linear system:

(20)

The flowchart of the reconstruction procedure is shown in Fig. 3.

RESULTS AND DISCUSSION

In the experiment, two datasets, each contains 12 LR images are used to compare the proposed algorithm with Fast and robust SR8, TV regularization SR9 and variational Bayesian SR10.

In order to present the performance of proposed system more comprehensively, original LR images interpolated by nearest neighbor and bicubic are shown in Fig. 4a and b, respectively. Reconstruct results by using fast and robust SR, TV regularization SR and variational Bayesian SR, proposed method are shown in Fig. 4c-f, respectively. As can be seen from the figure that fast and robust is able to increase the image resolution with more details, however, it cannot avoid ringing artifacts at boundaries with discontinuity of the blur map. TV regularization SR and variational Bayesian SR do not have the ring artifacts, however, compare with the proposed method shown in Fig. 4f, the proposed method successfully restores more details with minimized artifacts. Results of the second dataset are shown in Fig. 5 and the input image is magnified two times by the proposed SR algorithm and other methods as shown in Fig. 5c-f.

All experiments have been run on an Intel Core i7 2600 3.40 GHz processor. Proposed algorithm is able to upscale an image sequence of 12 frames sized 60×80 to 120×160 in less than 30 sec on average. So the computational complexity of the solution is not high.

Proposed method is able to obtain a more detailed HR image than other algorithms is because,1: The mechanism is able to remove the non-key frames by calculating blur degree and relative sub-pixel parameter, 2: The MM algorithm is used to optimize the TV SR model in the reconstruction procedure. Compared with the proposed algorithm, both of the interpolation algorithms can only increase the amount of pixels without introducing more real information to the final HR image. Although fast and robust SR, TV regularization SR and variational Bayesian SR is able to increase the image resolution with more details, they do not have a mechanism of removing the non-key frame and also, the reconstruction procedure is not ideal compared with the proposed method. Through the comparison of the methods, proposed method provides obvious good subjective visual quality with rich textures and sharp edges and the increase in resolution and image quality is evident. In practice, the resolution increasing is also determined by optical distortion, sub-pixel registration accuracy and degree of redundancy.

Fig. 4(a-f):
An example of HR reconstruction results with different SR algorithms, (a) Original LR image interpolated by NN, (b) Original LR image interpolated by bicubic, (c) Fast and robust SR, (d) TV regularization SR, (e) Variational Bayesian SR and (f) Proposed SR

Fig. 5(a-f):
An example of HR reconstruction results with different SR algorithms, (a) Original LR image interpolated by NN (PSNR = 22.74, SSIM = 0.62), (b) Original LR image interpolated by bicubic (PSNR = 23.33, SSIM = 0.67), (c) Fast and robust SR (PSNR = 24.51, SSIM = 0.79), (d) TV regularization SR (PSNR = 25.70, SSIM = 0.84), (e) Variational Bayesian SR (PSNR = 25.82, SSIM = 0.83) and (f) Proposed SR (PSNR = 28.89, SSIM = 0.92)

CONCLUSION

Visual hyperacuity is a similar phenomenon to the technic of SR. Although visual hyperacuity and SR both share the same objective, there is not much cross application between them. In this study a novel method enhancing the image resolution that relates the conventional super resolution with the biological visual hyperacuity is presented. By analyzing the properties of the involuntary eye-movement, which is a key element in human visual hyperacuity, an optical hyperacuity mechanism is proposed to construct a more detailed HR image from a sequence of LR images. The proposed mechanism is able to increase the image resolution obtained by a sensor built on a vibrating system beyond the Nyquist limit. Future work will include using other degradation model to describe the imaging process that can be used in SR reconstruction problem.

ACKNOWLEDGMENTS

This study was supported by National Natural Science Foundation of China (Grant No. 61171155 and 61571364).

REFERENCES
Chen, W.L., L. Guo and W.L. Xia, 2013. A novel super-resolution reconstruction algorithm based on subspace projection. J. Comput., 8: 1893-1897.
CrossRef  |  Direct Link  |  

Dimigen, O., M. Valsecchi, W. Sommer and R. Kliegl, 2009. Human microsaccade-related visual brain responses. J. Neurosci., 29: 12321-12331.
CrossRef  |  Direct Link  |  

Farsiu, S., D. Robinson, M. Elad and P. Milanfar, 2003. Fast and robust super-resolution. Proceedings of the IEEE International Conference on Image Processing, Volume 2, September 14-17, 2003, Barcelona, Spain, pp: 291-294.

Mairal, J., 2015. Incremental majorization-minimization optimization with application to large-scale machine learning. SIAM J. Optimiz., 25: 829-855.
CrossRef  |  Direct Link  |  

Maiseli, B.J., O.A. Elisha and H. Gao, 2015. A multi-frame super-resolution method based on the variable-exponent nonlinear diffusion regularizer. EURASIP J. Image Video Process. 10.1186/s13640-015-0077-2

Martinez-Conde, S., S.L. Macknik, X.G. Troncoso and D.H. Hubel, 2009. Microsaccades: A neurophysiological analysis. Trends Neurosci., 32: 463-475.
CrossRef  |  Direct Link  |  

Ono, S. and I. Yamada, 2013. Optimized JPEG image decompression with super-resolution interpolation using multi-order total variation. Proceedings of the IEEE International Conference on Image Processing, September 15-18, 2013, Melbourne, VIC., pp: 474-478.

Shao, W.Z. and Z.H. Wei, 2013. Variational bayesian super-resolution based on composite prior modeling. Proceedings of the IEEE International Conference on Signal Processing, Communications and Computing, August 5-8, 2013, KunMing, China, pp: 1-5.

Tang, L., 2015. Video super-resolution reconstruction algorithm based on total variation regularization. Chem. Eng. Trans., 46: 169-174.
CrossRef  |  Direct Link  |  

Van Ouwerkerk, J.D., 2006. Image super-resolution survey. Image Vision Comput., 24: 1039-1052.
CrossRef  |  Direct Link  |  

©  2019 Science Alert. All Rights Reserved
Fulltext PDF References Abstract