Subscribe Now Subscribe Today
Research Article

An Efficient Structural Mouse Gesture Approach for Recognizing Hindi Digits

N.F. Shilbayeh, G. Raho and M. Alkhateeb
Facebook Twitter Digg Reddit Linkedin StumbleUpon E-mail

An efficient structural approach for recognizing Hindi digits drawn by the mouse is proposed. Our Mouse Gesture Hindi Digit (MGHD) system is designed and tested successfully. Our system deals with representation of shape based on a new boundary Freeman Chain Code (FCC) with eight connectivity and then use templates to recognize the Hindi digit. Freeman chain code techniques are widely used to represent an object because they preserve information such as detecting corners, straight lines. The FCC algorithm has been used to produce vector chain code that represented a thinned binary image of the Hindi digit object. In this paper FCC has been modified to extract the boundary of the shape and specify an area of the object were it has been drawn in an image as first check, then matching the result of the recognizer with the result of matching templates as a second check for improving accuracy of the recognition. The proposed method is tested on a sample of 1350 digits written by 27 different writers selected from different ages, genders and jobs, each one wrote 10 digits 5 times. An experimental result shows high accuracy of about 89.5% on the sample test. Experiments showed that this approach is flexible and can achieve high recognition rate for the shapes of the digits represented in this study.

Related Articles in ASCI
Similar Articles in this Journal
Search in Google Scholar
View Citation
Report Citation

  How to cite this article:

N.F. Shilbayeh, G. Raho and M. Alkhateeb, 2009. An Efficient Structural Mouse Gesture Approach for Recognizing Hindi Digits. Journal of Applied Sciences, 9: 3469-3479.

DOI: 10.3923/jas.2009.3469.3479



Gesture recognition has emerged as one of the most important research areas in the field of motion-based image processing and recognition. Latest keyboard have been replaced by handwriting technology recognition in a palm and pocket PDA’s, in this study, we are focusing in developing a system to recognize Hindi digits using a mouse device as an input device for the digits.

A mouse gesture is a way of moving the computer mouse in a certain predefined way, which the software recognizes as a specific command. The directions are very important, so the system is traced to recognize the gestures movements {left, right, top, down…}, Mouse gestures are used to provide quick access to common functions of a computer application.

Mouse device can be useful to recognize Hindi digits via capturing the cursor movements that has been used in drawing the Hindi digit. When the user starts drawing the digit the system starts firing an events which they had been specified to interact with a mouse device such as {mouse move, mouse leave,…etc.} until he picked up his finger from the mouse left button then the system start manipulating an image. Online concept refers to the system that can be used to capture and detect the interaction between the user and the mouse device.

East Arabs are mostly used Hindi digits in their writing. The first column in Table 1 shows an ideal sample of Hindi digits drawn by the mouse. The second column in Table 1 shows how to pronounce in Arabic these numerals and third column shows the corresponding pronunciation in English.

Regarding to the understanding of the problem with young children and people they preferred to use mouse instead of a keyboard, designing a system that can be user friendly very simple to use and flexible enough to use according to the user knowledge and conditions well be helpful. In another way, GUI must be simple were a user must not feel uncomfortable when using it. Some technical requirements should be taken into consideration such as recognizing a Hindi digit as quickly as possible and have high degree to input recognizing (availability).

Table 1: Hindi digit drawn by a mouse

Arabic digit recognition attracted many researchers (Decong et al., 2007; Cakmakov, 2002), but Hindi digits recognition has been started recently. Researchers in this field have proposed different approaches, such as statistical, structural, Hidden Markov model (Eickeler et al., 1998; Yang and Yangsheng, 1994) and neural network approaches (Shilbayeh and Iskandarani, 2005; Subri et al., 2006). The main primitives that form digits are lines and corners. Different arrangements of these primitives form different digits. To recognize a digit, we should determine the structural relation and the connectivity between these digits. The syntactic and structural approach requires efficient extraction of these primitives (Alon et al., 2005).

In this study, we proposed an efficient structural approach for recognizing Hindi digits drawn by the mouse. The proposed approach shows an efficient way for extracting the boundary of the shape and specify the area of the recognition digit were it has been drawn in an image then use matching template to recognize the digit.


The MGHD recognition system is constructed around the modular architecture of preprocessing, feature extraction and digit recognition. Mainly the modeling of the system focuses on two concepts: testing, a model is constructed from the digits drawn by a mouse and recognizing a Hindi digit by applying the modified Freeman Chain Code algorithm then displaying the recognized Hindi digit. Figure 1 shows the main steps in developing a mouse recognition system.

This section presents a general framework for MGHD recognition system. The proposed system is divided into the following main phases:-

Drawing Hindi digit: A user draw a Hindi digit inside a special toolbox (window) using the mouse. Then the pixels representing the Hindi digit are saved on a file (Adrien et al., 2007)
Preprocessing: The objectives of this stage is to prepare and clean the image to be more concise representation for feature extraction stage doing operations such as filtering, convert to gray image and normalization
Feature extractions: Done after the preprocess stage is completed. The objective of this stage is generating information that will subsequently feed to the classifier in order to classify the image. FCC with 8-connectivity has been used to extract feature
Digit recognition: Our recognition system is a double check recognition system. In the first check, FCC is used to provide information about the object such as corners, straight lines and area of the object. In the second check, the object is matched with templates to increase the recognition rate
Display a recognized digit: Display the recognized Hindi digit

Fig. 1: The mouse recognition system architecture

Drawing Hindi digit: The interface of the MGHD recognition system is designed using C#. The user can draw the Hindi digit inside [250x250] pixels window starting from the red point and following the limitation shown in Table 2. The possible beginning of a gesture is recorded by pressing the left button. From that moment, until the release of the button, each change in location of the curser is registered and added to a vector of a point. The order of the points in vectors defines directions. The vector of registered points is called the raw data of the image.

As each vector of raw data has practically a different length. The vector must be transformed to gray level and then passed to Matlab tool to start the next stage.

An example of the result of stage 1 in the MGHD recognition system is shown in Fig. 2. Figure 2a shows how to draw seta_6 (six) starting from the red point existed almost in the middle of the window and then the drawn number converted into gray level as shown in Fig. 2b.

Preprocessing: The objective of the stage is to prepare and clean the image to a more concise representation prior extracting features in the next step. The preprocessing attempts to eliminate some variability related to the writing process and that are very significant under the point of the view of the recognition (Subri et al., 2006).

Table 2: Limitation of drawing Hindi digits

Fig. 2: (a) color image that drawn by a user (b) gray image to the same image

Preprocessing operations are as follows:

Filtering the image: Eliminate the noises and to remove the borders of the image itself due to different reasons such as bad drawing means that the user do not started drawing the Hindi digit shape exactly from the red point
Binarization: Converting the gray image into binary image. The importance of this binarization is then simply a matter of choosing threshold values [0,0]
Dilating the image: Adds pixels to the boundaries of objects in an image
Normalization: Since the size and the length of Hindi digit shape is variable. Its normalization is often used to space out digits to a uniform size
Classification: Active contour method has been used to detect the digit border. The importance of detecting the digit inside the image is tracking the boundary of the object

Active contour method is used in this system which is known as the Freeman chain code of the digit border (Zahir and Dhou, 2008). In this method chain code stores the absolute position of the first pixel and the relative positions of successive pixels along the digit border after eliminating all small regions (image border) that can be seen as a noise. Then remove all the entire pixels inside the boundary of the digit object. This method has thinning algorithm whereas the contour approach avoids these problems since no shape information is lost (Teredesai et al., 2002).

Active contour in Hindi digit: Active contour Snake is a methodology based on the use of deformable contours, which adapt its border to the diverse shapes of the objects in the images. The results are very conditional by the selection of the initial position of the contour (Shilbayeh Iskandarani, 2005). Boundary detection is one in which the contours of the digit images are detected. Any standard edge detection algorithm can be used for this purpose. But for the sake of accuracy, contour detection by point processing is undertaken.

Active contour provides a flexible tracking mechanism. A snake is developed by Kass et al. (1988) parametrically defined as:

V(s) = (X(s), Y(s))

where, X(s) and Y(s) are x, y coordinates along the contour and s represents path length with values in [0, 1] (in another words arc-length of the border) (Chun and Shiu, 1998; Pons et al., 2008).

The energy function to be minimized is a weighted sum of internal and external forces, which can be written as in Eq. 1 and 2.


Internal energy function is needed to move onto the object border as shown by Eq. 2.

Eint = (α(s)|vs(s)|2 + β(s)|vss(s)|2/

where, Vs(s) is the first derivative with respect to s and Vss(s) is the second derivative with respect to s. The first term is treated as elastic energy shown in Fig. 3, which causes the snake to shrink like an elastic rubber band. Weight α(s) allows us to control elastic energy along different parts of the contour. The second term is bending energy, which is defined as the sum of squared curvature of the contour. Bending energy makes the snake to behave like a thin metal strip and is responsible for smoothness of the contour. Weight β(s) plays a similar role as α(s).

Image (External) function, the image energy is derived from the image data over which the snake lies. This energy functional attracts snake to salient features in images such as lines and edges. For line drawing objects, image energy can be defined as in Eq. 3

Eimage = – I (x, y)

This drives the snake to move towards regions with high intensity. The image energy for objects with homogeneous regions image energy can be defined as in Eq. 4 if we want snake to be attracted to edges:

Eimage = – | Δ I (x, y)|

Fig. 3: Elastic energy

Fig. 4: Binary image (number Tessa “9”) generated by MGHD

As a matter of fact, the key point of the active contour lies in the definition of the image energy. In theory, the image energy can be formulated in any form and derived from any image properties as long as the minimum of snake energy corresponds to desired image properties.

For example, consider a digit image with a black background and a white foreground as shown in Fig. 4. The detection procedure encompasses a horizontal scanning technique. Every pixel of the Hindi digit image is scanned horizontally. If the pixel under consideration is white, then the colors of its 8 connected pixels are taken into account as shown in Fig. 5 (edge with a blue color of number tessa_”9”).

Feature extraction: Feature extraction plays one of the most important roles in any recognition system. It refers to a kind of image analysis, where it focuses on basic directions, with a wide range of applications in mind, such as medical imaging, remote sensing, character and digit recognition. Features (image information) are fed to a digit recognition stage in order to classify the possible Hindi digit.

The importance of information in the MGHD system is the shape and the size of an object within the image. For example, the shape of the object is of the major importance in the automatic digit recognition in Optimal Digit Recognition (ODR) system.

Fig. 5: Contour object of number Tessa “9” detected by MGHD

Fig. 6: (a) Freeman 8-direction chain code (b) generated code by running through the boundaries

Although, there are ODR systems already employing our familiar regional features, where is a large class of techniques that use the shape information residing in the boundary curve of the digit. The recognition of the digit must be insensitive to its position, size and orientation.

From the previous section the image in Fig. 4 is the result of the binarization phase, in which all gray levels of the digit region below a certain threshold become 0 and all above the threshold become 1 and the image in Fig. 5 (the object in blue color) shows the boundaries tracked by the contouring algorithm.

Freeman Chain Code (FCC) algorithm has been used for getting features information from the image boundaries (Mongkolnam et al., 2007; Zahir and Dhou, 2008).

Chain codes: Chain coding is one of the most widely used techniques for boundary shape description and representation; the boundary curve is approximated via a sequence of connected straight line of preselected direction and length. Every line is coded with a specific coding number depending on its direction as shown in Fig. 6a. These directions are numbered form 0 to 7. The red point is considered as the starting point in our system. Figure 6b shows the sequence of chain codes obtained by running along the boundaries in clockwise direction. The disadvantage of this description is that the resulting chain codes are usually long and at the same time very sensitive to the presence of noise.

Fig. 7: Starting point

Fig. 8: Vector generated by FCC algorithm for number Tessa_”9”

Fig. 9: Simplifying the output of a FCC for number Tessa “9”

Representing a position of point according to the eight neighbors pixels is an array of the coding number of the direction of the line that connected boundary pixels (xi, yi), (xi+1, yi+1)) were (xi, yi) represent the current pixel and next pixel is (xi+1, yi+1), sweeping the boundary in clockwise direction. The importance of this feature extraction stage is that the accuracy of recognition depends on the information passed from this preprocessing stage to the classifier (recognizer). The extracted features are represented as a sequence of 3 tuples (xi, yi, pi) where (xi, yi) are the coordinates of the pixel and pi is the binary number that represents the line drawn in the image according to the angle theta (θ).

Freeman chain code has a several limitation used to represent the boundaries such as connected sequence of straight lines of specified length and direction. Also, it depends on the starting point that is usually represented by a sequence of direction numbers (Subri et al., 2006). If the chain code is used for matching it must be independent of the choice of the first border pixel in the sequence. One possibility for normalizing the chain code is to find the pixel in the border sequence which results in the minimum integer number if the description chain is interpreted as a base four number. That pixel is then used as the starting pixel. A mod 4 or mod 8 differences are called a chain code derivative.

An example of using FCC with 8-connectivity in our MGHD system is shown in Fig. 8 as a result of drawing number tessa_”9” as shown in Fig. 7. Then, our MGHD system well simplifies the directions of the list. Simplifying consists of finding consecutive movements in the same direction and joining them in one direction.

Fig. 10: Modified freeman chain code algorithm

Figure 9 shows the simplified FCC for number tessa_"9". Figure 10 shows the Modified Freeman Chain code Algorithm used in our MGHD.

Distinguishing ambiguous digits: There are multiple Hindi digits that have the same number of corners and straight lines, these Hindi numbers

Ethnan_”2” and seta_”6”
Saba_”7” and thamanya_”8”

Using FCC algorithm reduces the detection rate of our system. To solve this ambiguity we add more constraints on these digits by drawing the digits in a special area.

Fig. 11: Image subdivided into areas

Fig. 12: Determining area in MGHD

The drawing window is divided into four areas {Area1, Area2, Area3 and Area4} as shown in Fig. 11.

So, our MGHD system can distinguish these ambiguous Hindi digits by checking the area of the drawn digit as shown in Fig. 12 and then apply the digit templates in these areas.

Digit recognition stage: Recognizer Stage is the last and important stage for recognizing the Hindi digit. MGHD gives a weight for each Hindi digit. However, each Hindi digit has a different weight at a different size of a number shape. Hindi digit weight is the number of corners plus the number of straight lines. Assume that C represents the number of corners and SL represents the number of straight lines. Then Hindi digit weight is given by Eq. 5.

Weight = C + SL

Drawn area that has been described in previous section helps the system for recognizing a Hindi digit and increases the detection rate for the ambiguous digits. MGHD use the Hindi templates shown in Fig. 13 to identify the Hindi digits by matching these templates with the drawn Hindi digit.

Fig. 13: Templates

A set of templates (patterns) has been used in the MGHD to make the best possible match and a sort of reducing the rejected rates. These templates are certain objects forming Hindi digits in a certain place in our MGHD system.

The flowchart is shown in Fig. 14 shows the steps followed in our MGHD system to recognize each Hindi digit according to the previous discussion. Once our MGHD system can determine the results from FCC, the templates can easily recognize Hindi digit and display the digit at the screen.


Drawing a Hindi digit is the most important technique. A user should start drawing a Hindi digit from the red circle. Drawing the digit needs to stick with the drawing limitation is shown in Table 2 to get the highest accuracy.

The MGHD recogni tion system well capture the drawn image [250x250] size then pass the gray image to Matlab tool as a matrix. Then preprocessing stage will start running, by doing the preprocessing operations filtering, binarization, dilation, normalization and contouring. Filtration done by removing the border of the image and the small objects if exists. After that, MGHD recognition will convert the gray image into binary image. then Dilation step will start executing after the binarization were done; dilating the image is necessary for fixing the Hindi digit border by adding a “1” value to the near pixel that do not contain “1”. After dilation were done, normalization is important for determining the size of the object and contouring for tracking the border. An example of the resulted binary matrix for number thalatha_”3” is shown in Fig. 15.

Fig. 14: Recognizer flowchart

Fig. 15:Binary matrix for number thalatha_”3”

Feature extraction fire after the preprocessing stage is done. Feature extraction will execute the matrix that is received form preprocessing stage by applying the FCC algorithm for tracking the border of the object. FCC algorithm used to generate a VCC, corners, straight lines and the area as shown in Fig. 16a of the object. Figure 16b shows the VCC generated by MGHD recognition system.

The first step of the construction of the chain code is to extract the boundaries of the image. Chains can represent the boundaries or contours of any Hindi digit. Chain code represents closed boundaries. Extracting the contour depends on the connectivity.

Fig. 16: VCC generated by MGHD recognition system

Table 3: Number of corners and lines for Hindi digits

In this study we use the FCC with 8-connectivity algorithm and a [3×3] area of pixels. The MGHD recognition system generates a real output of the VCC and then simplifying the VCC.

Applying the modified chain code is very useful for detecting number of corners, straight lines and specifies the drawing area. Table 3 shows the total number of corners, straight lines and processing time needed to recognize each Hindi digits.


Mouse gesture hindi digit has been designed for classifying the Hindi digit. The system works in virtual mouse position and action are interestingly controlled by mouse to save some expenses. Technically MGHD has been designed and developed on different platform by using first Microsoft visual c# 2005 software as a side of generating a powerful GUI and Matlab software as a side for handling and manipulating the image.

The trains set were filled by 27 writers selected from different ages, genders and jobs. Each one of them had drawn the numbers 5 times.

Fig. 17: Recognition accuracy chart

Table 4: Recognition accuracy testing results

So, for each number the program attempts to recognize it 135 times. As results, there is only one possible outcome. The testing accuracy resulted from the MGHD recognition system for all Hindi digits are summarized in Table 4. From The recognition accuracy chart shown in Fig. 17, the ability of the system to recognize the drawn Hindi digits correctly is about 89.46%.

The MGHD proposed recognition system is mouse-based recognition system. Mouse is the most difficult hand written tool and used by many people especially the children and the handicapped. The proposed system use a simple structural approach in comparison with other complicated approaches used by many authors in the literature such as SVM, HMM and MLP Neural network. The MGHD system not only gives more advantage in performance when recognizing a digit but also recognition rate above 89.5% on average. The training in MGHD is not an essential part in comparisons with other approaches such as SVM, HMM and MLP neural network. In addition to that, there is no rejection case in the testing results as other systems.

The correct recognition percentages reported in Javad et al. (2003), Bhattacharya et al. (2002) and Taani and Hammad (2008) are respectively 94.14, 93.025 and 95%. The reject recognition percentages reported are respectively (5.86% misclassification rate), 6.975% (1.97% rejection rate and 5.005% misclassification) and (5% incorrect results). It can be noticed that the accuracy of the proposed approach is lower than the accuracy of reported approaches due to the following reasons:

MGHD has been tested using the mouse gesture which is the most difficult handwritten tool
The writing process is subjective and depends on the person writing style. So, The result could be improved if our testing people have been trained and carefully selected and if we exclude the small children from the testing data
Some of the reported results have been tested on different numerals written by different languages and used by different people. As example, Taani and Hammad (2008) proposed a structural approach for recognizing Arabic numeral. Arabic digits has been studied and improved by many authors in comparison with Hindi digits
Other approaches is more complicated and needs to be trained in comparison with our approach

Despite these factors our approach has a good recognition rate,increase number of work done in Hindi digits, use the most complicated handwritten tool and can be extended to work in characters and mathematical symbol.


An efficient on-line structural pattern recognition system has been designed and tested. The MGHD recognition system deals with representation of shape based on a new boundary FCC with 8-connectivity and then use the drawing areas and templates to improve the accuracy of the recognition. The FCC algorithm has been used to produce vector chain code that represented a thinned binary image of the Hindi digit object. In this paper FCC has been modified to return the vector of chain code with 8-connectivity concatenating with area after determining the object inside an image with a size [30x30]. Then the recognizer uses the weight for each Hindi digit and the size of number shape to identify the Hindi digit. Templates are used to fulfill this task, to improve the accuracy results and to avoid the rejected rate. If MGHD system has been rejected or failed to recognize the Hindi digit for any reason, MGHD can guess the drawn Hindi digit by area and templates.

An experimental result shows high accuracy of about 89.5% on the sample test. Experiments showed that this approach is flexible and can achieve high recognition rate for the shapes of the digits represented in this study.

1:  Adrien, C., S. Schimke, J. Vanderdonckt and C. Vielhauer, 2007. Trainable sketch recognizer for graphical user interface design. IFIP, 11: 124-135.
CrossRef  |  

2:  Alon, J., V. Athitsos and S. Sclaroff, 2005. Accurate and efficient gesture spotting via pruning and subgesture reasoning. Proceedings of the ICCV HCI Workshop, October 2005, Boston, pp: 1-10.

3:  Ahmad, A.T. and H. Maen, 2008. Recognition of on-line handwritten arabic digits using structural features and transition network. Informatica, 32: 275-281.
Direct Link  |  

4:  Bhattacharya, U., T. Das and B. Chaudhuri, 2002. Recognition of Handprinted Bangla Numerals Using Neural Network Models. In: Advances in Soft Computing-AFSS 2002, Pal, N.R. and M. Sugeno (Eds.). Springer-Verlag, Berlin, Heidelberg, ISBN: 978-3-540-43150-3, pp: 139-161.

5:  Cakmakov, D., V. Radevski, B. Younes and G. Dejan, 2002. Decision fusion and reliability control in handwritten digit recognition system. J. Comp. Inform. Technol., 4: 283-293.
Direct Link  |  

6:  Chun, L. and Y. Shiu, 1998. An active contour algorithm for object tracking. Pattern Recognition Lett., 19: 491-498.
Direct Link  |  

7:  Decong, Y., M. Lihong and L. Hanqing, 2007. Lottery digit recognition based on multi-features. Proceedings of the Systems and Information Engineering Design Symposium, Apr. 27, IEEE, Charlottesville, VA., pp: 1-4.

8:  Eickeler, S., A. Kosmala and G. Rigoll, 1998. Hidden markov model based continuous online gesture recognition. Proceedings of the Pattern Recognition 1998, Fourteenth International Conference, Aug. 16-20, Springer, pp: 1206-1208.

9:  Pons, S.V., J. Rodriguez and O. Perez, 2008. Active contour algorithm for texture segmentation using a texture feature set. Proceedings of the 19th International Conference on Pattern Recognition, Dec. 8-11, Tampa, FL., pp: 1-4.

10:  Javad, S., Y. Ching and D. Tien, 2003. Application of support vector machines for recognition of handwritten arabic persian digits. Proceedings of the 2nd Conference on Machine Vision and Image Processing and Applications, February 2003, K.N. Toosi Univ. of Tech., Tehran, IranTehran, pp: 300-307.

11:  Kass, M., A. Witkin and D. Terzopoulos, 1988. Snake active contour models. Int. J. Comp. Vision, 4: 321-331.

12:  Mongkolnam, P., C. Nukoolkit and T. Dechsakulthorn, 2007. Represent image contents using curves and chain code. Proceedings of the MVA2007 IAPR Conference on Machine Vision Applications, May 16-18, Tokyo, Japan, pp: 355-358.

13:  Shilbayeh, N.F. and M.Z. Iskandarani, 2005. An intelligent multilingual mouse gesture recognition system. J. Comput. Sci., 3: 346-350.

14:  Subri, S., H. Haron and R. Sallehuddin, 2006. Neural network corner detection of vertex chain code. AIML J., 1: 37-43.
Direct Link  |  

15:  Teredesai, A., E. Ratzlaff, J. Subrahmonia and V. Govindaraju, 2002. On-line digit recognition using off-line features.

16:  Yang, J. and X. Yangsheng, 1994. Hidden Markov Model for Gesture Recognition. CMU-RI-TR, The Robotics Institute Carnegie Mellon University, Pittsburgh, Pennsylvania, pp: 30.

17:  Zahir, S. and K. Dhou, 2008. A new chain coding based method for binary image compression and reconstruction.

©  2021 Science Alert. All Rights Reserved