Abstract: Background and Objective: Natural walk and topological analysis of human being have respective and certainly unique key features that allow identifications when other biometric techniques are not visible. The objective of this paper is to draw attention towards a simple and novel feature extractor for gait recognition that is based on a deep learning approach. Materials and Methods: Different from conventional ways and means, the gait is designated as regular and intermittent motion taken out directly from silhouettes. Before the use of convolutional neural network to learn human gait representations, two important data pre-processing stages are enforced to enhance the characteristics of gait patterns obtained from grayscale images. Results: The proposed gait recognition approach achieves impressive results in terms of training/validation accuracy and mean square errors. Conclusion: The conducted experimental outcomes report competitive performance as compared to many traditional machine learning methods and previous deep gait models specifically for the case of low-image resolutions and large-scale dataset of input images.
INTRODUCTION
This article describes a part of a project to provide a practical smart system to recognize every student on the Arab Open University (AOU) campus by observing and recording all the time. This system, when completed, will give facilities for direct access to student information for more efficient and productive processes such as door control systems, security systems, classroom participation, exams attendances and site detections. Since most of men on campus wear headscarf and women wear veils, the normal face recognition techniques are not practical solutions for matching data captured by cameras. For more accurate matching model and, in addition, for the purpose of verification and recognition of any strange human being on campus, the gait recognition model introduced in Center for Biometrics and Security Research1, is integrated with deep learning architecture, which is considered as the state-of-the-art in the last 5 years.
Gait based human identification, which was first introduced by Anonymous2, as a biometric feature aims to recognize human beings by analyzing their manner of walk using sequences of images. It allows the identification at a long distance using normal cameras even with videos of low resolutions and in any conditions. Gait recognition is applicable to unfavorable far observation in shopping malls, banks, airports, military departments, homeland security and other control environments as valuable methods for forensic identification and crime prevention. In some European countries, gait analysis has already contributed to the collection of evidence for convicting criminals. There are many human gait based identification techniques such as image and video processing and floor sensors and sensor attached to the body3. Further, there exist two main types of gait features: Model-based features that involve static and dynamic of users’ body parameters and model-free features that use the dark shape and outline of user, see for instance4,5. On the other hand, gait appears to be unbalanced and it is subject to change in mode when the identified human is relaxed, in hurry or even carrying objects in his/her hands. Furthermore, differentiated external factors, in particular, clothing, footwear and walking surface might affect the behavior of gait recognition. Furthermore, it is sensitive to the quality of gait sequence as well as the use of small dataset. It should be stated that the motion-analysis is closely related to several domains of computer science namely artificial intelligence, computer vision, image processing and pattern recognition. For related works, interested readers are referred to Bhandare et al.6, Boyd and Little7, Das and Saharia8, Dupuis et al.9, Glorot and Bengio10, Hannink et al.11, Jain and Aggarwal12, Javed et al.13, Kale et al.14 and Kim15.
Convolutional Neural Networks (CNNs) are superior types of multi-layer neural networks (NN) that made up of neurons with learnable weights and biases16-22. In fact, CNNs are at the heart of deep learning's current leap forward in computer vision as CNNs provide an optimal architecture for pattern detection and image recognition. Mutual with advances in graphic processing units (GPUs) and parallel computing23, CNNs represent the underlying of key technologies of new developments in facial recognition and automated driving. The architecture of CNNs involve convolutional layers followed by fully connected layers as in the NNs. The CNNs are designed to recognize 2D visual patterns straight from pixel images with negligible preprocessing and reduced number of parameters in the inputs. Moreover, they can be easily trained and have less parameters (with the same number of hidden units) comparing with fully connected NNs. In this way all neurons, which in a feature share the same weights, detect the same feature at different positions. Moreover, CNNs, like almost all other NNs, are trained with a version of the back-propagation algorithm.
The problem statement can be properly summarized as follows: Capturing all relevant data by fixed cameras, filtering and transforming the data to a useful information and building an intelligent decision making process after separating the background image. Unnecessary data has to be filtered and removed and the data dimension need to be reduced to select features relevant to the application province. Therefore, feature that can be used for pattern recognition are extracted from individuals segmented walking. There are several dimensional reduction methods including linear and nonlinear techniques. For example, one of the most common methods for linear reduction is principle component analysis, while CNN algorithms are most suitable for nonlinear applications24.
Thus, the main objective of current work is to take the advantages of deep learning algorithms that are well established in handwritten, document analysis, human pose estimation, voice and facial recognitions for comparison and identification of features of a realistic gait data.
CNNs can extract and recognize gait features, moreover, comparing with different gait recognition approaches, can achieve better performance and accuracy1,25.
MATERIALS AND METHODS
The gait biometric as a pattern recognition system is proposed and validated by applying a deep machine-learning algorithm in order to recognize individuals based on dynamics and shapes as a training set using a captured gallery sequence. For the proposed method, image-processing techniques are applied to analyze the gait images in an efficient and accurate manner. Therefore, the histogram stretching on is first applied on the images to improve their appearance quality26. Then, the image flipping is introduced to insert more images to the provided samples. Finally, the CNN model is adopt to classify gait with better than 98% accuracy.
Study duration: The study duration since pre-study preparations and data acquisition until end was about 2 years.
Biometric gait models: Gait is defined to be a complete walking cycle that is obtained from a sequence of images. A gait cycle represents the time duration of heel-strike between the identical legs27. A gait recognition system involves three steps: User tracking and detection, gait feature extraction and training testing and classification. There are two approaches to analyze gait28,29. The first approach is to model gait as the human body structure or motion using knowledge of the body component and shape. Thus, the gait features are extracted using joint positions rather than dynamics from movements. This approach has the capacity to regulate gait feature free of the inspiration of model limitations in particular clothing. However, this model-based approach needs high computational complexity and high quality of taken gait sequences. The second approach is to model gait as the whole motion pattern of human body without considering the underlying structure. Hence, the features are extracted using static gait characteristics such as centroid, height and width of the outline of a moving object. This outline is referred as a silhouette30. In this model-free gait recognition approach31, features are extracted from the pixel level in silhouettes obtained from image sequences. This approach has less computational complexity and comparatively easy to follow and apply. Furthermore, it is less sensitive to the quality of silhouettes.
It is worth mentioning that feature extraction is essential in gait recognition systems. In addition, the reduction of the dimensionality of features is a pivotal in saving invaluable running time and making classification more efficient. Accordingly, gait sequences are captured from arbitrary walking directions and selected angles. Then, silhouettes are obtained by using background subtraction and shadow removal for each gait sequence. Subsequently, the binary image frames that contain Boolean pixel values are computed using rational operation block. The proposed deep learning system is, therefore, set up to decrease the storage and computational costs by reducing the dimensions of the 2D outlines of training and testing data using convolution operations. As a supplementary phase of the deep learning process, fully connected stages are employed for the purpose of classifications and identifications.
Histogram stretching and image flipping: Due to the short colour band covered in some images, the histogram stretching32,33, is manipulated to widen the covered band and increase its standard deviation. The histogram of a grayscale image provides information about the distribution of the intensity values of the image over the pixels in the image. This information about the intensity distribution is important in determining the characteristics of the image. For instance, an image whose histogram values are concentrated or skewed towards the lower values will tend to be ‘dark’ in nature compared to an image whose histogram values are skewed towards the higher values. On the other hand, an image whose histogram is concentrated in the centre would tend to be a low-contrast image. Hence, histogram information can be used to perform image-processing tasks to achieve image enhancement and image segmentation. For image enhancement, histogram equalization is commonly used to achieve contrast enhancement. For image segmentation, the thresholding operations using histogram information are commonly used. In addition, the image flipping process has been developed to increase the size of the dataset. The process is designed to be significantly straightforward by reflecting the image horizontally.
Consequently, each continuous image (silhouette) data is simply represented as a reduced 2D convolutional feature. What follows is to approve more or less deep-learning algorithms intended for classification and therefore, gait recognition.
Conventional neural network algorithm: By convolution, it mean that an elementary operation in the convolutional neural network. Convolutional layers are diverse to completely associated layers; they utilize a couple traps to lessen the quantity of parameters that should be educated while holding high expressiveness. These are nearby connectivity: Neurons are associated just to a subset of neurons in the past layer, sharing weights are common between a subset of neurons in the convolutional layer (these neurons frame are known as a feature map) and pooling is static subsampling of sources of information. A reduced 2D feature map is obtained to represent the characteristics of each gait higher-level image after image processing and to carry out some measurements. Here, it should be emphasized that the process of feature extraction is important for improving the effectiveness of the classification process. Then, CNN classifier technique is applied to understand the correspondence between these features. Now CNNs as deep learning deliver the best solutions to several problems in natural language processing and complex pattern recognitions. The architecture of the network simply consists of convolution and subsampling layers followed by a fully connected output layers, which feed into the softmax classifier. The back-propagation algorithm will be used to compute the gradient with respect to the model parameters. It is worth mentioning here that one of the most important stage in the algorithm is to set up the training parameters (options) for the network and to inspect the influences of these parameters.
Data source: An overview of the type of data used in motion analysis and the proposed gait recognition system that is bases on the model-free approach are given. In order to acquire 2D feature of a moving object, a silhouette is detected after performing the background subtraction of the captured image obtained by a fixed camera during the object detection and tracking step. Some assumptions should be taken into consideration such as the walking route has to be straight lines and only one moving object appears in the color video cameras. The background subtraction33,34, is applied to identify a moving object against a static background by estimating pixel properties of this static background. In fact, there are different background subtraction techniques35,36, such as frame difference, real time background subtraction and shadow detection and adaptive background mixture model for real time tracking.
Here, silhouette analysis based recognition system is asserted to extract the moving object. For each frame, the true colour image (RGB) is converted into grayscale intensity image. Then, the super-bounding rectangular frame is located and the background is conjectured only for pixels inside this frame. Thus, a threshold scheme is practiced to contract binary form (BW) of such image as shown in Fig. 1. So that pixel values within the frames are used to get the nxn binary matrix A in Eq. 1, where 1 denotes the foreground and 0 denotes the background:
| Fig. 1: | Feature extractions from silhouettes |
| (1) |
A specific fixed size for the binary images in the input layer of the CNN architecture is defined. For computational reasons, the size of each image is 32×32 pixels.
Feature learning: A feature reduction algorithm is necessary to extract useful and informative features for classification. In fact, the dimensionality of features extracted from gait sequences usually plays an important role in conventional classification algorithms and deserves more attention from the literature. The result of a convolution between a weight matrix called the kernel and a small region of same size in the image called the receptive field represents the first hidden neuron in a convolutional layer. The kernel matrix acts as a filter in images extracting particular information from the original image matrices. With respect to the convolution operation, it is a simple element wise multiplication of two matrices. It is important here to note that there are various kernels or feature detectors with different effects on the images. In order to calculate the second hidden neuron in the same convolutional layer, the kernel shifts by a unit or more (one stride or more) on the input image from left to right and applies the similar convolution operation. Different stride lengths might be sometimes experimented. As Fig. 2 showed an example of a 3×3 kernel matrix that works on a 6×6 image (across the top two positions) with a stride of one and produces a 4×4 convolutional feature matrix. While Fig. 3 gave an example of using the same convolutional kernel to extract features from a 4×4 image with a stride of one into a 2×2 dimension space.
| Fig. 2: | A simple illustration of two-dimensional convolutional operations |
| Fig. 3: | Applying the receptive field calculation on 4×4 example |
| Fig. 4: | An illustration of the proposed deep convolutional network for walking gait recognition |
The number of output features, say, in each dimension can be intended using Eq. 2. Here, is the number of input feature, is the convolutional kernel size and is convolutional stride size:
| (2) |
Convolutional neural network architecture: A trainable multistage CNN might have tens or hundreds of hidden layers such that each layer learns to identify different features of the input image called feature maps. The feature learning simply consists of multiple stages each of one convolutional layer and one polling layer. The convolution layer puts the input images through a set of convolutional operations, each of which activates certain features from the input images. It is occasionally common to insert a pooling layer in between two successive convolutional layers. The pooling layer simplifies the output by performing nonlinear down sampling, reducing the number of needed parameters (and computations as well) for training the network. Moreover, pooling layers control the network over-fitting. It is worth noting here that stride might be used instead of max pooling in order to reduce layer size in network architecture. Then, the connected convolutional layers are trailed by the 2×2 max-pooling (pooling images) layers that after converting to one-dimensional vectors. We call this conversion the flattening stage. Thus, after learning features in many layers and flattening, the architecture of the CNN shifts to single or multiple fully connected layers, which, by the end of the process, computes the class scores of the gait images. These layers, which combine all the features learned by the previous layers across the images to identify the larger patterns, are similar to hidden layers in regular NNs. More specifically, after the network is trained, the last hidden layer outputs are used as gait characterizations to construct the gait classification. The deep convolutional network architecture for the gait identification system is shown in Fig. 4.
RESULTS AND DISCUSSION
In order to extract walking characteristics of a person for classification and forthcoming identification, a complete gait cycle is analyzed and a sequence of images are produced. Subsequently dataset is created in form of binary image frames. This dataset is stored for oversight situation wherever there is no previous information about the object. The design cycle of the proposed overall deep gait recognition approach, as shown in Fig. 5, was composed of a dataset collection and feature selection, as well as a deep learning algorithm and a powerful evaluation model.
Now, to test the performance of the CNN algorithm and assess different gait features, dissimilar datasets from different gender as well as ages are prepared. Furthermore, indoor silhouettes from the identifiable video clips and the outdoor silhouettes from the three different sets of open gait CASIA database are used37. Video clips of individuals are captured from different viewing angles and each clip is divided into 25 frame per second. All image frames (more than 16,821 images) for the participants are divided into 2 disjoint sets; the first set for training, the second set for validation and testing. In the proposed experiment, a subset of 80% of the data is presented to the network during the training and the network is adjusted according to its error. A subset of 20% of the data is used for the network validation and for providing an independent measurement and testing of the CNN performance. It is considered that the data for the same participant does not exist in both of the training and validation and testing sets.
Convolutional neural network analyses the images in a feature-based approach by reducing the dimensionality of super-bounding pixel frames. Employing, the mentioned above, CNN models took about 35 sec per epoch (one epoch equals one forward pass and one backward pass of all the training examples). However, in order to enhance the performance, more epochs must be processed and this will take more time to overcome this slow training by using GPU computing. The GPU computing is the utilization of a representation-handling unit together with the central processing unit (CPU) to squeeze machine-learning, test and design the applications. The process ends up when the network reaches minimum errors which are calculated by the mean square errors (MSE) as shown in Eq. 3, between the network output values and the desired output values .
| (3) |
where, the network structure consists of 16 convolutional/ pooling layers with 3×3 small convolution filter and 3 fully connected layers, where the softmax is the classifier. Beginning with 32 channels, the quantity of channels with each convolutional layer is twofold. The thickly associated hidden layers both have 800 units by applying a model been figured. There are two major challenges: The model implementation is time-consuming and over-fitting. An instinct behind beginning with a higher learning rate and diminishing it over the span of preparing is the following: As training begins, results are far from the ideal and major steps ought to be taken towards the ideal and learning the system rapidly. Yet, the nearer the outcomes get to the ideal, the lighter the need for step is required. The next step is to apply a popular regularization technique called dropout. It drives an artificial neural strategy to take in different autonomous portrayals of similar information by, on the other hand, arbitrarily handicapping neurons in the learning stage.
To evaluate the proposed method, the analysis in MATLAB R2016a on a 64-bit Windows PC with Intel®2.8 GHz x-64 based processor and 16 GB RAM is conducted. As Fig. 6 demonstrated the smoothed curves of training and validation accuracy percentage. It was noticed from the smoothed curves in Fig. 7 that the training loss (= MSE) and validation loss continue and enhance until the end. Note also that the training loss vibrates at the beginning of iterations and converges at the later stage. Most of the computer time and memory are used in the feature learning stages and that most of the parameters are in the last classifications stages. In this context, it hypothesized that deep learning modelling and classifiers play an essential role in this such development.
| Fig. 5: | Overall flow chart of the proposed deep gait recognition system |
| Fig. 6: | CNN training and validation accuracy |
| Fig. 7: | Testing smoothed loss curves |
The major challenge in the existing methods of gait recognitions is to find reliable representations of gait patterns and high efficient gait feature extractions. In addition, most of these methods focuses on the achievable classification accuracy using different machine learning algorithms. Computing efficiency in these methods are always significant issues. Although substantial progress on gait features extraction has been achieved, existing traditional methods can only detect human gait with classification accuracies less than 91%. Now, to correlate this work with the previous literature, it should be emphasized that a different silhouette analysis based strategy has been used representing one significant contribution of the research. Based on the experimental results, the proposed CNN would lead to the best results with an average accuracy 98.7%. In fact, this achieved accuracy is the research ultimate goal. An improved CNN performance can be anticipated once the network is larger. Therefore, by adding an optional fully connected layer the average recognition rate might override 99%. Moreover, the proposed method is efficient in terms of computer time. The Table 1 showed the comparison between average accuracy of the proposed algorithm and some state-of-the-art gait recognition approaches1,17,20,25,32,35,38.
| Table 1: | Accuracy of gait recognition techniques with full training sets |
Overall, however, the current gait recognition methods still need more enhancement in terms of pre-processing operations before they can be employed for the purpose of biometric identification.
In this project, human gait classification was accomplished by using deep learning architectures designed for images. The CNN was applied to discover the deep information of multi-layer network and to improve the recognition performance in the gait identification process. Based on this notion, a series of comprehensive studies and calculations on silhouettes have been proposed. In fact, this research topic exposes the gait analysis and recognition problem to the modern computer vision applications and it would stimulate the involvement of more researchers in gait researches in the future. Although the current method has many advantages, some limitations may affect gait recognition performance, such as camera viewing angles.
CONCLUSION AND FUTURE RECOMMENDATIONS
It can be concluded that gait analysis is not only to increase the level of security and safety in the community level but also to classify age and gender.
In the recent future, gait will be deployed for human being recognition in conjunction with other biometrics and in many other applications. The proposed method compared to a broader variation view and in conjunction with a new appropriate database will be confirmed.
SIGNIFICANCE STATEMENT
This study discover unpretentious human motion analysis algorithm that is beneficial for the research in games industry and computer graphics. Furthermore, the study might help the researcher to uncover and explore critical areas of gender classification and security monitoring using human motion that many researchers were not able to explore. Thus, a new theory on the human-body measurements and calculations may be arrived at.