Towards Discriminative Training Estimators for HMM Speech Recognition System

INTRODUCTION

Automatic Speech Recognition (ASR) is a pattern recognition problem. Statistical pattern recognition techniques have been successfully applied to many problems, including speech recognition. The majority of current automatic speech recognition systems employ statistical techniques to model speech (Ben-Yishai and Burshtein, 2004). The Hidden Markov Model (HMM) method is statistically based and its success has triggered a renewed urge for a better understanding of the traditional statistical pattern recognition approach to speech recognition problems. The advent of powerful computing devices and the success of Hidden Markov Modeling released a renewed pursuit for more powerful statistical methods to further reduce the recognition Word Error Rate (WER) and build more robust speech recognition systems across various conditions. The accuracy of state of the art recognition systems relies on properly trained parameters with minimum training data (Tong, 2001). That’s why a tremendous research work is actually intended on finding new Discriminative Training (DT) algorithms yielding to an increase of speech recognition accuracies in clean and noisy environments. Approaches to improve discrimination between confusable classes can be categorised in two ways. In the first category, discrimination is improved by operating the recogniser in a feature space in which the acoustic units of interest are inherently better separated. In a second category, the problem of discrimination is addressed at the model level by building better classifiers. Discriminatively-trained hidden Markov models fall under this category (Ben-Yishai and Burshtein, 2004; Juang and Katagiri, 1992).

The present research is motivated by the following observation; robust parameter estimates from insufficient training data is a research topic itself, then we intended to give an answer to the following question: how much training data do we need to be able to construct a good classifier? Of course, increasing the size of the database nearly always results in a decrease of error rate. The reason for this is that the more training data one has, the more parameters one can afford to train and, consequently, the more detailed the models can be. Unfortunately, the use of much training data, will result many drawbacks such that the increase of processing time, or other logistic difficulties.

So, we focused in this research study, an improvement, in the model training level, of the HMM recognition system described by Frikha et al. (2007). Our major concern was basically oriented on studying the efficiency of the training estimator by varying its capacity represented by the size of data as well as the complexity of the estimated function. So, we studied in this present research:

STATISTICAL SPEECH RECOGNITION OVERVIEW

A typical statistical speech-recognition system is shown in Fig.1. The goal in a statistically-based speech recognition system is to find the most likely word sequence given the acoustic data. If O is the acoustic evidence that is provided to the system and W = w₁, …,w_N is a sequence of words, then the recognition system must choose a word string that maximizes the probability that the word string W was spoken given that the acoustic data was observed (Rabiner, 1989):

(1)

P(W|O) is known as the a posteriori probability since it represents the probability of occurrence of a sequence of words after observing the acoustic signal. The above approach to speech recognition, where the word hypothesis is chosen within a probabilistic framework, is what makes most present recognizers statistical pattern recognition systems.

Fig. 1:	Schematic overview of a statistical speech recognition system

It is difficult to directly compute the maximization of Eq. 1 since there are effectively an infinite number of word sequences for a given language from which the most likely word sequence needs to be chosen. This problem can be significantly simplified by applying a Bayesian approach to find (Rabiner, 1989):

(2)

The probability P(O|W), that the data was observed if a word sequence was spoken is typically provided by an acoustic model. The likelihood that gives the a priori chances of the word sequence being spoken is determined using a language model (Rabiner, 1989). Probabilities for word sequences are generated as a product of the acoustic and language model probabilities.

The process of combining these two probability scores and sorting through all plausible hypotheses to select the one with the maximum probability, or likelihood score, is called decoding or search.

Statistical Acoustic Modeling: Hidden Markov Modeling of Speech: In most current speech recognition systems, the acoustic modeling components of the recogniser are almost exclusively based on Hidden Markov Models (HMM).

HMM provide a statistical framework for modeling speech patterns using a Markov process that can be represented as a state machine. The temporal evolution of speech is modeled by an underlying Markov process (Rabiner, 1989). Hidden Markov modeling is a powerful statistical framework for time-varying quasi-stationary process and a popular choice for statistical modeling of speech signal.

Given a speech utterance, let O = (o₁, o₂.....o_T) be a feature vector sequence extracted from the speech waveform, where o_t denotes a short-time acoustic vector measurement. Further, consider a first-order N-state Markov chain governed by a state transition probability matrix A = [a_ij], where a_ij is the probability of making a transition from state i to state j. Assume that at t = 0 the state of the system q₀ is specified by an initial state probability π_i = P(q₀ = i). Then, for any state sequence q = (q₁, q₂.....q_T), the probability of q’s being generated by the Markov chain is:

(3)

Suppose the system, when at state q_t, puts out an observation o_t according to a distribution b_q(o_t) = P(o_t|q_t), q_t = 1,2,.....,T. If we denote B the matrix containing the probability distribution in each state, B=[b_jk], where b_jk = P(o_k|q_t =j). Therefore, we can use the compact notation λ = (π, A, B) to fully define the HMM. The HMM used as a distribution for the speech utterance can then be defined as (Rabiner, 1989):

(4)

Where, b_qt(o_t) defines the distribution for short-time observations, this output probability distribution represents the probability of observing an input feature vector in a given state q_t. At the core of the HMM is a Bayes classifier where classification is done using a simple likelihood ratio test. The output probability distribution could be parametrized in several ways. The most commonly used form of the output for continuous distribution is a multivariate Gaussian distribution.

Acoustic model estimation: The estimation of the parameters of the HMM acoustic models plays a vital role in the accuracy of the ASR system. A key to the widespread use of HMM to model speech can be attributed to the availability of efficient parameter estimation procedures (Rabiner, 1989; Baum, 1972). Maximum Likelihood Estimation (MLE) is one such optimisation criterion (Dempster et al., 1977). The motivation to use MLE comes from the probabilistic definition of the speech recognition process which attempts to find a word sequence which maximises a cost (likelihood) function: we should be able to estimate the model parameters λ_MLE = to maximise the probability of the observation sequence given the model (Saul and Rahim, 2002):

(5)

The Expectation-Maximisation (EM) algorithm provides an iterative framework for MLE (Dempster et al., 1977); given the structure of HMM and training data, the algorithm finds the parameter values of the HMM according to the maximum likelihood (ML) criterion.

In the EM algorithm, an auxiliary function is defined as:

(6)

The EM algorithm iteratively estimates a new parameter set by computing an auxiliary function based on an old parameter set (E-step) and then maximizing the auxiliary function over (M-step):

(7)

The convergence of this iterative procedure to a local maximum of the objective function is guaranteed (Baum, 1972; Dempster et al., 1977).

A speech recognizer trained by the ML criterion achieves good recognition rate only if training data are sufficient to estimate model parameters reliably and the modeling assumptions are correct. However, in reality, speech is not produced by a hidden Markov process and the training sample size is not large enough. In this situation, training by maximum likelihood estimation may not lead to the best possible model that maximizes the recognition rate (Juang et al., 1997). The maximum recognition rate may be indirectly achieved by separating different classes as much as possible.

STATISTICAL LEARNING THEORY

Supervised learning refers to learning from examples, in the form of input-output pairs (x,y), by which a system that isn’t programmed in advance can estimate an unknown function and predict its values for inputs outside the training set (Cherkassky and Malier, 1999; Tong, 2001).

The central issue of statistical supervised learning problem is presented as follows:

That’s have a set of measures D = {(x_i, y_i) εRⁿ x {0,1}, i = 1,…,m} according to unknown jointed probability distribution P (x, y). F is a set of functions such that F = {f_α(x)/αεΛ}.

Our main objective is to find f_α* ε F such that the estimation = f_α*(x) will be the best possible. The choice of that function depends therefore on m training data iid according to the joint probability distribution P(x,y) = P(x).P(y|x). In order to select the function f_α* from the given set of functions F, we need a loss function L(f_α(x),y) which corresponds to the loss caused by using f_α(x) to predict y. The traditional loss function for classification problems is (Vapnik, 1999):

(8)

Here, α is the generalized parameter of functions. Therefore, f_α* will correspond to the function which minimizes the risk functional R(α) defined by Vapnik (1999):

(9)

In order to minimise the risk functional given by Eq. 9, for an unknown probability measure P(x,y). The expected risk functional is replaced by the empirical risk functional (or training error) which is constructed on the basis of the samples. This principle is called the Empirical Risk Minimization (ERM). The empirical risk functional is defined as (Vapnik, 1999).

(10)

Minimization of the above functional is one of the most commonly used optimization procedures in machine learning. ERM is computationally simpler than attempting to minimize the actual risk as defined in Eq. 10. Note that according to the law of large numbers:

(11)

This convergence is the main motivation for the Empirical Risk Minimisation (ERM): one hope that the function minimising the empirical risk (training error) will also have a small risk; then the ERM principle is said to be consistent.

Suppose we now receive a new set of data that does not include any of the examples used previously. For a machine that generalizes well, we should be able to predict with a high degree of confidence that the empirical risk obtained using this new data (or generalization error), will also be small. However this is not sufficient to guarantee a small generalization error. This phenomenon is called overfitting (Fig. 2).

Fig. 2:	Underfitting and overfitting phenomena

To avoid it, we need to restrict the class of functions on which the empirical error is minimized in order to have some guarantee on the efficiency of the algorithm.

This restriction to a prescribed set of functions called the model can lead to underfitting, i.e., to an estimator which has high empirical and expected risks. Therefore, to choose the adequate size (also called capacity or complexity) of the model is a key issue to build consistently efficient estimators. This implies that the expected risk (generalization error) tends towards the empirical risk. With this, we can guarantee both a small empirical risk (training error) and good generalisation with which we guarantee an ideal situation for a learning machine.

DISCRIMINATIVE ADAPTIVE TRAINING

Maximum-likelihood point estimation is by far the most prevailing training method. However, due to the problems of unknown speech distributions, such as sparse training data, high spectral and temporal variability of speech signal and possible mismatch between training and testing conditions, a dynamic training strategy is needed. To cope with the changing speakers and speaking conditions in real operational conditions for high-performance speech recognition, such paradigms incorporate a small amount of speaker and environment specific adaptation data into the training process. Bayesian adaptive learning is an optimal way to combine prior knowledge in an existing collection of general models with a new set of condition-specific adaptation data.

Maximum likelihood linear regression: Bayesian adaptive learning is an optimal way to combine prior knowledge in an existing collection of general models with a new set of condition-specific adaptation data. The most often used structure is through an affine transformation such as finding linear regression transformation of the mean vectors of the original HMM.

Maximum likelihood linear regression or MLLR computes a set of transformations that will reduce the mismatch between an initial model set and the adaptation data. More specifically MLLR is a model adaptation technique that estimates a set of linear transformations for the mean and variance parameters of a Gaussian mixture HMM system. The effect of these transformations is to shift the component means and alter the variances in the initial system so that each state in the HMM system is more likely to generate the adaptation data.

The transformation matrix used to get a new estimate of the adapted mean is given by Leggetter and Woodland (1995):

(12)

(13)

Hence, W can be decomposed into:

(14)

The transformation matrix W is obtained by solving a maximisation problem using the Expectation- Maximisation (EM) technique (Bilmes, 1998). This technique is also used to compute the variance transformation matrix. The use of EM algorithm results a maximisation of a standard auxiliary function.

MLLR and regression classes: MLLR makes use of a regression class tree to group the Gaussians in the model set, so that the set of transformations to be estimated can be chosen according to the amount and type of adaptation data that is available (Leggetter and Woodland, 1995; Lee and Huo, 2000). The tying of each transformation across a number of mixture components makes it possible to adapt distributions for which there were no observations at all. With this process all models can be adapted and the adaptation process is dynamically refined when more adaptation data becomes available.