The ATNN was proposed by Lin et al. (1992),
Inspired of TDNN architecture, which appeared as an adequate proposition remedying
the randomized initialization problem of the fixed delays. The resulting network
is trained to distinguish the temporal properties and spatiotemporal correlations
of various input patterns. The ATNN was used successfully in many works concerning
the automatic targets recognition (Lin, 2004), aircraft
target recognition (Huaitie et al., 1997) as
well as the phonetic classification (Ourdighi et al.,
In theory, this approach is based on estimation of universal approximation function and derived gradient descent algorithm, protecting the external temporal aspect, which was founded in its basic conception. However, this algorithm as any optimization discriminate method remains limited in the large-sized problems, where the probability to converge on a global minimum at reasonable time depends essentially on the good knowledge of the problem. In these cases, in lack of good knowledge, it is advised to turn to methods of stochastic optimizations.
By referring on the works (Spalanzani, 1999; Seiffert,
2001; Tlemsani et al., 2005; Whitley,
1988; Kamp and Savenije, 2006), a Genetic Algorithms
(GA) can be used as a learning device on Multi-Layers Perceptron (MLP). Motivated
by what GA seems to be like a good alternative training solution, which allow
cover important space research for convex problems in several local minimums,
we going to eliminate the original learning algorithm and replace it by an evolutionary
In this study, the network training will be in charge of genetic algorithms in multiple populations. Each chromosome represents network structure proposition and their evolution into several population will provide the final optimal structure network according to optimal criteria. We tested a model in the chaotic series Mackey-Glass equation and compared the results with other methods. Also, we extended the experiment to phonetic classification based on TIMIT database.
||An example of three layered ATNN
|| Representation of bloc delay between two units
ATNN approach: With similar architecture of TDNN, an adaptive time delay neural network employs modifiable time delays along the interconnections between two processing units. For more illustration, Fig. 1 showed an example of three layered ATNN.
Each node j of layer h (e.g., j ε Nh), receives nji inputs from each i ε ij, h-1 (Fig. 2), where, Ij, h-1 is the subset of nodes on layer h-1 (Ij, h-1 ⊆ Nh-1) that connects to unit j on layer h.
Time frame ji is a set of time delays (τji1, ..., τjinji)
employed on the connections to node j from node i. The time frame can have an
additional index h (τji,h) to signify layer h. Node i of layer
h is connected to node j of the next layer, with the connection line having
an independent and modifiable time-delay τjik,h and weight wjik,h.
Each node sums the net inputs from the activation values of the previous neurons,
through the corresponding time-delays on each connection line. Then the output
of node j is governed by a symmetric sigmoid function f of the net input. The
adaptation of both time delays and weight parameters is derived based on the
gradient descent method to minimize the cost function E during training based
on error back-propagation (Lin, 1994). The cost function
is defined as the instantaneous mean square error measure of the desired output
dj and network output aj, L at time tn as:
where, L denotes the output layer. So, it obvious that we have at least one criteria to optimize in training, which consent to integration of evolutionary algorithm possible.
Integration of genetic algorithm on ATNN training: There are various
methods to integrate GA into the neural network. These methods get generally
the network topology (e.g., hidden layers number or neurons number) or on training
network by adapting a parameters (Branke, 1995; Castillo
et al., 2003).
The unpredictable network parameters initialization is mainly the fault which
limits the performance of gradient descent algorithm. In contrast, a GA assures
a research in the complete domain. According to the generations, this research
space is refined towards some various sub-spaces (Belew et
Genetic algorithms develop a set of candidate solutions, called a population,
where a chromosome is other one than a possible solution of the problem to be
resolved (Goldberg, 1988). Every chromosome of this population
sees attributing a function called adaptation function or fitness which allows
measuring its quality or its weight. Then, the best chromosome selected, undergo
crossings and mutation and new population of solutions is produced for the following
generation. This process continues, until sufficient fitness achieved or time
delayed for example.
In parallel genetic algorithms using multiple populations, instead of using
a single population, we build sets small sub-populations which evolve separately
(Grosso, 1985; Cantu-Paz, 1998;
Tinos and Yang, 2007). With this insulation, every subpopulation
can evolve with its own parameters, in different directions, that are towards
different solutions. Three different models exist: the global model, the diffusion
model and the migration model (Yang, 2008).
Using in this neural integration the migration model, the principle is to divide
the population in multiple subpopulations (Fig. 3). These
subpopulations evolve independently from each other for a certain number of
generations (isolation time).
|| Migration model
After the isolation time a number of individuals is distributed between the
subpopulations (migration). The number of exchanged individuals (migration rate),
the selection method of the individuals for migration and the scheme of migration
determines how much genetic diversity can occur in the subpopulations and the
exchange of information between subpopulations (Ahn et
In this case of serial-time prediction, using of each subpopulation between 40 and 80 chromosomes, which are representing ATNN network structure, the chromosome is a serial of weight and delay of connections coded in binary code. The adaptation function will be error inverse 1/E.
Experimental results: A very interesting learning problem is prediction of a deterministic, but chaotic, time-series that we want to take as an application example of ATNN with parallel genetic algorithms training. The particular time series we choose is the time-series generated by Mackey-Glass equation. The function is generated by the following differential equation:
With: a = 0.2, b = 0.1, c = 10, t = 17.
For ATNN architecture, we used one neuron for input layer representing x(t)
with 200 sequences to learning. Using receptive fields of four delays size with
one shift to memorize signal, the three neurons of hidden layer used it to accumulate
the activation of input neuron. Concerning out-put layer, with six receptive
fields, we used one neuron to reproducing x(t+1). For prediction, the initial
segment was fixed to 9 delays. This structure is used to define number of synaptic
weight and delays for coding into chromosome. Finally, for parallel genetic
algorithms, we used four subpopulations with different size between 40 and 80
genotypes. The migration frequency was fixed to 1 migration every 10 generations
with individual number estimated at 5.
For a better demonstration of each models performance, after several experiments, the graphs shown below represent the worst graphic generations of the used networks.
First, we applied integration into TDNN with same parameters for more comparison and validation. The result was satisfied, like is shown in Fig. 4, comparing to TDNN training by descent gradient algorithm shown in Fig. 4a with error rate estimated at 0.451 after 1500 iterations, parallel genetic algorithms training shown in Fig. 4b given less error rate estimated at 0.04.
Figure 5 showed with more of details that error rate convergence was perfectly assured during training. Also, we could see that training by parallel genetic algorithms converged better than classic method.
After this trial, We experimented the training to ATNN for knowing if this integration would provided more stability for learning and parallel adaptation of parameters, consequently, if it would converged at better solution.
The results was very satisfied, as shown at Fig. 6a and b,
the error attained 0.006 after only 507 iterations with the evolutionary algorithms
training when it was estimated at 0,18 after 1500 iteration which is an optimization
by 10-2 in the result. Also, we observed that TDNN which given less
good result than ATNN using descent gradient algorithm, become more better with
parallel genetic algorithms training.
||Comparison of evolution results of x(t) using (a) TDNN with
descent of gradient algorithm (b) and TDNN training with parallel GA after
|| Comparison of evolution error rate between simple TDNN et
TDNN training with GA on 1500 iterations
||Comparison of evolution results of x(t) using ATNN with descent
of gradient algorithm (at left) and ATNN training with parallel GA (at right)
after 1500 iterations
||Comparison of evolution error rate between simple ATNN et
ATNN training with parallel GA on 1500 iterations
The error rate was less stable for parallel genetic algorithm but convergence was faster than descent of gradient algorithm, like is shown in Fig. 7.
Trying to extend this experimentation, we moved towards a phonetic classification. Of the TIMIT database, we have selected three phonemes of plosives categories.
|| Results obtained by applying ATNN and ATNN with AG
||Results obtained by applying ATNN training with AG with different
However, the /b/, /d/ and /g/ are known for difficulty of separability (Weibel,
1988; Weibel et al., 1989).
Unlike the first case, different fitness choices are offered. So, we could attribute for each subpopulation a fitness for varying the evolution. We could use the error function or the total train rate or the rate of some classes.
The result showed comparison of AG integration with different forms of fitness in several experimentations. We opted for using the rate or error.
For using AG with subpopulations, the results werent so good. We noticed the speed of the convergence, but there was a decline in return on a generalization. The results obtained by simple GA integration with two different forms of fitness shown by Table 1 confirmed the efficient of the method to optimize the result where the rate augmented to 10% compare to the results of temporal gradient descent rules. In Table 2 with GA using subpopulations, the results moved back and were less encouraging in spite of the 2% of augmentation in training rate, the result shown issues.
The integration of multiple-deme parallel genetic algorithms in ATNN training finally strengthened its performance in a tangible way. In fact, we can really notice it on the results of the applications approached on our work, especially on the prediction.
We consider that the results obtained, in the classification case, are in the preliminary state and that our work amounts to a trial phase. We plan on one hand to continuous the experiment to the entire TIMIT database and on the other hand, to take into account the effect of the noise through NTIMIT database.