Abstract: For the industrial complex processes featured with multi-extremums of output quality characteristics and nonlinear relationship between parameters and characteristics, how to optimize the parameters via small sample experiments and global modeling is critical to product quality improvement. This study proposes an approach for randomly-oriented sequential design and regression modeling of complex processes. Firstly, an initial sample is generated by using uniform design and a primary model is set up by using Support Vector Regression (SVR). Secondly, new points are added according to the generation population of Genetic Algorithm (GA) and the SVR model is rebuilt correspondingly. Thirdly, the second stage is iterated sequentially until the mean square error of successive SVR models is reached to a certain lower limit. Finally, the global optimum of characteristics and therefore, the parameters are optimized by applying GA on the latest SVR models. The theoretical analysis and the simulation study show that, compare with the one stage design and modeling approach, the proposed approach requires a smaller sample size and can get a smaller prediction error by adding new points sequentially and compare with the path-oriented sequential design and modeling approach, the proposed approach can effectively avoid the limitation of reaching local optimums by adding new points randomly.
INTRODUCTION
In modern industry, there exist a large number of complex processes (Egea et al., 2010) featured with many input parameters, multi-extremums of output quality characteristics and nonlinear relationship between parameters and characteristics. It is very hard to find the best operating parameters due to the lack of true physical or engineering models of the processes. Therefore, in order to optimize the parameters, regression modeling of complex processes based on Design of Experiments (DOE) must be done at first.
Traditional regression modeling methods include parametric ones such as first or second order polynomials (Myers, 1999), nonparametric ones such as Kriging regression and kernel regression and machine learning ones such as Artificial Neural Networks (ANN) and Support Vector Regression (SVR). The polynomial method assumes the processes can be fitted by first-order or second-order polynomials and is widely used in the regression of simple processes featured with nearly linear or quadratic relationship between parameters and characteristics. While fitting complex processes, especially those with more than one extremum of characteristics, low order polynomials often lead to poor fitting and prediction performance. Although Kriging and ANN methods are widely used in complex processes modeling for their flexible model forms (Chakraborty, 2012; Turan et al., 2011), they usually need a large sample size to set up regression models. When the experimental sample is small, these methods always lead to poor prediction performance due to overfitting. In most cases, however, only a relatively small sample size of experiments can be got because of the restriction of time and cost. As a result, it is improper to use these methods in the optimization of parameters for complex processes. SVR (Shawe-Taylor and Sun, 2011) is the latest machine learning for regression which is based on the principle of structural risk minimization. It can get better fitting and prediction performance with small-sample experiments. This property enables SVR to be widely used in many fields such as function fitting and process modeling (Tirelli et al., 2012; Banerjee and Das, 2012).
Samples for regression modeling of complex processes are usually obtained through DOE (Montgomery, 2001) which can be classified into one stage design and sequential design according to the number of design stages. Commonly used one stage DOE methods include Uniform Design (UD) (Fang and Ma, 2002), space-grid design, orthogonal design and Latin Hypercube Sampling (LHS). UD which is based on the uniform dispersion principle, can arrange a small number of trials over a large number of parameters with multi-levels and is widely used as the main DOE manner in modeling complex processes. Sequential design is a multistage DOE strategy. In this method, new points are added sequentially to an initial sample which is generated by a one stage DOE manner. For the complex processes, sequential design can gradually improve the regression model through the process information gained from successive sampling, so it can effectively reduce the number of trials compare with one stage DOE strategy.
Sequential design has been studied by a number of researchers. Schwaab et al. (2006) produced a new sequential discrimination procedure which makes use of model probabilities and concentrates the efforts on models with higher probabilities. Dos Santos and Dos Santos (2008) proposed a simulation scenarios oriented sequential design strategy. Keys and Rees (2004) proposed a space-grid design, in which the existing points are constructed after modeling and then the best sub-bit point is selected as the new center point of sequential points. Wang et al. (2011) proposed a sequential RBF modeling method, in which new points are generated by the weighted average of the existing test point and the nearest neighbor point. Jiang et al. (2007) proposed a sequential design method in which the new points are added based on the criterion of maximum entropy. New test points are produced based on existing points and adjusted covariance function.
In most studies, the sequential criteria of sequential designs are deterministic path-oriented. In these designs, new points are added according to certain deterministic rules around existing points. For the path-oriented manners, however, due to lack of randomness, subsequent points will always follow the path given by the deterministic rules which is mostly impacted by initial points. So the initial sample has a great influence on the optimizing results. For example, if the initial sample is unsuitable, the optimizing result will be trapped in a local optimum inevitably. How to avoid the limitations of the path-oriented sequential design is still an important issue in sequential design studies. Actually, in the field of function optimization, there exists an effective search manner named randomly-oriented search which can reach the global optimum of the function by jumping out of local optimums according to a certain probability. Because of path independence, the randomly-oriented search pattern can be a reasonable alternative of path-oriented search pattern in sequential design. Typical algorithms used in the randomly-oriented search manner are Genetic Algorithm (GA) Holland (1975) and Simulated Annealing Algorithm (SAA) but these algorithms can only be applied in optimizing functions with explicit forms. For the complex processes with implicit relationship, how to adapt randomly-oriented search in the sequential design and regression modeling and thereby optimizing the process parameters globally remains uninvestigated.
This study proposes a three stage approach based on GA and SVR for randomly-oriented sequential design and modeling of complex processes.
THEORY OF SUPPORT VECTOR REGRESSION
Traditional statistical theory is inadequate to solve problems with limited sample size. New Statistic Learning Theory (SLT) proposed by Vapnik (1998) can overcome this deficiency. It specially researches machine learning principles in the case of small sample. Vpanik and Chervonenkis proposed Structural Risk Minimization (SRM) guidelines which define the compromise between the approximation and complexity of function. SVR is one of the implementation tools of SRM guidelines. The theory of SVR can be described as follows.
Assume that there are m factors of a process. For the input variable x = [x1, x2,…,xm]T, there is a certain dependency F(x, y) between x and the corresponding output variable y. The essence of regression is to find a proper function expression f(x) to estimate the corresponding y. For linear regression f(x) = w•x+b, solve:
| (1) |
where, ε is a non-sensitive parameter and C is a penalty coefficient. ε-insensitive loss function is expressed as:
| (2) |
Put the solution (α, α*) = {α1, α1*, α2, α2*,þ,αl, αl*} into:
to get the final regression function f(x) and f(x) is described as:
| (3) |
Nonlinear regression is the expansion of the linear regression. Its essence is using the nonlinear mapping φ to map data x to high dimensional space and then constructing a linear regression in the space. The kernel function is used to avoid complex dot product operation in high-dimensional space. After the kernel function k(xi, xj) = φ(xi)φ(xj), the insensitive coefficient ε and the penalty coefficient C are chosen, solve the following equation:
| (4) |
The final solution f(x) of the nonlinear problem is shown as:
| (5) |
MATERIALS AND METHODS
Basic ideas: In the path-oriented sequential design, new points are added around existing points according to certain rules which always lead to the deficiency of trapping into local optimum, GA can be chosen as the randomly-oriented design manner to overcome the disadvantage. However, one of the premises of using GA is that there must be an explicit model to be the target function. So, it is unsuitable to apply GA directly in optimizing complex processes with implicit relationship. Here, a three stage sequential design approach is proposed to carry out randomly-oriented search. In stage I, an initial sample is generated by using uniform design and a primary model is set up by using SVR. In stage II, GA is applied to find the interim global optimum of the current stage SVR model and meanwhile, a new generation population in GA is produced. Then the sample is renewed by adding new points according to the generation population and the SVR model is rebuilt correspondingly. Stage II is iterated sequentially until the mean square error of successive SVR models reaches to a certain goal. In stage III, GA is still applied to optimize the SVR model to find the final global optimum of characteristics and therefore, the parameters are optimized correspondingly.
Algorithm steps of randomly-oriented sequential modeling design based on GA and SVR: According to the theoretical analysis above, the steps of each stage of the approach are given as follows:
Stage I: Initial modeling stage:
| Step 1: | Use uniform design to generate initial points to reform the initial sample for regression as: |
| (6) |
| Step 2: | Set test sample which only includes x dimension for the sequential termination criterion: |
| (7) |
| Step 3: | Choose Gaussian function: |
| as the kernel function of SVR, calculate C, ε, σ of Eq. 4 according to the method given in literature Cui et al. (2008) |
| Step 4: | Use SVR according to S1, C, ε and k(x, x') to obtain the initial regression model SVR1 denoted by f1(x) |
| Step 5: | Substitute xpre into f1(x) and then get: |
| (8) |
Stage II: Sequential design stage:
| Step 6: | For i≥1, let Si be the initial population. Use GA to optimize the regression model SVRi. After iterating several times until GA is terminated, the points of the latest population are collected and incorporated into Si to form the i+1th sample Si+1 for regression |
| Step 7: | Use SVR and Si+1 to obtain the i+1th regression model SVRi+1 denoted by fi+1(x) |
| Step 8: | Substitute xpre into fi+1(x) and then get: |
| (9) |
| Step 9: | Calculate mean square error MSEi+1 of |
| (10) |
Stage III: Final optimization stage:
| Step 10: | Iterate the steps in stage II until one of the following termination conditions is reached: | |
| • | The iteration time i reach to a predefined upper limit | |
| • | The mean square error MSEi of successive stage reaches to a predefined lower limit | |
| Step 11: | Use GA for the optimization of the last regression model after sequential design stage is finished and then gets the finial optimizing settings of parameters | |
SIMULATION STUDY
Process description: In this section, a complex function is used to represent an actual complex industrial process. Assume there are two parameters (or factors) with the range of x = [x1, x2]T∈[-5, 5]x[-2, 8].
The relationship between y and the x can be expressed by the following function:
| (11) |
The surface and the contour graphics of the process are shown as section (a) of Fig. 1.
Reference to literature Fang et al. (2009), in the process, there are three local extremums with the coordinate (x1, x2, y) of (0, 5, 2.000), (-3, 0, 1.0124) and (3, 0, 1.0013), among which (0, 5, 2.0000) is the global optimum. In the following sections, the proposed approach and one stage design modeling approach is applied to optimize the parameters of the process.
| Fig. 1(a-e): | True and modeling graphics of the relationship between inputs and outputs in complex process, (a) True graphics of the relationship between inputs and outputs in complex process, (b) Modeling graphics the relationship between inputs and outputs in complex process, Sample size 23, (c) Sample size 31, (d) Sample size 32 and (e) Sample size 37 |
RESULTS AND DISCUSSION
Results by using the proposed approach: Considering the scale of the parameters, a 15 runs uniform design is used to generate the initial sample points. To further illustrate, the proposed approach is repeated for four times. All of the results are shown in Table 1. In the table, x1, x2 and y denote the value of each factor and the output value, respectively; Dis represents the distance between obtained optimum and actual optimum and MSEpre which is calculated from a 41x41 uniform interval test sample set, represents the mean square error between the regression model and the true values of the process output y.
As can be seen from Table 1, the sample size of each repeat is less than 40. The best optimal value obtained in four sequential designs is 1.9995 which is 99.99% of the actual optimum. That is to say, the proposed approach can achieve good modeling and optimization performance via small sample experiments.
Results by using one stage design and modeling approach: As a comparison, one stage design manner of UD is used to generate 4 groups of samples. Each group consists of ten samples generated by UD with the same size as that in sequential design. The mean of the ten optimization results of each group is calculated and recorded as shown in Table 2.
Comparison and analysis of the results: According to Table 1 and 2, conclusions can be made that:
| • | With the same sample size, the optimal value of the proposed Sequential Design (SD) approach is 5.43% higher than that of UD on average, the distance of SD is 21.29% of that of UD and the prediction error of SD is 67.58% of that of UD |
| • | When the prediction errors are close, the optimal value of SD is 2.74% higher than that of UD on average, the distance of SD is 27.06% of that of UD and the number of trials of SD is 86% of that of UD |
The comparison between the results (i.e., optima value, distance and prediction error) of SD and UD is plotted in Fig. 2.
| Table 1: | Results of sequential design modeling optimization |
| Table 2: | Results of UD modeling optimization |
| Fig. 2(a-c): | Comparison charts of optima value, distance and prediction error between the results got from SD and UD, (a) Comparison of optima value between the results got from SD and UD, (b) Comparison of distance between the results got from SD and UD and (c) Comparison of prediction between the results got from SD and UD |
As can be seen from all the comparison results of different sections, the optimal value of SD is higher than that of UD, the distance and prediction error of SD are lower than those of UD.
Surface and contour graphic of each model obtained by the proposed approach are shown as section (b) to section (e) of Fig. 1. These sections show that, in the case of small sample size, the surface graphics of models obtained by the proposed approach are very similar to that of the actual function. The three extremums are shown on each of the contour graphics as well. The distribution of the sample points can be observed by counting the points distributed in the neighborhood of the actual global optimum in each graphic. The Results indicate that the sample points of SD are obviously clustered around the optimum compare with those of UD.
CONCLUSION
In this study, a randomly-oriented sequential design approach based on GA and SVR is proposed for regression modeling and optimization of complex processes. When using the approach, uniform design makes the initial sample globally distributed within the range of parameters, GA enables the points sequentially be added into the sample and SVR improves the prediction performance of regression modeling. The simulation studies show that the model response surface obtained by the approach is similar to the real surface of the process and the obtained optimum is very close to the global optimum as well. Compared with one stage design, when the sample size is same, the proposed approach can reach better optimal value and get smaller prediction error. All of these demonstrate the advantage and effectiveness of the proposed approach.
ACKNOWLEDGMENT
This study was supported by the National Natural Science Foundation of China under Grant 71171180 and Grant 71272225.