
Research Article


On the Adequacy of Variable Selection Techniques on Model Building


M.P. Iwundu
and
O.P. Efezino



ABSTRACT

The adequacy of variable selection techniques on model building is examined using the drying characteristics of fresh grains. Three selection techniques (forward selection, backward elimination and stepwise methods) are employed each of which identifies a series of models assumed to be adequate. From the resulting models for each technique, the most suitable model is determined using standard assessment criteria namely, R^{2}, R^{2}_{adj}, PRESS, AIC and C_{p}statistic. In addition to the standard assessment criteria, the Doptimality criterion is examined and presented as a criterion for measuring the goodness and adequacy of regression models. Results show that under the forward selection and stepwise regression methods, each assessment criterion locates the same model. Variation seems to exist using backward elimination technique.





Received:
June 15, 2015; Accepted: August 06, 2015;
Published: October 19, 2015 

INTRODUCTION Response Surface Methodology (RSM) as an important branch of experimental design and a critical technology in developing new processes plays a vital role in model building. According to Myers and Montgomery (2002), Response Surface Methodology (RMS) is seen as a combination of statistical and mathematical techniques useful for developing, improving and optimizing processes. Carley et al. (2003) observed the usefulness of RSM in model calibration and validation. As pointed out by Kadane and Lazar (2004), model selection plays a major role in modern scientific enterprise. The question of model selection then becomes apparent as the variables that affect the outcome of interest become a major concern.
Building statistical model of a response as a function of multiple explanatory variables is a common practice in various professions. Gerritsen et al. (1996) observed its usefulness in risk estimation. Generally, variable selection and modelbuilding techniques are used to identify the best subset of regressors to include in a regression model (Montgomery et al., 2001; Myers, 1990; Burnham and Anderson, 2001a; Hobbs and Hilborn, 2006; Ward, 2008; Lee and Ghosh, 2009; Olden and Jackson, 2000). Leaving out important regressors introduces bias into the parameter estimates, while including unimportant variables weakens the prediction or estimation capability of the model. In model building, the process of identifying and fitting an appropriate response surface model from experimental data requires some use of experimental design fundamentals. As pointed out by Burnham and Anderson (2001b), three general principles guild modelbased inferences and these principles are simplicity and parsimony, several working hypotheses and strength of evidence. Kadane and Lazar (2004) presented a review of methods and criteria for model selection with emphasis on Bayesian and frequentist approaches. Evaluating all possible regressions can be computationally burdensome hence various methods have been developed for evaluating only a small number of subset regression models by either adding or deleting regressors one at a time. These methods are generally referred to as stepwisetype procedures and can be classified into three broad categories namely (i) forward selection method, (ii) backward elimination method and (iii) stepwise regression method. Sauerbrei et al. (2007), Ward (2008), Lee and Ghosh (2009) and Raffalovich et al. (2008) employed the steptype procedures in selecting best subset of regressors. Carley and Kamneva (2004) used the simulated annealing method for searching the best subset of regressors. Hastie et al. (2009) used the Sequential Search (SS) method to find optimal subsets of variables for a specified model size. The various stepwise procedures are among popular and widespread techniques of model selection. They provide systematic ways of searching through models. With these procedures new models are obtained at each stage by adding to or deleting from the model one variable at each previous stage. Although, the stepwise procedures were originally designed to aid in variable selection problems for regression models, they have been applied to generalized linear models as seen in Lawless and Singhal (1978) as well as to contingency tables as in Agresti (1990). An advantage of using the stepwise procedures is that they are readily available in most statistical packages. Unfortunately, as reported in Weisberg (1985), none of the stepwise regressions corresponds to a specified criterion for choosing a model. Hence, the selected model need not be optimal in any other sense than that it is the result of the algorithm applied to the dataset. In fact, Graybill (1976) demonstrated that forward selection and backward elimination might not result in the same final model while working on the same dataset. Other frequentist procedures include the risk inflation criterion and the covariance inflation criterion (Donoho and Johnstone, 1994; Foster and George, 1994; Tibshirani and Knight, 1999). The aim of this study is to access the adequacy of three stepwise procedures in model building using the drying characteristics of fresh grains (melon seed). This problem shall be viewed from a regression context. As noted in Foster and Georges (1994), variable selection problem in regression is a two stage process involving the selection of a best subset of predictors and the estimation of the model coefficients by least squares. MATERIALS AND METHODS Stepwise procedures shall be employed to identify models that suitably describe the relationship between drying characteristics of fresh grains. The adequacy of the resulting models shall be tested using standard assessment criteria namely, Cpstatistics, Akaike Information Criteria (AIC), R^{2} statistics, R^{2}_{adj} statistics, MSE and PRESS. Furthermore, Doptimality criterion shall be introduced as an assessment criterion for testing for adequacy of regression models and its performance shall be compared with those of the standard assessment criteria. Stepwisetype procedures: The stepwisetype procedures are based on three different strategies, namely Forward Selection (FS), Backward Elimination (BE) and Stepwise Regression (SR). Forward selection: Forward selection strategy begins with the assumption that there are no regressors in the model other than the intercept. An effort is made to find an optimal subset by inserting regressors into the model one at a time. The first regressor selected for entry into the model is the one that has the largest simple correlation with the response variable y. Suppose that regressor is x_{1}, this is also the regressor that will produce the largest value of the Fstatistic for testing significance of regression. This regressor is entered if the Fstatistic exceeds a preselected Fvalue, say F_{in} (or Ftoenter). The second regressor chosen for entry is the one that now has the largest correlation with the response variable after adjusting for the effect of the first regressor entered on y. We refer to these correlations as partial correlations. In general, at each step the regressor having the highest partial correlation with y (or equivalently the largest partial Fstatistic given the other regressors already in the model) is added to the model if its partial Fstatistic exceeds the preselected entry level F_{in}. The procedure terminates either when the partial Fstatistic at a particular step does not exceed F_{in} or when the last candidate regressor is added to the model. The technique is evaluated using the Ftest defined by: where, RSS_{p} and RSS_{p+j} are the residuals sum of squares of the models with p and p+j variables, S^{2}_{p+j} is the variance of the model built with p+j variables and F_{in} is used as a stop criterion. Backward elimination: Backward elimination method proceeds in the opposite way. It begins with a model that includes all K candidate regressors. Then the partial Fstatistic (or tstatistic, which is equivalent) is computed for each regressor as if it were the last variable to enter the model. The smallest of these partial Fstatistic is compared with a preselected value, F_{out} (or Ftoremove) and if the smallest partial Fvalue is less than F_{out}, that regressor is removed from the model. At this instance, a regression model with K1 regressors is fitted, the partial Fstatistic for this new model calculated and the procedure repeated. The backward elimination algorithm terminates when the smallest partial Fvalue is not less than the preselected cutoff value F_{out}. The technique is evaluated using the Ftest defined by: where, F_{out} is used as stop criterion. Stepwise regression: Stepwise regression is the modification of forward selection in which at each step all regressors entered into the model previously are reassessed via their partial Fstatistic or tstatistics. A regressor added at an earlier step may now be redundant because of the relationship between it and regressors now in the equation. If the partial Fstatistic for a variable is less than F_{out}, that variable is dropped from the model. The stepwise regression requires two cutoff values, F_{in} and F_{out}. Assessment criteria
Rsquare (R^{2}): Rsquare is a measure of the proportion of variability in the data set that is accounted for by a regression model. It assumes that every independent variable in the model helps to explain variation in the dependent variable (y) and thus gives the percentage of explained variation if all independent variables in the model affect the dependent variable (y). The R^{2} statistic is defined as:
Where:
SS_{T} 
= 
Σ_{i }(y_{i})^{2} is the total sum of squares 
SS_{R} 
= 
Σ_{i }()^{2} is the regression sum of squares 
y_{i} and 
= 
Original and modeled data values 
Adjusted Rsquare (R^{2}_{adj}): In least squares regression, increasing the number of regressors in the model leads to increase in R^{2}. Hence, R^{2} alone cannot be employed as a meaningful comparison of models. The adjusted Rsquare, R^{2}_{adj}, tells us the percentage of variation explained by only those independent variables that truly affect the dependent variable (y) and penalizes for adding independent variable (s) that do not belong to the model. The adjusted Rsquare is defined as: where, n is the sample size and p is the model parameter. Akaike information criterion: Akaike Information Criterion (AIC) measures the relative quality of a statistical model for a given set of data. Consequently, AIC provides a means for model selection as it estimates the quality of each model, relative to the other models. The Akaike Information Criterion (AIC) is given as:
where, n denotes the sample size, p denotes the number of parameters and
Operationally, one computes AIC for each of the identified models and selects as best, the model with the smallest AIC value. For small sample sizes (i.e., n/p<~40), the secondorder Akaike Information Criterion (AIC_{c}) is used and is defined by: The secondorder Akaike Information Criterion (AIC_{c}) is simply AIC with a correction for finite sample sizes. As sample size increases, the last term of the AIC_{c} approaches zero and the AIC_{c} tends to yield the same conclusions as the AIC (Burnham and Anderson, 2002).
Predicted residual sum of squares: The Predicted Residual Sum of Squares (PRESS) can be used as a data validation procedure. It provides insight into the quality and potential influence of individual observations on the estimates. The PRESS statistic, derived from the jackknifed residuals e_{i} = is defined as:
where, is the estimated dependent variable when the regression model is fitted to a sample of n1 observations with the ith observation omitted. Mallow’s C_{p} statistic: Mallow’s C_{p} statistic is defined as a criterion for assessing fits when models with different numbers of parameters are being compared. It is given by:
where, RSS (p) is the error sum of squares for the pterm model and = MS_{E } (for the full model). The C_{p} statistic will tend to be close to or smaller then p if the pparameter model is adequate. Doptimality criterion: The use of Doptimality criterion as model adequacy technique has not been well established in the literature. The Doptimality criterion is a determinant criterion and provides low variance estimates for parameters and low correlation estimates among parameters. A design is Doptimal if it maximizes the determinant of the information matrix and equivalently minimizes the variancecovariance matrix. By definition, a design, say ξ*, is Doptimal if for all designs ξ^{i}, i = 1, 2, …
det M(ξ*) = max {det M(ξ^{i})}; i = 1, 2, … 
(8) 
where, det(.) is the determinant and M(.) is the information matrix.
RESULTS The experimental data set used in assessing the adequacy of variable selection techniques in model building comprises of drying characteristics of fresh grains tabulated in Appendix A. The key variables are time, temperature and moisture content. The data was analyzed using SPSS 21 and series of models resulted using each of the stepwisetype techniques. The analyses for forward selection techniques are presented in Table 15.
Table 1:  SPSS 21 outputs for forward selection   ^{a}Predictors: (constant), TEMP. MC, ^{b}Predictors:(constant), TEMP. MC, MC, ^{c}Predictors:(constant), TEMP. MC, MC, MC. MC, ^{d}Predictors: (constant), TEMP. MC, MC, MC. MC, TEMP, ^{e}Predictors:(constant), TEMP. MC, MC, MC. MC, TEMP, TIME. TEMP, ^{f}Predictors: (constant), TEMP. MC, MC, MC. MC, TEMP, TIME. TEMP, TEMP. TEMP, df: Degree of freedom 
Table 2:  Variables entered/removed^{a}   ^{a}Dependent variable: DRYING.RATE 
Table 3:  ANOVA^{a}   ^{a}Dependent variable: DRYING.RATE, ^{b}Predictors: (constant), TEMP.MC, ^{c}Predictors: (constant), TEMP.MC, Mc, ^{d}Predictors: (constant), TEMP.MC, MC, MC.MC, ^{e}Predictors: (constant), TEMP.MC, MC, MC.MC, TEMP, ^{f}Predictors: (constant), TEMP.MC, MC, MC.MC, TEMP, TIME.TEMP, ^{g}Predictors: (constant), TEMP.MC, MC, MC.MC, TEMP, TIME.TEMP, TEMP.TEMP, df: Degree of freedom 

The analysis for backward elimination techniques are presented in Table 69. 

The analysis for stepwise regression techniques are presented in Table 1014. 

From the statistical analysis, the resulting models using the forward selection method are 
• 
y = a_{o}+a_{23}x_{2}x_{3+}e 
• 
y = a_{o}+a_{3}x_{3 }+a_{23}x_{2}x_{3+}e 
Table 4:  Coefficients^{a}   ^{a}Dependent variable: DRYING.RATE 
• 
y = a_{o}+a_{3}x_{3}+a_{23}x_{2}x_{3}+a_{33}x_{3}^{2}_{+}e 
• 
y = a_{o}+a_{2}x_{2}+a_{3}x_{3}+a_{23}x_{2}x_{3}+a_{33}x_{3}^{2}_{+}e 
• 
y = a_{o}+a_{2}x_{2}+a_{3}x_{3}+a_{12}x_{1}x_{2}+a_{23}x_{2}x_{3}+a_{33}x_{3}^{2}_{+}e 
• 
y = a_{o}+a_{2}x_{2}+a_{3}x_{3}+a_{12}x_{1}x_{2}+a_{23}x_{2}x_{3}+a_{2}x_{2}^{2}+a_{3}x_{3}^{2}_{+}e 
The resulting models using the backward elimination method are
• 
y = a_{0}+a_{1}x_{1}+a_{2}x_{2}+a_{3}x_{3}+a_{12}x_{1}x_{2}+a_{13}x_{1}x_{3}+a_{23}x_{2}x_{3}+a_{11}x_{1}^{2}+a_{22}x_{2}^{2}+a_{33}x_{3}^{2}_{+}e 
• 
y = a_{0}+a_{1}x_{1}+a_{2}x_{2}+a_{3}x_{3}+a_{12}x_{1}x_{2}+a_{23}x_{2}x_{3}+a_{11}x_{1}^{2}+a_{22}x_{2}^{2}+a_{33}x_{3}^{2}_{+}e 
• 
y = a_{0}+a_{1}x_{1}+a_{2}x_{2}+a_{3}x_{3}+a_{12}x_{1}x_{2}+a_{23}x_{2}x_{3}+a_{11}x_{1}^{2}+a_{33}x_{3}^{2}_{+}e 
• 
y = a_{0}+a_{1}x_{1}+a_{2}x_{2}+a_{3}x_{3}+a_{23}x_{2}x_{3}+a_{11}x_{1}^{2}+a_{33}x_{3}^{2}_{+}e 
The resulting models using the stepwise regression method are
• 
y = a_{o}+a_{23}x_{2}x_{3+}e 
• 
y = a_{o}+a_{3}x_{3}+a_{23}x_{2}x_{3+}e 
• 
y = a_{o}+a_{3}x_{3}+a_{23}x_{2}x_{3}+a_{33}x_{3}^{2}_{+}e 
• 
y = a_{o}+a_{2}x_{2}+a_{3}x_{3}+a_{23}x_{2}x_{3}+a_{33}x_{3}^{2}_{+}e 
Table 5:  Excluded variables^{a}   ^{a}Dependent Variable: DRYING.RATE, ^{b}Predictors in the Model: (constant), TEMP.MC, ^{c}Predictors in the Model: (constant), TEMP.MC, MC, ^{d}Predictors in the Model: (constant), TEMP.MC, MC, MC.MC, ^{e}Predictors in the Model: (constant), TEMP.MC, MC, MC.MC, TEMP, ^{f}Predictors in the Model: (constant), TEMP.MC, MC, MC.MC, TEMP, TIME.TEMP, ^{g}Predictors in the Model: (constant), TEMP.MC, MC, MC.MC, TEMP, TIME.TEMP, TEMP.TEMP 
Table 6:  SPSS 21 outputs for backward elimination (variables entered/removed^{a})   ^{a}Tolerance: 0.000 limits reached, ^{b}Predictors: (constant), MC.MC, TEMP.TEMP, TIME.TIME,TEMP.MC, TIME.TEMP, TIM.MC,MC, TEMP, TIME 
• 
y = a_{o}+a_{2}x_{2}+a_{3}x_{3}+a_{12}x_{1}x_{2}+a_{23}x_{2}x_{3}+a_{33}x_{3}^{2}_{+}e 
• 
y = a_{o}+a_{2}x_{2}+a_{3}x_{3}+a_{12}x_{1}x_{2}+a_{23}x_{2}x_{3}+a_{2}x_{2}^{2}+a_{3}x_{3}^{2}_{+}e 
Table 7:  Model summary^{d}   ^{a}Dependent Variable: DRYING.RATE, ^{b}Predictors: (constant), MC.MC, TEMP.TEMP, TIME.TIME, TEMP.MC, TIME.TEMP, TIM.MC, MC, TEMP, TIME, ^{c}Predictors: (constant), MC.MC, TEMP.TEMP, TIM.TIM, TIM.TEMP, TEMP.MC, TIM, TEMP, TIM.MC, ^{d}Predictors: (constant), MC.MC, TIM.TIM, TIM.TEMP, TEMP.MC, TIM, TEMP, TIM.MC, ^{e}Predictors: (constant), MC.MC, TIM.TIM, TIM.TEMP, TEMP.MC, TIM, TEMP 
Table 8:  ANOVA^{a}   ^{a}Dependent variable: DRYING. RATE, ^{b}Predictors: (constant), MC.MC, TEMP.TEMP, TIME.TIME, TEMP.MC, TIME.TEMP, TIM.MC, MC, TEMP, TIME, ^{c}Predictors: (constant), MC.MC, TEMP.TEMP, TIM.TIM, TIM.TEMP, TEMP.MC, TIM, TEMP, TIM.MC, ^{d}Predictors: (constant), MC.MC, TIM.TIM, TIM.TEMP, TEMP.MC, TIM, TEMP, TIM.MC, ^{e}Predictors: (constant), MC.MC, TIM.TIM, TIM.TEMP, TEMP.MC, TIM, TEMP 
Table 9:  Coefficients^{a}   ^{a}Dependent Variable: DRYING.RATE, ^{b}Predictors: (constant), MC.MC, TEMP.TEMP, TIME.TIME, TEMP.MC, TIME.TEMP, TIM.MC, MC, TEMP, TIME, ^{c}Predictors: (constant), MC.MC, TEMP.TEMP, TIM.TIM, TIM.TEMP, TEMP.MC, TIM, TEMP, TIM.MC, ^{d}Predictors: (constant), MC.MC, TIM.TIM, TIM.TEMP, TEMP.MC, TIM, TEMP, TIM.MC, ^{e}Predictors: (constant), MC.MC, TIM.TIM, TIM.TEMP, TEMP.MC, TIM, TEMP 
Table 10:  SPSS 21 outputs for stepwise regression variables entered/removed^{a}   ^{a}Dependent variable: DRYING. RATE 
Table 11:  Model summary   ^{a}Predictors: (constant), TEMP.MC, ^{b}Predictors: (constant), TEMP.MC, MC, ^{c}Predictors: (constant), TEMP.MC, MC, MC.MC, ^{d}Predictors: (constant), TEMP.MC, MC, MC.MC, TEMP, ^{e}Predictors: (constant), TEMP.MC, MC, MC.MC, TEMP, TIME.TEMP, ^{f}Predictors: (constant), TEMP.MC, MC, MC.MC, TEMP, TIME.TEMP, TEMP.TEMP, df: Degree of freedom 
The assessment of the adequacy of each model using the selected criteria are presented in Table 1517.
Table 12:  ANOVA^{a}   ^{a}Dependent Variable: DRYING.RATE, ^{b}Predictors: (constant), TEMP.MC, ^{c} Predictors: (constant), TEMP.MC, MC, ^{d}Predictors: (constant), TEMP.MC, MC, MC.MC, ^{e}Predictors: (constant), TEMP.MC, MC, MC.MC, TEMP, ^{f}Predictors: (constant), TEMP.MC, MC, MC.MC, TEMP, TIME.TEMP, ^{g}Predictors: (constant), TEMP.MC, MC, MC.MC, TEMP, TIME.TEMP, TEMP.TEMP, df: Degree of freedom 
Table 13:  Coefficients^{a}   ^{a}Dependent variable: DRYING.RATE 
Table 14:  Excluded variables^{a}   ^{a}Dependent variable: DRYING.RATE, ^{b}Predictors in the Model: (constant), TEMP.MC, ^{c}Predictors in the Model: (constant), TEMP.MC, MC, ^{d}Predictors in the Model: (constant), TEMP.MC, MC, MC.MC, ^{e}Predictors in the Model: (constant), TEMP.MC, MC, MC.MC, TEMP, ^{f}Predictors in the Model: (constant), TEMP.MC, MC, MC.MC, TEMP, TIME.TEMP, ^{g}Predictors in the Model: (constant), TEMP.MC, MC, MC.MC, TEMP, TIME.TEMP, TEMP.TEMP 
Table 15:  Assessment of adequacy of the models obtained using forward selection method   RSS: Error sum of square for the pterm model, MSE: Error sum of square for full model, AIC: Alkaike information criterion, PRESS: Predicted residual sum of square, R^{2}: Rsquare, R^{2}_{adj}: Adjusted Rsquare 
Table 16:  Assessment of adequacy of the models obtained using Backward Elimination method   RSS: Error sum of square for the pterm model, MSE: Error sum of square for full model, AIC: Alkaike information criterion, PRESS: Predicted residual sum of square, R^{2}: Rsquare, R^{2}_{adj}: Adjusted Rsquare 
Table 17:  Assessment of adequacy of the models obtained using stepwise regression   RSS: Error sum of square for the pterm model, MSE: Error sum of square for full model, AIC: Alkaike information criterion, PRESS: Predicted residual sum of square, R^{2}: Rsquare, R^{2}_{adj}: Adjusted Rsquare 
DISCUSSION In assessing the adequacy of variable selection techniques on model building using the drying characteristics of fresh grains, each of the forward selection, backward elimination and stepwise regression methods identified a series of models assumed to be adequate. From the resulting models for each technique, the most suitable model was determined using six assessment criteria. For forward selection method, the predicted R^{2 }and adjusted R^{2 }criteria identified the model: y = a_{o}+a_{2}x_{2}+a_{3}x_{3}+a_{12}x_{1}x_{2}+a_{23}x_{2}x_{3}+a_{22}x_{2}^{2}+a_{33}x_{3}^{2}_{+}e as best out of a series of five models. The measure of the proportion of variability in the data set that is accounted for by the regression model is 98.30 and 98.20%, respectively. The least PRESS value of 0.3006 was also associated with the same model. The AIC further confirmed the model as most suitable. The C_{p} statistic as well as the Doptimality criterion identified the same model as best. These results were also true for the stepwise regression. There seemed to be no perfect agreement using the assessment criteria for the backward elimination method. The predicted R^{2}, adjusted R^{2}, AIC and Doptimality criteria identified the model: y = a_{0}+a_{1}x_{1}+a_{2}x_{2}+a_{3}x_{3}+a_{12}x_{1}x_{2}+a_{13}x_{1}x_{3}+a_{23}x_{2}x_{3}+a_{11}x_{1}^{2}+a_{22}x_{2}^{2}+a_{33}x_{3}^{2}_{+}e_{i} as best. However, variations existed with the PRESS and C_{p} statistic assessment criteria. Hence, for the experimental data used, all assessment criteria considered identified the same model as best for the forward selection method as well as the stepwise regression method but several models were identified by the assessment criteria using the backward elimination method. The Doptimality criterion has been successfully employed as a new criterion for assessing the adequacy of regression models. Its discriminating ability is as with the R^{2}_{adj}, PRESS, AIC and C_{p}statistic.
Appendix A: 
Drying rate of melon seed 


REFERENCES 
Agresti, A., 1990. Categorical Data Analysis. 1st Edn., John Wiley and Sons, Inc., New York, USA.
Burnham, K.P. and D.R. Anderson, 2001. Model Selection and Multimodel Inference: A Practical Information Theoretical Approach. Spring, New York.
Burnham, K.P. and D.R. Anderson, 2001. KullbackLeibler information as a basis for strong inference in ecological studies. Wildlife Res., 28: 111119. CrossRef  Direct Link 
Burnham, K.P. and D.R. Anderson, 2002. Model Selection and Inference, a Practical Information Theoretical Approach. 2nd Edn., SpringerVerlag, Berlin, Heidelberg.
Carley, K.M. and N.Y. Kamneva, 2004. A network optimization approach for improving organizational design. Technical Report CMUISRI04102, Institute of for Software Research International, Carnegie Mellon Uuniversity.
Carley, K.M., D. Fridsman, E. Casman, N. Altman and J. Chag et al., 2003. Biowar scalable multiagents and epideiological simulation of bioterrorism trends. Proceedings of the NAACSOS Conference, October 2003, Pittsbugh, PA .
Donoho, D.L. and I.M. Johnstone, 1994. Ideal spatial adaptation by wavelet shrinkage. Biometrika, 81: 425455. CrossRef  Direct Link 
Foster, D.P. and E.I. George, 1994. The risk inflation criterion for multiple regression. Ann. Stat., 22: 19471975. Direct Link 
Gerritsen, J., J.M. Dietz and H.T. Wilson Jr, 1996. Episodic acidification of coastal plain streams: An estimation of risk to fish. Ecol. Applic., 6: 438448. CrossRef  Direct Link 
Graybill, F.A., 1976. Theory and Application of Linear Models. Duxbury Press, North Scituate, MA.
Hastie, T., R. Tibshirani and J. Friedman, 2009. The Elements of Statistical Learning: Data Mining, Inference and Prediction. 2nd Edn., Springer, New York, ISBN: 9780387848587, Pages: 767.
Hobbs, N.T. and R. Hilborn, 2006. Alternatives to statistical hypothesis testing in ecology: A guide to self teaching. Ecol. Applic., 16: 519. CrossRef  Direct Link 
Kadane, J.B. and N.A. Lazar, 2004. Methods and criteria for model selection. J. Am. Stat. Assoc., 99: 279290. CrossRef  Direct Link 
Lawless, J.F. and K. Singhal, 1978. Efficient screening of nonnormal regression models. Biometrics, 42: 318327. CrossRef  Direct Link 
Lee, H. and S.K. Ghosh, 2009. Performance of information criteria for spatial models. J. Stat. Comput. Simulation, 79: 93106. CrossRef  Direct Link 
Montgomery, D.C., E.A. Peck and G.G. Vining, 2001. Introduction to Linear Regression Analysis. 3rd Edn., Jon Wiley and Sons, New York, USA., ISBN13: 9780471315650, Pages: 672.
Myers, R.H. and D.C. Montgomery, 2002. Response Surface Methodology: Process and Product Optimization Using Designed Experiments. 2nd Edn., Wiley, New York, ISBN: 9780471412557, Pages: 824.
Myers, R.H., 1990. Classical and Modern Regression with Applications. 2nd Edn., Duxbury/Thompson Learning, Boston, USA., ISBN13: 9780534380168, Pages: 488.
Olden, J.D. and D.A. Jackson, 2000. Torturing data for the sake of generality: How valid are our regression models? Ecoscience, 7: 501510. Direct Link 
Raffalovich, L.E., G.D. Deane, D. Armstrong and H.S. Tsao, 2008. Model selection procedures in social research: MonteCarlo simulation results. J. Applied Stat., 35: 10931114. CrossRef  Direct Link 
Sauerbrei, W., P. Royston and H. Binder, 2007. Selection of important variables and determination of functional form for continuous predictors in multivariable model building. Stat. Med., 26: 55125528. CrossRef  Direct Link 
Tibshirani, R. and K. Knight, 1999. The covariance inflation criterion for adaptive model selection. J. R. Stat. Soc. Ser. B, 61: 529546. Direct Link 
Ward, E.J., 2008. A review and comparison of four commonly used Bayesian and maximum likelihood model selection tools. Ecol. Modell., 211: 110. CrossRef  Direct Link 
Weisberg, S., 1985. Applied Linear Regression. 2nd Edn., John Wiley and Sons, New York, USA.



