Abstract: The new Italian regulation for the distribution of the Ordinary Financing public Fund (FFO) is included in the framework of the reformation of the University System financing politics. A part of the FFO is bound by a specific merit rating performed by the C.N.V.S.U. (forthcoming A.N.V.U.R.) for the competent Ministry. Such an evaluation results in a periodical monitoring of achieved results for each University, compared with the triennial operating programme in which the main targets, in terms of improvement and efficiency of the public service quality, are presented. From the quantitative viewpoint, this ambitious goal is reached by means of a Quality Aggregate Index (QAI), obtained by the weighted synthesis of twenty-one parameters, measured on scale of ratios and assembled for homogeneity in five macro-areas. The present proposal for the QAI quantification does not seem to be statistically clear and is tends to create discordant results compared with the inspiring regulation. In this study, the authors, preventatively examine carefully the procedure indicated by the Ministry, pointing out the statistical problems implied in it. Such statistical problems can arise also in assessment systems for other countries where the Public Funds distribution is correlated with the evaluation of the efficiency of University (public and/or private) activities. We focus our attention on the theoretical and computational problems of a generic QAI, suggesting an alternative methodology. At the end of the paper, numerical simulations are computed in order to show the effectiveness of the adopted procedure.
INTRODUCTION
The recent (L.43/2005) Italian regulation for the distribution of the Ordinary Financing public Fund (FFO) is included in the framework of the reformation of the University System financing politics and pursues the public service improvement aim. This approach is strengthened by the recent law L. 1/2009 which aims to bring the present internal system of resource distribution for higher education and scientific research near to a qualitative logic of productiveness, already adopted in other industrialised countries. It is well-established (29 industrialized countries declaration) the need to individualize a comparable evaluation system among the European countries (Quality assurance), nevertheless a common and shareable evaluation procedure is not still available, the last should be methodologically accepted and, meanwhile, easy to be interpreted at an international level. The Italian attempt in this direction is undoubtedly a first remarkable step forward.
At present, only the assignment of 1/3 of FFO is related to a specific evaluation (ex-post) of every single Italian University. Such an evaluation is inspired by the aims-results binomial.
Each University decides (ex-ante) the aims, in self government, during the triennial operating programme while the results are quantified (ex-post) by a specific Quality Aggregate Index (QAI).
The Italian Government (Ministry of University) has fixed2 both parameters and criteria in order to quantify the QAI3. There are 21 parameters, measured on scale of ratios and assembled for homogeneity in macro-areas (Fig. 1). The 5 macro-areas are: a) Degree Curricula (3 parameters); b) Scientific Research (5); c) Services for students (5); d) Internationalization (4); e) Academic and Administrative Staff (4).
| Fig. 1: | Nidification scheme of variables |
Following the mentioned aims-results binomial:
| • | Each parameter is linked with a Simple Quality Index (SI) which is derived by the improvements or worsening 4that characterize the results of each monitored activity |
| • | Each macro-area is linked with a Partial Quality Index (PI), by the un-weighted mean of SIs computed for the single variables identificated in it |
The QAI5 derives from the weighted mean of the partial indicators6.
PARTIAL INDEX AND QAI COMPUTATION
The SI for each variable are computed relating the value in the year 2008 (time m = 1) to the mean of the results in the triennium 2004-2006 (time m = 0). The presence of a reference value (base) calls for, from the statistical viewpoint, the construction of a simple index number.
According to national regulation conditions, the real quantitative problem is substantiated in the identification of the best data synthesis procedure (complex and/or weighted index number).
Such a synthesis must lead to the correct valorisation of the Partial Index first, then of the QAI. In literature (Cecchi, 1995; Balk, 2008), two main index numbers aggregation techniques exist (Ratios of the means - RM; Mean of the Ratios-MR), on the other hand there are plenty of synthesis functions and all of them can be traced to the power means Mt.
In general, as shown in the solution of similar problems (Russo, 2007), let xvj1 be the value of the v.th parameter (v = 1, 2, … Vj) in the macro-area j (j = a, b, c, d, e; so Va = 3, Vb = 5, Vc = 5, Vd = 4, Ve = 4) at time m = 1 (year 2008); let μvj0 be the v.th parameter’s mean in the macro-area j at time m = 0 (triennium 2004-2006); let pvj and pj be respectively the weight of the v.th parameter in its macro-area and the weight in the whole macro-area, according to the national regulation criteria it results: pvj = 1/ Vj and 10% ≤ pj ≤ 30%.
As a consequence, the partial indicator for each j.th macro-area can be derived from one of the following equations:
| (1) |
| (2) |
It is well known that the two methods lead to the same result in the case t→0 (geometric mean). In order to avoid tendentiously positive (or negative) interpretations7, it would be better to apply the following equation:
| (3) |
Moreover, as shown in Russo and Gismondi (2007) the geometric mean also presents more sensitiveness8.
In this way, the complex index (QAI) computation is immediate:
| (4) |
PRELIMINARY SPECIFICATIONS
Let us consider a quantitative variable x, the simplest way to compare the registered value with a reference base is the so called Indexation, that is to consider the ratio between the value and a characteristic value (usually the mean)9. The main consequence of this simple procedure is that the new variable is independent from the former system of measurement. In order to take in due account also the range of variation of the variable is necessary to adopt the so-called Normalization.
Generally speaking, a transformation based on Indexation seems to be more suitable when the purpose consists in building up a series of independent index numbers, without need to synthesise them into a unique overall performance indicator.
However, also Normalization results could be heavily affected by potential outlying values at the minimum and/or maximum (Russo and Gismondi, 2007). Moreover, effects of Normalization are less immediately clear when original variables are characterised at same time by different mean, minimum and maximum. This potential problem can be reduced using as minimum and maximum ad hoc theoretical values instead of empirical ones (as in the case proposed by Italian Government); however, this choice could not completely eliminate the problem if even one of them is not representative, being quite distant from the mass of data of the observed distribution. In general, a useful preliminary step is given by an explorative analysis of data and their density distributions (Hardy et al., 1988; Hoaglin et al., 2000).
NORMALIZATION OF VARIABLES
Linear transformation: The resort to absolute value variables, also in the totalization of simple index numbers (Spada and Russo, 2006) does not solve some problems such as the different unit of measurement and/or the different variables range of variation; this is more evident when the variables minimum value is different from zero. With this preamble, the real weights of single variables should undergo a dangerous modification. This is the reason why, standardisation or better pre-emptive normalization is necessary (Grilli and Russo, 2007, 2008), also by virtue of the exact individualization, in the specific case, of minimum and maximum values in the reference frame. Formally, the classic normalization is:
| (5) |
where, zvj1 is the normalized value of the v.th variable in the j.th macro-area at time m = 1 and where xvj,min and xvj,max are, respectively, the minimum and maximum registered value in the same variable in the whole University system.
More in general, defining a linear transformation of the former variable: zvj1= a + bxvj1, the aforementioned case represents one of the possible variable transformations, in particular when:
a = - xvj,min/( xvj,max-xvj,min)
and b = 1/(xvj,max - xvj,min) |
In order to show the positive effects of a simple normalization let us consider the following similar case (Grilli and Russo, 2007): let us consider the problem of decision making in financial markets in the case in which, for example, the investor makes her decision taking into account only two different financial indicators for each asset: (first variable) CAP (in billion of euros) and (second variable) EPS (in euros). The authors consider 38 assets from Italian Stock Market. In Fig. 2 the cloud of data is plotted, the EPS value is in the first axis and in the second axis the CAP. The parameter CAP, as a consequence of its range of variation, is over-weighted compared with the parameter EPS. In Fig. 3 the data cloud has been considerably affected by the normalization procedure and, as a result, the two variables are now in the same range of variation and they are equally weighted. The authors compute the two ordering procedures, using data in original scale and in the case of normalized data. The ordering procedure shows very different results in particular for assets that have bad performance in CAP, that is the over-weighted variable. The last statement can apply to any of the 21 parameter considered in the evaluation of University activities performance.
| Fig. 2: | Data are plotted in the original scale, it is evident how the parameter CAP is the most relevant one (Grilli and Russo, 2007) |
| Fig. 3: | Applying the normalization procedure the two variables are now in the same range of variation (Grilli and Russo, 2007) |
Non-linear transformation: In addition to the above, it is evident that the classic normalizing transformation assigns a specific meaning to the minimum and maximum variables value, disregarding, actually, the distribution associated with the entire variation range. As a consequence it is preferable (Russo and Gismondi, 2007) a non-linear transformation z = f(x) satisfying the following condition:
| (6) |
where, λx in ]0,1[ is an adjustment value for the variable x and it varies according to the weight that must be assigned to the difference of the observed value x and its minimum, that is according to the shape (asymmetry and dis-normality) of the distribution. The biggest is the value of λx the lowest is, by contrary, the relative weight assigned to the difference (xvj,max - xvj). The underlying idea in Eq. 6 is that, when λx = 0,5, after the transformation into the new scale z, the ratio between the difference of the transformed value from its minimum and the distance of its maximum from z must equalize the same ratio computed in the former variable x. From Eq. 6 it is easy to obtain the non-linear transformation:
| (7) |
which can be seen as the arithmetic weighted mean of zvj,max and zvj,min. In particular, fixing λx=0,5, it is possibile to obtain, after some algebra, the classic normalization Eq. 4.
Consequently, for any xvj, xvj,max and xvj,min, the value of zvj will be bigger the higher the coefficient λx, pointing out a bigger weight to the distance of xvj from its minimum, rather than the distance from its maximum. The last solution is useful in the case in which the minimum in the scale is more representative in comparison with the maximum, as happens in the case of positive asymmetric distribution (with a negative asymmetric distribution the opposite case must be considered). As a consequence, the QAI indicator can be obtained from Eq. 4, subject to the substitution, in equation Eq. 3, of xvj1 and μvj0 by zvj1 and μzj0 computed from equation Eq. 7. This procedure, at last, releases the variables, that must be quantitatively manipulated using a synthesis indicator, from: the influence of a different unit of measurement; the minimum and maximum value in the reference scale; the different distribution of data which characterizes them (asymmetry and dis-normality)
RESULTS
Using data (The name of each University (except University of Foggia) is replaced by a randomly assigned code) from the Italian Minister of University, a numerical test is computed in order to show the negative outcomes of the global index proposed by the Italian Government.
We consider the 59 Italian Public Universities and we compute, for each one, the complex index (QAI) following the Italian Government procedure and the procedure suggested in this paper both in the simple (index numbers) and in the normalized version. The resulting ordering procedures are computed; the problems indicated in the previous sections of the paper are self-evident. The results of the simulation are presented in Table 1.
The Italian Government indicator seems to be very inflexible and it does not appropriately discriminate Universities since it moves in the range [0.15, 0.22], as a consequence many Universities achieve the same position and the results, in performance terms, are not easily readable. Consequently it is not an efficient performance indicator. The proposed indicators (both simple and normalized) generate rankings much more flexible and correlated with University performance. The resulting rankings are very different compared with the QAI generated using the Italian Government procedure. It is worth noting that the correlation of the proposed methods is high while the correlation between the two methods and the Italian Government one is very low and this is due to the presence of over-weighted variables, as shown in the previous section. The case of Univ. 3 is emblematic (49th in the Italian Government ranking and 1st and 18th in the proposed rankings), in this case the presence of a variable (e4) which decuples its value in the triennium determines, in the normalized indicator, an enormously positive result; this situation is very frequent also because the initial value for some variable is zero.
| Table 1: | The QAI computed following the procedure suggested by the Italian Government and the consequent ranking together with QAI and rankings computed following the procedure suggested in this paper both using index numbers (QAI*) and normalization (QAI**) |
| QAI*: Complex index obtained using the simple index numbers procedure. QAI**: Complex index obtained using the normalization procedure | |
The numerical simulation confirms, unequivocally, that the problem of standardisation or better pre-emptive normalization is very important since it influences the synthesis results that are the Italian Government goal. The following discussion on which should be the computational method to be adopted is still open, in this paper we have suggested a possible solution.
DISCUSSION
The need to individualize a comparable evaluation system of high formation among the European countries (Quality assurance) is still without a common answer at international level. The forthcoming procedure for the redistribution of a part of the FFO to Italian Universities represents a simple (initial) answer, even if it is not without quantitative problems. Every evaluation system of a public service undergoes the same problems.
As shown in the numerical computation, if not solved, such problems can lead to decisions that are not coherent compared with the inspiring regulation (apart from Statistics).
CONCLUSIONS
Considering the lack of a methodological note from the National Government, a critical discussion on the choice of parameters, macro-areas and weights assigned by the Government is necessary, in order to avoid, during the monitoring period, distorted and politically dangerous assessments.
Since, parameters and macro-area are usually fixed by the National Regulator, each Country should focus on the choice of appropriate normalizing procedure of former variables and on the most coherent synthesis function of the variables into a partial and global index. In this study we have presented the theoretical motivations and, using a numerical simulation, the effects of the synthesis procedure on the assessment process. A complete study on which index number and, above all, which normalization (linear or non-linear) should be the most suitable is our future research address. Another direction for future research is a detailed statistical analysis of data and results in order to evaluate the contribute of each variable in the aggregate index and also to underline the presence of cluster structures among Universities.
ACKNOWLEDGMENTS
The authors deeply thank the Administrative Direction of University of Foggia for providing us data. We also thank the two anonymous referees, they have considerably contributed to the improvement of this study.