Abstract: We suggest considering the occupational outcome as an additional dimension to the ranking of academic institutions. In this study we analyze the Italian situation: we use the ISTAT graduate employment national survey to compute the occupational measure and show how the Censis - la Repubblica ranking of Italian Universities changes when accounting for it.
INTRODUCTION
For many years some organizations have come up with different ratings of universities for specific purposes. As a consequence, the same university can be differently ranked by several rating systems. The purpose of an overall ranking is to establish partnerships among universities in teaching or research fields. In such case the ranking is built by specialised organizations like The Institute of Higher Education of the Shangai Jiao Tong University, The Centre for Science and Technology Studies CWTS of Leiden University or the Centre for Measuring University Performance. On the other hand, if the purpose of the ranking system is to assess the quality of research from the point of view of a public authority with spending responsibilities, we have national rankings like the RAE (Research Assessment Exercise) in the United Kingdom or the CIVR (Comitato di Indirizzo per la Valutazione della Ricerca) in Italy. In some cases various organizations provide more specifical type of ranking. Notable examples of this trend are the Webometrics Ranking Universities by the Cybermetrics Lab and the Racial Diversity Ranking by the Journal of Black Higher Education. Finally, daily newspapers and magazines offer some interesting national ranking. In this case rankings are supposed to help students and their families in choosing the right university. The most important mass-media rankings are: the Maclean’s in Canada, the U.S.News and World Report in USA, the Times Higher Education Supplement of Guardian and Financial Times and The Complete University Guide of The Independent in United Kingdom, the Censis-la Repubblica in Italy. It is worthwhile to point out that rankings in Anglo-Saxon countries are at least 20 years old, while the first Italian ranking is only 8 years old. In fact, since 2000 the Censis (Centro Studi Investimenti Sociali) has been issuing a ranking of the universities that is published by the newspaper la Repubblica to monitor the performances of the large number of universities and of the different courses offered by each of them. These courses were born as a consequence of a period of extensive reforms, started in 1999 into Italian University System. This is a likely symptom of the fact that the amount of competition among universities increased as well. In the presence of fiercer competition, there is an increasingly strong need to have reliable university rankings that are particularly useful for students to make the best choice. In turn, this ranking is based upon a set of sub-rankings which tap into specific aspects of the academic activity: Productivity, Education, Research, International Relationships, Teacher Profile (In this paper we consider the 8th edition (2007-2008) of the Censis Guide). In our opinion this ranking neglects an important aspect, i.e., the Occupational Outcome of those graduating of one academic institution. The Italian Ministry of the Education explicitly remarks (DD.MM. 26/07/07) that the faculties have to guarantee their graduates appropriate job and placement activities. Moreover, the general public seems quite interested to the occupational prospective. Hence, some Italian magazines tend to highlight this very aspect (See for example Panorama on 29 May 2008 and Il Sole 24 Ore on 14 July 2008). So, we suggest a new index which captures this occupational outcome; then we combine it with the others which are already embedded in the Censis ranking, in order to obtain a new overall rating. Finally, we compare the new ranking with the old one, with the purpose of pointing out possible differences. In our opinion this new index is better at capturing the multi-faceted reality of academic institutions and hence more useful to the students.
OCCUPATIONAL OUTCOMES EVALUATION
An important aspect which should be taken into account when evaluating and comparing universities is the occupational outcome of those graduating at each institution. Some other authors have considered these topics in recent years (D'Hombres et al., 2008). Since, there are several surveys which ask about graduates’ employment, an index capturing occupational outcomes is feasible. In the case of Italy three authoritative agencies conduct this kind of research: the Italian National Statistical Institute (ISTAT) leads graduates’ employment national survey every three years; the ALMALAUREA and the CILEA with the project STELLA run a similar survey every year, albeit not at national level. At international level one should also mention the REFLEX project The Flexible Professional in the Knowledge Society New Demands on Higher Education in Europe (Reflex, 2007).
There is a host of papers in literature which are focused on job opportunities of Italian graduates: Ballarino and Bratti (2009) study how students’ occupational outcomes differ, as a function of the chosen field of study (faculties) and how this relationship has changed over time. They also emphasize the impact of graduates’ socio-economic background on employment outcomes. Biggeri et al. (2001) examine the factors determining the transition from university to work and analyse the effectiveness of universities and course programmes on labour market outcomes; in particular, they focus their attention on how times may vary when someone’s trying to get their first job. Again for the Italian case, Brunello and Cappellari (2008) study the effect of the attended university on earnings and employment prospects three years after graduation.
Nevertheless, when evaluating university performance, the occupational outcome of graduates is seldom taken into account. For example, Agasisti (2009), in order to investigate the effects of increasing competition on the performance of Italian universities, adopts as an indicator of performance the main indices appearing in the annual report of the National Evaluation Committee. These are the rate of first-year students with higher scores in secondary schools, the rate of regular students and regular graduates and drop out rates: in fact, an indicator concerning graduates’ occupational outcomes is remarkably absent. Moreover, the university rankings cited in the previous paragraph do not take into account graduates’ employment opportunities when evaluating the overall university performance. An exception is represented by the Times Higher Education Supplement of Guardian and Financial Times in United Kingdom and The Complete University Guide of The Independent, an online ranking of UK universities. In particular the international ranking of the Times Higher Education Supplement considers 4 principal criteria: Research Quality, Graduate Employability, International Outlook and Teaching Quality. To each one is assigned a different weight: 60, 10, 10 and 20%, respectively. For our work is important to analyse the Employer Review Indicator, linked to the Graduate Employability: it accounts for only 10 per cent of the possible score but is of burning interest to students and parents alike, as well as to universities themselves. For 2007 World University Rankings 1,471 recruiters of graduates from around the world said where they would rather get their employees from. The Recruiter Review is based on online surveys. Respondents are asked to select up to 30 institutions they consider to be excellent in their area(s) of knowledge. They can choose among All Faculty Areas, Arts and Humanities, Engineering and IT, Life Sciences and Biomedicine, Natural Sciences and finally Social Sciences. The Complete University Guide, in association with The Independent (2009), publishes the League Tables of UK Universities since 15 years ago. The raw data for the League Table come from sources in the public domain. The League Table measures nine key aspects of university activity: Student Satisfaction, Research Assessment, Entry Standards, Student-Staff Ratio, Academic Services Spending, Facilities Spending, Good Honours, Graduate Prospects and Completion. In particular, for our research is important to analyse how the Graduate Prospects measure is compiled. It is a measure of the employability of a university’s graduate. It comes from HESA (Higher Education Statistics Agency) data and it reflects the number of graduates who take up employment or further study divided by the total number of graduates with a known destination expressed as a percentage.
In our opinion the graduates’ employment opportunities are especially important to choose the university and the field of studies; thus, rankings dedicated to prospective students should definitely incorporate this relevant dimension of university performance, alongside other indicators.
THE CENSIS RANKING ACCORDING TO ISTAT OCCUPATIONAL SURVEY
Here, following the Censis approach (Censis, 2007), we will calculate a new ranking which adds to the Censis indices a new dimension which is meant to capture the occupational aspect. The Censis-la Repubblica is split up in two: a ranking of the universities and a ranking of the faculties. Here we consider the latter. The rating model used by Censis is complex and some critics are moved about it. The university system is represented by five dimensions. In the Censis methodological note attached to the ranking, the five dimensions are described with the following five indices:
| • | Productivity: to ensure a regular course of studies |
| • | Education: to ensure a balancing ratio between teachers and students, to provide adequate facilities and adequate training, etc. |
| • | Research: to assess the ability of teachers to engage in research activities and establish how many chances a student may have with such professors |
| • | Teacher profile: to appreciate the faculties that direct their attentions on intergenerational turnover and on the international relations of its staff |
| • | International relationships: to establish the degree of openness to the international experiences of both students and staff |
Each index is a synthesis of some pointers (Biolcati Rinaldi et al., 2008) and the Censis methodological note. The indices are normalized by Censis in the range 66-110 (as in the degree evaluation), while the final rating is calculated as the simple mean of those.
Here we add to the five Censis dimensions another dimension. This one describes the occupational outcome.
In this exercise we get the pointers from the questionnaire of the ISTAT graduates’ employment national survey (ISTAT, 2005). This is the only survey with national coverage and it is carried out by ISTAT every three years; so it is now the most updated survey about the graduates’ employment. The aim of the paper is to improve the Censis ranking, so it seems useful to give to the students data about the University situation at the moment of their choice and concurrently the most updated information concerning the graduates’ employment. The survey is carried out on a random sample of graduates, stratified for gender, university and course of study drown from a target population of 155,664 individuals; the respondents in the sample are 26,006 (answered rate is of 67,6%). The questionnaire is composed by 5 sections: the first is devoted to the study curriculum and qualifying activities, the second to the job, the third to the job research, while in the fourth and fifth information about family and personal data are collected. The pointers used for building the occupational index are in the section 2 of the questionnaire, except for the question about the potential reply of the chosen courses, which is in the first section.
Actually, the matter is complex; the literature suggests that the socio-economic and cultural characteristics of the graduates affect their occupational outcome (Checchi and Flabbi, 2007; Breen, 2004). So, according with Ballarino and Bratti (2006), we have built two variables which consider these aspects: family’s cultural capital, a dummy variable distinguishing people who have at least one parent with a university degree from the others and family’s social class, coded according to the highest occupation of their parents when respondents were 14 years old. We consider 3 social classes: upper class (entrepreneurs, managerial and high-level dependent workers, professionals); middle class (white collars, namely mid-level non manual dependent workers, and petty bourgeoisie, that is self-employed workers); working class (manual work, also if qualified and low-level non manual positions such as shopkeepers or babysitters). Therefore, we have developed a model in which the variable to be employed /not to be employed (see the question 2.1 in the ISTAT questionnaire) is the dependent variable and the variables cultural capital, social class and university (a dummy variable for each university) are the independent variables, for each faculty group. Specifically, the model is the well-known binomial logistic model (Agresti, 2002), that assumes a dichotomous dependent variable Y with probability π, that the i-th case is:
| (1) |
where X is the vector of independent variables and β the vector of parameters, which will be estimated with the maximum likelihood method.
We use as a case study the performance of Humanities faculties in Italy, which are 42. The number of graduates in the considered period varies from a minimum of 117 to 1334, the dimension of the sample used in the analysis is of 1525 graduates, with an average sampling ratio by faculty of almost 7,6%.
Table 1A shows the logistic regression coefficient of the model and Table 1B shows the occupational index that in the following will be calculated and commented.
The results obtained with the logistic model are very interesting, as shown in Table 1A. For Humanities (We estimate the model for each faculties group, in almost all the cases the university is significant) faculties the university is significant (The considered significance level is α = 0.05); in other words, net of cultural capital and social class, there is generally a university effect on the probability of to be/not to be employed.
| Table 1: | Coefficients estimated for the logistic model and occupational index for the humanities faculties |
To have a parent with university degree increases the probability to have a job; more over to belong to the Upper class increases the probability to have a job with respect to belong to the lower class of 36% (Exp(B)). The increasing for the middle class is 15%, but significant too.
Moreover, some universities (in italic in the Table) do not have significant coefficients, see for example Vercelli, Genova, Venezia, ect. For these faculties of Humanities there is no changing in the probability to find a job that depends on the university. Indeed Parma, Padova, Trento, Milano Catholic are the faculties of Humanities that mainly increase the occupational opportunities. Lecce and L’Aquila are the ones with the worst situation.
The result justifies and improves our intention to introduce the occupational dimension in the university ranking building.
According to the Censis method the new dimension is the synthesis of some pointers. Analysing the ISTAT data and according to the international literature, quoted in Occupational Outcomes Evaluation, we have chosen some questions and we have built appropriate pointers. The building pointers are the following:
| • | Percentage of those who would choose again the same degree (question 1.24)-O1 |
| • | Percentage of those who declare to be employed (question 2.1)-O2 |
| • | Percentage of those who declare to have a stable job (question 2.4 categories 1 and 4)-O3 |
| • | Percentage of those who declare to have a high job category (question 2.5 (categories 1 and 2) and 2.8 (categories 1 and 2 and 3) compared with question 2.4 categories 1 and 4)-O4 |
| • | Percentage of those who declare to have a high level of income (higher than 1000 euro net) (question 2.22a)-O5 |
| • | Percentage of those who declare that the obtained degree is necessary for their job (question 2.27)-O6 |
| • | Percentage of those who are satisfied with their job (simple mean from questions 2.31a to question 2.31f and then simple mean lower or equal to 2)-O7 |
| • | Percentage of those who declare that the university had a role in finding job (question 2.32, categories 3 and 4)-O8 |
A first attempt considers all these 8 pointers and the occupational index is obtained as simple mean of them. The values of indicators are not here presented for each Humanities faculty, but their mean and variability are reported in Table 2. It is important to notice that those indicators are positively correlated with each other. In the Table 1 (B) it is shown the value of occupational index for all the faculties of Humanities and its mean and variability are reported in Table 2 (last column).
Probably the occupational index is influenced from the socio economic context in which the faculty is located. In this exercise we consider the faculty of Humanities, but the same exercise can clearly be replicated for all faculties. Faculties of Humanities located in the same city have different values for the index; for example, there are three faculties of Humanities in Rome and three in Naples. It turns out that Tor Vergata in Rome and Napoli Orientale in Naples get the highest ranking within their respective cities. Moreover, in Milan the index for the Catholic University is 0.58, while the one for the State University is 0.45. This difference is further confirmation of the result obtained by model (1): it is realistic to deduce that the index describes also the ability of the faculty to improve the competitiveness of its graduates.
The new rating is obtained combining the five Censis indices with the occupational outcome one, by their simple mean; in this analysis, in conformity with the Censis methodology, private faculties are not considered. The rho Spearman between the old rating and the new one is 0.920. Figure 1 shows a scatter plot of the two rankings.
It is clear from this example that the ranking changes for effect of the inclusion of the dimension occupational outcome. Bari, Catania and Napoli Orientale improve their positions and the same happens with Parma, Verona, etc. On the other hand, some others, such as Macerata, Cassino and Firenze, get worse (Fig. 1).
The developed index is composed by very heterogeneous pointers; some of them are related to the subjects and then are more affected by their socio-economic and cultural characteristics, or by the physical features of the land; the others are connected with the university peculiarities. To measure distinctly the two aspects we have developed two specific indices, said respectively job index and placement index. The first is a synthesis (simple mean) of the pointers O2, O3, O4, O5 and O7 and it seizes the job aspects more affected by the graduates’ personal characteristics and by the land context, while the second is a synthesis (simple mean) of the pointers O1, O6 and O8 and it is strictly related to the university.
In the following plot (Fig. 2) the new indices obtained for each University are reported. The interesting situations are various: the best is Trento, where the values of the two indices are among the higher; Napoli Orientale seems to be the best university for placement, but it is beaten by many others in job, for example Parma and Padova, according with the result in Table 1. The University of Firenze has the worst placement value and it is no good for job as well. L’Aquila and Potenza have the lower values for job, but they are quite good for placement.
Then we have built two new rankings, inserting respectively the job index and the placement one. In the Fig. 3, 4 and 5 we will show the graphical comparison between them and the correlations between the various rankings: Censis, with global occupational index, with job and placement indices.
| Table 2: | Mean, standard deviation and coefficient of variation of occupational outcome pointers and occupational index for humanities faculties |
| Fig. 1: | New ranking compared with the old one for the faculties of humanities |
| Fig. 2: | Job and placement indices for the faculties of humanities |
The plot in Fig. 3 supports what is shown in Fig. 2. Trento University is now down and on the left in the graph: it is the university with the biggest rating, so it fills the first position in both rankings. Potenza has a better position in the ranking considering placement than in the ranking considering job, as previously observed.
About the correlations among the various rankings, the strongest correlation (0.971) is between the Censis ranking and the ranking considering job. Between the Censis ranking and the ranking considering placement there is a correlation equal to 0.944, bigger than the correlation between the Censis ranking and the ranking with global occupational index (0.921). The latter is obviously correlated with the ranking considering placement (0.942) and the ranking considering job (0.960). Also between the two rankings considering respectively placement and job there is an high correlation (0.957).
| Fig. 3: | Censis + job ranking compared with censis + placement ranking for the faculties of humanities |
| Fig. 4: | Censis ranking compared with censis + placement ranking for the faculties of humanities |
It is to notice that the placement is the dimension which most affects the ranking; as a matter of fact the correlation with the Censis ranking is lower.
It is interesting to graphically compare the Censis ranking with the two rankings considering respectively job and placement. The two new indices don’t spawn very different rankings in the extreme situations, confirming in this way the Censis ranking, but they cause meaningful changes in the intermediate positions. For example, Firenze is in fifteenth position in the Censis ranking but it goes down to the twenty-fifth position in the ranking considering placement ant to the twentieth position in the ranking considering job, confirming that this university is not good as far as the placement is concerned. On the contrary, Urbino is in the thirteenth position in the Censis ranking and in the ranking considering job, but it is in the seventh position in the ranking considering placement, earning six positions.
| Fig. 5: | Censis ranking compared with censis + job ranking for the faculties of humanities |
REMARKS
Our starting point was to remark that the university ranking system has to take into account the occupational outcome of those graduating at each institution. The problem is not simple because the variables used to measure the occupational outcome, for example those reported in the ISTAT questionnaire and analysed in this paper, derived from a lot of factors: the faculty ability to find a job to their graduates, the land peculiarities, the individuals’ socio-economic and cultural characteristics.
In the study, building the logistic model and the partial job and placement indices, we have tried to break apart the various elements. The logistic model has shown that, net of cultural capital and social class, there is generally a university effect on the probability of to be/not to be employed. So, it seems appropriate to consider the occupational dimension in the university ranking construction, to take also into account the new government guidelines and the main interest of a student who has to choose a study course. In the case of Italy the only available ranking of faculties is the Censis - la Repubblica one. In this study we show that including the occupational dimension in the Censis score is not inessential, since it would change the final ranking. The new rankings obtained inserting job and placement indices are different compared to the Censis ranking. This preliminary result, regarding only the faculties of humanities, prompts us to carry on this research direction. The future issues are various: it will be important to suggest an appropriate system of weights, to join the dimension indices and to obtain the final rating. It will be necessary to decide the way to insert both the job index and the placement one. The opportunity to consider also the territorial market labour dimension in the ranking construction will be weighed up: specifically, the use of the multilevel logit model, by territorial area and by university sites, will be assessed. Finally, the effects of the graduates’ socio-economic and cultural characteristics on the job and placement indices will have to be checked.