Abstract: Background and Objective: Items evaluation towards intellectual quotient (IQ) instruments is rarely focused on technical students for local contexts. Empirical evidence is essential for the purpose of assessing the quality of cognitive testing items. This study aimed at evaluating the quality of multiple choices of Ravens Advanced Progressive Matrices (RAPM) items in measuring IQ using Rasch model analysis in the polytechnics context. Methodology: The research design used a survey design with the fully quantitative approach. The total of 150 students from one of the polytechnics in Perak was selected by convenience sampling technique. Results: The results showed that 23 items have been dropped from 36 items of RAPM and 13 items found to fulfil Rasch model assumptions such as item fit, uni-dimensionality, local independence, item polarity, gender differential item functioning with evaluating the aspects of reliability and separation index. Conclusion: The items assessed are applicable for Malaysian polytechnics in order to measure IQ for the technical workers in future.
INTRODUCTION
The term intelligence is initially introduced at the end of 1800. The intelligence is inherited but nevertheless fails to form an intelligence test that can be used in his research. Intelligence also defined as the comprehensive capacity or ability to act intentionally, to think rationally and to deal effectively with the environment. Intelligence refers to an individual's cognitive ability to learn from experience, well-reasoned and coping effectively with the pressure of daily life1. To focusing on producing technical workers, polytechnic has their aim to develop and transform human capital to meet a requirement by the industry through the transformation agenda until 2020 for skill and education 2. This transformation also requires polytechnic students to improve their intellectual quotient (IQ) through rapid changes in their high order thinking style.
In order to achieve that, it’s important to have an instrument to measure IQ with good psychometric characteristics. Ravens Advanced Progressive Matrices (RAPM) is one of the potential instruments which can measure general intelligence among polytechnic students. This research will be focusing on the assessing the psychometric properties of items for RAPM using Rasch model to confirm the suitability of items for replication for the polytechnics. This can deliver signs for improving the items to have a better psychometric quality. Several studies from abroad3-6 and local7 are used RAPM in their study. Unfortunately, the study in assessing psychometric properties using Rasch model on local context for polytechnic students are limited. One of the research in Indonesian context related to this just focusing on classical test theory and not the modern theory like Rasch model3.
The previous paper published was not discussed about the psychometric properties of Raven advanced progressive matrices specifically. This paper will be provided all the empirical piece of evidences in the context of Rasch model and not just the correlation with other variables. Previous research also measuring IQ using Raven standard progressive matrices (with 60 items) and this paper is relying on Raven advanced progressive matrices (with 36 items). Some of the study not even similar in terms of the educational settings. It was conducted in different countries but this study was in Malaysia and for polytechnic context (technical students). In terms of analysis, the previous study covered only for descriptive statistics and principal components analysis but this paper covered the aspect of Rasch. This research work is significantly advance the current knowledge in terms of Gender Differential Item Functioning (GDIF) in Rasch which provide items fairness and unbiased. Nevertheless, this paper also mentioned about items that favor to male or female and not only focusing on Rasch assumptions only.
The objective of the research was to evaluate the quality of multiple choice of items in measuring IQ using Rasch model analysis such as item fit, uni-dimensionality, local independence, item polarity, gender differential item functioning, item-person map, reliability and separation index in the context of polytechnic in Malaysia.
Intellectual quotient (IQ): This study uses two intelligence theories, namely the Spearman general intelligence theory by Charles Spearman and the Horn and Cattell two-factor theory, introduced by John Horn and Raymond Cattell. Intelligence can be defined as the general ability to underlie the various behaviors. It was also as a form of mental energy generated by the brain. Horn and Cattell’s two-factor theory categorized two intelligence factors, namely the g factor which is a fluid intelligence (gf) and a crystallized intelligence (gc). Fluid intelligence (gf) means the ability to think and the capacity to remember and process information accurately. This intelligence is not affected by the environment8. Fluid intelligence is easily measured using culture free testing such as non-verbal tests, namely progressive matrices that have always been used and were introduced in 1938. This test involved matching solutions to diagram sketches. Crystallized intelligence (gc), on the other hand reflected the acquisition of skills and knowledge through education in school and daily experience1,9. Crystallized intelligence able to increased or remained the same at the end of adulthood1. This intelligence is measured through a more detailed specialization such as from the aspects of verbal, numerical and memory8.
Ravens Advanced Progressive Matrices (RAPM): IQ is measured by using fluid intelligence (gf) using Ravens Advanced Progressive Matrices (RAPM). It contains 36 questions to test intelligence in general10. The RAPM is very appropriate in the context of the study because the purpose of IQ measurement is to consider the intellectual quotient in general only. Students would be measured by their ability to arrange and match a series of image diagrams in which the last piece of image is missing. Candidates were then requested to select the missing piece of the image from a series of possible answers. A total of eight options were given to students10. The overall IQ score of RAPM is expected to give an overview of the IQ intelligence of polytechnic students. The RAPM score is dichotomous, whether the respondents’ answers are correct (1) or incorrect (0).
Rasch measurement model: Rasch model always applied in many fields, especially in education and psychology to measure the achievement and cognitive assessment11. The uniqueness of the Rasch is the arrangement of the characteristic of the model that based on person abilities and difficulty of the items12. Rasch basically helped in building a scale based on a set of items and concentrated on items and individuals as compared to the test scores13. Equation 1 showed that the model involves two important parameters in testing, that were, (1) item difficulty (observable trait) and (2) respondents’ ability (unobservable trait):
| (1) |
| Pi | = | Probability of getting a correct answer for item i |
| βn | = | Ability parameter for respondent n |
| δI | = | Difficulty parameter of an item I |
Item difficulty parameter is defined as the proportion of the number of students who answer incorrectly. Respondent’s ability parameter is calculated based on the ratio of the number of correct items. Modelling of both parameters is conducted through a procedure called calibration, where responses for each item is transformed into equal interval score call ‘measure’ using natural log (ln). The measure for both parameters is defined in logits unit. Many previous researchers are using Rasch for evaluating items for their instrument14,15 and also conducted other research that based on Rasch model concept16,17.
MATERIALS AND METHODS
Research design: The study was using a quantitative approach to survey research design. This technique was chosen because of their suitability for getting information with less cost and effective.
Sampling technique: The respondents are 150 respondents from one polytechnic in Perak that were selected with 99% confidence level and item calibrations stable within ±0.5 logits18. Convenient sampling technique was used based on the appropriateness of the sample, their free time and willingness to respond to a given inventory. The samples included 40 students from each Department of Civil, Electrical, Mechanical and Commerce. Each department was divided to 20 students from semesters one or two and another 20 were students from semesters five and six.
Return rate: The return rate was good which at the level of 93.75% or 150 students returning back the questionnaire from 160 and it was acceptable19.
Research limitations: The limitations of the study are only involved (a) One polytechnic in Perak, (b) The instrument used was the Ravens Advanced Progressive Matrices (RAPM) in this study, which consists of 36 items, (c) The department involved were only Department of Civil Engineering, Electrical Engineering, Mechanical Engineering and Commerce and (e) The students were from semesters 1, 2, 5 and 6.
Research procedures and administrations: The test was administered for 1 h via monitoring from the polytechnic lecturers. All students' successfully completed the instrument. The responses of all items in the constructs are dichotomous by arranging and matching a series of image diagrams in which the last piece of the image is missing. Eight options were given to students. Then, the score will be calculated by (0 = incorrect and 1 = correct). The researchers have applied for permission to run a study from the centre for research and Innovation and The Director for the Polytechnic. Researchers also get approval from the student affairs officer in obtaining the student population data for sampling calculation. The instrument was distributed with the face to face briefing with respondents. The instruments collection was made by the researchers after the test. The motivation or encouragement is also an important part of the students to respond to the items. Letters of appreciation were given to lecturers and gifts for students as a sign of obligation.
Statistical analysis: The data were inserted using SPSS 19.0 and WINSTEPS version 3.71.0.1 for getting Rasch model findings with the level of significance 0.05.
RESULTS AND DISCUSSION
The tests were discussed in this section are item fit, uni-dimensionality, local independence, item polarity, Gender Differential Item Functioning (GDIF) with the evaluation of reliability and separation index.
| Table 1: | Item fit for RAPM |
MNSQ: Mean square, Zstd: Zstandard, SE: Standard error, PTMEA: Point measure, Corr.: Correlation, Exp: Expected values, *Negative values in measure shows that the items are easier endorsed by the respondents |
|
| Table 2: | Standardized residual variance (in eigenvalue units) |
Item fit: The item fit showed the MNSQ value set in the acceptable range of 0.77-1.3020. Data are matched with Rasch and helped researchers to decide on the suitability of an item. Table 1 showed all 13 items of RAPM that fit the requirement of Rasch. This analysis proved that the items are fit with the model.
Uni-dimensionality: Table 2 showed the information regarding uni-dimensionality of RAPM. The PCA findings showed that raw variance as explained was found to be same with the expected model. The analysis considered the aspects of raw variance explained by measures the noise, ratio of variance explained by measurement and eigenvalues. The raw variance as explained by measures revealed that 27.2% items were represented "true" variance of the person measures or how much the person measures are really spread out along the latent variable and that was fulfilled the requirement of minimum 20%. For eigenvalue of 1.6 was good and showed the value less than 2.0. The 9.1% represented the noise of the items but still acceptable under 10%. Generally, the items successfully fulfilled the requirements of uni-dimensionality and proved that items are measuring IQ in RAPM.
Local independence: The findings showed all pairs of items met the standard correlation of residual values from -0.34-0.16. The results for local independence less than 0.3021-24. This showed that person ability to any item was not associated with the responses of other items for the same construct25. Low correlation does not give any implication on the item and correlation between unequal measure theoretically should be low26. Otherwise, the results were in line with RAPM that had only one construct for measuring IQ (general intelligence or fluid intelligence aspect only). The analysis proved that items are not correlated each other.
Polarity item: The polarity item showed positive value12,27 and more than 0.3028. The values were recorded in the range of 0.36 and 0.55. This value corresponds to a point biserial between 0.30 and 0.60 for a good testing and proved the item measuring the constructs to be measured and all items work towards the measurement of a single construct12. This analysis proved that the items moved in one direction meant by a construct29.
| Table 3: | Gender differential item functioning for 13 items accepted |
DIF: Differential item functioning, *GDIF contrast means the difference of DIF measure between groups. Negative values means DIF measure for G2 is larger than G1 |
|
These qualities demonstrated that all things get together to be characterizing a uni-dimensional develop of estimation.
Reliability and separation index: The person reliability index considered average, which is 0.65 and the items reliability index is estimated high with 0.94. The Cronbach Alpha is 0.68 and nearly 0.70 as suggested30 which indicates the internal consistency of the scale. The item reliability index that more than 0.90 considered as sufficient. For individual reliability, the items can distinguish between one individual to another for a given measured variable12,31. The items’ reliability indicates the possibility to obtain those items at similar locations throughout the scale, even though the same items are given to samples with the same level of ability12. Person separation index for RAPM was recorded at a value of 1.36. The results are coherent based on previous study that stated separation should exceed 1.032, with higher values of separation index with greater distributions of items and persons along a continum. The items separation index was 4.05. The ranges of four to five were considered very good20. The findings indicated that the person separation index was excellent and the items’ separation index is good. The items distributions along the scale able to separate persons based to their abilities and item difficulty31.
Gender Differential Item Functioning (GDIF): Table 3 showed 13 items of RAPM that fulfilled the requirement of GDIF analysis. These items are not favoured to both of the genders. GDIF helps to identify items that indicated early signs of biases when groups of students vary in the same gender of competency 12.
Results showed that DIF value of contrast between +0.5 logits up to -0.5 logits is significant for item testing33,34. Three DIF indicators are t-value between -2 to +2, contrast DIF values -0.5 logits to +0.5 logits and p<0.0512. The GDIF size less than 0.5 are considered unimportant and can be ignored35. This GDIF proved that these items were not favour to any gender.
Items removal: Table 4 shows the items of RAPM that been removed by MNSQ (13 items), Zstd (2 items) and PTMEA (2 items). Most of the items removed because of MNSQ (item fit) which are the main consideration of Rasch. Technically, some items can be considered to be accepted if the item fit ranges are wider.
The GDIF analysis shown in Table 5 removed 6 items and suggested that these items need to be restructuring to ensure that it will not favour to any gender. Technically, the items should be refer back to experts in order to create the pattern of items be more natural in terms on gender preferences.
Item person map (Wright Map): The Wright Map shown in Fig. 1 mentioned the most difficult item to be endorsed was Q26 (+1.52 logit) and the easiest item was Q6 (-1.27 logit). The range of logit between +1.52 logit to -1.27 logit are fulfilled the acceptance range of +3.00 logits to -3.00 logits that considered as good and adequate18,36,37. It can be suggested that items need to be generated between Q26 (measure = +1.52 logit) to Q21 (measure = +1.12 logit), Q13 (measure = +1.04 logit) to Q18 (measure = +0.47 logit), Q18 (measure = +0.47 logit) to Q15 (measure = +0.10 logit), Q20 (measure = -0.11 logit) to Q12 (measure = -0.50 logit), Q7 (measure = -0.57 logit) to Q5 (measure = -0.80 logit), Q5 (measure = -0.80 logit) to Q9 (measure = -1.01 logit) and Q9 (measure = -1.01 logit) to Q6 (measure = -1.27 logit) in order to make sure that RAPM can have sufficient items of IQ to be tested among students with different abilities. Results were reported that no items can be tested for higher person ability with more than +1.52 logit and also for persons with less than -1.27 logit.
| Fig. 1: | Item person map (Wright Map) |
| Table 4: | Items removed by MNSQ, Zstd and PTMEA |
MNSQ: Mean square, Zstd: Zstandard, SE: Standard error, PTMEA: Point measure, Corr.: Correlation, Exp: Expected values, *Negative values in measure shows that the items are easier endorsed by the respondents |
|
| Table 5: | Items removed by GDIF |
| Fig. 2: | Relationship of score of IQ with logits |
The analysis proved that RAPM requires more items to test the person with variety of abilities.
The graph of the relationship between the IQ score with the logit values was shown in Fig. 2. The figure proved that when items difficulty increase (the IQ score decrease), then the logits value will increase, which was in line with Rasch model concept.
CONCLUSION
The sufficiency of IQ items in RAPM as an empirical evidence needs extra concern, especially for polytechnic context. Further study needs to be done in building more IQ items with various difficulties in suiting with diverse abilities. To identify the strengths researchers should apply RAPM and ability of polytechnic students IQ to generate more items. In order to expand the usage of RAPM in a different context, the planning to examine the psychometric properties of different kind of institutions will be beneficial.
SIGNIFICANCE STATEMENT
This study found that the RAMP as an effective method for determining the IQ with different level of intelligence in Malaysia and it would be helpful for the researchers in examining the psychometric properties in different institutions. Thus best theory on it may be arrived at.