Abstract: The present study attempts to shed light on the role of depth of vocabulary knowledge in reading comprehension and its relationship with grammatical knowledge. To this end, a pre-1995 TOEFL and a measure of depth of vocabulary knowledge was administered to 57 EFL learners (17 male and 40 female). The analysis of the results showed that (a) language proficiency influences performance on depth of vocabulary knowledge tests (b) although depth of vocabulary knowledge is significantly related to reading, grammatical knowledge explains the greatest amount of variance in tests takers` performance on reading comprehension tests and (c) knowledge of collocation is related to grammatical knowledge. Having presented the findings in detail, the present study advises language teachers to increase the grammatical knowledge of language learners through diverse means such as focus on form and explicit grammar instruction.
INTRODUCTION
It has long been accepted that vocabulary and grammatical knowledge are instrumental in reading comprehension (Alderson and Urquhert, 1984; Alderson, 2000; Alavi, 2001; Alavi and Kaivanpanah, 2007). With respect to the relationship between depth and breadth of vocabulary knowledge and reading comprehension, less attention is paid to the role of depth (Qian, 2002; Read, 2007). Thus, we intend to address the role of depth of vocabulary knowledge in reading comprehension.
In the past three decades the filed of applied linguistics has witnessed a renaissance in research into the testing of vocabulary (Bachman, 2000; Read, 2000) because it has a determining role in formal and informal reading, listening, speaking and writing and contributes to achievement in the school education (Schmitt, 1999). Therefore, the issue of vocabulary assessment in the second language learning has become one of the great concerns of test developers. However, few have reflected on the problems language teachers and researchers face in developing proper tests for measuring learners’ vocabulary knowledge.
The first difficulty relates to defining word knowledge. Vocabulary is one of the most elusive terms to define. Laufer and Goldstein (2004) refer to earlier definitions of Lexical knowledge by Nation (1990, 2001), Richards (1976) and Ringbom (1987), who describe lexical knowledge as the sum of interrelated subknolwedges: knowledge of spoken and written form, morphological knowledge, collocational and grammatical knowledge, connotative and associational knowledge and the knowledge of social or other constraints to be observed in the use of words.
As Qian and Schedl (2004) argue there is a clear tendency that L2 vocabulary researchers no longer view vocabulary knowledge as a single dimension. Instead, there is a growing consensus that vocabulary knowledge should be regarded as a multidimensional construct. Thus, researchers Greidanus and Nienhuis (2001), Read (2000), Vermmer (2001) and Wolter (2001) have tended to view vocabulary knowledge as consisting of two dimensions of breadth (how many words are known) and depth (how well is a word known). The vocabulary depth measures were proposed to account for the inadequacy of tests of breadth in measuring the quality of test-takers’ knowledge; in these tests a single lexical item is measured on several components of knowledge (Laufer and Goldstein, 2004).
Researchers have also focused on how well test takers know a word. According to Read (2004):
Depth of knowledge focuses on the idea that for useful higher-frequency words learners need to have more than just a superficial understanding of the meaning; they should develop a rich and specific meaning representation as well as knowledge of the word’s formal features, syntactic functioning, collocational possibilities, register characteristics and so on (pp: 153).
As mentioned before vocabulary is regarded as the sum of interrelated subknolwedges. Of particular interest is the knowledge of polysemy, synonymy and collocation that are part of the depth dimension. Polysemy refers to fact that many words have multiple meanings. Synonymy and collocation are both aspects of lexical cohesion. Synonymy refers to the relation between words which have nearly the same meaning. According to Halliday and Hasan (1976) collocation is an aspect of lexical cohesion which embraces a relationship between lexical items that regularly co-occur.
In a study concerning construct validation, Schmitt (1999) examined TOEFL vocabulary tests in order to investigate the relationship between TOEFL vocabulary, items and meaning association, collocation and word-class knowledge. The analysis of the results indicated that TOEFL vocabulary provided the researchers with a limited amount of information. Performance on TOEFL did not indicate test takers’ association, word-class and collocation knowledge of a specific word. Therefore, he called for a deeper analysis of what vocabulary test items actually measure.
The second problem in developing proper vocabulary tests relates to the relationship between vocabulary knowledge and other related skills. Psychologists and researchers who have addressed various aspects of reading skill for more than two decades (Alderson, 1990a, b; Alderson and Urquhart, 1984; Johnston, 1983; Alavi, 2001; Martinez-Lang, 1995; Grabe, 1991; Gascoigne, 2005; Brantmeier, 2002; Block, 1986), have not reached an agreement regarding the nature of the skill, or skills that are involved in reading comprehension. It seems that reading researchers propose one of the two views; while some regard reading as a unitary, holistic and indivisible skill that cannot be divided into different sub-skills (Alderson, 1990a, b; Lunzer et al., 1979; Rost, 1993), there are other researchers who argue that reading skill consists of various sub-skills (Davis, 1968; Munby, 1979) including vocabulary and grammar. Hence, researchers have been encouraged to investigate the relationship between vocabulary knowledge and reading comprehension on the one hand and vocabulary and grammatical knowledge on the other hand.
The significant role of vocabulary knowledge in reading comprehension has been well recognized in first language (L1) situations and this has appeared to be true of second language (L2) settings as well. Grabe (1991) asserts that one of the most important areas of research for reading comprehension is research in vocabulary development; Grabe and Stoller (2001) emphasize the role of large vocabulary knowledge in reading comprehension. They stress that students need to recognize a large number of words automatically to become fluent readers and advise teachers to explicitly teach the key words of the text to be read in class to help students better understand the texts they read.
These and similar studies (Zhang and Annual, 2008) show that L2 vocabulary research has been mainly concerned with measuring vocabulary size because it is easier to develop measures of size than depth (Qian, 2002). At the same time, there have only been a handful of studies investigating the relationship between depth of knowledge and reading comprehension (Qian, 2002; Wesche and Paribakht, 1996). One of the recent studies was carried out by Qian (2002). His study confirmed that depth of knowledge is as important as vocabulary breadth in predicting performance in the reading section of TOEFL. In a more recent study, Qian and Schedl (2004), empirically evaluated an in-depth vocabulary knowledge measure in order to find out whether the measure could provide a basis for designing appropriate and useful item types for assessing test takers’ reading comprehension. It was found that depth of vocabulary knowledge and TOEFL had the same difficulty level and both had similar relationships with reading comprehension tests. It was also found that compared to existing TOEFL vocabulary measures, the new measure had a similar difficulty level and provided a similar amount of prediction of ESL test-takers’ reading performance.
Studies investigating the relationship between grammatical knowledge and reading comprehension have aimed at finding out whether knowledge of the relationship between the constituents of the sentence aids comprehension. For example, in a recent study by Gascoigne (2005) the relationship between reading comprehension and grammatical competence were investigated. The results indicated that there was no significant difference between performance on the reading and grammar tasks.
In another study Alavi (2001) investigated the relationship between grammar knowledge and FCE and TOEFL reading. The performance of 2059 university students from different educational backgrounds was compared across different sections of a proficiency test. The analysis of the results indicated that performance on written expression items was more related to TOEFL reading, implying that grammar knowledge plays a significant role in reading comprehension.
Thus, if we agree that grammatical knowledge is intrinsically involved in reading comprehension, its relationship with vocabulary knowledge that influences reading comprehension should be specified; there are some theoretical grounds to assume that the two components are interrelated.
The present study is motivated by the study of Qian and Schedl (2004), who raise some questions regarding the generalizablity of their study to other EFL contexts, where, the exposure to English is limited to a few hours a week in classroom and development of depth of Vocabulary Knowledge (VK) is not as important a goal as the development of breadth of VK for students. Thus, of particular interest is finding out whether the predictive power of Depth of Vocabulary Knowledge Test (DVKT) developed by Qian and Schedl (2004) and vocabulary section of TOEFL for reading comprehension is the same in other EFL and ESL contexts. Moreover, the DVKT has basically two parts: one related to polysemy and synonymy and the other related to collocation. Therefore, one may want to know which one better predicts performance on a test of reading comprehension. Hence, the present study was carried out with the aim of shedding light on the relation of depth of VK with reading comprehension as far as variation in proficiency levels is concerned; in other words, it seems that depth of VK might play a less conspicuous role in reading comprehension at lower proficiency levels because the depth of VK is not well developed in such levels. In addition, the relative importance of depth of VK in predicting reading in EFL contexts is still ripe for investigation. Based on the above argumentation, the present study addresses the following research questions:
| • | Is there any difference in test takers’ performance on DVKT and vocabulary section of TOEFL? |
| • | Which of the three vocabulary measures i.e., MC polysemy, MC collocation, or MC vocabulary section of TOEFL and grammar section of TOEFL better predicts performance on reading comprehension? |
| • | Is knowledge of collocation related to performance on the grammar section of TOEFL? |
MATERIALS AND METHODS
Participants: The participants were 57 (17 = male and 40 = female) students of English as a foreign language. They had studied (or were studying) English as a part of the national curriculum. However, they had enrolled in language classes in order to compensate for the deficiency of English education at high school. Their age ranged from 13 to 28; they represented different proficiency groups as evidenced by their scores on the TOEFL test.
Instruments: The following instruments were used to elicit data:
(1). An English language proficiency test: A TOEFL test consisting of 40 grammar items, 30 vocabulary items and 30 reading comprehension items was administered to participants in order to measure their level of proficiency in English; the test took 75 min to complete.
The grammar section had 40 multiple choice questions of which 15 were incomplete sentences for which the test-takers had to choose the one word or phrase that completed the sentence and 25 were sentences with four underlined words or phrases; the test-takers had to choose the one word or phrase that needed to be changed in order for the sentence to be correct. Twenty five minutes was allotted for the completion of this sub-section.
The TOEFL reading comprehension subtest was selected as the criterion measure of reading comprehension. The subtest contained five passages related to general academic materials at introductory levels, including biology, astronomy, chemistry, art history and biography. Multiple choice questions followed each passage. The questions aimed at finding out whether a basic comprehension of the texts had taken place. Produced before July 1995, this subtest did not contain vocabulary items in the comprehension questions. As such, the questions were judged appropriate for the purpose of measuring reading comprehension.
TOEFL vocabulary section consisted of 30 vocabulary tests that were selected from the earlier versions of TOEFL. The test was a criterion against which other vocabulary measures were compared. The tests were developed for TOEFL administrations before July 1995, when there was a separate section on vocabulary within the Reading Comprehension section. All the items were multiple choice type and each target word was underlined within a sentence context. For example:
Becoming a parent is a major lifestyle change that can be very rewarding yet very stressful:
| • | Satisfying |
| • | Sacrificing |
| • | Responding |
| • | Respecting |
As far as Read’s (2000) framework for vocabulary assessment is concerned, the vocabulary section of pre-1995 TOEFL is a discrete measure of vocabulary and is tested by a separate subsection of the test which means that for the test developers the construct of vocabulary is separate from other constructs; the measure is selective rather than comprehensive which means specific words are the focus of assessment and finally, the test is more context independent than dependent: when a vocabulary test item is context dependent, it implies that context should help inferencing the meaning of a word , not the mere superficial co-text. Regarding vocabulary items of this test, Read (2000) claims that in such tests, apparently, context is of limited, if of no, help in guessing the meaning of the word. As for the construct, since person characteristics are specified independent of context, the test can well fall on the trait category.
(2)Depth of vocabulary knowledge test developed by Qian and Schedl (2004): Each item in the test consisted of one word in italics and two boxes each of which contained four words. The test asked participants to look at the words in italics and choose any number of the words which were similar in meaning from A, B, C, or D and then choose any number of the words from E, F, G or H which could be used after the word in italics in a phrase or a sentence. The test takers were informed that they could select only 4 options from all 8 available to show the degree of their knowledge.
Each DVKT item consisted of one stimulus word, which was an adjective and two boxes, each containing four words. Among the four words in the left box, one to three words could be synonymous to one aspect of, or the whole meaning of, the stimulus word, whereas among the four words in the right box there could be one to three words that collocated with the stimulus word. The instruction sheet for the test taker explained that there were always four correct answers in each item. However, these answers were not evenly spread. Three situations were possible: (a) the left and right boxes both contain two correct answers; (b) the left box contains one correct answer and the right box contains three correct answers and (c) the left box contains three correct answers and the right box contains only one correct answer. This arrangement effectively reduced the chances of guessing.
(3)For example, for the word Sound they had the following options:
Sound: (A. logical B. healthy C. bold D. solid) (E. snow F. temperature G. sleep D. dance).
In this item sound is synonymous with logical, healthy and solid. More over, it collocates with sleep, so, the correct choices are A, B, D and G.
According to Read’s (2000) framework, DVKT is also a discrete measure of vocabulary knowledge and is tested separately i.e., it is not part of a section of another test which implies that the construct it measures i.e., vocabulary, is separate from other constructs; the measure is selective rather than comprehensive which means specific words e.g., adjectives are the focus of assessment and, finally, the test is context independent. Since person characteristics are specified independent of context, the test falls on the trait category.
Scoring: Participants received a point for each correctly answered item; in scoring DVKT, each word correctly chosen was awarded one point. Test takers were not penalized for providing incorrect answers. The maximum possible score was 160 for the total test. The maximum scores they received were 81 for the 40 items for collocation, 79 for the 40 item for polysemy and synonymy.
The maximum score for TOEFL was 100: 40 grammar, 30 vocabulary and 30 reading comprehension.
Procedure: The tests were administered to 57 EFL learners in pencil-and-paper format. The participants received all instruments in one session. First, they took the two-part vocabulary test; they were informed that they can complete the test in 30 min; then, after a 15 min break and refreshment, they completed the TOEFL test in 75 min. The test scores for all the subparts of the general proficiency and each of the vocabulary tests were calculated.
RESULTS
To investigate the impact of level of language proficiency on the performance on the vocabulary measures, on the basis of the mean score (33.35) and standard deviation (10.70) on TOEFL, participants were divided into three groups: 23 students whose scores were in the range of half a standard deviation below and above mean, i.e., those who scored in the range of 28.00 to 38.00, were regarded as intermediate; 19 participants whose scores were half a standard deviation below the mean, i.e., those scoring below 28.00, were considered low and 15 test takers who obtained scores half a standard deviation above the mean, i.e., those scoring above 38.00, were judged as upper-intermediate. Table 1 reports the descriptive statistics from the data analysis for each test.
To find out whether there is a significant difference in test takers’ performance from different proficiency levels on the two parts of DVKT and vocabulary section of TOEFL, a repeated measures ANOVA was run. Since the number of the items in the two formats was not the same, the scores were standardized and converted into a scale of 50.
| Table 1: | Descriptive statistics of tests takers’ performances in different parts of the TOELF tests and measures of vocabulary breadth and depth |
| Table 2: | Tests of within-subjects effects for the three vocabulary tests |
The results, as presented in Table 2, point to a significant difference in the performance of the participants in the three measures of vocabulary knowledge. This indicates that the impact of the level of proficiency was significant. The interaction between collocation part of DVKT and level of language proficiency was not significant; therefore, it can be argued that irrespective of the level of language proficiency, all participants performed better on the collocation section of DVKT.
To see where the differences lie, a post hoc Scheffe test was run. As seen in Table 3, the differences lie in performance of lower intermediate and upper intermediate, lower intermediate and intermediate and intermediate and upper intermediate test takers.
The next research question attempted to identify the variables i.e., synonymy, polysemy, collocation, TOEFL vocabulary, or grammar that could better predict performance on reading comprehension tests. To answer this question a regression analysis was run.
The results of regression analysis in Table 4 indicate that among predicting variables i.e., vocabulary section of TOEFL, synonymy and polysemy, collocation and grammatical knowledge, grammatical knowledge accounts for the greatest amount of variation in reading performance. The standardized Beta coefficient in Table 4 gives a measure of the contribution of each of the variables to the performance on reading comprehension test. Therefore, the larger the value of Beta coefficient, the greater the predictive power and contribution of the variable. The t-value of grammatical knowledge as a predictor variable is meaningful at p<0.000 level implying that the identified variable has a significant impact on the reading comprehension.
The results in Table 4, hence, indicate that grammatical knowledge and an awareness of the syntactic information contributes more to reading comprehension. The beta coefficient of 4.48 explains that grammatical knowledge is a stronger predictor than the depth of vocabulary knowledge in predicting reading comprehension (4.48 vs. 0.04, 0.39 and 0.29, respectively).
This finding is in line with earlier findings that indicate syntactic behavior is more related to reading comprehension than vocabulary knowledge (Shiotsu and Weir, 2007; Alderson, 2000; Alavi, 2001).
| Table 3: | Multiple comparisons Scheffe test |
| Based on observed means. *The mean difference is significant at the 0.05 level | |
| Table 4: | Regression coefficients for predicting performance on reading comprehension test |
| Dependent variable: S_READ | |
| Table 5: | Correlations between knowledge of collocation, synonymy, polysemy and grammar |
To answer the third research question that sought to find out whether knowledge of collocation is related to performance on the grammar section, the scores on the tests were correlated. The results as presented in Table 5 confirm that the knowledge of collocation, synonymy, polysemy and grammar are significantly correlated. However, the findings also show that the strength of association between knowledge of vocabulary as measured by conventional TOEFL type test and grammar is more than that of the knowledge of collocation and synonymy, polysemy and grammar. The results also indicate that the relationship between reading and grammatical knowledge is stronger than the link between TOEFL vocabulary and reading comprehension. These findings point to the age old problem of vocabulary tests: researchers have not been able to demonstrate that vocabulary tests, especially the decontextulaised ones of the TOEFL type, measure vocabulary knowledge and are independent of the knowledge of grammar. This finding, as Shiotsu and Weir (2007) argue highlight the fact that the test instruments are not reflective of the constructs they measure; as they argue, in several occasions, there is a great overlap between tests of grammar and tests of vocabulary.
The stronger correlation between knowledge of TOEFL vocabulary and grammar on the one hand and the close link between knowledge of vocabulary as measured by TOEFL test and reading comprehension on the other hand motivates us to conclude that vocabulary of TOEFL section is sensitive to both vocabulary knowledge and reading comprehension and addresses both knowledge sources. This might make us more cautious in regarding Pre-1995 TOEFL vocabulary section as being a context independent measure; in other words, pre-1995 TOEFL vocabulary items seem to be more context dependent and hence, this contradicts Read’s (2000) assertion that the items are context independent and the surrounding co-text dose not provide clues for choosing the correct answer.
Further, it was found that relationship between polysemy and synonymy and reading is stronger than the relationship between collocation with reading (0.387> 0.175). More research is needed to add to the understanding of this relationship.
DISCUSSION
The findings of the present study indicate that the test takers’ level of proficiency significantly impacts their performance on DVKT. Test takers with lower proficiency did not have an adequate knowledge to take depth of VK tests because this knowledge is incremental in nature and develops as the proficiency of the learners grows by being exposed to language for longer periods of time. This finding is consistent with several studies that have demonstrated performance on any language test is influenced by linguistic knowledge of test takers. As Alderson and Urquhart (1984) argue a certain level of proficiency or competency in the language should be achieved to allow the learners to take advantage of their knowledge, otherwise learners will not be able to transfer reading skills, strategies and prior knowledge to second language reading.
The comparison of the performance of the test takers on DVKT and TOEFL vocabulary section implies that there is a significant difference in test takers’ performance on the two tests. Test takers performed better on the two parts of vocabulary measures of DVKT rather than vocabulary section of TOEFL test. This is hard to explain and further qualitative research is needed to explain the results.
The results also indicated that the relationship between knowledge of collocation and synonymy, polysemy and grammatical knowledge is significant; this finding lends support to Lewis (2000, as cited in Westfall and Weber, 2005) who argues for inseparability of grammar and lexis and the relationship between collocation and grammar.
The results also show that the strength of association between knowledge of vocabulary as measured by conventional TOEFL type tests and grammar is more than the strength of the relationship between knowledge of collocation and polysemy on the one hand and knowledge of grammar on the other hand. This finding might relate to the inadequacy of the tests claiming to measure a certain construct. The high correlations between knowledge of vocabulary as measured by conventional TOEFL type test and grammar as Shiutsu and Weir (2007) have also noted raises question regarding the (in)divisibility of the two components. It is quite possible that the developments of these putatively distinct elements of linguistic knowledge overlap. Thus, this finding encourages us to doubt claims regarding the independence of the construct of vocabulary knowledge; it seems that in the majority of cases the tests of vocabulary knowledge measure grammatical knowledge as well.
Since the correlation between variables was high, it was decided that variation in one of them could be explained by variation in the others. Therefore, the researchers sought to identify the variable that could account for the greatest amount of variation in reading comprehension. The premise underlying this question is that vocabulary and grammar as components of linguistic proficiency are involved in reading comprehension. Given the earlier research findings concerning the role of vocabulary and grammar in reading comprehension are inconsistent (Shiotsu and Weir, 2007), we sought to address this question once again and find out which one has a greater impact on reading comprehension. The scores of test takers on each vocabulary test and grammar subsection of TOEFL were correlated with their reading comprehension scores. The results revealed that grammar accounts for greater variation in reading comprehension than vocabulary knowledge. This finding of the present study is, therefore, consistent with those of Alavi (2001), who found that grammar is more influential in reading than vocabulary knowledge. On the basis of the findings it may be argued that the strong relationships between reading comprehension and grammar is related to the type of mental processes involved in test performance; in both tests, test takers need to focus on intertextual ties and logical relationships between sentences and the way these relationships influence meaning (pp: 121).
It was also found that grammar accounts for the greatest amount of variation in reading performance than vocabulary section of TOEFL, vocabulary synonymy and polysemy and vocabulary collocation. This is acceptable on the grounds that grammatical knowledge and lexical knowledge are two critical aspects of linguistic competence that seem to account for a better reading performance. This is echoed by Shiotsu and Weir (2007), who reported that syntactic knowledge is superior to vocabulary knowledge in predicting performance on reading comprehension. As they argue “syntactic knowledge remains one of the deciding factors in the performance on texts reading comprehension [especially] for learners up to certain level (pp: 121).
Implications: The findings of the present study suggest that language learners who have not reached a certain proficiency level experience difficulties in taking DVKT tests and comprehension of the written texts. Therefore, language teachers are recommended to help learners take advantage of their linguistic and lexical knowledge in comprehending the texts.
On the basis of the findings, it may be argued that grammatical knowledge predicts better comprehension and may be used as an indicator of success in reading. The explicit knowledge seems to help them realize the relationship between sentences (Alavi and Kaivanpanah, 2007). Therefore, if language teachers aim at helping students read better and comprehend faster, they are advised to increase the grammatical knowledge through diverse means such as focus on form and explicit instruction. However, this does not imply that grammatical knowledge is the only and the most important factor influencing reading; the results only encourage teachers focus on grammar, because as this research indicates successful reading is also influenced by vocabulary knowledge. Hence, the responsibility of language teachers in EFL contexts is to help students expand the number of words they know while keeping in mind that in addition to helping students improve their lexical and grammatical knowledge, they should equip learners with reading strategies.