本 科 生 毕 业 设 计 (论 文)

题目：Diachronic Content Validity Study on Reading Comprehension Tests of NMET in Shandong Province(2007-2011).

教学单位 外国语学院__ _ _______

姓 名 _刘洁 ________

学 号 _200830701021__________

年 级 _2008级 ________

专 业 _英语 _____

指导教师 _张书奎________________

职 称 _讲师_________________

20xx年 4 月 30日

Contents

Abstract …………. ……………………………………………………………III Chapter 1 Introduction ………………...…………………...………………….1

Chapter 2 Content Validity……………………………………….……….…...1

2.1 Definition of Validity…………………………………………………. 1

2.2 Types of Validity……………………………………………………….2

2.2.1 Face Validity……………………………………………………….2

2.2.2 Content Validity…………………………………………………....2

2.2.3 Criterion Validity…………………………………………………...2

2.2.4 Construct Validity………………………………………………….2

2.3 National Matriculation English Tests…………………………………...2 and Research Framework………….……….3 .….………………3

3.2ResearchFramework………………………….………………………...3

3.2.1 Data Collection……………………………………………………3

3.2.2 Data Analysis……………………………….……..……………....3

Chapter 4 Findings and Discussions ..…………………………………………..4

4.1 Length………………………………………….………………………4

4.2 Speed…………………………………….………….……….……..….5

4.3 New Words………………………………………………….…………5

4.4 Readabilities………………………………………………..........……..6

4.5 Topics………………………………………………………………….7

4.6 Genres…………………………………………………………………8

4.7 Reading Skills…………………………………………………………8

Chapter 5 Conclusions and Suggestions…………………………..…………...9

5.1 Conclusions……………………..………………….…………….…....9

5.2 Suggestions…………………….…………………………….….........10

5.2.1 Suggestions for the Testing Service Centre…………….………10

5.2.2 Recommendations and Implications for English Teaching and

Learning………..………………………………………….……….11

5.3 Limitation of the Study…………………….…………………..…….11

References …………………………………………………….......................12

Acknowledgement…………………………………………………...……....13

Abstract: This paper aims to explore the content validity of the Shandong NMET(National Matriculation English Tests)reading comprehension component based on the framework of task characteristics and related content validity theories, the senior high English teaching syllabus and the Syllabus for NMET test. The results show that reading comprehension parts of 2007-2011 Shandong NMET have well met the requirements of the two syllabi and have got a well content validity in many aspects such as topics, new words and reading skills. However, the author also finds some weaknesses. Based On the findings, this author puts forward some suggestions in order to improve language testing as well as teaching and learning English reading in high schools.

摘要：本文以山东省2007～20xx年高考英语阅读理解测试为研究对象，结合任务特征框架、内容效度的相关研究, 再根据全日制高级中学英语教学大纲( 试验修订版) 和2007- 20xx年的高考英语考试大纲, 对其阅读理解内容效度进行研究分析。 结果表明: 这五年高考英语山东省英语阅读理解试题在选材，生词量，以及考察能力等方面符合考试大纲和教学大纲的要求, 具有较好的内容效度。但同时也发现还存在一些不足，文章对此提出几点建议，希望能对高考英

语阅读测试的命题质量的提高以及今后的高中英语教学有所促进。

Key words: National Matriculation English Tests; Reading Comprehension Tests; Content Validity

关键词：高考英语；阅读理解测试；内容效度

Diachronic Content Validity Study on Reading Comprehension

Tests of NMET in Shangdong Province（2007-2011）.

Chapter 1 Introduction

Now, the National Unified Enrollment Examinations (NUEE) is one of the largest-scale tests in China. Due to that National Matriculation English Tests (NMET) is one of the three major subjects, it is necessary to make well analyses about it. As a high-stakes test, NMET, is used as agents to change or attempt to change the classroom teaching(Qi Luxia,2005：142)．It is widely acknowledged that the central problem of foreign language testing is validity. Reading comprehension is a main language skill. What’s more, it has a significant percentage in NMET. Therefore, the content validity of reading comprehension has a directly influence on the realization of NUEE Function. Currently, scholars do many researches on validation study about TOFEL，IELTS，TEM and CET. But, few are related to the NMET, let alone the English test paper of Shandong province.

Owing to the significant role reading plays, this paper has chosen to analyze the content validity of reading comprehension component of 2007-2011 Shandong NMET to make a clear analysis of it and throw some light on teaching and learning English reading in high schools as well as on the NMET.

Chapter 2 Content Validity

2.1 Definition of Validity

Many scholars give many kinds of definitions about validity. ‘The problem of validity is that of whether a test really measures what it purports to measures’.

Kelly(1927：14)defines. Henning(2001：89)notes，validity usually refers to a given test’s fitness or any of its component parts as a measure of what it is purported to measure．Henning states：‘tests are more or less valid for their purposes；validity is not an all—or-nothing matter’(Alderson，Clapham&Wall，2000：1 70)． In sum, the validity of a test is the extent to which it measures what it is supposed to measure and nothing else．Every test, whether it is a short，informal classroom test or a public examination, should be as valid as the constructor call make it．The test must aim to provide a true measure of the particular skills which it is intended to measure：to the extent that it measures external knowledge and other skills at the same time. It will not be a valid test．

2.2 Types of Validity

2.2.1 Face Validity

is a measure of how representative a research project is ‘at face value,’ and whether it appears to be a good project. Ingram(1 977：1 8)notes face validity as“the test's surface credibility or public acceptability". In generally, if a test seems to cater for other testers, teachers, moderators, and testees, it can be described as having at least face validity. measures what it should measures we can be said that it has face validity．Therefore, directly tests have high face validity, while indirectly tests have low face validity.

2.2.2 Content Validity

is the estimate of how much a measure represents every single element of a construct. Kerlinger(1973:458)states that content validity is a tool to measure the representativeness of the content and the adequacy of the samplings. In other words, it refers to whether the tests the requirements of the syllabus or not. Content validity is an important issue for both language testing and language teaching. Hughes(2000：27)states that‘'the greater a test’s content validity is，the more likely it is to accurately measure what it is supposed to measure．

2.2.3 assesses whether a test reflects a certain set of abilities. Usually, there are two kinds of - and . measures the test against a benchmark test and high correlation indicates that the test has strong criterion validity. While, is a measure of how well a test predicts abilities. It involves testing a group of subjects for a certain construct and then comparing them with results obtained at some point in the future.

2.2.4 Construct Validity defines how well a test or experiment measures up to its claims.

A test designed to measure depression must only measure that particular construct, not closely related ideals such as anxiety or stress. It means that if a test has construct validity, it is capable of measuring certain specific characteristics in accordance with a theory of a language behavior and learning.

2.3 National Matriculation English Tests

The National Unified Enrollment Examinations (NUEE) is one of the largest-scale tests in China. Due to National Matriculation English Tests (NMET) is one of the three major subjects, many authors pay attention on it. There are many articles about it. A list of them is followed.

? Validation Study on 2008-2009 National Matriculation English Test Papers of Three Provinces is written by Yan Jing in 2010.

? An Investigation of Washback Effect of the National Matriculation English Test is written by Hong Xiaoxiang in 2008.

? Content Validity Study on Reading Comprehension Tests of NMET of Chongqing is written by Qiu Jin in 2010.

? A study on the Backwash Effect of the Nmet (shandong Province)reading and Expression is written by Zuo Peilan in 2010.

? An Empirical Study on the Washback of the Grammar Part in the National Matriculation English Test(Shanghai) is written by Zhao Meijuan in 2009.

? Diachronic Study on NMET Reading Comprehension Discourse Characteristics and Question Design is written by Zhang Quan in 2010.

? A Study of Characteristics and Content Validity of Nmet Reading Tests in Shaan Xi Province (2006-2009) is written by Bai Haiyu in 2011

? A Study of Content Validity on Reading Comprehension of Matriculation English Test in Guangdong Province (2004-2010) is written by Zhou Yiqiong in 2011.

There are also many other articles about NMET. However, by carefully analysis author finds that a majority of them regard the NMET as whole to analyze. Of course, there are some articles are about reading comprehension. But few concerns Shangdong province let alone the diachronic research about it. Therefore this paper has chosen to analyze the content validity of reading comprehension component of 2007-2011 Shandong NMET to make a clear analysis of it and throw some light on teaching and learning English reading in high schools as well as on the NMET.

Chapter 3 s and Research Framework

3.1

The author combines the framework of task characteristics and relied content validity study theories proposed by Bachman and Palmer ( 1999 ) , the senior high English teaching syllabus and the syllabus for NMET test.

3.2 Research Framework

3.2.1 Data Collection

The author collects reading comprehension parts of 2007-2011 Shandong NMET, then makes a careful analyses both qualitative and quantitative data which are about length, readability, reading speed, new words, topics and genres. Computers have also been adopted in the data collecting.

3.2.2 Data Analysis

By comparing the collected data with the requirements of the senior high English teaching syllabus and the syllabus for NMET test. The author also invited a classmates and a lecturer to help me when it is difficult for the author to analyze some questions.

Chapter 4 Findings and Discussions

The author combines the framework of task characteristics and relied content validity study theories proposed by Bachman and Palmer (1999), the senior high English teaching syllabus and the Syllabus for NMET test to put forward the objects for testing the content validity (Table1).

4.1 Length

The syllabus for NMET test stipulates that the total length of the passages should be above 1000 words (Dong 2008). Careful counting is shown in the table 2.

(PT: Passage Total; Q: Question; T: Total)

First, it is obviously the total length of the reading tests is above 1000 words. Alderson (2000:256) recommends that the length of each test passage is suitable when it is between 150~350 words. Liu and Han also mentioned that reading

material should be controlled at about 200-500 words(Dong 2008).The table 2 shows that the length of each reading test is between 250-320 words. Therefore, it meets the requirement of scholars.

As for the layout of the articles, it should follow the general principle that from short to long and from easy to difficult. However, this study found that some years the articles’ layout is not very reasonable, such as the 2009 and 2010, which begin with the longest passage of the whole reading test. This may give candidates "slap" feeling, which will affect students’ normal performance.

4.2 Speed

As for reading speed of NMET, two ways can be considered. First is the speed of reading articles, and the second is finishing the reading tests. The syllabus for high school students has specific demands. Senior middle school goals are divided into two levels, and the first level is 50~70 words per minute, then, the second level is 70~80 words per minute. Yang Huizhong and Weir(1998)submit that the proportion of reading articles and answering questions is 1 to 0.75. Syllabus requires that reading comprehension section should be finished by 35 minutes. According to this, reading the article takes up 20 minutes, and answering the questions takes 15 minutes. Based on these, the author calculates the reading speed according to the proportion of twenty to fifteen minutes (WPM1). In addition, the writer also adopts the usual way to calculate t he reading speed, or the total input amount is divided by reading time (WPM2). The two calculations are shown in t he table below:

Through the first calculation, the read speed of the five-year is from54.3-61.0 wpm. Basically，the speed fully meets the curriculum requirements. And the reading speed is well balanced, except for 2008 which is a slightly slower.

As for the second calculation method, generally speaking, reading speed for every paper almost meets the requirement of the teaching syllabus, except for the speed of 2008 which is lower that 50 wpm.

By analyzing the Table 3, the author finds out that they just reach the first level requirement of the first grade ( 50- 70 wpm) , not reach the second level ( 70-80 wpm ) . NMET is a top level test which is used to select the qualified candidates for university. Therefore, reading speed should sometime rise to second level. Based on the analysis, a suggestion may be made that reading speed of NMET needs to be slightly raised.

4.3 New Words

The teaching and testing syllabi(判决理由书的要旨) stipulate clearly that the new words of each reading passage should be less than 3%. The new words and vocabulary beyond the teaching syllabus of each year are shown in the table 4.

(N: New Words A: Average)

From the Table 4, the author finds that the new words rates of 2007-2011 are lower than 3%. What’s more, the new words of each year are stable except the year of 2009. Reading comprehension tests of NMET in Shangdong province gets better control of the amount of new words which matches well in line with the requirements of the syllabus.

4.4 Readability

The readability of the article refers to the level of reading material that is easy to read and understand. The readability of reading comprehension should be appropriate because it can not reflect the actual reading abilities if it is too difficult or too easy.

Nowadays widely used formula was the Flesh, which is first used in 1948 and still in use today. The formula is like this: RE = 206. 835-(0. 846 x NSYLL) - (1. 015 x W/S) (qt d in Alderson 2000: 71). NSYLL refers to the number of syllables per 100 words, and W/S refers to the average number of words per sentence Although, the formula is designed f or the English native learners not for the foreigners, Yang and Weir (1998) put forward that it can be regarded as a reference for the test designers and provide us a basis to build our comparison with a uniform criterion. The formula describes the readability with seven levels from easy to difficult.

(R: Readability; D: Description of difficulty)

Based on the Flesh formula, the readability of reading passages of NMET in Shangdong province is described according to readability yardstick as follows:

(AR: Average Readability)

By analyzing the table shown above the readability of reading comprehension tests of NMET in Shangdong province is from 50-70. According to the Flesh

reference scale, the readability of the reading materials is from fairly difficult to standard. So, a conclusion can be made that it well meets the requirement of the two syllabi. What’s more, it is characterized with stationarity.

4.5 Topics

Alderson (2000) notes that usually good reading tests and good assessment procedures will make sure that readers have been assessed for their ability to make a well known of texts in a range of different topics. The senior high English teaching syllabus and the syllabus for NMET test require that senior middle school graduates can read and understand articles on different topics including science and technology, biography, story, social life and culture, literature and history. Based on these, the topics will be divided into six parts and the collected data will show in the form 7.

(S&C: Social life and Culture; S: Story; E: Education; S&T: Science and Technology; H&G: History and Geography; PE: Personal Experience. )

By analyzing the table 7, the author concludes some characteristics about the topics of reading materials as follows:

First, the selection of topics is extensive. The table 7 shows that almost the six topics are included. By this, it can ensure to test the candidates’ reading abilities more scientifically, objectively and thoroughly.

Second, the selection of topics is comfort to the times. There are some heated topics which can stimulate students’ interest in reading and help to guide the students to focus on social and current affairs.

Third, the selection of topics is realization. Many topics are concerned with the students ' daily life which they are so familiar with. This can make sure the fairness of the test. Just as Bachman (1990: 138) proposes that if the subject matter of input is familiar to some test takers and not others, these individuals may have an unfair advantage, resulting in better performance.

Fourth, the selection of topics is ideology. The table 7 shows that different topics account for different proportions. Among of them story occupies a large proportion. Every story has been carefully selected, and every of them has its ideology, which can make significant influence on the students.

In short, all kinds of topics involved can not only favor students ' reading skills, inspire students ' interest in reading but also can help develop their good qualities. However, the author also finds that there are little topics about History and

Geography, which is also very important for students.

4.6 Genres

People generally divide article genre into narration, exposition and argumentation. And Yang Huizhong &Weir also divide it in the same way. However, The senior high English teaching syllabus and the Syllabus for NMET test require that students also should make a well known of some simple practical writing for example, letters, invitations, notices, notes and some common signs. Based on this, the author goes to divided it into four aspects: narration, argumentation and exposition and practical writing. The result of the collection will show in the form8.

(N: Narration; E: Exposition; A: Argumentation; P: Practical Writing)

First，from Table 8 ,it clearly shows that genre of reading materials is very extensive. Except practical writing, the last three genres all can be found.

Second, narration and exposition occupy a large proportion，which reflect the requirement of the senior high English teaching syllabus and the syllabus for NMET test.

Third, the argumentation has a proportion of 15%. Argumentation, which is characterized by deep stationery plays a very significant role in developing students’ logical reasoning and thinking ability. Practical Writing is also required by the two syllabi, so in the future practical writing should also get its place in the NMET test.

4.7 Reading Skills

Alderson (2000) puts forward that the act of reading is composed of the deployment of a range of separate skills, abilities or strategies. There are six reading skills described in the testing syllabus of NMET (2000). They can be described as follows:

01understanding explicitly stated information 02 making simple judgment and inference 03 Understanding essence theme,

04 inferring the meanings of new words depending on the context 05 Understand the author's intention and attitude 06 understanding the basic structure of passages

According to the six reading skills listed above, the author make a simple analyse as follows:

Table 9 Reading skills coverage of reading tests from 2007 to 2011

Table 9 shows that Shangdong province focus on understanding explicitly stated information, which shares 44%. Specific information refers to the prescribed information that appears in the text. The shill of Understanding the specific information, which belongs to the lower reading skills, is the most basic requirement in reading comprehension. The largest proportion of it shows that it is the basic skill for high school students to understand the specific information and details.

The proportion for the skill of making simple judgment and inference is just after the skill of understanding explicitly stated information. Simple judgment and reasoning is a high level of reading skill which requires the candidates have a well understanding of the passages. Therefore, it can reflect the candidates’ performance more realistically and objectively. Then, the skill of understanding essence theme also occupies 14%. This can well develop the students’ inductive and synthetic analysis abilities.

However, it is obviously that the skills of understanding the author's intention and attitude understanding the basic structure of passages make up a small proportion even none. Discourse structure as an important inherent characteristic of text has a positive effect on improving the accuracy and efficiency of reading. What’s more the two syllabi also have clearly requirement for it. So, the test designers should pay more attention on the two skills.

Chapter 5 Conclusions and Suggestions

5.1 Conclusions

Through the analysis of the collected data of NMET, conclusions can be drawn as followed.

Firstly readability of reading comprehension tests fits the students well. And it is characterized with stationarity.

Secondly, the selection of topics is extensive，realized and comfort to the time. Thirdly, reading materials are selected with appropriate difficulty .Vocabulary and readability are the two most important evidences to prove that most of the items have got a proper difficulty. ,

Lastly，almost all the papers attach more importance to textual understanding and examination of integrated abilities，such as analysis，judgment, inference and induction．

Therefore, the reading tests of NMET from 2007 to 2011 have well met the requirements described by the senior high English teaching syllabus and the syllabus for NMET test.

5.2 Suggestions

5.2.1 Suggestions for the Testing Service Centre

Although reading comprehension part of NMET has a high content validity, the author also finds some shortages. On the basis of the previous quantitative and qualitative analyses of the reading test，the author makes some recommendations in the hope of further improvement and development of the reading tests in the future.

First the arrangement of reading passage should be appropriately.

It should follow the general principle that from short to long and from easy to difficult.

Second, reading speed should be slightly raised.

By analyzing the Table 3, the author finds out that they just reach the first level requirement of the first grade , not reach t he second level, and even the reading speed of 2008 is lower that 50 wpm. In order to select the qualified candidates, reading speed should be slightly raised.

There are little topics about History and Geography. This kind of topic can help spread students’ wings a bit. Therefore, the test designers should select some passages about them.

Fourth, the genres should be diversified.

Practical Writing is also required by the two syllabi. What’s more, this kind of genre can help students in their daily life. And the test has an important influence on the teaching, so in the future practical writing should also get it place in the NMET test.

Fifth, the test of the reading skills should be more balanced.

Discourse structure has a positive effect on improving the accuracy and efficiency of reading. What’s more, the two syllabi also have clearly requirement for it. So, the test designers should pay more attention on the two skills.

However, though the collected data, the skills of understand the author's intention and attitude understanding the basic structure of passages make up a small proportion even none. So it is significant to balance the test of reading skills. Sixth, a title should be involved in each passage.

It is well known that title can show the main content of the passage. In skills and strategies testing it is important that students should know the main content of the passage. Therefore, a title is necessary for every reading passage.

5.2.2 Recommendations and Implications for English Teaching and Learning in high school.

First, the students should strengthen and widen reading.

The topics and genres of NMET became more and more enriched. The topics are closely associated with the real social life，which undoubtedly call stir

students’interest to read carefully. Reading newspapers，reading English story books，reading the simplified English versions，listening to English news on the radio，watching English TV programmers and surfing the internet are the easy way to widen reading.

Second, emphasize training students’reading strategies and reading skills in everyday English teaching．

Teachers should focus on the six skills training in their daily reading. When students finish reading an article, teachers should guide them to get specific information and do some simple inference, as well as analysis discourse structure and understand the author's intention and attitude.

5.3 Limitations of the Study

Although the author has made great efforts in the study of reading comprehension parts of NMET, there still exist some problems that the author cannot solve at present. Firstly, the author has insufficient experience in this research. Secondly, although, the author tries the best to make a full research of the NMET, it is still insufficient to depict the full picture of the reading comprehension parts of NMET. Finally, regardless the limitation of the study, the author hopes that it can boost the improvement and development of the reading tests in the future. At the same time, the author also sincerely hopes that the problems existing in the study can be solved by some other authors.

Reference

[1] Alderson，J. C. ，Clapham，C. ＆ Wall，D. Language Testing Construction and Evaluation［M］. Beijing: Foreign Language Teaching and Research Press，2000.

[2]Alderson，J. C.. Assessing Reading［M］. Cambridge: Cambridge University Press，2000.

[3] Bachman，L. F. ＆ Palmer，A. S. Language Testing in Practice［M］. Shanghai: Shanghai Foreign Language Education Press，1999.

[4] Dong Manxia, Content Validity Study on Reading Comprehension Tests of NMET,2008.

[5]Hughes，A. Testing for Language Teachers［M］.Cambridge: Cambridge University Press，1989.

[6]Qi, Luxia. Stakeholders conflicting aims undermine the washback function of a high-stakes test. Language Testing. 2005.

[7]Weir, C. J. Communicative Language Testing. London: Prentice Hall International ( UK) Ltd. 1990.

[8]Weir, C. J. Understanding and Developing Language Tests. Hemel Hempstead: Prentice Hall International.1993.

[9]教育部. 普通高等学校招生全国统一考试英语考试大纲［M］. 北京：人民教育出版社，2007-2011.

[10]教育部.全日制普通高级中学英语教学大纲（实验修订版）［M］. 北京：人民教育出版社，2000.

[11]杨惠中，Weir，C. J. 大学英语四、六级考试效度研究［M］. 上海：上海外语教育出版社，1998.

Acknowledgements

First, I would like to say that it is my pleasure to spend my past four years in

Southwest University for Nationalities. Second, I am grateful for the help and

encouragement of all the professors and lecturers in the school of Foreign Language, such as Professor Wang Xiangdong, Lecturer Zeng Yuhong, Zhu Ping and Feng Yujuan.

I would like to express my deepest gratitude to my supervisor, Lecturer Zhang Shukui ,for his most valuable guidance，insightful comments and warm-hearted encouragement during the whole writing process of this thesis，without which the completion of this thesis would never have been possible．