INSTITUTE OF EDUCATIONAL SCIENCES
PhD PROGRAM IN ENGLISH LANGUAGE EDUCATION
PhD Thesis Defense
ABSTRACT
RATER DISCREPANCY RESOLUTION IN SECOND LANGUAGE WRITING ASSESSMENT: AN ANALYSIS OF RATER NEGOTIATIONS
by
Ece Sevgi
Date & Time: Thursday, June 7th 2018, 10.00
Place: Faculty of Education, Room 530
All interested are cordially invited.
The purpose of this study is to analyze rater negotiations for discrepancy resolution to investigate: a) the rater moves used in rater negotiations, b) functional parts of rater negotiations and specific rater moves within them, c) how rater tendencies affect the scoring decisions of rater pairs, and d) what raters report on the possible causes of discrepancy. Thirty non-native English teachers participated in the study. Data came from audio-recorded rater negotiations for discrepancy resolution following a B2 level module exit exam at the language school of an English-medium university in Turkey, and it was transcribed and coded by the researcher and co-researcher. The Rater Negotiation Scheme (RNS) was developed at the end of the recursive coding process, and it was validated through field testing in authentic settings. The validation process of the RNS consisted of two phases: a) training of the participants for using the RNS to code transcribed rater negotiations through exercises, and b) asking the participants to code five purposefully sampled rater dialogues by using the RNS. At the end of the field testing, the RNS was validated with 89% participant coding-key agreement. The RNS is proposed both as a research and instructional tool for the analysis of rater negotiations. Through the use of the rater moves listed in the RNS, the study also investigated the stages of rater negotiations and the rater moves within them. Three functional parts of rater negotiations namely the opening stage, the discussion stage, and the concluding stage were identified, and the types of moves used at each stage were listed. The rater negotiations were also analyzed in this study to investigate how rater tendencies towards accuracy vs. fluency orientation affect the scoring decisions of rater pairs. The results showed that a majority of the raters were fluency oriented in the examined cases. However, for the cases where linguistic features of the test taker essay frequently reported to be problematic, an overriding effect of accuracy was observed, and the winners of these cases were always the accuracy-oriented raters. The study also investigated the raters’ reports on factors that cause rater discrepancy, and presented a list of fifteen discrepancy creating factors. The results obtained throughout this study had significant implications for test developers, test administrators, raters, and rater training.
Key words: writing assessment, rater negotiations, discrepancy resolution, stages of rater negotiations, rater effects, causes of discrepancy in writing assessment