Document Type : Research Paper
Authors
1 West Tehran Branch, Islamic Azad University
2 Sharif University of Technology
Abstract
Conveying a strong message within a language stems from not only a linguistically appropriate utterance but also a pragmatically appropriate discourse. Broadly considering various facets of pragmatics, pragmatic assessment has not been potentially brought into perspective. To address this discourse gap, this study, guided by the principles of mixed-method design, pursued three purposes: to inspect the matches and mismatches, to explore rating variations, and to assess the rater consistency between the holistic and analytic rating methods of disagreement speech acts in L2 by non-native English teachers. As a result, 12 different pragmatic situations for disagreement DCTs accompanied by EFL learners' responses to each situation were rated by 50 non-native English teachers. Initially, they were asked to rate it holistically, incorporating both ratings and providing comments. The content analysis of raters' comments indicated sixteen disagreement criteria. The descriptive statistics also revealed variations across different situations. Moreover, the teachers were asked to rate it analytically based on the assessment rubrics adopted from Ishihara and Cohen (2010). The findings of intra-class correlations implied that respondents were more consistent in analytic rating. Moreover, the results indicated that there was a convergence between the two rating methods suggesting that the raters adopted the same level of leniency and severity in rating. Overall, the results accentuated the significance of pragmalinguistic and sociopragmatic aspects of language for EFL raters. Finally, the results of the present study place a premium on the importance of pragmatic assessment training as well as cultural awareness.
Keywords
Alcon Soler, E, & Martinez Flor, A. (Eds.), (2008). Investigating pragmatics in foreign language learning, teaching, and testing. Clevedon: Multilingual Matters.
Alemi, M. (2012). Patterns in interlanguage pragmatic rating: Effects of rater training, intercultural Proficiency, and self-assessment.Unpublished Ph.D. dissertation, Allameh Tabataba’i University,Tehran, Iran.
Alemi, M., Eslami, Z. R. & Rezanejad, A. (2014). Rating EFL learners’ interlanguage pragmatic competence by non-native English speaking teachers. Procedia-Social and Behavioral Sciences, 98, 171-174.
Alemi, M., Khanlarzadeh, N. (2015). Rating EFL Learners' Production of Request and Complaint by Native and Nonnative English Speaking Teachers. (Unpublished master's thesis). The Sharif University of Technology, Tehran, Iran.
Alemi, M., & Tajeddin, Z. (2013). Pragmatic rating of L2 refusal: Criteria of native and non-native English teachers. TESL Canada Journal, 30, 63–81.
Alemi, M., & Tajeddin, Z. (2014). Pragmatic rating of L2 refusal: Criteria of native and non- native English teachers. TESL Canada Journal, 30, 63-81.
Austin, J. L. (1975). How to do things with words. Oxford: Oxford University Press.
Bachman, L. F. (1990). Fundamental considerations in language testing. New York: Oxford University Press.
Bachman, L. F., Lynch, B. K., & Mason, M. (1995). Investigating variability on tasks and rater judgment in a performance test of foreign language speaking. Language Testing, 12 (2), 238-257.
Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice. Oxford: Oxford University Press.
Beebe, L. M., & Takahashi, T. (1989). Sociolinguistic variation in face-threatening speech acts. New York: Springer.
Chastisement and disagreement. In M. R. Eisenstein (Ed.), The dynamic interlanguage: Empirical studies in second language variation (pp. 199-218). New York: Plenum Press.
Behnam, B., & Niroomand, M. (2011). An investigation of Iranian EFL learners’ use of politeness strategies and power relations in disagreement across different proficiency levels. English Language Teaching, 4(4), 204-220. http://dx.doi.org/10.5539/elt.v4n4p204
Blum-Kulka, S., & Olshtain, E. (1984). Requests and apologies: A cross-cultural study of speech act realization patterns (CCSARP). Applied Linguistics, 5(3), 196-213.
Brown, J. D. (2001). Pragmatics tests: Different purposes, Different tests. In K. R. Rose & G. Kasper (Eds.), Pragmatics in language teaching. New York: Cambridge University Press.
Brown, P., & Levinson, S. (1987). Politeness: Some universals in language usage. Cambridge: Cambridge University Press.
Carr, N. T. (2000). A Comparison of the Effects of Analytic and Holistic Rating Scale Types in the Context of Composition Tests. Issues in Applied Linguistics, 11(2), 207-229.
Cohen, A. D. (2008). Teaching and assessing L2 pragmatics: What can we expect from learners? Language Teaching, 41, 213-235.
Creswell, J. W., & Clark, V. L. P. (2011). Designing and conducting mixed methods research (2nd Ed.). California: Sage.
Douglas, D., & Smith, J. (1997). Theoretical underpinnings of the Test of Spoken English Revision Project (TOEFL Monograph Series Report No. 9).Princeton, New Jersey: Educational Testing Service.
Eslami, A., Jafari, D. S., & Mehregan, M. (2012). How do you react to the breakdown after it happens? Do you complain about it?: A contrastive study on the complaint behavior in American English and Persian. Procedia-Social and Behavioral Sciences, 47, 34-40.
Eslami, Z.R., & Eslami-Rasekh, A. (2008). Enhancing the pragmatic competence of non-native English-speaking teacher candidates (NNESTCs) in an EFL context. In E. Alcón Soler & A. Martínez-Flor (Eds.), Investigating Pragmatics in Foreign Language Learning, Teaching and Testing (pp. 178-197). Great Britain: Cromwell Press.
Fraser, B. (1981). On apologizing. In F. Coulmas (Ed.), Conversational Routine: Explorations in standardized communication situations and pre-patterned speech (pp. 259–271). New York: Mouton.
Holmes, J. (1990). Apologies in New Zealand English. Language in Society, 19(2), 155–199.
Hudson, T., Detmer, E., & Brown, J. D. (1995). Developing prototypic measures of cross-cultural pragmatics. (Tech. Rep. No. 7). Honolulu: the University of Hawai'i at Manoa, Second Language Teaching and Curriculum Center.
Hughes, A. (1989). Testing for language teachers. Cambridge Cambridge University Press.
Ishihara, N., & Cohen, A. D. (2010). Teaching and learning pragmatics: Where language and culture meet. Harlow, UK: Pearson Education.
Ishihara, N. (2013). Teacher-based assessment of L2 Japanese pragmatics: Classroom applications. In S. Ross & G. Kasper (Eds.), Assessing second language pragmatics (pp. 124–148). Basingstoke, UK: Palgrave Macmillan.
Jalilifar, A. (2009). Request strategies: a cross-sectional study of Iranian EFL learners and Australian native speakers. English Language Teaching, 2(1), 46-61.
Jianda, L. (2006b). Assessing EFL Learners' Interlanguage Pragmatic Knowledge: Implications for Testers and Learners. Reflections on English Language Teaching 5 (1), 1-22.
Kasper, G. (1997). The role of pragmatics in language teaching education. In K. Bardovi-Harlig and B. Hartford (Eds.) Beyond Methods: Components of Second Language Teacher Education (pp. 113-136). New York: McGraw-Hill.
Kasper, G. (1998). Interlanguage pragmatics. In H. Byrnes (Ed.), Learning foreign and second languages: Perspectives in research and scholarship (pp. 183-208). New York: The Modern Language Association of America.
Klein, S. P., Stecher, B. M., Shavelson, R. J., McCaffrey, D., Ormseth, T., Bell, R. M., & Othman, A. R. (1998). Analytic versus holistic scoring of science performance tasks. Applied Measurement in Education, 11(2), 121-137.
Knoch, U. (2007) Diagnostic writing assessment. The development and validation of a writing scale (Unpublished doctoral dissertation). The University of Auckland, New Zealand. Retrieved from https://researchspace.auckland.ac.nz
Kreutel, K. (2007). I'm Not Agree with You. ESL Learners' Expressions of Disagreement. TESL-EJ, 11(3), 1-35.
Liu, J. (2007). Developing a pragmatic test for Chinese EFL learners. Language Testing, 24(3), 391-415.
Liu, J., & Xie, L. (2014). Examining rater effects in a WDCT pragmatics test. Iranian Journal of Language Testing, 4(1), 50–65.
Locastro, V. (1986). Yes, I agree with you, but…: Agreement and disagreement in Japanese and American English. Paper presented at the JapanAssociation Language Teachers' International Conference on Language Teaching andLearning, Hamamatsu, Japan.
Locher, M. (2004). Power and Politeness in action: Disagreements in oral communication. Berlin: Mouton de Gruyter.
Murphy, B., & Neu, J. (1996). My grade is too low: The speech act set of complaining. In S. M. Gass, & J. Neu (Eds.), Speech acts across cultures: Challenges to communication in a second language (pp. 191-216). Berlin: Mouton de Gruyter.
Olshtain, S., & Cohen, A. D. (1983). Apology: A speech act set. In N. Wolfson & E. Judd (Eds.), Sociolinguistics and language acquisition (pp. 18–35). Rowley, MA: Newbury House.
Paramasivam, S. (2007). Managing disagreement while managing not to disagree: Polite disagreement in negotiation discourse. Journal of Intercultural Communication Research, 36(2), 91-116.
Pattrawut, C. (2014). A Cross-Cultural Pragmatic Study: Politeness Strategies and Realizations of the Strategies Used to Perform Student-Lecturer Multiple Disagreements by Native Speakers of Thai and English. Journal of Educational and Social Research, 4(1), 147-158.
Pearson, E. (1986). Agreement/disagreement: An example of the results of discourse analysis applied to the oral English classroom. International Review of Applied Linguistics, 74(1), 47-61.
Roever, C. (2008). Rater, item, and candidate effects in discourse completion tests: A FACETS approach. In E. Alcon Soler & A. Martinez-Flor (Eds.), Investigating pragmatics in foreign language learning, teaching, and testing (pp. 249–266). Clevedon, UK: Multilingual Matters.
Rose, K. R. & Kasper, G. (Eds.). (2001). Pragmatics in language teaching. New York: Cambridge University Press.
Samar, R. G., Abaszadeh, A., & Pourmohamadi, F. (2013). Investigating Disagreement through a Context-Specific Approach: A Case of Iranian L2 Speaker. Applied Research on the English Language, 3(1), 87100.
Scollon, R., & Scollon, S.W. (2001). 27 Discourse and Intercultural Communication. In D. Schiffrin, D. Tannen, & H. Hamilton (Eds.), The handbook of discourse analysis, (pp. 538-547). Oxford: Blackwell.
Stalpers, J. (1995). The expression of disagreement. In K. Ehrlich & J. Wagner (Eds.), The discourse of business negotiation (pp. 275–289). Berlin: Mouton de Gruyter.
Sydorenko, T., Maynard, C., & Guntly, E. (2015). Rater Behaviour When Judging Language Learners’ Pragmatic Appropriateness in Extended Discourse. TESL Canada Journal, 32(1), 19-41.
Taguchi, N. (2006). Analysis of appropriateness in a speech act of request in L2 English. Pragmatics, 16(4), 513-533.
Taguchi, N. (2011). Rater variation in the assessment of speech acts. Pragmatics, 21(3), 453–471.
Tajeddin, Z., Alemi, M., & Pashmforoosh, R. (2011). Non-native teachers’ rating criteria for L2 speaking: Does a rater training program make a difference. Teaching English Language and Literature Society of Iran (TELLSI), 5(1), 125-153.
Tajeddin, Z., & Alemi, M. (2014). Criteria and bias in native English teachers’ assessment of L2 pragmatic appropriacy: Content and FACETS analyses. The Asia-Pacific Education Researcher, 23(3), 425-434.
Tajeddin, Z., Alemi, M., & Razzaghi, S. (2014). Cross-cultural Perceptions of Impoliteness by Native English Speakers and EFL Learners: The Case of Apology Speech Act. Journal of Intercultural Communication Research, 43(4), 304-326.
Tajeddin, Z., & Alemi, M. (2015). Functional Communication in English. Tehran, Jungle Press.
Walkinshaw, I. (2007). Power and disagreement: Insights into Japanese learners of English. RELC Journal, 38(3), 278–301
Walters, F. S. (2007). A conversation-analytic hermeneutic rating protocol to assess L2 oral pragmatic competence. Language Testing, 24(2), 155-183. doi: 10.1177/0265532207076362
Weigle, S. C. (1994b). Effects of training on raters of ESL compositions. Language Testing, 11(2), 197–223.
Widdowson, H. G. (2007). Discourse analysis, Oxford: Oxford University Press.
Yamashita, S. (2008). Investigating interlanguage pragmatic ability: What are we testing? In E. AlcónSoler & A. Martínez-Flor (Eds.), Investigating Pragmatics in Foreign Language Learning, Teaching and Testing (pp. 201–223). Bristol, England: Multilingual Matters.
Youn, S. J. (2007). Rater bias in assessing the pragmatics of KFL learners using facets analysis. Second Language Studies, 26(1), 85–163.
', './data/jtls/coversheet/cover_en.jpg'))