Document Type : Research Paper
Authors
1 English Department Vali-e-Asr University of Rafsanjan
2 Vali-Asr University
3 Human literature faculty, Vali-e-asr university of Rafsanjan, Rafsanjan, Iran.
Abstract
The aim of the current study was twofold: (1) to validate the internal structure of the general English (GE) section of the university entrance examination for Ph.D applicants into the English programs at state universities in Iran (Ph.D. UEE), and (2) to examine the factor structure invariance of the Ph.D. UEE across two proficiency levels. Structural equation modeling (SEM) was used to analyze the responses of a random sample of participants (N=1009) who took the test in 2014 to seek admission to English programs at Iranian state universities. First, four models (unitary, uncorrelated, correlated and higher-ordered) were estimated and compared to find the model that best represented the data. Then, the factor structure invariance of the test across two proficiency levels was explored using multigroup confirmatory factor analysis. The higher-order and correlated three-factor model showed the best fit to the data. The result also showed that the structure of the test remained invariant across both proficiency levels. These results supported the multi-componential view of language proficiency. It was found that there is no relationship between levels of language proficiency and the structure of the test. However, the results called into question the score-reporting policy for the PhD UEE and led to the conclusion that a single total score does not reflect the structure of the test.
Keywords
Ahmadi, A., Darabi Bazvand, A., Sahragard, R., & Razmjoo, A. (2015). Investigating the Validity of PhD. Entrance Exam of ELT in Iran in Light of Argument-Based Validity and Theory of Action. Journal of Teaching Language Skills, 34(2), 1-37.
Alavi, T. (2012). The Predictive Validity of Final English Exams as a Measure of Success in Iranian National University Entrance English Exam. Journal of Language Teaching and Research, 3(1), 224-228.
Alderson, J. C. (1991): Language testing in the 1990s: How far have we come? How much further have we to go? In S. Anivan (Ed.), Current developments in language testing (p. 18). Singapore: SEAMEO Regional Language Centre.
Alibakhshi, G., & Ali, H. G. (2011). External Validity of TOEFL Section of Doctoral Entrance Examination in Iran: A Mixed Design Study. Theory and Practice in Language Studies, 1(10), 1304-1310.
Bachman, L. F. (1990). Fundamental considerations in language testing. Oxford University Press.
Bachman. L. F. (2005). Building and supporting a case for test use. Language Assessment Quarterly, 2(1), 1–34.
Bachman, L. F. &Palmer. A. S. (1981a). The construct validation of the FSI oral interview. Language Learning, 31(1), 67-86.
Bachman, L. F., & Palmer, A. (1982). The construct validation of some components of communicative proficiency. TESOL Quarterly, 16(1), 449–465.
Bachman, L. F., Davidson, F., Ryan, K., & Choi, I.-C. (1995). An investigation into the comparability of two tests of English as a foreign language: The Cambridge-TOEFL comparability study.Cambridge, England: Cambridge University Press.
Bae, J., & Bachman, L. F. (1998). A latent variable approach to listening and reading: Testing factorial invariance across two groups of children in the Korean/English two-way immersion program. Language Testing, 15(3), 380-414.
Barati, H., & Ahmadi, A. R. (2010). Gender-based DIF across the Subject Area: A Study of the Iranian National University Entrance Exam. The Journal of Teaching Language Skills (JTLS), 2(3), 1-22.
Barati, H., Ketabi, S., & Ahmadi, A. (2006). Differential item functioning in high-stakes tests: the effect of field of study. International Journal of American Linguistics (IJAL), 9 (3), 27-49.
Barbour, R. P. (1983). An exploratory study of the hypothesis of divisible versus unitary competence in second language proficiency (Doctoral dissertation, University of British Columbia), 1-147.
Beauducel, A., & Wittmann, W. W. (2005). Simulation study on fit indexes in CFA based on data with a slightly distorted simple structure. Structural Equation Modeling, 12(1), 41-75.
Bennett, R. E. (2010). Cognitively based assessment of, for, and as learning: A preliminary theory of action for summative and formative assessment. Measurement: Interdisciplinary Research and Perspectives, 8 (1), 70-91.
Bentler, P. M. (1992). On the fit of models to covariances and methodology to the Bulletin. Psychological Bulletin, 112(3), 400.
Birjandi, P., & Amini, M. (2007). Differential item functioning (test bias) analysis paradigm across manifest and latent examinee groups (on the construct validity of IELTS). Journal of Human Sciences, 8(2), 1-20.
Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen, & J. S. Long (Eds.), Testing structural equation models (pp. 136–162). Newbury Park, CA: Sage.
Byrne, B.M. (1989): A primer of LISREL: basic applications and programming for confirmatory factor analytic models. New York: Springer Verlag.
Byrne, B. M. (1994). Structural equation modeling with EQS and EQS/Windows: Basic concepts, applications, and programming. University of Ottawa, Canada. Sage.
Byrne, B. M. (2013). Structural equation modeling with AMOS: Basic concepts, applications, and programming. Routledge.
Canale, M., & Swain, M. (1980). Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics, 1(1), 1-47.
Carroll, J. B. (1983). Psychometric theory and language testing. Issues in Language Testing Research, 80-107.
Chen, F. F., Sousa, K. H., & West, S. G. (2005). Teacher's corner: Testing measurement invariance of second-order factor models. Structural Equation Modeling, 12(3), p.474.
Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9(2), 233-255.
Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272.
Farhady, H. (1983). On the plausibility of the unitary language proficiency factor. Issues in Language Testing Research, 11-28.
Farhady, H., & Abbassian, G. R. (2000). The test method, level of language proficiency and the underlying structure of language ability. Al Zahra Journal, 9(29), 27-32.
Farnsworth, T. L. (2013). An investigation into the validity of the TOEFL iBT speaking test for international teaching assistant certification. Language Assessment Quarterly, 10(3), 274-291.
Fouly, K. A., Bachman, L. F., & Cziko, G. A. (1990). The divisibility of language competence: A confirmatory approach. Language Learning, 40(1), 1-21.
Garrido, L. E., Abad, F. J., & Ponsoda, V. (2012). A new look at Horn’s parallel analysis with ordinal variables. Psychological Methods, in press. Epub ahead of print retrieved December 10, 2012.
Ginther, A., & Stevens, J. (1995). Language Background, Ethnicity, and the Internal Construct Validity of the Advanced Placement Spanish Language Examination. Education Resource Information Center (ERIC). 1-27.
Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2), 179-185.
Horn, J. L., McArdle, J. J., & Mason, R. (1983). When is invariance not invariant: A practical scientist's look at the ethereal concept of factor invariance? Southern Psychologist, 4(2), 179-188.
Hoyle, R. H. (Ed.). (1995). Structural equation modeling: Concepts, issues, and applications. Sage Publications.
Hu, L-T., & Bentler, P. M. (1999). Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria versus New Alternatives. Structural Equation Modeling, 6(1), 1–55.
In'nami, Y., & Koizumi, R. (2011). Factor structure of the revised TOEIC test: A multiple-sample analysis. Language Testing, 29(1), 131-152.
Jamalifar, G., Tabrizi, H. H., & Chalak, A. (2014). Islamic Azad University Entrance Examination of Master Program in. The Iranian EFL Journal, 29(1), 386.
Jiao, H. (2004). Evaluating the dimensionality of the Michigan English language assessment battery. Spaan Fellow Working Papers in Second or Foreign Language Assessment. 2004(2), 27-155.
Kaiser, H. F. (1960). The application of electronic computers to factor analysis. Educational and psychological measurement, 20(1), 141-151.
Kane, M. T. (1992). An argument-based approach to validity. Psychological Bulletin, 112(3), 527-535.
Kline, R. B. (1989). Principles and practice of structural equation modeling. London: The Guilford Press.
Kunnan, A. J. (1992). An investigation of a criterion-referenced test using G-theory, and factor and cluster analyses. Language Testing, 9(1), 30-49.
Mahmoudi, L., & Bakar, K. A. (2013). Iranian Pre-university English Teachers' Perceptions and Attitudes towards the Iranian National University Entrance Exam: A Washback Study. International Journal of Education & Literacy Studies, 1(2), 47.
Meredith, W. (1993). Measurement invariance, factor analysis, and factorial invariance. Psychometrika, 58(4), 525-543.
Mesick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (pp. 13–103). Washington, DC: American Council on Education and National Council on Measurement in Education.
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741.
Oller, J. W., Jr. (1978). The language factor in the evaluation of bilingual education. In J. Alatis (Ed.), International dimension of bilingual education. Washington, D.C.: Georgetown University Press.
Oller, J. W. (1979). Language tests at school: A pragmatic approach. Addison-Wesley Longman Ltd.
Oller, J. W., & Hinofotis, F. A. (1980). Two mutually exclusive hypotheses about second language ability: Factor analytic studies of a variety of language subtests. In J. W. Oller, Jr., & K. Perkins (Eds.), Research in language testing (pp. 13–23). Rowley, MA:
Newbury House.
Ravand, H., & Firoozi, T. (2016). Investigating Validity of UEE using the Rasch Model. International Journal of Language Testing, 6(1), 1-23.
Rindskopf, D., & Rose, T. (1988). Some theory and applications of confirmatory second-order factor analysis. Multivariate Behavioral Research, 23(1), 51-67.
Römhild, A. (2008). Investigating the invariance of the ECPE factor structure across different proficiency levels. Spaan Fellow, 6(1), 29-54.
Saito, Y. (2003). Investigating the construct validity of the cloze section in the Examination for the Certificate of Proficiency in English. Spaan Fellow Working Papers in Second or Foreign Language Assessment, 2003(1), 39-82.
Salehi, H., & Yunus, M. M. (2012). The washback effect of the Iranian universities entrance exam: Teachers’ insights. GEMA: Online Journal of Language Studies, 12(2), 609-628.
Shin, S. K. (2005). Did they take the same test? Examinee language proficiency and the structure of language tests. Language Testing, 22(1), p.31.
Sasaki, M. (1996). Second language proficiency, foreign language aptitude, and intelligence: Quantitative and qualitative analyses. New York, NY: Peter Lang Publishing, Inc.
Sawaki, Y., Stricker, L. J., & Oranje, A. H. (2009). Factor structure of the TOEFL Internet-based test. Language Testing, 26 (1), 5-30.
Song, M. Y. (2008). Do distinct subskills exist in second language (L2) comprehension? A structural equation modeling approach. Language Testing, 25(4), 435-464.
Steiger, J. H., & Lind, J. C. (1980). Statistically based tests for the number of common factors. In annual meeting of the Psychometric Society, Iowa City, IA. 758(1), 424-453.
Stricker, L. J., & Rock, D. A. (2008). Factor Structure of the TOEFL Internet‐Based Test across Subgroups. ETS Research Report Series, 2008(2), i-38.
Stricker, L. J., Rock, D. A., & Lee, Y. W. (2005). Factor structure of the languedge™ test across language groups. ETS Research Report Series, 2005(1), i-43.
Swinton, S. S., & Powers, D. E. (1980). Factor analysis of the Test of English as a Foreign Language for several language groups. ETS Research Report Series, 1980(2), i-79.
Ullman, J. B. (2001). Structural equation modeling. In B. G. Tabachnick & L. S. Fidell, Using Multivariate statistics (pp. 653–771). Needham Heights, MA: Allyn and Bacon.
Velicer, W. F., Eaton, C. A., & Fava, J. L. (2000). Construct explication through factor or component analysis: A re-review and evaluation of alternative procedures for determining the number of factors or components. In Problems and solutions in human assessment (pp. 41-71). Springer US.
Vollmer, H. J. (1983). The structure of foreign language competence. Current developments in language testing, 3-30.
Wagner, E. (2004). A construct validation study of the extended listening sections of the ECPE and MELAB. Spaan Fellow Working Papers in Second or Foreign Language Assessment 2004(1), 1-155.
Wang, S. (2006). Validation and Invariance of Factor Structure of the ECPE and MELAB across Gender. SPAAN FELLOW, 4 (1), 41-56.
Yuan, K. H. (2005). Fit indices versus test statistics. Multivariate Behavioral Research, 40(1), 115-148.
Zhang, B. (2010). Assessing the accuracy and consistency of language proficiency classification under competing measurement models. Language Testing, 27(1), p.120.
Zwick, W. R., & Velicer, W. F. (1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99(3), 432.
', './data/jtls/coversheet/cover_en.jpg'))