Document Type : Research Paper
Authors
1 Yasouj University/ Shiraz University, Iran
2 Department of Language and Linguistics, Islamic Azad University, Fereshtegaan International Branch, Tehran, Iran
3 Florida International University, USA
Abstract
Second language (L2) researchers often rely on quantitative methods and measurement instruments like questionnaires and scales to explore latent constructs. They usually borrow sophisticated statistical techniques from disciplines like psychology and education to conduct quantitative analyses in L2 research. However, there are growing concerns about the inappropriate use and reporting practices of such statistical procedures like exploratory factor analysis (EFA). This study aimed to describe and evaluate the methodological issues in EFA research practices, examine the relationship between study features and outcomes, and improve future L2 research practices. This study presented the results of a methodological synthesis study on exploratory factor analysis use published in five universally reputable psychology-related journals since 2000. Specifically, the researchers identified 93 EFA studies and developed a coding scheme of key EFA considerations to analyze how EFA was used and reported in these articles. The article discussed the several results and provides essential recommendations for L2 researchers intending to employ EFA in their works.
Keywords
Main Subjects
Adelson, J. L., & McCoach, D. B. (2011). Development and psychometric properties of the Math and Me Survey: Measuring third through sixth graders’ attitudes toward mathematics. Measurement and Evaluation in Counseling and Development, 44(4), 225-247.
Alamer A, Schuberth F, Henseler J. (2024). When and how to use confirmatory composite analysis (CCA) in second language research. Studies in Second Language Acquisition, 1-20. doi:10.1017/S0272263124000020
Auerswald, M., & Moshagen, M. (2019). How to determine the number of factors to retain in exploratory factor analysis: A comparison of extraction methods under realistic conditions. Psychological Methods. Advance online publication.
Bandalos, D. L., & Finney, S. J. (2010). Factor Analysis. In G. R. Hancock & R. O. Mueller (Eds.), The reviewer’s guide to quantitative methods in the social sciences (pp. 93-114). Routledge.
Bandalos, D. L., & Finney, S. J. (2018). Factor analysis: Exploratory and confirmatory. In The reviewer’s guide to quantitative methods in the social sciences (pp. 98-122). Routledge.
Bandalos, D. L., & Finney, S. J. (2019). Factor analysis: Exploratory and confirmatory. In G. R. Hancock, L. M. Stapleton, & R. O. Mueller (Eds.), The reviewer’s guide to quantitative methods in the social sciences (2nd ed., pp. 98–122). Routledge. https://doi.org/10.4324/9781315755649-8
Braeken, J., van Assen, M. L. A. M. (2017). An empirical kaiser criterion. Psychological Methods, 22(3), 450-466.
Briggs, N. E., & MacCallum, R. C. (2003). Recovery of weak common factors by maximum likelihood and ordinary least squares estimation. Multivariate Behavioral Research, 38, 25-56.
Brown, T. A. (2013). Latent variable measurement models. In T. D. Little (Ed.), The Oxford handbook of quantitative methods (Vol. 2, Statistical analysis) (pp.257-280). Oxford University Press.
Comrey, A. L., & Lee, H. B. (1992). The first course in factor analysis (2nd ed.). Lawrence Erlbaum.
Cortina, J. M. (2002). Big things have small beginnings: An assortment of “minor” methodological misunderstandings. Journal of Management, 28, 339-362.
De Bot, K. (2015). A history of applied linguistics: From 1980 to the present. Routledge.
DeVellis, R. F. (2022). Scale development: Theory and applications (3rd ed.). Sage.
Dimitrov, D. M. (2012). Statistical methods for validation of assessment scale data in
counseling and related fields. American Counseling Association.
Ding, C. S. (2013). Multidimensionality scaling. In T. D. Little (Ed.), The Oxford handbook of quantitative methods (Vol. 2, Statistical analysis) (pp. 235-256). Oxford University Press.
Dörnyei, Z., & Dewaele, J.-M. (2022). Questionnaires in second language research: Construction, administration, and processing (3rd ed.). Routledge.
Fabrigar, L. R., & Wegener, D. T. (2012). Exploratory factor analysis. Oxford University Press.
Field, A. (2009). Discovering statistics using SPSS (3rd ed.). Sage
Field, A. (2013). Discovering statistics using SPSS (4th ed.). Sage.
Flake, J., Pek, J., & Hehman, E. (2017). Construct validation in social and personality research: Current practice and recommendations. Social Psychological and Personality Science, 8, 370–378. https://doi.org/10.1177/1948550617693063
Ghanbar, H., & Rezvani, R. (2023). Structural Equation Modeling in L2 Research: A Systematic Review. International Journal of Language Testing, 13, 79-108. 10.22034/ijlt.2023.381619.1224
Ghanbar, H. & Rezvani, R. (2024). A Methodological Synthesis of Research Articles in the English for Specific Purposes’s Lifespan. Language Related Research,15 (1), 119-143. 20.1001.1.23223081.1401.0.0.376.8
Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
Gorsuch, R. L. (1990). Common factor analysis versus component analysis: Some well and little known facts. Multivariate behavioral research, 25(1), 33-39.
Gorsuch, R.L. (2003). Factor analysis. In A. Schinka & W.F. Velicer (Vol. Eds.), Handbook of psychology: Vol. 2. Research methods in psychology (pp. 143–164). Wiley
Haig, B. D. (2013). The philosophy of quantitative methods. In T. D. Little (Ed.), The Oxford handbook of quantitative methods (Vol. 1, Foundations) (pp. 7-31). Oxford University Press.
Hair, J. F., Black,W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate data analysis (7th ed.). Prentice Hall.
Hancock, G. R., & Mueller, R. O. (2001). Rethinking Construct Reliability within Latent Variable Systems. In R. Cudeck, S. du Toit, & D. Sorbom (Eds.), Structural Equation Modeling: Present and Future—A Festschrift in Honor of Karl Joreskog (pp. 195-216). Scientific Software International.
Henson, R. K., Capraro, R. M., & Capraro, M. M. (2004). Reporting practices and use of exploratory factor analyses in educational research journals: Errors and explanation. Research in the Schools, 11(2), 61-72.
Henson, R. K., & Roberts, J. K. (2006). Use of exploratory factor analysis in published research: Common errors and some comment on improved practice. Educational and Psychological Measurement, 66(3), 393-416.
Iwaniec, J. (2019). Questionnaires: Implications for effective implementation. In J. McKinley & H. Rose (Eds.), The Routledge handbook of research methods in applied linguistics (pp. 324–335). Routledge.
Jolliffe, I. T. (2002). Principal component analysis (2nd ed.). Springer.
Kaiser, H. F. (1970). A second-generation Little Jiffy. Psychometrika, 35, 401-415.
Kaiser, H. F. (1974). An index of factorial simplicity. Psychometrika, 39, 31-36.
Kieffer, K. M. (1999). An introductory primer on the appropriate use of exploratory and confirmatory factor analysis. Research in the Schools, 6, 75-92
Kim, Y. (2009). Validation of psychometric research instruments: The case of information science. Journal of the American Society for Information Science and Technology, 60, 1178–1191. https://doi.org/10.1002/asi.21066
Lance, C. E., Butts, M. M., & Michels, L. C. (2006). The Sources of Four Commonly Reported Cutoff Criteria. What Did They Really Say? Organizational Research Methods, 9, 202-220.
Lazaraton, A. (2000). Current trends in research methodology and statistics in applied
linguistics. TESOL Quarterly, 34, 175–181.
linguistics. TESOL Quarterly, 34, 175–181.
Lazaraton, A. (2005). Quantitative research methods. In E. Hinkel (Ed.), Handbook of
research in second language teaching and learning (pp. 109–224). Erlbaum.
research in second language teaching and learning (pp. 109–224). Erlbaum.
Liu, Y., Zumbo, B. D., & Wu, A. D. (2012). A demonstration of the impact of outliers on the decisions about the number of factors in exploratory factor analysis. Educational and Psychological Measurement, 72(2), 181-199.
Loewen, S., & Gass, S. (2009). Research timeline: The use of statistics in L2 acquisition research. Language Teaching, 42(2), 181–196. doi: 10.1017/S0261444808005624
Loewen, S., Gonülal, T., Isbell, D. R., Ballard, L., Crowther, D., Lim, J., et al. (2020). How knowledgeable are applied linguistics and SLA researchers about basic statistics? Data from North America and Europe. Studies in Second Language Acquisition, 42(4), 871-890. https://doi.org/10.1017/S0272263119000548
Loewen, S., Lavolette, E., Spino, L. A., Papi, M., Schmidtke, J., Sterling, S., & Wolff, D. (2014).
Statistical literacy among applied linguists and second language acquisition researchers.
TESOL Quarterly, 48(2), 360–388. doi: 10.1002/tesq.128
Statistical literacy among applied linguists and second language acquisition researchers.
TESOL Quarterly, 48(2), 360–388. doi: 10.1002/tesq.128
Loewen, S., & Plonsky, L. (2016). An A–Z of applied linguistics research methods. Bloomsbury Publishing.
Marsh, H., & Alamer, A. (2024). When and how to use set-exploratory structural equation modelling to test structural models: A tutorial using the R package lavaan. British Journal of Mathematical and Statistical Psychology, 1–18. https://doi.org/10.1111/bmsp.12336
Mertler, C. A., & Vannatta, R. A. (2013). Advanced and multivariate statistical methods. Practical application and interpretation (5th ed.). Pyrczak Publishing.
Meyers, L. S., Gamst, G. C., & Guarino, A. J. (2013). Performing data analysis using IBM SPSS. Wiley.
Morea, N., & Ghanbar, H. (2024). Q methodology in applied linguistics: A systematic research synthesis. System, 120, 103194.
Morin, A. J. (2023). Exploratory structural equation modeling. In R. Hoyle (Ed.), Handbook of structural equation modeling (2nd ed.). Guilford.
Pedhazur, E. J., & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach. Lawrence Erlbaum.
Pett, M. A., Lackey, N. R., & Sullivan, J. J. (2003). Making sense of factor analysis: The use of factor analysis for instrument development in health care research. Sage.
Pituch, K. A., & Stevens, J. P. (2016). Applied multivariate statistics for social sciences (6th ed.). Routledge.
Plonsky, L., & Ghanbar, H. (2018). Multiple regression in L2 research: A methodological synthesis and guide to interpreting R2 values. The Modern Language Journal, 102, 713-731.
DOI: 10.1111/modl.125090026-7902/18/1-19
Plonsky, L., & Gonulal, T. (2015). Methodological synthesis in quantitative L2 research: A review of reviews and a case study of exploratory factor analysis. Language Learning, 65(1), 9-36.
Ponto, J. (2015). Understanding and evaluating survey research. Journal of the Advanced Practitioner in Oncology, 6, 168–171.
Russell, D. W. (2002). In search of underlying dimensions: The use (and abuse) of factor analysis in Personality and Social Psychology Bulletin. Personality and Social Psychology Bulletin, 28, 1626-1646.
Selinker, L., & Lakshmanan, U. (2001). How do we know what we know? Why do we believe what we believe? Second Language Research, 17, 323–325.
Spearman, C. (1904). General intelligence’ objectively determined and measured. The American Journal of Psychology, 5, 201–293.
Stevens, J. P. (2009). Applied multivariate statistics for the social sciences (5th ed.). Routledge.
Sudina, E. (2021). Study and scale quality in second language survey research, 2009–2019: The case of anxiety and motivation. Language Learning, 71(4), 1149–1193. https://doi.org/10.1111/lang.12468
Sudina, E. (2023). Scale quality in second-language anxiety and WTC: A methodological synthesis. Studies in Second Language Acquisition, 1–29. https://doi.org/10.1017/S0272263122000560
Tabachnick, B., & Fidell, L. (2007). Using multivariate statistics (5th ed.). Allyn & Bacon.
Tabachnick, B., & Fidell, L. (2013). Using multivariate statistics (6th ed.). Pearson Education.
Tabachnick, B. G., & Fidell, L. S. (2019). Using Multivariate Statistics (7th ed.). Pearson.
Thompson, B. (2004). Exploratory and confirmatory factor analysis: Understanding concepts and applications. American Psychological Association.
Thorkildsen, T. A. (2005). Fundamentals of measurement in applied research. Pearson Education, Inc.
Thurstone, L. L. (1935). The vectors of the mind. University of Chicago Press.
Thurstone, L. L. (1947). Multiple factor analysis. University of Chicago Press.
Velicer, W. F., Eaton, C. A., & Fava, J. L. (2000). Construct explication through factor or component analysis: A review and evaluation of alternative procedures for determining the number of factors or components. In R. D. Goffin & E. Helmes (Eds.), Problems and solutions in human assessment: Honoring Douglas N. Jackson at seventy (pp. 41-71). Kluwer Academic/Plenum Publishers.
Wang, L. L., Watts, A. S., Anderson, R. A., & Little, T. D. (2013). Common fallacies in quantitative research methodology. In T. D. Little (Ed.), The Oxford handbook of quantitative methods (Vol. 2, Statistical analysis) (pp. 718-758). Oxford University Press.
Watson, J. C. (2017). Establishing evidence for internal structure using exploratory factor analysis. Measurement and Evaluation in Counseling and Development, 50(4), 32-238.
', './data/jtls/coversheet/cover_en.jpg'))