1887
image of Pitfalls of production data analysis for investigating L2 cognitive mechanism

Abstract

Abstract

This article addresses the pitfalls of performance analysis in investigating cognitive processing during second language (L2) learning. The problems that we discuss in this paper are twofold: (1) Assuming psychological variables to be ontological entities without meeting the criteria for ontological reality and (2) Inappropriateness of assessing abilities based on learner’s speaking or writing performance to investigate cognitive processes. By addressing these problems, we argue that some latent variables postulated by observing L2 performance do not exist in reality and emphasize the difficulty of interpreting cognitive mechanisms through performance analysis. We also enumerate some problems that arise from the epistemological perspectives of previous research practice (e.g., the bifurcation of contradictory hypotheses and their indeterminacy). Finally, two alternative approaches treating L2 performance are proposed. The implications of this line of discussion for future research are also discussed.

Available under the CC BY-NC 4.0 license.
Loading

Article metrics loading...

/content/journals/10.1075/jsls.21013.fuk
2022-03-22
2022-05-28
Loading full text...

Full text loading...

/deliver/fulltext/10.1075/jsls.21013.fuk/jsls.21013.fuk.html?itemId=/content/journals/10.1075/jsls.21013.fuk&mimeType=html&fmt=ahah

References

  1. Biber, D., Gray, B., & Poonpon, K.
    (2011) Should we use characteristics of conversation to measure grammatical complexity in L2 writing development?TESOL Quarterly, 31(1), 5–35. 10.5054/tq.2011.244483
    https://doi.org/10.5054/tq.2011.244483 [Google Scholar]
  2. Bollen, K., & Lennox, R.
    (1991) Conventional wisdom on measurement: A structural equation perspective. Psychological Bulletin, 110(2), 305–314. 10.1037/0033‑2909.110.2.305
    https://doi.org/10.1037/0033-2909.110.2.305 [Google Scholar]
  3. Borsboom, D., & Cramer, A. O. J.
    (2013) Network analysis: An integrative approach to the structure of psychopathology. Annual Review of Clinical Psychology, 9, 91–121. 10.1146/annurev‑clinpsy‑050212‑185608
    https://doi.org/10.1146/annurev-clinpsy-050212-185608 [Google Scholar]
  4. Borsboom, D., Cramer, A. O. J., Kievit, R. A., Zand Scholten, A., & Franić, S.
    (2009) The end of construct validity. InR. W. Lissitz (Ed.), The concept of validity (pp.135–170). Charlotte, NC: Information Age Publishers.
    [Google Scholar]
  5. Borsboom, D., Mellenbergh, G. J., & van Heerden, J.
    (2004) The concept of validity. Psychological Review, 111(4), 1061–1071. 10.1037/0033‑295X.111.4.1061
    https://doi.org/10.1037/0033-295X.111.4.1061 [Google Scholar]
  6. Branigan, H. P., & Pickering, M. J.
    (2017) An experimental approach to linguistic representation. Behavioral and Brain Sciences, 40, e282. 10.1017/S0140525X16002028
    https://doi.org/10.1017/S0140525X16002028 [Google Scholar]
  7. Bulté, B., & Housen, A.
    (2018) Syntactic complexity in L2 writing: Individual pathways and emerging group trends. International Journal of Applied Linguistics, 28(1), 147–164. 10.1111/ijal.12196
    https://doi.org/10.1111/ijal.12196 [Google Scholar]
  8. Cartwright, N.
    (1983) How the laws of physics lie. Oxford: Clarendon Press. 10.1093/0198247044.001.0001
    https://doi.org/10.1093/0198247044.001.0001 [Google Scholar]
  9. Churchland, P.
    (1995) The engine of reason, the seat of the soul: A philosophical journey into the brain. Cambridge, MA: MIT Press.
    [Google Scholar]
  10. Coltman, T., Devinney, T. M., Midgley, D. F., & Venaik, S.
    (2008) Formative versus reflective measurement models: Two applications of formative measurement. Journal of Business Research, 61(12), 1250–1262. 10.1016/j.jbusres.2008.01.013
    https://doi.org/10.1016/j.jbusres.2008.01.013 [Google Scholar]
  11. Declerck, M., & Kormos, J.
    (2012) The effect of dual task demands and proficiency on second language speech production. Bilingualism: Language and Cognition, 15(4), 782–796. 10.1017/S1366728911000629
    https://doi.org/10.1017/S1366728911000629 [Google Scholar]
  12. Edwards, J. R., & Bagozzi, R. P.
    (2000) On the nature and direction of relationships between constructs and measures. Psychological Methods, 5(2), 155–174. 10.1037/1082‑989X.5.2.155
    https://doi.org/10.1037/1082-989X.5.2.155 [Google Scholar]
  13. Ellis, R.
    (2003) Task-based language learning and teaching. Oxford: Oxford University Press.
    [Google Scholar]
  14. Fornell, C., & Bookstein, F. L.
    (1982) Two structural equation models: LISREL and PLS applied to consumer exit-voice theory. Journal of Marketing Research, 19(4), 440–452. 10.1177/002224378201900406
    https://doi.org/10.1177/002224378201900406 [Google Scholar]
  15. Foster, P., & Tavakoli, P.
    (2009) Native speakers and task performance: Comparing effects on complexity, fluency, and lexical diversity. Language learning, 59(4), 866–896. 10.1111/j.1467‑9922.2009.00528.x
    https://doi.org/10.1111/j.1467-9922.2009.00528.x [Google Scholar]
  16. Fukuta, J., Goto, A., Kawaguchi, Y., Kurita, A., & Murota, D.
    (2018) Syntactically–driven algorithmic processing of PP-attachment ambiguity in a second language. International Review of Applied Linguistics in Language Teaching, 56(3), 253–277. 10.1515/iral‑2017‑0038
    https://doi.org/10.1515/iral-2017-0038 [Google Scholar]
  17. Fukuta, J., Goto, A., Kawaguchi, Y., Murota, D., & Kurita, A.
    (2015) Japanese EFL learners’ implicit knowledge and algorithmic processing of dative alternation: Perspective from syntactic priming in reading comprehension. ARELE: Annual Review of English Language Education in Japan, 26, 221–236. 10.20581/arele.26.0_221
    https://doi.org/10.20581/arele.26.0_221 [Google Scholar]
  18. Fukuta, J., & Yamashita, J.
    (2015) Effects of cognitive demands on attention orientation in L2 oral production. System, 53, 1–12. 10.1016/j.system.2015.06.010
    https://doi.org/10.1016/j.system.2015.06.010 [Google Scholar]
  19. García Mayo, M. D. P.
    (Ed.) (2006) Investigating tasks in formal language learning. Multilingual Matters. 10.21832/9781853599286
    https://doi.org/10.21832/9781853599286 [Google Scholar]
  20. Gilabert, R.
    (2007a) The simultaneous manipulation of task complexity along planning time and (+ / - here-and-now): Effects on L2 oral production. InM. P. García Mayo (Ed.), Investigating tasks in formal language learning (pp.44–68). Clevedon, UK: Multilingual Matters.
    [Google Scholar]
  21. (2007b) Effects of manipulating task complexity on self-repairs during L2 oral production, International Review of Applied Linguistics in Language Teaching, 45(3), 215–240. 10.1515/iral.2007.010
    https://doi.org/10.1515/iral.2007.010 [Google Scholar]
  22. Gilabert, R., Barón, J., & Llanes, À.
    (2009) Manipulating cognitive complexity across task types and its impact on learners’ interaction during oral performance. International Review of Applied Linguistics in Language Teaching, 47(3), 367–395. 10.1515/iral.2009.016
    https://doi.org/10.1515/iral.2009.016 [Google Scholar]
  23. Hacking, I.
    (1983) Representing and intervening: Introductory topics in the philosophy of science. Cambridge: Cambridge University Press. 10.1017/CBO9780511814563
    https://doi.org/10.1017/CBO9780511814563 [Google Scholar]
  24. Hartsuiker, R. J., Pickering, M. J., & Veltkamp, E.
    (2004) Is Syntax Separate or Shared Between Languages?: Cross-Linguistic Syntactic Priming in Spanish-English Bilinguals. Psychological Science, 15(6), 409–414. 10.1111/j.0956‑7976.2004.00693.x
    https://doi.org/10.1111/j.0956-7976.2004.00693.x [Google Scholar]
  25. Hiver, P., & Al-hoorie, A. H.
    (2016) A dynamic ensemble for second language research: Putting complexity theory into practice. The Modern Language Journal, 100(4), 741–756. 10.1111/modl.12347
    https://doi.org/10.1111/modl.12347 [Google Scholar]
  26. Housen, A., & Kuiken, F.
    (2009) Complexity, accuracy, and fluency in second language acquisition. Applied Linguistics, 30(4), 461–473. 10.1093/applin/amp048
    https://doi.org/10.1093/applin/amp048 [Google Scholar]
  27. Housen, A., Kuiken, F., & Vedder, I.
    (Eds.) (2012) Dimensions of L2 performance and proficiency: Complexity, accuracy and fluency in SLA. Philadelphia: John Benjamins. 10.1075/lllt.32
    https://doi.org/10.1075/lllt.32 [Google Scholar]
  28. Inhelder, B., & Piaget, J.
    (1958) An essay on the construction of formal operational structures. the growth of logical thinking: From childhood to adolescence. A. Parsons & S. Milgram (Trans.). New York: Basic Books. 10.1037/10034‑000
    https://doi.org/10.1037/10034-000 [Google Scholar]
  29. Ishikawa, T.
    (2007) The effect of manipulating task complexity along the [+ / - here-and-now] dimension on L2 written narrative discourse. InM. P. García Mayo (Ed.), Investigating tasks in formal language learning (pp.136–156). Clevedon, UK: Multilingual Matters.
    [Google Scholar]
  30. (2008) The effect of task demands of intentional reasoning on L2 speech performance. The Journal of Asia TEFL, 5, 29–63.
    [Google Scholar]
  31. Izumi, E., Uchimoto, K., & Isahara, H.
    (2004) The NICT-JLE Corpus: Exploiting the language learners’ speech database for research and education. International Journal of the Computer, the Internet and Management, 12(2), 119–125.
    [Google Scholar]
  32. Jackson, D. O., & Suethanapornkul, S.
    (2013) The Cognition Hypothesis: A synthesis and meta-analysis of research on second language task complexity. Language Learning, 63(2), 330–367. 10.1111/lang.12008
    https://doi.org/10.1111/lang.12008 [Google Scholar]
  33. Jolliffe, I. T.
    (2002) Principal component analysis (2nd ed). New York: Springer.
    [Google Scholar]
  34. Kaan, E., & Chun, E.
    (2018) Priming and adaptation in native speakers and second-language learners. Bilingualism: Language and Cognition, 21(2), 228–242. 10.1017/S1366728916001231
    https://doi.org/10.1017/S1366728916001231 [Google Scholar]
  35. Kaan, E., Futch, C., Fuertes, R. F., Mujcinovic, S., & Fuente, E. Á. D. L.
    (2019) Adaptation to syntactic structures in native and nonnative sentence comprehension. Applied Psycholinguistics, 40(1), 3–27. 10.1017/S0142716418000437
    https://doi.org/10.1017/S0142716418000437 [Google Scholar]
  36. Khezrlou, S.
    (2020) The role of task repetition with direct written corrective feedback in L2 writing complexity, accuracy and fluency. Journal of Second Language Studies, 3(1), 31–54. 10.1075/jsls.19025.khe
    https://doi.org/10.1075/jsls.19025.khe [Google Scholar]
  37. Kim, Y.
    (2009) The effects of task complexity on learner-learner interaction. System, 37, 254–268. 10.1016/j.system.2009.02.003
    https://doi.org/10.1016/j.system.2009.02.003 [Google Scholar]
  38. Koizumi, R., & In’nami, Y.
    (2014) Modeling complexity, accuracy, and fluency of Japanese Learners of English: A structural equation modeling approach. JALT Journal, 36(1), 25–46. 10.37546/JALTJJ36.1‑2
    https://doi.org/10.37546/JALTJJ36.1-2 [Google Scholar]
  39. Kormos, J., & Trebits, A.
    (2012) The role of task complexity, modality, and aptitude in narrative task performance. Language Learning, 62, 439–472. 10.1111/j.1467‑9922.2012.00695.x
    https://doi.org/10.1111/j.1467-9922.2012.00695.x [Google Scholar]
  40. Kuiken, F., & Vedder, I.
    (2007) Task complexity and measures of linguistic performance in L2 writing. International Review of Applied Linguistics in Language Teaching, 45, 261–284. 10.1515/iral.2007.012
    https://doi.org/10.1515/iral.2007.012 [Google Scholar]
  41. (2008) Cognitive task complexity and written output in Italian and French as a foreign language. Journal of Second Language Writing, 17, 48–60. 10.1016/j.jslw.2007.08.003
    https://doi.org/10.1016/j.jslw.2007.08.003 [Google Scholar]
  42. (2012) Syntactic complexity, lexical variation and accuracy as a function of task complexity and proficiency level in L2 writing and speaking. InA. Housen, F. Kuiken & I. Vedder (Eds.), Dimensions of L2 performance and proficiency: Complexity, accuracy, and fluency in SLA (pp.143–170). Amsterdam: John Benjamins. 10.1075/lllt.32.07kui
    https://doi.org/10.1075/lllt.32.07kui [Google Scholar]
  43. Lambert, C., & Kormos, J.
    (2014) Complexity, accuracy, and fluency in task-based L2 research: Toward more developmentally based measures of second language acquisition. Applied Linguistics, 35(5), 607–614. 10.1093/applin/amu047
    https://doi.org/10.1093/applin/amu047 [Google Scholar]
  44. Lambert, C., Kormos, J., & Minn, D.
    (2017) Task repetition and second language speech processing. Studies in Second Language Acquisition, 39(1), 167–196. 10.1017/S0272263116000085
    https://doi.org/10.1017/S0272263116000085 [Google Scholar]
  45. Larsen-Freeman, D.
    (2006) The emergence of complexity, fluency, and accuracy in the oral and written production of five Chinese learners of English. Applied Linguistics, 27(4), 590–619. 10.1093/applin/aml029
    https://doi.org/10.1093/applin/aml029 [Google Scholar]
  46. Laudan, L.
    (1981) A confutation of convergent realism. Philosophy of Science, 48(1), 19–49. 10.1086/288975
    https://doi.org/10.1086/288975 [Google Scholar]
  47. Markus, K. A., & Borsboom, D.
    (2013) Frontiers of test validity theory: Measurement, causation, and meaning. Abingdon, UK: Routledge. 10.4324/9780203501207
    https://doi.org/10.4324/9780203501207 [Google Scholar]
  48. Mehnert, U.
    (1998) The effects of different lengths of time for planning on second language performance. Studies in Second Language Acquisition, 20(1), 83–108. 10.1017/S0272263198001041
    https://doi.org/10.1017/S0272263198001041 [Google Scholar]
  49. Nishimura, Y., Kawaguchi, Y., & Abe, D.
    (2018) The relationship between complexity and fluency in L2 writing: An approach using network analysis. Language Education & Technology, (55), 171–198.
    [Google Scholar]
  50. Nishimura, Y., Tamura, Y., & Fukuta, J.
    (2018) Network structures in L2 oral performance: A learner corpus study. ARELE: Annual Review of English Language Education in Japan, 29, 113–128. CitetononCRdoi:10.20581/arele.29.0_113
    https://doi.org/Cite to nonCR doi: 10.20581/arele.29.0_113 [Google Scholar]
  51. Norris, J. M., & Ortega, L.
    (2009) Towards an organic approach to investigating CAF in instructed SLA: The case of complexity. Applied Linguistics, 30(4), 555–578. 10.1093/applin/amp044
    https://doi.org/10.1093/applin/amp044 [Google Scholar]
  52. Ong, J., & Zhang, L. J.
    (2010) Effects of task complexity on the fluency and lexical complexity in EFL students’ argumentative writing. Journal of Second Language Writing, 19(4), 218–233. 10.1016/j.jslw.2010.10.003
    https://doi.org/10.1016/j.jslw.2010.10.003 [Google Scholar]
  53. Pallotti, G.
    (2009) CAF: Defining, refining, and differentiating constructs. Applied Linguistics, 30(4), 590–601. 10.1093/applin/amp045
    https://doi.org/10.1093/applin/amp045 [Google Scholar]
  54. (2015) A simple view of linguistic complexity. Second Language Research, 31(1), 117–134. 10.1177/0267658314536435
    https://doi.org/10.1177/0267658314536435 [Google Scholar]
  55. Pickering, M. J., & Branigan, H. P.
    (1999) Syntactic priming in language production. Trends in cognitive sciences, 3(4), 136–141. 10.1016/S1364‑6613(99)01293‑0
    https://doi.org/10.1016/S1364-6613(99)01293-0 [Google Scholar]
  56. Plonsky, L., & Kim, Y.
    (2016) Task-Based Learner Production: A Substantive and Methodological Review. Annual Review of Applied Linguistics, 36, 73–97. 10.1017/S0267190516000015
    https://doi.org/10.1017/S0267190516000015 [Google Scholar]
  57. Polat, B., & Kim, Y.
    (2014) Dynamics of complexity and accuracy: A longitudinal case study of advanced untutored development. Applied Linguistics, 35(2), 184–207. 10.1093/applin/amt013
    https://doi.org/10.1093/applin/amt013 [Google Scholar]
  58. Révész, A.
    (2014) Towards a fuller assessment of cognitive models of task-based learning: Investigating task-generated cognitive demands and processes. Applied Linguistics, 35(1), 87–92. 10.1093/applin/amt039
    https://doi.org/10.1093/applin/amt039 [Google Scholar]
  59. Reynolds, C. W.
    (1987) Flocks, herds and schools: A distributed behavioral model. SIGGRAPH ’87: Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques. Association for Computing Machinery. pp.25–34. 10.1145/37401.37406
    https://doi.org/10.1145/37401.37406 [Google Scholar]
  60. Robinson, P.
    (1995) Task complexity and second language narrative discourse. Language Learning, 45(1), 99–140. 10.1111/j.1467‑1770.1995.tb00964.x
    https://doi.org/10.1111/j.1467-1770.1995.tb00964.x [Google Scholar]
  61. (2001) Task complexity, task difficulty, and task production: Exploring interactions in a componential framework. Applied Linguistics, 22(1), 27–57. 10.1093/applin/22.1.27
    https://doi.org/10.1093/applin/22.1.27 [Google Scholar]
  62. (2003) Attention and memory during SLA. InC. J. Doughty & M. H. Long (Eds.), The handbook of second language acquisition (pp.631–678). Malden, MA: Blackwell. 10.1002/9780470756492.ch19
    https://doi.org/10.1002/9780470756492.ch19 [Google Scholar]
  63. (2007) Criteria for classifying and sequencing pedagogic tasks. InM. P. García Mayo (Ed.), Investigating tasks in formal language learning (pp.7–26). Clevedon, UK: Multilingual Matters.
    [Google Scholar]
  64. Robinson, P. J.
    (2011) Second language task complexity, the cognition hypothesis, language learning, and performance. InP. J. Robinson (Ed.), Second language task complexity: Researching the cognition hypothesis of language learning and performance (pp.2–38). Amsterdam: John Benjamins. 10.1075/tblt.2.05ch1
    https://doi.org/10.1075/tblt.2.05ch1 [Google Scholar]
  65. Romeijn, J. W.
    (2008) The all-too-flexible abductive method: ATOM’s normative status. Journal of Clinical Psychology, 64(9), 1023–1036. 10.1002/jclp.20516
    https://doi.org/10.1002/jclp.20516 [Google Scholar]
  66. Schmittmann, V. D., Cramer, A. O. J., Waldorp, L. J., Epskamp, S., Kievit, R. A., & Borsboom, D.
    (2013) Deconstructing the construct: A network perspective on psychological phenomena. New Ideas in Psychology, 31(1), 43–53. 10.1016/j.newideapsych.2011.02.007
    https://doi.org/10.1016/j.newideapsych.2011.02.007 [Google Scholar]
  67. Sheppard, C.
    (2004) The measurement of second language production: The validity of fluency, accuracy, and complexity. ICU Language Research Bulletin, 19, 139–156.
    [Google Scholar]
  68. Skehan, P.
    (1996) A framework for the implementation of task-based instruction. Applied Linguistics, 17(1), 38–62. 10.1093/applin/17.1.38
    https://doi.org/10.1093/applin/17.1.38 [Google Scholar]
  69. (1998) A cognitive approach to language learning. Oxford: Oxford University Press.
    [Google Scholar]
  70. (2009) Modelling second language performance: Integrating complexity, accuracy, fluency, and lexis. Applied Linguistics, 30(4), 510–532. 10.1093/applin/amp047
    https://doi.org/10.1093/applin/amp047 [Google Scholar]
  71. (2014) The context researching a processing perspective on task performance. InP. Skehan (Ed.), Processing perspectives on task performance (pp.1–26). Amsterdam: John Benjamins. 10.1075/tblt.5.01ske
    https://doi.org/10.1075/tblt.5.01ske [Google Scholar]
  72. Skehan, P., & Foster, P.
    (1997) Task type and task processing conditions as influences on foreign language performance. Language Teaching Research, 1(3), 185–211. 10.1177/136216889700100302
    https://doi.org/10.1177/136216889700100302 [Google Scholar]
  73. (1999) The influence of task structure and processing conditions on narrative retellings. Language learning, 49(1), 93–120. 10.1111/1467‑9922.00071
    https://doi.org/10.1111/1467-9922.00071 [Google Scholar]
  74. (2005) Strategic and on-line planning: The influence of surprise information and task time on second language performance. InR. Ellis (Ed.), Planning and task performance in a second language (pp.193–216). Amsterdam: John Benjamins. 10.1075/lllt.11.12ske
    https://doi.org/10.1075/lllt.11.12ske [Google Scholar]
  75. Spoelman, M., & Verspoor, M.
    (2010) Dynamic patterns in development of accuracy and complexity: A longitudinal case study in the acquisition of Finnish. Applied Linguistics, 31(4), 532–553. 10.1093/applin/amq001
    https://doi.org/10.1093/applin/amq001 [Google Scholar]
  76. Tavakoli, P., & Foster, P.
    (2011) Task design and second language performance: The effect of narrative type on learner output. Language Learning, 61, 37–72. 10.1111/j.1467‑9922.2011.00642.x
    https://doi.org/10.1111/j.1467-9922.2011.00642.x [Google Scholar]
  77. Van Fraassen, B.
    (1980) The scientific image. Oxford: Oxford University Press. 10.1093/0198244274.001.0001
    https://doi.org/10.1093/0198244274.001.0001 [Google Scholar]
  78. Vercellotti, M. L.
    (2017) The development of complexity, accuracy, and fluency in second language performance: A longitudinal study. Applied Linguistics, 38(1), 90–111. 10.1093/applin/amv002
    https://doi.org/10.1093/applin/amv002 [Google Scholar]
  79. Wolfe-Quintero, K., Inagaki, S., & Kim, H.-Y.
    (1998) Second language development in writing: Measures of fluency, accuracy, and complexity. Honolulu University of Hawai’i, Second Language Teaching and Curriculum Center.
    [Google Scholar]
  80. Wright, C.
    (2020) Effects of task type on L2 Mandarin fluency development. Journal of Second Language Studies, 3(2), 157–179. 10.1075/jsls.00010.wri
    https://doi.org/10.1075/jsls.00010.wri [Google Scholar]
  81. Xu, T. S., Zhang, L. J., & Gaffney, J. S.
    (2021) Examining the relative effectiveness of task complexity and cognitive demands on students’ writing in a second language. Studies in Second Language Acquisition. Advance online publication. 10.1017/S0272263121000310
    https://doi.org/10.1017/S0272263121000310 [Google Scholar]
  82. Yu, H., & Lowie, W.
    (2019) Dynamic paths of complexity and accuracy in second language speech: A longitudinal case study of Chinese learners. Applied Linguistics, 41(6), 855–877. 10.1093/applin/amz040
    https://doi.org/10.1093/applin/amz040 [Google Scholar]
  83. Yuan, F., & Ellis, R.
    (2003) The effects of pre-task planning and on-line planning on fluency, complexity and accuracy in L2 monologic oral production. Applied Linguistics, 24(1), 1–27. 10.1093/applin/24.1.1
    https://doi.org/10.1093/applin/24.1.1 [Google Scholar]
http://instance.metastore.ingenta.com/content/journals/10.1075/jsls.21013.fuk
Loading
/content/journals/10.1075/jsls.21013.fuk
Loading

Data & Media loading...

  • Article Type: Research Article
Keywords: performance analysis ; accuracy ; fluency ; complexity ; ontological realism
This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error