1887
Volume 1, Issue 2
  • ISSN 2950-189X
  • E-ISSN: 2950-1881

Abstract

Abstract

While previous machine learning protocols have failed to achieve even observational adequacy in acquiring natural language, generative large language models (LLMs) now produce large amounts of free text with few grammatical errors. This is surprising in view of what is known as “the logical problem of language acquisition”. Given the likely absence of negative evidence in the training process, how would the LLM acquire the information that certain strings are to be avoided as ill-formed? We attempt to employ Dutch-speaking ChatGPT as a linguistic informant by capitalizing on the documented “few shot learning” ability of LLMs. We then investigate whether ChatGPT has acquired familiar island constraints, in particular the CNPC, and compare its performance to that of native speakers. Although descriptive and explanatory adequacy may remain out of reach, initial results indicate that ChatGPT performs well over chance in detecting island violations.

Available under the CC BY 4.0 license.
Loading

Article metrics loading...

/content/journals/10.1075/nb.00015.mul
2025-01-24
2025-02-15
Loading full text...

Full text loading...

/deliver/fulltext/nb.00015.mul.html?itemId=/content/journals/10.1075/nb.00015.mul&mimeType=html&fmt=ahah

References

  1. Baker, C. LeRoy
    1979 Syntactic theory and the projection problem. Linguistic Inquiry10(4). 533–581.
    [Google Scholar]
  2. Bates, Douglas, Martin Maechler, Ben Bolker & Steve Walker
    2015 Fitting linear mixed-effects models using lme4. Journal of Statistical Software67(1). 1–48. 10.18637/jss.v067.i01
    https://doi.org/10.18637/jss.v067.i01 [Google Scholar]
  3. Berwick, Robert C., Paul Pietroski, Beracah Yankama & Noam Chomsky
    2011 Poverty of the stimulus revisited. Cognitive Science351. 1207–1242. 10.1111/j.1551‑6709.2011.01189.x
    https://doi.org/10.1111/j.1551-6709.2011.01189.x [Google Scholar]
  4. Braine, Martin D. S.
    1971 On two types of models of the internalization of grammars. InD. I. Slobin (ed.), The Ontogenesis of Grammar: A Theoretical Symposium. New York: Academic Press.
    [Google Scholar]
  5. Brown, Tom B., Benjamin Mann, Nick Ryder, Malanie Subbiah, Jared D. Kaplan
    2020 Language models are few-shot learners. InH. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan & H. Lin (eds.), Advances in Neural Information Processing Systems331.
    [Google Scholar]
  6. Chomsky, Noam
    2013 Problems of projection. Lingua1301. 33–49. 10.1016/j.lingua.2012.12.003
    https://doi.org/10.1016/j.lingua.2012.12.003 [Google Scholar]
  7. 2016What kind of creatures are we?Columbia University Press.
    [Google Scholar]
  8. 2023 The false promise of ChatGPT. The New York Times, 2023-03-08.
    [Google Scholar]
  9. Clark, Alexander & Eyraud, Rémi
    2007 Polynomial identification in the limit of substitutable context-free languages. Journal of Machine Learning Research81. 1725–1745.
    [Google Scholar]
  10. Cowie, Fiona
    1997 The logical problem of language acquisition. Synthese1111. 17–51. 10.1023/A:1004975305820
    https://doi.org/10.1023/A:1004975305820 [Google Scholar]
  11. Dentella, Vittoria, Fritz Günther & Evelina Leivada
    2023 Systematic testing of three Language Models reveals low language accuracy, absence of response stability, and a yes-response bias. Proceedings of the National Academy of Sciences, 120(51). 10.1073/pnas.2309583120
    https://doi.org/10.1073/pnas.2309583120 [Google Scholar]
  12. Fox, John & Sanford Weisberg
    2019An R Companion to Applied Regression, Third edition. Sage, Thousand Oaks CA. https://socialsciences.mcmaster.ca/jfox/Books/Companion/
    [Google Scholar]
  13. Hamaker, Ellen L., Pascal van Hattum, Rebecca M. Kuiper & Herbert Hoijtink
    2011 Model selection based on information criteria in multilevel modeling. Handbook of advanced multilevel analysis, 231–255.
    [Google Scholar]
  14. Hornstein, Norbert & David Lightfoot
    (eds.) 1981Explanation in Linguistics, The Logical Problem of Language Acquisition. Longman.
    [Google Scholar]
  15. Hu, Jennifer, Kyle Mahowald, Gary Lupyan, Anna Ivanova & Roger Levy
    2024Language models align with human judgments on key grammatical constructions. 10.48550/arXiv.2402.01676
    https://doi.org/10.48550/arXiv.2402.01676 [Google Scholar]
  16. Huang, C.-T. James
    1982 Logical Relations in Chinese and the Theory of Grammar. Diss. MIT.
  17. Huijbregts, Riny
    2008Linguistic Argumentation and Poverty of the Stimulus Arguments. Ms., Utrecht University.
    [Google Scholar]
  18. Kojima, Takeshi, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo & Yusuke Iwasawa
    2022Large Language Models are zero-shot reasoners. 10.48550/arXiv.2205.11916
    https://doi.org/10.48550/arXiv.2205.11916 [Google Scholar]
  19. Laurence, Stephen & Eric Margolis
    2001 The poverty of the stimulus argument. The British Journal for the Philosophy of Science52(2). 217–276. 10.1093/bjps/52.2.217
    https://doi.org/10.1093/bjps/52.2.217 [Google Scholar]
  20. Lenth, Russell V.
    2023emmeans: Estimated Marginal Means, aka Least-Squares Means. R package version 1.9.0, https://CRAN.R-project.org/package=emmeans
    [Google Scholar]
  21. OpenAI
    OpenAI 2023GPT-4 Technical Report. 10.48550/arXiv.2303.08774
    https://doi.org/10.48550/arXiv.2303.08774 [Google Scholar]
  22. Ouyang, Long
    2022Training language models to follow instructions with human feedback. 10.48550/arXiv.2203.02155
    https://doi.org/10.48550/arXiv.2203.02155 [Google Scholar]
  23. Ozaki, Satoru, Dan Yurovsky & Lori Levin
    2022 How well do LSTM language models learn filler-gap dependencies?Proceedings of the Society for Computation in Linguistics (SCiL)5(1). 76–88. 10.7275/414y‑1893
    https://doi.org/10.7275/414y-1893 [Google Scholar]
  24. Pinheiro, José, Douglas Bates & R Core Team
    2023nlme: Linear and Nonlinear Mixed Effects Models. R package version 3.1–164. https://CRAN.R-project.org/package=nlme
    [Google Scholar]
  25. Pinheiro, José & Douglas Bates
    2000Mixed-Effects Models in S and S-PLUS. New York: Springer. 10.1007/978‑1‑4419‑0318‑1
    https://doi.org/10.1007/978-1-4419-0318-1 [Google Scholar]
  26. Pinker, Steven
    1986 Productivity and conservatism in language acquisition. InW. Demopoulos & A. Marras (eds.), Language Learning and Concept Acquisition: Foundational Issues, 54–79. Norwood, NJ: Ablex.
    [Google Scholar]
  27. Posit team
    Posit team 2024RStudio: Integrated Development Environment for R. Boston, MA: PBC. www.posit.co/
    [Google Scholar]
  28. R Core Team
    R Core Team 2023R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/
    [Google Scholar]
  29. Reali, Florencia & Morten H. Christiansen
    2005 Uncovering the richness of the stimulus: Structure dependence and indirect statistical evidence. Cognitive Science291. 1007–1028. 10.1207/s15516709cog0000_28
    https://doi.org/10.1207/s15516709cog0000_28 [Google Scholar]
  30. Ross, John R.
    1967 Constraints on variables in syntax. Doctoral dissertation, MIT, Cambridge, MA. Reprinted as Infinite Syntax!Norwood, NJ: Ablex 1986.
  31. Sprouse, Jon, Matt Wagers & Colin Phillips
    2012 A test of the relation between working-memory capacity and syntactic island effects. Language. 82–123. 10.1353/lan.2012.0004
    https://doi.org/10.1353/lan.2012.0004 [Google Scholar]
  32. Sprouse, Jon & Diogo Almeida
    2017 Design sensitivity and statistical power in acceptability judgment experiments. Glossa: A Journal of General Linguistics2(1).
    [Google Scholar]
  33. Wendler, Chris, Veniamin Veselovsky, Giovanni Monea & Robert West
    2024 Do Llamas work in English? On the latent language of multilingual transformers. 10.18653/v1/2024.acl‑long.820
    https://doi.org/10.18653/v1/2024.acl-long.820 [Google Scholar]
/content/journals/10.1075/nb.00015.mul
Loading
/content/journals/10.1075/nb.00015.mul
Loading

Data & Media loading...

This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error