1887
  1. Home
  2. e-Journals
  3. International Journal of Corpus Linguistics
  4. Online First Articles
  5. Online First Article
image of Can BERT predict fillers for construction elements?
USD
Buy:$35.00 + Taxes

Abstract

The starting point of this paper is a central problem in constructicography: on the one hand, constructicon projects aim at describing a broad spectrum of constructions based on usage data; on the other hand, the established method for determining salient fillers of construction elements (slots) requires comprehensive annotations of entire corpora. In this paper, we propose a new method to tackle this problem. Using the pre-trained language model BERT and a limited number of example sentences, the method predicts typical fillers of slots, so-called collo-profiles. A critical evaluation of the method reveals not only its potentials and limitations but also differences between such a prediction-based method and classical count-based methods.

© John Benjamins Publishing Company
Loading

Article metrics loading...

/content/journals/10.1075/ijcl.24108.fel
2026-05-21
2026-06-07

  • From This Site

    /content/journals/10.1075/ijcl.24108.fel
    dcterms_title,dcterms_subject,pub_keyword
    -contentType:Journal -contentType:Contributor -contentType:Concept -contentType:Institution
    10
    5
Loading full text...

Full text loading...

References

  1. Bengio, Y., Ducharme, R., Vincent, P., & Janvin, C.
    (2003) A neural probabilistic language model. The Journal of Machine Learning Research, , –. 10.5555/944919.944966
     https://doi.org/10.5555/944919.944966 [Google Scholar]
  2. Boas, H. C., Lyngfelt, B., & Torrent, T. T.
    (2019) Framing constructicography. Lexicographica, , –. 10.1515/lex‑2019‑0002
     https://doi.org/10.1515/lex-2019-0002 [Google Scholar]
  3. Brants, S., Dipper, S., Eisenberg, P., Hansen-Schirra, S., König, E., Lezius, W., Rohrer, C., Smith, G., & Uszkoreit, H.
    (2004) TIGER: Linguistic interpretation of a German corpus. Research on Language and Computation, , –. 10.1007/s11168‑004‑7431‑3
     https://doi.org/10.1007/s11168-004-7431-3 [Google Scholar]
  4. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., & Amodei, D.
    (2020) Language models are few-shot learners. InH. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, & H. Lin (Eds.), NIPS’20: Proceedings of the 34th international conference on neural information processing systems (pp.–). Curran Associates, Inc.https://papers.nips.cc/paper_files/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html
     [Google Scholar]
  5. Bubenhofer, N.
    (2020) Semantische Äquivalenz in Geburtserzählungen: Anwendung von Word Embeddings [Semantic equivalence in birth narratives: Application of word embeddings]. Zeitschrift Für Germanistische Linguistik, (), –. 10.1515/zgl‑2020‑2014
     https://doi.org/10.1515/zgl-2020-2014 [Google Scholar]
  6. Czulo, O., Willich, A., Ziem, A., & Torrent, T. T.
    (2023) A multilingual approach to the interaction between frames and constructions: Towards a joint framework and methodology. Constructions and Frames, (), –. 10.1075/cf.00067.czu
     https://doi.org/10.1075/cf.00067.czu [Google Scholar]
  7. de Kok, D.
    (2020) TüBa-D/DP: Wikipedia. [Dataset]. University of Tübingen. 10.57754/FDAT.56d4k‑9zc14
     https://doi.org/10.57754/FDAT.56d4k-9zc14 [Google Scholar]
  8. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K.
    (2019) BERT: Pre-training of deep bidirectional transformers for language understanding. InJ. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: Human language technologies, Volume 1 (Long and short papers) (pp.–). Association for Computational Linguistics. 10.18653/v1/N19‑1423
     https://doi.org/10.18653/v1/N19-1423 [Google Scholar]
  9. Fankhauser, P., & Kupietz, M.
    (2022) Count-based and predictive language models for exploring DeReKo. InP. Bański, A. Barbaresi, S. Clematide, M. Kupietz, & H. Lüngen (Eds.), Proceedings of the LREC 2022 workshop on challenges in the management of large corpora (pp.–). European Language Resources Association (ELRA). https://aclanthology.org/2022.cmlc-1.5/
     [Google Scholar]
  10. Fehrmann, I.
    (2018) Kausative Konstruktionen mit dem Verb “machen” im Deutschen [Causative constructions with the verb “machen” in German]. [Doctoral dissertation, The Humboldt University of Berlin]. 10.18452/19403
     https://doi.org/10.18452/19403
  11. Firth, J. R.
    (1957) A Synopsis of linguistic theory 1930–1955. InPhilological Society (ed.), Studies in linguistic analysis: Special volume of the Philological society. (pp.–). Blackwell.
     [Google Scholar]
  12. Flach, S.
    (2021) Collostructions: An R implementation for the family of collostructional methods (v.0.2.0) [Computer software]. https://sfla.ch/collostructions/
     [Google Scholar]
  13. Goldberg, A.
    (1995) Constructions: A Construction grammar approach to argument structure. University of Chicago Press.
     [Google Scholar]
  14. (2005) Constructions at work: The nature of generalization in language. Oxford University Press. 10.1093/acprof:oso/9780199268511.001.0001
     https://doi.org/10.1093/acprof:oso/9780199268511.001.0001 [Google Scholar]
  15. Gries, S. T.
    (2019) 15 years of collostructions: Some long overdue additions/corrections (to/of actually all sorts of corpus-linguistics measures). International Journal of Corpus Linguistics, (), –. 10.1075/ijcl.00011.gri
     https://doi.org/10.1075/ijcl.00011.gri [Google Scholar]
  16. Harris, Z. S.
    (1954) Distributional structure. WORD, (), –. 10.1080/00437956.1954.11659520
     https://doi.org/10.1080/00437956.1954.11659520 [Google Scholar]
  17. Herbst, T.
    (2018) Is language a collostructicon? — A proposal for looking at collocations, valency, argument structure and other constructions. InP. Cantos-Gómez & M. Almela-Sánchez (Eds.), Lexical collocation analysis: Advances and applications (pp.–). Springer. 10.1007/978‑3‑319‑92582‑0_1
     https://doi.org/10.1007/978-3-319-92582-0_1 [Google Scholar]
  18. (2020) Constructions, generalizations, and the unpredictability of language: Moving towards colloconstruction grammar. Constructions and Frames, (), –. 10.1075/cf.00035.her
     https://doi.org/10.1075/cf.00035.her [Google Scholar]
  19. Herbst, T. & Hoffmann, T.
    (2018) Construction Grammar for students: A constructionist approach to syntactic analysis (CASA). Yearbook of the German Cognitive Linguistics Association, (), –. 10.1515/gcla‑2018‑0010
     https://doi.org/10.1515/gcla-2018-0010 [Google Scholar]
  20. Hoffmann, T., & Trousdale, G.
    (Eds.) (2013) The Oxford handbook of construction grammar. Oxford University Press. 10.1093/oxfordhb/9780195396683.001.0001
     https://doi.org/10.1093/oxfordhb/9780195396683.001.0001 [Google Scholar]
  21. Jackendoff, R.
    (2008) Construction after construction and its theoretical challenges. Language, , –. https://www.jstor.org/stable/40071010
     [Google Scholar]
  22. Jurafsky, D., & Martin, J. H.
    (2009) Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition (2nd ed.). Prentice-Hall.
     [Google Scholar]
  23. Kupietz, M., Diewald, N., & Margaretha, E.
    (2020) RKorAPClient: An R package for accessing the German reference corpus DeReKo via KorAP. InN. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, & S. Piperidis (Eds.), Proceedings of the twelfth language resources and evaluation conference (pp.–). European Language Resources Association. https://aclanthology.org/2020.lrec-1.867
     [Google Scholar]
  24. Kupietz, M., Lüngen, H., Kamocki, P., & Witt, A.
    (2018) The German reference corpus DeReKo: New developments — new opportunities. InN. Calzolari, K. Choukri, C. Cieri, T. Declerck, S. Goggi, K. Hasida, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis, & T. Tokunaga (Eds.), Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018) (p.). European Language Resources Association. https://aclanthology.org/L18-1689/
     [Google Scholar]
  25. Lebani, G. E., & Lenci, A.
    (2017) Modelling the meaning of argument constructions with distributional semantics. Papers from the 2017 AAAI spring symposium (pp.–). Association for the Advancement of Artificial Intelligence. https://aaai.org/papers/15345-15345-modelling-the-meaning-of-argument-constructions-with-distributional-semantics/
     [Google Scholar]
  26. Lyngfelt, B.
    (2018) Introduction: Constructicon and constructicography. InB. Lyngfelt, L. Borin, K. Ohara, & T. T. Torrent (Eds.), Constructicography: Constructicon development across languages (pp.–). John Benjamins. 10.1075/cal.22.01lyn
     https://doi.org/10.1075/cal.22.01lyn [Google Scholar]
  27. Lyngfelt, B., Borin, L., Ohara, K., & Torrent, T. T.
    (Eds.) (2018) Constructicography: Constructicon development across languages. John Benjamins. 10.1075/cal.22
     https://doi.org/10.1075/cal.22 [Google Scholar]
  28. Mikolov, T., Chen, K., Corrado, G., & Dean, J.
    (2013) Efficient estimation of word representations in vector space. arXiv:1301.3781. 10.48550/arXiv.1301.3781
     https://doi.org/10.48550/arXiv.1301.3781 [Google Scholar]
  29. Perek, F., & Hilpert, M.
    (2017) A Distributional semantic approach to the periodization of change in the productivity of constructions. International Journal of Corpus Linguistics, (), –. 10.1075/ijcl.16128.per
     https://doi.org/10.1075/ijcl.16128.per [Google Scholar]
  30. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L.
    (2018) Deep contextualized word representations. InM. Walker, H. Ji, & A. Stent (Eds.), Proceedings of the 2018 conference of the North American chapter of the association for computational linguistics: Human language technologies, Volume 1 (Long papers) (pp.–). Association for Computational Linguistics. 10.18653/v1/N18‑1202
     https://doi.org/10.18653/v1/N18-1202 [Google Scholar]
  31. Rambelli, G., Chersoni, E., Blache, P., Huang, C.-R., & Lenci, A.
    (2019) Distributional semantics meets construction grammar: Towards a unified usage-based model of grammar and meaning. InN. Xue, W. Croft, J. Hajič, C.-R. Huang, S. Oepen, M. Palmer, & J. Pustejovsky (Eds.), Proceedings of the first international workshop on designing meaning representations (pp.–). Association for Computational Linguistics.10.18653/v1/W19‑3312
     https://doi.org/10.18653/v1/W19-3312 [Google Scholar]
  32. Sahlgren, M.
    (2008) The distributional hypothesis. Rivista Die Linguistica, (), –. https://www.italian-journal-linguistics.com/app/uploads/2021/05/Sahlgren-1.pdf
     [Google Scholar]
  33. Stefanowitsch, A.
    (2007) Wortwiederholungen im Englischen und Deutschen: Eine korpuslinguistische Annäherung [Word repetitions in English and German: A corpus linguistic approach]. InA. Ammann & A. Urdze (Eds.), Wiederholung, Parallelismus, Reduplikation: Strategien der multiplen Strukturanwendung (pp.–). Brockmeyer.
     [Google Scholar]
  34. Stefanowitsch, A., & Gries, S. T.
    (2003) Collostructions: Investigating the interaction of words and constructions. International Journal of Corpus Linguistics, (), –. 10.1075/ijcl.8.2.03ste
     https://doi.org/10.1075/ijcl.8.2.03ste [Google Scholar]
  35. Tayyar Madabushi, H., Romain, L., Divjak, D., & Milin, P.
    (2020) CxGBERT: BERT meets construction grammar. InD. Scott, N. Bel, & C. Zong (Eds.), Proceedings of the 28th international conference on computational linguistics (pp.–). International Committee on Computational Linguistics. 10.18653/v1/2020.coling‑main.355
     https://doi.org/10.18653/v1/2020.coling-main.355 [Google Scholar]
  36. Torrent, T. T., Hoffmann, T., Almeida, A. L. & Turner, M.
    (2024) Copilots for linguists: AI, constructions, and frames. Cambridge University Press. 10.1017/9781009439190
     https://doi.org/10.1017/9781009439190 [Google Scholar]
  37. Veenboer, T., & Bloem, J.
    (2023) Using collostructional analysis to evaluate BERT’s representation of linguistic constructions. InA. Vlachos & I. Augenstein (Eds.), Findings of the association for computational linguistics: EACL 2023 (pp.–). Association for Computational Linguistics. 10.18653/v1/2023.findings‑acl.819
     https://doi.org/10.18653/v1/2023.findings-acl.819 [Google Scholar]
  38. Wada, T., Baldwin, T., & Lau, J.
    (2023) Unsupervised lexical simplification with context augmentation. InH. Bouamor, J. Pino, & K. Bali (Eds.), Findings of the association for computational linguistics: EMNLP 2023 (pp.–). Association for Computational Linguistics. 10.18653/v1/2023.findings‑emnlp.627
     https://doi.org/10.18653/v1/2023.findings-emnlp.627 [Google Scholar]
  39. Wada, T., Baldwin, T., Matsumoto, Y., & Lau, J. H.
    (2022) Unsupervised lexical substitution with decontextualised embeddings. InN. Calzolari, C.-R. Huang, H. Kim, J. Pustejovsky, L. Wanner, K.-S. Choi, P.-M. Ryu, H.-H. Chen, L. Donatelli, H. Ji, S. Kurohasi, P. Paggio, N. Xue, S. Kim, Y. Hahm, Z. He, T. K. Lee, E. Santus, F. Bond, & S.-H. Na (Eds.), Proceedings of the 29th international conference on computational linguistics (pp.–). International Committee on Computational Linguistics. https://aclanthology.org/2022.coling-1.366
     [Google Scholar]
  40. Wartena, C.
    (2019) A Probabilistic morphology model for German lemmatization. Inthe Chair of Computational Corpus Linguistics (Eds.), Proceedings of the 15th conference on natural language processing (KONVENS 2019) (pp.–). German Society for Computational Linguistics & Language Technology. 10.25968/opus‑1527
     https://doi.org/10.25968/opus-1527 [Google Scholar]
  41. Webber, W., Moffat, A., & Zobel, J.
    (2010) A similarity measure for indefinite rankings. ACM Transactions on Information Systems, (), –. 10.1145/1852102.1852106
     https://doi.org/10.1145/1852102.1852106 [Google Scholar]
  42. Weissweiler, L., He, T., Otani, N. R., Mortensen, D., Levin, L., & Schütze, H.
    (2023) Construction grammar provides unique insight into neural language models. InC. Bonial & H. Tayyar Madabushi (Eds.), Proceedings of the first international workshop on construction grammars and NLP (CxGs+NLP, GURT/SyntaxFest 2023) (pp.–). Association for Computational Linguistics. https://aclanthology.org/2023.cxgsnlp-1.10
     [Google Scholar]
  43. Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, Ł., Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C., Smith, J., Riesa, J., Rudnick, A., Vinyals, O., Corrado, G., Hughes, M., & Dean, J.
    (2016) Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv:1609.08144. arxiv.org/abs/1609.08144
     [Google Scholar]
  44. Xu, L., Wu, J., Peng, J., Gong, Z., Cai, M., & Wang, T.
    (2023) Enhancing language representation with constructional information for natural language understanding. InA. Rogers, J. Boyd-Graber, & N. Okazaki (Eds.), Proceedings of the 61st annual meeting of the association for computational linguistics (Volume 1: Long papers) (pp.–). Association for Computational Linguistics. 10.18653/v1/2023.acl‑long.258
     https://doi.org/10.18653/v1/2023.acl-long.258 [Google Scholar]
  45. Ziem, A.
    (2018) Tag für Tag Arbeit über Arbeit: Konstruktionsgrammatische Zugänge zu Reduplikationsstrukturen im Deutschen [Day by day, work upon work: Constructionist approaches to reduplication structures in German]. InK. Steyer (ed.), Sprachliche Verfestigung. Wortverbindungen, Muster, Phrasem-Konstruktionen (pp.–). Narr.
     [Google Scholar]
  46. Ziem, A. & Flick, J.
    (2019) Constructicography at work: Implementation and application of the German Constructicon. Yearbook of the German Cognitive Linguistics Association, (), –. 10.1515/gcla‑2019‑0012
     https://doi.org/10.1515/gcla-2019-0012 [Google Scholar]
  47. Ziem, A. & Feldmüller, T.
    (2023) Dimensions of constructional meanings in the German constructicon: Why collo-profiles matter. Yearbook of the German Cognitive Linguistics Association, (), –. 10.1515/gcla‑2023‑0010
     https://doi.org/10.1515/gcla-2023-0010 [Google Scholar]
/content/journals/10.1075/ijcl.24108.fel
Loading
Can BERT predict fillers for construction elements?
John Benjamins ; https://doi.org/10.1075/ijcl.24108.fel
/content/journals/10.1075/ijcl.24108.fel
/content/journals/10.1075/ijcl.24108.fel
Loading

Data & Media loading...

  • Article Type: Research Article
Keywords: constructicography ; BERT ; construction grammar ; collostructional analysis ; language models

Most Cited

This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error