Volume 34, Issue 1
  • ISSN 0774-5141
  • E-ISSN: 1569-9676
Buy:$35.00 + Taxes



This squib briefly explores how contextualized embeddings – which are a type of compressed token-based semantic vectors – can be used as semantic retrieval and annotation tools for corpus-based research into constructions. Focusing on embeddings created by the Bidirectional Encoder Representations from Transformer model, also known as ‘BERT’, this squib demonstrates how contextualized embeddings can help counter two types of retrieval inefficiency scenarios that may arise with purely form-based corpus queries. In the first scenario, the formal query yields a large number of hits, which contain a reasonable number of relevant examples that can be labeled and used as input for a sense disambiguation classifier. In the second scenario, the contextualized embeddings of exemplary tokens are used to retrieve more relevant examples in a large, unlabeled dataset. As a case study, this squib focuses on the - construction (e.g. ).


Article metrics loading...

Loading full text...

Full text loading...


  1. Baroni, Marco , Georgiana Dinu , and Germán Kruszewski
    2014 “Don’t Count, Predict! A Systematic Comparison of Context-Counting vs. Context-Predicting Semantic Vectors.” InProceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ed. by Kristina Toutanova , and Hua Wu , 238–247. Baltimore, Maryland: Association for Computational Linguistics. doi:  10.3115/v1/P14‑1023
    https://doi.org/10.3115/v1/P14-1023 [Google Scholar]
  2. Bergen, Benjamin K. , and Nancy Chang
    2005 “Embodied Construction Grammar in Simulation-Based Language Understanding.” InConstruction Grammars: Cognitive Grounding and Theoretical Extensions, ed. by Jan-Ola Östman , and Mirjam Fried , 147–190. Amsterdam/Philadelphia: John Benjamins. doi:  10.1075/cal.3.08ber
    https://doi.org/10.1075/cal.3.08ber [Google Scholar]
  3. Boleda, Gemma
    2020 “Distributional Semantics and Linguistic Theory.” Annual Review of Linguistics6 (1): 213–234. doi:  10.1146/annurev‑linguistics‑011619‑030303
    https://doi.org/10.1146/annurev-linguistics-011619-030303 [Google Scholar]
  4. Bolukbasi, Tolga , Kai-Wei Chang , James Zou , Venkatesh Saligrama , and Adam Kalai
    2016 “Man Is to Computer Programmer as Woman Is to Homemaker? Debiasing Word Embeddings.” ArXiv:1607.06520 [Cs, Stat], July. arxiv.org/abs/1607.06520
    [Google Scholar]
  5. Clark, Kevin , Urvashi Khandelwal , Omer Levy , and Christopher D. Manning
    2019 “What Does BERT Look At? An Analysis of BERT’s Attention.” ArXiv:1906.04341 [Cs], June. arxiv.org/abs/1906.04341. 10.18653/v1/W19‑4828
    https://doi.org/10.18653/v1/W19-4828 [Google Scholar]
  6. Croft, William
    2001Radical Construction Grammar: Syntactic Theory in Typological Perspective. Oxford: Oxford University Press. 10.1093/acprof:oso/9780198299554.001.0001
    https://doi.org/10.1093/acprof:oso/9780198299554.001.0001 [Google Scholar]
  7. De Pascale, Stefano
    2019 “Token-Based Vector Space Models as Semantic Control in Lexical Sociolectometry”. PhD dissertation, KU Leuven.
    [Google Scholar]
  8. Devlin, Jacob , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova
    2019 “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding”. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), ed. by Jill Burstein , Christy Doran , and Thamar Solorio , 4171–4186. Minneapolis: Association for Computational Linguistics.
    [Google Scholar]
  9. Giulianelli, Mario , Marco Del Tredici , and Raquel Fernández
    2020 “Analysing Lexical Semantic Change with Contextualised Word Representations.” ArXiv:2004.14118 [Cs], April. arxiv.org/abs/2004.14118. 10.18653/v1/2020.acl‑main.365
    https://doi.org/10.18653/v1/2020.acl-main.365 [Google Scholar]
  10. Goldberg, Adele E.
    1995Constructions: A Construction Grammar Approach to Argument Structure. Chicago: University of Chicago Press.
    [Google Scholar]
  11. Gupta, Abhijeet , Gemma Boleda , Marco Baroni , and Sebastian Padó
    2015 “Distributional Vectors Encode Referential Attributes.” InProceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, ed. by Lluís Màrquez , Chris Callison-Burch , and Jian Su , 12–21. Lisbon: Association for Computational Linguistics. doi:  10.18653/v1/D15‑1002
    https://doi.org/10.18653/v1/D15-1002 [Google Scholar]
  12. Heylen, Kris , Thomas Wielfaert , Dirk Speelman , and Dirk Geeraerts
    2015 “Monitoring Polysemy: Word Space Models as a Tool for Large-Scale Lexical Semantic Analysis.” Lingua157 (April): 153–172. doi:  10.1016/j.lingua.2014.12.001
    https://doi.org/10.1016/j.lingua.2014.12.001 [Google Scholar]
  13. Hilpert, Martin
    2013 “Corpus-Based Approaches to Constructional Change.” In: The Oxford Handbook of Construction Grammar, ed. by Thomas Hoffmann , and Graeme Trousdale , 458–475. Oxford: Oxford University Press. doi:  10.1093/oxfordhb/9780195396683.013.0025
    https://doi.org/10.1093/oxfordhb/9780195396683.013.0025 [Google Scholar]
  14. Hilpert, Martin , and David Correia Saavedra
    2017 “Using Token-Based Semantic Vector Spaces for Corpus-Linguistic Analyses: From Practical Applications to Tests of Theoretical Claims.” Corpus Linguistics and Linguistic Theory16(2): 393–424. doi:  10.1515/cllt‑2017‑0009
    https://doi.org/10.1515/cllt-2017-0009 [Google Scholar]
  15. Johnson, Jeff , Matthijs Douze , and Hervé Jégou
    2017 “Billion-Scale Similarity Search with GPUs.” ArXiv:1702.08734 [Cs], February. arxiv.org/abs/1702.08734
    [Google Scholar]
  16. Linzen, Tal , Grzegorz Chrupała , Yonatan Belinkov , and Dieuwke Hupkes
    (eds) 2019Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Florence: Association for Computational Linguistics. https://www.aclweb.org/anthology/W19-4800
    [Google Scholar]
  17. Louwerse, Max M. , and Rolf A. Zwaan
    2009 “Language Encodes Geographical Information.” Cognitive Science33 (1): 51–73. doi:  10.1111/j.1551‑6709.2008.01003.x
    https://doi.org/10.1111/j.1551-6709.2008.01003.x [Google Scholar]
  18. Manning, Christopher D. , Prabhakar Raghavan , and Hinrich Schütze
    2009An Introduction to Information Retrieval. Cambridge: Cambridge University Press.
    [Google Scholar]
  19. Perek, Florent
    2016 “Using Distributional Semantics to Study Syntactic Productivity in Diachrony: A Case Study.” Linguistics54 (1): 149–188. doi:  10.1515/ling‑2015‑0043
    https://doi.org/10.1515/ling-2015-0043 [Google Scholar]
  20. Peters, Matthew , Mark Neumann , Mohit Iyyer , Matt Gardner , Christopher Clark , Kenton Lee , and Luke Zettlemoyer
    2018 “Deep Contextualized Word Representations.” InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), ed. by Marilyn Walker , Heng Ji , and Amanda Stent , 2227–2237. New Orleans: Association for Computational Linguistics. doi:  10.18653/v1/N18‑1202
    https://doi.org/10.18653/v1/N18-1202 [Google Scholar]
  21. Schlechtweg, Dominik , Stefanie Eckmann , Enrico Santus , Sabine Schulte im Walde , and Daniel Hole
    2017 “German in Flux: Detecting Metaphoric Change via Word Entropy.” InProceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), ed. by Roger Levy , and Lucia Specia , 354–367. Vancouver: Association for Computational Linguistics. doi:  10.18653/v1/K17‑1036
    https://doi.org/10.18653/v1/K17-1036 [Google Scholar]
  22. Sommerauer, Pia , and Antske Fokkens
    2018 “Firearms and Tigers Are Dangerous, Kitchen Knives and Zebras Are Not: Testing Whether Word Embeddings Can Tell”. InProceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, ed. by Tal Linzen , Grzegorz Chrupała , and Afra Alishahi , 276–286. Brussels: Association for Computational Linguistics. doi:  10.18653/v1/W18‑5430
    https://doi.org/10.18653/v1/W18-5430 [Google Scholar]
  23. Stefanowitsch, Anatol , and Stefan Gries
    2003 “Collostructions: Investigating the Interaction between Words and Constructions.” International Journal of Corpus Linguistics8 (2): 209–243. 10.1075/ijcl.8.2.03ste
    https://doi.org/10.1075/ijcl.8.2.03ste [Google Scholar]
  24. Vaswani, Ashish , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N. Gomez , Lukasz Kaiser , and Illia Polosukhin
    2017 “Attention Is All You Need”. ArXiv:1706.03762 [Cs], December. arxiv.org/abs/1706.03762
    [Google Scholar]
  25. Yoon, Jiyoung , and Stefan Th Gries
    (eds) 2016Corpus-Based Approaches to Construction Grammar. Amsterdam/Philadelphia: John Benjamins. 10.1075/cal.19
    https://doi.org/10.1075/cal.19 [Google Scholar]

Data & Media loading...

  • Article Type: Research Article
Keyword(s): BERT; corpus linguistics; data retrieval; distributional semantics; prepositions
This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error