Volume 27, Issue 2
  • ISSN 0929-9971
  • E-ISSN: 1569-9994
Buy:$35.00 + Taxes



In this paper, we address the system evaluation issue for commercial term extraction tools from the users’ perspective. We first revisit the gold standard approach commonly practised among researchers, and discuss the challenges it may pose on end users, taking translators as a typical example. Considering the very different motivations and needs of users and researchers, a user-driven approach is proposed as a variation and alternative to the gold standard approach to allow users to assess and understand the performance of commercial tools more objectively. Its feasibility and usefulness are demonstrated by deploying a benchmarking dataset of English-Chinese financial terms, produced by multiple annotators, in a case study with SDL MultiTerm Extract. The results also provide insight for future development of term extractors designed for translators, which will hopefully generate more accurate candidates, offer more customised features, enable better user experience, and enjoy wider popularity as a computer-aided translation tool.


Article metrics loading...

Loading full text...

Full text loading...


  1. Agirre, Eneko, Xabier Arregi, Xabier Artola, Arantza Díaz de Illarraza, Kepa Sarasola, and Aitor Soroa
    2000 “A Methodology for Building Translator-oriented Dictionary Systems.” Machine Translation15: 295–310. 10.1023/A:1012092229448
    https://doi.org/10.1023/A:1012092229448 [Google Scholar]
  2. Baldwin, Timothy, and Takaaki Tanaka
    2004 Translation by machine of complex nominals: Getting it right. InProceedings of the Second ACL Workshop on Multiword Expressions: Integrating Processing, 24–31. Barcelona, Spain. 10.3115/1613186.1613190
    https://doi.org/10.3115/1613186.1613190 [Google Scholar]
  3. Bernier-Colborne, Gabriel, and Patrick Drouin
    2014 “Creating a test corpus for term extractors through term annotation.” Terminology20(1): 50–73. 10.1075/term.20.1.03ber
    https://doi.org/10.1075/term.20.1.03ber [Google Scholar]
  4. Bertels, Ann, and Dirk Speelman
    2014 “Clustering for semantic purposes: Exploration of semantic similarity in a technical corpus.” Terminology20(2): 279–303. 10.1075/term.20.2.07ber
    https://doi.org/10.1075/term.20.2.07ber [Google Scholar]
  5. Black, E., S. Abney, D. Flickenger, D. C. Gdaniec, R. Grishman, P. Harrison, D. Hindle, R. Ingria, F. Jelinek, J. Klavans, M. Liberman, M. Marcus, S. Roukos, B. Santorini, and T. Strzalkowski
    1991 “A Procedure for Quantitatively Comparing the Syntactic Coverage of English Grammars.” InProceedings of the DARPA Workshop on Speech and Natural Language, 306–311. Pacific Grove, California. 10.3115/112405.112467
    https://doi.org/10.3115/112405.112467 [Google Scholar]
  6. Blancafort, Helena, Francis Bouvier, Béatrice Daille, Ulrich Heid, and Anita Ramm
    2013 “TTC Web Platform: from Corpus Compilation to Bilingual Terminologies for MT and CAT Tools.” InProceedings of TRALOGY II, Paris.
    [Google Scholar]
  7. Bourigault, Didier
    1992 “Surface Grammatical Analysis for the Extraction of Terminological Noun Phrases.” InProceedings of the Fourteenth International Conference on Computational Linguistics (COLING ’92), 977–981. Nantes, France. 10.3115/992383.992415
    https://doi.org/10.3115/992383.992415 [Google Scholar]
  8. Bowker, Lynne
    2015 “Computer-aided Translation: Translator training.” InThe Routledge Encyclopedia of Translation Technology, ed. bySin-Wai Chan. Routledge.
    [Google Scholar]
  9. Cabré Castellví, M. Teresa, Rosa Estopà Bagot, and Jordi Vivaldi Palatresi
    2001 “Automatic term detection: A review of current systems.” InRecent Advances in Computational Terminology, ed. byDidier Bourigault, Christian Jacquemin, and Marie-Claude L’Homme, 53–87. Amsterdam/Philadelphia: John Benjamins Publishing Company. 10.1075/nlp.2.04cab
    https://doi.org/10.1075/nlp.2.04cab [Google Scholar]
  10. Cabré, M. Teresa
    1996 “Terminology today.” InTerminology, LSP and Translation: Studies in Language Engineering in Honour of Juan C. Sager, ed. byHarold Somers, 15–35. Amsterdam/Philadelphia: John Benjamins Publishing Company. 10.1075/btl.18.04cab
    https://doi.org/10.1075/btl.18.04cab [Google Scholar]
  11. Cao, Yunbo, and Hang Li
    2002 “Base noun phrase translation using Web data and the EM algorithm.” InProceedings of the 19th International Conference on Computational Linguistics (COLING 2002), Taipei. 10.3115/1072228.1072239
    https://doi.org/10.3115/1072228.1072239 [Google Scholar]
  12. Chung, Teresa Mihwa
    2003 “A corpus comparison approach for terminology extraction.” Terminology9(2): 221–246. 10.1075/term.9.2.05chu
    https://doi.org/10.1075/term.9.2.05chu [Google Scholar]
  13. Daille, Béatrice
    1996 “Study and implementation of combined techniques for automatic extraction of terminology.” InThe Balancing Act: Combining symbolic and statistical approaches to language, ed. byJudith L. Klavans and Philip Resnik, 49–66. Cambridge, MA: MIT Press.
    [Google Scholar]
  14. 2005 “Variations and Application-oriented Terminology Engineering.” Terminology, 11(1): 181–197. 10.1075/term.11.1.08dai
    https://doi.org/10.1075/term.11.1.08dai [Google Scholar]
  15. 2012 “Building bilingual terminologies from comparable corpora: The TTC TermSuite.” InProceedings of the 5th Workshop on Building and Using Comparable Corpora, 29–32. Istanbul, Turkey.
    [Google Scholar]
  16. Daille, Béatrice, and Emmanuel Morin
    2005 “French-English Terminology Extraction from Comparable Corpora.” InNatural Language Processing – IJCNLP 2005. Lecture Notes in Artificial Intelligence, ed. byR. Dale, K-F. Wong, J. Su and O. Y. Kwong, Volume3651, 707–718. Springer-Verlag. 10.1007/11562214_62
    https://doi.org/10.1007/11562214_62 [Google Scholar]
  17. Drouin, Patrick
    2003 “Term extraction using non-technical corpora as a point of leverage.” Terminology9(1): 99–115. 10.1075/term.9.1.06dro
    https://doi.org/10.1075/term.9.1.06dro [Google Scholar]
  18. Erdmann, Maike, Kataro Nakayama, Takahiro Hara, and Shojiro Nishio
    2009 “Improving the Extraction of Bilingual Terminology from Wikipedia.” ACM Transactions on Multimedia Computing, Communications and Applications5(4): Article 31. 10.1145/1596990.1596995
    https://doi.org/10.1145/1596990.1596995 [Google Scholar]
  19. Estopà, Rosa
    2001 « Les unités de signification spécialisées: élargissant l’objet du travail en terminologie » [Units of Specialised Meaning: Broadening the Scope of Terminology Work]. Terminology7(2): 217–237. 10.1075/term.7.2.06est
    https://doi.org/10.1075/term.7.2.06est [Google Scholar]
  20. Fernandez Parra, M., and P. Hacken
    2010 “Identifying Fixed Expressions: A Comparison of SDL MultiTerm Extract and Déjà Vu’s Lexicon.” InProceedings of Translating and the Computer32, ASLIB.
    [Google Scholar]
  21. Foo, J., and M. Merkel
    2010 “Using machine learning to perform automatic term recognition.” InProceedings of the LREC 2010 Workshop on Methods for Automatic Acquisition of Language Resources and their Evaluation Methods, 49–54. Valletta, Malta.
    [Google Scholar]
  22. Fulford, Heather
    2001 “Exploring Terms and Their Linguistic Environment in Text: A Domain-Independent Approach to Automated Term Extraction.” Terminology7(2): 259–279. 10.1075/term.7.2.08ful
    https://doi.org/10.1075/term.7.2.08ful [Google Scholar]
  23. Fung, Pascale
    1998 “A statistical view on bilingual lexicon extraction: from parallel corpora to non-parallel corpora.” Lecture Notes in Artificial Intelligence, Volume 1529, 1–17. Springer. 10.1007/3‑540‑49478‑2_1
    https://doi.org/10.1007/3-540-49478-2_1 [Google Scholar]
  24. Hätty, Anna, and Sabine Schulte im Walde
    2018 “A Laypeople Study on Terminology Identification across Domains and Task Definitions.” InProceedings of the 2018 Conference of the North America Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), 321–326. New Orleans, Louisiana. 10.18653/v1/N18‑2052
    https://doi.org/10.18653/v1/N18-2052 [Google Scholar]
  25. Hazem, Amir, and Emmanuel Morin
    2016 “Efficient Data Selection for Bilingual Terminology Extraction from Comparable Corpora.” InProceedings of the 26th International Conference on Computational Linguistics: Technical Papers, 3401–3411. Osaka, Japan.
    [Google Scholar]
  26. Hazem, Amir, Mérième Bouhandi, Florian Boudin, and Béatrice Daille
    2020 “TermEval 2020: TALN-LS2N System for Automatic Term Extraction.” InProceedings of the 6th International Workshop on Computational Terminology (COMPUTERM 2020), 95–100.
    [Google Scholar]
  27. Hippisley, Andrew R., David Cheng, and Khurshid Ahmad
    2005 “The head-modifier principle and multilingual term extraction.” Natural Language Engineering11(2): 129–157. 10.1017/S1351324904003535
    https://doi.org/10.1017/S1351324904003535 [Google Scholar]
  28. Kageura, Kyo
    1996 “Methods of Automatic Term Recognition – A Review.” Terminology, 3(2): 259–289. 10.1075/term.3.2.03kag
    https://doi.org/10.1075/term.3.2.03kag [Google Scholar]
  29. Kageura, Kyo, Masaharu Yoshioka, Keita Tsuji, Fuyuki Yoshikane, Koichi Takeuchi, and Teruo Koyama
    1999 “Evaluation of the Term Recognition Task.” InProceedings of the First NTCIR Workshop on Research in Japanese Text Retrieval and Term Recognition, 417–434. Tokyo, Japan.
    [Google Scholar]
  30. Kilgarriff, Adam, and Joseph Rosenzweig
    2000 “Framework and results for English SENSEVAL.” Computers and the Humanities34(1–2): 15–48. 10.1023/A:1002693207386
    https://doi.org/10.1023/A:1002693207386 [Google Scholar]
  31. Kim, J.-D., T. Ohta, Y. Tateisi, and J. Tsujii
    2003 “GENIA corpus – a semantically annotated corpus for bio-textmining.” Bioinformatics19(1): i180–i182. 10.1093/bioinformatics/btg1023
    https://doi.org/10.1093/bioinformatics/btg1023 [Google Scholar]
  32. Kit, Chunyu, and Xiaoyue Liu
    2008 “Measuring mono-word termhood by rank difference via corpus comparison.” Terminology14(2): 204–229. 10.1075/term.14.2.05kit
    https://doi.org/10.1075/term.14.2.05kit [Google Scholar]
  33. Krauthammer, Michael, and Goran Nenadić
    2004 “Methodological review: Term identification in the biomedical literature.” Journal of Biomedical Informatics37(6): 512–526. 10.1016/j.jbi.2004.08.004
    https://doi.org/10.1016/j.jbi.2004.08.004 [Google Scholar]
  34. Kwong, Oi Yee
    2018a “Evaluating Term Extraction Tools: System Performance vs Use Perception.” InThe Human Factor in Machine Translation, ed. bySin-Wai Chan. Routledge.
    [Google Scholar]
  35. 2018b “Analysis and Annotation of English-Chinese Financial Terms for Benchmarking and Language Processing.” InProceedings of the First Financial Narrative Processing Workshop (FNP 2018), 10–16. Miyazaki, Japan.
    [Google Scholar]
  36. Kwong, Oi Yee, Benjamin K. Tsou, and Tom B. Y. Lai
    2004 “Alignment and extraction of bilingual legal terminology from context profiles.” Terminology10(1): 81–99. 10.1075/term.10.1.05kwo
    https://doi.org/10.1075/term.10.1.05kwo [Google Scholar]
  37. Laroche, Audrey, and Philippe Langlais
    2010 “Revisiting context-based projection methods for term-translation spotting in comparable corpora.” InProceedings of the 23rd International Conference on Computational Linguistics (COLING 2010), 617–625. Beijing, China.
    [Google Scholar]
  38. Lossio-Ventura, Juan Antonio, Clement Jonquet, Mathieu Roche, and Maguelonne Teisseire
    2016 “Biomedical term extraction: overview and a new methodology.” Information Retrieval Journal19(1): 59–99. 10.1007/s10791‑015‑9262‑2
    https://doi.org/10.1007/s10791-015-9262-2 [Google Scholar]
  39. Macken, Lieve, Els lefever, and Véronique Hoste
    2013 “TExSIS: Bilingual terminology extraction from parallel corpora using chunk-based alignment.” Terminology19(1): 1–30. 10.1075/term.19.1.01mac
    https://doi.org/10.1075/term.19.1.01mac [Google Scholar]
  40. Meyer, Ingrid
    1991 “Knowledge Management for Terminology-Intensive Applications: Needs and Tools.” InProceedings of Workshop on Lexical Semantics and Knowledge Representation, 20–33. Berkeley, California, USA.
    [Google Scholar]
  41. QasemiZadeh, Behrang, and Anne-Kathrin Schumann
    2016 “The ACL RD-TEC 2.0: A Language Resource for Evaluating Term Extraction and Entity Recognition Methods.” InProceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), 1862–1868. Portorož, Slovenia.
    [Google Scholar]
  42. QasemiZadeh, Behrang, and Siegfried Handschuh
    2014 “The ACL RD-TEC: A Dataset for Benchmarking Terminology Extraction and Classification in Computational Linguistics.” InProceedings of the 4th International Workshop on Computational Terminology, 52–63. Dublin, Ireland.
    [Google Scholar]
  43. Quirchmayr, Thomas, Barbara Paech, Roland Kohl, Hannes Karey, and Gunar Kasdepke
    2018 “Semi-automatic rule-based domain terminology and software feature-relevant information extraction from natural language user manuals: An approach and evaluation at Roche Diagnostics GmbH.” Empirical Software Engineering23: 3630–3683. 10.1007/s10664‑018‑9597‑6
    https://doi.org/10.1007/s10664-018-9597-6 [Google Scholar]
  44. Resnik, Philip, and I. Dan Melamed
    1997 “Semi-automatic acquisition of domain-specific translation lexicons.” InProceedings of the Fifth Conference on Applied Natural Language Processing, 340–347. Washington DC, USA. 10.3115/974557.974607
    https://doi.org/10.3115/974557.974607 [Google Scholar]
  45. Rigouts Terryn, Ayla, Véronique Hoste, and Els Lefever
    2019 “In No Uncertain Terms: A Dataset for Monolingual and Multilingual Automatic Term Extraction from Comparable Corpora.” Language Resources and Evaluation54: 385–418. doi:  10.1007/s10579‑019‑09453‑9
    https://doi.org/10.1007/s10579-019-09453-9 [Google Scholar]
  46. Rigouts Terryn, Ayla, Véronique Hoste, Patrick Drouin, and Els Lefever
    2020 “TermEval 2020: Shared Task on Automatic Term Extraction Using the Annotated Corpora for Term Extraction Research (ACTER) Dataset.” InProceedings of the 6th International Workshop on Computational Terminology (COMPUTERM 2020), 85–94.
    [Google Scholar]
  47. Sager, Juan C.
    1990A Practical Course in Terminology Processing. Amsterdam: John Benjamins Publishing Company. 10.1075/z.44
    https://doi.org/10.1075/z.44 [Google Scholar]
  48. Smadja, Frank, Vasileios Hatzivassiloglou, and Kathleen McKeown
    1996 “Translating collocations for bilingual lexicons: A statistical approach.” Computational Linguistics22(1): 1–38.
    [Google Scholar]
  49. Sproat, Richard, and Thomas Emerson
    2003 “The First International Chinese Word Segmentation Bakeoff.” InProceedings of the Second SIGHAN Workshop on Chinese Language Processing, 133–143. Sapporo, Japan. 10.3115/1119250.1119269
    https://doi.org/10.3115/1119250.1119269 [Google Scholar]
  50. Vivaldi, Jorge, and Horacio Rodríguez
    2007 “Evaluation of Terms and Term Extraction Systems: A Practical Approach.” Terminology13(2): 225–248. 10.1075/term.13.2.06viv
    https://doi.org/10.1075/term.13.2.06viv [Google Scholar]
  51. Voorhees, Ellen M., and Donna K. Harman
    (eds) 2005TREC: Experiment and Evaluation in Information Retrieval. Cambridge, MA: The MIT Press.
    [Google Scholar]
  52. Wang, Rui, Wei Liu, and Chris McDonald
    2016 “Featureless Domain-specific Term Extraction with Minimal Labelled Data.” InProceedings of Australasian Language Technology Association Workshop, 103–112.
    [Google Scholar]
  53. Warburton, Kara
    2020 “Supporting Translators through Keyword Mining.” InBook of Abstracts of Translation in Transition (TT5): Human and Machine Intelligence, Virtual Conference, 34–38.
    [Google Scholar]
  54. Xu, Ran, and Serge Sharoff
    2014 “Evaluating Term Extraction Methods for Interpreters.” InProceedings of the 4th International Workshop on Computational Terminology, 86–93. Dublin, Ireland. 10.3115/v1/W14‑4811
    https://doi.org/10.3115/v1/W14-4811 [Google Scholar]
  55. Zaretskaya, Anna
    2017 Translators’ Requirements for Translation Technologies: User Study on Translation Tools. Doctoral Thesis, Universidad de Málaga.
    [Google Scholar]

Data & Media loading...

This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error