International Journal of Corpus Linguistics - Volume 10, Issue 4, 2005

This article looks at how a comprehensive list of one category of idioms, that of ‘core idioms’, was established. When the criteria to define a core idiom were strictly applied to a dictionary of idioms, the result was that the large number of ‘idioms’ was reduced to a small number of ‘core idioms’. The original list from the first source dictionary was added to by applying the same criteria to other idiom dictionaries, and other sources of idioms. Once the list was complete, a corpus search of the final total of 104 ‘core idioms’ was carried out in the British National Corpus (BNC). The search revealed that none of the 104 core idioms occurs frequently enough to merit inclusion in the 5,000 most frequent words of English.

With a brief outline of the history of frequency dictionaries of the past, a renewed need for these dictionaries, based on modern large corpora, is observed. A linguistic background and some criticism of what has been done is offered. It is stressed that, next to the traditional and predominant paradigmatic approach, at least some attention should be paid to syntagmatic or combinatorial aspects of the lexis as well, as these are readily available in modern corpora. Against this background, solution to this need and projection of corpus possibilities is presented as it has been realized in the recent large Frequency Dictionary of Czech (Frekvencní slovník ceštiny). A brief discussion of the linguistic and lexicographic problems is followed by a brief step-by-step description of technical solution.

The purpose of this paper is to discuss actual usage of the two Korean causal conjunctive suffixes, -(e)seand-(u)nikka, and to propose their multi-layered semantics based on analysis of corpus data. To account for the functional differences of the two conjunctives, most previous studies focused on different syntactic distributions or semantic contrast by employing an objectivist viewpoint, failing to incorporate the polyfunctionality, semantic overlapping and pragmatic ambiguities of them. This paper advances that the meanings of the two causal suffixes are distributed on four different cognitive-discourse levels: content, epistemic, speech act, and discourse level. Corpus analysis does reveal that all four levels are accessible to both conjunctive suffixes but the difference between the two suffixes lies in the different degree of accessibility of these four levels in their sentence semantics. This finding suggests that we treat these linguistic categories more flexibly by accepting their gradient and pragmatically ambiguous status.

A chatbot is a machine conversation system which interacts with human users via natural conversational language. Software to machine-learn conversational patterns from a transcribed dialogue corpus has been used to generate a range of chatbots speaking various languages and sublanguages including varieties of English, as well as French, Arabic and Afrikaans. This paper presents a program to learn from spoken transcripts of the Dialogue Diversity Corpus of English, the Minnesota French Corpus, the Corpus of Spoken Afrikaans, the Qur'an Arabic-English parallel corpus, and the British National Corpus of English; we discuss the problems which arose during learning and testing. Two main goals were achieved from the automation process. One was the ability to generate different versions of the chatbot in different languages, bringing chatbot technology to languages with few if any NLP resources: the corpus-based learning techniques transferred straightforwardly to develop chatbots for Afrikaans and Qur'anic Arabic. The second achievement was the ability to learn a very large number of categories within a short time, saving effort and errors in doing such work manually: we generated more than one million AIML categories or conversation-rules from the BNC corpus, 20 times the size of existing AIML rule-sets, and probably the biggest AI Knowledge-Base ever.

The Web has recently been used as a corpus for linguistic investigations, often with the help of a commercial search engine. We discuss some potential problems with collecting data from commercial search engine and with using the Web as a corpus. We outline an alternative strategy for data collection, using a personal Web crawler. As a case study, the university Web sites of three nations (Australia, New Zealand and the UK) were crawled. The most frequent words were broadly consistent with non-Web written English, but with some academic-related words amongst the top 50 most frequent. It was also evident that the university Web sites contained a significant amount of non-English text, and academic Web English seems to be more future-oriented than British National Corpus written English.