1887
Volume 10, Issue 1
  • ISSN 1871-1340
  • E-ISSN: 1871-1375
USD
Buy:$35.00 + Taxes

Abstract

This short paper discusses shortcomings of the capture-recapture (CR) method of estimating vocabulary size (Meara & Olmos Alcoy, 2010; Williams, Segalowitz & Leclair, 2014). When sampling from a population generated by a power-law process (e.g., a Zipf distribution), the probability that any given member is selected is dependent on its rank, such that higher frequency rank (i.e., 1st, 2nd, 3rd) members are much more likely to be selected than lower rank (i.e., 100th, 1000th) members. Because of this, sampling is much more likely to select from the same limited group of words. The CR measure, however, assumes a uniform distribution, and so drastically underestimates the size of the vocabulary when applied to power-law data. Work with simulated data shows ways that the degree of underestimation may be lessened. Applying these methods to real data shows effects parallel to those in the simulations.
Loading

Article metrics loading...

/content/journals/10.1075/ml.10.1.06nel
2015-01-01
2019-10-22
Loading full text...

Full text loading...

References

http://instance.metastore.ingenta.com/content/journals/10.1075/ml.10.1.06nel
Loading
  • Article Type: Research Article
Keyword(s): corpora , measurement , vocabulary and Zipf’s law
This is a required field
Please enter a valid email address
Approval was successful
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error