Using Large Corpora

Book excerpt: Using Large Corpora identifies new data-oriented methods for organizing and analyzing large corpora and describes the potential results that the use of large corpora offers. Today, large corpora consisting of hundreds of millions or even billions of words, along with new empirical and statistical methods for organizing and analyzing these data, promise new insights into the use of language. Already, the data extracted from these large corpora reveal that language use is more flexible and complex than most rule-based systems have tried to account for, providing a basis for progress in the performance of Natural Language Processing systems. Using Large Corpora identifies these new data-oriented methods and describes the potential results that the use of large corpora offers. The research described shows that the new methods may offer solutions to key issues of acquisition (automatically identifying and coding information), coverage (accounting for all of the phenomena in a given domain), robustness (accommodating real data that may be corrupt or not accounted for in the model), and extensibility (applying the model and data to a new domain, text, or problem). There are chapters on lexical issues, issues in syntax, and translation topics, as well discussions of the statistics-based vs. rule-based debate. ACL-MIT Series in Natural Language Processing.