Introduction to Probability and Information Theory

Linguistics as field is in the process of undergoing a dramatic methodological shift. Quantitative advances in allied fields (such as psycholinguistics, sociolinguistics, and computational linguistics) have led to a new interest in empirical methods among theoretical linguists. In order to understand and contribute to current research in linguistics, practitioners need a solid grounding in probability, statistics and information theory, areas which get little attention in traditional (under)graduate linguistics education. This course aims to fill that gap, giving students some of the mathematical background they need to be a linguist in the 21st century.

Readings

• Charles Grinstead and J. Laurie Snell. 1997. Introduction to Probability. American Mathematical Society. [pdf]

• John Verzani. 2001. simpleR: Using R for Introductory Statistics. Math dept., College of Staten Island, CUNY. [pdf]

• John R. Pierce. 1980. An Introduction to Information Theory. Dover Publications. [ch2, ch3, ch4, ch5, ch6, ch7, ch8]

• R. Harald Baayen. 2001. Word Frequency Distributions. Kluwer. [ch1]

• R. Harald Baayen. 2003. "Probabilistic approaches to morphology." In Rens Bod, Jennifer Hay, Stefanie Jannedy (eds.) 2003. Probabilistic Linguistics. MIT Press. [pdf]

Syllabus

1. Probability and counting [slides]
2. Conditional probability, Bayes' Theorem [slides]
3. Probability distributions [slides]
4. Statistics and hypothesis testing [slides]
5. Word frequencies [slides]
6. Information theory: entropy and compression [slides]
7. Information theory: noisy channels and complexity [slides]
8. Applications to morphology [slides]