6.891 Machine Learning Approaches for Natural Language Processing, Fall 2003

6.891 (Fall 2003): Machine Learning Approaches for Natural Language Processing

Instructor: Michael Collins (home)

Class times: Monday, Wednesday 4-5.30, Room 1-379.

Office hours: Thursdays 10.30-12, Room NE43-723. If you'd like to meet with me at other times, please send me email at mcollins at ai dot mit dot edu.

Course description

General information (grading policy etc.) .ps, .pdf.

New Announcements (November 5th, 2003)

The instructions for Assignment II (the final project) are now posted here. Please read the instructions carefully! The project is due by Wednesday, December 10th.

Lecture Slides

Lecture 1 (9/3/03):	Introduction .ps,.pdf
Lecture 2 (9/8/03):	Language Modeling, and Smoothed estimation .ps,.pdf
Lecture 3 (9/10/03):	Stochastic Parsing I .ps,.pdf
Lecture 4 (9/15/03):	Stochastic Parsing II .ps,.pdf
Lecture 5 (9/17/03):	Stochastic Parsing III .ps,.pdf
Lecture 6 (9/24/03):	Log-Linear Models .ps,.pdf
Lecture 7 (9/29/03):	Tagging .ps,.pdf
Lecture 8 (10/1/03):	Log-Linear Models for Parsing, and the EM Algorithm Part I .ps,.pdf
Lecture 9 (10/6/03):	The EM Algorithm, Part II .ps,.pdf
Lecture 10 (10/8/03):	Machine Translation, Part I .ps,.pdf
Lecture 11 (10/15/03):	Machine Translation, Part II .ps,.pdf
Lecture 12 (10/20/03):	Machine Translation, Part III .ps,.pdf
Lecture 13 (10/22/03):	Machine Translation, Part IV .ps,.pdf
Lecture 14 (10/27/03):	Global Linear Models, Part I .ps,.pdf
Lecture 15 (10/29/03):	Global Linear Models, Part II .ps,.pdf
Lecture 16 (11/3/03):	Global Linear Models, Part III .ps,.pdf
Lecture 17 (11/5/03):	Theories of Generalization .ps,.pdf
Lecture 18 (11/12/03):	Word Sense Disambiguation .ps,.pdf
Lecture 19 (11/17/03):	Guest Lecture (Regina Barzilay)
Lecture 20 (11/19/03):	Information Extraction, and Partially Supervised Approaches .ps,.pdf
Lecture 21 (11/24/03):	Relation Extraction .ps,.pdf
Lecture 22 (12/1/03):	Dialogue Systems .ps,.pdf
Lecture 23 (12/3/03):	Optimality Theory .ps,.pdf
Lecture 24 (12/8/03):	Kernel Methods .ps,.pdf

Lecture Slides + Lecture Readings:

Lecture 1 (3rd September, 2003): Introduction

Slides .ps, .pdf.

Syllabus .ps, .pdf.

Questionnaire .ps, .pdf.

Lecture 2 (8th September, 2003): Language Modeling, and Smoothed estimation

Slides .ps, .pdf.

Lecture 3 (10th September, 2003): Stochastic Parsing I

Slides .ps, .pdf.

Lecture 4 (15th September, 2003): Stochastic Parsing II

Slides .ps, .pdf.

Two readings for the lecture:

Statistical parsing with a context-free grammar and word statistics. Eugene Charniak. 1997.Proceedings of the Fourteenth National Conference on Artificial Intelligence AAAI Press/MIT Press, Menlo Park (1997). pdf version.
Three Generative, Lexicalised Models for Statistical Parsing. Michael Collins. 1997. In proceedings of ACL 1997. pdf version.

Lecture 5 (17th September, 2003): Stochastic Parsing III

Slides .ps, .pdf.

Lecture 6 (24th September, 2003): Log-Linear Models

Slides .ps, .pdf.

There is one reading for the lecture:

A maximum entropy approach to natural language processing . Adam Berger, Stephen Della Pietra, and Vincent Della Pietra. Computational Linguistics, (22-1), March 1996.
A couple of notes about the reading: there appears to be an error in section 3.4, the derivation of the primal/dual relationship (there's a sign flipped in equation 11). So don't worry too much about this derivation -- we'll go through some of it in class. Also, the paper emphasizes the "maximum entropy" property of these models, whereas I'll emphasize the parametric/maximum-likelihood interpretation (section 3.5) in class.

Lecture 7 (29th September, 2003): Tagging

Slides .ps, .pdf.

Two readings for the lecture:

Please read chapter 5 of Maximum entropy models for natural language ambiguity resolution. Adwait Ratnaparkhi. 1998. PhD thesis, University of Pennsylvania. pdf version.
Maximum Entropy Markov Models for Information Extraction and Segmentation. Andrew Mccallum, Dayne Freitag, Fernando Pereira. 2000. Machine Learning: Proceedings of the Seventeenth International Conference (ICML 2000)

Lecture 8 (1st October, 2003): Log-Linear Models for Parsing, and the EM Algorithm Part I

Slides .ps, .pdf.

Notes on Lagrange multipliers (thanks to Ali Mohammad for putting these together): .ps, .pdf.

Lecture 9 (6th October, 2003):The EM Algorithm, Part II

Slides .ps, .pdf.

Lecture 10 (8th October, 2003): Machine Translation, Part I

Slides .ps, .pdf.

There are two readings for the lecture. Please note: I've given both pdf and ps versions of each paper, but the .ps versions may be better (the conversion from postscript to pdf is not always perfect). If you can't view postscript files, you may want to download a copy of ghostview (try a search for ghostview on google).

A survey of current paradigms in machine translation. Bonnie Dorr, Pamela Jordan, and John Benoit. Advances in Computers, Vol 49, M. Zelkowitz (Ed), Academic Press, London, pp. 1--68, 1999. .ps version
Automating Knowledge Acquisition for Machine Translation. Kevin Knight. AI Magazine 18(4), 1997. ps version. I thought this was a fun article! If you can, try following the Centauri/Arcturan translation example with some care as you read the text!

Lecture 11 (15th October, 2003): Machine Translation, Part II

Slides .ps, .pdf.

Lecture 12 (20th October, 2003): Machine Translation, Part III

Slides .ps, .pdf.

A reading for the lecture:

Fast Decoding and Optimal Decoding for Machine Translation. U. Germann, M. Jahr, K. Knight, D. Marcu, and K. Yamada. Proc. of the Conference of the Association for Computational Linguistics (ACL), 2001. Best Paper award.

Lecture 13 (22nd October, 2003): Machine Translation, Part IV

Slides .ps, .pdf.

Lecture 14 (27th October, 2003): Global Linear Models, Part I

Slides .ps, .pdf.

Lecture 15 (29th October, 2003): Global Linear Models, Part II

Slides .ps, .pdf.

Lecture 16 (3rd November, 2003): Global Linear Models, Part III

Slides .ps, .pdf.

Lecture 17 (5th November, 2003): Theories of Generalization

Slides .ps, .pdf.

Lecture 18 (12th November, 2003): Word Sense Disambiguation

Slides .ps, .pdf.

6.891 (Fall 2003): Machine Learning Approaches for Natural Language Processing

New Announcements (November 5th, 2003)

Lecture Slides

Lecture Slides + Lecture Readings:

Lecture 1 (3rd September, 2003): Introduction

Lecture 2 (8th September, 2003): Language Modeling, and Smoothed estimation

Lecture 3 (10th September, 2003): Stochastic Parsing I

Lecture 4 (15th September, 2003): Stochastic Parsing II

Lecture 5 (17th September, 2003): Stochastic Parsing III

Lecture 6 (24th September, 2003): Log-Linear Models

Lecture 7 (29th September, 2003): Tagging

Lecture 8 (1st October, 2003): Log-Linear Models for Parsing, and the EM Algorithm Part I

Lecture 9 (6th October, 2003):The EM Algorithm, Part II

Lecture 10 (8th October, 2003): Machine Translation, Part I

Lecture 11 (15th October, 2003): Machine Translation, Part II

Lecture 12 (20th October, 2003): Machine Translation, Part III

Lecture 13 (22nd October, 2003): Machine Translation, Part IV

Lecture 14 (27th October, 2003): Global Linear Models, Part I

Lecture 15 (29th October, 2003): Global Linear Models, Part II

Lecture 16 (3rd November, 2003): Global Linear Models, Part III

Lecture 17 (5th November, 2003): Theories of Generalization

Lecture 18 (12th November, 2003): Word Sense Disambiguation

Lecture 19 (17th November, 2003): Guest Lecture (Regina Barzilay)

Lecture 20 (19th November, 2003): Information Extraction, and Partially Supervised Approaches

Lecture 21 (24th November, 2003): Relation Extraction

Lecture 22 (1st December, 2003): Dialogue Systems

Lecture 23 (3rd December, 2003): Optimality Theory

Lecture 24 (8th December, 2003): Kernel Methods