Johns Hopkins Computer Science Home
Johns Hopkins University The Whiting School of Engineering

Natural Language Processing
Prof. Jason Eisner
Course # 600.465 - Fall 2006

parse trees

Announcements


Vital Statistics [1]

Welcome! This course is designed to introduce you to some of the problems and solutions of NLP, and their relation to linguistics and statistics. You need to know how to program (e.g., 600.120) and use common data structures (600.226). It might also be nice to have some previous familiarity with automata (600.271) and probabilities (550.310). At the end you should agree (I hope!) that language is subtle and interesting, feel some ownership over some of NLP's formal and statistical techniques, and be able to understand research papers in the field.

Course catalog entry: This course is an in-depth overview of techniques for processing human language. How should linguistic structure and meaning be represented? What algorithms can recover them from text? And crucially, how can we build statistical models to choose among the many legal answers? The course covers methods for trees (parsing and semantic interpretation), sequences (finite-state transduction such as morphology), and words (sense and phrase induction), with applications to practical engineering tasks such as information retrieval and extraction, text classification, part-of-speech tagging, speech recognition and machine translation. There are a number of structured but challenging programming assignments. Prerequisite: 600.226. [Eisner, Applications, Fall] 3 credits

Lectures:MTW 2-3 pm, Shaffer 301
Prof:Jason Eisner - (image of email address) ((image of email address))
TA:Omar Zaidan - (image of email address) (ozaidan at cs dot jhu dot edu)
CA: Asheesh Laroia - (image of email address) (asheesh at asheesh dot org)
Constantinos Michael - (image of email address) (stvorph at gmail dot com)
Office hrs: For Prof: MT 3-4pm, or by appt, in NEB 324A
For TA: T 4-5:30, W 12:15-1:45, in undergrad lab (NEB 225)
Mailing list: (image of email address) ... public questions, discussion, announcements
Web page:http://cs.jhu.edu/~jason/465
Textbook: Jurafsky & Martin (required - online partial draft of next edition wants your comments!)
Manning & Schütze (recommended - online PDF version is accessible for free from within JHU)
Policies: Grading: homework 45%, participation 10%, midterm 15%, final 30%
Submission: via this web form
Lateness: floating late days policy
Cheating: here's what it means
Intellectual engagement: much encouraged
Announcements: Read mailing list and this page!
Related
course sites:
List of many courses
Bob Berwick (MIT)
Steven Bird (Penn)
Jan Hajic (JHU)
Alon Lavie (CMU)
Lillian Lee (Cornell)
Chris Manning (Stanford)
Jim Martin (Colorado)
Philip Resnik (Maryland)
Ellen Riloff (Utah)

Schedule

Warning: For future lectures and assignments, the links below take you to last year's versions, which are subject to change.

Week Monday Tuesday Wednesday Readings
9/11 Introduction (ppt)
  • Why is NLP hard?
  • Levels of language
  • NLP applications
  • Random language via n-grams
  • Questionnaire
  • Assignment 1 given: Designing CFGs
    Chomsky hierarchy (ppt)
  • What's wrong with n-grams?
  • Regular expressions, CFGs, & more
  • Lists, trees, and vectors
  • Language models (ppt)
  • Language ID
  • Text categorization
  • Spelling correction
  • Segmentation
  • Speech recognition
  • Machine translation
  • J&M chapters 1, 13, 6.2; for assignment, J&M 9 (or M&S 3)
    9/18 Probability concepts (ppt)
  • Joint & conditional prob
  • Entropy, cross-entropy, perplexity
  • Continuous distributions?
    Bayes' Theorem (ppt)
  • Smoothing n-grams (ppt)
  • Add-one or add-lambda smoothing
  • Good-Turing discounting
  • Smoothing with backoff
  • Deleted interpolation
  • Postponed this material till 9/27
    Human sentence processing (ppt)
  • Methodology
  • Frequency sensitivity
  • Incremental interpretation
  • M&S chapters 2, 6
    9/25
  • Assignment 1 due
        (& another sign meant 3 ... ?)
    Assignment 2 given: Using n-Grams
    Limitations of CFG
  • Discussion of Asst. 1
  • Improving CFG with features (ppt)
  • Morphology
  • Lexicalization
  • Tenses
  • Gaps (slashes)
  • Skipped this material since we were behind
    Extending CFG (summary (ppt))
  • CCG
  • TSG, TAG, TIG
  • J&M 11.1-11.4
    10/2 No class (Yom Kippur)
    But could have a Q&A / homework help session with the TA ...
    Context-free parsing (ppt)
  • What is parsing?
  • Why is it useful?
  • Brute-force algorithm
  • CKY and Earley algorithms
  • Context-free parsing
  • From recognition to parsing
  • Incremental strategy
  • Dotted rules
  • Sparse matrices
  • J&M 10
    10/9 Earley's algorithm (ppt)
  • Top-down parsing
  • Earley's algorithm
  • PCFGs
  • Probabilistic parsing (ppt)
  • PCFG parsing
  • Dependency grammar
  • Lexicalized PCFGs
  • Parsing tricks (ppt)
  • Pruning; best-first
  • Left-corner strategy
  • Rules as regexps
  • Smoothing
  • Evaluation
    Assignment 2 due on Friday --->
  • J&M 12 (or M&S 11.1-11.3 and 12.1.1-12.1.5)
    10/16 No class (fall break) Semantics (ppt)
  • What is understanding?
  • Lambda terms
  • Semantic phenomena and representations
  • Semantics continued
  • Adding semantics to CFG rules
  • Compositional semantics
  • J&M 14-15; also this web page, up to but not including "denotational semantics" section
    10/23 Midterm exam Assignment 3 given: Parsing and Semantics
    Finite-state functions (ppt)
  • Regexp review
  • Properties
  • Functions, relations, composition
  • Simple applications
  • Finite-state implementation (ppt)
  • Finite-state operators
  • Uses of composition
  • Implementing the operators
  • chap 2 of xfst book draft (only accessible from CS research and undergrad networks; don't distribute)
    10/30 Programming with Regexps (ppt)
  • Analogy to programming
  • Extended finite-state operators
  • Date parsing
  • FASTUS
  • Noisy Channels and FSTs (ppt)
  • Regexps and segmentation
  • The noisy channel generalization
  • Applications of the noisy channel
  • Implementation using FSTs
  • Morphology and Phonology (ppt)
  • English, Turkish, Arabic
  • Stemming
  • Compounds, segmentation
  • Two-level morphology
  • Punctuation
  • Rewrite rules
  • OT
  • chap 3 of xfst book draft; perhaps also this paper
    11/6 Finite-state parsing
  • CFG approximation
  • Multipass chunking
  • (Wo)man vs. machine
  • Finite-state tagging (ppt)
  • The task
  • Hidden Markov Models
  • Transformation-based
  • Constraint-based
  • HMMs
  • Tagging continued
  • Chunking
  • Speech
  • J&M 8 or M&S 10
    11/13 Assignment 3 due
    Assignment 4 given: Finite-State Grammars
    Forward-backward algorithm (Excel spreadsheet; Viterbi version; lesson plan)
  • Ice cream, weather, words and tags
  • Forward and backward probabilities
  • Inferring hidden states
  • Controlling the smoothing effect
  • Forward-backward continued
  • Reestimation
  • Likelihood convergence
  • Symmetry breaking
  • Local maxima
  • Uses of states
  • Expectation Maximization (ppt)
  • Generalizing the forward-backward strategy
  • Inside-outside algorithm
  • Allen pp. 195-208 (handout); M&S 11
    11/20 Grouping words (ppt; Excel spreadsheet)
  • Words vs. senses
  • Class-based / HMM
  • Clustering words as vectors
  • More on learning (ppt)
  • Learning in the limit (Gold's theorem)
  • Assignment 4 due
    Assignment 5 given: Training an HMM
    No class (Thanksgiving coming)
    M&S 14
    11/27 Splitting words (ppt)
  • WordNet
  • Supervised disambiguation (e.g., naive Bayes)
  • Unsupervised
  • Words vs. senses in IR (ppt)
  • Information retrieval
  • Query expansion
  • Disambiguation
  • Latent Semantic Analysis
  • Final FSM Examples (ppt)
  • Baby talk
  • Edit distance
  • Back-transliteration
  • Machine translation
  • M&S 7, 5, 15.2, 15.4 (since J&M 16-17 covers only some of this)
    12/4 Machine Translation (ppt)
  • Competitive linking algorithm
  • Translingual projection
  • Noisy channel model
  • Text categorization (ppt)
  • Task and its variants
  • Accuracy, precision, recall ...
  • Clusters, k-nearest neighbor
  • Naive Bayes
  • Decision trees
  • Maximum entropy (ppt) M&S 13, 16
    12/11 Assignment 5 due
    Current and Future Research (ppt)
  • Current state of the art
  • Deploying NLP
  • IE for the masses?
  • Dialogue Systems: Eliza, Loebner Prize, TRIPS
  • Sun 12/17 is absolute deadline for late assignments --->

    Final exam is Thu 12/21, 9am-noon --->
    subject to change