6.835 Intelligent Multimodal Interfaces

Prerequisites: 6.034, fluency in Python, 6.813 is useful but not required.

G (Spring)
3-0-9 units

Implementation and evaluation of intelligent multi-modal user interfaces, taught from a combination of hands-on exercises and papers from the original literature. Topics include basic technologies for handling speech, vision, pen-based interaction, and other modalities, as well as various techniques for combining modalities. Substantial readings and a term project, where students build an interface to illustrate one or more themes of the course.

Work in the course has several components:

  • Readings: There are substantial readings from original literature that go with each lecture; you are expected to read them in preparation for the lecture.
  • Reading summary assignments: There are short written assignments covering the readings for a particular lecture (posted on the Stellar site); they will help you focus on important issues in papers that are sometimes overly dense. You are expected to write about a half page (in total) and to turn in a hardcopy (yes, really, hardcopy) at the beginning of the lecture covering that topic.
  • Lectures: You are expected to have done the reading associated with a lecture and to participate in class discussion. You are responsible for what happens in class; while the course has a web site, not everything announced in class will necessarily appear on the web site.
  • Projects: There will be four mini-projects of increasing complexity during the term. They involve working with real data in both single and multiple modalities, using a variety of tools.
  • Term project: you will be expected to build something appropriate for the course (i.e., a program that demonstrates novel interaction). We will distribute a list of previous projects as a source of inspiration and to help with brainstorming, and will gladly help you select and refine your proposed project idea.
  • Course load: the classroom and mini-project work is front-loaded in the course, with the second half of the course focused on term project development, with studio sessions for feedback on your designs and demonstrations of your systems as they evolve.
  • Project presentation: The last two or three classes will be devoted to brief presentations of your final projects.
  • Overlap with your thesis, etc.: we have no problem if your term project for this course overlaps with your thesis research or other academic requirements, as long as it is appropriate for this course.

Grades in the course will be determined by a combination of the above along with participation in class.

Sample Syllabus, from Spring 2018

(The course is run off of Stellar; the appropriate Stellar pages will be available each year by late January.)

February  2018

  • 06 Tue Introduction and Course Organization
  • 08 Thu Interfaces: dumb/smart, unnatural/natural, multimodal
  • 13 Tue Sketch Understanding: strokes as gestures, ink, and drawings
  • 13 Tue Mini Project 1   assignment posted
  • 15 Thu Sketch Understanding: temporal and visual representations
  • 20 Tue No Class: Virtual Monday
  • 22 Thu Body Tracking and Gesture Recognition
  • 22 Thu Term Project Proposal   assignment posted
  • 23 Fri  Mini Project 2   assignment posted
  • 23 Fri  Mini Project 1   assignment due
  • 27 Tue Gestures: natural and artificial

March  2018

  • 01 Thu Face Detection and Recognition
  • 01 Thu Term Project Proposal   assignment due
  • 05 Mon Mini Project 3   assignment posted
  • 05 Mon Mini Project 2   assignment due
  • 06 Tue Affective Computing
  • 08 Thu Speech Understanding
  • 12 Mon Mini Project 3   assignment due
  • 13 Tue Cross-modal Understanding: speech and writing; Multimodal Fusion
  • 13 Tue Mini Project 4   assignment posted
  • 15 Thu Brain-based Interaction
  • 20 Tue Design Studio: feedback on term project conceptions
  • 22 Thu Design Studio: feedback on term project conceptions
  • 25 Sun Mini Project 4   assignment due
  • 27 Tue No Class: Spring Break
  • 29 Thu No Class: Spring Break

April  2018

  • 03 Tue Cross-modal understanding; multimodal grounding of speech
  • 05 Thu Virtual Reality and Physical Reality
  • 10 Tue Prototype Studio: demonstrations of project prototypes and feedback
  • 12 Thu Prototype Studio: demonstrations of project prototypes and feedback
  • 17 Tue No Class: Patriot’s Day
  • 19 Thu Designing and executing a user study
  • 24 Tue Implementation Studio: demonstration and feedback on first complete run-through
  • 26 Thu Implementation Studio: demonstration and feedback on first complete run-through

May  2018

  • 01 Tue Haptic Interaction
  • 01 Tue Creating a user study plan   assignment due
  • 03 Thu No Class: run your user study
  • 08 Tue No Class (user study report due)   assignment due
  • 10 Thu Final Presentations
  • 15 Tue Final Presentations
  • 17 Thu Final Presentations
  • 18 Fri Term Paper Submission   assignment due
  • 18 Fri Project Code Submission   assignment due
  • 18 Fri Detailed system information   assignment due
  • 18 Fri Final presentation video   assignment due