James Pustejovsky, Professor of Computer Science, Language and Linguistics, Brandeis University

The weekly 'Issues in Cognitive Science' series provides an informal setting in which faculty, graduate students and invited speakers present their current research or plans for future research, and discuss recent journal articles of general interest to cognitive scientists. The 'catered brown bag' meetings take place every other Wednesday, 12-1:30pm, Sage 4101. Many of these presentations may be viewed in our video archiveGeneral information for speakers

James Pustejovsky, Professor of Computer Science, Language and Linguistics, Brandeis University

Sage 4101

May 24, 2018 12:00 PM - 1:30 PM

The demand for more sophisticated human-computer interactions is rapidly increasing, as users become more accustomed to conversation-like interactions with their devices. In this paper, we examine this changing landscape in the context of human-machine interaction in a shared workspace to achieve a common goal. In our prototype system, people and avatars cooperate to build blocks world structures through the interaction of language, gesture, vision, and action. This provides a platform to study computational issues involved in multimodal communication.  In order to establish elements of the common ground in discourse between speakers, we have created an embodied 3D simulation, enabling both the generation and interpretation of multiple modalities, including: language, gesture, and the visualization of objects moving and agents acting in their environment. The simulation is built on the modeling language VoxML that encodes objects with rich semantic typing and action affordances, and actions themselves as multimodal programs, enabling contextually salient inferences and decisions in the environment.  We illustrate this with a walk-through of multimodal communication in a shared task.

 

Download Paper 1, Paper 2, Paper 3

 

 

 

Add to calendar
Share|