Past Projects

This project will build and develop JOKER, a generic intelligent user interface providing a multimodal dialogue system with social communication skills including humor, empathy, compassion, charm, and other informal socially-oriented behavior.

Talk during social interactions naturally involves the exchange of propositional content but also and perhaps more importantly the expression of interpersonal relationships, as well as displays of emotion, affect, interest, etc. This project will facilitate advanced dialogues employing complex social behaviors in order to provide a companion-machine (robot or ECA) with the skills to create and maintain a long term social relationship through verbal and non verbal language interaction. Such social interaction requires that the robot has the ability to represent and understand some complex human social behavior. It is not straightforward to design a robot with such abilities. Social interactions require social intelligence and ‘understanding’ (for planning ahead and dealing with new circumstances) and employ theory of mind for inferring the cognitive states of another person.

JOKER will emphasize the fusion of verbal and non-verbal channels for emotional and social behavior perception, interaction and generation capabilities. Our paradigm invokes two types of decision: intuitive (mainly based upon non-verbal multimodal cues) and cognitive (based upon fusion of semantic and contextual information with non-verbal multimodal cues.) The intuitive type will be used dynamically in the interaction at the non-verbal level (empathic behavior: synchrony of mimics such as smile, nods) but also at verbal levels for reflex small- talk (politeness behavior: verbal synchrony with hello, how are you, thanks, etc). Cognitive decisions will be used for reasoning on the strategy of the dialog and deciding more complex social behaviors (humor, compassion, white lies, etc.) taking into account the user profile and contextual information.

JOKER will react in real-time with a robust perception module (sensing user’s facial expressions, gaze, voice, audio and speech style and content), a social interaction module modelling user and context, with long-term memories, and a generation and synthesis module for maintaining social engagement with the user.

The research will provide a generic intelligent user interface for use with various platforms such as robots or ECAs, a collection of multimodal data with different socially-oriented behavior scenarios in two languages (French and English) and an evaluation protocol for such systems. Using the database collected in a human-machine context, cultural aspects of emotions and natural social interaction including chat, jokes, and other informal socially-oriented behavior will be incorporated.

Go to the project page.

The IMOTION project develops and evaluates innovative multi-modal user interfaces for interacting with augmented videos. Starting with an extension of existing query paradigms (keyword search in manual annotations), image search (query by example in key frames), IMOTION considers novel sketch- and speech-based user interfaces.

Go to the project page

The goal of this project is to build the pen-based interfaces for the classroom of the future. Currently there is little interaction and personalized feedback between instructors and pupils. We use realtime processing of pen input to create consolidated representations of student interactions and allow teachers to give timely and to-the-point feedback to students to enhance the learning experience.

TVs are slowly morphing into powerful set-top computers with internet connections. As such, they slowly take over roles and functions that were traditionally associated with desktop computers. TV users, for example, can use their TV for browsing the internet. Unfortunately, the vast majority of the content in the internet has been designed for desktop viewing, hence they have to be adapted for viewing on a TV. In this Project, we aim to develop a semi-automatic content retargeting system, which is expected to work with minimal intervention of an expert.

The main goal of this project is to develop a computer software program that will assist children with Autism Spectrum Conditions (ASC) to understand and express emotions through facial expressions, tone-of-voice and body gestures.This software will assist them to understand and interact with other people, and as a result, will increase their inclusion in society.

Go to the project page

Deep Green is a project that ran under the Information Processing Technology Office of the Defense Advanced Research Projects Agency. The purpose of the project was to develop a decision-making support system for United States Army commanders. The systems developed feature advanced predictive capabilities to enable computers to efficiently and accurately predict possible future scenarios, based on an analysis of the current situation, in order to give army commanders a better view of possible outcomes for their decisions [1][2][3] Deep Green is composed of four major components: Blitzkrieg – Battlefield model which analyzes current situation and determines possible future outcomes for use in planning. When a plan is presented, Blitzkrieg analyzes the plan to point out possible results of that course of action to the commander. Blitzkrieg itself does not do planning, it merely determines the likely results of a plan formulated by a human commander. Crystal Ball – Performs analysis of possible futures generated from blitzkrieg, and determines the “best” choices by measuring flexibility, usefulness, and likelihood of each. It picks the best of these choices and presents them to the commander. Also updates model of battlefield situation with information pulled from the field. This might include reports from soldiers, through a program similar to the Communicator program that was developed under the Information Awareness Office or through automated RSTA systems such as HART. Commander’s Associate – this is the user interface and visualization component. It consists of “Sketch-to-decide” which presents the commander with a list of options, and “Sketch-to-plan” which is a screen on which the commander can draw up a plan, which Deep Green will interpret and put into action

Go to the project page