Supported by National Science Foundation (2016 – Present)
This research project develops novel causality models for concrete action verbs to capture the intended change of state of the physical world.
It augments meanings of concrete verbs based on how they might change the environment (i.e., causality) and meanings of concrete nouns based on how they might be changed by actions (i.e., affordance). It incorporates causality models into learning and inference algorithms for grounding language to the physical world.
This work will provide a new dimension to connect verb semantics to perception and action. Verb causality models will allow the robot to predict a potential change of state from human linguistic utterances. This prediction will provide top-down information to guide visual processing and action modeling.