AUTHOR=Krishnaswamy Nikhil , Pustejovsky James 

TITLE=Affordance embeddings for situated language understanding

JOURNAL=Frontiers in Artificial Intelligence

VOLUME=Volume 5 - 2022

YEAR=2022

URL=https://www.frontiersin.org/journals/artificial-intelligence/articles/10.3389/frai.2022.774752

DOI=10.3389/frai.2022.774752

ISSN=2624-8212

ABSTRACT=Much progress in AI over the last decade has been driven by advances in natural language processing technology, in turn facilitated by large datasets and increased computation power used to train large neural language models. These systems demonstrate apparently sophisticated linguistic understanding or generation capabilities, but often fail to transfer their skills to situations they have not encountered before. We argue that computational situated grounding of linguistic information to real or simulated scenarios provide a solution to some of these learning challenges by creating situational representations that both serve as a formal model of the salient phenomena, and contain rich amounts of exploitable, task-appropriate data for training new, flexible computational models. Our model reincorporates methods of structured and symbolic reasoning into the framework of neurosymbolic intelligence, using multimodal contextual modeling of interactive situations, events, and object properties. We discuss how situated grounding provides diverse data and multiple levels of modeling AI learning challenges involving situated grounding, reasoning, and communication, including transferring knowledge of objects and situations to novel entities, and learning how to recognize and generate linguistic and gestural denotations of object affordances.