IPAB Workshop - 21/11/2024

Speaker: Kevin Denamganai

Title: Reasoning about Physics through Natural Language

Abstract: We aim to build AI systems that can reason about physics. They should be able to do it through natural language interactions with a human user, like in a brainstorming discussion.  However, AI systems struggle with physics understanding due to a lack of grounding. Indeed, they are not embodied and cannot develop intuitive understanding like humans do. Moreover, extensive evidence starts to show that their reasoning capabilities are also limited. We consider reasoning capabilities of different kinds. Either as step-by-step problem solving or as an insights-building exploration process using a mind’s eye.

This presentation will focus on our proposed architecture to address both of those limitations. It leverages application-specific natural language-specified guidelines and physics simulators to build internal representations. A language model then uses them for grounding and reasoning purposes. These internal representations are built at test-time, as needed, using approximate, cheap simulation roll-outs, on little computing resources. Test-time adaptation can be achieved by changing the application-specific simulator and/or guidelines. As a result, our architecture gains in versatility.

This presentation will detail two applications. First, an application on the game of Pool. It involves complex physics of rigid- and deformable-bodies, and abstract strategies common to multi-players games. Then, an application on a cartpole system, which only involves rigid-body physics. Each application investigates, respectively, one of the above limitations, in grounding or in reasoning.