Friday 26th September 2025 - 11am

Speaker: Vagrant Gautam (Heidelberg Institute for Theoretical Studies)

Title: Evaluating Reference in NLP

Abstract: Names (“Vagrant”), pronouns (“they”), and definite descriptions (“the birder”) are examples of referring expressions, linguistic forms that point to referents. The complexity of reference lies in how we can, in different contexts, map the same referring expression to different individuals, and the same individual to different referring expressions. This makes reasoning about reference a challenge for systems that deal exclusively with linguistic forms, despite impressive advances in natural language processing. In the technical parts of this talk, I will present evaluations of LLMs on reference in English: When multiple individuals are involved, with different architectures, and using probability- as well as generation-based evaluations. Throughout, I will interweave the social aspects of evaluating reference in NLP, including the social impacts of incorrect reference (e.g., misgendering), and the difficulties of defining and measuring “correct” reference in the first place.

Biography: Vagrant Gautam is the 2025 recipient of the Independent Postdoctoral Fellowship at the Heidelberg Institute for Theoretical Studies, prior to which they did their PhD at Saarland University. Their interdisciplinary research is broadly about social and technical aspects of trustworthy natural language processing – especially fairness, reasoning and robustness. On the technical side, they propose methods for and evaluations of these aspects inspired by linguistics. On the social side, they investigate gaps and opportunities in how we define and operationalize abstract concepts like ‘gender’, 'bias', etc., as well their societal implications.