Friday, 20th September - 11am Mario Giulianelli : Seminar Title: From Information Rate to Information Value: Enhancing Information-Theoretic Models of Online Language Processing Abstract In this talk, I will outline the development from the classic information-theoretic model of linguistic communication to a new framework with improved theoretical properties. I will demonstrate how this new framework results in improved predictions for both behavioural and neural markers of human language processing. The traditional information-theoretic framework posits a relationship between predictive uncertainty, measured as the next-unit surprisal of the human language model, and processing cost—a link supported by extensive empirical evidence. I will start by challenging a core prediction of this model: that information rate should be constant or uniform in dialogue. Using neural-network language models as a proxy for the human construct model, I show that, contrary to this assumption, information rate often decreases throughout interactions, with speakers progressively reducing the processing cost for their interlocutors (Giulianelli & Fernández, 2021, Giulianelli et al. 2021, Giulianelli et al. 2022, Yee et al. 2024). I will then introduce a generalisation of surprisal that enables a family of measures of predictive uncertainty, computable with modern language models through Monte Carlo simulation. I will focus, in particular, on information value (Giulianelli et al., 2023), a more nuanced measure of processing cost that considers the communicative equivalence of linguistic expressions and provides better predictions for human comprehension behaviour. Finally, I will present the Incremental Alternative Sampling model, an extension of information value that incorporates the representational and temporal dimensions of online linguistic prediction (Giulianelli et al. 2024). This model demonstrates superior predictive power compared to surprisal for most neural and behavioural measurements of processing cost and offers new insights into the predictive mechanisms driving language comprehension. Altogether, this work advances our understanding of the information processing mechanisms underlying linguistic communication, laying the groundwork for more refined computational models and a deeper understanding of the cognitive processes that enable language use. Bio Mario is a postdoctoral fellow at ETH Zürich, working in the Institute for Machine Learning within the Department of Computer Science. He is also an associated researcher at the ETH AI Center and a member of the ELLIS Society. Mario’s research integrates natural language processing, machine learning, and information theory to study language interaction in humans and computational models. Sep 20 2024 11.00 - 12.00 Friday, 20th September - 11am Mario Giulianelli : Seminar This event is co-organised by ILCC and by the UKRI Centre for Doctoral Training in Natural Language Processing, https://nlp-cdt.ac.uk. IF G.03 and on Teams Contact
Friday, 20th September - 11am Mario Giulianelli : Seminar Title: From Information Rate to Information Value: Enhancing Information-Theoretic Models of Online Language Processing Abstract In this talk, I will outline the development from the classic information-theoretic model of linguistic communication to a new framework with improved theoretical properties. I will demonstrate how this new framework results in improved predictions for both behavioural and neural markers of human language processing. The traditional information-theoretic framework posits a relationship between predictive uncertainty, measured as the next-unit surprisal of the human language model, and processing cost—a link supported by extensive empirical evidence. I will start by challenging a core prediction of this model: that information rate should be constant or uniform in dialogue. Using neural-network language models as a proxy for the human construct model, I show that, contrary to this assumption, information rate often decreases throughout interactions, with speakers progressively reducing the processing cost for their interlocutors (Giulianelli & Fernández, 2021, Giulianelli et al. 2021, Giulianelli et al. 2022, Yee et al. 2024). I will then introduce a generalisation of surprisal that enables a family of measures of predictive uncertainty, computable with modern language models through Monte Carlo simulation. I will focus, in particular, on information value (Giulianelli et al., 2023), a more nuanced measure of processing cost that considers the communicative equivalence of linguistic expressions and provides better predictions for human comprehension behaviour. Finally, I will present the Incremental Alternative Sampling model, an extension of information value that incorporates the representational and temporal dimensions of online linguistic prediction (Giulianelli et al. 2024). This model demonstrates superior predictive power compared to surprisal for most neural and behavioural measurements of processing cost and offers new insights into the predictive mechanisms driving language comprehension. Altogether, this work advances our understanding of the information processing mechanisms underlying linguistic communication, laying the groundwork for more refined computational models and a deeper understanding of the cognitive processes that enable language use. Bio Mario is a postdoctoral fellow at ETH Zürich, working in the Institute for Machine Learning within the Department of Computer Science. He is also an associated researcher at the ETH AI Center and a member of the ELLIS Society. Mario’s research integrates natural language processing, machine learning, and information theory to study language interaction in humans and computational models. Sep 20 2024 11.00 - 12.00 Friday, 20th September - 11am Mario Giulianelli : Seminar This event is co-organised by ILCC and by the UKRI Centre for Doctoral Training in Natural Language Processing, https://nlp-cdt.ac.uk. IF G.03 and on Teams Contact
Sep 20 2024 11.00 - 12.00 Friday, 20th September - 11am Mario Giulianelli : Seminar This event is co-organised by ILCC and by the UKRI Centre for Doctoral Training in Natural Language Processing, https://nlp-cdt.ac.uk.
