Natural Language Processing and Computational Linguistics

Concurrency in (computational) linguistics

Improving understanding of synchronic and diachronic aspects of phonology.

Supervisor: Julian Bradfield

In several aspects of linguistic analysis, it is natural to think of some form of concurrent processing, not least because the brain is a massively concurrent system. This is particularly true in phonology and phonetics, and descriptions such as feature analyses, and especially autosegmental phonology, go some way to recognizing this. Although there has been some work on rigorous formal models of such descriptions, there has been little if any application of the extensive body of research in theoretical computer science on concurrent processes. Such a project has the potential to give better linguistic understanding of synchronic and diachronic aspects of phonology and perhaps syntax, and even to improve speech generation and recognition, by adding formal underpinning and improvement to the existing agent-based approaches.

Topics in morphology (NLP or cognitive modelling)

Supevisor:  Sharon Goldwater

Many NLP systems developed for English ignore the morphological structure of words and (mostly) get away with it. Yet morphology is far more important in many other languages. Handling morphology appropriately can reduce sparse data problems in NLP, and understanding human knowledge of morphology is a long-standing scientific question in cognitive science. New methods in both probabilistic modeling and neural networks have the potential to improve word representations for downstream NLP tasks and perhaps to shed light on human morphological acquisition and processing. Projects in this area could involve working to combine distributional syntactic/semantic information with morphological information to improve word representations for low-resource languages or sparse datasets, evaluating new or existing models of morphology against human behavioral benchmarks, or related topics.

Adapting NLP interpretability for deep speech models

Supervisor: Sharon Goldwater

Deep models for speech and language are incredibly successful, yet the nature of their learned representations, and how those representations contribute to model behavior, are still poorly understood. Work on interpretability in NLP and machine learning has begun to make progress on this question, for example by showing that features and concepts are often represented in low-dimensional subspaces which can be manipulated to change the model's representation of that feature/concept without affecting other aspects of the representation (a "causal intervention"). There is a considerable literature on interpretability and causal interventions in text-based language models, yet little comparable work on speech-based models. Projects in this space will seek to address this gap. Research questions could include: How do speech models maintain phonetic or other context (such as speaker properties) over time? How and to what extent are larger units (such as words) represented in self-supervised representations? Projects could also focus more on methodology, for example looking at how intervention methods designed for text models (which take symbolic input and are fully generative) could be adapted to speech models (which have neither property)? Inspiration may come not just from NLP and machine learning but also from the human speech perception literature, and projects in this area could lean more toward AI interpretability or cognitive science, depending on student interest.

Neural Network Models of Human Language and Visual Processing

Supervisor: Frank Keller

Recent neural models have used attention mechanisms as a way of focusing the processing of a neural networks on certain parts of the input. This has proved successful for diverse applications such as image description, question answering, or machine translation. Attention is also a natural way of understanding human cognitive processing: during language processing, humans attend words in a certain order; during visual processing, they view image regions in a certain sequence. Crucially, human attention can be captured precisely using an eye-tracker, a device that measures which parts of the input the eye fixates, and for how long. Projects within this area will leverage neural attention mechanisms to model aspects of human attention. Examples include reading: when reading text, humans systematically skip words, spend more time on difficult words, and sometimes re-read passages. Another example is visual search: when looking for a target, human make a sequence of fixations which depend a diverse range of factors, such as visual salience, scene type, and object context. Neural attention models that capture such behaviors need to combine different types of knowledge, while also offering a cognitively plausible story how such knowledge is acquired, often based on only small amounts of training data.

Neural Network Models of Long-form, Multimodal Narratives

Supervisor: Frank Keller

Deep learning approaches are very successful in classical NLP tasks. However, they often assume a limited input context, and are designed to work on short texts. Standard architectures such as LSTMs or transformers therefore struggle to process long-form narratives such as books or screenplays. Prior work shows that such narratives have a particular structure, which can be analyzed in terms of events and characters. This structure can be used for applications such as question answering or summarization of long-form texts. Projects within this area will leverage recent advances in language modeling, such as retrieval-based or memory-based models, to analyze narrative structure. The analysis can take the form of sequences or graphs linking events or characters. Based on such structures, higher level concepts (e.g., schemas or tropes) can be identified, and user reaction such as suspense, surprise, or sentiment can be predicted. Multimodal narratives (illustrated stories, comics, or movies) pose a particular challenge, as narrative elements need to be grounded in both the linguistic and the visual modality to infer structure.

Multi-Sentence Questions

Supervisor: Bonnie Webber

A multi-sentence question (MSQ) is a short text specifying a question or set of related questions. Evidence suggests that the sentences in an MSQ relate to each other in difference ways and that recognizing these relations can enable a system to produce a better response. We have been gathering a corpus of MSQs and beginning to characterize relations within them. Research will involve collecting human responses to MSQs and using them to design and implement a system that produces similar responses.

Concurrent Discourse Relations

Supervisor: Bonnie Webber, Hannah Rohde (LEL)

Evidence from crowd-sourced conjunction-completion experiments shows that people systematically infer implicit discourse relations that hold in addition to discourse relations signalled explicitly. Research on shallow discourse parsing has not yet reflected these findings. It is possible that enabling a shallow discourse parser to recognize implicit relations that hold concurrently with explicitly signalled relations may also help in the recognition of implicit relations without additional signals.  Work in this area could also involve crowd-sourcing additional human judgments on discourse relations.

Knowledge Graph Completion with Rich Semantics

Supervisor: Jeff Pan

Knowledge Graphs have been shown useful for improving the performance and explain-ability of machine learning methods (such as transfer learning and zero-shot learning) and their downstream tasks, such as NLP tasks.  The present project seeks to investigate how to integrate rich semantics into knowledge graph completion models and methods. Projects in this area could involve integrations of schema of knowledge graph, temporal and spatial constraints, updates of knowledge graphs, or information from language models.

Complex Query Answering over Knowledge Graphs

Supervisor: Jeff Pan

Answering complex queries on large-scale incomplete knowledge graphs is a fundamental yet challenging task. There have been two extremes in current research: one is to completely relying on logical reasoning of both schema and data sub-graphs but suffers from knowledge incompleteness; the other is to mainly relying embeddings but paying less attention to schema information.  This present project will investigate alternative approaches including those that can make use of schema and embedding effectively. 

Open-Domain Complex Question Answering at Scale

Supervisor: Pasquale Minervini

Open-Domain Question Answering (ODQA) is a task where a system needs to generate the answer to a given general-domain question, and the evidence is not given as input to the system. A core limitation of modern ODQA models is that they remain limited to answering simple, factoid questions, where the answer to the question is explicit in a single piece of evidence. In contrast, complex questions involve aggregating information from multiple documents, requiring some form of logical reasoning and sequential, multi-hop processing in order to generate the answer. Projects in this area involve proposing new ODQA models for answering complex questions, for example, by taking inspiration from models for answering complex queries in Knowledge Graphs (Arakaleyan et al., 2021; Minervini et al., 2022) and Neural Theorem Provers (Minervini et al., 2020a; Minervini et al., 2020b) and proposing methods by which neural ODQA models can learn to search in massively large text corpora, such as the entire Web.

Neuro-Symbolic and Hybrid Discrete-Continuous Natural Language Processing Models

Supervisor: Pasquale Minervini

Incorporating discrete components, such as discrete decision steps and symbolic reasoning algorithms, in neural models can significantly improve their interpretability, data efficiency, and predictive properties — for example, see (Niepert et al., 2021; Minervini et al., 2022; Minervini et al., 2020a,b). However, approaches in this space rely either on ad-hoc continuous relaxations (e.g. (Minervini et al., 2020a,b)) or on gradient estimation techniques that require some assumptions on the distributions of the discrete variables (Niepert et al., 2021; Minervini et al., 2022). Projects in this area involve devising neuro-symbolic approaches for solving NLP tasks that require some degree of reasoning and compositionality and identifying gradient estimation techniques (for back-propagating through discrete decision steps) that are both data-efficient, hyper parameter-free, accurate, and require fewer assumptions on the distribution of the discrete variables.

Learning from Graph-Structured Data

Supervisor: Pasquale Minervini

Graph-structured data is everywhere – e.g. consider Knowledge Graphs, social networks, protein and drug interaction networks, and molecular profiles. In this project, we aim to improve models for learning from graph-structured data and their evaluation protocols. Projects in this area involve incorporating invariances in graph machine learning models (e.g. see (Minervini et al., 2017)), proposing methods of transferring knowledge between graph representations, automatically identifying functional inductive biases for learning from graphs from a given domain (such as Knowledge Graphs) and proposing techniques for explaining the output of black-box graph machine learning methods (such as graph embeddings).