Turing AI Postdocs Workshop #2

Project: Causal Knowledge Graphs for Counterfactual Claims Reasoning, Led by Björn Ross and Vaishak Belle

Talk Title: Knowledge Graphs based Misinformation Detection on Counterfactual Claims about Climate Change

Abstract: Climate change is a crisis that requires global action. As social media significantly impacts people's opinions, it is essential to reduce the harmful misinformation about climate change spreading there. However, a single AI approach is not enough for misinformation detection (MD), especially for counterfactual claims. In this project, we aim to develop a system that will apply NLP techniques to parse a given counterfactual claim and then query knowledge graphs (KGs) to get evidence for the MD in the climate change domain. In addition, probability will be computed to represent how much the conclusion is trusted. In this talk, we will present the logic for determining the truth value of a given counterfactual claim, the current pipeline, and the prepared data for annotation and then summarize future work.

Project: Grounding Actions and Action Modifiers in Instructional Videos, Led by Hakan Bilen, Frank Keller, and Laura Sevilla

Talk Title: Learning Action Changes by Measuring Verb-Adverb Textual Relationships

Abstract: The goal of this work is to understand the way actions are performed in videos. That is, given a video, we aim to predict an adverb indicating a modification applied to the action (e.g. cut “finely”). We cast this problem as a regression task. We measure textual relationships between verbs and adverbs to generate a regression target representing the action change we aim to learn. We test our approach on a range of datasets and achieve state-of-the-art results on both adverb prediction and antonym classification. Furthermore, we outperform previous work when we lift two commonly assumed conditions: the availability of action labels during testing and the pairing of adverbs as antonyms. Existing datasets for adverb recognition are either noisy, which makes learning difficult, or contain actions whose appearance is not influenced by adverbs, which makes evaluation less reliable. To address this, we collect a new high quality dataset: Adverbs in Recipes (AIR). We focus on instructional recipes videos, curating a set of actions that exhibit meaningful visual changes when performed differently. Videos in AIR are more tightly trimmed and were manually reviewed by multiple annotators to ensure high labeling quality. Results show that models learn better from AIR given its cleaner videos. At the same time, adverb prediction on AIR is challenging for models, demonstrating that there is considerable room for improvement. 

Project: Multimodal Interpretability from Partial Sight, Led by Siddharth Narayanaswamy and Ivan Titov

Talk Title: Multimodal Interpretability from Partial Sight

Abstract: We seek to build DGMs that capture the joint distribution over co-observed visual and language data (e.g. abstract scenes, COCO, VQA), while faithfully capturing the conceptual mapping between the observations in an interpretable manner. This relies on two key observations: (a) perceptual domains (e.g. images) are inherently interpretable, and (b) a key characteristic of useful abstractions are that they are low(er) dimensional (than the data) and correspond to some conceptually meaningful component of the observation. We will seek to leverage recent work on conditional neural processes (Garnelo et al, 2018) to develop partial-image representations to mediate effectively, and in an interpretable manner, between vision and language data. Evaluation of this framework will involve both the ability to generate multimodal data against state-of-the-art approaches, as well as on human-measured interpretability of the learnt representations.  Our project image represents multi-modal data(images, text) as a "partial specification" that allows effective encoding and reconstruction of data.

Project: Asking Your Self-Driving Car to Explain its Decisions, Led by Stefano V. Albrecht, Chris Lucas, and Shay Cohen

Talk Title: Towards a Safe and Trustworthy Autonomous Vehicle: a Human-centric Reasoning System to Explain its Decisions

Abstract: Artificial Intelligence (AI) has shown considerable success in autonomous vehicle (AV) perception, localization and decision-making. For such a safety-critical system, safe AI is a prerequisite before bringing AVs to market. Thus, investigating effective methods to improve AI safety is critical. One step towards safe AI is to make AI explainable and transparent. By introducing explainable AI into AV decision-making, we can understand the causality behind AV decisions, which will help us troubleshoot, discover faults and ultimately improve AI-involved AVs. As a result, we develop a reasoning system to explain autonomous vehicle decisions based on the existing interpretable motion planning called IGP2, which applies Monte Carlo tree search for AV decision-making. By running a parallel Monte Carlo tree search in the background, we can generate past, present and future explanations for users' why, why-not, what, and what-if queries. Preliminary results demonstrate the proposed reasoning system's capability.