Moonshot: Assistive Collaborative Manipulation

March 20 2025

Contribution type: Outreach

As part of a visit to The University of Edinburgh, and supported by PhD student Marina Aoyama, Dr. Antonova recently led an inspiring Women in Robotics career development session, designed to support early-career researchers and students navigating their futures in robotics and AI. The interactive session offered a candid look into Dr. Antonova’s own journey from academia to industry, providing valuable insights into the evolving landscape of tech careers.

December 11, 2024

Contribution Type: Outreach and Demonstrations

The Women in Computing event at St George's School (Edinburgh, UK) was an inspiring initiative to spark curiosity and encourage women to explore careers in STEM. The event aimed to highlight the exciting possibilities in technology, engineering, robotics, and computing through interactive talks, engaging panel discussions, and hands-on demonstrations. Our team was proud to participate, engaging with young students in order to provide career advice and role models that will inspire the next generation and awaken their interest in computing.

November 23, 2024

Contribution Type: Conference Paper Publication

The number of ways to climb a staircase depends on the perspective: infinite in a continuous context and finite in a discrete one, determined by step placement and sequence. We introduce NAS, an algorithm that integrates these aspects to compute all solutions for contact planning problems under standard assumptions. NAS is the first to deliver a globally optimal policy, enabling real-time humanoid footstep planning. Results on the Talos robot (in simulation and hardware) demonstrate that NAS reduces theoretical exponential complexity to a manageable bilinear form, supporting efficient GPU parallelisation while maintaining completeness.

Jiayi Wang, Saeid Samadi, Hefan Wang, Pierre Fernbach, Olivier Stasse, Sethu Vijayakumar and Steve Tonneau, NAS: N-step computation of All Solutions to the footstep planning problem, Proc. IEEE-RAS 23rd International Conference on Humanoid Robots (Humanoids 2024), Nancy, France (2024). [pdf] [video] [citation]

November 23, 2024

Contribution Type: Conference Paper Publication

Humans can use their entire body surface to manipulate heavy, distant, or multiple objects simultaneously, but replicating this ability in robots—known as Whole-Body Contact-Rich Manipulation (WBCRM)—is highly complex. This is due to the challenge of managing numerous contact modes and enabling contact anywhere on the robot's surface, which complicates planning within a reasonable time. To address this, we reformulate planar WBCRM as hierarchical continuous optimization problems, using a novel explicit representation of the robot surface. This approach significantly improves convergence, planning efficiency, and feasibility.

Victor Leve, Joao Moura, Namiko Saito, Steve Tonneau and Sethu Vijayakumar, Explicit Contact Optimization in Whole-Body Contact-Rich Manipulation, Proc. IEEE-RAS 23rd International Conference on Humanoid Robots (Humanoids 2024), Nancy, France (2024). [pdf] [video] [citation]

November 8, 2024

Contribution Type: Conference Paper Publication

Manipulation without grasping, known as non-prehensile manipulation, is essential for dexterous robots in contact-rich environments, but presents many challenges relating with underactuation, hybrid-dynamics, and frictional uncertainty. Additionally, object occlusions in a scenario of contact uncertainty becomes a critical problem, which previous literature fails to address. We present a method for learning visuotactile state estimators and uncertainty-aware control policies for non-prehensile manipulation under occlusions, by leveraging diverse interaction data from privileged policies trained in simulation. Unlike prior non-prehensile research that relies on complex external perception set-ups, our method handles occlusions after sim-to-real transfer to robotic hardware with an onboard camera.

Juan Del Aguila Ferrandis, Joao Moura, and Sethu Vijayakumar, Learning Visuotactile Estimation and Control for Non-prehensile Manipulation under Occlusions, Proc. The Conference on Robot Learning (CoRL), Munich, Germany (2024). [pdf] [video] [citation]

October 14, 2024

Contribution Type: Interactive Workshop and Seminar

We invited world-leading research scientists for a workshop at the International Conference on Intelligent Robots and Systems (IROS), in Abu Dhabi, UAE, to discuss collecting, managing and utilizing data through embodied robots.

In recent years, embodied robots have been increasingly playing a significant role in real-world scenarios such as daily tasks, healthcare, caregiving, and agriculture, and interacting with humans and environments. Nevertheless, the collection and application of sensorimotor data in real-world settings pose challenges, given the associated high costs, inherent random noise, requisite tailored processing, and potential inclusion of personal data. This workshop would provide a platform to discuss efficient data collection, high-quality data assurance, sophisticated processing and utilization, and ethical considerations.

Check our workshop website for more information: Data curation workshop

October 2024

Contribution Type: Conference Paper Publication

Recognising the characteristics of objects while a robot handles them is crucial for adjusting motions that ensure stable and efficient interactions with containers.  Ahead of realising stable and efficient robot motions for handling/transferring the containers, this work aims to recognise the unobservable latent object characteristics. We propose a cross-modal transfer learning approach from vision to haptic-audio, which facilitates the representation of object characteristics using indirect sensor data, thereby improving recognition accuracy. For evaluation, we handled predicting shape, position, and orientation of an object inside a closed box.

Namiko Saito, Joao Moura, Hiroki Uchida and Sethu Vijayakumar, Latent Object Characteristics Recognition with Visual to Haptic-Audio Cross-modal Transfer Learning , Proc. 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2024), Abu Dhabi, UAE (2024). [pdf] [video] [citation]

September 2024

Contribution type: Conference Paper Publication

The research on long-horizon manipulation in environments with numerous objects and subtasks falls under the framework of task and motion planning (TAMP). We propose a concept of hierarchical framework combining deep neural networks (DNN) for higher-level subgoal decisions and optimization for lower-level motion control to enhance robustness, scalability, and generalizability. This will be evaluated on a latent state box transport and stacking task – where the robot needs to change the order of actions and speed to control during motion execution.

Namiko Saito, Joao Moura, and Sethu Vijayakumar, Long-horizon Manipulation through Hierarchical Motion Planning with Subgoal Prediction, 40th Anniversary of the IEEE Conference on Robotics and Automation (ICRA@40), Rotterdam, Netherlands (2024). [pdf] [video] [citation]

September 2024

Contribution Type: Conference Paper

Shared Autonomy presents a paradigm for solving the problem of tele-operating complex robotic systems. We propose a framework for combining human user/operator input with autonomous reasoning for remote handling of contact-rich manipulation tasks, consisting of an optimal control (OC) formalism that incorporates models from hybrid contact dynamics, compliant interaction, and operator intention as a means of expanding current robotic manipulation capabilities whilst ensuring safe and stable task execution. 

Joao Moura, Theodoros Stouraitis, Namiko Saito, and Sethu Vijayakumar, Optimal Shared Autonomy for Contact-rich Robotic Manipulation , 40th Anniversary of the IEEE Conference on Robotics and Automation (ICRA@40), Rotterdam, Netherlands (2024). [pdf] [citation]

May 30 2024

Contribution Type: Seminar and Live Demonstrations

AI and embodied robotics solutions are playing an increasingly significant role in addressing some of the most pressing grand challenges in healthcare and assisted living in our society. For successful, safe and effective deployment of these fast-changing technologies, it is important that we work closely and collaboratively with different stakeholders.

In this workshop, we bring together end-users in care homes, hospitals and rehabilitation centres; world-leading researchers from academia including European and international consortia; startup companies and established industry players in this space; as well as policymakers and researchers who look at the ethics and governance structures that inform deployment strategies.

Check our workshop website for more information: AI in Healthcare Workshop

Please check out the detailed news item for some pictures of the event.

May 20 2024

Contribution Type: Symposium and Presentations

As robot learning approaches are getting more and more complex and sophisticated, we co-organised a mini-symposium in collaboration with the Advanced Telecommunications Research Institute International (ATR), in Kyoto, for discussing challenges and the latest developments on robot learning. Our host was one of leading human motor control researchers in the world, Prof. Mitsuo Kawato of ATR.

In this symposium, we had the chance of hearing from many robot learning researchers at ATR as well as renowned international researchers, such as Prof. Jan Peters (TUD, Germany) and Prof Ales Ude (Jožef Stefan Institute, Slovenia). We also presented some of the work we have been developing in the context of the Moonshot programme, with presentations from our UoE SLMC group including Prof. Sethu Vijayakumar, Dr. Namiko Saito, Dr. Joao Moura, and Marina Aoyama, among others.

Please check out the detailed news item.

May 17 2024

Contribution Type: Interactive Workshop and Seminar

We invited world-leading research scientists and practitioners for a workshop at the International Conference on Robotics and Automation (ICRA), in Yokohama, Japan, to explore the new frontiers of “robots in cooking” ! We will address various scientific research questions, including hardware, multimodal perception, motion planning and control, experimental methodologies, and benchmarking approaches. The workshop will count with networking opportunities through poster session from young researchers and even exciting live robotic demos from Waseda University and Kawada Robotics Corporation!

Check our workshop website for more information: Cooking Robotics Workshop

May 16 2024

Contribution Type: Conference Paper Publication

Existing Learning from Demonstration (LfD) approaches require a large number of human demonstrations for enabling robots to generalise to unseen objects. In this work, we propose a semi-supervised LfD approach that decouples the learnt model into a haptic representation encoder and a motion generation decoder. This enables us to pre-train the first using a large amount of unsupervised data, easily accessible, while using few-shot LfD to train the second, leveraging the benefits of learning skills from humans. We validate our approach on the wiping task, demonstrating improved performance for sponges with unseen stiffness and surface friction.

Marina Y. Aoyama, Joao Moura, Namiko Saito and Sethu Vijayakumar, Few-Shot Learning of Force-Based Motions From Demonstration Through Pre-training of Haptic Representation, Proc. IEEE International Conference on Robotics and Automation (ICRA 2024), Yokohama, Japan (2024). [pdf] [video] [citation]

April 12 2024

Contribution Type: Outreach Event

We opened our laboratory to the public to showcase our cutting-edge research platforms to inform and engage visitors about the best ways to deliver human-centric assistance and effective human-robot collaborations.

Visitors, both adults and children, had  the opportunity to learn more about the world of robotics, through an interactive tour that included demonstrations of humanoid robots, exoskeletons, a showcase of the factory of the future, and finally a chance to directly teleoperate the robots themselves.

Please check out the detailed news item.

March 2025

Contribution type: Integration activities

Our team, Victor Leve, Joao Pousa De Moura, and Sethu Vijayakumar, spent 3 weeks at Waseda University integrating our framework on whole-body contact-rich motion planning on the AIREC dual arm robot with the help of Sachiya Fujita and Tamon from Waseda.

After solving many sim2real and perception challenges, we finally got AIREC to use both its arms and torso to move a heavy shopping box.