IPAB Workshop - 6/11/25 Speaker: Wanming Yu Title: Preference-Based Long-Horizon Robotic Stacking with Multimodal Large Language Models Abstract: Pretrained large language models (LLMs) can work as high-level robotic planners by reasoning over abstract task descriptions and natural language instructions, etc. However, they have shown a lack of knowledge and effectiveness in planning long-horizon robotic manipulation tasks where the physical properties of the objects are essential. An example is the stacking of containers with hidden objects inside, which involves reasoning over hidden physics properties such as weight and stability. To this end, this paper proposes to use multimodal LLMs as high-level planners for such long-horizon robotic stacking tasks. The LLM takes multimodal inputs for each object to stack and infers the current best stacking sequence by reasoning over stacking preferences. Furthermore, in order to enable the LLM to reason over multiple preferences at the same time without giving explicit instructions, we propose to create a custom dataset considering stacking preferences including weight, stability, size, and footprint, to fine-tune the LLM. Compared to the pretrained LLM with prompt tuning, we demonstrate the improved stacking completion of the LLM fine-tuned with our custom dataset via large-scale simulation evaluation. Furthermore, we showcase the effectiveness of the proposed framework for the long-horizon stacking task on a real humanoid robot in an online manner. Speaker: Zhaoxing Deng Title: A Model-Based Framework for Assessing Operator Abstract: Bronchoscopy is a critical procedure for diagnosing and treating pulmonary diseases, but its safe and effective execution demands substantial operator training. This work introduces a model-based framework for objective skill evaluation in navigational bronchoscopy. By leveraging pose data from electromagnetic trackers and embedding kinematic constraints of expert motion, the system generates optimal reference trajectories and computes deviation-based error metrics to assess operator performance. Tested on a phantom lung dataset with 11 operators and 98 procedures, the proposed method effectively distinguishes between expert and novice skills. This approach offers a data-driven, interpretable alternative to subjective expert assessments and provides a public benchmark dataset to advance objective skill evaluation in bronchoscopy training and robotic systems. Nov 06 2025 13.00 - 14.00 IPAB Workshop - 6/11/25 Wanming Yu & Zhaoxing Deng G.03
IPAB Workshop - 6/11/25 Speaker: Wanming Yu Title: Preference-Based Long-Horizon Robotic Stacking with Multimodal Large Language Models Abstract: Pretrained large language models (LLMs) can work as high-level robotic planners by reasoning over abstract task descriptions and natural language instructions, etc. However, they have shown a lack of knowledge and effectiveness in planning long-horizon robotic manipulation tasks where the physical properties of the objects are essential. An example is the stacking of containers with hidden objects inside, which involves reasoning over hidden physics properties such as weight and stability. To this end, this paper proposes to use multimodal LLMs as high-level planners for such long-horizon robotic stacking tasks. The LLM takes multimodal inputs for each object to stack and infers the current best stacking sequence by reasoning over stacking preferences. Furthermore, in order to enable the LLM to reason over multiple preferences at the same time without giving explicit instructions, we propose to create a custom dataset considering stacking preferences including weight, stability, size, and footprint, to fine-tune the LLM. Compared to the pretrained LLM with prompt tuning, we demonstrate the improved stacking completion of the LLM fine-tuned with our custom dataset via large-scale simulation evaluation. Furthermore, we showcase the effectiveness of the proposed framework for the long-horizon stacking task on a real humanoid robot in an online manner. Speaker: Zhaoxing Deng Title: A Model-Based Framework for Assessing Operator Abstract: Bronchoscopy is a critical procedure for diagnosing and treating pulmonary diseases, but its safe and effective execution demands substantial operator training. This work introduces a model-based framework for objective skill evaluation in navigational bronchoscopy. By leveraging pose data from electromagnetic trackers and embedding kinematic constraints of expert motion, the system generates optimal reference trajectories and computes deviation-based error metrics to assess operator performance. Tested on a phantom lung dataset with 11 operators and 98 procedures, the proposed method effectively distinguishes between expert and novice skills. This approach offers a data-driven, interpretable alternative to subjective expert assessments and provides a public benchmark dataset to advance objective skill evaluation in bronchoscopy training and robotic systems. Nov 06 2025 13.00 - 14.00 IPAB Workshop - 6/11/25 Wanming Yu & Zhaoxing Deng G.03
