Keywords: Intelligent Autonomous Vehicles, Human-Machine and Human-Robot Systems, Automotive Systems
Abstract: Personalized driving refers to an autonomous vehicle's ability to adapt its driving behavior or control strategies to match individual users' preferences and driving styles while maintaining safety and comfort standards. However, existing works either fail to capture every individual preference precisely or become computationally inefficient as the user base expands. Vision-Language Models (VLMs) offer promising solutions to this front through their natural language understanding and scene reasoning capabilities. In this work, we propose a lightweight yet effective on-board VLM framework that provides low-latency personalized driving performance while maintaining strong reasoning capabilities. Our solution incorporates a Retrieval-Augmented Generation (RAG)-based memory module that enables continuous learning of individual driving preferences through human feedback. Through comprehensive real-world vehicle deployment and experiments, our system has demonstrated the ability to provide safe, comfortable, and personalized driving experiences across various scenarios and significantly reduce takeover rates by up to 76.9%. To the best of our knowledge, this work represents the first end-to-end VLM-based motion control system in real-world autonomous vehicles.

Keywords: Path Planning and Motion Control, Machine Learning in modeling, estimation, and control
Abstract: In this talk, we explore recent works from my group that tackle complex real-world planning challenges with large language models (LLMs). First, we introduce AutoTAMP, a method that LLMs as translators and checkers to bridge natural language task descriptions with task-and-motion planning (TAMP), enabling robots to jointly reason about tasks and motion under intricate environmental and temporal constraints. Second, we introduce an LLM-based planning framework that formalizes and solves complex multi-constraint planning problems as constrained satisfiability problems, which are further consumed by sound and complete satisfiability solvers. When user input queries are infeasible, our framework can identify the unsatisfiable core, provide failure reasons, and offer personalized modification suggestions. Finally, we discuss how to extend the above to a general-purpose framework that leverages LLMs to capture key information from planning problems and formally formulate and solve them as optimization problems from scratch without task-specific in-context examples. We demonstrate how this general LLM-based planner can generate zero-short plans for cross-domain tasks, ranging from multi-constraint decision-making to multi-step planning problems.

Keywords: Adaptive and Learning Systems, Human-Machine and Human-Robot Systems, Stochastic Systems
Abstract: Autonomous systems must operate safely in uncertain, interactive, and non-stationary environments, particularly when collaborating with humans. In this talk, I will present recent advances from our group that address key challenges in risk-aware decision-making and adaptive learning for such systems. I will begin with a novel double Bayesian framework for the sequential fine-tuning of transformers. This technique enables robust and data-efficient learning by formulating fine-tuning as a posterior inference in time and across layers and adaptively balancing new information with pre-trained knowledge based on quantified uncertainty. Next, I will introduce our work on physics-informed learning to estimate long-term risk probability and probabilistic reachability. This technique integrates data with partial differential equation characterizations to achieve provable generalization and to infer long-term safety despite short-term data with limited risk events. I will then introduce safety certificates that ensure long-term safety using myopic control. These safety certificates are developed based on a novel technique, probabilistic invariance. Such techniques can be used to ensure latent risks arising from occlusions or unknown opponent intentions, despite unobservable information. Finally, I will conclude by highlighting the gaps in current methods for intelligent autonomous agents operating with adaptive and strategic humans and possible directions for future research.

Keywords: Machine Learning in modeling, estimation, and control, Human-Machine and Human-Robot Systems, Robotics
Abstract: Ensuring safety in learning-enabled control systems is increasingly vital as neural networks become integral to robotics and other complex dynamical systems. This talk presents recent advances in the formal verification and synthesis of neural control barrier functions (neural CBFs) that guarantee safety in nonlinear systems represented by neural networks. I begin by introducing a verified neural dynamic modeling framework that employs Bernstein polynomial-based over-approximations, which enables real-time safe control and achieves significant speedup over the complete verifier while maintaining safety guarantees. Empowered by our recent neural network verification toolbox, I will then introduce symbolic derivative bound propagation techniques to verify neural CBFs with improved tightness and scalability of formal verification. These methods form the mathematical backbone of two frontier applications. First, in PDE boundary control, I show how neural boundary CBFs ensure constraint satisfaction for systems governed by unknown PDEs via neural operator modeling. Second, I extend these safety concepts to human-AI conversations, modeling dialogue context state transitions in large language models (LLMs) as controllable dynamical systems. Using neural CBFs, we conduct online safety steering that provably prevents LLMs from multi-turn jailbreaking attacks. By unifying methods across physical and symbolic domains, this line of work opens the door to verifiable dynamical safety in a wide range of autonomous systems— from robotic systems to conversational AI agents. The talk emphasizes verifiable forward invariance induced by neural barrier function and the integration of neural dynamics with control theory, demonstrating a scalable path toward trustworthy and safe AI-driven control.

Keywords: Machine Learning in modeling, estimation, and control, Path Planning and Motion Control
Abstract: Foundation models are rapidly reshaping how we interact with and certify complex autonomous systems. In this talk, I will present two recent efforts that leverage large language models (LLMs) and neural network verification to advance planning and perception in robotic control pipelines.

First, I introduce Nl2Hltl2Plan, a framework that enables non-experts to specify long-horizon, collaborative tasks in natural language. By translating these instructions into hierarchical Linear Temporal Logic (LTL) using LLMs, the framework creates structured specifications that align with off-the-shelf planners. The translation occurs in two stages: first, extracting a hierarchical task tree, and second, converting sub-tasks into LTL formulas. This hierarchical abstraction simplifies multi-robot planning and improves both task success rates and efficiency in simulation and real-world deployments with human users.

Second, I present a method for certifying the local robustness of vision-based 6D object pose estimation, focusing on two-stage pipelines that rely on keypoint detection followed by PnP computation. We frame the certification problem as one of neural network verification, introducing tractable input and output specifications—including convex hull representations and pixel-wise output bounds. Our system-level sensitivity analysis propagates robustness constraints from pose estimation down to pixel-level tolerances, enabling the first certification framework for keypoint-based object pose estimators under real-world image perturbations.

Together, these works highlight how learning-enabled models can be integrated into both reasoning and verification, offering new tools for designing intelligent, robust, and user-friendly autonomous systems.