Keywords: Shared control, Potential impact of automation and open problems, Semi-autonomous and mixed-initiative systems
Abstract: Public spaces are some of the most challenging environments in which autonomous systems operate. Unlike workspaces or controlled environments where humans can be trained to work with autonomous systems, or where systems can be built to specifications based on the users, public environments feature a wide variety of people and events. In these environments, interaction challenges cannot be solved with mere interface or instructional improvements. To operate in these environments, robots need to be savvy about the norms and social signals that people use to jointly negotiate on the road, sidewalk and hallway. More concerningly, these norms and signals vary by location and context. In this talk, I will talk about current research from my lab and beyond on human robot interaction and human-AV interaction which elicits interactive behaviors from people, and discuss attempts to capture regional differences, norm enforcing behaviors. This research informs ways that autonomous systems can be designed and developed to more safely operate in public.

Keywords: Advanced control design-linear, non-linear, stochastic, large scale control systems, Public policies, Biomedical implants
Abstract: The paper considers continuous-time networked susceptible-infected-susceptible (SIS) diseases spreading over a population. Each agent represents a subpopulation and has its own healing rate and infection rate; the state of the agent at a time instant denotes what fraction of the said subpopulation is infected with the disease at the said time instant. By taking account of the changes in behaviors of the agents in response to the infection rates in real-time, our goal is to devise a feedback strategy such that the infection level for each agent strictly stays below a pre-specified value. Furthermore, we are also interested in ensuring that the closed-loop system converges either to the disease-free equilibrium or, when it exists, to the endemic equilibrium. The upshot of devising such a strategy is that it allows health administration officials to ensure that there is sufficient capacity in the healthcare system to treat the most severe cases. We demonstrate the effectiveness of our controller via numerical examples.

Keywords: Smart prosthetics, Shared control, Cognitive control
Abstract: Neural interfaces provide novel opportunities for augmenting human capabilities in domains like human-machine interaction, brain-computer interface, and rehabilitation. However, the performance of these interfaces varies significantly across users. Decoders that adapt to individual users have the potential to reduce variability and improve performance but introduce a “two-learner” problem as the user simultaneously adapts to the changing decoder. We propose and experimentally test a game-theoretic framework to optimize closed-loop performance of a myoelectric interface for continuous control (based on surface electromyography, sEMG) through the co-adaptation of user and decoder. Human subjects learned to use our interface to perform a two-dimensional trajectory-tracking task. Closed-loop performance was affected by decoder learning rate, but not initialization or decoder cost weights. Our study indicates the potential for co-adaptation in humans and machines to optimize the performance of neural interfaces.

Keywords: Aircraft control, Human-Machine interaction in aircraft, Automotive cooperated control (ADAS, etc), Control in hazardous environments
Abstract: With the rapid advancement of Formal Methods, Model-based Safety Analysis (MBSA) has been gaining tremendous attention for its ability to rigorously verify whether the safety-critical scenarios are adequately addressed by the design solution of a cyber-physical human system. However, there is a gap. If specific safety-critical scenarios are not included in the given design solution (i.e., the model) in the first place, the results of MBSA cannot be trusted for safety assurance. To tackle this problem, we propose a new safety-guided design methodology (called STPA+) to complement MBSA. Inspired by STPA, STPA+ treats a system as a control structure, which is particularly fit for systems with complex interactions between human, machine, and automation. Three methods are developed in STPA+ to tackle the possible omissions of safety-critical scenarios caused by incorrectly de ned safety constraints, improperly constrained process model, and inadequately designed controller. In this way, STPA+ directly derives an adequately de ned design solution as the input to an MBSA verification program and bridges the gap between current MBSA approaches and safety assurance.

Keywords: Cognitive control, Shared control
Abstract: Autonomous systems are increasingly being used for the purpose of training humans to attain new skills or perform new tasks. In these contexts, autonomous systems should be responsive to, and guide, human behavior such that skill or task performance is maximized. These systems generally rely on human performance to determine if assistance is needed. However, it is recognized that these systems should also respond to human cognitive factors, such as self--confidence, that are relevant for human learning. We propose and experimentally validate a heuristic control strategy, based on both a user's performance and self-reported self-confidence as they, that determines whether or not they receive automated assistance in learning how to land a quadrotor in a simulated environment. Through a human subject study involving a benchmark strategy that is solely performance-based, we show that the proposed strategy not only successfully calibrates the self-confidence of the participants, but also leads to statistically significant improvements in participants' task performance and consistency after 20 trials relative to outcomes for participants who experience the benchmark strategy.

Keywords: Aircraft control, Human-Machine interaction in aircraft, Semi-autonomous and mixed-initiative systems, Shared control
Abstract: There are several critical challenges to the success of cyber-physical human systems in aerospace applications. What technological advances are needed to address these challenges? What are the key industry and government lab needs driving the increasing importance of human-machine teaming in aerospace systems?

Keywords: Robotic surgery, Shared control, Assistive robotics
Abstract: Cyber-physical human systems play an increasing role in healthcare systems. What role should cyber-physical human systems play in healthcare? How can practitioner-centric design be realized in cyber-physical human systems?

Keywords: Aircraft control, Human-Machine interaction in aircraft
Abstract: Interaction between the adaptive behavior of a human pilot and an adaptive flight control system (FCS) usually leads to undesirable adverse effects such as pilot-induced oscillations (PIO). This paper presents an analysis of this interaction in the case of a fast adapting FCS. The analysis shows that, although a pilot will have some nonlinear behavior at the moment when a failure occurs, the fast adapting FCS allows to keep the physical workload low and the pilot behavior remains the same as before the failure. This effectively mitigates the adverse effects seen in the literature.

Keywords: Cognitive control, Aircraft control, Human-Machine interaction in aircraft
Abstract: Reachability analysis is a widely used method to analyze the safety of a Human-in-the-Loop Cyber Physical System (HiLCPS). It allows the HiLCPS to respond against an imminent threat in advance by predicting reachable states of the system. However, it could lead to an unnecessarily conservative reachable set if the prediction only relies on the system dynamics without explicitly considering human behavior, and thus the risk might be overestimated. To avoid the conservativeness, we present a state probability distribution function (pdf) prediction method which takes into account a stochastic human behavior model represented as a Gaussian Mixture Model (GMM). In this paper, we focus on the multi-rotor controlled by a human operator in a near-collision situation. The stochastic human behavior model is trained using experimental data to represent the human operators’ evasive maneuver. Then, we can retrieve a human control input pdf from the trained stochastic human behavior model using the Gaussian Mixture Regression (GMR). The proposed algorithm predicts the multi-rotor’s future state pdf by propagating the pdf of the retrieved human control input according to the given dynamics, which yields closed-loop analysis of the HiLCPS. Human subject experiment results are provided to demonstrate the effectiveness of the proposed algorithm.

Keywords: Shared control
Abstract: In human/machine interfaces (HMI), humans can interact with dynamic machines through a variety of sensory and motor modalities. Redundant motor modalities are known to have advantages in both human sensorimotor control and human-computer interaction: motor redundancy in sensorimotor control provides abundant solutions to achieve tasks; and incorporating diverse features from different modalities has improved the performance of movement-, gesture-, and brain-controlled computer interfaces. Our objective is to investigate whether redundant motor modalities enhance performance for a continuous trajectory-tracking task. We designed a multimodal human/machine interface with combined manual (joystick) and muscle (surface electromyography, sEMG) inputs and evaluated its closed-loop performance for tracking trajectories through second-order machine dynamics. In a human subjects experiment with 15 participants, we found that the multimodal interface outperformed the manual-only interface while performing comparably to the muscle-only interface; and that the multimodal interface enabled users to coordinate sensorimotor noise in individual modalities to improve performance. Multimodal human/machine interfaces could be beneficial in systems that require stability and robustness against perturbations such as motor rehabilitation and robotic manipulation.

Keywords: Aircraft control, Human-Machine interaction in aircraft
Abstract: The effectiveness of alternative displays (primary flight and predictive displays) were studied in experiments with and without motion cues. The ground-based simulations involved the task of landing in wake turbulence performed by a medium-haul and a second-generation supersonic aircraft.

Keywords: Cognitive control, Shared control
Abstract: With the emergence of safe autonomous vehicles and systems, there is a demand for creating systems that are aware of and responsive to the human. There have been decades of work dedicated to human-in-the loop studies. However, when it comes to systems that are responsive to the human based on learning parameters, there is a need for the appropriate input parameters to assess learning. In this work, signal processing methods were used to analyze game controller input signals in response to humans completing a simulated quadrotor landing task with three levels of difficulty (easy, medium, and difficult) over 30 trials. Data collected from twelve adults were analyzed using the energy of the controller input signal; 2) non-dimensional velocity; and 3) dominant frequency analysis. The landing trajectories were also mapped graphically revealing three categories of learners: beginner, intermediate, and trained. The results from the signal processing analysis procedure provided supporting evidence for these categories. The results of this work suggests that input parameters from a game controller can be used as a proxy for learning and can provide an additional means for enabling human aware systems.

Keywords: Automotive cooperated control (ADAS, etc), Decision-support for human operators, Intelligent road transportation
Abstract: In this paper, we investigate coordination of an autonomous vehicle (AV) and an intelligent human vehicle (IHV). The IHV is a human-driven vehicle that can communicate and collaborate with other vehicles while also providing advisory directives to the driver to optimize its maneuver. The objective is to optimize control inputs for the AV and advisory directives for the driver on the IHV to coordinate their motions. We consider a coordinated lane merging example where the two vehicles need to reach a prescribed separation before the lane merging maneuver. We model the motion of the IHV and the AV using a Discrete Hybrid System Automata (DHSA) and formulate a model predictive control (MPC) problem to generate optimal inputs to the two vehicles. In particular, the input to the IHV is advisory commands that stochastically transition the human state. Since solving the MPC involves mixed-integer programming (MIP), we leverage a machine learning approach to predict optimal integer values, thereby reducing the computational time of the optimization. Preliminary simulation results and experimental findings from a driving simulator reveal successful coordination between the IHV and the AV and enhanced merging performance when compared to the `no advising' scenario.

Keywords: Automotive cooperated control (ADAS, etc), Cognitive control, Shared control
Abstract: Recently, the use of artificial potential fields, known as risk fields, has been proposed for modeling human driver decision making. Such potential fields map from vehicle states and control inputs to a numerical risk measure such that the probability of choosing a control decreases as the risk associated increases. In this paper, we show that such a model can be used in a natural manner to also capture aspects of the driver's situational awareness, assuming that the risk fields govern their underlying behavior. We demonstrate our ideas on a specific obstacle avoidance scenario wherein obstacles to be avoided are placed in front of a driver at predicable intervals. Using data collected on a pilot experiment involving six different drivers using a high-fidelity driving simulator, we demonstrate the ability of our approach to capture the likelihood that the driver has perceived/reacted to the obstacle. Our approach works for scenarios when the driver collides with the obstacle as well as scenarios involving successful collision avoidance.

Keywords: Advanced control design-linear, non-linear, stochastic, large scale control systems, Shared control, Assistive devices
Abstract: Quite recently, considerable attention has been paid to the concept of prediction of manual wheelchair locomotion by a predictive model simulation, with the aim of facilitating the synthesis of the biomechanics of propulsion movement and improving the wheelchair ergonomics. Generally, the biomechanical modeling of wheelchair propulsion is a highly nonlinear dynamic that depends on the upper limb motion and the contact force between the human hand and the handrim, which will increase the complexity of implementing these models to evaluate the biomechanics of propulsion in the simulation. In this context, a new approach for the prediction of the propulsion motion of a manual wheelchair based on nonlinear model predictive control (NMPC) has been proposed. Based only on the kinematics of the hand on the handrim and the dynamics of the rear wheels, this approach is able to predict the optimal hand contact on the handrim, during the push phase and the optimal hand trajectory during the recovery phase. Thus, for a given speed configuration, the predictive model simulation is able to generate the most suitable propulsion scheme. The advantage of using a simple model of the hand-wheel interaction could allow for an easier implementation while providing a fully predictive simulation of the wheelchair propulsion. Simulation results are presented, showing the effectiveness of the proposed approach.

Keywords: Automotive cooperated control (ADAS, etc), Semi-autonomous and mixed-initiative systems, Potential impact of automation and open problems
Abstract: A theoretical framework is proposed to manage an equilibrium between human engagement levels and automation capability levels. Engagement level is expressed as a two-dimensional space comprised of readiness factors and the driver’s “intention to act.” When the human agent’s (e.g., the driver’s) engagement is below the required threshold (the driver is not monitoring the automation and/or aware of the traffic situation), some form of intervention is necessary to bring the human-automation system back into equilibrium. The automation capability level also changes, if for example, despite intervention attempts, the driver’s engagement level is still below requirements. Finally, just as the driver’s engagement can degrade as a result of disturbances, the automation can also suffer from system faults and sensing limitations as well as breakdowns in the updates provided to the driver. The last section proposes a formal method to analyze the automation interface and information design using via a generic cruise control system.

Keywords: Automotive cooperated control (ADAS, etc), Intelligent road transportation, Advanced control design-linear, non-linear, stochastic, large scale control systems
Abstract: We develop algorithms for inferring long-term intentions and parameters of local collision-avoidance behavior of agents in a multiagent system from their trajectories. This problem is challenging because an agent's observed trajectory only partially manifests its long-term task; it also contains adjustments made by the agent to ensure collision avoidance with other agents and obstacles in the environment. Since an observer would have no means to determine the magnitude of these adjustments, it is difficult to isolate the task-oriented component from the observed motion. To circumvent this problem, we model the agent's dynamics using a reactive optimization whose objective function captures the long-term task while its constraints capture collision-avoidance behavior. We develop two robust mixed-integer programming algorithms that infer the task and safety-related parameters of this optimization problem from the positions and velocities of the agents. These algorithms are validated on synthetic datasets using parameter estimation errors, displacement errors, and computation time as metrics. We further test these algorithms on a dataset of real human trajectories. We show that the learned parameters capture the true underlying pedestrian dynamics by rolling out the learned model and showing similarities between the ground truth trajectories and the reconstructed trajectories.

Keywords: Flexible manufacturing
Abstract: Human-robot interaction (HRI) is an important component to improve the flexibility of modern production lines. However, in real-world applications, the task (ie the conditions that the robot needs to operate on, such as the environmental lighting condition, the human subjects to interact with, and the hardware platforms) may vary and it remains challenging to optimally and efficiently configure and adapt the robotic system under these changing tasks. To address the challenge, this paper proposes a task-agnostic adaptable controller that can 1) adapt to different lighting conditions, 2) adapt to individual behaviors and ensure safety when interacting with different humans, and 3) enable easy transfer across robot platforms with different control interfaces. The proposed framework is tested on a human-robot handover task using the FANUC LR Mate 200id/7L robot and the Kinova Gen3 robot. Experiments show that the proposed task-agnostic controller can achieve consistent performance across different tasks.

Keywords: Flexible manufacturing, Assistive robotics, Shared control
Abstract: When robots work with humans for a collaborative task, they need to plan their actions while taking humans' actions into account. The state-of-the-art optimization-based human-aware task planner plans robot actions by prioritizing the actions that are parallel to the human's actions. However, the limitation of this approach is that the robot execution time for each action, which is an important parameter for the optimization problem, is fixed. In this paper, we investigate how online robot execution time estimation can increase the time efficiency of such a human-aware task planner. We propose a direct time estimation method and an analytical time estimation method. An integrated task and trajectory planning framework is also presented. Experiments show that online robot execution time estimation increases the time efficiency of the collaborative task. Moreover, the results of the direct time estimation method are more accurate than those of the analytical method.

Keywords: Exoskeletons, Assistive devices, Advanced control design-linear, non-linear, stochastic, large scale control systems
Abstract: Lower-limb hybrid exoskeletons integrate powered mechanisms and functional electrical stimulation (FES) to provide assistive forces and activate muscles for restoring gait function after a neurological injury. Particularly, improving the load-bearing ability is a primary rehabilitation goal. Hence, the control of the exoskeleton and FES is critical within the stance phase of walking to ensure smooth weight transfer between limbs, and achieve a sound loading response and leg propulsion. This paper develops a concurrent learning adaptive control technique to provide torque assistance about the hip and knee joints using a cable-driven exoskeleton, and activate the quadriceps and hamstrings muscle groups via FES for treadmill walking. The human-exoskeleton dynamics are modeled with phase-dependent switching dynamics to strategically update the concurrent learning controller at early stance (heel strike), late stance (toe-off), and the swing phases of walking. Thus, the adaptive switching controller compensates for the dynamic changes within the stance phase and its transition to swing, while achieving joint kinematic tracking and estimating a subset of the leg’s uncertain parameters. A multiple Lyapunov function analysis is developed to demonstrate stability of the overall phase-dependent switched system requiring a dwell-time condition that guarantees exponential tracking and parameter estimation convergence.