The ability of autonomous robots to model, communicate, and act on semantic “soft data” remains challenging. The human-assisted robotic planning and sensing (HARPS) framework is presented for active semantic sensing and planning in human–robot teams to address these gaps by formally combining the benefits of online sampling-based partially observable Markov decision process policies, multimodal human–robot interaction, and Bayesian data fusion. HARPS lets humans impose model structure and extend the range of soft data by sketching and labeling new semantic features in uncertain environments. Dynamic model updating lets robotic agents actively query humans for novel and relevant semantic data, thereby improving model and state beliefs for improved online planning. Simulations of a unmanned aerial vehicle-enabled target search in a large-scale partially structured environment show significant improvements in time and beliefs required for interception versus conventional planning with robot-only sensing. A human subject study in the same environment shows an average doubling in dynamic target capture rate compared to the lone robot case and highlights the robustness of HARPS over a range of user characteristics and interaction modalities.

In search applications, autonomous unmanned vehicles must be able to efficiently reacquire and localize mobile targets that can remain out of view for long periods of time in large spaces. As such, all available information sources must be actively leveraged - including imprecise but readily available semantic observations provided by humans. To achieve this, this work develops and validates a novel collaborative human-machine sensing solution for dynamic target search. Our approach uses continuous partially observable Markov decision process (CPOMDP) planning to generate vehicle trajectories that optimally exploit imperfect detection data from onboard sensors, as well as semantic natural language observations that can be specifically requested from human sensors. The key innovation is a scalable hierarchical Gaussian mixture model formulation for efficiently solving CPOMDPs with semantic observations in continuous dynamic state spaces. The approach is demonstrated and validated with a real human-robot team engaged in dynamic indoor target search and capture scenarios on a custom testbed.