The Hogweed of Sosnowskyi (lat. Heracleum sosnówskyi) is poisonous for humans, dangerous for farming crops, and local ecosystems. This plant is fast-growing and has already spread all over Eurasia: from Germany to the Siberian part of Russia, and its distribution expands year-by-year. In-situ detection of this harmful plant is a tremendous challenge for many countries. Meanwhile, there are no automatic systems for detection and localization of hogweed. In this article, we report on an approach for fast and accurate detection of hogweed. The approach includes the Unmanned Aerial Vehicle (UAV) with an embedded system on board running various Fully Convolutional Neural Networks (FCNN). We propose the optimal architecture of FCNN for the embedded system relying on the trade-off between the detection quality and frame rate. We propose a model that achieves ROC AUC 0.96 in the hogweed segmentation task, which can process 4K frames at 0.46 FPS on NVIDIA Jetson Nano. The developed system can recognize the hogweed on the scale of individual plants and leaves. This system opens up a wide vista for obtaining comprehensive and relevant data about the spreading of harmful plants allowing for the elimination of their expansion.

Unmanned Aerial Vehicles (UAV) have been recently applied in several various civilian applications. Based on this, there is a growing need for intuitive UAV control interfaces. In this work, we report on the Body-Machine Interface (BMI), helping a human operator to control a quadrotor through the gesture commands. We perform the human motion capture through wearable sensors and Kalman filter to reduce the noise. For the gesture command recognition, we designed the Recurrent Neural Network recognizing gestures within 65 ms. For the quadrotor orientation estimation, we designed the Extended Kalman Filter (EKF). We assess the proposed BMI via the simulations and experiments: the standard deviation of the trajectories varies for up to 10 cm.

We report on the teleoperation system DronePick which provides remote object picking and delivery by a human- controlled quadcopter. The main novelty of the proposed system is that the human user continuously gets the visual and haptic feedback for accurate teleoperation. DronePick consists of a quadcopter equipped with a magnetic grabber, a tactile glove with finger motion tracking sensor, hand tracking system, and the Virtual Reality (VR) application. The human operator teleoperates the quadcopter by changing the position of the hand. The proposed vibrotactile patterns representing the location of the remote object relative to the quadcopter are delivered to the glove. It helps the operator to determine when the quadcopter is right above the object. When the “pick” command is sent by clasping the hand in the glove, the quadcopter decreases its altitude and the magnetic grabber attaches the target object. The whole scenario is in parallel simulated in VR. The air flow from the quadcopter and the relative positions of VR objects help the operator to determine the exact position of the delivered object to be picked. The experiments showed that the vibrotactile patterns were recognized by the users at the high recognition rates: the average 99% recognition rate and the average 2.36s recognition time. The real-life implementation of DronePick featuring object picking and delivering to the human was developed and tested.

We propose SlingDrone, a novel Mixed Reality interaction paradigm that utilizes a micro-quadrotor as both pointing controller and interactive robot with a slingshot motion type. The drone attempts to hover at a given position while the human pulls it in desired direction using a hand grip and a leash. Based on the displacement, a virtual trajectory is defined. To allow for intuitive and simple control, we use virtual reality (VR) technology to trace the path of the drone based on the displacement input. The user receives force feedback propagated through the leash. Force feedback from SlingDrone coupled with visualized trajectory in VR creates an intuitive and user friendly pointing device. When the drone is released, it follows the trajectory that was shown in VR. Onboard payload (e.g. magnetic gripper) can perform various scenarios for real interaction with the surroundings, e.g. manipulation or sensing. Unlike HTC Vive controller, SlingDrone does not require handheld devices, thus it can be used as a standalone pointing technology in VR.

We propose a concept of a novel interaction strategy for providing rich haptic feedback in Virtual Reality (VR), when each user’s finger is connected to micro-quadrotor with a wire. Described technology represents the first flying wearable haptic interface. The solution potentially is able to deliver high resolution force feedback to each finger during fine motor interaction in VR. The tips of tethers are connected to the centers of quadcopters under their bottom. Therefore, flight stability is increasing and the interaction forces are becoming stronger which allows to use smaller drones.

We propose a novel interaction strategy for a human-swarm communication when a human operator guides a formation of quadrotors with impedance control and receives vibrotactile feedback. The presented approach takes into account the human hand velocity and changes the formation shape and dynamics accordingly using impedance interlinks simulated between quadrotors, which helps to achieve a life-like swarm behavior. Experimental results with Crazyflie 2.0 quadrotor platform validate the proposed control algorithm. The tactile patterns representing dynamics of the swarm (extension or contraction) are proposed. The user feels the state of the swarm at his fingertips and receives valuable information to improve the controllability of the complex life-like formation. The user study revealed the patterns with high recognition rates. Subjects stated that tactile sensation improves the ability to guide the drone formation and makes the human-swarm communication much more interactive. The proposed technology can potentially have a strong impact on the human-swarm interaction, providing a new level of intuitiveness and immersion into the swarm navigation.