Robotic endoluminal surgery has gained tremendous attention for its enhanced treatments in gastrointestinal intervention, where navigating surgeons with monocular camera-based metric depth estimation is a vital sector. However, existing methods either rely on external sensors or perform poorly in terms of visual navigation. In this work, we present our M$^{3}$-Degrees Net, a novel monocular vision-guided and graph learning-based network tailored for accurate metric marching depth (MD) estimation. We first leverage a generative model to output a scale-free depth map, providing a depth basis in a coarse granularity. To achieve an optimized and metric MD prediction, a relational graph convolutional network with multi-modal visual knowledge fusion is devised. It utilizes shared salient features between keyframes and encodes their pixel differences on the depth basis as the main node, while a projection length-based node that predicts the MD on a proportional relationship basis is introduced, aiming to enable the network with explicit depth awareness. Moreover, to compensate for rotation-induced MD estimation bias, we model the endoscope's orientation changes as image-level feature shifts, formulating an ego-motion correction node for MD optimization. Lastly, a multi-layer regression network for the metric MD estimation with finer granularity is devised. We validate our network on both public and in-house datasets, and the quantitative results reveal that it can limit the overall MD error under 27.3%, which vastly outperforms the existing methods. Besides, our M$^{3}$-Degrees Net is qualitatively tested on the in-house clinical gastrointestinal endoscopy data, demonstrating its satisfactory performance even under cavity mucus with varying reflections, indicating promising clinical potentials.

Robots with active variable stiffness (VS) capabilities can potentially achieve safer interactions with humans and better adaptabilities to uncertainties in complex environments. Currently, the conventional jamming or phase-change-based VS mechanisms simultaneously act on the stiffnesses of the robotic joint in all axes, making it difficult to achieve decoupled stiffness programming in different directions/axes. To overcome this challenge, a bio-inspired stiffness-programmable robotic flexible joint (SPRFJ) based on the electro-adhesive (EA) clutches is proposed. By programming the ON/OFF states of the EA clutches on different surfaces around the SPRFJ, customization of stiffness profiles in different directions/axes can be realized, and therefore, the load-bearing capacity and flexibility of the robotic arm can be adjusted. A SPRFJ prototype consisting of four EA clutch units is developed, and through extensive experiments, we demonstrate that it can achieve a stiffness change of 21 times and can withstand resisting forces up to 13.41 N at 1 kV. The reliable multi-directional stiffness programmability of the SPRFJ is shown via extensive tests. Demonstrations on stable position locking at different angles and free movements while carrying payloads are conducted to showcase its application in soft robotics. This SPRFJ developed in this work processes the potential in industrial robots, search-and-rescue missions, and space explorations.

Virtual reality (VR) technology has enormous applications in education, entertainment, and healthcare. Haptic feedback can significantly enhance the immersive experience in VR. However, most commercial hand/fingertip wearable VR haptic devices rely on bulky rigid structures, which are limited in the offered stimuli and cause fatigue. This study introduces a novel soft wearable fingertip device, T-Touch, that provides both thermal and multi-frequency haptic feedback for more realistic VR experiences. A flexible electrohydraulic actuator (EHA) is adopted for multi-frequency mechanical stimuli, and a flexible thermoelectric array (Flex-TEA) is utilized for distinct thermal stimuli. The EHA and Flex-TEA can be independently controlled to activate simultaneously or independently, thereby rendering ON/OFF contact stimuli, vibrations, controlled temperature stimuli, or any combination of the three modalities. Our T-Touch device features a compact form factor of 35 mm × 25 mm × 22 mm and weighs only ∼8 g. It can generate mechanical stimuli with the maximum stroke of ∼1 mm, a force of 0.47 N, at a bandwidth >10 Hz, and can render precise thermal stimuli in the range of 20 to 40 °C. The main performance of the EHA and Flex-TEA modules is characterized in extensive experiments and the effects of the key design and actuation parameters are investigated to optimise performance. Preliminary user tests verify the efficacy of our T-Touch design in immersive VR applications.

Pouch motors continue to attract research attention owing to their simple fabrication process, low cost, and excellent energy density. Existing pouch motors based on the liquid-gas phase transition (LGPT) principle exhibit significant stroke and force outputs but suffer from slow responses. Pouch motors that rely on the electrohydraulic actuation (EHA) demonstrate rapid responses and broad bandwidths, yet their stroke/force outputs remain limited. This paper presents a novel thermal-electrostatic dual-modal soft pouch motor (TES-SPM) that synergistically combines the advantages of LGPT and EHA. The output performance of the TES-SPM in both the LGPT and EHA modes is characterized by extensive experiments. The effects of key parameters including the liquid volumes and actuation voltage/current amplitudes are also investigated in experiments. In the EHA mode, the TES-SPM can exert a stroke of 2.5 mm within a rapid ~ 0.06 s, while in the LGPT mode, it is able to exhibit a maximum stroke of 22.8 mm and a blocking force of ~ 80 N. A novel folding fan-inspired actuator and accordion-inspired soft gripper based on the serially attached TES-SPM units are developed to demonstrate the potentials of soft robotic applications. The TES-SPM designed in this paper is envisioned to have promising applications in industrial soft grippers and wearable assistive devices.

Currently, short video platforms have become the primary place for individuals to share experiences and obtain information. To better meet users' needs for acquiring information while browsing short videos, some apps have introduced a search entry at the bottom of videos, accompanied with recommended relevant queries. This scenario is known as query recommendation in video-related search, where core task is item-to-query (I2Q) recommendation. As this scenario has only emerged in recent years, there is a notable scarcity of academic research and publicly available datasets in this domain. To address this gap, we systematically examine the challenges associated with this scenario for the first time. Subsequently, we release a large-scale dataset derived from real-world data pertaining to the query recommendation in video- related search on the Kuaishou app (KuaiRS). Presently, existing methods rely on embeddings to calculate similarity for matching short videos with queries, lacking deep interaction between the semantic content and the query. In this paper, we introduce a novel LLM-based framework named GREAT, which guides que ry g ener ation with a trie to address I2Q recommendation in related search. Specifically, we initially gather high-quality queries with high exposure and click-through rate to construct a query-based trie. During training, we enhance the LLM's capability to generate high-quality queries using the query-based trie. In the inference phase, the query-based trie serves as a guide for the token generation. Finally, we further refine the relevance and literal quality between items and queries via a post-processing module. Extensive offline and online experiments demonstrate the effectiveness of our proposed method.

Conflict-Based Search is one of the most popular methods for multi-agent path finding. Though it is complete and optimal, it does not scale well. Recent works have been proposed to accelerate it by introducing various heuristics. However, whether these heuristics can apply to non-grid-based problem settings while maintaining their effectiveness remains an open question. In this work, we find that the answer is prone to be no. To this end, we propose a learning-based component, i.e., the Graph Transformer, as a heuristic function to accelerate the planning. The proposed method is provably complete and bounded-suboptimal with any desired factor. We conduct extensive experiments on two environments with dense graphs. Results show that the proposed Graph Transformer can be trained in problem instances with relatively few agents and generalizes well to a larger number of agents, while achieving better performance than state-of-the-art methods.

We consider the problem of navigating a mobile robot towards a target in an unknown environment that is endowed with visual sensors, where neither the robot nor the sensors have access to global positioning information and only use first-person- view images. In order to overcome the need for positioning, we train the sensors to encode and communicate relevant viewpoint information to the mobile robot, whose objective it is to use this information to navigate to the target along the shortest path. We overcome the challenge of enabling all the sensors (even those that cannot directly see the target) to predict the direction along the shortest path to the target by implementing a neighborhood-based feature aggregation module using a Graph Neural Network (GNN) architecture. In our experiments, we first demonstrate generalizability to previously unseen environments with various sensor layouts. Our results show that by using communication between the sensors and the robot, we achieve up to 2.0 × improvement in SPL (Success weighted by Path Length) when compared to a communication-free baseline. This is done without requiring a global map, positioning data, nor pre-calibration of the sensor network. Second, we perform a zero-shot transfer of our model from simulation to the real world. Laboratory experiments demonstrate the feasibility of our approach in various cluttered environments. Finally, we showcase examples of successful navigation to the target while both the sensor network layout as well as obstacles are dynamically reconfigured as the robot navigates. We provide a video demo11https://www.youtube.com/watch?v=kcrnr6RUgucw, the dataset, trained models, and source code22https://github.com/proroklab/sensor-guided-visual-nav.

Psychological distress is a significant and growing issue in society. In particular, depression and anxiety are leading causes of disability that often go undetected or late-diagnosed. Automatic detection, assessment, and analysis of behavioural markers of psychological distress can help improve identification and support prevention and early intervention efforts. Compared to modalities such as face, head, and vocal, research investigating the use of the body modality for these tasks is relatively sparse, which is partly due to the limited available datasets and difficulty in automatically extracting useful body features. To enable our research, we have collected and analyzed a new dataset containing full body videos for interviews and self-reported distress labels. We propose a novel approach to automatically detect self-adaptors and fidgeting, a subset of self-adaptors that has been shown to correlate with psychological distress. We perform analysis on statistical body gestures and fidgeting features to explore how distress levels affect behaviors. We then propose a multi-modal approach that combines different feature representations using Multi-modal Deep Denoising Auto-Encoders and Improved Fisher Vector Encoding. We demonstrate that our proposed model, combining audio-visual features with detected fidgeting behavioral cues, can successfully predict depression and anxiety in the dataset.

Graph Neural Networks (GNNs) are a paradigm-shifting neural architecture to facilitate the learning of complex multi-agent behaviors. Recent work has demonstrated remarkable performance in tasks such as flocking, multi-agent path planning and cooperative coverage. However, the policies derived through GNN-based learning schemes have not yet been deployed to the real-world on physical multi-robot systems. In this work, we present the design of a system that allows for fully decentralized execution of GNN-based policies. We create a framework based on ROS2 and elaborate its details in this paper. We demonstrate our framework on a case-study that requires tight coordination between robots, and present first-of-a-kind results that show successful real-world deployment of GNN-based policies on a decentralized multi-robot system relying on Adhoc communication. A video demonstration of this case-study can be found online11youtube.com/watch?v=COh-WLn4i04.

Effective communication is key to successful, decentralized, multi-robot path planning. Yet, it is far from obvious what information is crucial to the task at hand, and how and when it must be shared among robots. To side-step these issues and move beyond hand-crafted heuristics, we propose a combined model that automatically synthesizes local communication and decision-making policies for robots navigating in constrained workspaces. Our architecture is composed of a convolutional neural network (CNN) that extracts adequate features from local observations, and a graph neural network (GNN) that communicates these features among robots. We train the model to imitate an expert algorithm, and use the resulting model online in decentralized planning involving only local communication and local observations. We evaluate our method in simulations by navigating teams of robots to their destinations in 2D cluttered workspaces. We measure the success rates and sum of costs over the planned paths. The results show a performance close to that of our expert algorithm, demonstrating the validity of our approach. In particular, we show our model's capability to generalize to previously unseen cases (involving larger environments and larger robot teams).