This paper investigates the hybrid beamforming for reconfigurable intelligent surface (RIS)-assisted multiuser multiple-input multiple-output (MIMO) systems. In order to stabilize the quality of service (QoS) for each user, the transmit power is minimized under multiple mean squared error (MSE) constraints. The optimization problems of analog beamforming and RIS reflection beamforming are solved by the proposed inner approximation-iterative coordinate ascent (IA-ICA) method. Compared with the semidefinite relaxation algorithm (SDR) and latest inner majorization-minimization (iMM) method, our approach always converges to lower transmit power. Moreover, a modified subgradient method (MSM) is developed to tackle the classical convex digital precoder optimization problem and obtain the optimal values. Simulation results verify that the proposed method can obtain higher performance solutions than the existing work. The proposed IA-ICA method is more suitable for solving the non-convex quadratically constrained quadratic program (QCQP) with extra constant modulus constraints.

Click-based interactive segmentation is the most concise and widely used data labeling method. While existing interactive segmentation methods excel in handling simple targets, they encounter challenges in obtaining high-quality masks from some complex scenes, even with a large number of clicks. Also, the cost of retraining the model from scratch for special scenarios is unacceptably high. To address these issues, we propose ClickAdapter, a simple yet powerful interactive segmentation model adapter without the need for no pre-training. Through introducing a small number of additional parameters and computations, the adapter module effectively enhanced the ability of interactive segmentation models to obtain high-quality prediction with limited clicks. Specifically, we incorporate a detail extractor that aims to extract spatial correlations and local detail features of images. These fine-grained data are then integrated into a model with our adapter to generate segmentation masks with sharp and precise edges. During the training process, only the parameters of our adapter are learnable, thereby reducing the training cost. Features in special scenarios can also be infused more efficiently. To verify the efficiency and performance advantages of the proposed method, a series of experiments on a wide range of benchmarks were conducted, demonstrating that the proposed algorithm achieved cutting-edge performance compared to current state-of-the-art (SOTA) methods.

Interactive segmentation has gained significant attention due to its applications in human-computer interaction and data annotation. To address the challenge of target scale variations in interactive segmentation, we propose a novel multi-scale token fusion algorithm. This algorithm selectively fuses only the most important tokens, enabling the model to better capture multi-scale characteristics in important regions. To further enhance the robustness of multi-scale token selection, we introduce a token learning algorithm based on contrastive loss. This algorithm fully utilizes the discriminative information between target and background multi-scale tokens, effectively improving the quality of selected tokens. Extensive benchmark testing demonstrates the effectiveness of our approach in addressing multi-scale issues. The code and data have been made publicly available athttps://github.com/hahamyt/mst.

Vehicular Ad Hoc Network (VANET) is an important component of the Internet of Vehicles (IoV), providing a wide range of traffic safety, efficiency, and infotainment applications. These applications pose challenges to onboard computation and energy resources. To address these issues, Vehicle Edge Computing (VEC) has been introduced as a novel computing paradigm. Given the rapid and frequent changes in the topology of VANET networks due to the highspeed movement of vehicles, most VANET applications adopt broadcast communication, except for a few employing unicast communication. However, traditional task offloading models struggle to support broadcast applications. Therefore, this paper proposes a VANET broadcast task model to offload tasks from On-Board Units (OBUs) to Road-Side Units (RSUs) for the first time. To meet the requirement of low latency, an efficient polynomial-time heuristic algorithm is further proposed. Numerical results demonstrate the outstanding performance of the algorithm, with all instances being solved in less than 6ms, which significantly surpasses the performance of existing solvers. Moreover, the solutions obtained from the algorithm exhibit gaps below 1%, indicating their high acceptability and applicability in practical VANET applications. Furthermore, extensive numerical experiments provide valuable insights into the practical implementation of task offloading, suggesting an appropriate ratio of 6 between the number of OBUs and RSUs.

In this paper, we study beamforming design for multi-user MIMO cognitive radio systems, where a secondary base station transmits multiple data streams to multiple secondary users while imposing interference on primary users. We focus on the weighted sum rate (WSR) maximization problem with the sum power constraint (SPC) and the interference constraints (ICs) by optimizing the beamforming matrices. Firstly, through an analysis of the generalized utility optimization problem, we prove that the WSR maximization problem with a single quadratic constraint can be simplified to an unconstrained WSR maximization problem with adaptive covariance matrices, which can be further solved by the weighted minimal mean square error (WMMSE) method with much lower complexity. Then, we propose the modified subgradient method (MSM)-reduced (R)-WMMSE algorithm for the general scenario and the null-space projection (NSP)-R-WMMSE algorithm for the special scenario with zero ICs. Finally, theoretical and numerical results show superior performances of the proposed algorithms compared to benchmark schemes in terms of computational complexity.

Depth estimation is a cornerstone for autonomous driving, yet acquiring per-pixel depth ground truth for supervised learning is challenging. Self-Supervised Surround Depth Estimation (SSSDE) from consecutive images offers an economical alternative. While previous SSSDE methods have proposed different mechanisms to fuse information across images, few of them explicitly consider the cross-view constraints, leading to inferior performance, particularly in overlapping regions. This paper proposes an efficient and consistent pose estimation design and two loss functions to enhance cross-view consistency for SSSDE. For pose estimation, we propose to use only front-view images to reduce training memory and sustain pose estimation consistency. The first loss function is the dense depth consistency loss, which penalizes the difference between predicted depths in overlapping regions. The second one is the multi-view reconstruction consistency loss, which aims to maintain consistency between reconstruction from spatial and spatial-temporal contexts. Additionally, we introduce a novel flipping augmentation to improve the performance further. Our techniques enable a simple neural model to achieve state-of-the-art performance on the DDAD and nuScenes datasets. Last but not least, our proposed techniques can be easily applied to other methods. The code is available at https://github.com/denyingmxd/CVCDepth.

Accurate localization and pose estimation remain challenging for autonomous robots in low-texture environment. This article proposes a tightly coupled direct depth-inertial odometry and mapping (DDIO-Mapping) framework to simultaneously tackle three crucial issues in such environments: 1) ineffective feature point extraction; 2) inefficient searching of feature points; and 3) imbalanced feature extraction under uneven illumination conditions. In DDIO-Mapping, a novel robust strategy is designed that combines grayscale and depth features for optimization instead of only the RBG features in the existing methods. To improve searching efficiency, a new RGBD feature extraction is applied to directly extract both the depth and grayscale features from the RGBD images, which only requires searching the feature points in the 2-D space rather than the enormous 3-D space in K-dimensional (KD) tree. To deal with imbalanced feature extraction, a feature filtering and selection strategy is proposed to adaptively adjust the depth and grayscale weightage. Finally, with the effectively extracted features from RGBD images, a new nonlinear tightly coupled inverse depth residual function is customized to accurately estimate the optimal pose in low-texture environments. The framework is highly robust, accurate, and efficient. Experiments demonstrate that DDIO-Mapping reduces the root-mean-square error by approximately 30% compared to other state-of-the-art algorithms while retaining the same efficiency of approximately 20–35 ms.

3D perception tasks, such as 3D object detection and Bird's-Eye-View (BEV) segmentation using multi-camera images, have drawn significant attention recently. Despite the fact that accurately estimating both semantic and 3D scene layouts are crucial for this task, existing techniques often neglect the synergistic effects of semantic and depth cues, leading to the occurrence of classification and position estimation errors. Additionally, the input-independent nature of initial queries also limits the learning capacity of Transformer-based models. To tackle these challenges, we propose an input-aware Transformer framework that leverages Semantics and Depth as priors (named SDTR). Our approach involves the use of an S-D Encoder that explicitly models semantic and depth priors, thereby disentangling the learning process of object categorization and position estimation. Moreover, we introduce a Prior-guided Query Builder that incorporates the semantic prior into the initial queries of the Transformer, resulting in more effective input-aware queries. Extensive experiments on the nuScenes and Lyft benchmarks demonstrate the state-of-the-art performance of our method in both 3D object detection and BEV segmentation tasks.

This paper considers a MIMO cognitive radio system, where a secondary base station (SBS) transmits signal to multiple secondary users (SUs), while imposing interference to multiple primary users (PUs) in the primary network. We aim at maximizing the weighted sum rate (WSR) of all the SUs subjected to the power constraint of SBS and interference constraints of multiple PUs, by optimizing the linear beamformers matrices associated with all the SUs. We first reformulate the original problem into a convex weighted minimum mean square error (WMMSE) problem, and then a modified subgradient method (MSM) is proposed to solve the WMMSE problem. Simulation results also show that the proposed MSM algorithm outperforms other existing algorithms with lower computational complexity.

In this era of videos, automatic video editing techniques attract more and more attention from industry and academia since they can reduce workloads and lower the requirements for human editors. Existing automatic editing systems are mainly scene-or event-specific, e.g., soccer game broadcasting, yet the automatic systems for general editing, e.g., movie or vlog editing which covers various scenes and events, were rarely studied before, and converting the event-driven editing method to a general scene is nontrivial. In this paper, we propose a two-stage scheme for general editing. Firstly, unlike previous works that extract scene-specific features, we leverage the pre-trained Vision-Language Model (VLM) to extract the editing-relevant representations as editing context. Moreover, to close the gap between the professional-looking videos and the automatic productions generated with simple guidelines, we propose a Reinforcement Learning (RL)-based editing framework to formulate the editing problem and train the virtual editor to make better sequential editing decisions. Finally, we evaluate the proposed method on a more general editing task with a real movie dataset. Experimental results demonstrate the effectiveness and benefits of the proposed context representation and the learning ability of our RL-based editing framework.

With the rapid development and popularization of intelligent terminals, app software has also developed rapidly. The research and practical value of mining user experience (UX) of app software form interaction information are becoming increasingly prominent. The interactive information of app software is multisource homogeneous and heterogeneous. In order to obtain more accurate and more comprehensive app software UX results, the fused multisource information should be analyzed. In this paper, the app software UX analysis method based on multisource information fusion is proposed. First, feature engineering is carried out to extract the features. Then, the feature combination tree is constructed after feature correlation mining. Finally, the multisource app software interactive data are fused, and the result is further analyzed to obtain the information of app software UX. The experiments clearly show that the method can effectively fuse multisource app software interaction data and help to comprehensively mine the app software UX embodied in the data.

While mobile application (app) software is becoming increasingly important in people's daily lives, researchers have the limitation of understanding the details of user operations inside the app. With the update of the Android system and application user interface, relying on manually defined user operation event templates or modifying the app source code can no longer meet the needs of fine-grained user operation analysis in multiparallel applications. In this article, a novel method is proposed for effectively analyzing user operations in parallel apps based on the temporal context of user operation sequences. The authors provide a general framework in the Android system to parse out fine-grained user operations. In addition, the authors build a deep learning model with LSTM-TextCNN to complete user operations in parallel app from global temporal context and app temporal context. The authors collected 240k operations of 12 users over a month. Comparative experiments with a baseline show that the proposed method can efficiently and accurately analyze parallel app user operations.

The pandemic of COVID-19 has increased the demand for the oropharynx sampling robots. For an automatic oropharynx sampling, detection and localization of the oropharynx objects are essential. First, in response to the small-object and real-time needs of visual oropharynx detection, a lightweight multi-attention single-shot multibox detector (MASSD) method is designed. This method can effectively improve the detection accuracy of oropharynx sampling regions, especially small regions, while ensuring sufficient speed by introducing spatial attention, channel attention, and feature fusion mechanisms into the single-shot multibox detector. Second, the proposed MASSD is applied to an oropharyngeal swab (OP-swab) robot system to detect oropharynx sampling regions and conduct autonomous sampling. In the experiment, training and validation based on a custom oropharynx dataset verify the effectiveness and efficiency of the proposed MASSD. The detection accuracy can reach 81.3% of mean average [email protected]:0.95 at 104 frames per second and the application experiment on the OP-swab robot system performs oropharynx sampling with 100% success accuracy in human–robot collaboration strategy.

Aiming at the problems of accurate and fast hand gesture detection and teleoperation mapping in the hand-based visual teleoperation of dexterous robots, an efficient hand gesture detection framework based on deep learning is proposed in this article. It can achieve an accurate and fast hand gesture detection and teleoperation of dexterous robots based on an anchor-free network architecture by using an RGB-D camera. First, an RGB-D early-fusion method based on the HSV space is proposed, effectively reducing background interference and enhancing hand information. Second, a hand gesture classification network (HandClasNet) is proposed to realize hand detection and localization by detecting the center and corner points of hands, and a HandClasNet is proposed to realize gesture recognition by using a parallel EfficientNet structure. Then, a dexterous robot hand-arm teleoperation system based on the hand gesture detection framework is designed to realize the hand-based teleoperation of a dexterous robot. Our method achieves high accuracy with fast speed on public and custom hand datasets and outperforms some state-of-the-art methods. In addition, the application of the proposed method in the hand-based teleoperation system can control the grasping of various objects by a dexterous hand-arm system in real time and accurately, which verifies the efficiency of our method.

Visual hand-based robot teleoperation provides a powerful guarantee for robots to complete complex tasks. However, detection and distinction of dual hands on images are difficult because of the small differences between left and right hands. To solve this problem, a parallel dual-hand detection and distinction method that combines the features of hands with the relationship features between the dual hands and body pose is proposed to achieve robust and accurate dual-hand detection. This parallel dual-hand detection method includes a hand detection module, a body pose estimation module, and a fusion module. In the hand detection module, a hand detector that realizes fast and accurate hand detection by detecting the center and corner points of hands is designed. In the body pose estimation module, a body pose estimator with dual-hand positions is proposed. The fusion module is designed to fuse hand detection and dual-hand estimation results to achieve distinction between left and right hands. Finally, the parallel dual-hand detection method is applied to a bimanual robot teleoperation system by using a designed dual-hand teleoperation framework. The proposed parallel dual-hand detection method can achieve 98.54% mAP of hand detection with 18 frames per second on a custom dual-hand detection dataset, and the bimanual robot teleoperation method can achieve 95.4% average accuracy for teleoperation tasks. Experimental results show the high accuracy and speed of our proposed parallel dual-hand detection method and its practicability in bimanual robot teleoperation.

The visual analysis of the mouth cavity plays a significant role in the pathogen specimen sampling and disease diagnosis of the mouth cavity. Aiming at performance defects of general detectors based on deep learning in detecting mouth cavity components, this article proposes a mouth cavity analysis network (MCNet), which is an instance segmentation method with spatial features, and a mouth cavity dataset (MCData), which is the first available dataset for mouth cavity detecting and segmentation. First, given the lack of a mouth cavity image dataset, the MCData for detecting and segmenting key parts in the mouth cavity was developed for model training and testing. Second, the MCNet was designed based on the mask region-based convolutional neural network. To improve the performance of feature extraction, a parallel multiattention module was designed. Besides, to solve low detection accuracy of small-sized objects, a multiscale region proposal network structure was designed. Then, the mouth cavity spatial structure features were introduced, and the detection confidence could be refined to increase the detection accuracy. The MCNet achieved 81.5% detection accuracy and 78.1% segmentation accuracy (intersection over union = 0.50:0.95) on the MCData. Comparative experiments with the MCData showed that the proposed MCNet outperformed state-of-the-art approaches with the task of mouth cavity instance segmentation. In addition, the MCNet has been used in an oropharyngeal swab robot for COVID-19 oropharyngeal sampling.

New challenges such as automation, connection, electrification, and sharing (ACES) have brought disruptive changes to vehicles, transportation, and mobility services, which urgently requires an ideal solution for sustainable transportation. This paper introduces the Internet as a paradigm and, for the first time, proposes the Transportation Internet (TI), inspired by the similarity between the Internet and transportation. Referring to the construction ideas of the Internet, this paper establishes the framework of TI, proposes the transportation router based on the transportation switching and routing models, and preliminarily forms a large-scale automatic transportation solution. Following the latest technologies of the Internet, this paper further presents the software-defined transportation (SDT) by separating the control plane and transport plane of the transportation router, which can enhance transportation routing and provide Internet-like capabilities such as centralized intelligent control, terminals plug-and-play, and open application ecology. The evaluation of the prototype system shows promising results. The software-defined signals (SDS) can save 36% energy compared to signal machines, and the software-defined vehicles (SDV) automatic driving can save 24% energy compared to manual driving. Overall, TI brings innovations to sustainable transportation, and provides a framework for a new generation of Intelligent Transportation Systems (ITS).

Semantic Scene Completion (SSC) aims to reconstruct complete 3D scenes with precise voxel-wise semantics from the single-view incomplete input data, a crucial but highly challenging problem for scene understanding. Although SSC has seen significant progress due to the introduction of 2D semantic priors in recent years, the occluded parts, especially the rear-view of the scenes, are still poorly completed and segmented. To ameliorate this issue, we propose a novel deep learning framework for 3D SSC, named Planar Convolution and Attention-based Network (PCANet), to effectively extend high-precision predictions of the front-view surface to the rear-view occluded areas. Specifically, we decompose the traditional convolutional layer into three successive planar convolutions to form a Planar Convolution Residual (PCR) block, which maintains the planar features of the 3D scene. Afterward, the Planar Attention Module (PAM) is proposed to capture three different planar attentions and harvest the global context from the front surface to the rear occluded areas to improve the overall accuracy. Extensive experiments on the real NYU and NYUCAD datasets and the synthetic SUNCG-RGBD dataset demonstrate that our proposed framework can generate high-quality SSC results in both front and rear views and outperforms the state-of-the-art approaches trained in an end-to-end manner without additional data.

Global temporal information and local semantic information are essential cues for high-performance online object detection in videos. However, despite their promising detection accuracy in most cases, most state-of-the-art approaches have following two limitations: invalid background/scale suppression and inadequate temporal information mining between frames. Many jobs currently focus on temporal information learning based on a single frame. In this article, we propose an attentional global–local information learning network; this is one of the first attempts to fully use both types of information between frames. Attention maps are creatively utilized to transfer temporal contexts between frames. This also effectively alleviates the adverse effects of scale changes. Furthermore, empowered by a detailed framework, a proposed detector effectively uses multilevel feature extraction. Given these contributions, the proposed detector achieves state-of-the-art performance on challenging benchmarks. Finally, practical experiments are conducted on a space human–robot interaction platform.

Superior dexterity and extreme flexibility are typical advantages for concentric cable-driven manipulators working in confined spaces. However, its inverse kinematics and configuration planning are very complicated. In this article, we propose a spatial biarc method for the above problem. The distinguishing feature of this method is that input parameters are two positions and two direction vectors in three-dimensional (3-D) space, and the output is a reasonable spatial biarc for controlling a concentric cable-driven manipulator in 3-D space. This method has the following three advantages. First, the positions and direction vectors of the base and inner distal tip are considered simultaneously. In addition, the length and ratio of the overlapped section and separated section can be adjusted by changing the length of the direction vectors. Furthermore, by judging the angular value of the direction vectors, one can predetermine whether the spatial configuration of the entire arm is C- or S-shaped. The proposed method realizes the parameterization of a concentric cable-driven manipulator, which makes it convenient to intuitively control the manipulator to achieve interference-free motion trajectory planning in confined spaces. Finally, trajectory tracking inspections are simulated and experimentally executed. It can be seen from results that the proposed spatial biarc method can provide reasonable solutions for concentric cable-driven manipulators. The method is especially favorable in terms of 3-D-pose-determination problem and trajectory-planning problem. It can also be applied to other manipulators with similar configurations. Without loss of generality, when the given points and direction vectors are coplanar, the proposed spatial biarc method can be transformed to a planar biarc method.

Driven by the SARS-CoV-2 pandemic, demand for oropharyngeal swab sampling (OP-swabs) is surging. However, medical staff can easily become infected by the virus during the sampling process. In an effort to combat this, we developed a novel, intrinsically safe rigid- flexible coupling (RFC) manipulator to improve the safety and reliability of OP-swab sampling to test for COVID-19, which is presented herein. Suitable sampling areas and the necessary contact force for OP-swab sampling tasks are carefully investigated, and three typical sampling paths outlined that could be performed by a robotic system. This is followed by a detailed description of an intrinsically safe bionic micro-pneumatic actuator (MPA) that was designed and fabricated as the main component of the RFC manipulator. The developed RFC manipulator’s kinematic modeling, motion planning, and force control capacities were designed for OP-swab sampling scenarios. The system was then validated using both an oral cavity phantom and human volunteers, with comparative experiments on the swab quality of the OP-swab sampling approach conducted in both robotic and manual modes. The results indicate that fully-automated sampling based on this design would be feasible.

The Canadarm2, also named Space Station Remote Manipulator System (SSRMS), is a 7-joint redundant manipulator. Without spherical wrists, the singularity analysis and avoidance of these manipulators are very difficult. In this paper, a method is presented to analytically identify its singular configurations based on the elementary transformation of Jacobian matrix. Firstly, we constructed a general kinematics model to describe them in a united manner. Correspondingly, the differential kinematics equation and the modified form are derived. Secondly, the singularity conditions are isolated and collected in a 3×4 sub-matrix by several times row transformation of the modified Jacobian matrix, which is partitioned into a block-triangle matrix. Finally, all the singularity configurations are determined by analyzing the rank degeneracy conditions of the 3×4 sub-matrix. The proposed method isolates the singularity conditions, and collects them in a 3×4 sub-matrix, largely reducing the computation workload.

In this paper, we present a novel Movable Gripper (MovGrip) which targets on climbing non-enclosable rectangular trusses such as bridges and space stations. It is designed with a transformation mechanism which provides the features of parallel grippers, rotatory grippers, and active wheels. For truss climbing, MovGrip acts as a parallel gripper which allows the change in gripping width according to different size of trusses. Since MovGrip is equipped with active wheels, fast climbing motion can therefore be realized. Moreover, MovGrip can be transformed into a mobile platform which is suitable for the navigation on ground. It weighs only 500 grams with a climbing speed of 3 cm/s. To maintain the climbing stability, we steer the rotation axis of wheels. By this, a directional pulling force (from MovGrip to truss surface) can be distributed from the drive force of wheels which pulls MovGrip towards the truss while climbing. Experimental results show that tilting of MovGrip can be auto-adjusted based on the proposed design. Also, it is shown that the load carrying capability of MovGrip is approximately 1 kg which is 2 times of its weight.