Recent developments in action recognition and object tracking have facilitated intelligent surveillance technologies. However, the two domains are predominantly studied separately due to their high computational complexity, which poses challenges for their concurrent deployment in real-world Internet of Things (IoT) surveillance scenarios. In this work, we propose an integrated skeleton-based framework for lightweight violent action recognition and attacker tracking, which leverages shared skeleton information between modules to reduce overall computation, enabling timely processing of surveillance videos. For violent action recognition, we introduce an efficient model, named skeletal topology refining graph convolution (STR-GC), which considers the human interaction, motor coordination and bone connectivity of joints to explore representative topologies of violent behaviors. For attacker tracking, we present a skeleton-guided correlation filter (SGCF), which retains the computational efficiency of the correlation filter and exhibits robustness under challenging conditions, such as large-scale variations, severe deformations, occlusions, boundary effects, and low resolution. Experiments on three benchmark datasets show that our framework not only outperforms the existing state-of-the-art methods for violent action recognition and attacker tracking, but also supports monitoring applications with reduced computational cost.

Training a universal controller for robots with different morphologies is a promising research trend, since it can significantly enhance the robustness and resilience of the robotic system. However, diverse morphologies can yield different dimensions of state space and action space, making it difficult to comply with traditional policy networks. Existing methods address this issue by modularizing the robot configuration, while do not adequately extract and utilize the overall morphological information, which has been proven crucial for training a universal controller. To this end, we propose GCNT, a morphology-agnostic policy network based on improved Graph Convolutional Network (GCN) and Transformer. It exploits the fact that GCN and Transformer can handle arbitrary number of modules to achieve compatibility with diverse morphologies. Our key insight is that the GCN is able to efficiently extract morphology information of robots, while Transformer ensures that it is fully utilized by allowing each node of the robot to communicate this information directly. Experimental results show that our method can generate resilient locomotion behaviors for robots with different configurations, including zero-shot generalization to robot morphologies not seen during training. In particular, GCNT achieved the best performance on 8 tasks in the 2 standard benchmarks.

Dynamic objects diversify the distribution of point cloud in the map, degrading the performance of the robotic downstream tasks. To address this problem, we present a novel real-time dynamic instance-aware static mapping framework called DynaInsRemover, which exploits the geometric discrepancies between instances to efficiently remove dynamic objects and preserve more details of static map. It contains the Instance Occupancy Check module for initial dynamic instance proposal and the Instance Belief Update module for reverting false positives. We quantitatively evaluate our approach performance on the SemanticKITTI dataset and validate it in a real-world environment. Experimental evaluations show that our method achieves very promising results in dynamic environments. The implementation of our method is available as open source at: https://github.com/Zhaohuanfeng/DynaInsRemover.git.

Martian terrain segmentation aims to assign all pixels of an input image with various terrain labels, which provides a firm support for the downstream research on rover traversing and geologic analysis tasks. However, existing studies in this field suffer from limitations in two aspects: one is the lack of large-scale and high-quality Martian terrain datasets, and the other is the over-reliance on purely supervised learning that is very data-hungry and sensitive to domain shifts among different datasets. In this article, we overcome these from the perspective of both data and methodology. First, we publish MarsScapes, a panorama dataset with appreciable data volume and fine-grained annotations for Martian terrain understanding. The dataset contains 195 terrain panoramas composed of 3779 subimages, and all pixels in the panoramas are split into nine semantic categories. Then, we propose the first transformer-based unsupervised domain adaptation (UDA) framework (UDAFormer) for the cross-domain terrain segmentation on Mars, which consists of a teacher–student model and an output-guided biased sampling (OGBS) module. The teacher–student model performs knowledge distillation to explore robust cross-domain features, where a modified augmentation regularization (MAR) is designed to alleviate the interference of undesirable augmentations to domain adaption. The OGBS helps the teacher–student network to emphasize the categories that tend to be ambiguous or submerged during the training, elevating the overall accuracy for the UDA segmentation of Martian terrains. Extensive experiments on the MarsScapes and another dataset called Mars-Seg demonstrate the superiority of UDAFormer over the state-of-the-art methods in UDA Martian terrain segmentation.

Martian surface, as a typical unstructured terrain, is extremely challenging for Mars exploration missions. Commonly, Mars rovers require multiple sensors to explore such harsh environment, such as depth cameras, range finder and other devices. However, the onboard load, power and storage of rovers are not sufficient to achieve high-level stereoscopic perception, which can be adverse to downstream tasks such as visual navigation and scientific exploration. To this end, in this paper we propose a high-level awareness perception light-weight framework using only close-shot monocular images to implement semantic 3D reconstruction of Martian landforms. This framework consists of two parts. One is a semantic segmentation module based on the proposed real-time Mars terrain segmentation (RMTS) network to extract intra-class and inter-class contexts by local supervision. The other is a depth generation module based on a dual-encoder pix2pix network to encode the visual and semantic information of monocular images, simultaneously. To validate the proposed framework, we construct a Martian planar-stereo dataset based on AI4Mars, an open-source semantic segmentation dataset for Mars surface. It contains monocular close-up Martian images, semantic images and depth images that match each other. After training, the accuracy of proposed semantic segmentation model can reach 84.0% mIoU, with 152.2 FPS on a single RTX6000-24GB GPU. The absolute relative error of pixels in depth images between generation model and the ground truth is 0.367, while the root mean square error gets to 0.510, and the accuracy is 0.753 with 42.9 FPS. The overall environment perception scheme is with 9.5FPS.

In this paper, we propose a multi-object tracking framework for videos captured by UAVs, considering motion imperfection in the following two aspects: 1) motion blurring of objects due to high-speed motion of the UAV and the objects, deteriorating the performance of the detector; 2) motion coupling of the global movement of the UAV camera with the object motion, resulting in the nonlinearity of objects trajectories in adjacent frames and further more difficult to predict. For motion blurring, this paper proposes a hybrid deblurring module that deals with the blurred frames while retaining the clear frames, trading off between video tracking performance and spatio-temporal consistency. For motion coupling, we proposed a motion compensation module to align adjacent frames by feature matching, and the corrected target position is obtained in the next frame to alleviate the interference of camera movement with tracking. We evaluate the proposed methods on VisDrone dataset and validate that our framework achieves new state-of-the-art performance on UAV-based MOT systems.

Martian rock segmentation aims to separate rock pixels from background, which plays a crucial role in downstream tasks, such as traversing and geologic analysis by Mars rovers. The U-Nets have achieved certain results in rock segmentation. However, due to the inherent locality of convolution operations, U-Nets are inadequate in modeling global context and long-range spatial dependencies. Although emerging Transformers can solve this, they suffer from difficulties in extracting and retaining sufficient low-level local information. These shortcomings limit the performance of the existing networks for Martian rocks that are variable in shape, size, texture, and color. Therefore, we propose RockFormer, the first U-shaped Transformer framework for Mars rock segmentation, consisting of a hierarchical encoder–decoder architecture with a feature refining module (FRM) connected between them. Specifically, the encoder hierarchically generates multiscale features using an improved vision Transformer (improved-ViT), where both abundant local information and long-range contexts are exploited. The FRM removes less representative features and captures global dependencies between multiscale features, improving RockFormer’s robustness to Martian rocks with diverse appearances. The decoder is responsible for aggregating these features for pixelwise rock prediction. For evaluation, we establish two Mars rock datasets, including both real and synthesized images. One is MarsData-V2, an extension of our previously published MarsData collected from real Mars rocks. The other is SynMars, a synthetic dataset sequentially photographed from a virtual terrain built referring to the TianWen-1 dataset. Extensive experiments on the two datasets show the superiority of RockFormer for Martian rock segmentation, achieving state-of-the-art performance with decent computational simplicity.

Semantic segmentation of Mars scenes has a crucial role in Mars rovers science missions. Current CNN-based composition of U-Net has powerful information extraction capabilities; however, convolutional localization suffers from the limited global context modeling capability. Although Transformer global modeling has performed well, it still encounters obstacles in the extraction and retention of low-level features. This issue is particularly relevant for Martian rocks with their varying shapes, textures, and sizes in Mars scenes. In this paper, we propose a novel Transformer semantic segmentation framework for Martian rock images, called MarsFormer, that consists of an encoder-decoder structure connected through a Feature Enhancement Module (FEM) and a Window Transformer Block (WTB). Specifically, multi-scale hierarchical features are generated by the Mix Transformer encoders, Upgraded-FFN decoder (UFD) fuses and filters features at different scales, preserving the rich local and global contextual information. FEM enhances the inter-multi-scale feature correlation from both spatial and channel perspectives. WTB captures the long-range contexts and preserves the local features. We built two datasets of synthetic and real Martian rocks. The synthetic dataset is SynMars, referencing data from the ZhuRong rover taken from its Virtual Terrain Engine. The other dataset is MarsData-V2, from real Mars scenes, and published recently in our previous study. Extensive experiments conducted on both datasets showed that MarsFormer achieves superiority in Martian rock segmentation, obtaining state-of-the-art performance with favorable computational simplicity. The data is available at: https://github.com/CVIR-Lab/SynMars.

This article presents two kernel-based rock detection methods for a Mars rover. Rock detection on planetary surfaces is particularly pivotal for planetary vehicles regarding navigation and obstacle avoidance. However, the diverse morphologies of Martian rocks, the sparsity of pixel-wise features, and engineering constraints are great challenges to current pixel-wise object detection methods, resulting in inaccurate and delayed object location and recognition. We therefore propose a region-wise rock detection framework and design two detection algorithms, kernel principle component analysis (KPCA)-based rock detection (KPRD) and kernel low-rank representation (KLRR)-based rock detection (KLRD), using hypotheses of feature and sub-spatial separability. KPRD is based on KPCA and is expert in real-time detection yet with less accurate performance. KLRD is based on KPRD with KLRR which can generate more precise rock detection results with less delay. To validate the efficiency of the proposed methods, we build a small-scale Martian rock dataset, MarsData, containing various rocks. Preliminary experimental results show that our methods are efficient in dealing with complex images containing rocks, shadows, and gravel. The code and data are available at: https://github.com/CVIR-Lab/MarsData.

Recently, intelligent monitoring technologies have been developing rapidly, among them, human action recognition and target tracking have made breakthroughs in their respective fields. However, the huge computational cost makes the integration of the two technologies difficult. In this paper, we design a skeleton-based monitoring system that realizes violent action recognition and abuser tracking with a relatively low overall complexity. For violence recognition, we put forward a novel Skeletal Context Attention Network (SCAN), which is lightweight yet effective to exploit spatial and temporal representations of skeleton data. For abuser tracking, we present a Skeleton-Guided Correlation Filter (SGCF) that can track a perpetrator continuously even in some extreme cases, such as drastic changes in speed or color. Experiments on two benchmark datasets show that the proposed system not only outperforms the existing state-of-the-art methods for violent action recognition and abuser tracking, but also implements the both tasks with less computation.

This paper presents a novel approach to fault tolerant reconfiguration of modular self-folding robots. Among various types of faults that probably occur in the modular system, we focus on the tolerance of complete actuation failure of active modules that might cause imprecise robotic motion and even reconfiguration failure. Our approach is to utilize the reconfigurability of modular self-folding robots and investigate intra-module connection to determine initial patterns that are inherently fault tolerant. We exploit the redundancy of actuation and distribute active modules in both layout-based and target-based scenarios, such that reconfiguration schemes with user-specified fault tolerant capability can be generated for an arbitrary input initial pattern or 3D configuration. Our methods are demonstrated in computer-aided simulation on the robotic platform of Mori, a modular origami robot. The simulation results validate that the proposed algorithms yield fault tolerant initial patterns and distribution schemes of active modules for several 2D and 3D configurations with Mori, while retaining generalizability for a large number of modular self-folding robots.

In this work, we propose a novel auto rock detection scheme via sparse-based background modeling for planetary rover. Instead of pixel-level image processing and pre-training, we deal with the detection issue by region contrast enhancement. Our method consists of a rough detection algorithm to auto-label conservative background regions and a background modeling algorithm to precisely detect rocks. Background is reconstructed in feature space with the labeled regions as dictionary via sparse representation. Contrast map is enhanced by the error of original image and reconstructed background. Afterwards, rock is directly segmented by an auto threshold. Our method is simple yet efficient, primary results show that the proposed algorithm yields highly precise detection results, compared with current state-of-art edge-based algorithms.

Modular self-folding robots are versatile systems that can change their own shape from two-dimensional patterns at instant commands. This reconfigurability is commonly restrained by power limitation in autonomous environments, The robotic systems with insufficient torque may lead to inaccurate movements and even transformation failures. This paper presents methodology for optimized reconfiguration planning with torque limitation in modular self-folding robots. We determine reconfiguration schemes with optimal initial pattern and robotic base that result in minimal peak torque by minimizing robotic inertia of the modular architecture. We present minimal bounding box and capacitated spanning tree heuristic algorithms to generate optimal initial patterns and propose 3 heuristic rules for robotic base selection. Our approach is demonstrated in simulation by applying the algorithms to the robotic concept of Mori, a modular origami robot. The simulation results show that the proposed algorithms yield reconfiguration schemes with low peak torque, thereby appropriate for real-time applications in modular robotic systems.