With the increasing demands for perception accuracy in autonomous driving, there is a growing focus on fine-grained 3D semantic occupancy prediction. Effectively representing detailed three-dimensional scenes has become a significant challenge in the development of this task. In this paper, we present a novel transformer-based framework named CVFormer, which leverages two-dimensional circum-views from the ego to excavate three-dimensional features of the surrounding environment. Circum-views provide a novel solution for effectively addressing the representation of dense and fine-grained scenes. Specifically, a multi-attention module CTMA is designed for fusing temporal features from circum-views to fully exploit the spatiotemporal correlations between frames and capture more comprehensive clues. Furthermore, a novel 2D projection constraint is established by observing objects from different perspective directions, and multiple 3D constraints based on object invariance and semantic consistency are also conducted for supervising the network, which enhances its performance of understanding the scene. Experimental results on nuScenes dataset demonstrate that the proposed CVFormer obviously outperforms existing methods for occupancy prediction.

Panoptic segmentation is a challenging perception task, which can help robots to comprehensively perceive the surrounding environment. In the task, we notice that semantic, instance, and panoptic have rich relations, however, which are rarely explored. In this work, we propose a novel panoptic, instance, and semantic bridged network to delve into the reciprocal relation. To make semantic and instance benefit from each other, we design a novel Gated Encoding (GE) module, incorporating complementary cues between semantic and instance heads through the gated mechanism. In addition, a novel edge-aware consistency constraint among edges of each task is presented, which exhaustedly exploits geometric constraints, to boost the segmentation quality of challenging edges. Experimental results on the Cityscapes and MS-COCO datasets demonstrate that our approach achieves state-of-the-art performance in an efficient CNN-based paradigm, attaining a balance between accuracy and efficiency.

The weakly-supervised audio-visual video parsing (AVVP) task aims to parse duration and categories of each snippet when only the video-level event labels are provided. Most methods either leverage attention mechanisms to explore cross-modal and cross-video event semantics or alleviate label noise to improve performance. However, the distributional modality discrepancy caused by the heterogeneity of signals remains a significant challenge. To this end, we propose a novel cross-modal common-specific feature learning method (cm-CS) to map the modal features into modality-common and modality-specific subspaces. The former aims to capture similar high-level scene cue across different modalities, while the later attempts to capture specific cue. The proposed method is applied among and across in-visual 2D-3D modalities, audio-visual modalities, respectively. In addition, we design a training strategy to strengthen the learning of similarity and differences across modalities. Experiments show a large improvement of our method against existing works on the Look, Listen, and Parse (LLP) dataset (e.g. from 58.9% to 62.9% in video-level visual metric).

Unsupervised domain adaptation (UDA) is proposed to better adapt the network trained on labeled synthetic data to unlabeled real-world data for addressing the annotation cost. However, most of these methods pay more attention to domain distributions in input and output stages while ignoring the important differences in semantic expressions and local details in middle feature stages. Therefore, a novel UDA network named FeatDANet is presented to align feature-level domain distributions at each encoder layer. Specifically, two attention-based modules abbreviated as IFAM and DFLM are designed and implemented by mixing queries and keys between domains for advisable domain adaptation. The former realizes Inter-domain Features Alignment by transferring feature style, and the latter achieves Domain-invariant Features Learning robustly for the domain shift. Furthermore, FeatDANet is constructed as a self-training network with three weight-sharing branches, and an improved pseudo-labels learning strategy is suggested by identifying more confident pseudolabels and maximizing the use of pseudo-labels. It increases the participation of unlabeled data and also ensures stability in training. Extensive experiments show that FeatDANet achieves state-of-the-art performances on the tasks of GTA→Cityscapes and Synthia→Cityscapes.

In recent times, as a result of the emergence of high-throughput satellites, there is an increasing need for high-speed and high-quality communication. The resources available on these satellites are valuable, and the rational allocation of these resources has become the focal point of research. This paper establishes a resource allocation model for OFDM multi-beam high-throughput satellites that takes into account inter-beam interference, inter-sub carrier interference, and satellite downlink loss. The objective of this model is to ensure fairness among satellite users, and this is achieved through the use of quantum genetic algorithms for resource allocation. Compared with the fixed allocation method, our algorithm has better performance.

In adaptive array beamforming, when the interference position changes rapidly, the antenna receiving platform vibrates and moves, or the update speed of adaptive weights is slow, the interference may move out of the null position and cannot be effectively cancelled, and then the conventional method may be completely invalid. Therefore, in this paper, we propose an interference null widening algorithm based on the U-V domain expansion of the steering vector of interference source. The algorithm solves the problem of coupling the pitch angle and the azimuth angle of the interference signal by adding the adjacent interference, introducing the covariance matrix taper method, and mapping the expansion of angle domain to the expansion of independent U-V domain. The simulation results show that the algorithm has a deep depth of null in the range of null widening, and can effectively improve the anti-interference ability of the system in dynamic environment.

Compression-sensing-based DOA estimation method can overcome the disadvantages of the traditional spatial spectral estimation algorithm, but the calculation task is heavy and time-consuming, which is limited by the power and processing capability on satellite. At the same time, with the change of the space environment and the aging of the devices, the satellite array error problem will appear. Therefore, we proposed a joint subspace decomposition and compressed sensing approximation algorithm to solve the problem of DOA estimation. We first use the Fast Root-MUSIC algorithm to correct the satellite array error and perform a preliminary DOA estimation to reduce the search range of the compressed sensing algorithm, then performed an accurate estimation of the DOA by the L1-svd algorithm. Simulation results show that the proposed algorithm has better performance and less computation complexity under low signal to noise ratio, small fast beat number, as well as in the presence of array error.

Occlusions have long been a hard nut to crack in optical flow estimation due to ambiguous pixels matching between abutting images. Current methods only take two consecutive images as input, which is challenging to capture temporal coherence and reason about occluded regions. In this paper, we propose a novel optical flow estimation framework, namely MFCFlow, which attempts to compensate for the information of occlusions by mining and transferring motion features between multiple frames. Specifically, we construct a Motion-guided Feature Compensation cell (MFC cell) to enhance the ambiguous motion features according to the correlation of previous features obtained by attention-based structure. Furthermore, a TopK attention strategy is developed and embedded into the MFC cell to improve the subsequent matching quality. Extensive experiments demonstrate that our MFCFlow achieves significant improvements in occluded regions and attains state-of-the-art performances on both Sintel and KITTI benchmarks among other multi-frame optical flow methods.

Category-level 6D object pose estimation aims to predict the full pose and size information for previously unseen instances from known categories, which is an essential portion of robot grasping and augmented reality. However, the core challenge of this task still is the enormous shape variation within each category. With regard to the challenge, we propose a novel framework SD-Pose, which utilizes the instance-category structural discrepancy and the potential geometric-semantic association to enhance the exploration of the intra-class shape information. Specifically, an information exchange augmentation (IEA) module is introduced to supplement the instance-category structural information by their structural discrepancy, thus facilitating the enhanced geometric information to contain both the character of instance shape and the commonality of category structure. For complementing the deficiencies of structural information adaptively, a semantic dynamic fusion (SDF) module is further designed to fuse semantic and geometric features. Finally, the proposed SD-Pose framework equipped with the IEA and SDF modules hierarchically supplements instance-category structural information in a stacked manner and achieves state-of-the-art performance on the CAMERA25 and REAL275 datasets.

Traffic sign recognition plays a crucial role in both unmanned vehicles and advanced driver assistance systems. Although many recent deep-learning-based approaches have made some progress on this task, it still suffers from the significant changes in scale and rotation, the presence of objects of the same color appearance, as well as the high similarity between classes. In this paper, we first propose a traffic sign detection network, named AAMNet, by adopting the generally framework of one-stage anchor-free instance segmentation models. For the feature extraction, a novel Aliased residual block is designed to support the encoder to retain more detailed information from the shallow low-level features. For decoder, we introduce a channel attention module into the mask generator to implement an Adaptive focus localizer head, which can filter out the irrelevant prototype masks. A Mask guided center loss is further constructed to improve the localization accuracy. Then, considering the difficulty of text based traffic signs recognition, for the first time, we developed a text-traffic-signs dataset TextTSD covering rich scenes and multiple languages. Extensive experiments both on TT100k and TextTSD show that our AAMNet gains a competitive performance compared with several state-of-the-art methods.

Estimating head pose from a single RGB image has recently attracted considerable research attention. Prior arts employ a CNN backbone to process face images and then directly output Euler angles. We argue that they may ignore essential features that are highly correlated to head pose due to the non-global perspective, and the ambiguity and discontinuity issues of Euler angles representation could interfere with the performance of challenging samples. In this paper, we formulate the head pose estimation problem into quaternion representation space and propose a novel framework named Structural Relation-aware Network (SRNet). Different from previous methods, our SRNet explicitly explores the correlation among different regions of the face for mining global facial structure information. Furthermore, in order to boost robustness and generalization of the model, a hard example mining (HEM) strategy is designed to mitigate the data imbalance issue by adjusting the contributions of examples in different states to loss. Extensive experiments demonstrate that our method outperforms the current state-of-the-art alternatives on the public benchmark datasets: AFLW2000 and BIWI.

Neighborhood construction plays a key role in point cloud processing. However, existing models only use a single neighborhood construction method to extract neighborhood features, which limits their scene understanding ability. In this paper, we propose a learnable Dual-Neighborhood Feature Aggregation (DNFA) module embedded in the encoder that builds and aggregates comprehensive surrounding knowledge of point clouds. In this module, we first construct two kinds of neighborhoods and design corresponding feature enhancement blocks, including a Basic Local Structure Encoding (BLSE) block and an Extended Context Encoding (ECE) block. The two blocks mine structural and contextual cues for enhancing neighborhood features, respectively. Second, we propose a Geometry-Aware Compound Aggregation (GACA) block, which introduces a functionally complementary compound pooling strategy to aggregate richer neighborhood features. To fully learn the neighborhood distribution, we absorb the geometric location information during the aggregation process. The proposed module is integrated into an MLP-based large-scale 3D processing architecture, which constitutes a 3D semantic segmentation network called DNFA-Net. Extensive experiments on public datasets containing indoor and outdoor scenes validate the superiority of DNFA-Net.

Depth estimation and semantic edge detection are two key tasks in computer vision, which have made great progress. To date, how to associatively predict the depth and the semantic edge is rarely explored. In this work, we first propose a flexible two-branch framework that can make the two tasks take advantage of each other, achieving a win-win situation. Specifically, for the semantic edge detection branch, an Enhanced Edge Weighting strategy (EEW) is designed, which learns weight information from the by-product of depth branch, depth edge, to enhance edge perception in features. Meanwhile, we make depth estimation benefit from semantic edge detection through introducing Depth Edge Semantic Classification module (DESC). Furthermore, a double reconstruction (D-reconstruction) approach is presented, together with semantic edge-guided disparity smoothing loss to mitigate the ambiguities of the self-supervised manner for depth estimation. Experiments on the Cityscapes dataset demonstrate that our framework outperforms the state-of-the-art method in depth estimation along with a significant improvement in semantic edge detection.

Recovering depth information from a single image is a long-standing challenge, and self-supervised depth estimation methods have gradually attracted attention due to not relying on high-cost ground truth. Constructing an accurate photometric loss based on photometric consistency is crucial for these self-supervised methods to obtain high-quality depth maps. However, the photometric loss in most studies treats all pixels indiscriminately, resulting in poor performance. In this paper, we propose two modules based on the spatial and temporal cues to refine the photometric loss. Delving into the geometric model of photometric consistency, we introduce a depth-aware pixel correspondence module (DPC) inside the monocular depth estimation pipeline. It reduces the uncertainty of photometric errors by applying the homography matrix to the projection of corresponding pixels in far regions instead of the fundamental matrix. Furthermore, we design an omnidirectional auto-masking module (OA) to boost the robustness of our model, which utilizes temporal sequences to generate disturbance poses and hypothetical views to distin-guish dynamic objects with different directions that violate the photometric consistency. Experiments on the KITTI and the Make3d datasets reveal that our framework achieves state-of-the-art performance.

The 3D semantic map plays an increasingly important role in a wide variety of applications, especially for many kinds of task-driven robots. In this paper, we present a semantic mapping methodology for 3D semantic map obtaining from RGB-D scans. In contrast to existing methods that use 3D annotated information as supervisory, we focus on accurate 2D frame labeling and combine labels in 3D space using semantic fusion mechanism. For scene parsing, a two-stream network with a novel discriminatory mask loss is proposed to explore sufficient extraction and fusion of RGB and depth information achieving steadily semantic segmentation. The discriminatory mask guides the cross-entropy loss function and interprets the influence of different pixels on back-propagation, which reduces the harmful effects of the depth noise or the fallible annotation at the edges of objects. After the correspondences between frames are provided, these semantic frames are fused in unified 3D coordinates using the novel label-oriented voxelgrid filter. It can ensure the intra-frame spatial continuity and the inter-frame spatiotemporal consistency through introducing the label-oriented statistical principle into labeled point clouds. In order to avoid the unfavorable interference between uncorrelated frames, we further propose an adaptive grouping algorithm by applying the view frustum filter to group frames with sufficient overlap as a segment. To this end, we demonstrate the effectiveness of the proposed method on the 2D/3D semantic label benchmark of ScanNetv2 and Cityscapes datasets.

Learning to distinguish independent moving objects from the observed optical flow with a moving camera remains challenging. In this work, we first present a novel camera pose compensation (CPC) scheme. With the help of ingenious geometric analysis, it breaks the observed optical flow into patterns that are easier to interpret for the motion segmentation network. Secondly, we further refine such compensation with a camera parameter aware (CPA) module to account for poses’ errors in the CPC processing and enhance the entire network’s tolerance to noises. Additionally, an MMPNet is developed to intensify the identification ability of overall motion patterns. It reaches a larger receptive field with a bottom-up information transmission structure and integrates motion information at different granularities. We demonstrate the benefits of our framework on FlyingThings3D and Monkaa datasets. Without the complement of semantic information, our approach outperforms the top methods for moving objects segmentation.

Neural networks have recently achieved impressive success in semantic and instance segmentation on 2D images. However, their capabilities have not been fully explored to address semantic instance segmentation on unstructured 3D point cloud data. Digging into the regional feature representation to boost point cloud comprehension, we propose a region-feature-enhanced structure consisting of adaptive regional feature complementary (ARFC) module and affinity-based regional relational reasoning (AR3) module. The ARFC module aims to complement low-level features of sparse regions adaptively. The AR3module emphasizes on mining the potential reasoning relationships between high-level features based on affinity. Both the ARFC and AR3modules are plug-and-play. Besides, a novel dual spatial-aware discriminative loss is proposed to improve the discrimination of instance embedding. Our proposal-free point cloud instance segmentation network (RegionNet) equipped with the region-feature-enhanced structure and dual spatial-aware discriminative loss achieves state-of-the-art performance on S3DIS dataset and ScanNet-v2 dataset.

The recent surge in computer science (CS) education for children has led to the popularization of blocks-based programming languages (BBPLs), which offer a syntax-directed approach to teach fundamental programming concepts. Most BBPL environments are designed with imperative programming in mind. The primary building blocks represent the key constructs that support sequencing, iteration, and selection, all in an imperative style. Some BBPL environments support bi-directionality and can show the source code of the same program represented as either blocks or in a general-purpose textual language. There have been few alternatives to the imperative primitives used in these environments. In this Vision paper, we propose a paradigmatic change to the underlying and front-facing structures of BBPL environments by replacing them with Model-driven Engineering (MDE) primitives, such as declarative transformation rules and metamodels. We have implemented a prototype of our envisioned system in a modeling environment to show the feasibility of the approach, which we believe could be used by young children. The contribution of our vision is a demonstration of the potential for a new thread of research that lies at the intersection of modeling and CS education for children. We aim to spark an interest in MDE among CS education researchers, while encouraging the modeling community to consider how MDE might be taught to younger students.

Abstract Hardware/software (HW/SW) partitioning, that decides which components of an application are implemented in hardware and which ones in software, is a crucial step in embedded system design. On modern heterogeneous embedded system platform, each component of application can typically have multiple feasible configurations/implementations, trading off quality aspects (e.g. energy consumption, completion time) with usage for various types of resources. This provides new opportunities for further improving the overall system performance, but few works explore the potential opportunity by incorporating the multiple choices of hardware implementation in the partitioning process. This paper proposes three algorithms for multiple-choice HW/SW partitioning of tree-shape task graph on multiple processors system on chip (MPSoC) with the objective of minimizing execution time, while meeting area constraint. Firstly, an efficient heuristic algorithm is proposed to rapidly generate an approximate solution. The obtained solution produced by the first algorithm is then further refined by a customized Tabu search algorithm. We also propose a dynamic programming algorithm to calculate the exact solutions for relatively smaller scale instances. Simulation results show that the proposed heuristic algorithm is able to quickly generate good approximate solutions, and the solutions become very close to the exact solutions after refined by the proposed Tabu search algorithm, in comparison to the exact solutions produced by the dynamic programming algorithm.

Compared to traditional camera-based computer vision and imaging, radio imaging based on wireless sensing does not require lighting and is friendly to privacy. This work proposes a deep learning radio imaging solution to visualize real-time user indoor activities. The proposed solution uses a low-power, MIMO Frequency Modulated Continuous Wave (FMCW) radar array to capture the reflected signals from human objects, and then constructs 3D human visualization through a serials of data analytics including: 1) a data preprocessing mechanism to remove background static reflection, 2) a signal processing mechanism to transfer received complex radar signals to a matrix containing spatial information, and 3) a deep learning scheme to filter abnormal frames resulted from rough surface of human body. This solution has been extensively evaluated in an indoor research lab. The constructed real-time human images are compared to the camera images captured at the same time. The results show that the proposed radio imaging solution can result in significantly high accuracy.

We consider the problem of expected cost analysis over nondeterministic probabilistic programs, which aims at automated methods for analyzing the resource-usage of such programs. Previous approaches for this problem could only handle nonnegative bounded costs. However, in many scenarios, such as queuing networks or analysis of cryptocurrency protocols, both positive and negative costs are necessary and the costs are unbounded as well.

A novel method is proposed to deal with the problem of building recognition in forward-looking infrared (FLIR) image sequences. Two computational models are deduced in this paper: one is the perspective transformation model, through which an image is transformed perspectively from downward-looking state to forward-looking state; the other is the indirect location model, through which the position of a building is computed in an FLIR image. In addition, in order to illustrate the application scope of our method, the error analysis is presented. The proposed approach is validated by extensive experiments, with images taken in different weather conditions, seasons and at different times. Superior recognition results were obtained on three FLIR image sequences we collected.

In the course of program development for virtual digital storage oscilloscope (DSO), if single thread programming technique is employed, there usually exist shortcomings of slow responding of software panel or control and unstable display of waveform. To solve this problem, based on the analysis of working principles of DSO, this paper introduced multithread technology to realize functions of signals acquisition, parameters setting and signals displaying, and experiments show that better control and display effect can be obtained.