Recent advances in deep learning-based 3D representation have achieved remarkable success, particularly in modeling static high-fidelity geometries. However, the extension of these techniques to dynamic 3D scenes introduces a critical challenge of effectively representing spatio-temporal dependencies, i.e., jointly modeling detailed spatial structures within frames and temporal dynamics across frames. To address this challenge, this paper proposes that the temporal evolution observed in dynamic 3D scenes is fundamentally attributable to the deformation of underlying spatial structures. To capture this relationship, we introduce a unified continuous 4D latent space representation incorporating a structure-equivalence prior, named SEP-4D. The core of SEP-4D is an efficient 4D tensor decomposition-fusion approach. This method fuses decomposed learnable 2D feature planes via a plane-wise spatio-temporal fusion mechanism of planar distributions, explicitly enforcing the principle that temporal evolution originates from geometric deformations of the 3D structure. To mitigate the associated computational demands, we sample the 3D probability volumes generated by VAE-based fusion into a spatio-temporally consistent 4D latent representation. The efficacy of our approach is validated through experiments on the fundamental task of 4D occupancy reconstruction. Extensive results demonstrate that, by leveraging the inherent equivalence of temporal dynamics and structural deformation, our method achieves high-quality reconstruction across various sequence lengths. Notably, for 4-frame scenes, we attain an impressive 91.68% mIoU, significantly outperforming state-of-the-art baselines on standard benchmarks.

The sparse cross-modality detector offers more advantages than its counterpart, the Bird’s-Eye-View (BEV) detector, particularly in terms of adaptability for downstream tasks and computational cost savings. However, existing sparse detectors overlook the quality of token representation, leaving it with a sub-optimal foreground quality and limited performance. In this paper, we identify that the geometric structure preserved and the class distribution are the key to improving the performance of the sparse detector, and propose a Sparse Selector (SS). The core module of SS is Ray-Aware Supervision (RAS), which preserves rich geometric information during the training stage, and Class-Balanced Supervision, which adaptively reweights the salience of class semantics, ensuring that tokens associated with small objects are retained during token sampling. Thereby, outperforming other sparse multi-modal detectors in the representation of tokens. Additionally, we design Ray Positional Encoding (Ray PE) to address the distribution differences between the LiDAR modality and the image. Finally, we integrate the aforementioned module into an end-to-end sparse multi-modality detector, dubbed CrossRay3D. Experiments show that, on the challenging nuScenes benchmark, CrossRay3D achieves state-of-the-art performance with 72.4% mAP and 74.7% NDS, while running$1.84\times $faster than other leading methods. Moreover, CrossRay3D demonstrates strong robustness even in scenarios where LiDAR or camera data are partially or entirely missing. The code is available onhttps://github.com/xuehaipiaoxiang/CrossRay3D

Computer-aided diagnosis (CAD) of pituitary microadenoma (PM) can assist doctors in decision-making, leading to improved lesion detection rates and diagnostic accuracy. However, the performance of existing CAD methods for PM detection has been hindered by the difficulty in obtaining high-quality segmentation results. This is primarily due to the small size of PM lesions and the relatively low resolution of Magnetic Resonance Imaging (MRI) images. To address these challenges, this paper proposes a new medical image detection and segmentation model based on spatio-temporal information. The proposed model aims to addresses the disease classification of PM by designing a network module based on multi-scale feature fusion. This module ensures comprehensive extraction of target semantic information while retaining clear spatial information, achieving classification from dynamic contrast-enhanced MRI(DCE-MRI) to identify positive PM samples. For the lesion segmentation of PM, after ROI Align alignment, the model further adds a semantic segmentation module named Dual-path Semantic Segmentation Module (DSSM) behind the mask head and classification head. This module captures more precise spatio-temporal semantic information, reducing accuracy loss and achieving pituitary segmentation. Finally, leveraging the results of pituitary detection, a feature pyramid network (FPN) layer is redesigned named Reuse Underlying Information Module (RUIM) to reuse low-level information, enhancing the detection capability for PM and thus achieving precise object detection and segmentation. The proposed model achieves an accuracy of 97.10% for PM, mAP of 50.24%, which is superior to multiple representative deep models for medical data. The code is available at https://github.com/BUCT-IUSRC/Research__PM-CAD .

During the process of driving, humans usually rely on multiple senses to gather information and make decisions. Analogously, in order to achieve embodied intelligence in autonomous driving, it is essential to integrate multidimensional sensory information in order to facilitate interaction with the environment. However, the current multi-modal fusion sensing schemes often neglect these additional sensory inputs, hindering the realization of fully autonomous driving. This paper considers multi-sensory information and proposes a multi-modal interactive perception dataset named MIPD, enabling expanding the current autonomous driving algorithm framework, for supporting the research on embodied intelligent driving. In addition to the conventional camera, lidar, and 4D radar data, our dataset incorporates multiple sensor inputs including sound, light intensity, vibration intensity and vehicle speed to enrich the dataset comprehensiveness. Comprising 126 consecutive sequences, many exceeding twenty seconds, MIPD features over 8,500 meticulously synchronized and annotated frames. Moreover, it encompasses many challenging scenarios, covering various road and lighting conditions. The dataset has undergone thorough experimental validation, producing valuable insights for the exploration of next-generation autonomous driving frameworks. Data, development kit and more details will be available athttps://github.com/BUCT-IUSRC/Dataset__MIPD

In recent years, researchers have focused on identifying tasks related to driver state, traffic environment, and others to enhance the safety of autonomous driving assistance systems. However, current research on these tasks is conducted independently, neglecting the interconnections between the driver, traffic environment, and vehicle. In this paper, we propose a Unified Multi-task Assistive Driving Network Based on Multimodal Fusion (UMD-Net), the first unified model capable of recognizing four tasks simultaneously by utilizing multimodal data: driver behavior recognition, driver emotion recognition, traffic context recognition, and vehicle behavior recognition. In order to better enhance the synergistic effects between multiple tasks, we designed the position-sensitive multi-directional attention feature extraction subnetwork and recursive dynamic feature fusion module. The former captures the key features of multi-view images by different directions of attention mechanism to improve the generalization of the model across multiple tasks. The latter dynamically adjusts the fusion weight according to the multimodal features to enhance the representation ability of important features in multi-task learning. Our model was evaluated on the public dataset AIDE, achieving the best performance across all four tasks and a high accuracy of 95.31% in the traffic context recognition task, demonstrating the superiority of our approach. The code is available on https://github.com/Wenzhuo-Liu/UMD-Net.

Traditional convolutional neural networks (CNNs) share their kernels among all positions of the input, which may constrain the representation ability in feature extraction. Dynamic convolution proposes to generate different kernels for different inputs to improve the model capacity. However, the total parameters of the dynamic network can be significantly huge. In this article, we propose a lightweight dynamic convolution method to strengthen traditional CNNs with an affordable increase of total parameters and multiply-adds. Instead of generating the whole kernels directly or combining several static kernels, we choose to "look inside," learning the attention within convolutional kernels. An extra network is used to adjust the weights of kernels for every feature aggregation operation. By combining local and global contexts, the proposed approach can capture the variance among different samples, the variance in different positions of the feature maps, and the variance in different positions inside sliding windows. With a minor increase in the number of model parameters, remarkable improvements in image classification on CIFAR and ImageNet with multiple backbones have been obtained. Experiments on object detection also verify the effectiveness of the proposed method.

Virtual simulated scenes are becoming a critical part of autonomous driving. In the context of knowledge automation and machine learning, simulated images are widely used for visual environmental perception. However, even the most inspirational applications have not fully exploited the potential of simulated images in solving real-world problems. In this article, we propose a novel framework “ParallelEye Pipeline,” which uses image-to-image translation and simulated images to automatically generate realistic synthetic images with multiple ground-truth annotations. Specifically, this method has three steps: first, we use Unity3D software to simulate driving scenarios and generate simulated image pairs (including raw images and six ground-truth labels) from the simulated scenes; second, advanced image-to-image translation algorithms can generate realistic and high-resolution synthetic images from simulated image pairs; third, we exploit publicly datasets, simulated images, and synthetic images to conduct experiments for visual perception. The experimental results suggest: 1) synthetic images and simulated images can improve the performance of detectors in real autonomous driving scenarios and 2) image-to-image translation algorithms can be affected by occlusion condition.

Abstract Instance segmentation aims to locate targets in the image and segment each target at the pixel level, which is one of the most important tasks in computer vision. Mask R‐CNN is a classic method of instance segmentation, but we find that its predicted masks are unclear and inaccurate near contours. To cope with this problem, we draw on the idea of contour matching based on distance transformation image and propose a novel loss function called contour loss. Contour loss is designed to specifically optimise the contour parts of the predicted masks, thus can assure more accurate instance segmentation. To make the proposed contour loss be jointly trained under modern neural network frameworks, we design a differentiable k‐step distance transformation image calculation module, which can approximately compute truncated distance transformation images of the predicted mask and the corresponding ground‐truth mask online. The proposed contour loss can be integrated into existing instance segmentation methods such as Mask R‐CNN, and combined with their original loss functions without modification of the structures of inference network, thus has strong versatility. Experimental results on COCO show that contour loss is effective, which can further improve instance segmentation performances.

Varying density of point clouds increases the difficulty of 3D detection. In this paper, we present a context-aware dynamic network (CADNet) to capture the variance of density by considering both point context and semantic context. Point-level contexts are generated from original point clouds to enlarge the effective receptive filed. They are extracted around the voxelized pillars based on our extended voxelization method and processed with the context encoder in parallel with the pillar features. With a large perception range, we are able to capture the variance of features for potential objects and generate attentive spatial guidance to help adjust the strengths for different regions. In the region proposal network, considering the limited representation ability of traditional convolution where same kernels are shared among different samples and positions, we propose a decomposable dynamic convolutional layer to adapt to the variance of input features by learning from the local semantic context. It adaptively generates the position-dependent coefficients for multiple fixed kernels and combines them to convolve with local features. Based on our dynamic convolution, we design a dual-path convolution block to further improve the representation ability. We conduct experiments on KITTI dataset and the proposed CADNet has achieved superior performance of 3D detection outperforming SECOND and PointPillars by a large margin at the speed of 30 FPS.

In this article, we propose a key secret-sharing technology based on generative adversarial networks (GANs) to address three major problems in the blockchain: 1) low security; 2) hard recovery of lost keys; and 3) low communication efficiency. In our scheme, the proposed network plays the role of a dealer and treats the secret-sharing process as a classification issue. The key idea is to view the secret as an image during the secret-sharing process. If the user's private key is text, we can covert the key text into an image called the original image. Specifically, we first divide the original image into original subimages by the image segmentation. Next, we encode each original subimage by DNA coding. Finally, we train the proposed network to find the key secret-sharing results. Our proposed scheme is not only a significant extension of the GANs but also a new direction for the key secret-sharing technology. The simulation results show that the scheme is secure, and both flexible and efficient in communication.

Object instance segmentation in traffic scenes is an important research topic. For training instance segmentation models, synthetic data can potentially complement real data, alleviating manual effort on annotating real images. However, the data distribution discrepancy between synthetic data and real data hampers the wide applications of synthetic data. In light of that, we propose a virtual-real interaction method for object instance segmentation. This method works over synthetic images with accurate annotations and real images without any labels. The virtual-real interaction guides the model to learn useful information from synthetic data while keeping consistent with real data. We first analyze the data distribution discrepancy from a probabilistic perspective, and divide it into image-level and instance-level discrepancies. Then, we design two components to align these discrepancies, i.e., global-level alignment and local-level alignment. Furthermore, a consistency alignment component is proposed to encourage the consistency between the global-level and the local-level alignment components. We evaluate the proposed approach on the real Cityscapes dataset by adapting from virtual SYNTHIA, Virtual KITTI, and VIPER datasets. The experimental results demonstrate that it achieves significantly better performance than state-of-the-art methods.

We propose Mask SSD, an efficient and effective approach to address the challenging instance segmentation task. Based on a single-shot detector, Mask SSD detects all instances in an image and marks the pixels that belong to each instance. It consists of a detection subnetwork that predicts object categories and bounding box locations, and an instance-level segmentation subnetwork that generates the foreground mask for each instance. In the detection subnetwork, multi-scale and feedback features from different layers are used to better represent objects of various sizes and provide high-level semantic information. Then, we adopt an assistant classification network to guide per-class score prediction, which consists of objectness prior and category likelihood. The instance-level segmentation subnetwork outputs pixel-wise segmentation for each detection while providing the multi-scale and feedback features from different layers as input. These two subnetworks are jointly optimized by a multi-task loss function, which renders Mask SSD direct prediction on detection and segmentation results. We conduct extensive experiments on PASCAL VOC, SBD, and MS COCO datasets to evaluate the performance of Mask SSD. Experimental results verify that as compared with state-of-the-art approaches, our proposed method has a comparable precision with less speed overhead.

Object instance segmentation can achieve preferable results, powered with sufficient labeled training data. However, it is time-consuming for manually labeling, leading to the lack of large-scale diversified datasets with accurate instance segmentation annotations. Exploiting the synthetic data is a very promising solution except for domain distribution mismatch between synthetic dataset and real dataset. In this paper, we propose a synthetic-to-real domain adaptation method for object instance segmentation. At first, this approach is trained to generate object detection and segmentation using annotated data from synthetic dataset. Then, a feature adaptation module (FAM) is applied to reduce data distribution mismatch between synthetic dataset and real dataset. The FAM performs domain adaptation from three different aspects: global-level base feature adaptation module, local-level instance feature adaptation module, and subtle-level mask feature adaptation module. It is implemented based on novel discriminator networks with adversarial learning. The three modules of FAM have positive effects on improving the performance when adapting from synthetic to real scenes. We evaluate the proposed approach on Cityscapes dataset by adapting from Virtual KITTI and SYNTHIA datasets. The results show that it achieves a significantly better performance over the state-of-the-art methods.

Dataset plays an essential role in the training and testing of traffic vision algorithms. However, the collection and annotation of images from the real world is time-consuming, labor-intensive, and error-prone. Therefore, more and more researchers have begun to explore the virtual dataset, to overcome the disadvantages of real datasets. In this paper, we propose a systematic method to construct large-scale artificial scenes and collect a new virtual dataset (named “ParallelEye”) for the traffic vision research. The Unity3D rendering software is used to simulate environmental changes in the artificial scenes and generate ground-truth labels automatically, including semantic/instance segmentation, object bounding boxes, and so on. In addition, we utilize ParallelEye in combination with real datasets to conduct experiments. The experimental results show the inclusion of virtual data helps to enhance the per-class accuracy in object detection and semantic segmentation. Meanwhile, it is also illustrated that the virtual data with controllable imaging conditions can be used to design evaluation experiments flexibly.

Image based eye detection and gaze estimation have a wide range of potential applications, such as medical treatment, biometrics recognition, human-computer interaction. Though a large number of researchers have attempted to solve the two problems, they still exist some challenges due to the variation in appearance and lack of annotated images. In addition, most related work perform eye detection first, followed by gaze estimation via appearance learning. In this paper, we propose a unified framework to execute the gaze estimation and the eye detection simultaneously by learning the cascade regression models from appearance around the eye related key points. Intuitively, there is coupled relationship among location of eye center, shape of eye related key points, appearance representation and gaze information. To incorporate these information, at each cascade level, we first learn a model to map the shape and appearance around current eye related key points to the three dimension gaze update. Then, with the help of estimated gaze, we further learn a regression model to map the gaze, shape and appearance information to eye location update. By leveraging the power of cascade learning, the proposed method can alternatively optimize the two tasks of eye detection and gaze estimation. The experiments are conducted on benchmarks of GI4E and MPIIGaze. Experimental results show that our proposed method can achieve preferable results in gaze estimation and outperform the state-of-the-art methods in eye detection.

Offline training and testing are playing an essential role in design and evaluation of intelligent vehicle vision algorithms. Nevertheless, long-term inconvenience concerning traditional image datasets is that manually collecting and annotating datasets from real scenes lack testing tasks and diverse environmental conditions. For that virtual datasets can make up for these regrets. In this paper, we propose to construct artificial scenes for evaluating the visual intelligence of intelligent vehicles and generate a new virtual dataset called “ParallelEye-CS”. First of all, the actual track map data is used to build 3D scene model of Chinese Flagship Intelligent Vehicle Proving Center Area, Changshu. Then, the computer graphics and virtual reality technologies are utilized to simulate the virtual testing tasks according to the Chinese Intelligent Vehicles Future Challenge (IVFC) tasks. Furthermore, the Unity3D platform is used to generate accurate ground-truth labels and change environmental conditions. As a result, we present a viable implementation method for constructing artificial scenes for traffic vision research. The experimental results show that our method is able to generate photorealistic virtual datasets with diverse testing tasks.

In order to achieve effective protection of digital image information and provide anti-attack capability for encrypted image, this paper proposed an ACP-based Approach to color image encryption using DNA sequence operation and hyper-chaotic system. By using the ACP method which is a way to solve the social computing problem, the influence of the chaotic data from the real world and the influence of the chaotic data from the simulation on the encryption were combined. First, obtaining chaotic data in reality, we made artificial random images by using cloud model; Then, chaotic data in reality were used to encrypt the artificial random image while chaotic data in simulation were used to encrypt the original image; Finally, using the method of parallel execution, combining with the influence of the chaotic data of the two groups, performing DNA-XOR operation on two groups encryption results and we get the final encrypted image. The simulation results show that the algorithm has a good encryption effect and a larger secret key space to the key. In addition, the algorithm can also resist the brute attack and differential attack, and achieve the hyper-chaotic image encryption in the disadvantages of low chaos.

This paper presents a vehicle license plate recognition method based on character-specific extremal regions (ERs) and hybrid discriminative restricted Boltzmann machines (HDRBMs). First, coarse license plate detection (LPD) is performed by top-hat transformation, vertical edge detection, morphological operations, and various validations. Then, character-specific ERs are extracted as character regions in license plate candidates. Followed by suitable selection of ERs, the segmentation of characters and coarse-to-fine LPD are achieved simultaneously. Finally, an offline trained pattern classifier of HDRBM is applied to recognize the characters. The proposed method is robust to illumination changes and weather conditions during 24 h or one day. Experimental results on thorough data sets are reported to demonstrate the effectiveness of the proposed approach in complex traffic environments.

This paper proposes a novel approach to visual tracking of moving objects based on the dynamic coupled conditional random field (DcCRF) model. The principal idea is to integrate a variety of relevant knowledge about object tracking into a unified dynamic probabilistic framework, which is called the DcCRF model in this paper. Under this framework, the proposed approach integrates spatiotemporal contextual information of motion and appearance, as well as the compatibility between the foreground label and object label. An approximate inference algorithm, i.e., loopy belief propagation, is adopted to conduct the inference. Meanwhile, the background model is adaptively updated to deal with gradual background changes. Experimental results show that the proposed approach can accurately track moving objects (with or without occlusions) in monocular video sequences and outperforms some state-of-the-art methods in tracking and segmentation accuracy.

This paper proposes a novel video-based vehicle detection approach with data-driven adaptive neuro-fuzzy networks. The key ideas include configuring several virtual loops as vehicle detection zones in the image, assuming moving vehicles will cause pixel intensities and local textures to change, and then identifying such changes to detect vehicles. In this work, vehicle detection is treated as a pattern classification problem. First, 14 image features (regarding foreground area, texture change, and environmental condition) are extracted to represent the distinction between vehicle and nonvehicle patterns. Then, three neuro-fuzzy networks are trained via incremental semi-supervised learning to build a data-driven adaptive classifier, which judges whether a vehicle is located in the virtual loop. The semi-supervised learning procedure is performed based on a modified tri-training approach, to automatically optimize the structures and parameters of the component neuro-fuzzy networks. Experimental results illustrate that the proposed approach is accurate and robust to detect vehicles in complex environments (e.g. adverse illumination and weather conditions), and thus can improve the performance of video-based vehicle detection.

Field data are important for convenient daily travel of urban residents, reducing traffic congestion and accidents, pursuing a low-carbon environment-friendly sustainable development strategy, and meeting the extra peak traffic demand of large sporting events or large business activities, etc. To meet the field data demand during the 2010 Asian (Para) Games held in Guangzhou, China, based on the novel Artificial systems, Computational experiments, and Parallel execution (ACP) approach, the Parallel Traffic Management System (PtMS) was developed. It successfully helps to achieve smoothness, safety, efficiency, and reliability of public transport management during the two games, supports public traffic management and decision making, and helps enhance the public traffic management level from experience-based policy formulation and manual implementation to scientific computing-based policy formulation and implementation. The PtMS represents another new milestone in solving the management difficulty of real-world complex systems.

For the last two decades, intelligent transportation systems (ITS) have emerged as an efficient way of improving the performance of transportation systems, enhancing travel security, and providing more choices to travelers. A significant change in ITS in recent years is that much more data are collected from a variety of sources and can be processed into various forms for different stakeholders. The availability of a large amount of data can potentially lead to a revolution in ITS development, changing an ITS from a conventional technology-driven system into a more powerful multifunctional data-driven intelligent transportation system (D2ITS) : a system that is vision, multisource, and learning algorithm driven to optimize its performance. Furthermore, D2ITS is trending to become a privacy-aware people-centric more intelligent system. In this paper, we provide a survey on the development of D2ITS, discussing the functionality of its key components and some deployment issues associated with D2ITS Future research directions for the development of D2ITS is also presented.

In this paper, a feedback algorithm based on the constant bandwidth server (CBS) is designed to support and meet the quality of service requirements of soft real-time tasks in vehicular application specific embedded operating systems (vASOS). Moreover, it realizes the temporal isolation of hard and soft real-time tasks and guarantees the reliability and safety of vehicles in vASOS. A proportional integrative derivative (PID) controller is applied to control the fraction of CPU bandwidth allocated to these tasks, and a precise mathematical model is provided. Finally, the system stability is analyzed and its effectiveness of our method is verified by simulation.