Automated waterway environment perception is crucial for enabling unmanned surface vessels (USVs) to understand their surroundings and make informed decisions. Most existing waterway perception models primarily focus on instance-level object perception paradigms (e.g., detection, segmentation). However, due to the complexity of waterway environments, current perception datasets and models fail to achieve global semantic understanding of waterways, limiting large-scale monitoring and structured log generation. With the advancement of vision-language models (VLMs), we leverage image captioning to introduce WaterCaption, the first captioning dataset specifically designed for waterway environments. WaterCaption focuses on fine-grained, multi-region long-text descriptions, providing a new research direction for visual geo-understanding and spatial scene cognition. Exactly, it includes 20.2k image-text pair data with 1.8 million vocabulary size. Additionally, we propose Da Yu, an edge-deployable multi-modal large language model for USVs, where we propose a novel vision-to-language projector called Nano Transformer Adaptor (NTA). NTA effectively balances computational efficiency with the capacity for both global and fine-grained local modeling of visual features, thereby significantly enhancing the model’s ability to generate long-form textual outputs. Da Yu achieves an optimal balance between performance and efficiency, surpassing state-of-the-art models on WaterCaption and several other captioning benchmarks. The project is available at https://github.com/GuanRunwei/WaterCaption.

Ensuring maritime safety and optimizing traffic management in increasingly crowded and complex waterways require effective waterway monitoring. However, current methods struggle with challenges arising from multimodal data, such as dimensional disparities, mismatched target counts, vessel scale variations, occlusions, and asynchronous data streams from systems like the automatic identification system (AIS) and closed-circuit television (CCTV). Traditional multi-vessel association methods often struggle with these complexities, particularly in densely trafficked waterways. To overcome these issues, we propose a graph learning-driven multi-vessel association (named GMvA) method tailored for maritime multimodal data fusion. By integrating AIS and CCTV data, GMvA leverages time series learning and graph neural networks to capture the spatiotemporal features of vessel trajectories effectively. To enhance feature representation, the proposed method incorporates temporal graph attention and spatiotemporal attention, effectively capturing both local and global vessel interactions. Furthermore, a multi-layer perceptron-based uncertainty fusion module computes robust similarity scores, and the Hungarian algorithm is adopted to ensure globally consistent and accurate target matching. To validate the efficacy of our method, we have also constructed a new maritime multimodal dataset (termed MaritimeMmD) for data fusion. Extensive experiments demonstrate that GMvA delivers superior accuracy and robustness in multi-vessel association, outperforming existing methods even in challenging scenarios with high vessel density and incomplete or unevenly distributed AIS and CCTV data. The source dataset and code are available athttps://github.com/LouisYxLu/GMvA

Pedestrian crossing intention prediction is crucial for ensuring safety in intelligent transportation systems, especially in autonomous driving scenarios. Most existing methods rely primarily on visual information; however, the quality of visual data deteriorates significantly at long distances due to limited resolution. Although multi-modal approaches can mitigate this issue by incorporating additional sensory data, they inevitably introduce extra computational overhead. To address these challenges, we propose a lightweight cascaded model for pedestrian crossing intention prediction based on text-trajectory alignment. The model employs a cascaded architecture that jointly performs coordinate and intention prediction, while leveraging a pre-trained large language model (LLM) to generate textual descriptions of videos, thereby enriching trajectory features. Furthermore, a center-aware classification module is integrated to enhance inter-class separability and intra-class compactness. Extensive experiments onJAADandPIEdemonstrate state-of-the-art performance: our method achieves 91% accuracy onPIEand 89% onJAAD, matching or surpassing recent multi-modal approaches with substantially fewer inputs. The source code will be released athttps://github.com/xyhhappy/TCP-prediction

In cross-view image geolocation tasks, traditional CNN models are limited in performance due to their inability to effectively capture global correlations. Although Transformer-based methods can address this shortcoming, their computational complexity and GPU memory consumption are relatively high. To address these limitations, we propose Swin Transformer for Cross-view Geo-localization (STG), a novel model based on the Swin Transformer (ST) that leverages its advantage of linear computational complexity scaling with image resolution. We design an adaptive window shift mechanism and introduce a pre-training strategy for STG, which can enhance STGs global modeling capability and its ability to learn general features. To further address the redundancy issues in our STG model, we propose the Distillation Swin Transformer for Cross-view Geo-localization model (DSTG), which is obtained through knowledge distillation using the STG model as the teacher model. To tackle the issue that the student model strugglings to learn from the teacher model’s output during distillation directly, we propose a multi-scale logit standardization distillation method. The proposed STG and DSTG models can obtain superior global embedding descriptions without relying on polar coordinate transformations. Experimental results on the CVUSA, CVACT_val, and CVACT_test datasets indicate that the DSTG model has significantly lower computational cost and GPU memory usage compared to the STG model. At the same time, both models achieve state-of-the-art performance. The source code is available at https://github.com/liangjxiong/DSTG.

In recent years, global climate change has intensified, leading to frequent occurrences of short-term extreme rain-fall events. How to accurately predict short-term precipitation within the next few hours has become a concern for people, especially for meteorologists. Traditional Numerical Weather Prediction (NWP) models rely on solving complex physical equations, resulting in significant computational costs and low temporal resolution, which limit their suitability for nowcasting tasks. Weather radar echo extrapolation, which forecasts future echoes from historical observations, has thus become the predominant approach for precipitation nowcasting. In the realm of deep learning-based methods, convolutional or recurrent neural network-based extrapolation pipelines inherently struggle in processing sequential data and capturing the multi-scale spatial features inherent in radar echo images. Additionally, current models often focus on enhancing dense prediction capabilities while neglecting to mine or utilize the evolutionary patterns of historical echoes, leading to diminished accuracy in long-term sequence forecasting. Moreover, high precipitation areas are critical for public safety, but current models often fail to forecast them accurately due to insufficient emphasis during learning. In this paper, we propose a memory-aware multi-scale predictive transformer (MMPFormer) for precipitation nowcasting. Specifically, we designed the Multi-Scale Spatial-Temporal Transformer (MS-STT), which incorporates multi-scale convolution techniques alongside self-attention mechanisms to effectively extract and model the spatiotemporal correlations within historical radar echoes. Additionally, we have designed a ConvGRU-Instructed Memory Mechanism (CIMM) to alleviate the degradation of radar echo details during extended extrapolation periods, enabling accurate forecasts over the next three hours. Furthermore, the Key Area Attention Module (KAAM), a plug-and-play module, has been introduced. It emphasizes high precipitation areas through attention mechanisms, mitigating the negative impacts of missing high precipitation by previous learning-based methods. Quantitative and qualitative experimental results on radar echo datasets demonstrate the superior performance of our method in modeling spatiotemporal dynamics and long-term extrapolation for precipitation forecasting.

Maritime surveillance plays a vital role in reducing maritime accidents and improving maritime safety. To enhance situational awareness for maritime movements, deep learning-based visual object detection has become an important part of maritime surveillance. However, the detection results are highly dependent on the training datasets collected from different departments (i.e., clients). If the sub-datasets from departments are sensitive and private in cross-department maritime surveillance, it will be intractable to directly combine these sub-datasets to train the learning-based object detection method. To solve this issue, we propose a federated learning-driven cross-spatial vessel detection model, called FLCSDet, for maritime surveillance with privacy preservation. In particular, an efficient multi-scale attention module is integrated into our FLCSDet to achieve local cross-spatial feature learning. To improve the federated-learning aggregation method, we propose an optimized algorithm based on the proportion of valid data on departments to adaptively select the allocating weights and preserve the specific characteristics of client data. In addition, we employ transfer learning to further improve the robustness and convergence of our FLCSDet under different experimental scenarios. Compared with several representative federated learning-based detection methods, our FLCSDet could achieve superior detection performance in terms of both quantitative and qualitative results. Moreover, comprehensive experiments conducted on real datasets from both inland waterways and open seas demonstrate the robustness and generalization of our method in intelligent transportation systems. The source code is available athttps://github.com/huangyanh/FLCSDet.

Vehicle trajectory prediction (VTP) is important for ensuring safe decision-making and planning in Internet of Vehicles (IoV). In complex traffic scenarios, accurate and reliable trajectory prediction requires comprehensive understanding of the interaction behaviors among vehicles. However, existing methods fail to effectively capture vehicle interaction features and fully explore their potential dependencies, limiting improvements in prediction accuracy. To this end, we propose a social attention network fused multipatch temporal–variable dependency (SAN-FTVD) model to tackle the above problems. In specific, we first design a variable token embedding module (VTEM) to extract the motion state information of vehicles, which independently embeds each variable of vehicle historical data into a variable token. After that, we propose a physical informed vehicle interaction encoder (PI-VIE) to capture vehicle interaction features over continuous time. The encoder is combined with physical priors to encode vehicle interaction features based on the correlations between the variable tokens. Following that, a temporal–variable dependency fusion module (TVDFM) is proposed to extract and fuse the multipatch temporal and variable dependencies, fully exploring potential dependencies in vehicle interaction features. Numerical results demonstrate that compared with the state-of-the-art model, the proposed model reduces the average prediction root mean square error over 5-s time range by 8% and 7% on two public data sets with 75% less inference cost. Furthermore, extensive ablation experiments validate the effectiveness of the above modules in the model.

Recently, low-rank tensor completion (LRTC) methods by exploiting the global low-rankness of the target tensor have shown great potential for traffic data imputation. However, in real-world transportation networks, traffic data usually suffer from more complicated structural missing patterns than random-missing patterns, e.g., tube-missing patterns due to disruptions in wireless connections or slice-missing mechanism caused by sensor maintenance. As the naturally low-rank structure of traffic data in several missing scenarios, the existing LRTC methods indeed refrain from desirable performance for imputing traffic data. To tackle the complicated missing scenarios, we propose a convolutional low-rank tensor representation (CLRTR). Especially, CLRTR represents each unfolding matrix of the tensor as a sum of convolutions between two-dimensional (2D) filters and the corresponding low-rank coefficients, which allows us to simultaneously reveal the local patterns and the low-rankness of traffic data. Based on the CLRTR, we introduce the corresponding low-rank metric CLRTR-rank. Based on the suggested low-rank metric, we propose a traffic data imputation model that is well-suited to the complicated missing data scenarios. To implement the resultant imputation model, we design the alternating direction method of multipliers (ADMM) based algorithm with a theoretical convergence guarantee. Extensive numerical experiments on several real-world traffic datasets for both traffic data imputation and downstream traffic data prediction highlight the superiority of our model over the existing state-of-the-art matrix/tensor models for extensive missing scenarios.

The visible-light camera, which is capable of environment perception and navigation assistance, has emerged as an essential imaging sensor for marine surface vessels in intelligent waterborne transportation systems (IWTS). However, the visual imaging quality inevitably suffers from several kinds of degradations (e.g., limited visibility, low contrast, color distortion, etc.) under complex weather conditions (e.g., haze, rain, and low-lightness). The degraded visual information will accordingly result in inaccurate environment perception and delayed operations for navigational risk. To promote the navigational safety of vessels, many computational methods have been presented to perform visual quality enhancement under poor weather conditions. However, most of these methods are essentially specific-purpose implementation strategies, only available for one specific weather type. To overcome this limitation, we propose to develop a general-purpose multi-scene visibility enhancement method, i.e., edge reparameterization- and attention-guided neural network (ERANet), to adaptively restore the degraded images captured under different weather conditions. In particular, our ERANet simultaneously exploits the channel attention, spatial attention, and reparameterization technology to enhance the visual quality while maintaining low computational cost. Extensive experiments conducted on standard and IWTS-related datasets have demonstrated that our ERANet could outperform several representative visibility enhancement methods in terms of both imaging quality and computational efficiency. The superior performance of IWTS-related object detection and scene segmentation could also be steadily obtained after ERANet-based visibility enhancement under complex weather conditions.

Real-time transportation surveillance is an essential part of the intelligent transportation system (ITS). However, images captured under low-light conditions often suffer poor visibility with types of degradation, such as noise interference and vague edge features, etc. With the development of imaging devices, the quality of the visual surveillance data is continually increasing, like 2K and 4K, which have more strict requirements on the efficiency of image processing. To satisfy the requirements on both enhancement quality and computational speed, this paper proposes a double domain guided real-time low-light image enhancement network (DDNet) for ultra-high-definition (UHD) transportation surveillance. Specifically, we design an encoder-decoder structure as the main architecture of the learning network. In particular, the enhancement processing is divided into two subtasks (i.e., color enhancement and gradient enhancement) via the proposed coarse enhancement module (CEM) and LoG-based gradient enhancement module (GEM), which are embedded in the encoder-decoder structure. It enables the network to enhance the color and edge features simultaneously. Through the decomposition and reconstruction on both color and gradient domains, our DDNet can restore the detailed feature information concealed by the darkness with better visual quality and efficiency. The evaluation experiments on standard and transportation-related datasets demonstrate that our DDNet provides superior enhancement quality and efficiency compared with state-of-the-art methods. Besides, the object detection and scene segmentation experiments indicate the practical benefits for higher-level image analysis under low-light environments in ITS. The source code is available at https://github.com/QuJX/DDNet.

The accurate and efficient vessel draft reading (VDR) is an important component of intelligent maritime surveillance, which could be exploited to assist in judging whether the vessel is normally loaded or overloaded. The computer vision technique with an excellent price-to-performance ratio has become a popular medium to estimate vessel draft depth. However, the traditional estimation methods easily suffer from several limitations, such as sensitivity to low-quality images, high computational cost, etc. In this work, we propose a multi-task learning-enabled computational method (termed MTL-VDR) for generating highly reliable VDR. In particular, our MTL-VDR mainly consists of four components, i.e., draft mark detection, draft scale recognition, vessel/water segmentation, and final draft depth estimation. We first construct a benchmark dataset related to draft mark detection and employ a powerful and efficient convolutional neural network to accurately perform the detection task. The multi-task learning method is then proposed for simultaneous draft scale recognition and vessel/water segmentation. To obtain more robust VDR under complex conditions (e.g., damaged and stained scales, etc.), the accurate draft scales are generated by an automatic correction method, which is presented based on the spatial distribution rules of draft scales. Finally, an adaptive computational method is exploited to yield an accurate and robust draft depth. Extensive experiments have been implemented on the realistic dataset to compare our MTL-VDR with state-of-the-art methods. The results have demonstrated its superior performance in terms of accuracy, robustness, and efficiency. The computational speed exceeds 40 FPS, which satisfies the requirements of real-time maritime surveillance to guarantee vessel traffic safety.

Autonomous driving on water surfaces plays an essential role in executing hazardous and time-consuming missions, such as maritime surveillance, survivor rescue, environmental monitoring, hydrography mapping and waste cleaning. This work presents WaterScenes, the first multi-task 4D radar-camera fusion dataset for autonomous driving on water surfaces. Equipped with a 4D radar and a monocular camera, our Unmanned Surface Vehicle (USV) proffers all-weather solutions for discerning object-related information, including color, shape, texture, range, velocity, azimuth, and elevation. Focusing on typical static and dynamic objects on water surfaces, we label the camera images and radar point clouds at pixel-level and point-level, respectively. In addition to basic perception tasks, such as object detection, instance segmentation and semantic segmentation, we also provide annotations for free-space segmentation and waterline segmentation. Leveraging the multi-task and multi-modal data, we conduct benchmark experiments on the uni-modality of radar and camera, as well as the fused modalities. Experimental results demonstrate that 4D radar-camera fusion can considerably improve the accuracy and robustness of perception on water surfaces, especially in adverse lighting and weather conditions. WaterScenes dataset is public onhttps://waterscenes.github.io.

As edge computing faces increasingly severe data security and privacy issues of edge devices, a framework called federated edge learning (FEL) has recently been proposed to enable machine learning (ML) model training at the edge, ensuring communication efficiency and data privacy protection for edge devices. In this paradigm, the training efficiency has long been challenged by the heterogeneity of communication conditions, computing capabilities, and available data sets at devices. Currently, researchers focus on solving this challenge via device selection from the perspective of optimizing energy consumption or convergence speed. However, the consideration of any one of them is insufficient to guarantee the long-term system efficiency and stability. To fill the gap, we propose an optimization problem to simultaneously minimize the total energy consumption of selected devices and maximize the convergence speed of the global model for device selection in FEL, under the constraints of training data amount and time consumption. For the accurate calculation of energy consumption, we deploy online bandit learning to estimate the CPU-cycle frequency availability of each device, based on an efficient algorithm, named fast-convergent energy-efficient device selection (FCE2DS), is proposed to solve the optimization problem with a low level of time complexity. Through a series of comparative experiments, we evaluate the performance of the proposed FCE2DS scheme, verifying its high training accuracy and energy efficiency.

Scene recovery is a fundamental imaging task with several practical applications, including video surveillance and autonomous vehicles, etc. In this article, we provide a new real-time scene recovery framework to restore degraded images under different weather/imaging conditions, such as underwater, sand dust and haze. A degraded image can actually be seen as a superimposition of a clear image with the same color imaging environment (underwater, sand or haze, etc.). Mathematically, we can introduce a rank-one matrix to characterize this phenomenon, i.e., rank-one prior (ROP). Using the prior, a direct method with the complexity$O(N)$is derived for real-time recovery. For general cases, we develop ROP$^+$to further improve the recovery performance. Comprehensive experiments of the scene recovery illustrate that our method outperforms competitively several state-of-the-art imaging methods in terms of efficiency and robustness.

The Internet of Things (IoT) has recently emerged as a revolutionary communication paradigm where a large number of objects and devices are closely interconnected to enable smart industrial environments. The tremendous growth of visual sensors can significantly promote the traffic situational awareness, traffic safety management, and intelligent vehicle navigation in intelligent transportation systems (ITSs). However, due to the absorption and scattering of light by the turbid medium in atmosphere, the visual IoT inevitably suffers from imaging quality degradation, e.g., contrast reduction, color distortion, etc. This negative impact can not only reduce the imaging quality, but also bring challenges for the deployment of several high-level vision tasks (e.g., object detection, tracking, recognition, etc.) in the ITS. To improve imaging quality under the hazy environment, we propose a deep network-enabled three-stage dehazing network (termed TSDNet) for promoting the visual IoT-driven ITS. In particular, the proposed TSDNet mainly contains three parts, i.e., multiscale attention module for estimating the hazy distribution in the RGB image domain, two-branch extraction module for learning the hazy features, and multifeature fusion module for integrating all characteristic information and reconstructing the haze-free image. Numerous experiments have been implemented on synthetic and real-world imaging scenarios. Dehazing results illustrated that our TSDNet remarkably outperformed several state-of-the-art methods in terms of both qualitative and quantitative evaluations. The high-accuracy object detection results have also demonstrated the superior dehazing performance of the TSDNet under hazy atmosphere conditions. The source code is available athttps://github.com/gy65896/TSDNet.

The speckle noise is a granular disturbance that often brings negative side effects on the detection and recognition of targets of interest in industrial imaging sensors. From the statistical point of view, this type of noise can be modeled as a multiplicative formula. The nonlinear multiplicative property makes despeckling more intractable with respect to noise reduction and details preservation. To blindly remove the undesirable speckle noise, we combine the gradient model and machine learning technology for despeckling. In particular, we first introduce the logarithmic transformation to transform the multiplicative speckle noise into an additive version. A gradient-guided despeckling transformer (termed GradDT) is then proposed to blindly reduce the additive noise in the transformed noisy images. To be specific, the proposed method mainly includes two modules, i.e., the spatial feature extraction module (SFEM) and the efficient transformer module (ETM). The SFEM can extract the spatial feature of speckle noise and the gradient maps corresponding to the noise-free image. The ETM module can calculate the spatial domain's cross-channel cross-covariance and produce global attention maps to reconstruct the sharp image. The proposed GradDT thus can effectively distinguish the speckle noise and vital image features (e.g., edge and texture) to balance the degree of noise suppression and details preservation. Extensive experiments have been implemented on both synthetic and realistic degraded images. Compared with several state-of-the-art speckle noise reduction methods, our GradDT could generate superior imaging performance in terms of both quantitative evaluation and visual quality.

The automatic identification system (AIS) and video cameras have been widely exploited for vessel traffic surveillance in inland waterways. The AIS data could provide vessel identity and dynamic information on vessel position and movements. In contrast, the video data could describe the visual appearances of moving vessels without knowing the information on identity, position, movements, etc. To further improve vessel traffic surveillance, it becomes necessary to fuse the AIS and video data to simultaneously capture the visual features, identity, and dynamic information for the vessels of interest. However, the performance of AIS and video data fusion is susceptible to issues such as data spatial difference, message asynchronous transmission, visual object occlusion, etc. In this work, we propose a deep learning-based simple online and real-time vessel data fusion method (termed DeepSORVF). We first extract the AIS-and video-based vessel trajectories, and then propose an asynchronous trajectory matching method to fuse the AIS-based vessel information with the corresponding visual targets. In addition, by combining the AIS-and video-based movement features, we also present a prior knowledge-driven anti-occlusion method to yield accurate and robust vessel tracking results under occlusion conditions. To validate the efficacy of our DeepSORVF, we have also constructed a new benchmark dataset (termed FVessel) for vessel detection, tracking, and data fusion. It consists of many videos and the corresponding AIS data collected in various weather conditions and locations. The experimental results have demonstrated that our method is capable of guaranteeing high-reliable data fusion and anti-occlusion vessel tracking. The DeepSORVF code and FVessel dataset are publicly available at https://github.com/gy65896/DeepSORVF and https://github.com/gy65896/FVessel, respectively.

Pedestrian trajectory prediction in multiple scenarios is of immense importance in autonomous driving and disentanglement of human behavior but is limited in catching human intention and initiative. Most previous works tend to predict the trajectory using only 2D coordinates, which generally cause two common problems: a) Overlooking the subjective initiative, including sudden swerve and erratic movement; b) A potential challenge called abnormal collision caused by unlabeled pedestrians on dataset is not being identified and resolved, which would ruin the model prediction. To break those limitations, we introduce visual localization and orientation as Visual Intention Knowledge to help the trajectory prediction, which is learned directly from visual scenarios. It benefits to comprehend human intention and formulates decision-making processes. Moreover, by learning from the visual information and decision-making policy, we construct the Visual Intention Knowledge associated spatio-temporal Transformer (VIKT) to predict human trajectory by combining the intention knowledge with the novel Transformer. Extensive experimental results demonstrate that our VIKT model could achieve competitive performance by the Visual Intention Knowledge through optimizing the model prediction compared with state-of-the-art methods in terms of prediction accuracy on ETH/UCY and SDD benchmarks.

Vision-based maritime surveillance has become an essential part of the vessel traffic services system. The images collected in low-light maritime conditions often suffer from poor visibility. These images may significantly degenerate the performance of high-level visual tasks and increase the uncertainty in maritime surveillance. To address this problem, we propose a lightweight neural network (Rep-Enhancer) for low-light image enhancement. Specifically, we first design a re-parameterizable multi-branch edge extraction module, i.e., spatial domain-oriented convolution block (SDCB). Furthermore, skip connections and spatial attention operations are employed to strengthen the features. By exploiting these well-strengthened edge features, we can enhance the low-light images effectively with the encoder-decoder structure. The experimental results have shown that Rep-Enhancer can enhance the low-light image qualifiedly while maintaining great inference efficiency.

The learning-based low-light image enhancement methods have remarkable performance due to the robust feature learning and mapping capabilities. This paper proposes a multi-branch topology residual block (MTRB)-based network (MTRBNet), which can alleviate training difficulties and more efficiently use the parameters between neurons. Compared with the previous residual block, the proposed MTRB increases the width of the network and simultaneously transmits information along with the depth and width directions, which can effectively select network nodes to promote the network learning capacity. Meanwhile, the feature information of neighbor nodes is transferred to each other, thereby maximizing the information flow of the convolution unit. The proposed information connection and feedback mechanism can improve the network’s ability to capture the global and local features. We analyze the pros and cons of two multi-feature fusion strategies (i.e., addition and concatenation) and three normalization methods on the quantitative results. In addition, we embed our MTRB into traditional Encoder-Decoder structure to improve the image enhancement results under different low-light imaging conditions. Experiments on the LOL image dataset have demonstrated that our MTRBNet achieves superior performance compared with several state-of-the-art methods.

The revolutionary advances in machine learning and data mining techniques have contributed greatly to the rapid developments of maritime Internet of Things (IoT). In maritime IoT, the spatio-temporal vessel trajectories, collected from the hybrid satellite-terrestrial automatic identification system (AIS) base stations, are of considerable importance for promoting traffic situation awareness and vessel traffic services, etc. To guarantee traffic safety and efficiency, it is essential to robustly and accurately predict the AIS-based vessel trajectories (i.e., the future positions of vessels) in maritime IoT. In this work, we propose a spatio-temporal multigraph convolutional network (STMGCN)-based trajectory prediction framework using the mobile edge computing (MEC) paradigm. Our STMGCN is mainly composed of three different graphs, which are, respectively, reconstructed according to the social force, the time to closest point of approach, and the size of surrounding vessels. These three graphs are then jointly embedded into the prediction framework by introducing the spatio-temporal multigraph convolutional layer. To further enhance the prediction performance, the self-attention temporal convolutional layer is proposed to further optimize STMGCN with fewer parameters. Owing to the high interpretability and powerful learning ability, STMGCN is able to achieve superior prediction performance in terms of both accuracy and robustness. The reliable prediction results are potentially beneficial for traffic safety management and intelligent vehicle navigation in MEC-enabled maritime IoT.

With the rapid development of cloud manufacturing, industrial production with edge computing as the core architecture has been greatly developed. However, edge devices often suffer from abnormalities and failures in industrial production. Therefore, detecting these abnormal situations timely and accurately is crucial for cloud manufacturing. As such, a straightforward solution is that the edge device uploads the data to the cloud for anomaly detection. However, Industry 4.0 puts forward higher requirements for data privacy and security so that it is unrealistic to upload data from edge devices directly to the cloud. Considering the abovementioned severe challenges, this article customizes a weakly supervised edge computing anomaly detection framework, i.e., federated learning-based transformer framework (FedAnomaly), to deal with the anomaly detection problem in cloud manufacturing. Specifically, we introduce federated learning (FL) framework that allows edge devices to train an anomaly detection model in collaboration with the cloud without compromising privacy. To boost the privacy performance of the framework, we add differential privacy noise to the uploaded features. To further improve the ability of edge devices to extract abnormal features, we use the transformer to extract the feature representation of abnormal data. In this context, we design a novel collaborative learning protocol to promote efficient collaboration between FL and transformer. Furthermore, extensive case studies on four benchmark datasets verify the effectiveness of the proposed framework. To the best of our knowledge, this is the first time integrating FL and transformer to deal with anomaly detection problems in cloud manufacturing.

The accurate and robust prediction of vessel traffic flow is gaining importance in maritime intelligent transportation system (ITS), such as vessel traffic services, maritime spatial planning, and traffic safety management, etc. To achieve fine-grained vessel traffic flow prediction, we will first generate the maritime traffic network (which is essentially a graph), and then propose a graph-driven neural network. In particular, to represent various correlations among spatio-temporal vessel traffic flow, we tend to extract the feature points (i.e., starting, way and ending points) by utilizing the knowledge of vessel positioning data. These feature points are essentially related to the geometrical structures of massive vessel trajectories collected from massive automatic identification system (AIS) records, contributing to the generation of maritime traffic network. We then propose a spatio-temporal multi-graph convolutional network (STMGCN)-based vessel traffic flow prediction method by exploiting multiple types of inherent correlations in the generated maritime graph. The proposed STMGCN mainly contains one spatial multi-graph convolutional layer and two temporal gated convolutional layers, beneficial for extracting spatial and temporal traffic flow patterns. The main benefit of our graph-driven prediction method is that it takes full advantage of the maritime graph and multi-graph learning. Comprehensive experiments have been implemented on realistic AIS dataset to compare our method with several state-of-the-art prediction methods. The fine-grained prediction results have demonstrated our superior performance in terms of both accuracy and robustness.

The predictive vessel surveillance is one of the indispensable functional components in intelligent maritime traffic system. Vessel trajectory prediction serves as a prerequisite for collision detection and risk assessment. Perceiving the forthcoming traffic situation in advance helps decide the succeeding actions to mitigate the potential risk. The availability of maritime big data brings great potential to extract vessel movement patterns to support trajectory forecasting. In this paper, a novel vessel trajectory and navigating state prediction methodology is proposed based on AIS data, which synergizes properly designed learning, motion modelling and knowledge base assisted particle filtering processes. The primary contributions of this work also comprise several critical research findings to handle the key challenges in vessel trajectory and navigating state prediction problem, such as the adaptive training window determination for the learning process and effective knowledge storage and searching algorithm intended to reduce the query time of waterway pattern retrieval. The studies for these challenges are still missing in the reported literatures but they are essentially important for improving the prediction accuracy, efficiency and practicality. With the maritime traffic data collected for Singapore water, a thorough evaluation of the prediction performance has been conducted for different navigating scenarios. It is also observed that better prediction outperforms on account of allowing earlier alert in risk detection.

Scene recovery is a fundamental imaging task for several practical applications, e.g., video surveillance and autonomous vehicles, etc. To improve visual quality under different weather/imaging conditions, we propose a real-time light correction method to recover the degraded scenes in the cases of sandstorms, underwater, and haze. The heart of our work is that we propose an intensity projection strategy to estimate the transmission. This strategy is motivated by a straightforward rank-one transmission prior. The complexity of transmission estimation is O(N ) where N is the size of the single image. Then we can recover the scene in real-time. Comprehensive experiments on different types of weather/imaging conditions illustrate that our method outperforms competitively several state-of-the-art imaging methods in terms of efficiency and robustness.

Future generation communication systems, such as 5G and 6G wireless systems, exploit the combined satellite-terrestrial communication infrastructures to extend network coverage and data throughput for data-driven applications. These ground-breaking techniques have promoted the rapid development of Internet of Things (IoT) in maritime industries. In maritime IoT applications, intelligent vessel traffic services can be guaranteed by collecting and analyzing high volume of spatial data flows from automatic identification system (AIS). This AIS system includes a highly integrated automatic equipment, including functionalities of core communication, tracking, and sensing. The increased utilization of shipboard AIS devices allows the collection of massive trajectory data. However, the received raw AIS data often suffers from undesirable outliers (i.e., poorly tracked timestamped points for vessel trajectories) during signal acquisition and analog-to-digital conversion. The degraded AIS data will bring negative effects on vessel traffic services (e.g., maritime traffic monitoring, intelligent maritime navigation, vessel collision avoidance, etc.) in maritime IoT scenarios. To improve the quality of vessel trajectory records from AIS networks, we propose to develop a two-phase data-driven machine learning framework for vessel trajectory reconstruction. In particular, a density-based clustering method is introduced in the first phase to automatically recognize the undesirable outliers. The second phase proposes a bidirectional long short-term memory (BLSTM)-based supervised learning technique to restore the timestamped points degraded by random outliers in vessel trajectories. Comprehensive experiments on simulated and realistic data sets have verified the dominance of our two-phase vessel reconstruction framework compared to other competing methods. It thus has the capacity of promoting intelligent vessel traffic services in 6G-enabled maritime IoT systems.

The communication in a 6G-enabled network in a box (NIB) needs to meet the characteristics of fast, convenient, and safe. Secret sharing scheme has become a hot topic nowadays due to unconditional security and simple decryption. At the same time, a quick response (QR) code as a popular carrier has been widely applied in various industrial applications because of the data payload and convenience. Thus, the combination of secret sharing and QR code provides a solution by satisfying the requirements of 6G-enabled NIB. In this article, we design a QR code secret sharing scheme with authentication to protect private data and prevent cheater. In this scheme, a secret image is first divided into a series of shadows based on a polynomial, and authentication bits are generated based on the generated shadows. Then shadows and the authentication bits are embedded into the cover QR codes according to the error correction redundancy and the homomorphism of the Reed-Solomon code in the QR code. In addition, the secret can be restored with the qualified shares and the authentication bits could verify the authenticity of the embedded shadows. Compared with existing schemes, the proposed scheme not only guarantees a high capacity but also embeds more authentication bits to improve the authentication ability. In addition, experimental results have demonstrated that the proposed scheme is both robust and secure.

Vehicle taillight detection is essential to analyze and predict driver intention in collision avoidance systems. In this article, we propose an end-to-end framework that locates the rear brake and turn signals from video stream in real-time. The system adopts the fast YOLOv3-tiny as the backbone model and three improvements have been made to increase the detection accuracy on taillight semantics, i.e., additional output layer for multi-scale detection, spatial pyramid pooling (SPP) module for richer deep features, and focal loss for alleviation of class imbalance and hard sample classification. Experimental results demonstrate that the integration of multi-scale features as well as hard examples mining greatly contributes to the turn light detection. The detection accuracy is significantly increased by 7.36%, 32.04% and 21.65% (absolute gain) for brake, left-turn and right-turn signals, respectively. In addition, we construct the taillight detection dataset, with brake and turn signals are specified with bounding boxes, which may help nourishing the development of this realm.

Multi-scale approach representing image objects at various levels-of-details has been applied to various computer vision tasks. Existing image classification approaches place more emphasis on multi-scale convolution kernels, and overlook multi-scale feature maps. As such, some shallower information of the network will not be fully utilized. In this paper, we propose the Multi-Scale Residual (MSR) module that integrates multi-scale feature maps of the underlying information to the last layer of Convolutional Neural Network. Our proposed method significantly enhances the characteristics of the information in the final classification. Extensive experiments conducted on CIFAR100, Tiny-ImageNet and large-scale CalTech-256 datasets demonstrate the effectiveness of our method compared with Res-Family.

Blind reduction of speckle noise has become a longstanding unsolved problem in several imaging applications, such as medical ultrasound imaging, synthetic aperture radar (SAR) imaging, and underwater sonar imaging, etc. The unwanted noise could lead to negative effects on the reliable detection and recognition of objects of interest. From a statistical point of view, speckle noise could be assumed to be multiplicative, significantly different from the common additive Gaussian noise. The purpose of this study is to blindly reduce the speckle noise under non-ideal imaging conditions. The multiplicative relationship between latent sharp image and random noise will be first converted into an additive version through a logarithmic transformation. To promote imaging performance, we introduced the feature pyramid network (FPN) and atrous spatial pyramid pooling (ASPP), contributing to a more powerful deep blind DeSPeckling Network (named as DSPNet). In particular, DSPNet is mainly composed of two subnetworks, i.e., Log-NENet (i.e., noise estimation network in logarithmic domain) and Log-DNNet (i.e., denoising network in logarithmic domain). Log-NENet and Log-DNNet are, respectively, proposed to estimate noise level map and reduce random noise in logarithmic domain. The multi-scale mixed loss function is further proposed to improve the robust generalization of DSPN et. The proposed deep blind despeckling network is capable of reducing random noise and preserving salient image details. Both synthetic and realistic experiments have demonstrated the superior performance of our DSPNet in terms of quantitative evaluations and visual image qualities.

The automatic identification system (AIS), an automatic vessel-tracking system, has been widely adopted to perform intelligent traffic management and collision avoidance services in maritime Internet-of-Things (IoT) industries. With the rapid development of maritime transportation, tremendous numbers of AIS-based vessel trajectory data have been collected, which make trajectory data compression imperative and challenging. This article mainly focuses on the compression and visualization of large-scale vessel trajectories and their graphics processing unit (GPU)-accelerated implementations. The visualization was implemented to investigate the influence of compression on vessel trajectory data quality. In particular, the Douglas-Peucker (DP) and kernel density estimation (KDE) algorithms, respectively, utilized for trajectory compression and visualization, were significantly accelerated through the massively parallel computation capabilities of the GPU architecture. Comprehensive experiments on trajectory compression and visualization have been conducted on large-scale AIS data of recording ship movements collected from three different water areas, i.e., the South Channel of Yangtze River Estuary, the Chengshan Jiao Promontory, and the Zhoushan Islands. Experimental results illustrated that: 1) the proposed GPU-based parallel implementation frameworks could significantly reduce the computational time for both trajectory compression and visualization; 2) the influence of compressed vessel trajectories on trajectory visualization could be negligible if the compression threshold was selected suitably; and 3) the Gaussian kernel was capable of generating more appropriate KDE-based visualization performance by comparing with other seven kernel functions.

In this study, we initiate a cyberinfrastructure solution by synergizing both the blockchain and Internet of Things (IoT) technologies for transportation insurance. The insurance premium related services are encapsulated in “on-chain” chaincodes to perform over the facts on vehicle's trip and driver's behavior, which are deduced through “off-chain” analytic services using the sensing data collected from vehicles' on-board sensors. A hybrid scheme coordinating both the permissioned (Hyperledger) and public (Ethereum) blockchains is proposed to exploit their respective capabilities in terms of high transaction throughput and built-in cryptocurrency. A working prototype platform is implemented with a basic premium calculation model. The prototype system is deployed across Amazon Web Services (AWSs) cloud in a real-world Internet environment. A comprehensive performance study from the aspects of throughput, latency, and resource usage under different configurations is presented to show the solution's feasibility. The design practice and research findings are concluded in consort with the experience gained for further enhancing the proposed solution and extending the functional features such as a more realistic insurance policy to be applied in generic vehicle insurance applications.

High-quality dehazing performance is highly dependent upon the accurate estimation of transmission map. In this work, the coarse estimation version is first obtained by weightedly fusing two different transmission maps, which are generated from foreground and sky regions, respectively. A hybrid variational model with promoted regularization terms is then proposed to assisting in refining transmission map. The resulting complicated optimization problem is effectively solved via an alternating direction algorithm. The final haze-free image can be effectively obtained according to the refined transmission map and atmospheric scattering model. Our dehazing framework has the capacity of preserving important image details while suppressing undesirable artifacts, even for hazy images with large sky regions. Experiments on both synthetic and realistic images have illustrated that the proposed method is competitive with or even outperforms the state-of-the-art dehazing techniques under different imaging conditions.

The study of patient behaviours (vital sign, physical action and emotion) is crucial to improve one’s quality of life. The only solution for handling and managing millions of people’s behaviours and health would be big data and IoT technology because most of the countries are lack of medical professionals. In this paper, a big data and IoT-based patient behaviour monitoring system have proposed. Qualitative studies are carried out on the selected behaviours analytics, cardiovascular disease identification and fall detection. At last, authors have summarised the general challenges like trust, privacy, security and interoperability as well as special challenges in various sectors: government, legislators, research institutions, information technology companies and patients.

Outdoor images captured in poor weather conditions (e.g., fog or haze) commonly suffer from reduced contrast and visibility. Increasing attention has recently been paid to single image dehazing, i.e., improving image contrast and visibility. It is generally thought that the dehazing performance highly depends on the accurate depth information. In this work, we first obtain the initial depth map by using the popular dark channel prior. A unified second-order variational framework is then proposed to refine the depth map and restore the haze-free image. The introduced second-order framework has the capacity of preserving important structures in both depth map and haze-free image. Furthermore, the proposed framework performs well for several different types of haze situations. The resulting optimization problems related to depth map estimation and latent image restoration can be effectively handled using the primal-dual algorithm under a two-step numerical framework. The effectiveness of our proposed method has been demonstrated by comparing the imaging performance with several state-of-the-art dehazing methods.

Single-image blind deblurring is a challenging ill-posed inverse problem which aims to estimate both blur kernel and latent sharp image from only one observation. This paper focuses on first estimating the blur kernel alone and then restoring the latent image since it has been proven to be more feasible to handle the ill-posed nature during blind deblurring. To estimate an accurate blur kernel, L0-norm of both first- and second-order image gradients is proposed to regularize the final estimation result. The proposed L0-regularized hybrid gradient sparsity priors obtain major benefit from the intrinsic sparsity properties of images and can assist in guaranteeing high-quality blur kernel estimation. Once the blur kernel is estimated, the final clean image is robustly generated using the combination of L1-norm data-fidelity term and total variation regularizer. Experimental results have demonstrated the satisfactory performance of the proposed method.

The multiplicative noise removal problem has received considerable attention recently. To solve this problem, various variational models have been proposed, which minimise an energy functional composed of the data term and the regularisation term. Regarding the regularisation term, a first‐order model is frequently used to remove multiplicative noise, which may cause staircase effect and loss of contrast in the output image. In this study, the authors use a second‐order model, the total curvature (TC), to solve the above problem. The TC model has the benefit of removing the staircase effect and maintaining image edges, contrasts and corners. The augmented Lagrange method is utilised to solve the proposed TC model by introducing auxiliary variables, Lagrange multipliers and using alternating optimisation strategy. In each loop of optimisation, the fast Fourier transform, generalised soft threshold formulas, projection method and gradient descent method are integrated effectively. The experimental results show that the TC model can effectively remove staircase effect and preserve smoothness, via comparison with the first‐order model (total variation regularisation and Perona–Malik regularisation). Furthermore, the TC model is better than another second‐order model based on bounded Hessian regularisation in preserving contrast and corner.

Single-image blind deblurring could be considered as an important preprocessing step in imaging information fusion. Its purpose is to simultaneously estimate blur kernel and latent sharp image from only one observed blurred image. Blind deblurring has been attracting increasing attention in the fields of image processing, computer vision, computational photography, etc. However, it is a typically ill-posed inverse problem, which requires regularization methods to guarantee stable image restoration results. We first proposed to robustly estimate the blur kernels by exploiting non-convex sparsity constraints on image gradients and blur kernels. The corresponding combined non-convex regularization term has the capacity of enhancing estimation accuracy. To guarantee the high-quality non-blind deblurring with estimated blur kernels, the hybrid non-convex first- and second-order TV regularizer was then introduced to stabilize the final image restoration process. The hybrid non-convex regularizer is able to achieve a good balance between sharp edges preservation and undesirable artifacts suppression. The resulting non-convex minimization problems related to blur kernel estimation and non-blind deblurring were handled using efficient numerical optimization algorithms in this paper. Numerous experiments on both synthetic and realistic images have demonstrated the good performance of the proposed blind deblurring method.

Magnetic resonance imaging (MRI) has been extensively used in clinical practice but suffers from long data acquisition time. Following the success of compressed sensing (CS) theory, many efforts have been made to accurately reconstruct MR images from undersampled k-space measurements and therefore dramatically reduce MRI scan time. To further improve image quality, we formulate undersampled MRI reconstruction as a least-squares optimization problem regularized by shearlet transform and overlapping-group sparsity-promoting total variation (OSTV). Shearlet transform, a directional representation system, is capable of capturing the optimal sparse representation for images with plentiful geometrical information. OSTV performs well in suppressing staircase-like artifacts often arising in traditional TV-based reconstructed images. To guarantee solution stability and efficiency, the resulting optimization problem is solved using an alternating direction methods of multipliers (ADMM)-based numerical algorithm. Extensive experimental results on both phantom and in vivo MRI datasets have demonstrated the superior performance of our proposed method in terms of both quantitative evaluation and visual quality.

Single-image blind deconvolution is a challenging illposed inverse problem which requires regularization techniques to stabilize the restoration process. Its purpose is to recover an underlying blur kernel and a latent image from only one blurred image. In most imaging situations, the blur kernel is not only spatially sparse, but also piecewise smooth with the support of a continuous curve. Thus this paper proposes a hybrid regularized method to robustly estimate the blur kernel by incorporating both L1-norm of kernel intensity and squared L2- norm of intensity derivative. Once the blur kernel is estimated, a total generalized variation based image restoration model is developed to guarantee robust non-blind image deconvolution. All optimization problems related to blur kernel estimation and non-blind deconvolution in this paper will be efficiently solved using fast numerical algorithms. Numerous experiments have been conducted to compare our proposed method with some state-of-the-art blind deconvolution methods on both synthetic and real-world datasets. The experimental results have illustrated the effectiveness of our proposed method in terms of quantitative and qualitative image quality evaluations.

Total variation (TV) is well capable of preserving edges and smoothing flat regions, however, often suffers from staircase artifacts in regions with gradual intensity variations. The established second-order TV can overcome this drawback but may lead to blurred edges and boundaries in restored images. In current literature, their nonconvex extensions have been proven to be effective for further enhancing image quality. This paper proposes an edge-preserving image restoration model by using both nonconvex first- and second-order TV regularizers, with a box constraint. The nonconvex hybrid regularizer is able to significantly suppress the staircase artifacts while preserving the valuable edge information. The addition of the box constraint provides a visible positive effect on image restoration, especially when there are many pixels with values lying on the predefined dynamic range boundaries. In what follows, to guarantee solution efficiency and stability, we develop an iteratively reweighted algorithm based on alternating direction method of multipliers (ADMM) to solve the proposed constrained nonconvex hybrid variational model. Numerous experimental results have demonstrated the superior performance of our proposed method in terms of quantitative and qualitative image quality evaluations.