The next-generation network is expected to interconnect multiple traditionally isolated networks, including aerial, sea surface, and underwater domains. However, achieving low-latency communication in such multi-domain networks is challenging because single-modal or non-adaptive multimodal approaches often fail, owing to disparate propagation characteristics across different channels and complex network conditions. This paper proposes an Adaptive Low-Latency Multi-Modal Cooperation Routing (ALL-M2CR) approach, which leverages the complementary strengths of multi-modal communications including Radio Frequency (RF), Underwater Acoustic Communication (UAC), and Optical Wireless Communication (OWC). By establishing real-time network state awareness and multi-modal cooperation, ALL-M2CR selects the globally optimal low-latency path, enabling opportunistic use of high-speed direct air-water OWC and RF links. Simulation results for cross-domain and intra-domain time-sensitive transmission tasks demonstrate that ALL-M2CR significantly outperforms modality-restricted and rule-based routing approaches in path connectivity and average end-to-end latency, reducing latency by up to 97% compared to the rule-based RF/UAC dual-modal routing.

Multi-modal image fusion, particularly between infrared (IR) and visible (VIS) images, integrates complementary information from diverse imaging sources to enhance perception in applications like autonomous driving and surveillance. While IR images capture thermal radiation and VIS images provide rich texture details, existing fusion methods face challenges in preserving infrared thermal signatures under high illumination, where overexposed VIS regions dominate fusion outputs. To address the above problems, we propose XOR-Fuse, a novel logical operation-driven fusion framework that explicitly captures complementary pixel-level discrepancies between IR and VIS modalities. The XOR-Fuse defines the pixel-wise analogy for the logical XOR operation, which rectifies the suppression of IR features caused by conventional maximum-intensity fusion rules under high illumination. To further reinforce IR feature preservation, we integrate multi-scale Gabor wavelet filtering and wavelet decomposition for illumination-invariant texture extraction and VGG-based semantic constraints, ensuring structural congruence between IR and VIS details. Experiments on MSRS, RoadScene, and TNO datasets demonstrate significant improvements over four baseline models (DeepFuse, SDNet, U2Fusion, and DATFuse). For instance, the enhanced DeepFuse achieves SD=48.27 and VIF=0.62 on MSRS, outperforming the original model (SD=33.79, VIF=0.42). Qualitative results under variable illumination confirm the recovery of suppressed IR details while retaining VIS textures.

Routing is essential in communication as it determines the most efficient path for data packets to travel from the source to the destination, ensuring reliable network connectivity. However, with the emergence of cross-domain communication scenarios involving air, surface, and underwater environments, the traditional routing protocol—which considers only limited information including bandwidth or delay as static link cost during route selection—falls short of meeting the rapid and efficient demands of cross-domain heterogeneous communication. To address this issue, this paper proposes a dynamic cost intelligent routing algorithm. The algorithm incorporates an effective bandwidth estimation module and a link priority prediction module, which dynamically process the extracted link information from air, surface, and underwater communication networks. The processed data is then fed into the deep learning model, where it takes multi-dimensional link attribute information as input and outputs a one-dimensional cost corresponding to each link. The effective cost are then integrated back into the network, dynamically adjusted the routing decisions using the shortest path algorithm to optimize overall network performance. The algorithm ensures more reliable data transmission and efficient utilization of network resources. To simulate real-world scenarios, this study utilized the NS3 platform to construct a cross-domain simulation environment that encompasses radio communication, underwater acoustic communication, and optical communication across air, surface, and underwater domains. We conducted experiments by constructing a heterogeneous network consisting of 36 nodes. The results show that this algorithm outperforms the routing algorithms based on the traditional OSPF routing algorithm, and several other algorithms, in relation to throughput, packet loss rate and average delay.

Skin cancer is one of the most prevalent cancer types. Training a convolutional neural network (CNN) for skin cancer diagnosis usually requires a large amount of labeled data to yield good performance. However, obtaining high-quality labels is laborious and expensive, as accurately annotating medical images demands expertise knowledge of the clinicians. In this paper, we present a novel semi-supervised framework for skin cancer diagnosis, SSCD-Net combines curriculum learning strategy, disease relation and the guidance of clinical information. Curriculum learning (CL) strategy is designed to help train from easy instances and then gradually handles harder ones to improve the generalization capacity and convergence rate. The whole training set is split into a number of subsets ranked from an easy one to a more complex one, in an unsupervised manner. The SSCD-Net is driven by disease relation apart from conventional individual consistency, which explicitly enforces the consistency of disease relation among different samples under perturbations, encouraging the model to explore extra semantic information from unlabeled data. At the end of the classification pipeline, a multimodal information fusion (MIF) module is designed to fuse the image features and clinical features, further introducing the guidance of clinical information. Comprehensive experiments are conducted on four publicly accessible dataset, i.e., PADUEFC-20, ISIC 2018, ISIC 2019 and ISIC-Archive. Simulations show that the SSCD-Net has better or comparable performance against that of the other state-of-the-art semi-supervised models.

For medical image segmentation, contrastive learning recently becomes the dominant method to get the class-separated visual representations by pulling augmented views of the same samples closer in a representation space, and pushing apart augmented views of different samples. However, most current contrastive learning methods may suffer from the issue of class collision because those images (pixels) that from the same class are forcefully contrasted due to the pretext task simply takes another augmentation of the same image as positive pairs and all other images are view as negative samples. Moreover, the problem looks more serious in pixel-level contrastive learning in that massive pixels from the same class are viewed as negative pairs to differ is intolerable. Therefore, to alleviate this, we propose a novel semi-supervised learning framework based on hierarchical data augmentation in contrastive learning for medical image segmentation. Specifically, We first propose a novel contrastive loss function based on class-aware patch by using the pseudo labels and source image effectively. With the novel loss, we learn better intra-class compactness and inter-class separability in the feature space. We then introduce a patch-based hierarchical data augmentation module to learn the hierarchical invariance by applying different strength of augment to different probability of patches, thus making the patches with a high probability of belong to a certain class more alike and avoiding the issue of class collision in contrastive learning. Experiments on several public accessible datasets from multiple domains reveals the superiority of our proposed method as compared to the state-of-the-art semi-supervised and contrastive learning methods.

With the rapid advancement of autonomous driving technology, there is an increasingly urgent demand for faster and more accurate object detection frameworks. Recently, numerous deep learning-based object detectors have shown impressive performance in real-time driving applications. However, the detection of small objects such as various traffic signs remains challenging due to the complex nature of these objects. This paper proposes a transformer-based detector TS-DETR to improve the accuracy of small object detection in autonomous driving systems. We introduce a constrained decoder structure to focus the model's attention on the predicted boxes. Additionally, a content-aware deformable cross-attention mechanism is proposed to obtain more comprehensive attention weights. Experimental results on challenging public datasets such as TT100K and CCTSDB2021 demonstrate that our method demonstrates substantial performance enhancements while introducing only a slight increment in parameter count compared to current algorithms.

Deep learning has been proved to be one of the most effective method in feature encoding for different intelligent applications such as video-based human action recognition. However, its nonconvex optimization mechanism leads large memory consumption, which hinders its deployment on the smart embedded systems with limited computational resources. To overcome this challenge, we propose a novel deep model compression technique for smart embedded systems, which realizes both the memory size reduction and inference complexity decrease within a small drop of accuracy. First, we propose an improved naive Bayes inference-based channel parameter pruning to obtain a sparse model with higher accuracy. Then, to improve the inference efficiency, the improved Tucker decomposition method is proposed, where an improved genetic algorithm is used to optimize the Tucker ranks. Finally, to elevate the effectiveness of our proposed method, extensive experiments are conducted. The experimental results show that our method can achieve the state-of-the-art performance compared with existing methods in terms of accuracy, parameter compression, and floating-point operations reduction.

Human behavior understanding is playing more and more important role in human-centered Industrial Internet of Video Things (IIoVT) system with the deep combination of artificial intelligence and video-based industrial Internet of Things. However, it requires expensively computational resources, including high-performance computing units and large memory, to train a deep computation model with a large number of parameters, which limits its effectiveness and efficiency for IIoVT applications. In this article, a tensor-train mechanism based deep model is presented for video human behavior understanding to meet the requirement of IIoVT applications. It can get competitive performance in accuracy and training efficiency with potentiality for combination of artificial intelligence and prefront IIoVT system. On the one hand, to achieve desirable accuracy, we improved the conventional CNN and adopted the recurrent neural network mechanism to enhance the video representation over time, which takes the correlation between consecutive deep feature into consideration. On the other hand, to enhance the inference capacity between the spatial and temporal features, we carry out the self-critical reinforcement learning mechanism in parameter learning stage. Meanwhile, to further reduce parameter storage size to meet requirement for the deployment of deep neural network and edge device, the tensor-train mechanism is used, which transforms the parameter matrix to a tensor space and carry out tensor decomposition mechanism to decrease the number of parameter generated in parameter training. Finally, we conduct extensive experiments to evaluate our scheme, and the results demonstrate that our method can improve the training efficiency and save the memory space for the deep computation model with better accuracy.

Video surveillance techniques like scene segmentation are playing an increasingly important role in multimedia Internet-of-Things (IoT) systems. However, existing deep learning-based methods face challenges in both accuracy and memory when deployed on edge computing devices with limited computing resources. To address these challenges, a tensor-train video scene segmentation scheme that compares the local background information in regional scene boundary boxes in adjacent frames is proposed. Compared to the existing methods, the proposed scheme can achieve competitive performance in both segmentation accuracy and parameter compression rate. In detail, first, an improved faster region convolutional neural network (faster-RCNN) model is proposed to recognize and generate a large number of region boxes with foreground and background to achieve boundary boxes. Then, the foreground boxes with sparse objects are removed and the rest are considered as optional background boxes used to measure the similarity between two adjacent frames. Second, to accelerate the training efficiency and reduce memory size, a general and efficient training way using tensor-train decomposition to factor the input-to-hidden weight matrix is proposed. Finally, experiments are conducted to evaluate the performance of the proposed scheme in terms of accuracy and model compression. Our results demonstrate that the proposed model can improve the training efficiency and save the memory space for the deep computation model with good accuracy. This work opens the potential for the use of artificial intelligence methods in edge computing devices for multimedia IoT systems.

Multimedia Internet-of-Things (IoT) systems have been widely utilized in various computer vision tasks and significantly integrated computer vision and networking capabilities. In these systems, convolutional neural networks (CNNs) perform a preliminary analysis of the collected video or image information in the edge devices. However, the high computational cost and huge storage consumption of the complex CNNs prevent their deployment on mobile-edge devices that have limited computational resource and memory. In this article, we aim to simultaneously accelerate and compress CNNs via a multilevel filter pruning (MFP) algorithm, to alleviate the dependence on the hardware of IoT edge nodes. First, a global pruning sensitivity order is defined, which could guide us to perform preliminary pruning from the perspective of convolutional layers' sensitivity. Then, the functional index of each filter is judged by the image entropy of its output feature map, which contributes to further pruning from the perspective of filter function importance. Finally, the moderate fine tuning is adopted to recover the network capability. The experimental results show that the proposed MFP algorithm could reduce 54.5% floating-point operations and 31.9% graphics memory for VGG-16 on CIFAR-10, and achieve 5.45 × floating-point acceleration and 19.70 × storage reduction for VGG-16 on ImageNet. In the reconstruction phase, the algorithm could recover the network capability much faster than the existing pruning algorithms.

Extracting road information from remote sensing images has been a popular issue for decades. However, most studies focus on cloudless datasets, without considering cloud occlusion. The thick clouds especially make it impossible to extract the road information from the blocked parts. To address this problem, we propose a new two-stage method. Since generative adversarial networks (GAN) have powerful image generation capabilities, the two-stage method comprises an edge prediction net relied on GAN and a color filling part. The edge prediction net sketches the contours of the region contaminated by thick clouds in road labels, and the second part fills colors in the missing part by using the edges predicted at the first stage. We evaluate our model over the DeepGlobe Road Extraction dataset. The results show that our model performs excellently on visual effects and evaluation indicators.

With the increasing demands of end-users to the visual perception in three-dimension (3D) image, quality assessment for 3D imageis dominantly required as the feedback information for multimedia transmission systems. In this paper, a novel full-reference quality assessment method by considering the depth and integral color information of 3D image under cloud computing environment is proposed. Based on the property of the depth information in 3D image, the depth map is firstly separated into different planes according to the perception of human visual system (HVS). Then, after express the image pixels of every separated plane through quaternions, the structural and energy information are separated by quaternion singular value decomposition (QSVD). The distortion of structural and energy in every plane are calculated in various formulas respectively. The final result is calculated in terms of the global score, which synthesizes the structural and energy distortion scores in every individual depth plane. It should be pointed out that the chrominance information is employed in our mechanism to evaluate the color image quality because of its useful characteristic for 3D color image quality assessment, and its spatial correlation is used for calculating structural distortion through vector cross-product. Our experimental results confirm that the proposed method has achieves better performance under cloud computing environments compared with other existing 3D image quality assessment methods.

C yber- P hysical- S ocial S ystems (CPSS) integrating the cyber, physical, and social worlds is a key technology to provide proactive and personalized services for humans. In this paper, we studied CPSS by taking h uman- i nteraction-aware b ig d ata (HIBD) as the starting point. However, the HIBD collected from all aspects of our daily lives are of high-order and large-scale, which bring ever-increasing challenges for their cleaning, integration, processing, and interpretation. Therefore, new strategies for representing and processing of HIBD become increasingly important in the provision of CPSS services. As an emerging technique, tensor is proving to be a suitable and promising representation and processing tool of HIBD. In particular, tensor networks, as a significant tensor decomposition technique, bring advantages of computing, storage, and applications of HIBD. Furthermore, T ensor- T rain (TT), a type of tensor network, is particularly well suited for representing and processing high-order data by decomposing a high-order tensor into a series of low-order tensors. However, at present, there is still need for an efficient Tensor-Train decomposition method for massive data. Therefore, for larger-scale HIBD, a highly-efficient computational method of Tensor-Train is required. In this paper, a d istributed T ensor- T rain (DTT) decomposition method is proposed to process the high-order and large-scale HIBD. The high performance of the proposed DTT such as the execution time is demonstrated with a case study on a typical form of CPSS data, C omputed T omography (CT) image data.

High efficiency video coding (HEVC), which is the newest video coding standard currently, achieves the best coding efficiency compared with all the other existing video coding standards. However, the computational complexity of the typical HEVC encoder dramatically increases because of the recursive searching scheme for finding the best coding unit (CU) partitions. In this paper, an adaptive fast CU size decision algorithm for HEVC Intra prediction is proposed based on CU complexity classification (CC) by using machine learning (ML) technology. Firstly, certain image features are extracted to characterize the CU complexity, which has a strong relationship with CU partitions, and then, the support vector machine is employed to analyze and construct the classification model according to the CU complexity. Finally, the proposed adaptive fast CU size decision algorithm, named as CCML, is released based on the complexity classification. The experimental results show that the proposed algorithm could achieve around 60% encoding time reduction for various test video sequences on average with only 1.26% Bjontegaard delta bit rate increase compared with the reference test model HM15.0 of HEVC.

Weather recognition is of great significance in traffic safety, environment and meteorology. However, the visual image features of the weather are highly abstractive, and the traditional method of weather recognition has a high computational complexity and low accuracy. In this paper, the edge deterioration phenomenon is introduced in convolution neural network (CNN) to solve the problem that common CNN cannot distinguish the specific weather. The proposed method used Mask R-CNN to extract the regions of interest including the foreground and foreground edges in the image, and superimposed them into the same-scale matrix and then input them into the network for classification. Outdoor traffic image experiments showed that this method can effectively improve the classification accuracy of the four weather conditions (sunny, foggy, rainy and snowy).

Because of the co-existing of H.264/AVC and high efficiency video coding standard (HEVC) in the coming long period, video transcoding technology has become an essential part of multimedia communication in the field of the Internet of Vehicles (IoV). However, due to the huge computational complexity of re-encoding processes, traditionally cascaded transcoders greatly increase the computing burden of the embedded devices and impact the real-time capability of transportation communication systems. In order to address this problem, a fast transcoding solution is proposed in this paper. First, we exploit the mapping relationship among H.264/AVC decoding information and HEVC coding unit (CU) depth decision and prediction unit (PU) mode decision. Then, a three-output classification model is built for CU depth decision processes, and a two-output classification model is built for PU mode selection processes by using support vector machine method. Finally, the models are applied into the cascaded transcoder to accelerate the re-encoding process. The experimental results show that our proposal averagely achieves up to 53.7% and 52.3% complexity reductions under Lowdelay_P_main and Randomaccess_main configurations, respectively, with the negligible rate-distortion degradation, which show a great potential in improving the transcoding efficiency in the real-time video communication system of IoV.

Machine-to-machine (M2M) communication, an automated communications technology for the equipment or devices, holds great promise in every corner of the modern society, such as civil transportation, smart homes, smart grids, and industrial automation. The M2M technology is still in its infancy, and further development and deployment of M2M systems hinges on establishing an efficient and secure information management system with a satisfactory security level. In this paper, we extend the idea of (t, n) secret sharing for information transmission in M2M with high security and efficiency. Specifically, a secret is divided into 2k shares, and then, transmitted through 2k node-disjoint paths constructed by the Latin square. Note that in our scheme, the secret key is simultaneously transmitted along with the encrypted message through these 2k paths from the source node to the destination node, which greatly improves the efficiency of point-to-point communications in M2M systems. Furthermore, owing to the properties of (t, n) secret sharing, the security of M2M communications is guaranteed. In addition, to avoid dishonest participants, verifiable secret sharing is supported in the proposed scheme. Sufficient theoretical proof and performance analysis demonstrate that our scheme is secure and efficient for M2M communications.

The latest High Efficiency Video Coding (HEVC) standard could achieve the highest coding efficiency compared with the existing video coding standards. To improve the coding efficiency of the intra frame, a quad-tree-based variable block size coding structure that is flexible to adapt to various texture characteristics of images and up to 35 intra-prediction modes for each prediction unit (PU) is adopted in HEVC. However, the computational complexity is increased dramatically because all the possible combinations of the mode candidates are calculated in order to find the optimal rate distortion cost using the Lagrange multiplier. To alleviate the encoder computational load, this paper proposes an adaptive mode decision algorithm based on texture complexity and direction for HEVC intra prediction. First, an adaptive coding unit selection algorithm according to each depth levels' texture complexity is presented to filter out unnecessary coding block. Then, the original redundant mode candidates for each PU are reduced according to its texture direction. The simulation results show that the proposed algorithm could reduce around 56% encoding time on average while maintaining the encoding performance efficiently with only a 1.0% increase in BD-rate compared with the test model HM16 of HEVC.

Recently, the deep computation model, as a tensor deep learning model, has achieved super performance for multimedia feature learning. However, the conventional deep computation model involves a large number of parameters. Typically, training a deep computation model with millions of parameters needs high-performance servers with large-scale memory and powerful computing units, limiting the growth of the model size for multimedia feature learning on common devices such as portable CPUs and conventional desktops. To tackle this problem, this article proposes a Tucker deep computation model by using the Tucker decomposition to compress the weight tensors in the full-connected layers for multimedia feature learning. Furthermore, a learning algorithm based on the back-propagation strategy is devised to train the parameters of the Tucker deep computation model. Finally, the performance of the Tucker deep computation model is evaluated by comparing with the conventional deep computation model on two representative multimedia datasets, that is, CUAVE and SNAE2, in terms of accuracy drop, parameter reduction, and speedup in the experiments. Results imply that the Tucker deep computation model can achieve a large-parameter reduction and speedup with a small accuracy drop for multimedia feature learning.

Video quality assessment (VQA) plays an important role in video processing applications, e.g., compression, archiving, restoration, transmission and enhancement. Based on the video content, we design an effective and efficient objective video quality metric. Up to now, many efforts have been made to develop the VQA that take advantages of the various characteristics of human visual system (HVS). Several objective quality assessment metrics have been proposed for VQA, such as mean structural similarity (MSSIM), visual-structural similarity (V-SSIM), motion-based video Integrity evaluation (MOVIE) and so on. However, motion information is of great importance in image processing, which has not been effectively studied and applied in VQA. In our algorithm, it is effectively used. Firstly, the video sequences content is divided into two parts: foreground and background according to its special property. Then the new video quality metric by utilizing the motion vector is established. Finally, the proposed metric is tested on the VQEG FRTV Phase 1 database. From the experimental results, it is concluded that our metric out performs the other state-of-art algorithms.

In order to apply high quality video communication in the mobile device with limited power and communication bandwidth, especially in wireless communication, an efficient solution for video communication should be proposed. So, the latest video coding standard, H.264/AVC, is used for Video Codec in mobile terminals. However, the H.264/AVC standard is much more complex than previous video coding standards because of utilizing improved inter and intra predication modes. In order to use the Video Codec in mobile devices, a high quality with low-cost interframe mode decision algorithm based on reducing redundant candidate modes and reducing reference frames for each block type by using coded macroblock (MB) information is proposed in this paper. From the results of our experiments, on average, our proposed algorithm can reduce 70.28% encoding time and decrease 0.97% bit rate (BR) with little quality loss.

In this paper, a complexity control scheme (CCS) for Inter frame mode decision (MD) is proposed for H.264/AVC encoder to speed-up the original encoding process. The information extracted from macroblock (MB), which can be used to pre-estimate the optimal mode of the MB is investigated and utilized to eliminate the redundant mode candidates. The simulation results show that the proposed algorithm can reduce over 80% Inter frame encoding time with little quality loss. It can be widely implemented in the mobile communication systems with H.264/AVC standard to realize the real-time video signal coding.

In this paper, we propose a full reference (FR) video quality assessment (VQA) algorithm for the broadcasting multimedia signal. The relationships between the VQA parameters and the quality score obtained by the subjective human visual system (HVS) model of the distorted videos are investigated. Considering the high correlations and the importance of the VQA parameters, a new FR VQA algorithmis worked out to estimate the quality of the video sequences. The simulation results show that our proposed algorithm can represent the quality of the video signals which broadcasted through network efficiently and give high correlation with the subjective video quality (VQ) score.

In this paper, we proposed a fast Intra prediction method for MPEG-2 to H.264/AVC transcoding. To reduce the computational complexity, we utilized the information of DCT coefficients in MPEG-2 decoder to measure the Intra macroblock (MB) energy trend. Base on the relationship between the energy trend and the intra prediction direction, we can pre-determine the Intra prediction for both luminance (luma) and chrominance (chroma) components. The numbers of the Intra prediction candidates of luma Intra16x16 and chroma Intra8x8 can be reduced from original four to one or two, and the numbers of prediction candidates of luma Intra 4x4 can be reduced from nine to four. With our proposed algorithm, the computational complexity of Intra prediction for MPEG-2 to H.264/AVC transcoding can be reduced about 60% compared with the conventional cascaded transcoding method at the negligible quality loss.

CROWN is a service-oriented grid computing middleware enabling resources integration in multiple heterogeneous domains and establishing dynamic cooperative relationship among researchers nationwide and worldwide. However, several security challenges should be addressed in CROWN due to the heterogeneous distribution of resources and the dynamic collaborations and resource sharing. In this paper, we present a security and trustworthiness architecture, CROWN-ST, for CROWN. The aim of this architecture is to provide a fine-grained and extensible framework for security and trustworthiness that enables employing distributed access control and dynamic trust establishment among service providers and consumers in a Grid environment. Based on this open and flexible architecture, a series of fundamental services which consist of secure communication, authentication, access control, credential federation, trust management and negotiation are implemented. Finally, comprehensive experimental studies are conducted to demonstrate the feasibility and performance of current CROWN-ST implementation.