The social relation is a fundamental attribute of human beings in daily life. The ability of humans to form large organizations and institutions stems directly from our complex social networks. Therefore, understanding social relationships in the context of multimedia is crucial for building domain-specific or general artificial intelligence systems. The key to reason social relations lies in understanding the human interactions between individuals through multimodal representations such as action and utterance. However, due to video editing techniques and various narrative sequences in videos, two individuals with social relationships may not appear together in the same frame or clip. Additionally, social relations may manifest in different levels of granularity in video expressions. Previous research has not effectively addressed these challenges. Therefore, this paper proposes aMulti-Granularity Relation Graph Aggregation Framework(MGRG) to enhance the inference ability for social relation reasoning in multimedia content, like video. Different from existing methods, our method considers the paradigm of jointly inferring the relations by constructing a social relation graph. We design a hierarchical multimodal relation graph illustrating the exchange of information between individuals' roles, capturing the complex interactions at multi-levels of granularity from fine to coarse. In MGRG, we propose two aggregation modules to cluster multimodal features in different granularity layer relation graph, considering temporal aspects and importance. Experimental results show that our method generates a logical and coherent social relation graph and improves the performance in accuracy.

Fine-grained image datasets have small inter-class differences and large intra-class differences, which is a difficulty of the fine-grained image classification. Traditional fine-grained image classification methods only focus on the visual features of images. However, this limitation can be eliminated when these methods are improved with multimodal information. This paper proposes an improved fine-grained image classification method with multimodal information that includes multimodal data preprocessing, multimodal feature extraction, multi-temporal feature fusion and decision correction. The preprocessing method proposed solves the problems of scattered distribution, difficult processing and uneven contribution to prediction of multimodal data through normalization, packing phrases and weighted concatenating methods. When extracting multimodal features, the SAMLP (Self-Attention MLP) module proposed combines self-attention with MLP to capture the internal correlation of multimodal information. The multi-temporal feature fusion proposed is divided into early feature fusion and late feature fusion. The former refers to adding multimodal information markers to the original image, and the latter refers to designing a multi-cascade dynamic MLP structure to fuse visual features and multimodal features. In view of the limitation of feature fusion, a decision strategy is proposed to revise the prediction results of fused features according to the prediction results of multimodal features. Ablation experiment on INAT18-1K and INAT21-1K datasets shows that our method is effective in improving classification with multimodal information. Experiments on the INAT2021_mini large dataset show that the comprehensive method in this article has higher accuracy and negligible efficiency loss compared with the state-of-the-art method.

Electroencephalogram (EEG) based seizure subtype classification plays an important role in clinical diagnostics. However, existing deep learning approaches face two challenges in such applications: 1) convolutional or recurrent neural network based models have difficulty learning long-term dependencies; and, 2) there are not enough labeled seizure sub-type data for training such models. This paper proposes a Transformer-based self-supervised learning model for EEG-based seizure subtype classification, which copes well with these two challenges. Filter bank analysis is first employed to improve Vision Transformer as a Wavelet Transformer (WaT) encoder, which generates multi-grained feature representations of EEG signals. Then, self-supervised learning is used to pre-train WaT from unlabeled EEG data. Experiments on two public datasets demonstrated that Wavelet2Vec outperformed several other supervised and self-supervised models in cross-subject seizure subtype classification.

Continual learning aims to learn on a sequence of new tasks while maintaining the performance on previous tasks. Source-free domain adaptation (SFDA), which adapts a pretrained source model to a target domain, is useful in protecting the source domain data privacy. Generalized SFDA (G-SFDA) combines continual learning and SFDA to achieve outstanding performance on both the source and the target domains. This paper proposes semi-supervised G-SFDA (SSG-SFDA) for domain incremental learning, where a pre-trained source model (instead of the source data), few labeled target data, and plenty of unlabeled target data, are available. The goal is to achieve good performance on all domains. To cope with domain-ID agnostic, SSG-SFDA trains a conditional variational auto-encoder (CVAE) for each domain to learn its feature distribution, and a domain discriminator using virtual shallow features generated by CVAE to estimate the domain ID. To cope with catastrophic forgetting, SSG-SFDA uses soft domain attention to improve the sparse domain attention in G-SFDA. To cope with insufficient labeled target data, SSG-SFDA uses MixMatch to augment the unlabeled target data and better exploit the few labeled target data. Experiments on three datasets demonstrated the effectiveness of SSG-SFDA.

Bootstrap aggregating (Bagging) and boosting are two popular ensemble learning approaches, which combine multiple base learners to generate a composite model for more accurate and more reliable performance. They have been widely used in biology, engineering, healthcare, etc. This article proposes BoostForest, which is an ensemble learning approach using BoostTree as base learners and can be used for both classification and regression. BoostTree constructs a tree model by gradient boosting. It increases the randomness (diversity) by drawing the cut-points randomly at node splitting. BoostForest further increases the randomness by bootstrapping the training data in constructing different BoostTrees. BoostForest generally outperformed four classical ensemble learning approaches (Random Forest, Extra-Trees, XGBoost and LightGBM) on 35 classification and regression datasets. Remarkably, BoostForest tunes its parameters by simply sampling them randomly from a parameter pool, which can be easily specified, and its ensemble learning framework can also be used to combine many other base learners.

Ecosystem monitoring involves fully understanding the complex interactions of ecosystem progress and providing massive scientific data to answer research questions. Fetching data in extreme areas without a public network largely relies on human efforts. Along with the ongoing advances of the Internet of Things (IoT), the Internet of Remote Things (IoRT), as a new paradigm, is being pursued. In this article, we improve the previous drone-enabled IoRT system and innovatively integrate the system with ecological monitoring devices for wildlife, phenology and environmental monitoring, and the monitored data are remotely retrieved by drones. The experimental results indicate that the data transmission rate between the drone relay and the terrestrial terminal reaches up to 10–15 MB/s. Experiments further demonstrate that in terms of wildlife monitoring, the required time to retrieve an image cached on a terrestrial terminal is approximately 0.35 s, the time required to transfer a set of phenology data is 0.29 and 0.34 s, and in regard to environmental monitoring data, the average time consumption reaches approximately 0.0302 s. Moreover, we introduce a signal strength-based priority strategy that can reduce the time consumption of data transmission between the drone relay and the terrestrial terminal. The demonstrated applications reveal that monitoring devices deployed in remote extreme regions can be connected in the IoRT network, and the way to retrieve data from these deployed devices is being revolutionized by the drone-enabled IoRT network.

In the automatic navigation robot field, robotic autonomous positioning is one of the most difficult challenges. Simultaneous localization and mapping (SLAM) technology can incrementally construct a map of the robot’s moving path in an unknown environment while estimating the position of the robot in the map, providing an effective solution for robots to fully navigate autonomously. The camera can obtain corresponding two‐dimensional digital images from the real three‐dimensional world. These images contain very rich colour, texture information, and highly recognizable features, which provide indispensable information for robots to understand and recognize the environment based on the ability to autonomously explore the unknown environment. Therefore, more and more researchers use cameras to solve SLAM problems, also known as visual SLAM. Visual SLAM needs to process a large number of image data collected by the camera, which has high performance requirements for computing hardware, and thus, its application on embedded mobile platforms is greatly limited. This paper presents a parallelization method based on embedded hardware equipped with embedded GPU. Use CUDA, a parallel computing platform, to accelerate the visual front‐end processing of the visual SLAM algorithm. Extensive experiments are done to verify the effectiveness of the method. The results show that the presented method effectively improves the operating efficiency of the visual SLAM algorithm and ensures the original accuracy of the algorithm.