With the rapid development of video conferencing technology, interactive multimedia services proliferate, resulting in a surge in business traffic. Remote work and online collaboration will become the mainstream office in the future, which poses new challenges to computer vision technology in the field of communication. At the same time, the appearance of semantic communication has solved the problem that the traditional video coding technology can not deal with semantic redundancy. Therefore, we propose the Semantic Transmission Optimization of Video Conference(STOVC) architecture to meet the interactivity requirements of multimedia services. We first use Iterated Integrated Attribution(IIA) and Segment Anything Model 2(SAM2) to understand the video and complete the semantic segmentation of the video. We then propose Neural Video Compression with Actor-Critic(NVCAC) to optimize video encoding and transmission strategies to balance video compression with semantic fidelity. Finally, the$\alpha$-fusion method is used to reconstruct the video at the receiving end. In addition, we use DeepCache accelerated stable diffusion to generate conference video to enrich the data set. The experimental results show that STOVC has a better performance than H.264 and H.265 at low bit rate.

The transition from 5G to beyond 5G (B5G) heralds a shift towards more pervasive and intelligent communication trends. This evolution necessitates a departure from traditional information theory towards semantic communication (SemCom), propelled by artificial intelligence (AI), aimed at enhancing capacity and optimizing resources. Concurrently, image SemCom (ISC) emerges to empower image applications. However, ISC demands suitable devices and ample computing resources to support complex neural network models and their training, presenting a significant challenge. In response, we propose a federated semantic feature distillation (FedSFD) architecture to enhance the overall performance of ISC. Combining federated learning (FL) and feature distillation (FD), FedSFD facilitates the transfer of group feature knowledge. Specifically, the powerful server model refines itself by approximating the distance between its middle layer features and those of the devices via FD. Subsequently, lightweight device ISC models leverage FD to incorporate the server model’s knowledge during training. This iterative process is bolstered by the information bottleneck (IB)-based loss function, enhancing image compression and reconstruction capabilities. Notably, this architecture operates without necessitating a unified model, thereby offering improved privacy protection. Simulation experiments demonstrate that compared to the baseline, our approach can achieve superior integration capability for ISC at the noisy edge.

Semi-supervised learning has made remarkable strides by effectively utilizing a limited amount of labeled data while capitalizing on the abundant information present in unlabeled data. However, current algorithms often prioritize aligning image predictions with specific classes generated through self-training techniques, thereby neglecting the inherent relationships that exist within these classes. In this paper, we present a new approach called OTMatch, which leverages semantic relationships among classes by employing an optimal transport loss function to match distributions. We conduct experiments on many standard vision and language datasets. The empirical results show improvements in our method above baseline, this demonstrates the effectiveness and superiority of our approach in harnessing semantic relationships to enhance learning performance in a semi-supervised setting.

Continual learning requires learning incremental tasks with dynamic data distributions. So far, it has been observed that employing a combination of contrastive loss and distillation loss for training in continual learning yields strong performance. To the best of our knowledge, however, this contrastive continual learning framework lacks convincing theoretical explanations. In this work, we fill this gap by establishing theoretical performance guarantees, which reveal how the performance of the model is bounded by training losses of previous tasks in the contrastive continual learning framework. Our theoretical explanations further support the idea that pre-training can benefit continual learning. Inspired by our theoretical analysis of these guarantees, we propose a novel contrastive continual learning algorithm called CILA, which uses adaptive distillation coefficients for different tasks. These distillation coefficients are easily computed by the ratio between average distillation losses and average contrastive losses from previous tasks. Our method shows great improvement on standard benchmarks and achieves new state-of-the-art performance.

Few-shot learning aims to adapt models trained on the base dataset to novel tasks where the categories were not seen by the model before. This often leads to a relatively concentrated distribution of feature values across channels on novel classes, posing challenges in determining channel importance for novel tasks. Standard few-shot learning methods employ geometric similarity metrics such as cosine similarity and negative Euclidean distance to gauge the semantic relatedness between two features. However, features with high geometric similarities may carry distinct semantics, especially in the context of few-shot learning. In this paper, we demonstrate that the importance ranking of feature channels is a more reliable indicator for few-shot learning than geometric similarity metrics. We observe that replacing the geometric similarity metric with Kendall’s rank correlation only during inference is able to improve the performance of few-shot learning across a wide range of methods and datasets with different domains. Furthermore, we propose a carefully designed differentiable loss for meta-training to address the non-differentiability issue of Kendall’s rank correlation. By replacing geometric similarity with differentiable Kendall’s rank correlation, our method can integrate with numerous existing few-shot approaches and is ready for integrating with future state-of-the-art methods that rely on geometric similarity metrics. Extensive experiments validate the efficacy of the rank-correlation-based approach, showcasing a significant improvement in few-shot learning.

Leveraging patch-level embedding in few-shot learning is widely studied by recent works. However, a fundamental challenge is that labels are actually assigned at image level, whereas patch-level annotations are missing. To deal with this problem, we observe that it exactly matches the applications of multiple instance learning (MIL) and novelly incorporate multiple instance learning with few-shot learning. Specifically, we propose a dynamic relation-aware multiple instance learning framework that explicitly models the spatial and semantic relation on instances and performs iterative aggregation. Extensive experiments demonstrate that the proposed method achieves competitive results compared with state-of-the-arts methods.