We introduce the BIR-Adapter, a parameter-efficient diffusion adapter for blind image restoration. Diffusion-based restoration methods have demonstrated promising performance in addressing this fundamental problem in computer vision, typically relying on auxiliary feature extractors or extensive fine-tuning of pre-trained models. Building on the observation that large-scale pretrained diffusion models can retain informative representations under image degradations, BIR-Adapter introduces a parameter-efficient, plug-and-play attention mechanism that substantially reduces the number of trained parameters. To further improve reliability, we adapt a sampling guidance mechanism that mitigates hallucinations during restoration. Experiments on synthetic and real-world degradations demonstrate that BIR-Adapter achieves competitive, and in several settings superior, performance compared to state-of-the-art methods while requiring up to 36 × fewer trained parameters. Moreover, the adapter-based design enables integration into existing models. We validate this generality by extending a super-resolution–only diffusion model to handle additional unknown degradations, highlighting the adaptability of our approach for broader image restoration tasks.

This paper introduces CVSC, a real-time-enabled and temporally consistent semantic video communication approach. We minimize the denoising steps of the diffusion model by using the most recent frame and motion information, enabling real-time-capable semantic video synthesis. To ensure temporal consistency, we employ windowed temporal cross-attention. While our objective evaluation highlights the limitations of existing metrics for generative models in semantic video communication, subjective evaluations demonstrate the superiority of our approach in terms of human preference at extremely low bit rates. (< 0.006 bpp).

This work addresses the fidelity problem in low-bitrate real-time semantic video communication, which is crucial for enhancing the user experience. We present I2V-SC, extending baseline diffusion-based Image2Video (I2V) synthesis with ControlNet and distillation tailored to semantic video coding, enabling real-time performance with high fidelity. Evaluations using perceptual, pixel-level, motion, and semantic metrics demonstrate that I2V-SC outperforms baseline I2V and a baseline semantic video communication approach, namely CVSC, in ultra-low-bitrate ($<0.006$bpp) and real-time-enabled settings. Subjective evaluations confirm that I2V-SC further improves the user QoE in terms of overall preferability.

Semantic Communication has recently emerged as a novel communication strategy that prioritizes transmitting meaning over conventional bit-based transmission. By significantly reducing resource requirements in communication tasks such as video conferencing, natural language, and audio transmission, Semantic Communication promises better utilization of challenging, near radio link failure (NRLF) channels. This paper introduces a novel semantic communication framework designed to further enhance the resilience of the transmission of semantics over NRLF channels. Unlike classical, Shannon-based communication that prioritizes the perfect reception of bits, our approach focuses on ensuring the successful synthesis of the message semantics. Our framework leverages the significance of discrete semantics, and a cross-layer strategy to ensure message integrity and comprehension, even under significant loss. Key to our framework is the novel, code block based Proactive Redundancy Transmission (PRT) mechanism prioritizing critical semantics, coupled with a novel error concealment step enabling meaningful reconstruction of non-critical semantics. We establish the resulting importance-driven resilience optimization problem, and introduce and validate preliminary heuristics as an initial attempt to optimize it. We formalize, implement, and evaluate our framework, demonstrating significant improvements in the resilience of semantic communication in NRLF environments. Our evaluations, leveraging a First Order Motion Model (FOMM) for video conferencing synthesis, underscore the resilience of our semantic communication framework against traditional H.265 compression under challenging Channel Block Error Rates (CBLERs). Unlike H.265, which fails to decode under significant CBLERs, our method exhibits remarkable resilience, maintaining perceptual quality even with CBLERs surpassing 75%.

This paper presents a real-time semantic video communication method for general scenes, combining lossy semantic map coding with motion compensation to achieve reduced bit rates while maintaining perceptual and semantic quality. Our findings show that semantic image synthesis effectively adapts to minute errors resulting from motion estimation, eliminating the need to transmit the residuals. We recommend the Group of Pictures approach as a more efficient alternative. Comparative assessments against HEVC and VVC confirm the method’s effectiveness. This research paves the way for efficient real-time semantic video communication, addressing the demands of data-intensive visual applications.

The increasing demand for high-quality, real-time visual communication and the growing user expectations, coupled with limited network resources, necessitate novel approaches to semantic image communication. This paper presents a method to enhance semantic image communication that combines a novel lossy semantic encoding approach with spatially adaptive semantic image synthesis models. By developing a model-agnostic training augmentation strategy, our approach substantially reduces susceptibility to distortion introduced during encoding, effectively eliminating the need for lossless semantic encoding. Comprehensive evaluation across two spatially adaptive conditioning methods and three popular datasets indicates that this approach enhances semantic image communication at very low bit rate regimes.

This article introduces a novel skill learning framework that learns rewards from very few demonstrations and uses them in policy search (PS) to improve the skill. The demonstrations are used to learn a parameterized policy to execute the skill and a goal model, as a hidden Markov model (HMM), to monitor executions. The rewards are learned from the HMM structure and its monitoring capability. The HMM is converted to a finite-horizon Markov reward process (MRP). A Monte Carlo approach is used to calculate its values. Then, the HMM and the values are merged into a partially observable MRP to obtain execution returns to be used with PS for improving the policy. In addition to reward learning, a black box PS method with an adaptive exploration strategy is adopted. The resulting framework is evaluated with five PS approaches and two skills in simulation. The results show that the learned dense rewards lead to better performance compared to sparse monitoring signals, and using an adaptive exploration lead to faster convergence with higher success rates and lower variance. The efficacy of the framework is validated in a real-robot settings by improving three skills to complete success from complete failure using learned rewards where sparse rewards failed completely.