The rapid emergence of the Metaverse requires higher network throughput and lower latency to deliver immersive and responsive virtual experiences. Traditional centralized data processing approaches are constrained by limited computational and bandwidth resources when handling large-scale user data. A Cloud-Edge-End transmission architecture is proposed in this study, tailored for Metaverse scenarios to optimize resource allocation, minimize latency, and enhance rendering efficiency. A real-time trajectory segment prediction scheme (FDK) was developed, which combines FastDTW with K-means by leveraging user behavior trajectories to determine subscene popularity and store them on GPU servers, thereby reducing user wait time. A two-tier cache optimization scheme (MAE2C) is also proposed, incorporating GCN for subscene feature identification. GPU servers employ the MADDPG strategy to cache popular subscenes, while edge servers utilize DDPG to cache missed scenes. This approach effectively reduces cloud access and cache replacement frequency. Simulation results demonstrate that the subscene cache hit rate of the MAE2C scheme significantly outperforms existing methods across various cache capacities, with a 6.9% reduction in cache replacement frequency. This research provides effective technical support for Metaverse scene rendering and offers insights into the development of generative Metaverse systems.

As large language models (LLMs) demonstrate impressive performance in various tasks and are increasingly integrated into the decision-making process, ensuring they align with human values has become crucial. This paper highlights that value priorities—the relative importance of different value—play a pivotal role in the decision-making process. To explore the value priorities in LLMs, this paper introduces INVP, a framework for INvestigating Value Priorities through decision-making in social scenarios. The framework encompasses social scenarios including binary decision-making, covering both individual and collective decision-making contexts, and is based on Schwartz’s value theory for constructing value priorities. Using this framework, we construct a dataset, which contains a total of 1613 scenarios and 3226 decisions across 283 topics. We evaluate seven popular LLMs and the experimental results reveal commonalities in the value priorities across different LLMs, such as an emphasis on Universalism and Benevolence, while Power and Hedonism are typically given lower priority. This study provides fresh insights into understanding and enhancing the moral and value alignment of LLMs when making complex social decisions.

With the rapid development of the Metaverse, the demand for high transmission rates and low latency is increasing. Traditional centralized data processing architectures, however, are unable to meet these demands due to resource and bandwidth bottlenecks. This paper proposes a cloud-edge-end collaborative transmission architecture to optimize resource allocation and improve rendering efficiency. A real-time trajectory segmentation prediction scheme (FDK) integrates the FastDTW algorithm with KMeans clustering to predict sub-scene popularity, enabling GPU cache allocation and reducing user wait times. Additionally, a two-tier cache optimization scheme (MAE2C) uses GCN to analyze sub-scene features, employing MADDPG to cache popular scenes on GPU servers and DDPG to cache missed scenes on edge servers. Simulation results show that the MAE2C scheme significantly improves cache hit rates, reducing cache replacement frequency by45.14%. This study provides efficient support for Metaverse scene rendering and insights for the development of generative Metaverse technologies.

Existing deep neural network (DNN)-based blind image quality assessment (BIQA) methods primarily rely on human-rated datasets for training. However, collecting human labels is extremely time-consuming and labor-intensive, posing a significant bottleneck for practical applications. To address this challenge, we propose a Deep opinion-Unaware BIQA model by learning and adapting from Multiple Annotators, termed DUBMA, thereby eliminating the need for human annotations. Specifically, we first generate a large-scale set of distorted image pairs and then assign relative quality rankings using existing full-reference IQA models. The resulting dataset is subsequently employed for training our DUBMA. Due to the inherent discrepancies between synthetic and real-world distortions, a domain shift may occur. To address this, we propose an outlier-robust unsupervised domain adaptation approach leveraging optimal transport. This strategy effectively reduces the gap between synthetic and real-world distortion domains, thereby boosting the model’s adaptability and overall performance. Extensive experiments show that DUBMA outperforms existing opinion-unaware BIQA methods in terms of prediction accuracy across multiple datasets.

The advancement of deep generation technology has significantly enhanced the growth of artificial intelligencegenerated content (AIGC). Among these, AI-generated omnidirectional images (AGOIs), hold considerable promise for applications in virtual reality (VR). However, the quality of AGOIs varies widely, and there has been limited research focused on their quality assessment. In this letter, inspired by the characteristics of the human visual system, we propose a novel viewportindependent blind quality assessment method for AGOIs, termed VI-AGOIQA, which leverages vision-language correspondence. Specifically, to minimize the computational burden associated with viewport-based prediction methods for omnidirectional image quality assessment, a set of image patches are first extracted from AGOIs in Equirectangular Projection (ERP) format. Then, the correspondence between visual and textual inputs is effectively learned by utilizing the pre-trained image and text encoders of the Contrastive Language-Image Pre-training (CLIP) model. Finally, a multimodal feature fusion module is applied to predict human visual preferences based on the learned knowledge of visual-language consistency. Extensive experiments conducted on publicly available database demonstrate the promising performance of the proposed method. The source code will be made available at https://github.com/LXLHXL123/VI-AGOIQA.

The ability to distinguish whether an image is generated by artificial intelligence (AI) is a crucial ingredient in human intelligence, usually accompanied by a complex and dialectical forensic and reasoning process. However, current fake image detection models and databases focus on binary classification without understandable explanations for the general populace. This weakens the credibility of authenticity judgment and may conceal potential model biases. Meanwhile, large multimodal models (LMMs) have exhibited immense vision-language capabilities on various tasks, bringing the potential for explainable fake image detection. Therefore, we pioneer the probe of LMMs for explainable fake image detection by presenting a multimodal database encompassing descriptions of textual authenticity, the FakeBench. For construction, we first introduce a fine-grained taxonomy of generative visual forgery concerning human perception, based on which we collect forgery descriptions in human natural language with a human-in-the-loop strategy. FakeBench examines LMMs with four evaluation criteria: detection, reasoning, explanation and fine-grained forgery analysis, to obtain deeper insights into image authenticity-relevant capabilities. Experiments on various LMMs confirm their merits and demerits in different aspects of fake image detection tasks. This research presents a paradigm shift towards transparency for the fake image detection area and reveals the need for greater emphasis on forensic elements in visual-language research and AI risk control. FakeBench will be available athttps://github.com/Yixuan423/FakeBench

Despite substantial efforts dedicated to the design of heuristic models for omnidirectional (i.e., 360°) image quality assessment (OIQA), a conspicuous gap remains due to the lack of consideration for the diversity of viewing behaviors that leads to the varying perceptual quality of 360° images. Two critical aspects underline this oversight: the neglect of viewing conditions that significantly sway user gaze patterns and the overreliance on a single viewport sequence from the 360° image for quality inference. To address these issues, we introduce a unique generative scanpath representation (GSR) for effective quality inference of 360° images, which aggregates varied perceptual experiences of multi-hypothesis users under a predefined viewing condition. More specifically, given a viewing condition characterized by the starting point of viewing and exploration time, a set of scanpaths consisting of dynamic visual fixations can be produced using an apt scanpath generator. Following this vein, we use the scanpaths to convert the 360° image into the unique GSR, which provides a global overview of gazed-focused contents derived from scanpaths. As such, the quality inference of the 360° image is swiftly transformed to that of GSR. We then propose an efficient OIQA computational framework by learning the quality maps of GSR. Comprehensive experimental results validate that the predictions of the proposed framework are highly consistent with human perception in the spatiotemporal domain, especially in the challenging context of locally distorted 360° images under varied viewing conditions. The code will be released at https://github.com/xiangjieSui/GSR.

Omnidirectional image quality assessment (OIQA) has been one of the hot topics in IQA with the continuous development of VR techniques, and achieved much success in the past few years. However, most studies devote themselves to the uniform distortion issue, i.e., all regions of an omnidirectional image are perturbed by the “same amount” of noise, while ignoring the non-uniform distortion issue, i.e., partial regions undergo “different amount” of perturbation with the other regions in the same omnidirectional image. Additionally, nearly all OIQA models are verified on the platforms containing a limited number of samples, which largely increases the over-fitting risk and therefore impedes the development of OIQA. To alleviate these issues, we elaborately explore this topic from both subjective and objective perspectives. Specifically, we construct a large OIQA database containing 10,320 non-uniformly distorted omnidirectional images, each of which is generated by considering quality impairments on one or two camera len(s). Then we meticulously conduct psychophysical experiments and delve into the influence of both holistic and individual factors (i.e., distortion range and viewing condition) on omnidirectional image quality. Furthermore, we propose a perception-guided OIQA model for non-uniform distortion by adaptively simulating users' viewing behavior. Experimental results demonstrate that the proposed model outperforms state-of-the-art methods.

With the explosive growth of multimedia streaming services and virtual reality devices, omnidirectional video (ODV) is becoming increasingly popular in practical applications. However, streaming the entire ODV with high definition and high frame rate induces a waste of bandwidth. The tile-based viewport adaptive streaming provides a solution to overcome volatile network conditions, while the scheme would lead to quality adaptation when the network changes dynamically. In this paper, we focus on investigating how the human visual quality of experience (QoE) changes with time-varying ODV quality. Specifically, we construct a new quality of experience database for viewport adaptive ODV streaming named JUFEOVQoE, which includes twelve original ODVs with diverse content, and corresponding 378 viewport videos generated by compressing the raw viewport videos using a variety of combinations of quantization parameter (QP), spatial (S), and temporal resolutions (T). We conduct a series of subjective experiments to collect the mean opinion scores of the viewport video sequences and the viewing direction data of the subjects. Furthermore, we test several state-of-the-art objective QoE models on the proposed database. Experimental results demonstrate that existing mainstream QoE methods cannot predict the QoE of the viewport adaptive streaming ODVs accurately. The database will be released to facilitate further research.

With the development of virtual reality (VR) technology, panoramic image (PI), which is an important digital form of immersive multimedia, has drawn much attention from researchers. However, distortions are inevitably introduced in the process of processing, encoding and compression, which damages their quality and affects the user’s experience. Therefore, assessing the quality of panoramic images is urgent. In this paper, with the consideration of viewing behavior, we propose a novel blind panoramic image quality assessment model, which consists of three parts, viewport generation, feature extraction and quality prediction. Specifically, inspired by the viewing process of PI, we first generate multiple viewport sequences according to the real viewing trajectory and then extract multilevel features with a pre-trained backbone. The concatenated features are taken as the input of a recurrent neural network to evaluate the perceptual quality of PI. To validate the effectiveness of the proposed method, objective experiments are conducted on the public subjective panoramic image quality database. Experimental results demonstrate that the proposed method outperforms state-of-the-art methods.

With the rapid development of multimedia processing and deep learning technologies, especially in the field of video understanding, video quality assessment (VQA) has achieved significant progress. Although researchers have moved from designing efficient video quality mapping models to various research directions, in-depth exploration of the effectiveness-efficiency trade-offs of spatio-temporal modeling in VQA models is still less sufficient. Considering the fact that videos have highly redundant information, this paper investigates this problem from the perspective of joint spatial and temporal sampling, aiming to seek the answer to how little information we should keep at least when feeding videos into the VQA models while with acceptable performance sacrifice. To this end, we drastically sample the video’s information from both spatial and temporal dimensions, and the heavily squeezed video is then fed into a stable VQA model. Comprehensive experiments regarding joint spatial and temporal sampling are conducted on six public video quality databases, and the results demonstrate the acceptable performance of the VQA model when throwing away most of the video information. Furthermore, with the proposed joint spatial and temporal sampling strategy, we make an initial attempt to design an online VQA model, which is instantiated by as simple as possible a spatial feature extractor, a temporal feature fusion module, and a global quality regression module. Through quantitative and qualitative experiments, we verify the feasibility of online VQA model by simplifying itself and reducing input.

While abundant research has been conducted on improving high-level visual understanding and reasoning capabilities of large multimodal models (LMMs), their image quality assessment (IQA) ability has been relatively under-explored. Here we take initial steps towards this goal by employing the two-alternative forced choice (2AFC) prompting, as 2AFC is widely regarded as the most reliable way of collecting human opinions of visual quality. Subsequently, the global quality score of each image estimated by a particular LMM can be efficiently aggregated using the maximum a posteriori estimation. Meanwhile, we introduce three evaluation criteria: consistency, accuracy, and correlation, to provide comprehensive quantifications and deeper insights into the IQA capability of five LMMs. Extensive experiments show that existing LMMs exhibit remarkable IQA ability on coarse-grained quality comparison, but there is room for improvement on fine-grained quality discrimination. The proposed dataset sheds light on the future development of IQA models based on LMMs. The codes will be made publicly available athttps://github.com/h4nwei/2AFC-LMMs.

This paper aims to address the issue of fixed-time consensus control for uncertain nonlinear multiagent systems (MASs), in which only a group of followers can directly access the leader's information. To ensure the minimum utilization of wireless channel without sacrificing system performance, a fixedtime event-triggered consensus scheme is developed and employed. Firstly, the fixed-time observer with the event-triggered mechanism is proposed to estimate the states of the leader for each follower, which eliminates the unexpected Zeno behavior. Secondly, based on the reconstructed leader's information, a fuzzy fixed-time controller via memory based learning is proposed, in which the fuzzy logic system (FLS) technology is utilized to handle the uncertainty in the MASs. Unlike most FLS based results, the historical memory instead of the single data point is utilized to update the FLS, which improves the learning ability of FLS. Moreover, in the proposed distributed controller, only one FLS parameter is required to be updated for each norder uncertain follower, effectively improving the computational efficiency. With the help of graph theory and Lyapunov stability theory, the fixed-time stability for the entire system is derived. Finally, the validity of the proposed control scheme is illustrated through simulation examples.

Compared with traditional 2D images, omnidirectional images (also referred to as 360 ∘ images) have more complicated perceptual characteristics due to the particularities of imaging and display. How humans perceive omnidirectional images in an immersive environment and form the immersive quality of experience are important problems. Thus, it is crucial to measure the quality of omnidirectional images under different viewing conditions, which suffer from realistic distortions. In this article, we build a large-scale subjective assessment database for omnidirectional images and carry out a comprehensive psychophysical experiment to study the relationships between different factors (viewing conditions and viewing behaviors) and the perceptual quality of omnidirectional images. In addition, we collect both subjective ratings and head movement data. A thorough analysis of the collected subjective data is also provided, where we make several interesting findings. Moreover, with the proposed database, we propose a novel transformer-based omnidirectional image quality assessment model. To be consistent with the human viewing process, viewing conditions and behaviors are naturally incorporated into the proposed model. Specifically, the proposed model mainly consists of three parts: viewport sequence generation, multi-scale feature extraction, and perceptual quality prediction. Extensive experimental results conducted on the proposed database demonstrate the effectiveness of the proposed method over existing image quality assessment methods.

With the popularity of virtual reality applications in daily life, a large number of users begin to acquaint themselves with panoramic videos. However, the demand for high bandwidth and low latency brings challenges to the transmission of panoramic videos. Although the tile-based viewport adaptive streaming scheme can mitigate this problem and improve bandwidth utilization, the sudden viewport switching of the users in an immersive environment will result in high transmission delay and video playback stall, which reduces the user’s quality of experience. Prefetching the content in panoramic videos that users most likely watch to an edge server can further reduce transmission latency. However, due to the limitation of edge server cache capacity, better caching strategies need to be proposed. In this paper, we propose an edge collaborative cache scheme, which performs cache and replacement decisions to improve cache utilization and minimize transmission delay and cost. Moreover, adjacent multiple mobile edge computing (MEC) servers form a collaboration domain and share the cache capacity. The proposed strategy determines the proactive caching content and selects the best cache server by taking the cache benefit and replacement cost of the whole cooperation domain into account. Experimental results show that compared with FIFO and LRU, the proposed cache replacement strategy has a higher cache hit ratio and viewport quality, and lower stall times, which can bring a good user experience.

Omnidirectional images, also called 360◦images, have attracted extensive attention in recent years, due to the rapid development of virtual reality (VR) technologies. During omnidirectional image processing including capture, transmission, consumption, and so on, measuring the perceptual quality of omnidirectional images is highly desired, since it plays a great role in guaranteeing the immersive quality of experience (IQoE). In this paper, we conduct a comprehensive study on the perceptual quality of omnidirectional images from both subjective and objective perspectives. Specifically, we construct the largest so far subjective omnidirectional image quality database, where we consider several key influential elements, i.e., realistic non-uniform distortion, viewing condition, and viewing behavior, from the user view. In addition to subjective quality scores, we also record head and eye movement data. Besides, we make the first attempt by using the proposed database to train a convolutional neural network (CNN) for blind omnidirectional image quality assessment. To be consistent with the human viewing behavior in the VR device, we extract viewports from each omnidirectional image and incorporate the user viewing conditions naturally in the proposed model. The proposed model is composed of two parts, including a multi-scale CNN-based feature extraction module and a perceptual quality prediction module. The feature extraction module is used to incorporate the multi-scale features, and the perceptual quality prediction module is designed to regress them to perceived quality scores. The experimental results on our database verify that the proposed model achieves the competing performance compared with the state-of-the-art methods.

Depth image super-resolution (DISR) is one of the hot topics in computer vision. Although great progress has been made in this research topic, the robustness of DISR models is not sufficiently investigated, which is of great importance in the real applications. Accordingly, in this paper, we make an initial attempt to investigate the robustness of DISR models. Specifically, we test their generalization ability when the input depth image suffers from visual quality degradation. To facilitate this study, we construct a large-scale depth image dataset in which the reference depth images are perturbed to generate the degraded depth images automatically. Then, we test six top-performing DISR models on the constructed dataset and then compare their strengths and weaknesses. By conducting comprehensive experiments, we find that depth image super-resolution models perform poorly on Gaussian noise, and that the higher the level, the lower the quality of the predicted depth map. Furthermore, some DISR models only outperform at lower magnifications (such as 2x and 4x).

Models for image quality assessment (IQA) are generally optimized and tested by comparing to human ratings, which are expensive to obtain. Here, we develop a blind IQA (BIQA) model, and a method of training it without human ratings. We first generate a large number of corrupted image pairs, and use a set of existing IQA models to identify which image of each pair has higher quality. We then train a convolutional neural network to estimate perceived image quality along with the uncertainty, optimizing for consistency with the binary labels. The reliability of each IQA annotator is also estimated during training. Experiments demonstrate that our model outperforms state-of-the-art BIQA models in terms of correlation with human ratings in existing databases, as well in group maximum differentiation (gMAD) competition.