Overdraw is inevitable in large-scale scatterplots. Current scatterplot abstraction methods lose features in medium-to-low density regions. We propose a visual abstraction method designed to provide better feature preservation across arbitrary abstraction levels for large-scale scatterplots, particularly in medium-to-low density regions. The method consists of three closely interconnected steps: first, we partition the scatterplot into iso-density regions and equalize visual density; then, we allocate pixels for different classes within each region; finally, we reconstruct the data distribution based on pixels. User studies, quantitative and qualitative evaluations demonstrate that, compared to previous methods, our approach better preserves features and exhibits a special advantage when handling ultra-high dynamic range data distributions.

Recovering spatial-varying bi-directional reflectance distribution function (SVBRDF) from as few as possible captured images has been a challenging task in computer graphics. Benefiting from the co-located flashlight-camera capture strategy and data-driven priors, SVBRDF can be estimated from few input images. However, this capture strategy usually requires a controllable darkroom environment, ensuring the flashlight is a single light source. It is often impractical during on-site capture in real-world scenarios. To support SVBRDF estimation in an uncontrolled environment, the key challenge lies in the high-precise estimation of unknown environment lighting and its effective utilization on SVBRDF recovery. To address this issue, we proposed a novel exemplar-based environment lighting representation, which is easier to use for neural networks. These exemplars are a set of rendered images of selected materials under the environment lighting. By embedding the rendering process, our approach transforms environment lighting represented in the spherical domain into the sample-surface domain, thereby achieving the domain alignment with input images. This significantly reduces the network’s learning burden, resulting in a more precise environment lighting estimation. Furthermore, after lighting prediction, we also present a dominant lighting extraction algorithm and an adaptive exemplar selection algorithm to enhance the guidance of environment lighting in SVBRDF estimation. Finally, considering the distant contribution of environment lighting and point lighting to SVBRDF recovery, we proposed a well-designed cascaded network. Quantitative assessments and qualitative analysis have demonstrated that our method achieves superior SVBRDF estimations compared to previous approaches. The source code will be released.

Transparent object reconstruction in an uncontrolled natural scene is a challenging task due to its complex appearance. Existing methods optimize the object shape with RGB color as supervision, which suffer from locality and ambiguity, and fail to recover accurate structures. In this paper, we present RCTrans, which uses ray-background intersection as a more efficient constraint to achieve high-quality reconstruction, while maintaining a convenient setup. The key technology to achieve this is a novel pre-trained correspondence estimation network, which allows us to acquire ray-background correspondence under uncontrolled scenes and camera views. In addition, a confidence evaluation is introduced to protect the reconstruction from inaccurate estimated correspondence. Extensive experiments on both synthetic and real data demonstrate that our method can produce highly accurate results, without any extra acquisition burden. The code and dataset will be publicly available.

Blind image inpainting is a challenging task aimed at reconstructing corrupted regions without relying on mask information. Due to the lack of mask priors, previous methods usually integrate a mask prediction network in the initial phase, followed by an inpainting backbone. However, this multi-stage generation process may result in feature misalignment. While recent end-to-end generative methods bypass the mask prediction step, they typically struggle with weak perception of contaminated regions and introduce structural distortions. This study presents a novel mask region perception strategy for blind image inpainting by combining adversarial training with forgery detection. To implement this strategy, we propose an attention-driven forgery adversarial network (AFAN), which leverages adaptive contextual attention (ACA) blocks for effective feature modulation. Specifically, within the generator, ACA employs self-attention to enhance content reconstruction by utilizing the rich contextual information of adjacent tokens. In the discriminator, ACA utilizes cross-attention with noise priors to guide adversarial learning for forgery detection. Moreover, we design a high-frequency omni-dimensional dynamic convolution (HODC) based on edge feature enhancement to improve detail representation. Extensive evaluations across multiple datasets demonstrate that the proposed AFAN model outperforms existing generative methods in blind image inpainting, particularly in terms of quality and texture fidelity.

Reflectance acquisition from sparse images has been a long-standing problem in computer graphics. Previous works have addressed this by introducing either material-related priors or illumination multiplexing with a general sampling strategy. However, fixed lighting patterns in multiplexing can lead to redundant sampling and entangled observations, making it necessary to adaptively capture salient reflectance responses in each shot based on material behavior. In this paper, we propose combining adaptive sampling with illumination multiplexing for SVBRDF reconstruction from sparse images lit by a planar light source. Central to our method is the modeling of a sampling importance distribution on lighting surface, guided by the statistical nature of microfacet theory. Based on this sampling structure, our framework jointly trains networks to learn an adaptive sampling strategy in the lighting domain, and furthermore, approximately separates pure specular-related information from observations to reduce ambiguities in reconstruction. We validate our approach through experiments and comparisons with previous works on both synthetic and real materials.

Obtaining high-quality labeled training data poses a significant bottleneck in the domain of machine learning. Data programming has emerged as a new paradigm to address this issue by converting human knowledge into labeling functions (LFs) to quickly produce low-cost probabilistic labels. To ensure the quality of labeled data, data programmers commonly iterate LFs for many rounds until satisfactory performance is achieved. However, the challenge in understanding the labeling iterations stems from interpreting the intricate relationships between data programming elements, exacerbated by their many-to-many and directed characteristics, inconsistent formats, and the large scale of data typically involved in labeling tasks. These complexities may impede the evaluation of label quality, identification of areas for improvement, and the effective optimization of LFs for acquiring high-quality labeled data. In this article, we introduce EvoVis, a visual analytics method for multi-class text labeling tasks. It seamlessly integrates relationship analysis and temporal overview to display contextual and historical information on a single screen, aiding in explaining the labeling iterations in data programming. We assessed its utility and effectiveness through case studies and user studies. The results indicate that EvoVis can effectively assist data programmers in understanding labeling iterations and improving the quality of labeled data, as evidenced by an increase of 0.16 in the average F1 score when compared to the default analysis tool.

This January 2025 issue of the IEEE Transactions on Visualization and Computer Graphics (TVCG) contains the proceedings of IEEE VIS 2024, held on October 1318 October, 2024 in St. Pete Beach, Florida, USA, with the three General Chairs Paul Rosen (University of Utah), Kristi Potter (U.S. National Renewable Energy Laboratory), and Remco Chang (Tufts University). With IEEE VIS 2024, the conference series is in its 35th year.

Current blockchain payment approaches face practical application challenges due to high decryption complexity, extended ciphertext lengths, and reliance on semi-honest operational models. To address these issues, this paper presents a batch payment scheme with denomination privacy using the Castagnos-Laguillaumie (CL) homomorphic encryption and batch zeroknowledge proofs. Our scheme reduces decryption complexity and ciphertext length while enabling malicious model operations. It supports efficient batch payments to multiple recipients and includes features for payment statistics. Experimental results show an average encryption and decryption time of 0.89 ms and 1.55 ms, respectively, confirming its practicality and speed.

Face normal estimation is a crucial step in the development of 3D facial applications, particularly for face modeling and relighting. U-shaped networks are widely used for the task and have witnessed remarkable success. However, CNN-based methods often suffer from unsatisfied generalization ability to out-of-distribution/unseen data, because they do not adequately model long-range dependencies. To address this limitation, Transformer-based approaches have been developed, which benefit from the global self-attention mechanism. Nevertheless, merely using them to learn face normal may lead to limited localization abilities due to insufficient low-level details. In this work, we customize a hybrid model called FNFormer that combines Transformer and CNN to achieve accurate face normal estimation. The proposed model encodes tokenized image patches from CNN feature maps as input to extract global context features using Transformer blocks. Additionally, it extracts detailed local spatial information from a U-shaped CNN. Both the CNN and Transformer features are then integrated for further learning, enabling the network to take both the local and global information into account effectively. Extensive experimental results demonstrate that our proposed FNFormer achieves state-of-the-art performance on various datasets. Our code is available at https://github.com/AutoHDR/FNFormer.

Recovering spatial-varying bi-directional reflectance distribution function (SVBRDF) from a few hand-held captured images has been a challenging task in computer graphics. Benefiting from the learned priors from data, single-image methods can obtain plausible SVBRDF estimation results. However, the extremely limited appearance information in a single image does not suffice for high-quality SVBRDF reconstruction. Although increasing the number of inputs can improve the reconstruction quality, it also affects the efficiency of real data capture and adds significant computational burdens. Therefore, the key challenge is to minimize the required number of inputs, while keeping high-quality results. To address this, we propose maximizing the effective information in each input through a novel co-located capture strategy that combines near-field and far-field point lighting. To further enhance effectiveness, we theoretically investigate the inherent relation between two images. The extracted relation is strongly correlated with the slope of specular reflectance, substantially enhancing the precision of roughness map estimation. Additionally, we designed the registration and denoising modules to meet the practical requirements of hand-held capture. Quantitative assessments and qualitative analysis have demonstrated that our method achieves superior SVBRDF estimations compared to previous approaches. All source codes will be publicly released.

Reconstructing the geometry of transparent objects has been a long-standing challenge. Existing methods rely on complex setups, such as manual annotation or darkroom conditions, to obtain object silhouettes and usually require controlled environments with designed patterns to infer ray-background correspondence. However, these intricate arrangements limit the practical application for common users. In this paper, we significantly simplify the setups and present a novel method that reconstructs transparent objects in unknown natural scenes without manual assistance. Our method incorporates two key technologies. Firstly, we introduce a volume rendering-based method that estimates object silhouettes by projecting the 3D neural field onto 2D images. This automated process yields highly accurate multi-view object silhouettes from images captured in natural scenes. Secondly, we propose transparent object optimization through differentiable refraction rendering with the neural SDF field, enabling us to optimize the refraction ray based on color rather than explicit ray-background correspondence. Additionally, our optimization includes a ray sampling method to supervise the object silhouette at a low computational cost. Extensive experiments and comparisons demonstrate that our method produces high-quality results while offering much more convenient setups.

Recovering spatial-varying bi-directional reflectance distribution function (SVBRDF) from a single hand-held captured image has been a meaningful but challenging task in computer graphics. Benefiting from the learned data priors, some previous methods can utilize the potential material correlations between image pixels to serve for SVBRDF estimation. To further reduce the ambiguity from single-image estimation, it is necessary to integrate additional explicit material correlations. Given the flexible expressive ability of basis material assumption, we propose DeepBasis, a deep-learning-based method integrated with this assumption. It jointly predicts basis materials and their blending weights. Then the estimated SVBRDF is their linear combination. To facilitate the extraction of data priors, we introduce a two-level basis model to keep the sufficient representative while using a fixed number of basis materials. Moreover, considering the absence of ground-truth basis materials and weights during network training, we propose a variance-consistency loss and adopt a joint prediction strategy, thereby enabling the existing SVBRDF dataset available for training. Additionally, due to the hand-held capture setting, the exact lighting directions are unknown. We model the lighting direction estimation as a sampling problem and propose an optimization-based algorithm to find the optimal estimation. Quantitative evaluation and qualitative analysis demonstrate that DeepBasis can produce a higher quality SVBRDF estimation than previous methods. All source codes will be publicly released.

We present a lightweight and efficient rain sound synthesis method for interactive virtual environments. Existing rain sound simulation methods require massive superposition of scene-specific precomputed rain sounds, which is excessive memory consumption for virtual reality systems (e.g. video games) with limited audio memory budgets. Facing this issue, we reduce the audio memory budgets by introducing a lightweight rain sound synthesis method which is only based on eight physically-inspired basic rain sounds. First, in order to generate sufficiently various rain sounds with limited sound data, we propose an exponential moving average based frequency domain additive (FDA) synthesis method to extend and modify the pre-computed basic rain sounds. Each rain sound is generated in the frequency domain before conversion back to the time domain, allowing us to extend the rain sound which is free of temporal distortions and discontinuities. Next, we introduce an efficient binaural rendering method to simulate the 3D perception that coheres with the visual scene based on a set of Near-Field Transfer Functions (NFTF). Various results demonstrate that the proposed method drastically decreases the memory cost (77 times compressed) and overcomes the limitations of existing methods in terms of interaction.

Abstract Multi-exposure image fusion has emerged as an increasingly important and interesting research topic in information fusion. It aims at producing an image with high quality by fusing a set of differently exposed images. In this article, we present a pixel-level method for multi-exposure image fusion based on an information-theoretic approach. In our scheme, an information channel between two source images is used to compute the Rényi entropy associated with each pixel in one image with respect to the other image and hence to produce the weight maps for the source images. Since direct weight-averaging of the source images introduce unpleasing artifacts, we employ Laplacian multi-scale fusion. Based on this pyramid scheme, images at every scale are fused by weight maps, and a final fused image is inversely reconstructed. Multi-exposure image fusion with the proposed method is easy to construct and implement and can deliver, in less than a second for a set of three input images of size 512$\times $340, competitive and compelling results versus state-of-art methods through visual comparison and objective evaluation.

Blockchain technology is increasingly utilized for data asset transactions, particularly for private data that should not be shared with unauthorized parties. To ensure auditability and traceability of transaction records and data, we propose a triple-receiver public key encryption scheme based on bilinear mapping. This scheme serves as a regulated mechanism for data asset transactions and is integrated into a blockchain-based platform with a digital asset exchange protocol. In our approach, the sender encrypts a message using its private key and the public keys of three designated receivers. Each receiver can then independently decrypt the message using their private key. We demonstrate that our scheme is secure against chosen-plaintext and adaptive chosen-ciphertext attacks, provided that the gap bilinear Diffie-Hellman problem and the computational Diffie-Hellman problem are computationally infeasible. In comparison to Diameter’s dual-receiver encryption scheme, our method offers similar decryption speeds. Additionally, we incorporate the significant feature that allows both the sender and all three receivers to decrypt independently. Performance evaluations indicate that our scheme is well-suited for data asset transactions in the Internet of Things (IoT), offering decentralized and trust management capabilities.

Audio-to-score alignment is a multi-modal task that aims at generating an accurate mapping between symbolic and signal-level representations of musical signals, which is important for music performance analysis and retrieval. Among numerous music genres, orchestral music is a category of music with complex performance characteristics such as multi-instrument, non-percussive instrument and music expressiveness. However, previous methods do not take sufficient account of the performance characteristics of orchestral music, leading to limitations in alignment accuracy on orchestral music of these methods. To solve this problem, we present a performance quantification guided audio-to-score alignment (PQG-A2SA) method with high alignment accuracy for orchestral music at note-level. Specially, the PQG-A2SA contains two parts, namely an Inter Onset Interval (IOI) guided conditionally-constrained Dynamic Time Wrapping (DTW) and an articulation guided onset and offset detection. Different from the previous work, the IOI-guided conditionally-constrained DTW is designed to achieve a preliminary mapping between symbolic and chord-level representations of musical signals. In the second module, the onset and offset detection model under different musical articulations are established, thus refining the alignment results. We provide extensive experimental validation and analysis of our method. Our PQG-A2SA method can improve 9.0% in onset align rate and 17.5% in offset align rate at most compared with the state-of-the-art methods.

Video summarization is one of the critical techniques in video retrieval, video browsing, and management. It is still a challenging research task due to user subjectivity, excessive redundant information, and lack of spatio-temporal dependency. In this paper, we propose an unsupervised video summarization approach via reinforcement learning with shot-level semantics. The primary idea of this unsupervised method is based on the encoder-decoder model. We use a novel field size dataset to train a convolutional neural network as an encoder to extract the convolutional feature matrix from the video. Then, a bidirectional LSTM is utilized as a decoder to obtain probability weights for selecting keyframes, which preserves the spatio-temporal dependence of video summarization. Specifically, to reduce the influence of user subjectivity, we design a shot-level semantic reward function to generate more representative summarization results. The shot-level semantics are the rules followed by the video shooting process without being changed by the preferences of different viewers. Finally, we evaluate our approach on four classical datasets, SumMe, TVSum, CoSum, and VTW. The results suggest that our algorithm outperforms others and achieves satisfactory results.

Shot boundary detection (SBD) is widely used in scene segmentation, semantic analysis, and video retrieval. However, existing SBD algorithms have certain applications in video processing, but they have the following three problems. First, these algorithms cannot effectively handle shot boundaries caused by sudden lighting changes. Second, when there are dimly lighting frames in the video, these algorithms cannot perform boundary detection well. Third, when there is object or camera motion in the video, these algorithms also fail to work. To resolve these issues, we propose an SBD algorithm with color clustering changes in small regions (CCSR) to detect the shot transitions, which are abrupt changes and gradual transitions (dissolve and fade). The main idea behind the CCSR algorithm is to compute the distance of color features and to preserve the spatio-temporal information as much as possible. This model has relatively less dependence on the threshold parameters and sliding windows. Unlike other SBD algorithms, the clustering results of CCSR weaken factors such as object motion and illumination changes between adjacent frames in the video, which is helpful for reducing false detections. Furthermore, we utilize an attention mechanism in the gradual transitions to improve detection efficiency and accuracy. Finally, we evaluated the SBD algorithm, which was tested on a standard TRECVID dataset. The experimental results suggest that our algorithm yields significant improvements in precision and recall compared to the current techniques, with an average improvement of 10.35% and 8.85%, respectively. Moreover, compared with state-of-the-art algorithms, the results prove that the proposed method improves the F-score by more than 2.64% and the computation time efficiency by over 10%.

The overdraw problem of scatterplots seriously interferes with the visual tasks. Existing methods, such as data sampling, node dispersion, subspace mapping, and visual abstraction, cannot guarantee the correspondence and consistency between the data points that reflect the intrinsic original data distribution and the corresponding visual units that reveal the presented data distribution, thus failing to obtain an overlap-free scatterplot with unbiased and lossless data distribution. A dual space coupling model is proposed in this paper to represent the complex bilateral relationship between data space and visual space theoretically and analytically. Under the guidance of the model, an overlap-free scatterplot method is developed through integration of the following: a geometry-based data transformation algorithm, namely DistributionTranscriptor; an efficient spatial mutual exclusion guided view transformation algorithm, namely PolarPacking; an overlap-free oriented visual encoding configuration model and a radius adjustment tool, namelyfrdraw. Our method can ensure complete and accurate information transfer between the two spaces, maintaining consistency between the newly created scatterplot and the original data distribution on global and local features. Quantitative evaluation proves our remarkable progress on computational efficiency compared with the state-of-the-art methods. Three applications involving pattern enhancement, interaction improvement, and overdraw mitigation of trajectory visualization demonstrate the broad prospects of our method.

Existing image-to-image translation methods can effectively deal with simple scenes or styles, such as horses-to-zebra, cat-to-dog, and summer-to-winter. However, the performance of night-to-day (N2D) translation remains unsatisfied due to imbalanced/poor visibility and thus translation ambiguity, although some progress has been made by GAN-based methods recently. This paper proposes a CycleGAN-based N2D translation scheme, namely N2D-GAN, in a weakly-supervised manner with consideration of both image and semantic information. Specifically, we first adjust the brightness of night-time images to boost the visibility, so that the generator can better extract content information. Then, the generator processes the translation by enforcing results to follow the distribution of daytime images in both image and semantic domains. Besides, the cycle consistency is introduced to preserve the fidelity between translations from two directions. Experimental results demonstrate that our strategy outperforms other state-of-the-art N2D methods both quantitatively and qualitatively.

Face inverse rendering, an important and challenging task in computer vision and computer graphics, attempts to decompose face image into shape, reflectance, and illuminance. This problem becomes fundamentally difficult under non-laboratory conditions without controlled illumination. Though recent works have produced compelling results, most of these techniques rely on multiple lighting images captured under contronlled lighting by complex equipment, such as Light Stage, which is not flexible and applicable to common users. In this paper, we propose a novel face inverse rendering framework, which neither relies on complex devices nor labeled training data. Instead, it learns reflectance, shape, and illuminance from its physical constraints. Extensive experiments on both synthetic and real image datasets demonstrate consistently superior performance of the proposed method. Our code will be made publicly available.

When viewing the surrounding environment, the human eyes perform different eye movements that reflect human's interest and attention. A large amount of research efforts have focused on Areas of Interest (AOIs)-based transition analysis of eye movement. However, to the best of our knowledge, there is little work on fixation duration from eye tracking data. In this paper, we investigate entropy based eye movement transition analysis of fixation duration. Specifically, we divide the fixation duration into three categories - express fixations (called short duration, SD), cognitive fixations (called medium duration, MD), and very long fixations (called long duration, LD), and model the eye tracking sequences of fixation duration as a Markov chain, which is further considered from the perspective of a discrete information channel. Subsequently, we exploit the information channel paradigm by computing the entropy measures from the fixation duration information channel to explore the visual behaviour. The proposed method, that allows a straightforward clustering of the results, is demonstrated on two eye tracking data sets, Van Gogh paintings and scientific posters, and shows that the differences between observers are more pronounced that the differences between the observed images. The findings from this work might also provide future insights to quantitative assess human visual cognitive process.

While existing face normal estimation methods have produced promising results on small datasets, they often suffer from severe performance degradation on diverse in-the-wild face images, especially for the high-fidelity face normal estimation. Training a high-fidelity face normal estimation model with generalization capability requires a large amount of training data with face normal ground truth. Since collecting such high-fidelity database is difficult in practice, which prevents current methods from recovering face normal with fine-grained geometric details. To mitigate this issue, we propose a coarse-to-fine framework to estimate face normal from an in-the-wild image with only a coarse exemplar reference. Specifically, we first train a model using limited training data to exploit the coarse normal of a real face image. Then, we leverage the estimated coarse normal as an exemplar and devise an exemplar-based normal estimation network to explore robust mapping from the input face image to the fine-grained normal. In this manner, our method can largely alleviate the negative impact caused by lacking training data, and focus on exploring the high-fidelity normal contained in natural images. Extensive experiments and ablation studies are conducted to demonstrate the efficacy of our design, and reveal its superiority over state-of-the-art methods in terms of both training data requirement and recovery quality of fine-grained face normal. Our code is available at \urlhttps://github.com/AutoHDR/HFFNE.

BACKGROUND: Bioinformatics has gained much attention as a fast growing interdisciplinary field. Several attempts have been conducted to explore the field of bioinformatics by bibliometric analysis, however, such works did not elucidate the role of visualization in analysis, nor focus on the relationship between sub-topics of bioinformatics. RESULTS: First, the hotspot of bioinformatics has moderately shifted from traditional molecular biology to omics research, and the computational method has also shifted from mathematical model to data mining and machine learning. Second, DNA-related topics are bridge topics in bioinformatics research. These topics gradually connect various sub-topics that are relatively independent at first. Third, only a small part of topics we have obtained involves a number of computational methods, and the other topics focus more on biological aspects. Fourth, the proportion of computing-related topics hit a trough in the 1980s. During this period, the use of traditional calculation methods such as mathematical model declined in a large proportion while the new calculation methods such as machine learning have not been applied in a large scale. This proportion began to increase gradually after the 1990s. Fifth, although the proportion of computing-related topics is only slightly higher than the original, the connection between other topics and computing-related topics has become closer, which means the support of computational methods is becoming increasingly important for the research of bioinformatics. CONCLUSIONS: The results of our analysis imply that research on bioinformatics is becoming more diversified and the ranking of computational methods in bioinformatics research is also gradually improving.

Abstract Danmu (Danmaku) is a unique social media service in online videos, especially popular in Japan and China, for viewers to write comments while watching videos. The danmu comments are overlaid on the video screen and synchronized to the associated video time, indicating viewers' thoughts of the video clip. This paper introduces an interactive visualization system to analyze danmu comments and associated viewer behaviors in a collection of videos and enable detailed exploration of one video on demand. The watching behaviors of viewers are identified by comparing video time and post time of viewers' danmu. The system supports analyzing danmu content and viewers' behaviors against both video time and post time to gain insights into viewers' online participation and perceived experience. Our evaluations, including usage scenarios and user interviews, demonstrate the effectiveness and usability of our system.

Due to existing blockchain systems concentrate mainly on privacy protection but lack payment statistics and supervision, we propose a blockchain-based transaction system with payment statistics and supervision. In the system, a payer uses a homomorphic encryption scheme to protect payment amounts. After the transaction is recorded in the blockchain, not only the payee can decrypt the payment amounts and use for future payment, but also the payer can even decrypt it and use for payment statistics. Besides, two supervisors can independently decrypt all users' payment amounts to master the whole economic dynamism and detect illegal transactions. Comparing with existing schemes, our homomorphic scheme only increases a little length of ciphertext, but supports payment amounts decryption by the payer, an additional two receivers. Finally, analyses show that our system is extremely efficient.

Abstract In this work, we introduce our project on Dunhuang cultural heritage protection using artificial intelligence. The Dunhuang Mogao Grottoes in China, also known as the Grottoes of the Thousand Buddhas, is a religious and cultural heritage located on the Silk Road. The grottoes were built from the 4th century to the 14th century. After thousands of years, the in grottoes decaying is serious. In addition, numerous historical records were destroyed throughout the years, making it difficult for archaeologists to reconstruct history. We aim to use modern computer vision and machine learning technologies to solve such challenges. First, we propose to use deep networks to automatically perform the restoration. Through out experiments, we find the automated restoration can provide comparable quality as those manually restored from an archaeologist. This can significantly speed up the restoration given the enormous size of the historical paintings. Second, we propose to use detection and retrieval for further analyzing the tremendously large amount of objects because it is unreasonable to manually label and analyze them. Several state-of-the-art methods are rigorously tested and quantitatively compared in different criteria and categorically. In this work, we created a new dataset, namely, AI for Dunhuang, to facilitate the research. Version v1.0 of the dataset comprises of data and label for the restoration, style transfer, detection, and retrieval. Specifically, the dataset has 10,000 images for restoration, 3455 for style transfer, and 6147 for property retrieval. Lastly, we propose to use style transfer to link and analyze the styles over time, given that the grottoes were build over 1000 years by numerous artists. This enables the possibly to analyze and study the art styles over 1000 years and further enable future researches on cross-era style analysis. We benchmark representative methods and conduct a comparative study on the results for our solution. The dataset will be publicly available along with this paper.

Semantic segmentation requires both speed and accuracy. This paper presents a two-branch network BiAttnNet with a unique Bilateral Attention structure that separates all attention modules into the Detail Branch to contribute semantic detail selections for specialized detail exploring. Specifically, the Detail Branch comprises AttnTrans entirely, which provides a better alternate for regular convolution. AttnTrans is a computationally efficient filtration entirely composed of concurrent spatial and channel attention. Meanwhile, a Context Branch is implemented with FCN-ResNet for rough segmentation. By combining two branches’ outputs, BiAttnNet achieves a good balance between speed and accuracy. Evaluations on the Cityscapes testing set conclude that BiAttnNet achieves 74.7% mIoU at 89.2 FPS at a quarter (512 × 1024) resolution with only 2.2 million parameters, running on a single GTX 2080 Ti card.

Semantic segmentation has been one of the essential tasks in computer vision. More complicated and computationally intensive mechanisms are integrated into segmentation models to get more accurate results, leading to increased processing time and resource usage. Based on the idea that pixels with similar high-level features are more likely to have similar semantic labels, we propose a computationally efficient mechanism named Hierarchical Semantic Broadcasting (HSB), which can infer earlier-stage semantic label maps from lower-level feature maps by referring to semantic label maps of higher-level feature maps. Since lower-level feature maps have higher resolution and richer context, HSB can provide additional details for better semantic segmentation. By integrating HSB into a general-purpose light network, we propose Hierarchical Semantic Broadcasting Network (HSB-Net) for real-time semantic segmentation, which achieves a good trade-off between accuracy and speed. Evaluated by the Cityscapes dataset, HSB-Net can run at 123.7 FPS for a 512$ \boldsymbol{\times }$1024 input on a single GeForce RTX 2080 Ti card while achieving 73.1% mean IoU.

Previous face inverse rendering methods often require synthetic data with ground truth and/or professional equipment like a lighting stage. However, a model trained on synthetic data or using pre-defined lighting priors is typically unable to generalize well for real-world situations, due to the gap between synthetic data/lighting priors and real data. Furthermore, for common users, the professional equipment and skill make the task expensive and complex. In this paper, we propose a deep learning framework to disentangle face images in the wild into their corresponding albedo, normal, and lighting components. Specifically, a decomposition network is built with a hierarchical subdivision strategy, which takes image pairs captured from arbitrary viewpoints as input. In this way, our approach can greatly mitigate the pressure from data preparation, and significantly broaden the applicability of face inverse rendering. Extensive experiments are conducted to demonstrate the efficacy of our design, and show its superior performance in face relighting over other state-of-the-art alternatives. Our code is available at https://github.com/AutoHDR/HD-Net.git.

Zero-shot classification is a promising paradigm to solve an applicable problem when the training classes and test classes are disjoint. Achieving this usually needs experts to externalize their domain knowledge by manually specifying a class-attribute matrix to define which classes have which attributes. Designing a suitable class-attribute matrix is the key to the subsequent procedure, but this design process is tedious and trial-and-error with no guidance. This paper proposes a visual explainable active learning approach with its design and implementation called semantic navigator to solve the above problems. This approach promotes human-AI teaming with four actions (ask, explain, recommend, respond) in each interaction loop. The machine asks contrastive questions to guide humans in the thinking process of attributes. A novel visualization called semantic map explains the current status of the machine. Therefore analysts can better understand why the machine misclassifies objects. Moreover, the machine recommends the labels of classes for each attribute to ease the labeling burden. Finally, humans can steer the model by modifying the labels interactively, and the machine adjusts its recommendations. The visual explainable active learning approach improves humans' efficiency of building zero-shot classification models interactively, compared with the method without guidance. We justify our results with user studies using the standard benchmarks for zero-shot classification.

Guqin is a plucked seven-string traditional Chinese musical instrument that exists for over 3,000 years. However, as an Intangible World Cultural Heritage, the inheritance of Guqin and its culture in modern society is in deep danger. According to our study with 1,006 Chinese worldwide, Guqin as an instrument is not well-known and barely accessible. To better promote Guqin, we developed two interactive systems: VirGuqin and MRGuqin. VirGuqin was developed using a low-cost motion tracking device and was tested in a museum. 89% of 308 participants expressed an increase in interest in learning Guqin after using our system. MRGuqin was developed as a mixed reality learning environment to reduce the entry barrier to Guqin, and was tested by 16 participants, allowing them to learn Guqin significantly faster and perform better than the current practice. Our study demonstrates how technology can be used to help the inheritance of this dying art.

Recent learning-based image retargeting methods have achieved significant improvement. However, two main is-sues remain in this challenging task: (i) it is difficult to build ground truth datasets for supervised learning; (ii) most methods are based on a certain operator, not suitable for various images with different target sizes. In this paper, for the first time, we address these issues by providing a deep supervised image retargeting solution. We introduce a new dataset1of 6, 576 pairs generated by multiple operators using Image Re-targeting Quality Assessment (IRQA) algorithm. We then develop a mult-operator image retargeting model named MR-GAN, which learns the deformation process of retargeted images using multiple methods and conducts retargeting operations in feature space. Experimental results validate the effectiveness as well as its superiority against state-of-the-art alternatives of the proposed approach.

Chinese character inpainting is a challenging task where large missing regions have to be filled with both visually and semantic realistic contents. Existing methods generally produce pseudo or ambiguous characters due to lack of semantic information. Given the key observation that Chinese characters contain visually glyph representation and intrinsic contextual semantics, we tackle the challenge of similar Chinese characters by modeling the underlying regularities among glyph and semantic information. We propose a semantics enhanced generative framework for Chinese character inpainting, where a global semantic supervising module (GSSM) is introduced to constrain contextual semantics. In particular, sentence embedding is used to guide the encoding of continuous contextual characters. The method can not only generate realistic Chinese character, but also explicitly utilize context as reference during network training to eliminate ambiguity. The proposed method is evaluated on both handwritten and printed Chinese characters with various masks. The experiments show that the method successfully predicts missing character information without any mask input, and achieves significant sentence-level results benefiting from global semantic supervising in a wide variety of scenes.

Revealing the evolution of science and the intersections among its sub-fields is extremely important to understand the characteristics of disciplines, discover new topics, and predict the future. The current work focuses on either building the skeleton of science, lacking interaction, detailed exploration and interpretation or on the lower topic level, missing high-level macro-perspective. To fill this gap, we design and implement Galaxy Evolution Explorer (Galex), a hierarchical visual analysis system, in combination with advanced text mining technologies, that could help analysts to comprehend the evolution and intersection of one discipline rapidly. We divide Galex into three progressively fine-grained levels: discipline, area, and institution levels. The combination of interactions enables analysts to explore an arbitrary piece of history and an arbitrary part of the knowledge space of one discipline. Using a flexible spotlight component, analysts could freely select and quickly understand an exploration region. A tree metaphor allows analysts to perceive the expansion, decline, and intersection of topics intuitively. A synchronous spotlight interaction aids in comparing research contents among institutions easily. Three cases demonstrate the effectiveness of our system.

We present a comprehensive study and evaluation of existing single image deraining algorithms, using a new large-scale benchmark consisting of both synthetic and real-world rainy images.This dataset highlights diverse data sources and image contents, and is divided into three subsets (rain streak, rain drop, rain and mist), each serving different training or evaluation purposes. We further provide a rich variety of criteria for dehazing algorithm evaluation, ranging from full-reference metrics, to no-reference metrics, to subjective evaluation and the novel task-driven evaluation. Experiments on the dataset shed light on the comparisons and limitations of state-of-the-art deraining algorithms, and suggest promising future directions.

Images captured under low-light conditions often suffer from (partially) poor visibility. Besides unsatisfactory lightings, multiple types of degradations, such as noise and color distortion due to the limited quality of cameras, hide in the dark. In other words, solely turning up the brightness of dark regions will inevitably amplify hidden artifacts. This work builds a simple yet effective network for Kindling the Darkness (denoted as KinD), which, inspired by Retinex theory, decomposes images into two components. One component (illumination) is responsible for light adjustment, while the other (reflectance) for degradation removal. In such a way, the original space is decoupled into two smaller subspaces, expecting to be better regularized/learned. It is worth to note that our network is trained with paired images shot under different exposure conditions, instead of using any ground-truth reflectance and illumination information. Extensive experiments are conducted to demonstrate the efficacy of our design and its superiority over state-of-the-art alternatives. Our KinD is robust against severe visual defects, and user-friendly to arbitrarily adjust light levels. In addition, our model spends less than 50ms to process an image in VGA resolution on a 2080Ti GPU. All the above merits make our KinD attractive for practical use.

Role–event videos are rich in information but challenging to be understood at the story level. The social roles and behavior patterns of characters largely depend on the interactions among characters and the background events. Understanding them requires analysis of the video contents for a long duration, which is beyond the ability of current algorithms designed for analyzing short-time dynamics. In this paper, we propose InSocialNet, an interactive video analytics tool for analyzing the contents of role–event videos. It automatically and dynamically constructs social networks from role–event videos making use of face and expression recognition, and provides a visual interface for interactive analysis of video contents. Together with social network analysis at the back end, InSocialNet supports users to investigate characters, their relationships, social roles, factions, and events in the input video. We conduct case studies to demonstrate the effectiveness of InSocialNet in assisting the harvest of rich information from role–event videos. We believe the current prototype implementation can be extended to applications beyond movie analysis, e.g., social psychology experiments to help understand crowd social behaviors.

Groups of enterprises can guarantee each other and form complex networks in order to try to obtain loans from banks. Monitoring the financial status of a network, and preventing or reducing systematic risk in case of a crisis, is an area of great concern for the regulatory commission and for the banks. We set the ultimate goal of developing a visual analytic approach and tool for risk dissolving and decision-making. We have consolidated four main analysis tasks conducted by financial experts: i) Multi-faceted Default Risk Visualization, whereby a hybrid representation is devised to predict the default risk and an interface developed to visualize key indicators; ii) Risk Guarantee Patterns Discovery. We follow the Shneiderman mantra guidance for designing interactive visualization applications, whereby an interactive risk guarantee community detection and a motif detection based risk guarantee pattern discovery approach are described; iii) Network Evolution and Retrospective, whereby animation is used to help users to understand the guarantee dynamic; iv) Risk Communication Analysis. The temporal diffusion path analysis can be useful for the government and banks to monitor the spread of the default status. It also provides insight for taking precautionary measures to prevent and dissolve systematic financial risk. We implement the system with case studies using real-world bank loan data. Two financial experts are consulted to endorse the developed tool. To the best of our knowledge, this is the first visual analytics tool developed to explore networked-guarantee loan risks in a systematic manner.

Sketch is one of the most important art expression forms for traditional Chinese painting. This paper presents a complete sketch generation framework for ancient mural paintings. First, we propose a deep learning network to perform mural-to-sketch prediction by combining meaningful convolutional features in a holistic manner. A dedicated mural database with fine-grained ground truth is built for network training and testing. Then we design a style-aware image fusion approach by detecting the specific feature region in a mural, from which the artistic style can be maximally preserved. Experimental results have demonstrated its validity in extracting style mural sketch. This work has the potential to provide a computer aided tool for artists and restorers to imitate and restore time-honored paintings.

Structure-texture decomposition from an image (a.k.a. structure-preserving image smoothing) is important for a variety of multimedia, computer vision and graphics tasks. Its performance heavily depends on the precision of indicating where are structural edges to maintain and where are textures to remove. An intuitive thought for constructing indication is to directly execute edge detection on the input image, which however would suffer from rich textures. Feeding inaccurate or erroneous indications into the smoother is at high risk of generating unsatisfactory results. It is almost sure that edge detectors can do a better job on inputs with textures removed. The above two components, say the smoother and the indicator, turn out to be in a chicken-egg situation. To address this issue, we propose a method to jointly detect structural edges and remove textures, by iteratively smoothing the input based on the edges detected from the previous smoothed result and refining the edges based on the newly processed image. Experiments on a number of challenging cases are conducted to show that the edge detection task and the smoothing task can benefit from each other, and reveal the superiority of our method over other state-of-the-art alternatives. Our code is publicly available at https://sites.google.com/view/xjguo/sdts.

Image smoothing, which aims to remove unwanted textures and preserve desired structures, plays an important role in many multimedia and computer vision tasks. The key to image smoothing, despite different applications, is to distinguish the structures from the textures. This paper presents a novel image smoothing method, following the principle that, for a certain pixel, its neighbors in both space and intensity should contribute more on smoothing, while the distant ones be insulated for avoiding over-smoothing. Intuitively, clustering is a good candidate to achieve the goal. However, due to rich textures and clutters within images, simply performing the clustering on the input likely obtains inaccurate results, and thus leads to unsatisfied smoothing results. In addition, for our task, using traditional hard clustering techniques is at high risk of generating staircase artifacts. For addressing these issues, an algorithm is customized, which on the one hand adopts the soft clustering to more faithfully assign pixels, on the other hand iterates the soft clustering and smoothing, expecting to improve each other. Experiments on several challenging images are provided to show the efficacy of our method, and its superiority over other prevailing approaches.

Line drawing is a unique drawing technique developed over millennia in China. Since ancient murals have line drawings of beautiful form and vast history, it is incredibly important to digitally curate these pieces. In this paper, we propose a line drawing generation method named Mural2Sketch (MS) for ancient mural paintings. MS first utilizes heuristic routing to detect the outer edge of a stroke, and then high frequency enhancement filtering is used to extract the information inside the stroke. A complete stroke is then generated by collaborative representation. MS is capable of outputting the result in vector form and producing different artistic styles. Experimental results show that our method is simple but effective. This research has the potential to support digital mural copying, mural protection, as well as related cultural research and application.

Co-saliency detection is a newly emerging research topic in multimedia and computer vision, the goal of which is to extract common salient objects from multiple images. Effectively seeking the global consistency among multiple images is critical to the performance. To achieve the goal, this paper designs a novel model with consideration of a hierarchical consistency measure. Different from most existing co-saliency methods that only exploit common features (such as color and texture), this paper further utilizes the shape of object as another cue to evaluate the consistency among common salient objects. More specifically, for each involved image, an intra-image saliency map is firstly generated via a single image saliency detection algorithm. Having the intra-image map constructed, the consistency metrics at object level and superpixel level are designed to measure the corresponding relationship among multiple images and obtain the inter saliency result by considering multiple visual attention features and multiple constrains. Finally, the intra-image and inter-image saliency maps are fused to produce the final map. Experiments on benchmark datasets are conducted to demonstrate the effectiveness of our method, and reveal its advances over other state-of-the-art alternatives.

These days, superpixel algorithms are widely used in computer vision and multimedia applications. However, existing algorithms are designed for planar images, which are less suited to deal with wide angle images. In this paper, we present a superpixel segmentation method for 360° spherical images. Unlike previous methods, our approach explicitly considers the geometry for spherical images and makes clustering to spherical image pixels. It starts with the seeds defined by Hammersley points sampled on the sphere, then iterates between assignment step and update step, which are both based on the distance metric respecting spherical geometry. We evaluate our method on the transformed Berkeley segmentation dataset and panorama segmentation dataset collected by ourselves. Experimental results show that our method can gain better performance in terms of adherence to image boundaries and superpixel structural regularity. Furthermore, superpixels generated by our method can reserve the coherence across image boundaries and all have closed contours.

Low rank matrix approximation, in the presence of missing data and outliers, has previously shown its significance as a theoretic foundation in a wide spectrum of tabulated information processing applications. To fit low rank models, minimizing the nuclear norm of matrices is a popular scheme, the computational load of which, however, is heavy. While bilinear factorization can largely mitigate the computational complexity. Unfortunately, without a known or precisely estimated target rank, this strategy often performs vulnerably when the given data is dirty. This paper attempts to simultaneously achieve the computational efficiency as well as the robustness to mild rank initialization and gross corruptions. Moreover, several Augmented Lagrange Multiplier based solvers and a heuristic rank estimator are customized to seek the optimal solution. Theoretical analysis on convergence and complexity, and experiments on both synthetic and real data are provided to reveal the efficacy of our method and show its superiority over the state-of-the-art alternatives.

In this paper, we propose the first model to be able to generate visually grounded questions with diverse types for a single image. Visual question generation is an emerging topic which aims to ask questions in natural language based on visual input. To the best of our knowledge, it lacks automatic methods to generate meaningful questions with various types for the same visual input. To circumvent the problem, we propose a model that automatically generates visually grounded questions with varying types. Our model takes as input both images and the captions generated by a dense caption model, samples the most probable question types, and generates the questions in sequel. The experimental results on two real world datasets show that our model outperforms the strongest baseline in terms of both correctness and diversity with a wide margin.

The blocking artifact frequently appears in compressed real-world images or video sequences, especially coded at low bit rates, which is visually annoying and likely hurts the performance of many computer vision algorithms. A compressed frame can be viewed as the superimposition of an intrinsic layer and an artifact one. Recovering the two layers from such frames seems to be a severely ill-posed problem since the number of unknowns to recover is twice as many as the given measurements. In this paper, we propose a simple and robust method to separate these two layers, which exploits structural layer priors including the gradient sparsity of the intrinsic layer, and the independence of the gradient fields of the two layers. A novel Augmented Lagrangian Multiplier based algorithm is designed to efficiently and effectively solve the recovery problem. Experimental results demonstrate the efficacy of our method.

In this paper, we present a superpixel generation method for spherical images, which cover 360° field-of-view. Unlike previous works that directly use existing superpixel algorithms on unrolled spherical images, our approach explicitly considers the geometry for spherical images and uses sphere as the underlying representation. For quantitative evaluation, we make a spherical image segmentation database by transforming Berkeley segmentation dataset to the spherical domain. Experimental results show that our method can get better performance in terms of adherence to image boundaries and spherical size variance. What's more, superpixels generated by our method all have closed contours.

Social media data with geotags can be used to track people's movements in their daily lives. By providing both rich text and movement information, visual analysis on social media data can be both interesting and challenging. In contrast to traditional movement data, the sparseness and irregularity of social media data increase the difficulty of extracting movement patterns. To facilitate the understanding of people's movements, we present an interactive visual analytics system to support the exploration of sparsely sampled trajectory data from social media. We propose a heuristic model to reduce the uncertainty caused by the nature of social media data. In the proposed system, users can filter and select reliable data from each derived movement category, based on the guidance of uncertainty model and interactive selection tools. By iteratively analyzing filtered movements, users can explore the semantics of movements, including the transportation methods, frequent visiting sequences and keyword descriptions. We provide two cases to demonstrate how our system can help users to explore the movement patterns.

Spectral calculation and analysis have very important practical applications in astrophysics. The main portion of spectral calculation is to solve a large number of one-dimensional numerical integrations at each point of a large three-dimensional parameter space. However, existing widely used solutions still remain in process-level parallelism, which is not competent to tackle numerous compute-intensive small integral tasks. This paper presented a GPU-optimized approach to accelerate the numerical integration in massive spectral calculation. We also proposed a load balance strategy on hybrid multiple CPUs and GPUs architecture via share memory to maximize performance. The approach was prototyped and tested on the Astrophysical Plasma Emission Code (APEC), a commonly used spectral toolset. Comparing with the original serial version and the 24 CPU cores (2.5GHz) parallel version, our implementation on 3 Tesla C2075 GPUs achieves a speed-up of up to 300 and 22 respectively.

In this paper, we propose a general algorithm to removing haze from single images using total variation minimization. Our approach stems from two simple yet fundamental observations about haze-free images and the haze itself. First, clear-day images usually have stronger contrast than images plagued by bad weather; and second, the variations in natural atmospheric veil, which highly depends on the depth of objects, always tend to be smooth. Integrating these two criteria together leads to a new effective dehazing model, which encourages the gradient ℓ1sparsity of atmospheric veil and implicitly maximizes the global contrast of haze-free image in the meanwhile. We also show that the proposed dehazing model can be efficiently solved using the TV-ℓ1minimization. Compared to alternative state-of-the-art methods, our approach is physically plausible and works well for all types of hazy situations. Comparative study and quantitative evaluation on both synthetic and natural images validate the superior performance and the generality of our approach.

As the increasing popularity of superpixel-based applications, measuring superpixel-level similarity becomes an important and commonly required problem. In this paper, we propose a general bag of squares (BoS) model for such particular purpose. Compared to existing methods, our approach provides a full scheme to both invariantly represent superpixels and accurately measure their pairwise similarities. In order to handle the split-and-merge variety of superpixels of same objects in different scenes, our model is based on superpixel pyramid. As a result, the BoS model of a superpixel is built upon a group of subregions consisting of the superpixel itself and its children subregions in the pyramid. For each subregion, we extract a proper number of maximum squares via distance transform, and then use a fast self-validated approach to clustering them into a small number of dominant squares, which together with a rotation and scale invariant square descriptor, jointly compose the BoS model for the particular superpixel. Finally, we measure the similarity between a pair of superpixels by the closeness of their BoS models. Experiments on interactive object segmentation and co-saliency detection show that the proposed BoS model can reliably capture the delicate differences among superpixels, thus always producing better segmentation results, especially for segmenting highly variant objects in clutter scenes.

We present a generalized linear light source solution to estimate both the local shading frame and anisotropic surface reflectance of a planar spatially varying material sample. We generalize linear light source reflectometry by modulating the intensity along the linear light source, and show that a constant and two sinusoidal lighting patterns are sufficient for estimating the local shading frame and anisotropic surface reflectance. We propose a novel reconstruction algorithm based on the key observation that after factoring out the tangent rotation, the anisotropic surface reflectance lies in a low rank subspace. We exploit the differences in tangent rotation between surface points to infer the low rank subspace and fit each surface point's reflectance function in the projected low rank subspace to the observations. We propose two prototype acquisition devices for capturing surface reflectance that differ on whether the camera is fixed with respect to the linear light source or fixed with respect to the material sample. We demonstrate convincing results obtained from reflectance scans of surfaces with different reflectance and shading frame variations.

We present "appearance-from-motion", a novel method for recovering the spatially varying isotropic surface reflectance from a video of a rotating subject, with known geometry, under unknown natural illumination. We formulate the appearance recovery as an iterative process that alternates between estimating surface reflectance and estimating incident lighting. We characterize the surface reflectance by a data-driven microfacet model, and recover the microfacet normal distribution for each surface point separately from temporal changes in the observed radiance. To regularize the recovery of the incident lighting, we rely on the observation that natural lighting is sparse in the gradient domain. Furthermore, we exploit the sparsity of strong edges in the incident lighting to improve the robustness of the surface reflectance estimation. We demonstrate robust recovery of spatially varying isotropic reflectance from captured video as well as an internet video sequence for a wide variety of materials and natural lighting conditions.

Issues about city utility services reported by citizens can provide unprecedented insights into the various aspects of such services. Analysis of these issues can improve living quality through evidence-based decision making. However, these issues are complex, because of the involvement of spatial and temporal components, in addition to having multi-dimensional and multivariate natures. Consequently, exploring utility service problems and creating visual representations are difficult. To analyze these issues, we propose a visual analytics process based on the main tasks of utility service management. We also propose an aggregate method that transforms numerous issues into legible events and provide visualizations for events. In addition, we provide a set of tools and interaction techniques to explore such issues. Our approach enables administrators to make more informed decisions.

This paper addresses a challenging problem of regularizing arbitrary super pixels into an optimal grid structure, which may significantly extend current low-level vision algorithms by allowing them to use super pixels (SPs) conveniently as using pixels. For this purpose, we aim at constructing maximum cohesive SP-grid, which is composed of real nodes, i.e SPs, and dummy nodes that are meaningless in the image with only position-taking function in the grid. For a given formation of image SPs and proper number of dummy nodes, we first dynamically align them into a grid based on the centroid localities of SPs. We then define the SP-grid coherence as the sum of edge weights, with SP locality and appearance encoded, along all direct paths connecting any two nearest neighboring real nodes in the grid. We finally maximize the SP-grid coherence via cascade dynamic programming. Our approach can take the regional objectness as an optional constraint to produce more semantically reliable SP-grids. Experiments on object localization show that our approach outperforms state-of-the-art methods in terms of both detection accuracy and speed. We also find that with the same searching strategy and features, object localization at SP-level is about 100-500 times faster than pixel-level, with usually better detection accuracy.

Content-based image retrieval (CBIR) has been receiving intensive research attention for many applications. In order to provide the users with more precise retrieval results, relevance feedback (RF) methods have been incorporated into CBIR which take the user's feedbacks into account. In general, explicit RF methods demand too much user effort while implicit RF methods suffer from lower retrieval accuracy. As such, we propose a hybrid RF method, adaptive-weight hybrid relevance feedback (AHRF) for content-based image retrieval. AHRF integrates explicit user grading and implicit user browsing histories to build a user preference model. The model is refined iteratively and used to train a preference classifier for the users. Moreover, an adaptive-weight mechanism is proposed to achieve a personalized preference model. Our proposed method is tested on a subset of the Corel Database and the experimental results reveal that AHRF can achieve good retrieval precision with less user effort.

As a famous cultural wealth, the sketch of fresco is one of the most important art expression forms in the World Heritage. To avoid the damage of natural and human factors, painters can only use photos and videos to depict sketch in most world culture heritage sites, otherwise real frescos. Therefore, a computational method for extracting sketch is helpful and meaningful in sketch copying and researching area. However, existing approaches with the incomplete fresco are not enough to deal with the challenge of sketch extraction. In this paper, we proposes a framework to generate sketch of fresco with frescos as input. To reduce noise and refine detail lines, we adopt hierarchical segmentation technology to extract sketches of different regions respectively, and splice those sketches into an integrated sketch. For replacing the missing content and getting a complete sketch, we provide recommendations from a database of many existing fresco sketches elements for users to select. At last, users can adjust sketch based on vectorization to achieve optimization. Our framework can be used for artists or learners to study painting sketch images and research fresco art. The experimental results demonstrate the effectiveness of our framework.

The study of digital Chinese calligraphy has become more valuable nowadays. While existing researches on Chinese calligraphy analysis are primarily focused on stroke-based character recognition and simulation, we propose a global feature descriptor to deal with style recognition problem in this paper. The proposed method extracts three categories of character features: position features, proportion features and projection features. These features are then used to train an SVM classifier of calligraphy style. We test the global feature based classifier on five-style Chinese calligraphy character set. The experimental results show that the proposed method can achieve good classification accuracy, proving the effectiveness of the global feature descriptor in calligraphy style recognition.

Online virtual navigation systems enable users to hop from one 360° panorama to another, which belong to a sparse point-to-point collection, resulting in a less pleasant viewing experience. In this paper, we present a novel method, namely Cube2Video, to support navigating between cubic panoramas in a video-viewing mode. Our method circumvents the intrinsic challenge of cubic panoramas, i.e., the discontinuities between cube faces, in an efficient way. The proposed method extends the matching-triangulation-interpolation procedure with special considerations of the spherical domain. A triangle-to-triangle homography-based warping is developed to achieve physically plausible and visually pleasant interpolation results. The temporal smoothness of the synthesized video sequence is improved by means of a compensation transformation. As experimental results demonstrate, our method can synthesize pleasant video sequences in real time, thus mimicking walking or driving navigation.

For preserving the Grotto wall paintings and protecting these historic cultural icons from the damage and deterioration in nature environment, a visual analytics framework and a set of tools are proposed for the discovery of degradation patterns. In comparison with the traditional analysis methods that used restricted scales, our method provides users with multi-scale analytic support to study the problems on site, cave, wall and particular degradation area scales, through the application of multidimensional visualization techniques. Several case studies have been carried out using real-world wall painting data collected from a renowned World Heritage site, to verify the usability and effectiveness of the proposed method. User studies and expert reviews were also conducted through by domain experts ranging from scientists such as microenvironment researchers, archivists, geologists, chemists, to practitioners such as conservators, restorers and curators.

This paper describes TabuVis, a light weight visual analytics system that provides a flexible, customizable and effective visualization for multidimensional data. A key aspect of visual analytics of data with multiple attributes is the quality and appropriateness of the analytical platforms whose presentation can be adjusted via domain experts. A commercial-free and comprehensive prototype was developed that used scatter-plot approach to support the visual analytics process. Our system consists of multiple components enabling the complete analysis process, including data processing, automatic marks, interactive visualization and control. We demonstrate the effectiveness of TabuVis with two case studies using medical and Oscars data sets.

Structure-aware image completion has the reputation of keeping the salient structure of images, except for the case in texture propagation of images with distinguishing texture characteristics and large missing region. This inspirits us to design this new algorithm that could intensify texture-aware functions in the structure completion processing in an optimal manner. To avoid the occurrence of visually inconsistent results, we consider the filling of missing region as the energy minimization, using Belief Propagation (BP), of discrete Markov random field with integrated constraints of the costs of label, structure coherence and texture coherence. Moreover, wavelet multi-resolution image pyramid is adopted in our method, which not only enhances the global geometric structures and detailed texture features within large missing region, but as well speeds up the rate of convergence in the synthesis process.

Texture synthesis is a core process of computer graphics applications, which can enhance the realistic rendering greatly. With the rapidly increasing demands of realistic rendering, single texture synthesis cannot meet the needs. Multi-exemplars synthesis is a challenging research topic to increase the richness of texture details. In this paper, we use exemplar graph to realize Multi-exemplars texture synthesis under users' control. Further to improve the quality of synthesis results, we adopt bilateral filtering technology to improve the exemplar analysis stage. It can maintain the boundary of the hybrids of multi-exemplars well while making the texture elements in the composite image more clear at the same time.

We present a navigatable multi-dimensional visualization of data named coaxial interactive visualization which is based on the coaxial axes. In this visualization, the dimensions of data in n-dimensional space are mapped to n concentric circular axes drawn in a planar space with polar coordinates. A data item in n-dimensional space is represented as a poly-curve with n vertices across all circular axes. Via the spatial distortion in polar system, the view of multi-dimensional data is interactively adjusted to satisfy viewer's interest. The viewer, thus, can effectively navigate the underlying visual pattern to reach the great detail for uncovering trends and patterns in the visualization. One novelties of this work is performing focus+context viewing in polar coordinate system.

Wavelet Analysis method is considered as one of the most efficient methods for detecting DDoS attacks. However, during the peak data communication hours with a large amount of data transactions, this method is required to collect too many samples that will greatly increase the computational complexity. Therefore, the real-time response time as well as the accuracy of attack detection becomes very low. To address the above problem, we propose a new DDoS detection method called Modified Wavelet Analysis method which is based on the existing Isomap algorithm and wavelet analysis. In the paper, we present our new model and algorithm for detecting DDoS attacks and demonstrate the reasons of why we enlarge the Hurst's value of the self-similarity in our new approach. Finally we present an experimental evaluation to demonstrate that the proposed method is more efficient than the other traditional methods based on wavelet analysis.

Transfer function plays an important role in volume rendering. Multi-dimension transfer function can achieve high rendering quality, but suffer from troublesome user specifications. In order to reduce the complexity of manipulations, a transfer function design method based on dimension reduction model is proposed in this paper. The idea of dimension reduction model is to integrate several data features into one index for transfer function. This paper also presents one instance of dimension reduction model called acting force model (AFM) to verity the effectiveness. In AFM, one integrated index is calculated by applying transformative universal gravitation formula and kinetic energy formula to three data features: scalar value, gradient and coherence distance. The experiments demonstrate the proposed method can produce flexible results without overloading the users.

In this work, we describe a new technique for detecting image composites by enforcing two-view geometrical constrains: H and F constraints on image pairs, where H denotes the planar homography matrix and F the fundamental matrix. Our approach detects fake regions efficiently on pictures taken at the same scene but with different camera configurations. Performance of this approach is demonstrated on real image pairs with visually plausible composites.

Image compositing technology has become popular for tampering with digital photographies. We describe how such composites can be detected by enforcing the geometric and photometric constraints from shadows. In particular, we explore (i) the imaged shadow relations that are modeled by the planar homology, and (ii) the color characteristics of the shadows measured by the shadow matte. Our approach efficiently extracts these constraints from a single image and makes use of them for the digital forgery detection. Experimental results on visually plausible images demonstrate the performance of the proposed method.

Bar chart is a very common and simple graph that is mainly used to visualize simple x, y plots of data for numerical comparisons by partitioning the categorical data values into bars and typically limited to operate on highly aggregated dataset. In todaypsilas growing complexity of business data with multi dimensional attributes using bar chart itself is not sufficient to deal with the representation of such business dataset and it also not utilizes the screen space efficiently.Nevertheless, bar chart is still useful because of its shape create strong visual attention to users at first glance than other visualization techniques. In this article, we present a treemap bar chart + tablelens interaction technique that combines the treemap and bar chart visualizations with a tablelens based zooming technique that allows users to view the detail of a particular bar when the density of bars increases. In our approach, the capability of the original bar chart and treemaps for representing complex business data is enhanced and the utilization of display space is also optimized.

Fast display of realistic dynamic cloud scene is a challenging task for researchers in computer graphics. This paper describes a novel method of simulating animations of cloud. We first model the motion of cloud based on the physical theory of cloud formation. Then, the physical model of cloud is solved in discrete sparse grids. Thus, we can get approximate motion of cloud. Next, we advect texture image using the calculated velocity field to add the local detail of dynamic cloud scene. Here the texture image we used is Perlin noise map. The color of arbitrary point in the simulation space is got by calculating the product of the density texture and the regenerated Perlin noise texture. Finally, by changing different noise maps, realistic dynamic cloud scenes in different light environment are generated at high rendering rates.

We propose to reconstruct time-sequence dynamic virtual images. At different time of a day or different weather conditions the same scene is differently lighted. Using general regression neural networks the rule of lighting conditions changing with outside conditions can be infered, thus the virtual images in virtual outside conditions can be obtained from a real image of the scene. So using the method presented here we can reconstruct virtual images at any appointed virtual time or weather condition. Any geometrical information of the scene is not needed in generating virtual images. Combining IBMR techniques or panoramic image techniques, the model of scene is recovered. Further, combining the dynamic virtual images obtained by our method, time-sequence dynamic virtual scene can be reconstructed and revisited in virtual reality.