Precise segmentation of brain tumors, particularly contrast-enhancing regions visible in post-contrast MRI (areas highlighted by contrast agent injection), is crucial for accurate clinical diagnosis and treatment planning but remains challenging. However, current methods exhibit notable performance degradation in segmenting these enhancing brain tumor areas, largely due to insufficient consideration of MRI-specific tumor features such as complex textures and directional variations. To address this, we propose the Harmonized Frequency Fusion Network (HFF-Net), which rethinks brain tumor segmentation from a frequency-domain perspective. To comprehensively characterize tumor regions, we develop a Frequency Domain Decomposition (FDD) module that separates MRI images into low-frequency components, capturing smooth tumor contours and high-frequency components, highlighting detailed textures and directional edges. To further enhance sensitivity to tumor boundaries, we introduce an Adaptive Laplacian Convolution (ALC) module that adaptively emphasizes critical high-frequency details using dynamically updated convolution kernels. To effectively fuse tumor features across multiple scales, we design a Frequency Domain Cross-Attention (FDCA) integrating semantic, positional, and slice-specific information. We further validate and interpret frequency-domain improvements through visualization, theoretical reasoning, and experimental analyses. Extensive experiments on four public datasets demonstrate that HFF-Net achieves an average relative improvement of 4.48% (ranging from 2.39% to 7.72%) in the mean Dice scores across the three major subregions, and an average relative improvement of 7.33% (ranging from 5.96% to 8.64%) in the segmentation of contrast-enhancing tumor regions, while maintaining favorable computational efficiency and clinical applicability. Our code is available at: https://github.com/VinyehShaw/HFF.

While traditional feature engineering for Human Activity Recognition (HAR) involves a trial-and-error process, deep learning has emerged as a preferred method for high-level representations of sensor-based human activities. However, most deep learning-based HAR requires a large amount of labelled data and extracting HAR features from unlabelled data for effective deep learning training remains challenging. We, therefore, introduce a deep semi-supervised HAR approach, MixHAR, which concurrently uses labelled and unlabelled activities. Our MixHAR employs a linear interpolation mechanism to blend labelled and unlabelled activities while addressing both inter- and intra-activity variability. A unique challenge identified is the activity-intrusion problem during mixing, for which we propose a mixing calibration mechanism to mitigate it in the feature embedding space. Additionally, we rigorously explored and evaluated the five conventional/popular deep semi-supervised technologies on HAR, acting as the benchmark of deep semi-supervised HAR. Our results demonstrate that MixHAR significantly improves performance, underscoring the potential of deep semi-supervised techniques in HAR.

Unsupervised domain adaptation methods have been proposed to tackle the problem of covariate shift by minimizing the distribution discrepancy between the feature embeddings of source domain and target domain. However, the standard evaluation protocols assume that the conditional label distributions of the two domains are invariant, which is usually not consistent with the real-world scenarios such as long-tailed distribution of visual categories. In this article, the imbalanced domain adaptation (IDA) is formulated for a more realistic scenario where both label shift and covariate shift occur between the two domains. Theoretically, when label shift exists, aligning the marginal distributions may result in negative transfer. Therefore, a novel cluster-level discrepancy minimization (CDM) is developed. CDM proposes cross-domain similarity learning to learn tight and discriminative clusters, which are utilized for both feature-level and distribution-level discrepancy minimization, palliating the negative effect of label shift during domain transfer. Theoretical justifications further demonstrate that CDM minimizes the target risk in a progressive manner. To corroborate the effectiveness of CDM, we propose two evaluation protocols according to the real-world situation and benchmark existing domain adaptation approaches. Extensive experiments demonstrate that negative transfer does occur due to label shift, while our approach achieves significant improvement on imbalanced datasets, including Office-31, Image-CLEF, and Office-Home.

The recent emergence of light field technology has led to new opportunities for immersive visual communication that has a need for high spatial and angular resolution, both of which contribute to a large image storage footprint and high-latency transmission. Task-driven downsampling methods have been proposed as a solution, and have shown improvements in single-image restoration. However, they are inevitable to disregard light field's intrinsic properties in the corresponding tasks. In this paper, we propose a light-field-specific task-driven downsampling framework, called the LFCrNet. The LFCrNet operates on a learning-based decreasing and increasing resolution in an end-to-end manner in order to utilize a cross-view asymmetric sampling technique. In detail, it separates raw data into disparity and non-disparity patterns by measuring pixel-wise residuals between the sub-aperture central view and auxiliary views. Then, a chain of 3-D deformable residual blocks (DRBs) is used to extract disparity features and manage these features regard of their intrinsic property individually. Afterwards, they are compacted into spatio-angular domains through a 3-D deformable downsampler (3-DDS). The non-disparity information is integrated into a separate pipeline that leverages spatial similarity across multiple light field views. This technique is capable of preserving specific occlusion components, and subsequently, restoring them using a learning-based upscaling method to generate a high-quality reconstruction. In general, our method has shown superior performance on multiple open-source datasets by a significant margin.

Glass-free compressive light field (CLF) display gains much attention due to their compatibility in holographic-like and three-dimensional (3D) demonstration. Opposite to other analogous devices, CLF display can provide binocular and motion parallaxes by stacking multiple liquid crystal screens without any extra accessories. It is possible to bring the immersive and accommodative experience upon a well-pleasing visual consequence. Conventionally, the excessive processing time impacts its practical value in commercial, along with the severe degradation of display brightness. Therefore, in this paper, we propose a learning-based factorization framework to promote the visual results and expedite the layer decomposition and display adaption. It utilizes the advantage of a dual-guided system and residual learning to implement pixel-wise information extraction and refinement. The experimental results illustrate the outperformance of our proposed method over the conventional iterative factorization. Furthermore, a three-layered CLF prototype has been assembled to verify the practicality of our method.

High-quality point clouds have practical significance for point-based rendering, semantic understanding, and surface reconstruction. Upsampling sparse, noisy and non-uniform point clouds for a denser and more regular approximation of target objects is a desirable but challenging task. Most existing methods duplicate point features for upsampling, constraining the upsampling scales at a fixed rate. In this work, the arbitrary point clouds upsampling rates are achieved via edge-vector based affine combinations, and a novel design of Edge-Vector based Approximation for Flexible-scale Point clouds Upsampling (PU-EVA) is proposed. The edge-vector based approximation encodes neighboring connectivity via affine combinations based on edge vectors, and restricts the approximation error within a second-order term of Taylor’s Expansion. Moreover, the EVA upsampling decouples the upsampling scales with network architecture, achieving the arbitrary upsampling rates in one-time training. Qualitative and quantitative evaluations demonstrate that the proposed PU-EVA outperforms the state-of-the-arts in terms of proximity-to-surface, distribution uniformity, and geometric details preservation.

Existing enhancement methods are empirically expected to help the high-level end computer vision task: however, that is observed to not always be the case in practice. We focus on object or face detection in poor visibility enhancements caused by bad weathers (haze, rain) and low light conditions. To provide a more thorough examination and fair comparison, we introduce three benchmark sets collected in real-world hazy, rainy, and low-light conditions, respectively, with annotated objects/faces. We launched the UG2+ challenge Track 2 competition in IEEE CVPR 2019, aiming to evoke a comprehensive discussion and exploration about whether and how low-level vision techniques can benefit the high-level automatic visual recognition in various scenarios. To our best knowledge, this is the first and currently largest effort of its kind. Baseline results by cascading existing enhancement and detection models are reported, indicating the highly challenging nature of our new data as well as the large room for further technical innovations. Thanks to a large participation from the research community, we are able to analyze representative team solutions, striving to better identify the strengths and limitations of existing mindsets as well as the future directions.

Multi-layer light field displays are a type of computational three-dimensional (3D) display which has recently gained increasing interest for its holographic-like effect and natural compatibility with 2D displays. However, the major shortcoming, depth limitation, still cannot be overcome in the traditional light field modeling and reconstruction based on multi-layer liquid crystal displays (LCDs). Considering this disadvantage, our paper incorporates a salience guided depth optimization over a limited display range to calibrate the displayed depth and present the maximum area of salience region for multi-layer light field display. Different from previously reported cascaded light field displays that use the fixed initialization plane as the depth center of display content, our method automatically calibrates the depth initialization based on the salience results derived from the proposed contrast enhanced salience detection method. Experiments demonstrate that the proposed method provides a promising advantage in visual perception for the compressive light field displays from both software simulation and prototype demonstration.

This work introduces a pixel rendering module (PRM) into an asynchronous event-based dynamic vision sensor (DVS) targeting for optical flow extraction. Optical flow using event-based cameras draws more attention since DVSs directly provide motion related information related and greatly reduce the data redundancy compared to conventional frame-based cameras. Although event-based optical flow has a high potential on real-time performance, its accuracy is limited by event sparseness and lack of intensity, especially for fast motion and highly textured areas. This paper presents a motion sensor with PRM and asynchronous gray-level events to settle these issues. The PRM enables each pixel to communicate with its neighbor pixels such that a single active pixel can force activate its neighboring inactive pixels to provide sufficient data for optical flow calculation. Furthermore, the sensor outputs asynchronous event packages including pixel position, time-stamp and its corresponding illumination. A 64 × 64 prototype was fabricated in 0.35um 2P4M Opto process. Each pixel occupies a footprint of 40 × 40 μm2with 17 7% fill factor.

In this paper, we propose an event-guided support vector machine (ESVM) for tracking high-speed moving objects. Tracking fast-moving objects with low frame rate cameras is always difficult due to motion blur and large displacements. The accuracy problem can be solved by using high frame rate cameras at the expense of tremendous computational cost. For this issue, our ESVM incorporates event-based guiding methods into the traditional structured support vector machine to improve the tracking accuracy at a relatively low-complexity level. The event-based guiding methods include two models, event position guided search localization and event intensity guided sample supplement, which are based on the event features of the CeleX motion sensor. The motion sensor continuously responds to intensity change, which is generally related to object motion. Once it has detected intensity change, the motion sensor outputs event packages, and each of them contains the pixel location, time stamp, and pixel illumination. The generated events are continuous in the temporal domain and thus record the motion trajectory of fast-moving objects, which cannot be fully captured by frame-based cameras. In this paper, we convert high-speed test sequences into sequences of spiking events recorded by the CeleX motion sensor. Our approach presents fairly high computational efficiency, and experiments over sequences from multiple tracking benchmarks demonstrate the superior accuracy and real-time performance of our method, compared to the state-of-the-art trackers.

Light field images can be conveniently captured by consumer-level plenoptic cameras. However, as the resulting data rates are very high, providing efficient compression for this type of data is of critical importance. This remains an open problem which has recently attracted a lot of attention from the coding community. State-of-the-art compression systems prove to be inefficient when directly applied on this type of data due to the inherent spatial discontinuities in light field images. In this paper, a novel intra-prediction method for disk-shaped pixel clusters is proposed. An L1 minimization of the prediction residuals is performed followed by clustering of the predictors, leading to an optimized set of predictors for the macro-pixels. Furthermore, directional intra-prediction modes based on HEVC are devised for the macro-pixels. Experimental results obtained on the EPFL light field image dataset demonstrate that the proposed coding scheme yields an average of 3.22 dB and 1.45 dB gain in PSNR, and 59.6% and 30.88% average rate savings compared to HEVC and the state-of-the-art in light field image coding respectively.

With the ability of representing 3D scene geometry, depth maps are used to synthesize virtual views in free view video (FVV) or 3DTV. However, compression artefact of the depth images always lead to seriously geometry distortions in synthesized view, which severely affects the visual perception of 3D display. To solve the problem caused by depth artifact, the bilateral filter based method is presented to alleviate the noise by local weighted sum of neighboring pixels. However, to overcome the unstable feature of local weight of noisy pixels, we propose a novel graph cuts algorithm for depth filter with the constraint of corresponding structure information in color image. As a depth in-loop filter, the filter is incorporated into the framework of H.264/MVC. The proposed approach offers 0.5dB and 0.8dB average PSNR gains in terms of video rendering quality and depth coding efficiency comparing with the state-of-the-art.

The advent of consumer-level plenoptic cameras has sparkled the interest towards the design of efficient compression techniques for light field images. State-of-the-art compression systems such as HEVC prove to be inefficient when directly applied on this type of data due to the inherent spatial discontinuities among neighboring microlens images. In this paper, a novel light field image compression system is proposed. The disk-shaped pixel clusters corresponding to each microlens in the light field image are efficiently predicted based on the neighboring disks. In this context, an optimized linear prediction design based on L1 minimization of the residuals is proposed. K-means clustering is employed on training data in order to determine the optimized set of predictors. The experimental results on an extensive set of light field images demonstrate that the proposed coding scheme yields an average of 2.93 dB and 3.22 dB gain in PSNR, and 52.67% and 57.27% average rate savings compared to HEVC and JPEG2000 respectively.

Summary form only given. We demonstrate a simple head tracked 3D display that is capable of providing motion parallax; the ability to look around objects. This is achieved by using a head tracked display that shows two images only at any particular time. These images can be observed in alternate viewing zones across the viewing field in front of the screen that are fixed in position. The display serves a single viewer and as the viewer traverses the viewing field a series of perspective views is displayed such that the appropriate pair of views is always directed to the viewer's left and right eyes with the use of a head position tracker.

Summary form only given. This paper presents a demo system that realistically displays the glasses-free light field 3D effect with a triple-layer structure. By combining multi-layer panels, high refresh rates, and directional backlighting together, we achieve a wide field of view and large depth of field with a thin form factor. Additionally, using some off-the-shelf hardware, this system demonstrates an interesting light field display.

In this paper, we propose a method to optimize two-layer light field display using depth initialization. In contrast to existing trade-off work between performance and processing time, this paper firstly models the display principle of layered light field display, and then performs layered initialization with the prior known depth of 3D objects, and finally optimizes the layered images for light field display. Experiments demonstrate that the proposed initialization method can obviously save the iterations and related processing time for the existing online or offline algorithms to achieve the same reconstructed peak signal to noise ratio (PSNR) and present a better subjective reconstructed performance using the same computation resource.

With the increasing volume of video data, how to analyze and browse video in a fast and effective way has become an urgent problem in applications. This letter proposes a novel video synopsis method in compressed domain for browsing video captured by static cameras. Synopsis video is a video abstraction, which displays moving objects from different periods simultaneously on the primary background contents of original video. To overcome the low efficiency of traditional video synopsis for compressed video, our method presents a new graph cut algorithm to extract objects tubes and meanwhile gives a fast solution to minimize energy function in compressed domain. Experimental results in H.264 video have demonstrated the high-efficiency of this new video synopsis scheme for massive video browsing.

Sparse representation-based face hallucination approaches proposed so far use fixed ℓ1norm penalty to capture the sparse nature of face images, and thus hardly adapt readily to the statistical variability of underlying images. Additionally, they ignore the influence of spatial distances between the test image and training basis images on optimal reconstruction coefficients. Consequently, they cannot offer a satisfactory performance in practical face hallucination applications. In this paper, we propose a weighted adaptive sparse regularization (WASR) method to promote accuracy, stability and robustness for face hallucination reconstruction, in which a distance-inducing weighted ℓqnorm penalty is imposed on the solution. With the adjustment to shrinkage parameter q , the weighted ℓqpenalty function enables elastic description ability in the sparse domain, leading to more conservative sparsity in an ascending order of q . In particular, WASR with an optimal q > 1 can reasonably represent the less sparse nature of noisy images and thus remarkably boosts noise robust performance in face hallucination. Various experimental results on standard face database as well as real-world images show that our proposed method outperforms state-of-the-art methods in terms of both objective metrics and visual quality.

Face hallucination methods based on low-resolution (LR) and high-resolution (HR) dictionary pair scheme infer HR patches by directly reusing coding coefficients trained by LR patches over LR dictionary. This scheme implies that LR and HR patch manifolds share highly similar local geometric structure. However, latest preliminary studies argue that the manifold assumption does not hold well such that face hallucination performance inevitably suffers from inconsistency of coding coefficients between LR and HR patches. In this paper, we are the first to observe that coding coefficients of LR patches are more relevant to latent those of HR patches under conditions of involving small magnifying factor. On the basis of this finding, we suggest a stepwise reconstruction scheme to minimize inconsistency risk in solution space. In particular, this scheme divides face hallucination process into multiple cascaded incremental training-synthesis steps, in which each individual step allows smaller magnifying factor as well as the corresponding intermediate resolution (IR) dictionary rather than merely LR and HR dictionary based learning. Moreover, in order to keep sparse representation (SR) sufficiently sparse while favoring its locality, we introduce a weighted ℓ1/ℓ2mixed norms minimization SR method and formulate a unified framework together with stepwise scheme. Experiments on commonly used face database demonstrate that our framework achieves state-of-the-art results.

With the increasing amount of surveillance data, moving object segmentation in the compressed domain has drawn broad attention from both academy and industry. In this paper, we propose a novel moving object segmentation method towards H.264 compressed surveillance videos. First, the motion vectors (MV) are accumulated and filtered to achieve reliable motion information. Second, considering the spatial and temporal correlations among adjacent blocks, spatio-temporal Local Binary Pattern (LBP) features of MVs are extracted to obtain coarse and initial object regions. Finally, a coarse-to-fine segmentation algorithm of boundary modification is conducted based on the DCT coefficients. The experimental results validate that the proposed method not only can extract fairly accurate objects in compressed video, but also has a relatively low computational complexity.

Pixel-domain analysis methods are widely adopted in background modeling, some of which are not only concerned by academia but also coming into view of industry. However, as the increasing data volume of video, how to process and analysis videos in a fast and effective way has still been an intractable problem in practical applications. Under this circumstance, surveillance video analysis in the compressed domain is indeed of strategic importance from the angle of balancing visual perception and processing speed, especially in modeling background and segmenting moving objects. Therefore, a background modeling method in the compressed domain is proposed to quickly extract moving objects in this paper. Our main contributions are: 1) a method to calculate MVLBP features based on MV amplitude in the compressed domain is presented; 2) a background modeling and moving objects extraction method is designed in the compressed domain based on Local Binary Patterns of Motion Vector (MVLBP). Experimental results show that our approach gives a stable performance in a shorter time in H.264 compressed domain.

With extending applications of digital surveillance video, the fast browsing technique for surveillance video has become a hot spot in the domain. As one of the most important supporting technologies, region of interest (ROI) information coding is a necessary part of fast browsing framework for surveillance video. Therefore, a novel coding method of object region flags and successful application of object mapping flags for the scalable browsing of surveillance video are proposed in this paper. This object region flags coding method includes both intra-frame coding and inter-frame coding, wherein, the former eliminates the redundancy in spatial domain by improving the scanning mode of region flags, the latter eliminates the redundancy in temporal domain, which cuts coding cost obviously and provides essential supports for video synopsis browsing of surveillance video.