Polysomnography (PSG) is the current gold standard for the diagnosis of sleep disorders. However, this multi-parametric sleep monitoring tool also has some drawbacks, e.g. it limits the patient's mobility during the night and it requires the patient to come to a specialized sleep clinic or hospital to attach the sensors. Unobtrusive techniques for the detection of sleep disorders such as sleep apnea are therefore gaining increasing interest. Remote photoplethysmography using video is a technique which enables contactless detection of hemodynamic information. Promising results in near-infrared have been reported for the monitoring of sleep-relevant physiological parameters pulse rate, respiration and blood oxygen saturation. In this study we validate a contactless monitoring system on eight patients with a high likelihood of relevant obstructive sleep apnea, which are enrolled for a sleep study at a specialized sleep center. The dataset includes 46.5 hours of video recordings, full polysomnography and metadata. The camera can detect pulse and respiratory rate within 2 beats/breaths per minute accuracy 92% and 91% of the time, respectively. Estimated blood oxygen values are within 4 percentage points of the finger-oximeter 89% of the time. These results demonstrate the potential of a camera as a convenient diagnostic tool for sleep apnea, and sleep disorders in general.

The time interval between consecutive heartbeats (interpulse interval, IPI) has previously been suggested for securing mobile-health solutions. This time interval is known to contain a degree of randomness, permitting the generation of a time- and person-specific identifier. It is commonly assumed that only devices trusted by a person can make physical contact with him/her, and that this physical contact allows each device to generate a similar identifier based on its own cardiac recordings. Under these conditions, the identifiers generated by different trusted devices can facilitate secure authentication. Recently, a wide range of techniques have been proposed for measuring heartbeats remotely, a prominent example of which is remote photoplethysmography (rPPG). These techniques may pose a significant threat to heartbeat-based security, as an adversary may pretend to be a trusted device by generating a similar identifier without physical contact, thus bypassing one of the core security conditions. In this paper, we assess the feasibility of such remote attacks using state-of-the-art rPPG methods. Our evaluation shows that rPPG has similar accuracy as contact PPG and, thus, forms a substantial threat to heartbeat-based-security systems that permit trusted devices to obtain their identifiers from contact PPG recordings. Conversely, rPPG cannot obtain an accurate representation of an identifier generated from electrical cardiac signals, making the latter invulnerable to state-of-the-art remote attacks.

This paper introduces a powerful filtering method that exploits the physiological and optical properties of skin reflections to improve the performance of remote photoplethysmography (rPPG). Based on the fact that the pulsatile and nonpulsatile (e.g., intensity and specular changes) components have different reflection-spectra in a multi-wavelength camera, we propose to use their different characteristic color changes as a soft criterion to filter the RGB-signals in the frequency domain, such that the AC-components containing clear color distortions can be suppressed before the actual pulse extraction. This leads to a novel “Color-Distortion Filter” (CDF) that can be used as a common pre-processing step for arbitrary rPPG algorithms to increase their robustness. The benchmark in challenging fitness recordings shows that CDF brings significant and consistent improvements to all benchmarked rPPG algorithms, and drives all multi-channel approaches to a similar high quality-level.

Camera-based remote pulse rate monitoring can be used during fitness exercise to optimize the effectiveness of a workout. However, such monitoring suffers from vigorous body motions and dynamic illumination changes due to exercise, which may lead to erroneous estimates. To better cope with this, we propose a quality metric, comprised of a front-end metric and a back-end metric, to indicate the monitoring conditions (e.g. luminance, skin property) and assess the reliability of pulse rate measurement (e.g. signal quality). The proposed quality metric has been thoroughly benchmarked on 78 videos recorded in a fitness setting. The experimental results show that (i) appropriate light source intensity variation and its angle variation in the front-end metric are critical indicators for pulse rate measurement accuracy, and (ii) the back-end metric can effectively indicate/reject unreliable estimates. The proposed method in this paper is the first quality metric for camera-based pulse rate monitoring, validated for the challenging use-case of fitness exercises.

In this paper, we propose a camera-based system combining video motion detection, motion estimation, and texture analysis with machine learning for sleep analysis. The system is robust to time-varying illumination conditions while using standard camera and infrared illumination hardware. We tested the system for periodic limb movement (PLM) detection during sleep, using EMG signals as a reference. We evaluated the motion detection performance both per frame and with respect to movement event classification relevant for PLM detection. The Matthews correlation coefficient improved by a factor of 2, compared to a state-of-the-art motion detection method, while sensitivity and specificity increased with 45% and 15%, respectively. Movement event classification improved by a factor of 6 and 3 in constant and highly varying lighting conditions, respectively. On 11 PLM patient test sequences, the proposed system achieved a 100% accurate PLM index (PLMI) score with a slight temporal misalignment of the starting time (<1 s) regarding one movement. We conclude that camera-based PLM detection during sleep is feasible and can give an indication of the PLMI score.

To assess sleep in the home situation, wrist actigraphy is often used. However, it requires an on-body sensor which may disturb sleep and primarily collects data on the movement of one wrist only. Video actigraphy, by estimating motion from captured infrared images, overcome these issues. In this paper, we compare activity levels from wrist and full body video actigraphy in a home setting. Video actigraphy correctly found 19 % more small and medium movements that were missed by the wrist sensor. We further show that similar values of sleep efficiency (SE, %) are obtained from simultaneous recordings of video and wrist actigraphy, and we compared both to reference SE values provided by a full polysomnography (PSG). The proposed video actigraphy proved convenient and easy to use in real home situations. It successfully found movements originating from under the blanket, and turned out robust to various sleeping positions, different illumination conditions, viewing angles, beds and blankets. Our results suggest that on-body actigraph sensors can be successfully replaced with the proposed video-based solution for the application of unobtrusive sleep monitoring and analysis.

Overall health and daily functioning deteriorate with poor sleep. To improve one's sleep, sleep monitoring can help identifying causes of sleep problems. As an advantage over traditional wrist actigraphy used in home sleep monitoring solutions, video contains more comprehensive movement information. Particularly, different body movements can be distinguished which is beneficial for a more detailed sleep analysis. We developed an efficient K-Means clustering method with a multi-distance seeding technique to find the dominant cluster candidates. An integrated multi-distance dissimilarity measure was used for the subsequent clustering. We present an automatic content-dependent weight tuning method for the dissimilarity measure to balance between different distance descriptors. This discriminative algorithm partitions similar body movements in the same cluster. We were able to produce several dissimilarity measures producing clusters that agreed 67% with manual clustering of motion vectors by one expert. Similar clustering characteristics were preferred by both the five expert annotators and the suggested clustering algorithm. This gives us confidence that the proposed optimization method can be used in the future.

Various important 3-D depth cues, such as focus, motion, occlusion, and disparity, can only be estimated reliably at distinct sparse image locations, such as edges and corners. Hence, for 2-D to 3-D video conversion, a stable and smooth sparse-to-dense conversion is required to propagate these sparse estimates to the complete video. To this end, optimization, segmentation, and triangulation-based approaches have been proposed recently. While optimization-based approaches produce accurate dense maps, the resulting energy functions are very hard to minimize within the stringent requirements of real-time video processing. In addition, segmentation and triangulation-based approaches can cause incorrect delineation of object boundaries. Dense maps that are independently estimated from video images suffer from temporal instabilities. To deal with the real-time issue, we propose an innovative low latency, line scanning based sparse-to-dense conversion algorithm with a low computational complexity. To mitigate the stability and smoothness issues, we additionally propose a recursive spatiotemporal postprocessing and an efficient joint bilateral up-sampling method. We illustrate the performance of the resulting sparse-to-dense converter on dense defocus maps. We also show a subjective assessment of 2-D to 3-D conversion results using a paired comparison on a variety of challenging low-depth-of-field test sequences. The results demonstrate that the proposed approach achieves equal 3-D depth and video quality as state-of-the-art sparse-to-dense converters with a significantly reduced computational complexity and memory usage.

Video-based sleep analysis is a topic with important applications, and shared-bed occurs frequently in the context of sleep. One difficulty for the shared-bed situation is to assign the movements to the correct person because they can occur in close proximity and even overlapping. To manage to achieve person segmentation in the shared-bed situation, in this paper we propose an approach to correctly segment the region of persons based on motion estimation. In our approach, considering the consistency of the motion vectors, specifically their length and angle, the adjacent blocks are clustered. The generated clusters are then assigned to a person according to temporal correlation. The occupied region of the person is updated each frame based on the assignment result of the clusters. The proposed approach tackles the segmentation issue when the two persons are close to each other or even overlap, and the accuracy of the segmentation is beyond 82% in the data set we acquired.

In computer vision, many applications could greatly benefit from multi-spectral image data. Our aim is to illustrate the effectiveness of multi-spectral analysis obtained from a simple and cost-effective system. While the proposed approach is broadly applicable, in this paper we focus on the specific case of skin detection. To obtain the multi-spectral data, we have assembled a system using multiple LEDs with different spectra to illuminate the scene and a conventional RGB camera to acquire images. A methodology is proposed to avoid strict requirements on the experimental environment, by adopting a simple training procedure which is tuned for the detection of human skin. Next a specific feature set is defined and a corresponding normalization method is designed to improve the robustness to changes in skin color and incident light, issues not addressed by available prior art. Finally, we use supervised learning to train our skin detector. We demonstrate the accuracy and effectiveness of our skin detector through extensive benchmarking. The proposed methodology enables a superior performance of skin detection compared to relevant alternative proposals.

Many motion compensation algorithms are based on block matching. The quality of the block correlation depends on the validity of the brightness constancy assumption and the assumption of fixed translational motion within a block. These assumptions are invalid in areas with texture changes, noise, lighting changes, and rapid deformations. Smoothness priors should enforce stable estimates in these regions by propagating neighboring estimates, while preserving hard object boundaries (piecewise smoothness). Most motion estimation algorithms that successfully implement these constraints are computationally complex. In this paper, we show an intuitive and computationally efficient way to implement them within the framework of (real-time) recursive search, targeting consumer-market embedded devices with limited resources.

Spectral regression discriminant analysis (SRDA) is an efficient subspace learning method proposed recently. One important unsolved issue of SRDA is how to automatically determine an appropriate regularization parameter. In this letter, we present a method to estimate the optimal regularization parameter for SRDA. We test our method in different applications including head pose estimation, face recognition, and text categorization. Our extensive experiments evidently illustrate the effectiveness and efficiency of our approach.

A compression artifacts removal algorithm that is adaptive to the artifact visibility level of the input video signal is proposed. The artifact visibility is determined per frame by the ratio of the accumulated gradient on the block edges to that of the remaining area. The filtering of each video frame is optimized using a least mean square mechanism which trains on pairs of target images and decompressed images of similar quality as the input frame. Experimental results show that the proposed approach outperforms several methods in coding artifact reduction.

One of the hallmarks of Alzheimer's disease (AD) is the loss of neurons in the brain. In many cases, the medical experts use MR (magnetic resonance) images to qualitatively measure the neuronal loss by the shrinkage (atrophy) of the structures-of-interest, or sometimes more easily by the enlargement of the fluid-filled structures, such as the ventricles. For quantitative analysis, volume is the common choice. Volume, or area in 2D, is a gross measure and it cannot capture shape differences that can improve the diagnostic accuracy. Because most existing methods use complex and difficult- to-reproduce shape descriptors, the experts prefer more easily and robustly extractable area and volume in their diagnosis. In this paper, we introduce several novel and easily-extractable 2D shape descriptors for brain ventricle, and show that they and some of the well-known simple shape descriptors, such as perimeter, are better descriptors in the classification of AD patients and healthy controls.

This work presents adomain-specificmemory subsystem based on a two-level memory hierarchy. It targets the application domain of video post-processing applications including video enhancement and format conversion. These applications are based on motion compensation and/or broad class of content adaptive filtering to provide the highest quality of pictures. Our approach meets the required performance and has sufficient flexibility for the application domain. It especially aims at the implementation-wise most challenging applications: compute-intensive and bandwidth-demanding applications that provide the highest quality at high picture resolutions. The lowest level of the memory hierarchy, closest to the processing element, the L0 scratchpad, is organized specifically to enable fast retrieval of an arbitrarily positioned 2-D block of pixels to the processing element. To guarantee the performance, most of its addressing logic is hardwired, leaving a user a set of API for initialization and storing/loading the data to/from the L0 scratchpad. The next level of the memory hierarchy, the L1 scratchpad, minimizes the off-chip memory bandwidth requirements. The L1 scratchpad is organized specifically to enable efficient aligned block-based accesses. With lower data rates compared to the L0 scratchpad and aligned block access, software-based addressing is used to enable full flexibility. The two-level memory hierarchy exploits prefetching to further improve the performance.

An overview of the classification-based least squares trained filters on picture quality improvement algorithms is presented. For each algorithm, the training process is unique and individually selected classification methods are proposed. Objective evaluation is carried out to single out the optimal classification method for each application. To optimize combined video processing algorithms, integrated solutions are benchmarked against cascaded filters. The results show that the performance of integrated designs is superior to that of cascaded filters when the combined applications have conflicting demands in the frequency spectrum.

Integral projections have been proposed as an efficient method to reduce the dimensionality of the search space in motion estimation (ME) algorithms. A number of papers describe methods extending direct (correlation based) block matching algorithms with projections, others derive optical flow methods based on the Radon transform. With a single exception all of the above methods use a single horizontal and vertical projection to keep the ME algorithm computationally simple. In this work we derive a generalization on the direct projection methods: a ME method based on backprojections, and investigate the added value of using multiple projection angles.

Bilateral filtering is a simple and non-linear technique to remove the image noise while preserving edges. However, it is difficult to optimize a bilateral filter to obtain desired effect by supervised training. In this paper, we propose a new type of trained bilateral filter, which possesses the essential characteristics of the original bilateral filter and can be optimized offline by least mean square optimization. In applications of JPEG and H.264/MEPG4 AVC deblocking, we compared the proposed filter with the original bilateral filter and other state-of-the-art methods. Experimental results show that the proposed method has a better performance at artifacts reduction and edge preserving.

In this paper, an integrated resolution up-conversion and compression artifacts removal algorithm is proposed. Local image patterns are classified into object details or coding artifacts using the combination of structure information and activity measure. For each pattern class, the weighting coefficients for up-scaling and artifact reduction are optimized by a least mean square (LMS) training technique, which trains on the combination of the original images and the compressed down-sampled versions of the original images. The proposed combined algorithm is proven to be more effective than previous classification based techniques in concatenation.

In this paper a novel local blur estimation method is presented. The focal blur process is usually modeled as a Gaussian low-pass filtering and then the problem of blur estimation is to identify the Gaussian blur kernel. In the proposed method, the blurred input image is first re-blurred by Gaussian blur kernels with different blur radii. Then the difference ratios between the multiple re-blurred images and the input image are used to determine the unknown blur radius. We show that the proposed method does not require edge detection preprocessing and can estimate a wide range of blur radius. Experimental results of the proposed method on both synthetic and natural images and a comparison with a state-of-the-art method are presented.

The gap between application specific integrated circuits (ASICs) and general-purpose programmable processors in terms of performance, power, cost and flexibility is well known. Application specific instruction-set processors (ASIPs) bridge this gap. In this work, we demonstrate the key benefits of ASIPs for several video applications. One of the most compute- and memory-intensive functions in video processing is motion estimation (ME). The focus of this work is on the design of a ME template, which is useful for several video applications like video encoding, obstacle detection, picturerate up-conversion, 2-D-to-3-D video conversion, etc. An instruction-set suitable for performing a variety of ME functions is developed. The ASIP is based on a very long instruction word (VLIW) processor template and meets low-power and low-cost requirements still providing the flexibility needed for the application domain. The ME ASIP design consumes 27 mW and takes an area of 1.1 mm/sup 2/ in 0.13 /spl mu/m technology performing picturerate up-conversion, for standard definition (CCIR601) resolution at 50 frames per second.

We present a method for true-motion estimation assisted by feature point correspondences. First the difference between true-motion estimation and motion estimation for coding applications is explained, and an earlier published efficient true-motion estimation algorithm, called 3DRS, is summarized. Then the convergence property of this algorithm is discussed. We present a method for improving the convergence, by using feature point correspondences and show that a significant quality increase can be obtained for sequences containing high velocities.

Yen's generalisation of the sampling theorem has been proposed as the theoretical solution for de-interlacing by Delogne and Vandendorpe. The solution results in a vertical interpolation filter with coefficients that depend on the motion vector value, which uses samples that exist in the current field and additional samples from a neighbouring field shifted over (part of) a motion vector. We propose a further generalisation, where we design vector-adaptive inseparable 2D filters, which use samples from the current and the motion compensated previous field that are not available for all vectors on a vertical line. The resulting inseparable filters give a better interpolation quality at a given number of input pixels. We will show that the algorithm can be made robust against the sensitivity to inaccurate motion vectors.

For analog color television standards such as PAL and NTSC, the transmission of color (C) takes place within the band available for the luminance (Y). At the television receiver, the required separation of Y and C can only be imperfect as both components now share the same frequency space. Modern televisions apply so-called comb-filters that exploit the opposite sub-carrier phase of correlated samples to separate both components. However, cross-talk artifacts and loss of resolution will occur in situations where no sufficiently correlated samples meet the strict opposite phase requirement. In this paper, a novel Y/C separation method is presented that is able to use samples with non-opposite sub-carrier phases.

In a Standard Definition (SD) Television system, the Y:U:V video format 4:2:2 with chrominance sub-sampling is widely used. With the advent of High Definition (HD) television, the 4:4:4 format is required for high-performance TV. High-quality up-sampling methods have been developed to perform a resolution conversion from Standard Definition (SD) signal to HD signal. Although these algorithms have been designed for spatial scaling of luminance, they may be adapted and used to up-sample the low-resolution components U,V (4:2:2) to a high-resolution UV-colour format (4:4:4). In this paper, a content-adaptive up-scaling method for chrominance is proposed, with interpolation filters that adapt to the local structure of both luminance and chrominance data. Optimal filters were computed from a large video data set in different colour formats, such that original high-resolution data in a 4:4:4 format was reconstructed from low-resolution colour data, on the basis of the Least Mean Square (LMS) criterion. By combining edge information of both luminance and chrominance, the edge in the chrominance signal can be detected more accurately, thus exploiting the wider bandwidth of the luminance signal.

Motion estimation algorithms play a key role in picture-rate up-conversion. To realise high quality interpolated images, while keeping the processing requirements low, we analyse the following parameters of the motion estimation algorithm: the number of motion estimation iterations per input image pair and the image scanning order of individual iterations. Furthermore, we tackle the issue of the optimal selection of the motion vector candidates. The selected motion estimation algorithm is the three-dimensional recursive search (3DRS) block matcher that offers a smooth and accurate motion vector field at relatively low computational cost.

Several state-of-the-art noise reduction algorithms are ranked using both subjective evaluation and 20 objective measures proposed in the literature. It is found that no regression model based on these objective quality measures can replace subjective assessment.

The number of video formats is increasing rapidly. As a consequence, the conversion of video signals from one format into another is of growing importance. The motion portrayal of up-converted sequences significantly improves using motion compensated interpolation techniques, but artifacts are generally visible at the boundary of moving objects where either covering or uncovering of picture parts occurs. A robust method to locate covered and uncovered areas, find the correct motion vector, and interpolate new pictures, is presented. Moreover, the paper provides an overview and an evaluation of some of the more relevant proposals for picture rate up-conversion.

The techniques for video format conversion have reached a maturity, where they no longer prohibit application in the consumer domain because of high price or clear picture degradation. Given the resulting new freedom, the question arises how to optimally choose a display format, i.e. free from the television history. We present an evaluation of various realistic display formats, all using the same pixel frequency, obtained from a subjective assessment of critical image material originated either for television, or obtained from Internet pages.

The key element in realizing low cost real-time software implementations of a H.263 videoconferencing system is a fast motion estimation algorithm, which only slightly decreases coding efficiency. We propose a spatio-temporal recursive estimator that combines an excellent coding efficiency with a high computational efficiency. Experimentally, the new algorithm proves to be comparable to full-search block matching when encoding typical videoconferencing sequences in presence of additive noise, even though the computational burden has been greatly reduced.

The question "to interlace or not to interlace" divides the television and the personal computer communities. A proper answer requires a common understanding of what is possible nowadays in deinterlacing video signals. This paper outlines the most relevant proposals, ranging front simple linear methods to advanced motion-compensated algorithms, and provides a relative performance comparison for 12 of these methods. Next to objective performance indicators, screen photographs have been used to illustrate typical artifacts of individual deinterlacers. The overview provides no final answer in the interlace debate, as such requires unavailable capabilities in balancing technical and nontechnical issues.

Some efficient motion estimation algorithms select their output motion vector from a limited number of likely correct candidate, or prediction, vectors. In this paper, next to the known spatial and temporal prediction vectors, an additional and independent prediction is proposed. This candidate is generated with a parametric model describing the global motion in a previously estimated motion vector field. The proposal is elaborated as an addition to the three-dimensional (3-D) recursive search block-matching algorithm. The evaluation shows that a subpixel accurate, true-motion estimator results with a very low operations count.

In this paper, recently developed algorithms for high quality motion-compensated up-conversion are combined in a new architecture closely resembling that of current 100 Hz consumer television sets. By merging the motion estimation and the motion compensation part, the estimated silicon area could be reduced to a level where the entire functionality can be realized with one processing chip replacing the currently used 100 Hz processing chip. This enables a simple evolution towards motion compensated 100 Hz TV, considered to be very attractive. The architectural choice, and the wish to share expensive memories, requires some modifications in the motion estimator and up-convertor design which are discussed. The specific case of movie programs is dealt with, and it is also shown how the evolutionary architecture can achieve a significantly improved motion portrayal for this movie material.>

A new recursive block-matching motion estimation algorithm with only eight candidate vectors per block is presented. A fast convergence and a high accuracy, also in the vicinity of discontinuities in the velocity plane, was realized with such new techniques as bidirectional convergence and convergence accelerators. A new search strategy, asynchronous cyclic search, which allows a highly efficient implementation, is presented. A new block erosion postprocessing proposal further effectively eliminates block structures from the generated vector field. Measured with criteria relevant for the field rate conversion application, the new motion estimator is shown to have a superior performance over alternative algorithms, while its complexity is significantly less.>