Quanta image sensors are an emerging type of image sensor offering single-photon sensitivity. In this paper, we propose a deep unfolding-based image reconstruction method that integrates an alternating direction method of multipliers optimization with a total variation prior. The proposed method effectively combines model-based approaches with deep learning, providing enhanced interpretability and performance. Experimental results demonstrate that our approach outperforms conventional methods based on the observation model for quanta image sensors.

Heart rate (HR) is a key parameter in evaluating the physiological and emotional states of a person. In this paper, we propose a novel video-based heart rate (HR) estimation method based on physiological modeling with multispectral imaging. To capture blood volume pulse (BVP) associated with a person’s heartbeat, we utilize a camera that records multispectral video consisting of red, green, blue, and near-infrared information. The novelty of the proposed method is the incorporation of a physiological BVP model into a multispectral HR estimation framework. The integration of a physiological model-based BVP signal extraction scheme into an adaptive multispectral framework enables the suppression of noise derived from ambient light and the accurate extraction of the BVP signal, thereby enhancing HR estimation performance. The experiments using RGB/NIR video datasets demonstrate the effectiveness of the proposed method.

Quanta image sensors are a novel paradigm in image sensor technology. Their direct application to quanta image sensors-based imaging systems is challenging because a bit-plane image is a set of binary images. In this paper, we introduce spatio-temporal priors based on the intensity invariance and smoothness characteristics of the motion vector. Specifically, we model when the image sequences align with the correct motion vector, the spatiotemporal structure becomes more consistent. Moreover, the spatial smoothness prior is incorporated through the smoothing filtering of the evaluation metrics of motion vector candidates. The experimental results show that the proposed method is more effective than conventional methods.

We propose a novel blood volume pulse (BVP) signal extraction method for heart rate estimation that incorporates the self-similarity properties of BVP in the spatial and temporal domains. The main novelty of the proposed method is the incorporation of the temporal self-similarity of BVP via low-rank approximation in the time-delay coordinate system for BVP signal extraction. To make a low-rank approximation of BVP in the time domain, we introduce knowledge of linear time-invariant systems, i.e., the autoregressive (AR) model lies in the low-rank subspace in the time-delay coordinate system. In the medical field, it is widely known that BVP has quasi-periodic temporal characteristics owing to the cardiac pulse and exhibits self-similarity properties in the temporal domain. Hence, we model the temporal behavior of BVP as an AR process, allowing for a low-rank approximation of BVP in the time-delay coordinate system. Low-rank approximation of BVP in the time and spatial domains enables reliable BVP signal extraction, resulting in accurate heart rate estimation. The experiments demonstrate the effectiveness of the proposed method.

This paper proposes a method for extracting diverse expressions from online movie review texts for a given keyword query. When people watch a movie that makes them cry, they generally do not say “I cried.” Instead, they use such euphemistic language as “I needed a handkerchief” or “My makeup was running.” To enable information retrieval based on audience reactions such as “movies that make me cry” using review texts, a variety of paraphrased expressions must be collected for arbitrary queries. Our proposed method extracts such expressions from review datasets by applying two extensions to Doc2Vec: 1) it changes the granularity of the training sentences to mitigate a lack of context, and 2) it applies query expansion for similarity calculation in advance. We conducted a large-scale experiment using crowdsourcing with 1.29 million actual sentences taken from Yahoo! Movies, Japan. The experimental result revealed that changing the training data granularity and adding the query expansion are both effective to accurately collect more diverse expressions that have a meaning similar to the given query.

This paper proposes a method for ranking movies by keyword queries and examples using machine learning techniques that analyze actual data from the online movie review site. Existing search methods cannot rank movies in “surprising order” for the keyword query “surprising.” People and critics created many “My best surprising movies” rankings on the web. Our proposed method uses a LambdaMART, one of the mainstream Learning to Rank techniques, to learn these personalized rankings and sort the movies through the viewpoint represented by a given query. To accept more complex information needs, we diverted the learning results to a search-by-example algorithm that enables users to input examples, such as “surprising movies like The Usual Suspects or Fight Club.” The experiment using the personal ranking data from the personal content curation service in Yahoo! Movies Japan suggests two findings: direct learning of personal ranking does not improve search performance, and the search-by-example-based application increases user satisfaction.

We propose a non-contact heart rate (HR) estimation method that is robust to various situations, such as bright, low-light, and varying illumination scenes. We utilize a camera that records red, green, and blue (RGB) and near-infrared (NIR) information to capture the subtle skin color changes induced by the cardiac pulse of a person. The key novelty of our method is the adaptive fusion of RGB and NIR signals for HR estimation based on the analysis of background illumination variations. RGB signals are suitable indicators for HR estimation in bright scenes. Conversely, NIR signals are more reliable than RGB signals in scenes with more complex illumination, as they can be captured independently of the changes in background illumination. By measuring the correlations between the lights reflected from the background and facial regions, we adaptively utilize RGB and NIR observations for HR estimation. The experiments demonstrate the effectiveness of the proposed method.

We propose a method for heart rate (HR) estimation that is robust to various situations such as bright, low-light, and varying illumination scenes. We capture temporal variations in the pixel values owing to person's cardiac pulse by using a camera that records red, green, and blue (RGB) and near-infrared (NIR) information. The key novelty of our method is to introduce a scheme for adaptive fusion of RGB and NIR signals for HR estimation, by analyzing variations in the background illuminations. RGB signals will be a good cue for HR estimation under bright scenes. In contrast, NIR signals are more reliable in HR estimation than RGB ones in complex illumination scenes, because NIR signals can be captured independent to changes in the background illuminations. By measuring correlations of signals between background and face regions, we adaptively utilize RGB and NIR signals for HR estimation. Experiments demonstrate the effectiveness of our method.

This paper proposes a new method of supplementing the context of short sentences for the training phase of Doc2Vec. Since CGM (Consumer Generated Media) sites and SNS sites become widespread, the importance of similarity calculation between a given query and a short sentence is increasing. As an example, a search by the query "sad" should find actual expressions such as "I needed a handkerchief" on a movie review site. Doc2Vec is one of the most widely used methods for vectorization of queries and sentences. However, Doc2Vec often exhibits low accuracy if the training data consists of short sentences, because they lack context. We modified Doc2Vec with the hypothesis that other posts for the same topic (i.e. reviews for the same movie in online movie review sites) may share the same background. Our method uses target-topic IDs instead of sentence IDs as the context in the training phase of the Doc2Vec with the PV-DM model; this model estimates the next term from a few previous terms and context. The model trained with item IDs vectorizes a sentence more accurately than a model trained with sentence IDs. We conducted a large-scale experiment using 1.2 million movie review posts and a crowdsourcing-based evaluation. The experimental result demonstrates that our new method achieves higher precision and nDCG than previous Doc2Vec variants and traditional topic modeling methods.