Mutual gaze detection, i.e., predicting whether or not two people are looking at each other, plays an important role in understanding human interactions. In this work, we focus on the task of image-based mutual gaze detection, and propose a simple and effective approach to boost the performance by using an auxiliary 3D gaze estimation task during the training phase. We achieve the performance boost without additional labeling cost by training the 3D gaze estimation branch using pseudo 3D gaze labels deduced from mutual gaze labels. By sharing the head image encoder between the 3D gaze estimation and the mutual gaze detection branches, we achieve better head features than learned by training the mutual gaze detection branch alone. Experimental results on three image datasets show that the proposed approach improves the detection performance significantly without additional annotations. This work also introduces a new image dataset that consists of 33.1K pairs of humans annotated with mutual gaze labels in 29.2K images.

Learning a classifier from ambiguously labeled face images is challenging since training images are not always explicitly-labeled. For instance, face images of two persons in a news photo are not explicitly labeled by their names in the caption. We propose a Matrix Completion for Ambiguity Resolution (MCar) method for predicting the actual labels from ambiguously labeled images. This step is followed by learning a standard supervised classifier from the disambiguated labels to classify new images. To prevent the majority labels from dominating the result of MCar, we generalize MCar to a weighted MCar (WMCar) that handles label imbalance. Since WMCar outputs a soft labeling vector of reduced ambiguity for each instance, we can iteratively refine it by feeding it as the input to WMCar. Nevertheless, such an iterative implementation can be affected by the noisy soft labeling vectors, and thus the performance may degrade. Our proposed Iterative Candidate Elimination (ICE) procedure makes the iterative ambiguity resolution possible by gradually eliminating a portion of least likely candidates in ambiguously labeled faces. We further extend MCar to incorporate the labeling constraints among instances when such prior knowledge is available. Compared to existing methods, our approach demonstrates improvements on several ambiguously labeled datasets.

In this paper, we present a new video-based face identification algorithm, where the target (i.e., person of interest) in the probe video is only annotated once with a face bounding box in a frame and the video may consist of multiple shots. Most video face identification techniques assume that the video is of single shot, and thus the bounding boxes of the target face can be extracted by tracking a face across the video frames. Nevertheless, such automatic annotation is vulnerable to the drifting of the face tracker, and the face tracking algorithm is inadequate to associate the face images of the target across multiple shots. In this paper, we propose a target face association (TFA) technique that retrieves a set of representative face images in a given video that are likely to have the same identity as the target face. These face images are then utilized to construct a robust face representation of the target face for searching the corresponding subject in the gallery. Since two faces that appear in the same video frame cannot belong to the same person, such cannot-link constraints are utilized for learning a target-specific linear classifier for establishing the intra/inter-shot face association of the target. Experimental results on the newly released JANUS challenge set 3 (JANUS CS3) dataset show that our method generates robust representations from target-annotated videos and demonstrates good performance for the task of video-based face identification problem.

We propose a non-local cost aggregation algorithm to recognize the identity of face and person tracks in a TV-series. In our approach, the fundamental element for identification is a track node, which is built on top of face and person tracks. Track nodes with temporal dependency are grouped into a knot. These knots then serve as the basic units in the construction of a k-knot graph for exploring the video structure. We build the minimum-distance spanning tree (MST) from the k-knot graph such that track nodes of similar appearance are adjacent to each other in MST. Non-local cost aggregation is performed on MST, which ensures information from face and person tracks is utilized as a whole to improve the identification performance. The identification task is performed by minimizing the cost of each knot, which takes into account the unique presence of a subject in a venue. Experimental results demonstrate the effectiveness of our method.

In real applications, data is not always explicitly-labeled. For instance, label ambiguity exists when we associate two persons appearing in a news photo with two names provided in the caption. We propose a matrix completion-based method for predicting the actual labels from the ambiguously labeled instances, and a standard supervised classifier can learn from the disambiguated labels to classify new data. We further generalize the method to handle the labeling constraints between instances when such prior knowledge is available. Compared to existing methods, our approach achieves 2.9% improvement on the labeling accuracy of the Lost dataset and comparable performance on the Labeled Yahoo! News dataset.

Defocus blur usually causes performance degradation in establishing the visual correspondence between stereo images. We propose a blur-aware disparity estimation method that is robust to the mismatch of focus in stereo images. The relative blur resulting from the mismatch of focus between stereo images is approximated as the difference of the square diameters of the blur kernels. Based on the defocus and stereo model, we propose the relative blur versus disparity (RBD) model that characterizes the relative blur as a second-order polynomial function of disparity. Our method alternates between RBD model update and disparity update in each iteration. The RBD model in return refines the disparity estimation by updating the matching cost and aggregation weight to compensate the mismatch of focus. Experiments using both synthesized and real datasets demonstrate the effectiveness of our proposed algorithm.

Multimedia Broadcast/Multicast Service (MBMS) is a bandwidth efficient broadcast scheme for multimedia communications. To support prioritized transmissions, the unequal error protection (UEP) for multi-resolution multimedia sources can be realized through MBMS. Nevertheless, the enhancement on the transmission fidelity in base layer typically sacrifices the fidelity of enhancement layers. Herein, a novel dual diversity space-time coding (DDSTC) is proposed to exploit the intrinsic UEP capability of space-time codes by utilizing a constellation mapping duo for two consecutive transmission periods in multiple-input multiple-output (MIMO) systems. As compared with Alamouti coding, the DDSTC achieves coding gains on the transmission error rates of base layer without significant degradations on the enhancement layers. At the transmission rates of base and enhancement layers equal to 2 bits per transmission, the DDSTC obtains 1.3 dB and 3.0 dB coding gains for base layer in 2 × 2 and 2 × 3 MIMO systems respectively. Besides, analytical analysis on symbol error probabilities verifies that 6 dB asymptotic coding gain is reachable in rich transmit diversity scenarios. While attaining the considerable improvements on error rates, the DDSTC avoids the high decoding complexity by adopting our proposed decoding schemes. Simulation results show that DDSTC outperforms conventional UEP schemes based on hierarchical modulations or power allocations.

We propose a novel joint source-channel coding (JSCC) framework that jointly optimizes encoding modes of macroblocks and unequal error protection (UEP) of packets for error-resilient video transmission, and address the problem of source packet loss due to bit errors incurred in error-prone channels. In the proposed framework, we consider the source packet loss probability as a function of the encoding configuration of a slice. The task of optimization is complicated by the inherent interdependency between macroblocks in the same slice, since the encoding mode optimization for each macroblock requires the encoding configuration of entire slice to be known in advance. We resolve such interdependency via an iterative method that alternates between the optimization of the slice size and the associated encoding modes, and we also utilize adaptive quantization for coding residual for source rate control in different channel conditions. Simulation results show that our framework outperforms existing approaches.

Spatial multiplexing is an efficient transmission technique for video streaming in multiple-input multiple-output (MIMO) wireless communication links due to its capability to support the high transmit rate. Nevertheless, the video quality is sensitive to packet losses resulting from channel fading and co-interference between transmit antennas in spatial multiplexing systems. Antenna selection techniques have been investigated to balance the spatial multiplexing gain and the link reliability. Herein, a novel cross-layer framework using dynamic antenna selection is proposed to control the multiplexing-diversity tradeoff with joint error-resilient source and channel coding. The cross-layer optimization is performed to optimize the physical, link, and application layers, and to minimize the end-to-end video distortion.

In this paper, we proposed a dynamic ordered subcarrier selection algorithm (DOSSA) for OFDM based video transmission system. The proposed scheme is shown to achieve lower bit error rate (BER) than the previously proposed OSSA by first selecting a fraction of the subcarriers with highest channel gain. The content information is then exploited in order to extend the OSSA to achieve unequal error protection (UEP) for packets of different importance. Simulation results show that system that utilizes the proposed scheme can achieve higher PSNR, especially at low SNR, compared to those that use the equal error protection (EEP) OSSA.

Recently, the development of mobile services is becoming more popular. The mobile applications will not only take advantage of contextual information, such as location-awareness, to offer greater services to a mobile host (MH) but maintain existing transport-layer connections as the MH moves from one location to another. This paper exhausts our most recent work: the AcoustaNomad project. AcoustaNomad not only uses the mobile IPv6 to maintain the existing connections even if the MH changes locations and addresses, but utilizes location-aware technique to detect what kind of services the new location provides. In addition, AcoustaNomad shows two mature mobile applications: mobile learning and audio blogging. This paper proposes the architecture of AcoustaNomad and experimental results that demonstrate the ability of AcoustaNomad to enable location-aware services and applications.