The recent advance in vision-language models is largely attributed to the abundance of image-text data. We aim to replicate this success for video-language models, but there simply is not enough human-curated video-text data available. We thus resort to fine-tuning a video-language model from a strong image-language baseline with syn-thesized instructional data. The resulting video model by video-instruction-tuning (VIIT) is then used to auto-label millions of videos to generate high-quality captions. We show the adapted video-language model performs well on a wide range of video-language benchmarks. For instance, it surpasses the best prior result on open-ended NExT-QA by 2.8%. Besides, our model generates detailed descriptions for previously unseen videos, which provide better textual supervision than existing methods. Experiments show that a video-language dual-encoder model contrastively trained on these auto-generated captions is 3.8% better than the strongest baseline that also leverages vision-language models. Our best model outperforms state-of-the-art methods on MSR-VTT zero-shot text-to-video retrieval by 6%. As a side product, we generate the largest video capation dataset to date.

A model that can authentically restore a low-quality face image to a high-quality one can benefit many applications. While existing approaches for face restoration make significant progress in generating high-quality faces, they often fail to preserve facial features that compromise the authenticity of reconstructed faces. Because the human visual system is very sensitive to faces, even minor changes may significantly degrade the perceptual quality. In this work, we argue that the problems of existing models can be traced down to the two sub-tasks of the face restoration problem, i.e. face generation and face reconstruction, and the fragile balance between them. Based on the observation, we propose a new face restoration model that improves both generation and reconstruction. Besides the model improvement, we also introduce a new evaluation metric for measuring models' ability to preserve the identity in the restored faces. Extensive experiments demonstrate that our model achieves state-of-the-art performance on multiple face restoration benchmarks, and the proposed metric has a higher correlation with user preference. The user study shows that our model produces higher quality faces while better preserving the identity 86.4% of the time compared with state-of-the-art methods.

Personalization is a long-standing problem in data mining and machine learning. Companies make personalized product recommendations to millions of users every second. In addition to the recommendation problem, with the emerging of personal devices, many conventional problems, e.g., recognition, need to be personalized as well. Moreover, as the number of users grows huge, solving personalization becomes quite challenging. In this paper, we formalize the generic personalization problem as an optimization problem. We propose several ADMM algorithms to solve this problem in a distributed way including a new Asynchronous ADMM that removes all synchronous barriers to maximize the training throughput. We provide a mathematical analysis to show that the proposed Asynchronous ADMM algorithm holds a linear convergence rate which is the best to our knowledge. The distributed personalization allows training to be performed in either a cluster or even on a user's device. This can improve the privacy protection as no personal data is uploaded, while personal models can still be shared with each other. We apply this approach to two industry problems, \emph{Facial Expression Recognition} and \emph{Job Recommendation}. Experiments demonstrate more than 30\% relative error reduction on both problems. Asynchronous ADMM allows faster training for problems with millions of users since it eliminates all network I/O waiting time to maximize the cluster CPU throughput. Experiments demonstrate 4 times faster than original synchronous ADMM algorithm.

Camera networks have been deployed to facilitate human tracking across multi-cameras in the modern surveillance systems. However, human privacy is an important concern on security surveillance. More specifically, in the real world, surveillance cameras are commonly installed by different entities (such as departments or companies), and any recorded video by one entity should not be shared with others to protect the privacy of tracked humans, while maintaining the knowledge of those moving trajectories of tracked humans in a centralized cloud server. This tracking across multi-cameras information can serve as a very powerful analysis tool for locating crime suspects or collecting business statistics. This paper is the first to aim at the importance of privacy-preserving in a multiple-camera tracking system. We address the problems of privacy-preserving human tracking based on Paillier encryption without revealing any recorded video or data, and introduce the secure multiple-camera system which consists of two stages: training and testing stages. Finally, the security analyses and simulations show the effectiveness of the proposed scheme.

We present sTrack, a system that can track objects across multiple cameras without sharing any visual information between two cameras except whether an object was seen by both. To achieve this challenging privacy goal, we leverage recent advances in secure two-party computation and multi-camera tracking. We derive a new distance metric learning technique that is more suited for secure computation. Compared to the existing methods, our technique has lower complexity in secure computation without sacrificing the tracking accuracy. We implement it using a new Boolean circuit for secure tracking. Experiments using real datasets show that the performance overhead of secure tracking is low, adding only a few seconds over non-private tracking.

A multiple-camera tracking system that tracks humans across cameras with nonoverlapping views is proposed in this paper. The systematically estimated camera link model, including transition time distribution, brightness transfer function, region mapping matrix, region matching weights, and feature fusion weights, is utilized to facilitate consistently labeling the tracked humans. The system is divided into two stages: in the training stage, based on an unsupervised scheme, we formulate the estimation of the camera link model as an optimization problem, in which temporal features, holistic color features, region color features, and region texture features are jointly considered. The deterministic annealing is applied to effectively search the optimal model solutions. The unsupervised learning scheme tolerates the presence of outliers in the training data well. In the testing stage, the systematic integration of multiple cues from the above features enables us to perform an effective reidentification. The camera link model can be continuously updated during tracking in the testing stage to adapt the changes of the environment. Several simulations and comparative studies demonstrate the superiority of our proposed estimation method to the others. Moreover, the complete system has been tested in a small-scale real-world camera network scenario.

We present a self-organized and scalable multiple-camera tracking system that tracks human across the cameras with nonoverlapping views. Given the GPS locations of uncalibrated cameras, the system automatically detects the existence of camera link within the camera network based on the routing information provided by Google Maps. The connected zones in any pair of directly-connected cameras are identified based on the feature matching between the camera's view and Google Street View. The camera link model is further estimated by an unsupervised learning scheme. Finally, multiple-camera tracking is performed. Thanks to the unsupervised pairwise learning and tracking in our system, the camera network is self-organized, and our proposed system is able to be scaled up efficiently when more cameras are added into the network.

Kernel based trackers have been proven to be a promising approach for video object tracking. The use of a single kernel often suffers from occlusion since the available visual information is not sufficient for kernel usage. In order to provide more robust tracking performance, multiple inter-related kernels have thus been utilized for tracking in complicated scenarios. This paper presents an innovative method, which uses projected gradient to facilitate multiple kernels, in finding the best match during tracking under predefined constraints. The adaptive weights are applied to the kernels in order to efficiently compensate the adverse effect introduced by occlusion. An effective scheme is also incorporated to deal with the scale change issue during the object tracking. Moreover, we embed the multiple-kernel tracking into a Kalman filtering-based tracking system to enable fully automatic tracking. Several simulation results have been done to show the robustness of the proposed multiple-kernel tracking and also demonstrate that the overall system can successfully track the video objects under occlusion.

Distributed camera networks have been deployed in the modern surveillance systems. The camera link model, including transition time distribution and brightness transfer function that represent the space-time relationship and color model between two cameras, is a critical element for tracking objects across the cameras. In this paper, we formulate the estimation of the camera link model as an optimization problem, where the deterministic annealing and the barrier method are applied to effectively extract the model parameters. Through the unsupervised scheme, our method utilizes the time stamps and color features together to establish the camera link model in presence of the outliers. Several simulations and comparative studies show the effectiveness of our approach.

In kernel-based video object tracking, the use of single kernel often suffers from the occlusion. In order to provide more robust tracking performance, multiple inter-related kernels have thus been utilized for tracking in complicated scenarios. This paper presents an innovative method that uses projected gradient to facilitate multiple kernels in finding the best match during tracking under predefined constraints. The adaptive weights are also applied to the kernels in order to efficiently compensate the adverse effect introduced by occlusion. An effective scheme is also incorporated to deal with the scale changing issue during the object tracking. Simulation results demonstrate that the proposed method can successfully track the video object under severe occlusion.