Biometric data is considered to be very private and highly sensitive. As such, many methods for biometric template protection were considered over the years — from biohashing and specially crafted feature extraction procedures, to the use of cryptographic solutions such as Fuzzy Commitments or the use of Fully Homomorphic Encryption (FHE).A key question that arises is how much protection these solutions can offer when the adversary can inject samples, and observe the outputs of the system. While for systems that return the similarity score, one can use attacks such as hill-climbing, for systems where the adversary can only learn whether the authentication attempt was successful, this question remained open.In this paper, we show that it is indeed possible to reconstruct the biometric template by just observing the success/failure of the authentication attempt (given the ability to inject a sufficient amount of faces). Our attack achieves negligible template reconstruction loss and enables full recovery of facial images through a generative inversion method, forming a pipeline from binary scores to high-resolution facial images that successfully pass the system more than 98% of the time. Our results are, of course, independent of the protection mechanism used by the system.

Nondestructive detection methods, based on vibrational spectroscopy, are vitally important in a wide range of applications including industrial chemistry, pharmacy and national defense. Recently, deep learning has been introduced into vibrational spectroscopy showing great potential. Different from images, text, etc. that offer large labeled data sets, vibrational spectroscopic data is very limited, which requires novel concepts beyond transfer and meta learning. To tackle this, we propose a task-enhanced augmentation network (TeaNet). The key component of TeaNet is a reconstruction module that inputs randomly masked spectra and outputs reconstructed samples that are similar to the original ones, but include additional variations learned from the domain. These augmented samples are used to train the classification model. The reconstruction and prediction parts are trained simultaneously, end-to-end with back-propagation. Results on both synthetic and real-world datasets verified the superiority of the proposed method. In the most difficult synthetic scenarios TeaNet outperformed CNN by 17%. We visualized and analysed the neuron responses of TeaNet and CNN, and found that TeaNet's ability to identify discriminant wavenumbers was excellent compared to CNN. Our approach is general and can be easily adapted to other domains, offering a solution to more accurate and interpretable few-shot learning.

Cross-modality recognition has many important applications in science, law enforcement and entertainment. Popular methods to bridge the modality gap include reducing the distributional differences of representations of different modalities, learning indistinguishable representations or explicit modality transfer. The first two approaches suffer from the loss of discriminant information while removing the modality-specific variations. The third one heavily relies on the successful modality transfer, could face catastrophic performance drop when explicit modality transfers are not possible or difficult. To tackle this problem, we proposed a compact encoder-decoder neural module (cmUNet) to learn modality-agnostic representations while retaining identity-related information. This is achieved through cross-modality transformation and in-modality reconstruction, enhanced by an adversarial/perceptual loss which encourages indistinguishability of representations in the original sample space. For cross-modality matching, we propose MarrNet where cmUNet is connected to a standard feature extraction network which takes as inputs the modality-agnostic representations and outputs similarity scores for matching. We validated our method on five challenging tasks, namely Raman-infrared spectrum matching, cross-modality person re-identification and heterogeneous (photo-sketch, visible-near infrared and visible-thermal) face recognition, where MarrNet showed superior performance compared to state-of-the-art methods. Furthermore, it is observed that a cross-modality matching method could be biased to extract discriminant information from partial or even wrong regions, due to incompetence of dealing with modality gaps, which subsequently leads to poor generalization. We show that robustness to occlusions can be an indicator of whether a method can well bridge the modality gap. This, to our knowledge, has been largely neglected in the previous works. Our experiments demonstrated that MarrNet exhibited excellent robustness against disguises and occlusions, and outperformed existing methods with a large margin (>10%). The proposed cmUNet is a meta-approach and can be used as a building block for various applications.

Coreset of a given dataset and loss function is usually a small weighed set that approximates this loss for every query from a given set of queries. Coresets have shown to be very useful in many applications. However, coresets' construction is done in a problem-dependent manner and it could take years to design and prove the correctness of a coreset for a specific family of queries. This could limit coresets' use in practical applications. Moreover, small coresets provably do not exist for many problems. To address these limitations, we propose a generic, learning-based algorithm for construction of coresets. Our approach offers a new definition of coreset, which is a natural relaxation of the standard definition and aims at approximating the average loss of the original data over the queries. This allows us to use a learning paradigm to compute a small coreset of a given set of inputs with respect to a given loss function using a training set of queries. We derive formal guarantees for the proposed approach. Experimental evaluation on deep networks and classic machine learning problems show that our learned coresets yield comparable or even better results than the existing algorithms with worst case theoretical guarantees (that may be too pessimistic in practice). Furthermore, our approach applied to deep network pruning provides the first coreset for a full deep network, i.e., compresses all the networks at once, and not layer by layer or similar divide-and-conquer methods.

Model compression is crucial for the deployment of neural networks on devices with limited computational and memory resources. Many different methods show comparable accuracy of the compressed model and similar compression rates. However, the majority of the compression methods are based on heuristics and offer no worst case guarantees on the tradeoff between the compression rate and the approximation error for an arbitrarily new sample. We propose the first efficient structured pruning algorithm with a provable tradeoff between its compression rate and the approximation error for any future test sample. Our method is based on the coreset framework, and it approximates the output of a layer of neurons/filters by a coreset of neurons/filters in the previous layer and discards the rest. We apply this framework in a layer-by-layer fashion from the bottom to the top. Unlike previous works, our coreset is data-independent, meaning that it provably guarantees the accuracy of the function for any input [Formula: see text], including an adversarial one.

We focus on attacks against a biometric authentication system aimed at reconstructing a biometric sample of the subject from the protected template. Such systems include three blocks: feature extraction, binarization, and protection. We propose a new white-box reversing attack on the binarization block that approximates a biometric template given the binary string obtained by the binarization block. The experiments show that the proposed attack reconstructs very accurate approximations that pass the verification threshold when compared to templates produced from the same and different samples of the subject. We then integrate this attack with known attacks on the other two blocks, namely, a variant of a guessing attack to extract the binary string and biometric inversion attack to reconstruct a sample from its template. We instantiate this end-to-end attack on a face authentication system using fuzzy commitments for protection. Facial images reconstructed by the end-to-end attack greatly resemble the original ones. In the simplest attack scenario, more than 83% of these reconstructed templates succeed in unlocking an account (when the system is configured to 0.1% FMR). Even in the “hardest” settings (in which we take a reconstructed image from one system and use it in a different system, with a different feature extraction process) the reconstructed image offers 170 to 210 times higher success rates than the system's FMR.

Biometrics traits such as faces, fingerprints, and irises, are becoming prevalent in computer security applications: from authentication systems to identification systems. Given the sensitive nature of biometrics, a great deal of effort is put into protecting the biometric data after it is acquired - from secure sketch and fuzzy extractors to the use of secure multiparty computations (in protocols such as SCiFI or GSHADE). While these solutions make sure that the extracted values (e.g., binary strings or vectors) that correspond to the biometrics are kept privately and securely, their practical implementations are not optimal with respect to privacy guarantees in the process of extracting the information from the raw biometric data. This paper analyses current solutions for protected systems and discusses the existing and potential problems in the security and privacy of their feature extraction and the binarization processes. As an illustrative example, we show a PoC of an attack on a feature extraction solution from facial images, used in several protected systems, and show that it reveals information which is very close to the training image of the user. As we argue in this paper, other solutions provide privacy for the system's users but make use of external set of biometric data which is often quite large, thus facing privacy and ownership issues associated with the external set of people. The take home message of this paper is: Many of the existing “privacy preserving” solutions neglect the privacy and security aspects of the feature extraction and binarization processes. Hence, we urge future research to close this gap in the security and privacy of biometric systems.

In this work we consider non-linear classifiers that comprise intersections of hyperplanes. We learn these classifiers by minimizing the “minimax” bound over the negative training examples and the hinge type loss of the positive training examples. These classifiers fit typical real-life datasets that consist of a small number of positive data points and a large number of negative data points. Such an approach is computationally appealing since the majority of training examples (belonging to the negative class) are represented by the statistics of their distribution, which is used in a single constraint on the empirical risk, as opposed to SVM, in which the number of variables is equal to the size of the training set. We first focus on intersection of $K$ hyperplanes, for which we provide empirical risk bounds. We show that these bounds are dimensionally independent and decay as $K/\sqrt{m}$ for $m$ samples. We then extend the K-hyperplane mixed risk to the latent mixed risk for training a union of $C$ $K$-hyperplane models, which can form an arbitrary complex, piecewise linear boundaries. We propose efficient algorithms for training the proposed models. Finally, we show how to combine hinge-minimax training with deep architectures and extend it to multi-class settings using transfer learning. The empirical evaluation of the proposed models shows their advantage over the existing methods in a small training labeled data regime.

Recent advances in deep learning (DL) allow for solving complex AI problems that used to be considered very hard. While this progress has advanced many fields, it is considered to be bad news for Completely Automated Public Turing tests to tell Computers and Humans Apart (CAPTCHAs), the security of which rests on the hardness of some learning problems. In this paper, we introduce DeepCAPTCHA, a new and secure CAPTCHA scheme based on adversarial examples, an inherit limitation of the current DL networks. These adversarial examples are constructed inputs, either synthesized from scratch or computed by adding a small and specific perturbation called adversarial noise to correctly classified items, causing the targeted DL network to misclassify them. We show that plain adversarial noise is insufficient to achieve secure CAPTCHA schemes, which leads us to introduce immutable adversarial noise-an adversarial noise that is resistant to removal attempts. In this paper, we implement a proof of concept system, and its analysis shows that the scheme offers high security and good usability compared with the best previously existing CAPTCHAs.

A canonical problem in computer vision is category recognition (e.g., find all instances of human faces, cars etc., in an image). Typically, the input for training a binary classifier is a relatively small sample of positive examples, and a huge sample of negative examples, which can be very diverse, consisting of images from a large number of categories. The difficulty of the problem sharply increases with the dimension and size of the negative example set. We propose to alleviate this problem by applying a "hybrid" classifier, which replaces the negative samples by a prior, and then finds a hyperplane which separates the positive samples from this prior. The method is extended to kernel space and to an ensemble-based approach. The resulting binary classifiers achieve an identical or better classification rate than SVM, while requiring far smaller memory and lower computational complexity to train and apply.

We explore a novel approach to upper bound the misclassification error for problems with data comprising a small number of positive samples and a large number of negative samples. We assign the hinge-loss to upper bound the misclassification error of the positive examples and use the minimax risk to upper bound the misclassification error with respect to the worst case distribution that generates the negative examples. This approach is computationally appealing since the majority of training examples (belonging to the negative class) are represented by the statistics of their distribution, in contrast to kernel SVM which produces a very large number of support vectors in such settings. We derive empirical risk bounds for linear and non-linear classification and show that they are dimensionally independent and decay as 1/\sqrtm for m samples. We propose an efficient algorithm for training an intersection of finite number of hyperplane and demonstrate its effectiveness on real data, including letter and scene recognition.

Biometric authentication is more secure than using regular passwords, as biometrics cannot be "forgotten" and contain high entropy. Thus, many constructions rely on biometric features for authentication, and use them as a source for "good" cryptographic keys. At the same time, biometric systems carry with them many privacy concerns.

We present a novel approach to pose estimation and model-based recognition of specular objects in difficult viewing conditions, such as low illumination, cluttered background, large highlights, and shadows that appear on the object of interest. In such challenging conditions, conventional features are unreliable. We show that under the assumption of a dominant light source, specular highlights produced by a known object can be used to establish correspondence between its image and the 3D model, and to verify the hypothesized pose and the identity of the object. Previous methods that use highlights for recognition make limiting assumptions such as known pose, scene-dependent calibration, simple shape, etc. The proposed method can efficiently recognize free-form specular objects in arbitrary pose and under unknown lighting direction. It uses only a single image of the object as its input and outputs object identity and the full pose. We have performed extensive experiments for both recognition and pose estimation accuracy on synthetic images and on real indoor and outdoor images.

We address the problem of 2D-3D pose estimation in difficult viewing conditions, such as low illumination, cluttered background, and large highlights and shadows that appear on the object of interest. In such challenging conditions conventional features used for establishing correspondence are unreliable. We show that under the assumption of a dominant light source, specular highlights produced by a known object can be used to establish correspondence between its image and the 3D model, and to verify the hypothesized pose. These ideas are incorporated in an efficient method for pose estimation from a monocular image of an object using only highlights produced by the object as its input. The proposed method uses no knowledge of lighting direction and no calibration object for estimating the lighting in the scene. The evaluation of the method shows good accuracy on numerous synthetic images and good robustness on real images of complex, shiny objects, with shadows and difficult backgrounds.

We introduce SCiFI, a system for Secure Computation of Face Identification. The system performs face identification which compares faces of subjects with a database of registered faces. The identification is done in a secure way which protects both the privacy of the subjects and the confidentiality of the database. A specific application of SCiFI is reducing the privacy impact of camera based surveillance. In that scenario, SCiFI would be used in a setting which contains a server which has a set of faces of suspects, and client machines which might be cameras acquiring images in public places. The system runs a secure computation of a face recognition algorithm, which identifies if an image acquired by a client matches one of the suspects, but otherwise reveals no information to neither of the parties. Our work includes multiple contributions in different areas: 1. A new face identification algorithm which is unique in having been specifically designed for usage in secure computation. Nonetheless, the algorithm has face recognition performance comparable to that of state of the art algorithms. We ran experiments which show the algorithm to be robust to different viewing conditions, such as illumination, occlusions, and changes in appearance (like wearing glasses). 2. A secure protocol for computing the new face recognition algorithm. In addition, since our goal is to run an actual system, considerable effort was made to optimize the protocol and minimize its online latency. 3. A system - SCiFI, which implements a secure computation of the face identification protocol. 4. Experiments which show that the entire system can run in near real-time: The secure computation protocol performs a preprocessing of all public-key cryptographic operations. Its online performance therefore mainly depends on the speed of data communication, and our experiments show it to be extremely efficient.

We consider the problem of visual categorization with minimal supervision during training. We propose a partbased model that loosely captures structural information. We represent images as a collection of parts characterized by an appearance codeword from a visual vocabulary and by a neighborhood context, organized in an ordered set of bag-of-features representations. These bags are computed in a local overlapping areas around the part. A semantic distance between images is obtained by matching parts associated with the same codeword using their context distributions. The classification is done using SVM with the kernel obtained from the proposed distance. The experiments show that our method outperforms all the classification methods from the PASCAL challenge on half of the VOC2006 categories and has the best average EER. It also outperforms the constellation model learned via boosting, as proposed by Bar-Hillel et al. on their data set, which contains more rigid objects.

We consider the problem of matching images to tell whether they come from the same scene viewed under different lighting conditions. We show that the surface characteristics determine the type of image comparison method that should be used. Previous work has shown the effectiveness of comparing the image gradient direction for surfaces with material properties that change rapidly in one direction. We show analytically that two other widely used methods, normalized correlation of small windows and comparison of multiscale oriented filters, essentially compute the same thing. Then, we show that for surfaces whose properties change more slowly, comparison of the output of whitening filters is most effective. This suggests that a combination of these strategies should be employed to compare general objects. We discuss indications that Gabor jets use such a mixed strategy effectively, and we propose a new mixed strategy. We validate our results on synthetic and real images.

In this paper we discuss object detection when only a small number of training examples are given. Specifically, we show how to incorporate a simple prior on the distribution of natural images into support vector machines. SVMs are known to be robust to overfitting; however, a few training examples usually do not represent well the structure of the class. Thus the resulting detectors are not robust and highly depend on the choice of the training examples. We incorporate the prior on natural images by requiring that the separating hyperplane will not only yield a wide margin, but also that the corresponding positive half space will have a low probability to contain natural images (the background). Our experiments on real data sets show that the resulting detector is more robust to the choice of training examples, and substantially improves both linear and kernel SVMwhen trained on 10 positive and 10 negative examples.

Lighting variation is commonly handled by methods invariant to additive and multiplicative changes in image intensity. It has been demonstrated that comparing images using the direction of the gradient can produce broader insensitivity to changes in lighting conditions, even for 3D scenes. We analyze two common approaches to image comparison that are invariant, normalized correlation using small correlation windows, and comparison based on a large set of oriented difference of Gaussian filters. We show analytically that these methods calculate a monotonic (cosine) function of the gradient direction difference and hence are equivalent to the direction of gradient method. Our analysis is supported with experiments on both synthetic and real scenes

We describe a novel method for real-time, simultaneous multi-view face detection and facial pose estimation. The method employs a convolu- tional network to map face images to points on a manifold, parametrized by pose, and non-face images to points far from that manifold. This network is trained by optimizing a loss function of three variables: im- age, pose, and face/non-face label. We test the resulting system, in a single configuration, on three standard data sets one for frontal pose, one for rotated faces, and one for profiles and find that its performance on each set is comparable to previous multi-view face detectors that can only handle one form of pose variation. We also show experimentally that the system's accuracy on both face detection and pose estimation is improved by training for the two tasks together. 1 Introduction The detection of human faces in natural images and videos is a key component in a wide variety of applications of human-computer interaction, search and indexing, security, and surveillance. Many real-world applications would profit from multi-view detectors that can detect faces under a wide range of poses: looking left or right (yaw axis), up or down (pitch axis), or tilting left or right (roll axis). In this paper we describe a novel method that not only detects faces independently of their poses, but simultaneously estimates those poses. The system is highly-reliable, runs at near real time (5 frames per second on standard hardware), and is robust against variations in yaw (90), roll (45), and pitch (60). The method is motivated by the idea that multi-view face detection and pose estimation are so closely related that they should not be performed separately. The tasks are related in the sense that they must be robust against the same sorts of variation: skin color, glasses, facial hair, lighting, scale, expressions, etc. We suspect that, when trained together, each task can serve as an inductive bias for the other, yielding better generalization or requiring fewer training examples [2]. To exploit the synergy between these two tasks, we train a convolutional network to map face images to points on a face manifold, and non-face images to points far away from that manifold. The manifold is parameterized by facial pose. Conceptually, we can view the pose parameter as a latent variable that can be inferred through an energy-minimization process [4]. To train the machine we derive a new type of discriminative loss function that is tailored to such detection tasks. Previous Work: Learning-based approaches to face detection abound, including real-time methods [16], and approaches based on convolutional networks [15, 3]. Most multi-view systems take a view-based approach, which involves building separate detectors for differ- ent views and either applying them in parallel [10, 14, 13, 7] or using a pose estimator to select a detector [5]. Another approach is to estimate and correct in-plane rotations before applying a single pose-specific detector [12]. Closer to our approach is that of [8], in which a number of Support Vector Regressors are trained to approximate smooth functions, each of which has a maximum for a face at a particular pose. Another machine is trained to con- vert the resulting values to estimates of poses, and a third is trained to convert the values into a face/non-face score. The resulting system is very slow. 2 Integrating face detection and pose estimation To exploit the posited synergy between face detection and pose estimation, we must design a system that integrates the solutions to the two problems. We hope to obtain better results on both tasks, so this should not be a mere cascaded system in which the answer to one problem is used to assist in solving the other. Both answers must be derived from one underlying analysis of the input, and both tasks must be trained together. Our approach is to build a trainable system that can map raw images X to points in a low-dimensional space. In that space, we pre-define a face manifold F (Z) that we para- meterize by the pose Z. We train the system to map face images with known poses to the corresponding points on the manifold. We also train it to map non-face images to points far away from the manifold. Proximity to the manifold then tells us whether or not an image is a face, and projection to the manifold yields an estimate of the pose. Parameterizing the Face Manifold: We will now describe the details of the parameter- izations of the face manifold. Let's start with the simplest case of one pose parameter Z = , representing, say, yaw. If we want to preserve the natural topology and geometry of the problem, the face manifold under yaw variations in the interval [-90, 90] should be a half circle (with constant curvature). We embed this half-circle in a three-dimensional space using three equally-spaced shifted cosines. Fi() = cos( - i); i = 1, 2, 3; = [- , ] (1) 2 2 When we run the network on an image X, it outputs a vector G(X) with three components that can be decoded analytically into corresponding pose angle: 3 G = arctan i=1 i(X ) cos(i) 3 (2) G i=1 i(X ) sin(i) The point on the manifold closest to G(X) is just F (). The same idea can be applied to any number of pose parameters. Let us consider the set of all faces with yaw in [-90, 90] and roll in [-45, 45]. In an abstract way, this set is isomorphic to a portion of the surface of a sphere. Consequently, we encode the pose with the product of the cosines of the two angles: Fij(, ) = cos( - i) cos( - j); i, j = 1, 2, 3; (3) For convenience we rescale the roll angles to the range of [-90, 90]. With these parame- terizations, the manifold has constant curvature, which ensures that the effect of errors will be the same regardless of pose. Given nine components of the network's output Gij(X), we compute the corresponding pose angles as follows: cc = G G ij ij (X ) cos(i) cos(j ); cs = ij ij (X ) cos(i) sin(j ) sc = G G ij ij (X ) sin(i) cos(j ); ss = ij ij (X ) sin(i) sin(j ) (4) = 0.5(atan2(cs + sc, cc - ss) + atan2(sc - cs, cc + ss)) = 0.5(atan2(cs + sc, cc - ss) - atan2(sc - cs, cc + ss)) Note that the dimension of the face manifold is much lower than that of the embedding space. This gives ample space to represent non-faces away from the manifold. 3 Learning Machine To build a learning machine for the proposed approach we refer to the Minimum Energy Machine framework described in [4]. Energy Minimization Framework: We can view our system as a scalar-value function EW (Y, Z, X), where X and Z are as defined above, Y is a binary label (Y = 1 for face, Y = 0 for a non-face), and W is a parameter vector subject to learning. EW (Y, Z, X) can be interpreted as an energy function that measures the degree of compatibility between X, Z, Y . If X is a face with pose Z, then we want: EW (1, Z, X) EW (0, Z , X) for any pose Z , and EW (1, Z , X) EW (1, Z, X) for any pose Z = Z. Operating the machine consists in clamping X to the observed value (the image), and finding the values of Z and Y that minimize EW (Y, Z, X): (Y , Z) = argminY {Y }, Z{Z}EW (Y, Z, X) (5) where {Y } = {0, 1} and {Z} = [-90, 90][-45, 45] for yaw and roll variables. Although this inference process can be viewed probabilistically as finding the most likely configu- ration of Y and Z according to a model that attributes high probabilities to low-energy configurations (e.g. a Gibbs distribution), we view it as a non probabilistic decision mak- ing process. In other words, we make no assumption as to the finiteness of integrals over {Y }and {Z}that would be necessary for a properly normalized probabilistic model. This gives us considerable flexibility in the choice of the internal architecture of EW (Y, Z, X). Our energy function for a face EW (1, Z, X) is defined as the distance between the point produced by the network GW (X) and the point with pose Z on the manifold F (Z): EW (1, Z, X) = GW (X) - F (Z) (6) The energy function for a non-face EW (0, Z, X) is equal to a constant T that we can interpret as a threshold (it is independent of Z and X). The complete energy function is: EW (Y, Z, X) = Y GW (X) - F (Z) + (1 - Y )T (7) The architecture of the machine is depicted in Figure 1. Operating this machine (find- ing the output label and pose with the smallest energy) comes down to first finding: Z = argminZ{Z}||GW (X) - F (Z)||, and then comparing this minimum distance, GW (X) - F (Z) , to the threshold T . If it smaller than T , then X is classified as a face, otherwise X is classified as a non-face. This decision is implemented in the architecture as a switch, that depends upon the binary variable Y . Convolutional Network: We employ a Convolutional Network as the basic architecture for the GW (X) image-to-face-space mapping function. Convolutional networks [6] are "end- to-end" trainable system that can operate on raw pixel images and learn low-level features and high-level representation in an integrated fashion. Convolutional nets are advantageous because they easily learn the types of shift-invariant local features that are relevant to image recognition; and more importantly, they can be replicated over large images (swept over every location) at a fraction of the cost of replicating more traditional classifiers [6]. This is a considerable advantage for building real-time systems. We employ a network architecture similar to LeNet5 [6]. The difference is in the number of maps. In our architecture we have 8 feature maps in the bottom convolutional and subsampling layers and 20 maps in the next two layers. The last layer has 9 outputs to encode two pose parameters. Training with a Discriminative Loss Function for Detection: We define the loss function as follows: 1 1 L(W ) = L L |S 1(W, Zi, X i) + 0(W, X i) (8) 1| |S iS 0| 1 iS0 Figure 1: Architecture of the Minimum Energy Machine. where S1is the set of training faces, S0the set of non-faces, L1(W, Zi, Xi) and L0(W, Xi) are loss functions for a face sample (with a known pose) and non-face, respectively1. The loss L(W ) should be designed so that its minimization for a particular positive training sample (Xi, Zi, 1), will make EW (1, Zi, Xi) < EW (Y, Z, Xi) for Y = Y i or Z = Zi. To satisfy this, it is sufficient to make EW (1, Zi, Xi) < EW (0, Z, Xi). For a particu- lar negative training sample (Xi, 0), minimizing the loss should make EW (1, Z, Xi) > EW (0, Z, Xi) = T for any Z. To satisfy this, it is sufficient to make EW (1, Z, Xi) > T . Let W be the current parameter value, and W be the parameter value after an update caused by a single sample. To cause the machine to achieve the desired behavior, we need the parameter update to decrease the difference between the energy of the desired label and the energy of the undesired label. In our case, since EW (0, Z, X) = T is constant, the following condition on the update is sufficient to ensure the desired behavior: Condition 1. for a face example (X, Z, 1), we must have: EW (1, Z, X) < EW (1, Z, X) For a non-face example (X, 1), we must have: EW (1, Z, X) > EW (1, Z, X) We choose the following forms for L1 and L0: L1(W, 1, Z, X) = EW (1, Z, X)2; L0(W, 0, X) = K exp[-E(1, Z, X)] (9) where K is a positive constant. Next we show that minimizing (9) with an incremental gradient-based algorithm will satisfy condition 1. With gradient-based optimization algorithms, the parameter update formula is of the form: W = W - W = -A L W . where A is a judiciously chosen symmetric positive semi-definite matrix, and is a small positive constant. For Y = 1 (face): An update step will change the parameter by W = -A EW (1,Z,X)2 = W -2EW (1, Z, X)A EW (1,Z,X) W . To first order (for small values of ), the resulting change in EW (1, Z, X) is given by: E T T W (1, Z, X ) E E W = -2E W (1, Z, X ) A W (1, Z, X ) < 0 W W (1, Z, X ) W W because EW (1, Z, X) > 0 (it's a distance), and the quadratic form is positive. Therefore EW (1, Z, X) < EW (1, Z, X). 1Although face samples whose pose is unknown can easily be accommodated, we will not discuss this possibility here. 100 100 95 95 90 90 85 85 80 80 75 75 70 70 65 65 Percentage of faces detected 60 Pose + detection 60 Pose + detection Detection only Pose only 55 Percentage of yaws correctly estimated 55 50 50 0 2 4 6 8 10 12 14 16 18 20 0 5 10 15 20 25 30 False positive rate Yaw-error tolerance (degrees) Figure 2: Synergy test. Left: ROC curves for the pose-plus-detection and detection-only networks. Right: frequency with which the pose-plus-detection and pose-only networks correctly estimated the yaws within various error tolerances. For Y = 0 (non-face): An update step will change the parameter by W = -A K exp[-E(1,Z,X)] = K exp[-E W W (1, Z, X )] EW (1,Z,X) W . To first order (for small values of ), the resulting change in EW (1, Z, X) is given by: E T T W (1, Z, X ) E E W = K exp[-E W (1, Z , X ) A W (1, Z , X ) > 0 W W (1, Z , X )] W W Therefore EW (1, Z, X) > EW (1, Z, X). Running the Machine: Our detection system works on grayscale images and it applies the network to each image at a range of scales, stepping by a factor of 2. The network is replicated over the image at each scale, stepping by 4 pixels in x and y (this step size is a consequence of having two, 2x2 subsampling layers). At each scale and location, the network outputs are compared to the closest point on the manifold, and the system collects a list of all instances closer than our detection threshold. Finally, after examining all scales, the system identifies groups of overlapping detections in the list and discards all but the strongest (closest to the manifold) from each group. No attempt is made to combine detections or apply any voting scheme. We have implemented the system in C. The system can detect, locate, and estimate the pose of faces that are between 40 and 250 pixels high in a 640 480 image at roughly 5 frames per second on a 2.4GHz Pentium 4. 4 Experiments and results Using the above architecture, we built a detector to locate faces and estimate two pose parameters: yaw from left to right profile, and in-plane rotation from -45 to 45 degrees. The machine was trained to be robust against pitch variation. In this section, we first describe the training regimen for this network, and then give the results of two sets of experiments. The first set of experiments tests whether training for the two tasks together improves performance on both. The second set allows comparisons between our system and other published multi-view detectors. Training: Our training set consisted of 52, 850, 32x32-pixel faces from natural images collected at NEC Labs and hand annotated with appropriate facial poses (see [9] for a description of how the annotation was done). These faces were selected from a much larger annotated set to yield a roughly uniform distribution of poses from left profile to right profile, with as much variation in pitch as we could obtain. Our initial negative training data consisted of 52, 850 image patches chosen randomly from non-face areas of a variety of images. For our second set of tests, we replaced half of these with image patches obtained by running the initial version of the detector on our training images and collecting false detections. Each training image was used 5 times during training, with random variations 100 100 95 95 90 90 85 85 80 80 75 75 70 70 65 65 Frontal Percentage of faces detected 60 Rotated in plane 60 In-plane rotation Profile Yaw 55 Percentage of poses correctly estimated 55 50 50 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 0 5 10 15 20 25 30 False positives per image Pose-error tolerance (degrees) Figure 3: Results on standard data sets. Left: ROC curves for our detector on the three data sets. The x axis is the average number of false positives per image over all three sets, so each point corresponds to a single detection threshold. Right: frequency with which yaw and roll are estimated within various error tolerances. in scale (from x 2 to x(1 + 2)), in-plane rotation (45), brightness (20), contrast (from 0.8 to 1.3). To train the network, we made 9 passes through this data, though it mostly converged after about the first 6 passes. Training was performed using LUSH [1], and the total training time was about 26 hours on a 2Ghz Pentium 4. At the end of training, the network had converged to an equal error rate of 5% on the training data and 6% on a separate test set of 90,000 images. Synergy tests: The goal of the synergy test was to verify that both face detection and pose estimation benefit from learning and running in parallel. To test this claim we built three networks with almost identical architectures, but trained to perform different tasks. The first one was trained for simultaneous face detection and pose estimation (combined), the second was trained for detection only and the third for pose estimation only. The "detection only" network had only one output for indicating whether or not its input was a face. The "pose only" network was identical to the combined network, but trained on faces only (no negative examples). Figure 2 shows the results of running these networks on our 10,000 test images. In both these graphs, we see that the pose-plus-detection network had better performance, confirming that training for each task benefits the other. Standard data sets: There is no standard data set that tests all the poses our system is designed to detect. There are, however, data sets that have been used to test more restricted face detectors, each set focusing on a particular variation in pose. By testing a single detector with all of these sets, we can compare our performance against published systems. As far as we know, we are the first to publish results for a single detector on all these data sets. The details of these sets are described below: MIT+CMU [14, 11] 130 images for testing frontal face detectors. We count 517 faces in this set, but the standard tests only use a subset of 507 faces, because 10 faces are in the wrong pose or otherwise not suitable for the test. (Note: about 2% of the faces in the standard subset are badly-drawn cartoons, which we do not intend our system to detect. Nevertheless, we include them in the results we report.) TILTED [12] 50 images of frontal faces with in-plane rotations. 223 faces out of 225 are in the standard subset. (Note: about 20% of the faces in the standard subset are outside of the 45 rotation range for which our system is designed. Again, we still include these in our results.) PROFILE [13] 208 images of faces in profile. There seems to be some disagreement about the number of faces in the standard set of annotations: [13] reports using 347 faces of the 462 that we found, [5] reports using 355, and we found 353 annotations. However, these discrepencies should not significantly effect the reported results. We counted a face as being detected if 1) at least one detection lay within a circle centered on the midpoint between the eyes, with a radius equal to 1.25 times the distance from that point to the midpoint of the mouth, and 2) that detection came at a scale within a factor of Figure 4: Some example face detections. Each white box shows the location of a detected face. The angle of each box indicates the estimated in-plane rotation. The black crosshairs within each box indicate the estimated yaw. Data set TILTED PROFILE MIT+CMU False positives per image 4.42 26.90 .47 3.36 .50 1.28 Our detector 90% 97% 67% 83% 83% 88% Jones & Viola [5] (tilted) 90% 95% x x Jones & Viola [5] (profile) x 70% 83% x Rowley et al [11] 89% 96% x x Schneiderman & Kanade [13] x 86% 93% x Table 1: Comparisons of our results with other multi-view detectors. Each column shows the detec- tion rates for a given average number of false positives per image (these rates correspond to those for which other authors have reported results). Results for real-time detectors are shown in bold. Note that ours is the only single detector that can be tested on all data sets simultaneously. two of the correct scale for the face's size. We counted a detection as a false positive if it did not lie within this range for any of the faces in the image, including those faces not in the standard subset. The left graph in Figure 3 shows ROC curves for our detector on the three data sets. Figure 4 shows a few results on various poses. Table 1 shows our detection rates compared against other systems for which results were given on these data sets. The table shows that our results on the TILTED and PROFILE sets are similar to those of the two Jones & Viola detectors, and even approach those of the Rowley et al and Schneiderman & Kanade non- real-time detectors. Those detectors, however, are not designed to handle all variations in pose, and do not yield pose estimates. The right side of Figure 3 shows our performance at pose estimation. To make this graph, we fixed the detection threshold at a value that resulted in about 0.5 false positives per image over all three data sets. We then compared the pose estimates for all detected faces (including those not in the standard subsets) against our manual pose annotations. Note that this test is more difficult than typical tests of pose estimation systems, where faces are first localized by hand. When we hand-localize these faces, 89% of yaws and 100% of in-plane rotations are correctly estimated to within 15.

This paper offers a natural extension of the newly introduced "anti-face" method to event detection, in both the gray-level and feature domains. For the gray-level domain, spatio-temporal templates are created by stacking the individual frames of the video sequence, and the detection is performed on these templates. In order to recognize the motion of features in a video sequence, the spatial locations of the features are modulated in time, thus creating a one-dimensional vector which represents the event in the detection process. The following applications are presented: 1) detection of an object under three-dimensional (3-D) rotations in a video sequence simulated from the COIL database; 2) visual recognition of spoken words; and 3) recognition of two-dimensional and 3-D sketched curves. The technique is capable of detecting 3-D curves in viewing directions which substantially differ from those in the training set. The resulting detection algorithm is very fast and can successfully detect events even in very low resolution. Also, it is capable of discriminating the desired event from arbitrary events, and not only from those in a negative training set. Possible applications of the techniques offered in this paper are in man-machine interaction, surveillance, and search and summarization in video databases.

Recognition systems have generally treated specular highlights as noise. We show how to use these highlights as a positive source of information that improves recognition of shiny objects. This also enables us to recognize very challenging shiny transparent objects, such as wine glasses. Specifically, we show how to find highlights that are consistent with a hypothesized pose of an object of known 3D shape. We do this using only a qualitative description of highlight formation that is consistent with most models of specular reflection, so no specific knowledge of an object's reflectance properties is needed. We first present a method that finds highlights produced by a dominant compact light source, whose position is roughly known. We then show how to estimate the lighting automatically for objects whose reflection is part specular and part Lambertian. We demonstrate this method for two classes of objects. First, we show that specular information alone can suffice to identify objects with no Lambertian reflectance, such as transparent wine glasses. Second, we use our complete system to recognize shiny objects, such as pottery.

This paper presents a natural extension of the newly introduced "anti-face" method to event detection, both in the image and in the feature domains. In the case of the image domain (video sequences) we create spatio temporal templates by stacking the video frames, and the detection is performed on these templates. In order to recognize the motion of features in a video sequence, the spatial locations of the features are modulated in time, thus creating a one-dimensional vector which re resents the event The following applications of anti-sequences are presented 1) Detection of an object under 3D rotations in a video sequence simulated from the COIL database, 2) Visual speech recognition of spoken words, and 3) Recognition of symbols sketched with a laser pointer. The resulting detection algorithm is very fast, and is robust enough to work on small images. Also, it is capable of discriminating the desired event-template from arbitrary events, and not only events in a "negative training set".

This paper focuses on the detection of objects with Lambertian surface under both varying illumination and pose. We offer to apply a novel detection method that proceeds by modeling the different illuminations from a small number of images in the training set; this automatically voids the illumination effects, allowing fast illumination invariant detection, without having to create a large training set. It is demonstrated that the method "fits in" nicely with previous work about the modeling of the set of object appearances under varying illumination. In the experiments, an object was correctly detected under image plane rotations in a 15-degrees range, and a wide variety of different illuminations.

This paper offers a novel detection method, which works well even in the case of a complicated image collection. It can also be applied to detect 3D objects under different views. The detection problem is solved by sequentially applying very simple filters (or detectors), which are designed to yield small results on the multitemplate (hence antifaces), and large results on "random" natural images. This is achieved by making use of a simple probabilistic assumption on the distribution of natural images, which is borne out well in practice. Only images which passed the threshold test imposed by the first detector are examined by the second detector, etc. The detectors are designed to act independently so that their false alarms are uncorrelated; this results in a false alarm rate which decreases exponentially in the number of detectors. The algorithm's performance compares favorably to the well-known eigenface and support vector machine based algorithms, but is substantially faster.