We present MOFI, Manifold OF Images, a new vision foundation model designed to learn image representations from noisy entity annotated images. MOFI differs from previous work in two key aspects: 1. pre-training data, and 2. training recipe. Regarding data, we introduce a new approach to automatically assign entity labels to images from noisy image-text pairs. Our approach involves employing a named entity recognition model to extract entities from the alt-text, and then using a CLIP model to select the correct entities as labels of the paired image. It's a simple, cost-effective method that can scale to handle billions of web-mined image-text pairs. Through this method, we have created Image-to-Entities (I2E), a new dataset with 1 billion images and 2 million distinct entities, covering rich visual concepts in the wild. Building upon the I2E dataset, we study different training recipes like supervised pre-training, contrastive pre-training, and multi-task learning. For constrastive pre-training, we treat entity names as free-form text, and further enrich them with entity descriptions. Experiments show that supervised pre-training with large-scale fine-grained entity labels is highly effective for image retrieval tasks, and multi-task training further improves the performance. The final MOFI model achieves 86.66\% mAP on the challenging GPR1200 dataset, surpassing the previous state-of-the-art performance of 72.19% from OpenAI's CLIP model. Further experiments on zero-shot and linear probe image classification also show that MOFI outperforms a CLIP model trained on the original image-text data, demonstrating the effectiveness of the I2E dataset in learning strong image representations. We release our code and model weights at https://github.com/apple/ml-mofi.

As an upgrade for H.264/AVC, high efficiency video coding (HEVC) achieves 50% bitrate reduction under the equivalent visual quality. However, high computational complexity increases dramatically for adopting up to 35 intra prediction modes. To deal with this issue, we explore spatial-temporal correlation between PUs to narrow rough mode decision (RMD) candidate list and rate distortion optimization (RDO) candidate list respectively. A fast PU intra mode decision scheme is proposed for HEVC fast intra encoding. For RMD, the proposed scheme early determines the impossible range of the PU optimal mode in line with spatial statistical analysis theory. It would narrow RMD candidate list and speed up RMD process. First, 4 subsets (Sb1, Sb2, Sb3 and Sb4) are defined in mode set with 33 directional prediction modes as shown in Figure 1. And then, on the basis of spatial statistical analysis theory, the impossible range of the parent PU optimal mode would be judged by its known optimal mode. Last, low probability subsets of current PU are removed, and candidate subsets are determined for current PU. Where ModeParBest and ModeCurBest are the parent PU optimal mode and the current PU optimal mode respectively. For RDO, the proposed scheme tries to combine temporal correlation with intra coding, which adds co-located optimal mode of the previous frame into the RDO list of the current PU. At the same time, the modes in RDO list are decreased focusing on most time-consuming PUs (4 4 PUs, 8 8 PUs) from 8 candidate modes to 3 candidate modes. Experimental results demonstrate that the proposed scheme yields average 29% encoding time reduction with average 1.19% BDBR gain and 0.06dB BDPSNR loss compared with HM16.9. Further, the proposed method implements fast CU encoding without additional computation during the encoding process.

User interaction provides useful information for solving challenging computer vision problems in practice. In this paper, we show that a very limited number of user clicks could greatly boost monocular depth estimation performance and overcome monocular ambiguities. We formulate this task as a deep structured model, in which the structured pixel-wise depth estimation has ordinal constraints introduced by user clicks. We show that the inference of the proposed model could be efficiently solved through a feed-forward network. We demonstrate the effectiveness of the proposed model on NYU Depth V2 and Stanford 2D-3D datasets. On both datasets, we achieve state-of-the-art performance when encoding user interaction into our deep models.

We present a multimodal sensor system for wound assessment and pressure ulcer care. Multiple imaging modalities including RGB, three- dimensional (3-D) depth, thermal, multispectral, and chemical sensing are integrated into a portable hand-held probe for real-time wound assessment. Analytic and quantitative algorithms for various assessments including tissue composition, wound measurement in 3-D, temperature profiling, spectral, and chemical vapor analysis are developed. After each assessment scan, 3-D models of the wound are generated on the fly for geometric measurement, while multimodal observations are analyzed to estimate healing progress. Collaboration between developers and clinical practitioners was conducted at the Charlie Norwood VA Medical Center for in-field data collection and experimental evaluation. A total of 133 assessment sessions from 23 enrolled subjects were collected, on which the multimodal data were analyzed and validated with respect to clinical notes associated with each subject. The system can be operated by nontechnical caregivers on a regular basis to aid wound assessment and care. A web portal front-end was developed for clinical decision and telehealth support, where all historical patient data including wound measurements and analysis can be organized online.

We present a real-time depth based computer vision system for pressure ulcer prevention, in-bed patient care and monitoring. Our system can effectively determine whether or not a mobility-compromised patient has been correctly repo-sitioned at the required frequency. A depth sensor is used to detect and recognize patient movements, motion patterns, and pose positions. If the patient has stayed in an unchanged pose for too long and needs pressure releasing movements, our system can notify caregivers for repositioning or assistance. Privacy concerns are mitigated by removing the RGB components of the video stream from the camera capturing, and only processing depth measurements. We collaborated with clinical practitioners at the Charlie Norwood VA Medical Center for in-field data collection and experimental evaluation. A web portal front-end is developed such that all historical patient movements, pose positions, and repositioning data can be organized to support telehealth applications.

Drift is the most difficult issue in object visual tracking based on framework of “tracking-by-detection”. Due to the self-taught learning, the mis-aligned samples are potentially to be incorporated in learning and degrade the discrimination of the tracker. This paper proposes a new tracking approach that resolves this problem by three multi-level collaborative components: a high-level global appearance tracker provides a basic prediction, upon which the structure preserved low-level local patches matching helps to guarantee precise tracking with minimized drift. Those local patches are deliberately deployed on the foreground object via foreground/background segmentation, which is realized by a simple and efficient classifier trained by super-pixel segments. Experimental results show that the three closely collaborated components enable our tracker runs in real time and performs favourably against state-of-the-art approaches on challenging benchmark sequences.

Egocentric vision has received increasing attention in recent years due to the vast development of wearable devices and their applications. Although there are numerous existing work on egocentric vision, none of them solve Optical Music Recognition (OMR) problem. In this paper, we propose a novel optical music recognition approach for egocentric device (e.g. Google Glass) with the assistance of MIDI data. We formulate the problem as a structured sequence alignment problem as opposed to the blind recognition in traditional OMR systems. We propose a linear-chain Conditional Random Field (CRF) to model the note event sequence, which translates the relative temporal relations contained by MIDI to spatial constraints over the egocentric observation. We performed evaluations to compare the proposed approach with several different baselines and proved that our approach achieved the highest recognition accuracy. We view our work as the first step towards egocentric optical music recognition, and believe it will bring insights for next-generation music pedagogy and music entertainment.

In this paper, our goal is to speed up a standard sliding window detector while maintaining detection accuracies. We do this by decomposing its weight template into multiple component detectors with no redundancy. Each component detector captures partial discriminative appearance, and they can be used in a cascade detection pipeline to aggressively reduce the size of search space. We formulate the decomposition problem as an orthogonal procrustes problem. We further approximate the component detector in the first cascade layer as separable 2D filters (i.e. a product between two vector filters). The running time of such component detector is thus reduced from quadratic to linear with the dimension lengths of the detector template. We conduct extensive experiments using our approach on two well-known object detection datasets: INRIA pedestrian detection dataset and PASCAL VOC 2007 detection dataset. We used HOG features and CNN features in our experiments. Our approach is almost "free lunch": without manipulating the feature representations, it makes the detection process several times faster.

In this paper, we propose a new approach for the person re-identification problem, discovering the correct matches for a query pedestrian image from a set of gallery images. It is well motivated by our observation that the overall complex inter-camera transformation, caused by the change of camera viewpoints, person poses and view illuminations, can be effectively modelled by a combination of many simple local transforms, which guides us to learn a set of more specific local metrics other than a fixed metric working on the feature vector of a whole image. Given training images in pair, we first align the local patches using spatially constrained dense matching. Then, we use a decision tree structure to partition the space of the aligned local patch-pairs into different configurations according to the similarity of the local cross-view transforms. Finally, a local metric kernel is learned for each configuration at the tree leaf nodes in a linear regression manner. The pairwise distance between a query image and a gallery image is summarized based on all the pairwise distance of local patches measured by different local metric kernels. Multiple decision trees form the proposed random kernel forest, which always discriminatively assign the optimal local metric kernel to the local image patches in re-identification. Experimental results over the public benchmarks demonstrate the effectiveness of our approach for achieving very competitive performances with a relatively simpler learning scheme.

Photo-sharing websites have become very popular in the last few years, leading to huge collections of online images. In addition to image data, these websites collect a variety of multimodal metadata about photos including text tags, captions, GPS coordinates, camera metadata, user profiles, etc. However, this metadata is not well constrained and is often noisy, sparse, or missing altogether. In this paper, we propose a framework to model these "loosely organized" multimodal datasets, and show how to perform loosely-supervised learning using a novel latent Conditional Random Field framework. We learn parameters of the LCRF automatically from a small set of validation data, using Information Theoretic Metric Learning (ITML) to learn distance functions and a structural SVM formulation to learn the potential functions. We apply our framework on four datasets of images from Flickr, evaluating both qualitatively and quantitatively against several baselines.

Vehicle recognition is a challenging task with many useful applications. State-of-the-art methods usually learn discriminative classifiers for different vehicle categories or different viewpoint angles, but little work has explored vehicle recognition using semantic visual attributes. In this paper, we propose a novel iterative multiple instance learning method to model local attributes and viewpoint angles together in the same framework. We expand the standard MISVM formulation to incorporate pairwise constraints based on viewpoint relations within positive exemplars. We show that our method is able to generate discriminative and semantic local attributes for vehicle categories. We also show that we can estimate viewpoint labels more accurately than baselines when these annotations are not available in the training set. We test the technique on the Stanford cars and INRIA vehicles datasets, and compare with other methods.

We introduce a new approach for part-based human pose estimation using multi-layer composite models, in which each layer is a tree-structured pictorial structure that models pose at a different scale and with a different graphical structure. At the highest level, the submodel acts as a person detector, while at the lowest level, the body is decomposed into a collection of many local parts. Edges between adjacent layers of the composite model encode cross-model constraints. This multi-layer composite model is able to relax the independence assumptions of traditional tree-structured pictorial-structure models while permitting efficient inference using dual-decomposition. We propose an optimization procedure for joint learning of the entire composite model. Our approach outperforms the state-of-the-art on the challenging Parse and UIUC Sport datasets.

Attributes are visual concepts that can be detected by machines, understood by humans, and shared across categories. They are particularly useful for fine-grained domains where categories are closely related to one other (e.g. bird species recognition). In such scenarios, relevant attributes are often local (e.g. “white belly”), but the question of how to choose these local attributes remains largely unexplored. In this paper, we propose an interactive approach that discovers local attributes that are both discriminative and semantically meaningful from image datasets annotated only with fine-grained category labels and object bounding boxes. Our approach uses a latent conditional random field model to discover candidate attributes that are detectable and discriminative, and then employs a recommender system that selects attributes likely to be semantically meaningful. Human interaction is used to provide semantic names for the discovered attributes. We demonstrate our method on two challenging datasets, Caltech-UCSD Birds-200-2011 and Leeds Butterflies, and find that our discovered attributes outperform those generated by traditional approaches.