Optical character recognition and machine translation are usually studied and applied separately. In this paper, we consider a new problem named cross-lingual text image recognition (CLTIR) that integrates these two tasks together. The core of this problem is to recognize source language texts shown in images and transcribe them to the target language in an end-to-end manner. Traditional cascaded systems perform text image recognition and text translation sequentially. This can lead to error accumulation and parameter redundancy problems. To overcome these problems, we propose a multihierarchy cross-modal mimic (MHCMM) framework for end-to-end CLTIR, which can be trained with a massive bilingual text corpus and a small number of bilingual annotated text images. In this framework, a plug-in machine translation model is used as a teacher to guide the CLTIR model for learning representations compatible with image and text modes. Via adversarial learning and attention mechanisms, the proposed mimic method can integrate both global and local information in the semantic space. Experiments on a newly collected dataset demonstrate the superiority of the proposed framework. Our method outperforms other pipelines while containing fewer parameters. Additionally, the MHCMM framework can utilize a large-scale bilingual corpus to further improve the performance efficiently. The visualization of attention scores indicates that the proposed model can read text images in a fashion similar to the machine translation model reading text tokens.

Despite the success of convolutional neural network (CNN) in conventional closed-set recognition (CSR), it still lacks robustness for dealing with unknowns (those out of known classes) in open environment. To improve the robustness of CNN in open-set recognition (OSR) and meanwhile maintain its high accuracy in CSR, we propose an alternative deep framework called convolutional prototype network (CPN), which keeps CNN for representation learning but replaces the closed-world assumed softmax with an open-world oriented and human-like prototype model. To equip CPN with discriminative ability for classifying known samples, we design several discriminative losses for training. Moreover, to increase the robustness of CPN for unknowns, we interpret CPN from the perspective of generative model and further propose a generative loss, which is essentially maximizing the log-likelihood of known samples and serves as a latent regularization for discriminative learning. The combination of discriminative and generative losses makes CPN a hybrid model with advantages for both CSR and OSR. Under the designed losses, the CPN is trained end-to-end for learning the convolutional network and prototypes jointly. For application of CPN in OSR, we propose two rejection rules for detecting different types of unknowns. Experiments on several datasets demonstrate the efficiency and effectiveness of CPN for both CSR and OSR tasks.

This paper considers recognizing texts shown in a source language and translating into a target language, without generating the intermediate source language text image recognition results. We call this problem Cross-Lingual Text Image Recognition (CLTIR). To solve this problem, we propose a multi-task system containing a main task of CLTIR and an auxiliary task of Mono-Lingual Text Image Recognition (MLTIR) simultaneously. Two different sequence to sequence learning methods, a convolution based attention model and a Bidirectional Long Short-Term Memory (BLSTM) model with Connectionist Temporal Classification (CTC), are adopted for these tasks respectively. We evaluate the system on a newly collected Chinese-English bilingual movie subtitle image dataset. Experimental results demonstrate the multi-task learning framework performs superiorly in both languages.

Classifying strokes into different categories is an essential preprocessing step in the automatic document understanding process. To tackle this task, it is crucial to integrate different types of contextual information. Previous methods which are based on conditional random fields or recurrent neural networks have some limitations in model capacity or computational cost. In this paper, we propose a novel framework based on graph attention networks to solve this problem, which casts the stroke classification problem into the node classification problem in a document graph. In the graph, each node represents a stroke and the edges are built from temporal and spatial interactions between strokes. Combined graph convolution with attention mechanisms to dynamically aggregate features from the neighborhood, our model is very flexible to control the message passing routine between different nodes and therefore has strong capability learning context-aware features. We perform comparison experiments on the IAMonDo dataset and experimental results demonstrate the superiority of our approach.

Oracle character is one kind of the earliest hieroglyphics, which can be dated back to Shang Dynasty in China. Oracle character recognition is important for modern archaeology, ancient text understanding, and historical chronology, etc. To overcome the limitation and class imbalance of training data in oracle character recognition, we propose a classification method based on deep metric learning. We use a convolutional neural network (CNN) to map the character images to an Euclidean space where the distance between different samples can measure their similarities such that classification can be performed by the Nearest Neighbor (NN) rule. Because new categories are still being discovered in reality, our model enables the rejection of unseen categories and the configuration of new categories. To accelerate NN classification, we also propose a prototype pruning method with little loss of accuracy. The proposed method exceeds the state of the art on the public dataset Oracle-20K and outperforms CNN with softmax layer on a new dataset Oracle-AYNU.

Most existing hashing methods resort to binary codes for large scale similarity search, owing to the high efficiency of computation and storage. However, binary codes lack enough capability in similarity preservation, resulting in less desirable performance. To address this issue, we propose Nonlinear Asymmetric Multi-Valued Hashing (NAMVH) supported by two distinct non-binary embeddings. Specifically, a real-valued embedding is used for representing the newly-coming query by an ideally nonlinear transformation. Besides, a multi-integer-embedding is employed for compressing the whole database, which is modeled by Binary Sparse Representation (BSR) with fixed sparsity. With these two non-binary embeddings, NAMVH preserves more precise similarities between data points and enables access to the incremental extension with database samples evolving dynamically. To perform meaningful asymmetric similarity computation for efficient semantic search, these embeddings are jointly learnt by preserving the pairwise label-based similarity. Technically, this results in a mixed integer programming problem, which is efficiently solved by a well-designed alternative optimization method. Extensive experiments on seven large scale datasets demonstrate that our approach not only outperforms the existing binary hashing methods in search accuracy, but also retains their query and storage efficiency.

Collaborative Filtering (CF) is the most popular method for recommender systems. The principal idea of CF is that users might be interested in items that are favorited by similar users, and most of the existing CF methods measure users' preferences by their behaviours over all the items. However, users might have different interests over different topics, thus might share similar preferences with different groups of users over different sets of items. In this paper, we propose a novel and scalable method CCCF which improves the performance of CF methods via user-item co-clustering. CCCF first clusters users and items into several subgroups, where each subgroup includes a set of like-minded users and a set of items in which these users share their interests. Then, traditional CF methods can be easily applied to each subgroup, and the recommendation results from all the subgroups can be easily aggregated. Compared with previous works, CCCF has several advantages including scalability, flexibility, interpretability and extensibility. Experimental results on four real world data sets demonstrate that the proposed method significantly improves the performance of several state-of-the-art recommendation algorithms.

This paper provides a deep analysis of user retweet behavior on Twitter. While previous works about analyzing retweet have mainly focused on predicting the retweetability of each tweet, they lacked interpretations at an individual level. In this paper, we perform a general analysis of retweet behavior from the perspective of individual users. Specifically, we train a prediction model to forecast whether a tweet will be retweeted by a given user, leveraging four different types of features: social-based, content-based, tweet-based and author-based features. By performing “leave-one-feature-out” comparisons, we identify factors that are strongly associated with user retweet behavior.

User generated content is the basic element of social media websites. Relatively few studies have systematically analyzed the motivation to create and share content, especially from the perspective of a common user. In this paper, we perform a comprehensive analysis of user posting behavior on a popular social media website, Twitter. Specifically, we assume that user behavior is mainly influenced by three factors: breaking news, posts from social friends and user's intrinsic interest, and propose a mixture latent topic model to combine all these factors. We evaluated our model on a large-scale Twitter dataset from three different perspectives: the perplexity of held-out content, the performance of predicting retweets and the quality of generated latent topics. The results were encouraging, our model clearly outperformed its competitors.

This paper focuses on the problem of discovering users' topics of interest on Twitter. While previous efforts in modeling users' topics of interest on Twitter have focused on building a "bag-of-words" profile for each user based on his tweets, they overlooked the fact that Twitter users usually publish noisy posts about their lives or create conversation with their friends, which do not relate to their topics of interest. In this paper, we propose a novel framework to address this problem by introducing a modified author-topic model named twitter-user model. For each single tweet, our model uses a latent variable to indicate whether it is related to its author's interest. Experiments on a large dataset we crawled using Twitter API demonstrate that our model outperforms traditional methods in discovering user interest on Twitter.

In this paper, we propose a novel partwise framework for cross-parameterization between 3D mesh models. Unlike most existing methods that use regular parameterization domains, our framework uses nonregular approximation domains to build the cross-parameterization. Once the nonregular approximation domains are constructed for 3D models, different (and complex) input shapes are transformed into similar (and simple) shapes, thus facilitating the cross-parameterization process. Specifically, a novel nonregular domain, the convex hull, is adopted to build shape correspondence. We first construct convex hulls for each part of the segmented model, and then adopt our convex-hull cross-parameterization method to generate compatible meshes. Our method exploits properties of the convex hull, e.g., good approximation ability and linear convex representation for interior vertices. After building an initial cross-parameterization via convex-hull domains, we use compatible remeshing algorithms to achieve an accurate approximation of the target geometry and to ensure a complete surface matching. Experimental results show that the compatible meshes constructed are well suited for shape blending and other geometric applications.

Accurately capturing user preferences over time is a great practical challenge in recommender systems. Simple correlation over time is typically not meaningful, since users change their preferences due to different external events. User behavior can often be determined by individual's long-term and short-term preferences. How to represent users' long-term and short-term preferences? How to leverage them for temporal recommendation? To address these challenges, we propose Session-based Temporal Graph (STG) which simultaneously models users' long-term and short-term preferences over time. Based on the STG model framework, we propose a novel recommendation algorithm Injected Preference Fusion (IPF) and extend the personalized Random Walk for temporal recommendation. Finally, we evaluate the effectiveness of our method using two real datasets on citations and social bookmarking, in which our proposed method IPF gives 15%-34% improvement over the previous state-of-the-art.

The peer-to-peer solution provides a service model that is attractive for video-on-demand (VOD). The key idea of using P2P technology is to aggregate storage and bandwidths capacity of peers to alleviate workload on the source server. Compared with P2P live streaming, it is much more challenge for VOD due to peers' asynchronous request, different interested portion and support for VCR-like operation. In this paper, we utilize two-scale list of peers, which is composed of R-peers and S-peers, to locate collaborator and support VCR operation. Demand-driven chunk swarming is proposed to exchange data with ragged collaborators in the VoD system. Simulation results show that the source sever load is significantly reduced, playback continuity is guaranteed and our method supports VCR operation quite well.

In recent years, time information is more and more important in collaborative filtering (CF) based recommender system because many systems have collected rating data for a long time, and time effects in user preference is stronger. In this paper, we focus on modeling time effects in CF and analyze how temporal features influence CF. There are four main types of time effects in CF: (1) time bias, the interest of whole society changes with time; (2) user bias shifting, a user may change his/her rating habit over time; (3) item bias shifting, the popularity of items changes with time; (4) user preference shifting, a user may change his/her attitude to some types of items. In this work, these four time effects are used by factorized model, which is called TimeSVD. Moreover, many other time effects are used by simple methods. Our time-dependent models are tested on Netflix data from Nov. 1999 to Dec. 2005. Experimental results show that prediction accuracy in CF can be improved significantly by using time information.

Color and texture have been widely used in image segmentation; however, their performance is often hindered by scene ambiguities, overlapping objects, or missing parts. In this paper, we propose an interactive image segmentation approach with shape prior models within a Bayesian framework. Interactive features, through mouse strokes, reduce ambiguities, and the incorporation of shape priors enhances quality of the segmentation where color and/or texture are not solely adequate. The novelties of our approach are in (i) formulating the segmentation problem in a well-defined Bayesian framework with multiple shape priors, (ii) efficiently estimating parameters of the Bayesian model, and (iii) multi-object segmentation through user-specified priors. We demonstrate the effectiveness of our method on a set of natural and synthetic images.

We propose a novel framework for consistent correspondence between arbitrary manifold meshes. Different from most existing methods, our approach directly maps the connectivity of the source mesh onto the target mesh without needing to segment input meshes, thus effectively avoids dealing with unstable extreme conditions (e.g. complex boundaries or high genus). In this paper, firstly, a novel mean-value Laplacian fitting scheme is proposed, which aims at computing a shape-preserving (conformal) correspondence directly in 3D-to-3D space, efficiently avoiding local optimum caused by the nearest-point search, and achieving good results even with only a few marker points. Secondly, we introduce a vertex relocation and projection approach, which refines the initial fitting result in the way of local conformity. Each vertex of the initial result is gradually projected onto the target model's surface to ensure a complete surface match. Furthermore, we provide a fast and effective approach to automatically detect critic points in the context of consistent correspondence. By fitting these critic points that capture the important features of the target mesh, the output compatible mesh matches the target mesh's profiles quite well. Compared with previous approaches, our scheme is robust, fast, and convenient, thus suitable for common applications.

In this paper, we present a novel active contour model, in which the traditional gradient descent optimization is replaced by graph cut optimization. The basic idea is to first define an energy function according to curve evolution and then construct a graph with well selected edge weights based on the objective energy function, which is further optimized via graph cut algorithm. In this fashion, our model shares advantages of both level set method and graph cut algorithm, which are "topological" invariance, computational efficiency, and immunity to being stuck in the local minima. The model is validated on synthetic images, applied to two-class segmentation problem, and compared with the traditional active contour to demonstrate effectiveness of the technique. Finally, the method is applied to samples imaged with transmission electron microscopy that demonstrate complex textured patterns corresponding subcellular regions and micro-anatomy.

Saliency is an important perceptual cue that occurs at different levels of resolution. Important attributes of saliency are symmetry, continuity, and closure. Detection of these attributes is often hindered by noise, variation in scale, and incomplete information. This paper introduces the iterative voting method, which uses oriented kernels for inferring saliency as it relates to symmetry. A unique aspect of the technique is the kernel topography, which is refined and reoriented iteratively. The technique can cluster and group nonconvex perceptual circular symmetries along the radial line of an object's shape. It has an excellent noise immunity and is shown to be tolerant to perturbation in scale. The application of this technique to images obtained through various modes of microscopy is demonstrated. Furthermore, as a case example, the method has been applied to quantify kinetics of nuclear foci formation that are formed by phosphorylation of histone gammaH2AX following ionizing radiation. Iterative voting has been implemented in both 2-D and 3-D for multi image analysis.

Most of spectral embedding algorithms such as Isomap, LLE and Laplacian Eigenmap only give map on training samples. One main problem of these methods is to find the embedding of new samples, which is known as the outof-sample problem of spectral embedding. In this paper, we propose a neural network based method to solve this problem. Neural network is used to train and perform both the forward map from high dimensional image space to low dimensional embedding space, and the backward map in the reverse direction. Additionally, combining the forward and backward network, this method is able to build auto-association model to retrieve high dimensional data, and cross association model to learn high dimensional correspondences. Experiments are conducted on real images for forward and backward map, auto-association and cross association.

We propose a novel region-based approach for building detection in high-resolution satellite image with densely build-up buildings. In our method, first the prior building model is constructed with texture and shape features from training building set. After over-segmentation of input image into many small regions, we identify regions which have a similar pattern with prior building model. These regions are called building like regions (BLRs). Then we group BLRs to get candidate building regions (CBRs), which have similar shape with prior building model. Next, lines which have strong relationship with each CBR are extracted. From these lines and CBR boundaries, 2-D rooftop hypotheses are generated. At last, shadows and geometrical rules are used to verify the hypotheses. Experimental results are shown on area with hundreds of buildings.

Landuse classification is an important problem in the remote sensing field. It can be used in a wide range of applications. In this paper, we propose a hybrid method fusing edges and regions information for the landuse classification of multispectral images. It mainly includes the steps of image pre-processing, initial segmentation and region merging. Especially, a novel spatial mean shift procedure is proposed so that some information can be extracted and used in the successive steps. Aiming at the multispectral images processing, we also design a band weighting strategy that give a proper weight to each band adoptively according to the region to be processed. Experimental results on the Landsat TM and ETM+ images validate the performance of the proposed method.

This paper presents a novel scheme for manifold learning. Different from the previous work reducing data to Euclidean space which cannot handle the looped manifold well, we map the scattered data to its intrinsic parameter manifold by semisupervised learning. Given a set of partially labeled points, the map to a specified parameter manifold is computed by an iterative neighborhood average method called anchor points diffusion procedure (APD). We explore this idea on the most frequently used close formed manifolds, Stiefel manifolds whose special cases include hyper sphere and orthogonal group. The experiments show that APD can recover the underlying intrinsic parameters of points on scattered data manifold successfully.

Scale space is a natural way to handle multi-scale problem. Yang and Ma considered the correspondence between scales, and proposed optical flow in the scale space. In this paper, we generalized Yang and Ma's work to generic images. We first generalize the Horn-Schunck algorithm to multidimensional multichannel image sequence. Since the global smoothness constraint for regularization is no longer suitable in general cases, we introduce localized smoothness weight regularization. At last, we apply the proposed method in color image scale-space pullback, together with another localized smoothness trick considering flow density.

A novel approach for reconstruction of sparse high-resolution data from lower-resolution dense spatio-temporal data is introduced. The basic idea is to compute the dense feature velocities from lower-resolution data and project them to the corresponding high-resolution data for computing the missing data. In this context, the basic flow equation is solved for intensity, as opposed to feature velocities at high resolution. Although the proposed technique is generic, we have applied our approach to sea surface temperature (SST) data at 18 km (low-resolution dense data) for computing the feature velocities and at 4 km (high-resolution sparse data) for interpolating the missing data. At low resolution, computation of the flow field is regularized and uses the incompressibility constraints for tracking fluid motion. At high resolution, computation of the intensity is regularized for continuity across multiple frames.

Organisms express their genomes in a cell-specific manner, resulting in a variety of cellular phenotypes or phenomes. Mapping cell phenomes under a variety of experimental conditions is necessary in order to understand the responses of organisms to stimuli. Representing such data requires an integrated view of experimental and informatic protocols. The proposed system, named BioSig, provides the foundation for cataloging cellular responses as a function of specific conditioning, treatment, staining, etc., for either fixed tissue or living cell studies. A data model has been developed to capture experimental variables and map them to image collections and their computed representation. This representation is hierarchical and spans across sample tissues, cells, and organelles, which are imaged with light microscopy. At each layer, content is represented with an attributed graph, which contains information about cellular morphology, protein localization, and cellular organization in tissue or cell culture. The Web-based multilayer informatics architecture uses the data model to provide guided workflow access for content exploration.

Our goal is to develop a feature based framework for data mining and forecasting from geophysical data fields. These data may be generated from either numerical simulation models or space based platforms. This paper focuses on pertinent features from sea surface temperature (SST) fields that are observed with the AVHRR satellite. Our contribution consist of three components: (1) A method for tracking feature velocities from from fluid motion with incompressibility constraint, (2) a method for localizing singular events such as vortices and saddle points from underlying feature velocities, and (3) application of our protocol to 12 years of high resolution real data to reveal novel seasonal and inter-annual trends based on computed events.

We propose to detect singular-features in order to generate an intelligent summary of high resolution spatio-temporal data that are obtained from satellite-based observations of the ocean. Toward this objective, we extend the Horn-Schunck model of flow field computation to incorporate incompressibility for tracking fluid motion. This is expressed as a zero-divergence constraint in the variational problem and an efficient multigrid implementation of it is introduced. Additionally, we show an effective localization of event features, such as vortices and saddle points, in the velocity field that can be used for subsequent abstraction, query and statistical analysis.

There exists an optical flow in the scale-space if the multiscale representation of an image is viewed as an ordinary image sequence in the time domain. This technique can be used to solve the ill-posed tracking problem in the scale-space.

This paper presents a novel multiscale representation of one-dimensional signal based on complex analysis. We show that a signal and its derivative at different scales can be represented by one analytic function defined on the unit disc in the complex plane, which is called the Schwarz representation of a signal. This representation is applied to the matching problem. Using the theory of analytic functions, we are able to define the inverse of a signal. The matching function between two signals can be defined as the composition of one signal's Schwarz representation and another signal's inverse. The matching function determined by this method has a group structure and is close-formed.