Graph-based recommender systems have become prevalent in micro-video recommendation by modeling user-item interactions as a bipartite graph. However, these methods face two inherent limitations: (1) their reliance on a fixed, pre-defined graph structure makes them susceptible to noisy interactions, and (2) the multi-modal representations they learn often contain redundant information that is not discriminative enough for the recommendation task. To overcome these issues, we propose a novel Multi-modal Bipartite Graph Structure Learning network (MBGSL), which leverages the information bottleneck principle for robust micro-video recommendation. Specifically, MBGSL first learns adaptive graph structures from multi-modal content (e.g., visual, acoustic, textual) through dedicated graph learners to mitigate noise. Then, it applies an intra-modality information bottleneck to learn minimal sufficient representations within each modality and an inter-modality information bottleneck to capture distinctive information across modalities, thereby eliminating redundancy. Furthermore, the model incorporates collaborative signals through a contrastive learning objective to guide the graph structure learning process. Extensive experiments on three real-world datasets demonstrate that MBGSL achieves state-of-the-art performance, significantly surpassing existing baselines.

Short-video platforms have gained immense popularity, captivating the interest of millions, if not billions, of users globally. Recently, researchers have highlighted the significance of analyzing the propagation of short-videos, which typically involves discovering commercial values, public opinions, user behaviors, etc. This paper proposes a new Short-video Propagation Influence Rating (SPIR) task and aims to promote SPIR from both the dataset and method perspectives. First, we propose a new Cross-platform Short-Video (XS-Video) dataset, which aims to provide a large-scale and real-world short-video propagation network across various platforms to facilitate research on short-video propagation. Our XS-Video dataset includes 117,720 videos, 381,926 samples, and 535 topics across 5 biggest Chinese platforms, annotated with the propagation influence from level 0 to 9. To the best of our knowledge, this is the first large-scale short-video dataset that contains cross-platform data or provides all of the views, likes, shares, collects, fans, comments, and comment content. Second, we propose a Large Graph Model (LGM) named NetGPT, based on a novel three-stage training mechanism, to bridge heterogeneous graph-structured data with the powerful reasoning ability and knowledge of Large Language Models (LLMs). Our NetGPT can comprehend and analyze the short-video propagation graph, enabling it to predict the long-term propagation influence of short-videos. Comprehensive experimental results evaluated by both classification and regression metrics on our XS-Video dataset indicate the superiority of our method for SPIR. Our dataset and code will be open upon acceptance.

Comprehending long-form videos, with their extensive temporal contexts and rich semantic complexities, remains a frontier challenge in video understanding. Recently, many existing methods offer promise for long video understanding yet often exhibit operational inefficiencies and suboptimal reasoning. These core challenges typically stem from fragmented multi-step reasoning, unreliable iterative control over information gathering, and visual retrieval strategies that inadequately adapt to query-aware granularities. In this paper, we propose a novel framework named VidCog that mimicks human cognitive processes to achieve robust and efficient long video understanding with Large Language Models (LLMs). VidCog features a Unified Reasoning Engine (URE) that transforms the discrete reasoning tasks into a single, cohesive LLM invocation, and a Contrastive Policy-Optimized Reasoning Gate (CPRG) that learns from relative preferences among contrastive query-exemplars to ensure reliable iterative decision-making. Furthermore, we propose Triadic Optimal Transport Visual Evidence Miner (TOT-VEM) to adaptively capture global-local temporal visual evidence by modeling it as a novel triadic optimal transport problem. Experiments on challenging long-video benchmarks demonstrate that VidCog consistently outperforms the strong baseline in both reasoning accuracy and efficiency, validating the superiority of the human-like cognitive loop.

With the rapid development of Internet and the burgeoning scale of social media, Social Event Classification (SEC) has garnered increasing attention. The existing study of SEC focuses on recognizing a fixed set of social events. However, in real-world scenarios, new social events continually emerge on social media, which suggests the necessity for a practical SEC model that can swiftly adapt to the evolving environment with incremental social events. Therefore, in this paper, we study a new yet crucial problem defined as Continual Social Event Classification (C-SEC), where new events continually emerge in the sequentially collected social data. Accordingly, we propose a novel Temporal Event Knowledge Network (TEKNet) to continually learn temporal event knowledge for C-SEC with temporally incremental events. First, we conduct present event knowledge learning to learn the classification of newly emerging events in the presently incoming data. Second, we design past event knowledge replay with self-knowledge distillation to consolidate the learned knowledge of past events and prevent catastrophic forgetting. Finally, we propose future event knowledge pretraining with a modality mixture mechanism to pretrain the classifiers for events that occur in the future. Comprehensive experiments on real-world social event datasets demonstrate the superiority of our proposed TEKNet for C-SEC.

With the increasing number of news uploaded to the internet daily, rumor detection has garnered significant attention in recent years. Existing rumor detection methods excel on familiar topics with sufficient training data (high resource) collected from the same domain. However, when facing emergent events or rumors propagated in different languages, the performance of these models is significantly degraded, due to the lack of training data and prior knowledge (low resource). To tackle this challenge, we introduce the Test-Time Training for Rumor Detection (T^3RD) to enhance the performance of rumor detection models on low-resource datasets. Specifically, we introduce self-supervised learning (SSL) as an auxiliary task in the test-time training. It consists of global and local contrastive learning, in which the global contrastive learning focuses on obtaining invariant graph representations and the local one focuses on acquiring invariant node representations. We employ the auxiliary SSL tasks for both the training and test-time training phases to mine the intrinsic traits of test samples and further calibrate the trained model for these test samples. To mitigate the risk of distribution distortion in test-time training, we introduce feature alignment constraints aimed at achieving a balanced synergy between the knowledge derived from the training set and the test samples. The experiments conducted on the two widely used cross-domain datasets demonstrate that the proposed model achieves a new state-of-the-art in performance. Our code is available at https://github.com/social-rumors/T3RD.

Graph Neural Networks (GNNs) have recently been utilized to build Collaborative Filtering (CF) models to predict user preferences based on historical user-item interactions. However, there is relatively little understanding of how GNN-based CF models relate to some traditional Network Representation Learning (NRL) approaches. In this paper, we show the equivalence between some state-of-the-art GNN-based CF models and a traditional 1-layer NRL model based on context encoding. Based on a Markov process that trades off two types of distances, we present Markov Graph Diffusion Collaborative Filtering (MGDCF) to generalize some state-of-the-art GNN-based CF models. Instead of considering the GNN as a trainable black box that propagates learnable user/item vertex embeddings, we treat GNNs as an untrainable Markov process that can construct constant context features of vertices for a traditional NRL model that encodes context features with a fully-connected layer. Such simplification can help us to better understand how GNNs benefit CF models. Especially, it helps us realize that ranking losses play crucial roles in GNN-based CF tasks. With our proposed simple yet powerful ranking loss InfoBPR, the NRL model can still perform well without the context features constructed by GNNs. We conduct experiments to perform detailed analysis on MGDCF.

With the rapid development of Internet and the expanding scale of social media, social event classification has attracted increasing attention. The key to social event classification is effectively leveraging the visual and textual semantics for classification. However, most of the existing approaches may suffer from the following limitations: (1) Most of them just simply concatenate the image features and text features to get the multimodal features and ignore the fine-grained semantic relationship between modalities. (2) The majority of them hold the closed-world assumption that all classes in test are already seen in training, while this assumption can be easily broken in real-world applications. In practice, new events on Internet may not belong to any existing/seen class, and therefore cannot be correctly identified by closed-world learning algorithms. To tackle these challenges, we propose an Open-World Social Event Classifier (OWSEC) model in this paper. Firstly, we design a multimodal mask transformer network to capture cross-modal semantic relations and fuse fine-grained multimodal features of social events while masking redundant information. Secondly, we design an open-world classifier and propose a cross-modal event mixture mechanism with a novel open-world classification loss to capture the potential distribution space of the unseen class. Extensive experiments on two public datasets demonstrate the superiority of our proposed OWSEC model for open-world social event classification.

Representation learning of knowledge graphs (KGs) aims to embed both entities and relations as vectors in a continuous low-dimensional space, which has facilitated various applications such as link prediction and entity retrieval. Most existing KG embedding methods focus on modeling the structured fact triples independently and ignore the multi-type relations among triples as well as the variety of data types (e.g., texts and images) associated with entities in KGs, and thus fail to capture the complex and multi-modal information that is inherently inside the entity-relation triples. In this paper, we propose a novel approach for knowledge graph embedding named Contrastive Multi-modal Graph Neural Network (CMGNN), which can encapsulate comprehensive features from multi-modal content descriptions of entities and high-order connectivity structures. Specifically, CMGNN first learns entity embeddings from multi-modal content and then contrasts encodings from multi-relational local neighbors and high-order connectivities to obtain latent representations of entities and relations simultaneously. Experimental results demonstrate that CMGNN can effectively model the multi-modalities and multi-type structures in KGs, and significantly outperforms existing state-of-the-art methods on benchmark datasets for the tasks of link prediction and entity classification.

Recently, user cold-start recommendations have attracted a lot of attention from industry and academia. In user cold-start recommendation systems, the user attribute information is often used by existing approaches to learn user preferences due to the unavailability of user action data. However, most existing recommendation methods often ignore the sparsity of user attributes in cold-start recommendation systems. To tackle this limitation, this article proposes a novel Inductive Heterogeneous Graph Neural Network (IHGNN) model, which utilizes the relational information in user cold-start recommendation systems to alleviate the sparsity of user attributes. Our model converts new users, items, and associated multimodal information into a Modality-aware Heterogeneous Graph (M-HG) that preserves the rich and heterogeneous relationship information among them. Specifically, to utilize rich and heterogeneous relational information in an M-HG for enriching the sparse attribute information of new users, we design a strategy based on random walk operations to collect associated neighbors of new users by multiple times sampling operation. Then, a well-designed multiple hierarchical attention aggregation model consisting of the intra- and inter-type attention aggregating module is proposed, focusing on useful connected neighbors and neglecting meaningless and noisy connected neighbors to generate high-quality representations for user cold-start recommendations. Experimental results on three real datasets demonstrate that the IHGNN outperforms the state-of-the-art baselines.

Medicine recommendation systems target to recommend a set of medicines given a set of symptoms which play a crucial role in assisting doctors in their daily clinics. Existing approaches are either rule-based or supervised. However, the former heavily relies on expert labeling, which is time-consuming and costly to collect, and the latter suffers from the data sparse problem. To automate medicine recommendation on sparse data, we propose MedRec, which introduces two graphs in modeling: (1) a knowledge graph connecting diseases, medicines, symptoms, and examinations; (2) an attribute graph connecting medicines via shared attributes and molecular structures. These two graphs enhance the connectivity between symptoms and medicines, which thus alleviate the data sparse problem. By learning the interrelationship between diseases, medicines, symptoms and examinations and the inner relationship within medicine, we can acquire unified embedding representations of symptoms and medicines which can be used in medicine recommendation. The experimental results show that the proposed model outperforms state-of-the-art methods. In addition, we find that these two tasks: learning graph representation and medical recommendation can benefit each other.

The cold-start recommendation is an urgent problem in contemporary online applications. It aims to provide users whose behaviors are literally sparse with as accurate recommendations as possible. Many data-driven algorithms, such as the widely used matrix factorization, underperform because of data sparseness. This work adopts the idea of meta-learning to solve the user’s cold-start recommendation problem. We propose a meta-learning-based cold-start sequential recommendation framework called metaCSR, including three main components: Diffusion Representer for learning better user/item embedding through information diffusion on the interaction graph; Sequential Recommender for capturing temporal dependencies of behavior sequences; and Meta Learner for extracting and propagating transferable knowledge of prior users and learning a good initialization for new users. metaCSR holds the ability to learn the common patterns from regular users’ behaviors and optimize the initialization so that the model can quickly adapt to new users after one or a few gradient updates to achieve optimal performance. The extensive quantitative experiments on three widely used datasets show the remarkable performance of metaCSR in dealing with the user cold-start problem. Meanwhile, a series of qualitative analysis demonstrates that the proposed metaCSR has good generalization.

The cross-modal image-text retrieval has attracted extensive attention in recent years, which contributes to the development of search engine. Fine-grained features and cross-attention have been widely used in past researches to reach the goal of cross-modal image-text matching. Although cross-related methods have achieved remarkable results, the features must be encoded again in evaluation phase due to the interaction of the two modalities, which is unsuitable for actual scenarios of search engine development. In addition, the aggregated feature does not contain sufficient semantics since it is merely obtained by simple mean pooling. Furthermore, connecting weights of self-attention blocks are target position invariant, which lacks the expected adaptability. To tackle these limitations, in this paper, we propose a novel Global Relation-aware Attention Network (GRAN) for image-text retrieval by designing Global Attention Module (GAM) and Relation-aware Attention Module (RAM) which play an important role in modeling the global feature and the relationships of local fragments. Firstly, we propose Global Attention Module (GAM) followed the fine-grained features to obtain meaningful global feature. Secondly, we use several stacked transformer encoders to further encode features separately. Finally, we propose Relation-aware Attention Module (RAM) to generate a vector which represents the relation information to infer the attention intensity of pairwise fragments. The local features, the global feature, and their relations are considered jointly to conduct an efficient image-text retrieval. Extensive experiments are conducted on the benchmark datasets of Flickr30K and MSCOCO, demonstrating the superiority of our method. On the Flickr30K, compared to the state-of-the-art method TERAN, we improve [email protected](K=1) metric by 5.8% and 4.0 on the image and text retrieval tasks, respectively.

Activity recognition based on egocentric multimodal data collected by wearable devices has become increasingly popular recently. However, conventional activity recognition methods face the dilemma of the lack of large-scale labeled egocentric multimodal datasets due to the high cost of data collection. In this paper, we propose a new task of few-shot egocentric multimodal activity recognition, which has at least two significant challenges. On the one hand, it is difficult to extract effective features from the multimodal data sequences of video and sensor signals due to the scarcity of the samples. On the other hand, how to robustly recognize novel activity classes with very few labeled samples becomes another more critical challenge due to the complexity of the multimodal data. To resolve the challenges, we propose a two-stream graph network, which consists of a heterogeneous graph-based multimodal association module and a knowledge-aware activity classifier module. The former uses a heterogeneous graph network to comprehensively capture the dynamic and complementary information contained in the multimodal data stream. The latter learns robust activity classifiers through knowledge propagation among the classifier parameters of different classes. In addition, we adopt episodic training strategy to improve the generalization ability of the proposed few-shot activity recognition model. Experiments on two public datasets show that the proposed model achieves better performances than other baseline models.

Nowadays, detecting fake news on social media platforms has become a top priority since the widespread dissemination of fake news may mislead readers and have negative effects. To date, many algorithms have been proposed to facilitate the detection of fake news from the hand-crafted feature extraction methods to deep learning approaches. However, these methods may suffer from the following limitations: (1) fail to utilize the multi-modal context information and extract high-order complementary information for each news to enhance the detection of fake news; (2) largely ignore the full hierarchical semantics of textual content to assist in learning a better news representation. To overcome these limitations, this paper proposes a novel hierarchical multi-modal contextual attention network (HMCAN) for fake news detection by jointly modeling the multi-modal context information and the hierarchical semantics of text in a unified deep model. Specifically, we employ BERT and ResNet to learn better representations for text and images, respectively. Then, we feed the obtained representations of images and text into a multi-modal contextual attention network to fuse both inter-modality and intra-modality relationships. Finally, we design a hierarchical encoding network to capture the rich hierarchical semantics for fake news detection. Extensive experiments on three public real datasets demonstrate that our proposed HMCAN achieves state-of-the-art performance.

Knowledge graph embedding (KGE) encodes components of a KG including entities and relations into continuous low vector space. Most existing methods focus on treating entities and relations in triples independently and thus failing to capture the complex and hidden information that is inherently implicit inside the local neighborhood surrounding a triple. In this paper, we present a new approach for knowledge graph completion called GAEAT (Graph Auto-encoder Attention Network Embedding), which can encapsulate both entity and relation features. Specifically, we construct a triple-level auto-encoder by extending graph attention mechanisms to obtain latent representations of entities and relations simultaneously. To justify our proposed model, we evaluate GAEAT on two real-world datasets. The experimental results demonstrate that GAEAT can outperform state-of-the-art KGE models in knowledge graph completion task, which validates the effectiveness of GAEAT. The source code of this paper can be obtained from https://github.com/TomersHan/GAEAT.

Nowadays, with the rapid development of social media, there is a great deal of news produced every day. How to detect fake news automatically from a large of multimedia posts has become very important for people, the government and news recommendation sites. However, most of the existing approaches either extract features from the text of the post which is a single modality or simply concatenate the visual features and textual features of a post to get a multimodal feature and detect fake news. Most of them ignore the background knowledge hidden in the text content of the post which facilitates fake news detection. To address these issues, we propose a novel Knowledge-driven Multimodal Graph Convolutional Network (KMGCN) to model the semantic representations by jointly modeling the textual information, knowledge concepts and visual information into a unified framework for fake news detection. Instead of viewing text content as word sequences normally, we convert them into a graph, which can model non-consecutive phrases for better obtaining the composition of semantics. Besides, we not only convert visual information as nodes of graphs but also retrieve external knowledge from real-world knowledge graph as nodes of graphs to provide complementary semantics information to improve fake news detection. We utilize a well-designed graph convolutional network to extract the semantic representation of these graphs. Extensive experiments on two public real-world datasets illustrate the validation of our approach.

Domain adaptation has received lots of attention for its high efficiency in dealing with cross-domain learning tasks. Most existing domain adaptation methods adopt the strategies relying on large amounts of source label information, which limits their applications in the real world where only a few label samples are available. We exploit the local geometric connections to tackle this problem and propose a Local Structure Alignment (LSA) guided domain adaptation method in this paper. LSA leverages the Nyström method to describe the distribution difference from the geometric perspective and then perform the distribution alignment between domains. Specifically, LSA constructs a domain-invariant Hessian matrix to locally connect the data of the two domains through minimizing the Nyström approximation error. And then it integrates the domain-invariant Hessian matrix with the semi-supervised learning and finally builds an adaptive semi-supervised model. Extensive experimental results validate that the proposed LSA outperforms the traditional domain adaptation methods especially when only sparse source label information is available.

Style text with decorative elements has a strong visual sense, and enriches our daily work, study and life. However, it introduces new challenges to text detection and recognition. In this study, we propose a text destylized framework, that can transform the stylized texts with decorative elements into a type that is easily distinguishable by a detection or recognition model. We arranged and integrate an existing stylistic text data set to train the destylized network. The new destylized data set contains English letters and Chinese characters. The proposed approach enables a framework to handle both Chinese characters and English letters without the need for additional networks. Experiments show that the method is superior to the state-of-the-art style-related models.

Natural language generation has become a fundamental task in dialogue systems. RNN-based natural response generation methods encode the dialogue context and decode it into a response. However, they tend to generate dull and simple responses. In this article, we propose a novel framework, called KAWA-DRG (Knowledge-aware Attentive Wasserstein Adversarial Dialogue Response Generation) to model conversation-specific external knowledge and the importance variances of dialogue context in a unified adversarial encoder-decoder learning framework. In KAWA-DRG, a co-attention mechanism attends to important parts within and among context utterances with word-utterance-level attention. Prior knowledge is integrated into the conditional Wasserstein auto-encoder for learning the latent variable space. The posterior and prior distribution of latent variables are generated and trained through adversarial learning. We evaluate our model on Switchboard, DailyDialog, In-Car Assistant, and Ubuntu Dialogue Corpus. Experimental results show that KAWA-DRG outperforms the existing methods.

Social event is something that occurs at specific place and time associated with some specific actions, and it consists of many stories over time. With the explosion of Web 2.0 platforms, a popular social event that is happening around us and around the world can spread very fast. As a result, social event analysis becomes more and more important for users to understand the whole evolutionary trend of social event over time. However, it is very challenging to do social event analysis because social event data from different social media sites have multi-modal, multi-domain, and large-scale properties. The goal of our research is to design advanced multimedia techniques to deal with the above issues and establish an effective and robust social event analysis framework for social event representation, detection, tracking and evolution analysis. (1) For social event representation, we propose a novel cross-domain collaborative learning algorithm based on non-parametric Bayesian dictionary learning model. It can make use of the shared domain priors and modality priors to collaboratively learn the data's representations by considering the domain discrepancy and the multi-modal property.(2) For social event detection, we propose a boosted multi-modal supervised Latent Dirichlet Allocation model. It can effectively exploit multi-modality information and utilize boosting weighted sampling strategy for large-scale data processing. (3) For social event tracking, we propose a novel multi-modal event topic model, which can effectively model the correlations between textual and visual modalities, and obtain their topics over time. (4) For social event evolution analysis, we propose a novel multi-modal multi-view topic-opinion mining model to conduct fined-grained topic and opinion analysis for social events from multiple social media sites collaboratively. It can discover multi-modal topics and the corresponding opinions over time to understand the evolutionary processes of social event. Extensive experimental results show that the proposed algorithms perform favorably against state-of-the-art methods for social event analysis.

With a smart phone at hand, it becomes easy now to snap pictures and publish them online with few lines of texts. The GPS coordinates and User-Generated Content (UGC) data embedded in the shared photos provide opportunities to exploit important knowledge to tackle interesting tasks like geographically organizing photos and location visualization. In this work, we propose to organize photos both geographically and semantically, and investigate the problem of location visualization from multiple semantic themes. The novel visualization scheme provides a rich display landscape for geographical exploration from versatile views. A two-level solution is presented, where we first identify the highly photographed places of interest (POI) and discover their focused themes, and then aggregate the lower-level POI themes to generate the higher-level city themes for location visualization. We have conducted experiments on crawled Flickr and Instagram data and exhibited the visualization for the cities of Singapore and Sydney. The experimental results have validated the proposed method and demonstrated the potentials of location visualization from multiple themes.

Newly emerging location-based social media network services (LBSMNS) provide valuable resources to understand users’ behaviors based on their location histories. The location-based behaviors of a user are generally influenced by both user intrinsic interest and the location preference, and moreover are spatial-temporal context dependent. In this article, we propose a spatial-temporal context-aware personalized location recommendation system (STCAPLRS), which offers a particular user a set of location items such as points of interest or venues (e.g., restaurants and shopping malls) within a geospatial range by considering personal interest, local preference, and spatial-temporal context influence. STCAPLRS can make accurate recommendation and facilitate people’s local visiting and new location exploration by exploiting the context information of user behavior, associations between users and location items, and the location and content information of location items. Specifically, STCAPLRS consists of two components: offline modeling and online recommendation. The core module of the offline modeling part is a context-aware regression mixture model that is designed to model the location-based user behaviors in LBSMNS to learn the interest of each individual user, the local preference of each individual location, and the context-aware influence factors. The online recommendation part takes a querying user along with the corresponding querying spatial-temporal context as input and automatically combines the learned interest of the querying user, the local preference of the querying location, and the context-aware influence factor to produce the top- k recommendations. We evaluate the performance of STCAPLRS on two real-world datasets: Dianping and Foursquare. The results demonstrate the superiority of STCAPLRS in recommending location items for users in terms of both effectiveness and efficiency. Moreover, the experimental analysis results also illustrate the excellent interpretability of STCAPLRS.

In social media, users have contributed enormous behavior data online which can be leveraged for user modeling and conduct personalized services. Temporal user modeling, which incorporates the timestamp of these behavior data and understands users' interest evolution, have attracted attention recently. With the recognition that user interests are vulnerable to transient events, many current temporal user modeling solutions propose to first identify the transient events and then consider the identified events into user behavior modeling. In this work, in the context of microblogs, we propose a unified probabilistic framework to simultaneously model the process of transient event detection and temporal user tweeting. The outputs of the framework include: (1) one long-term topic space spanning over general categories, (2) one short-term topic space for each time interval corresponding to the transient events, and (3) users' interest distributions over the long- and short-term topic spaces. Qualitative and quantitative experimental evaluation are conducted on a large-scale Twitter dataset, with more than 2 million users and 0.3 billion tweets. The promising results demonstrate the advantage of the proposed topic models.

The ever growing number of videos on YouTube makes recommendation an important way to help users explore interesting videos. Similar to general recommender systems, YouTube video recommendation suffers from typical problems like new user, cold-start, data sparsity, etc. In this paper, we propose a unified YouTube video recommendation solution via cross-network collaboration: users' auxiliary information on Twitter are exploited to address the typical problems in single network-based recommendation solutions. The proposed two-stage solution first transfers user preferences from auxiliary network by learning cross-network behavior correlations, and then integrates the transferred preferences with the observed behaviors on target network in an adaptive fashion. Experimental results show that the proposed cross-network collaborative solution achieves superior performance not only in term of accuracy, but also in improving the diversity and novelty of the recommended videos.

While on the go, people are using their phones as a personal concierge discovering what is around and deciding what to do. Mobile phone has become a recommendation terminal customized for individuals—capable of recommending activities and simplifying the accomplishment of related tasks. In this article, we conduct usage mining on the check-in data, with summarized statistics identifying the local recommendation challenges of huge solution space, sparse available data, and complicated user intent, and discovered observations to motivate the hierarchical, contextual, and sequential solution. We present a point-of-interest (POI) category-transition--based approach, with a goal of estimating the visiting probability of a series of successive POIs conditioned on current user context and sensor context. A mobile local recommendation demo application is deployed. The objective and subjective evaluations validate the effectiveness in providing mobile users both accurate recommendation and favorable user experience.

Vision-based sign language recognition has attracted more and more interest from researchers in the computer vision field. In this article, we propose a novel algorithm to model and recognize sign language performed in front of a Microsoft Kinect sensor. Under the assumption that some frames are expected to be both discriminative and representative in a sign language video, we first assign a binary latent variable to each frame in training videos for indicating its discriminative capability, then develop a latent support vector machine model to classify the signs, as well as localize the discriminative and representative frames in each video. In addition, we utilize the depth map together with the color image captured by the Kinect sensor to obtain a more effective and accurate feature to enhance the recognition accuracy. To evaluate our approach, we conducted experiments on both word-level sign language and sentence-level sign language. An American Sign Language dataset including approximately 2,000 word-level sign language phrases and 2,000 sentence-level sign language phrases was collected using the Kinect sensor, and each phrase contains color, depth, and skeleton information. Experiments on our dataset demonstrate the effectiveness of the proposed method for sign language recognition.

In this article, by taking a popular game, the Find-the-Difference (FiDi) game, as a concrete example, we explore how state-of-the-art image processing techniques can assist in developing a personalized, automatic, and dynamic game. Unlike the traditional FiDi game, where image pairs (source image and target image) with five different patches are manually produced by professional game developers, the proposed Personalized FiDi (P-FiDi) electronic game can be played in a fully automatic Snap & Play mode.Snapmeans that players first take photos with their digital cameras. The newly captured photos are used as source images and fed into the P-FiDi system to autogenerate the counterpart target images for users toplay. Four steps are adopted to autogenerate target images: enhancing the visual quality of source images, extracting some changeable patches from the source image, selecting the most suitable combination of changeable patches and difference styles for the image, and generating the differences on the target image with state-of-the-art image processing techniques. In addition, the P-FiDi game can be easily redesigned for the im-game advertising. Extensive experiments show that the P-FiDi electronic game is satisfying in terms of player experience, seamless advertisement, and technical feasibility.

This paper is devoted to detecting social, real-world events from the sharing images/videos on social media sites like Flickr and YouTube. The fast growing contents make the social media sites become gold mines for social event detection, but we still need to overcome the challenge of processing the associated heterogeneous metadata, such as time-stamp, location, visual content and textual content. Different from the traditional early or late fusion with different types of metadata, we represent them into a star-structured $K$-partite graph, that is, social media itself is regarded as the central vertices set and different types of metadata are treated as the auxiliary vertices sets which are pairwise independent with each other but correlated with the central one. Based on this graph, Social Event Detection with Robust High-Order Co-Clustering (SED-RHOCC) algorithm is proposed and it includes two steps: 1) coarse event detection, 2) clusters and samples refinement. In the first step, by revealing the inter-relationship on the constructed star-structured $K$-partite graph and the intra-relationship within some metadata sets such as time-stamp, we co-cluster social media and the associated metadata separately and iteratively to avoid information loss in early/late fusion. After that, a post process is utilized to refine the clusters and social media samples in the second step. MediaEval Social Event Detection Dataset [1] and its subset are selected to demonstrate the effectiveness of our proposed approach in handling the datasets with and without non-event samples.

In this paper, we propose a novel cross-space affinity learning algorithm over different spaces with heterogeneous structures. Unlike most of affinity learning algorithms on the homogeneous space, we construct a cross-space tensor model to learn the affinity measures on heterogeneous spaces subject to a set of order constraints from the training pool. We further enhance the model with a factorization form which greatly reduces the number of parameters of the model with a controlled complexity. Moreover, from the practical perspective, we show the proposed factorized cross-space tensor model can be efficiently optimized by a series of simple quadratic optimization problems in an iterative manner. The proposed cross-space affinity learning algorithm can be applied to many real-world problems, which involve multiple heterogeneous data objects defined over different spaces. In this paper, we apply it into the recommendation system to measure the affinity between users and the product items, where a higher affinity means a higher rating of the user on the product. For an empirical evaluation, a widely used benchmark movie recommendation data set—MovieLens—is used to compare the proposed algorithm with other state-of-the-art recommendation algorithms and we show that very competitive results can be obtained.

While on the go, people are using their phones as a personal concierge discovering what is around and deciding what to do. Mobile phone has become a recommendation terminal customized for individuals. While existing research predominantly focuses on one-step recommendation---recommending the next single activity according to current context, this work moves one step beyond by recommending a series of activities, which is a package of sequential Points of Interest (POIs). The recommended POIs are not only relevant to user context (i.e., current location, time, and check-in), but also personalized to his/her check-in history. We presents a probabilistic approach, which is highly motivated from a large-scale commercial mobile check-in data analysis, to ranking a list of sequential POI categories (e.g., "Japanese food" and "bar") and POIs (e.g., "I love sushi"). The approach enables users to plan consecutive activities on the move. Specifically, the probabilistic recommendation approach estimates the transition probability from one POI to another, conditioned on current context and check-in history in a Markov chain. To alleviate the discritization error and sparsity problem, we further introduce context collaboration and integrate prior information. Experiments on over 100k real-world check-in records and 20k POIs validate the effectiveness of the proposed approach.

Learning to rank has been demonstrated as a powerful tool for image ranking, but the issue of the "curse of dimensionality" is a key challenge of learning a ranking model from a large image database. This paper proposes a novel dimensionality reduction algorithm named ordinal preserving projection (OPP) for learning to rank. We first define two matrices, which work in the row direction and column direction respectively. The two matrices aim at leveraging the global structure of the data set and ordinal information of the observations. By maximizing the corresponding objective functions, we can obtain two optimal projection matrices mapping original data points into low-dimensional subspace, in which both global structure and ordinal information can be preserved. The experiments are conducted on the public available MSRA-MM image data set and "Web Queries" image data set, and the experimental results demonstrate the effectiveness of the proposed method.

In this paper, we propose a novel approach for automatic music video summarization based on audio-visual-text analysis and alignment. The music video is separated into the music and video tracks. For the music track, the chorus is detected based on music structure analysis. For the video track, we first segment the shots and classify the shots into close-up face shots and non-face shots, then we extract the lyrics and detect the most repeated lyrics from the shots. The music video summary is generated based on the alignment of boundaries of the detected chorus, shot class and the most repeated lyrics from the music video. The experiments on chorus detection, shot classification, and lyrics detection using 20 English music videos are described. Subjective user studies have been conducted to evaluate the quality and effectiveness of summary. The comparisons with the summaries based on our previous method and the manual method indicate that the results of summarization using the proposed method are better at meeting users' expectations.