Federated learning (FL) has received much attention in privacy-preserving and responsible recommender systems. Recent studies have shown promising results while federating widely used recommendation methods such as collaborative filtering. A major barrier when bringing FL into production is that the model complexity or the volume of gradients to be transmitted over the communication channel grows linearly as the number of items in a particular system increases. To address this challenge, we propose a communication-efficient neural collaborative filtering method for federated recommender systems. First, to align our solution with other deep neural architectures, we construct standard neural collaborative filtering in federated settings. Second, to solve the underlying model complexity challenge, a multi-armed bandit framework is used that intelligently selects a smaller set of payloads for each iteration of federated model training. The item selection is based on a carefully designed reward function that determines which portion of the overall payloads would be optimal for a particular user. The FL model only comprising of the selected items is transmitted over the network. The FL users train their local models in the regular federated learning way utilizing the payload-efficient global model, requiring no additional optimizations. The results show that using only 10% of the model’s payload, our method can achieve recommendation performance comparable with the standard federated neural collaborative filtering.

With the advent of large language models (LLMs), managing scientific literature via LLMs has become a promising direction of research. However, existing approaches often overlook the rich structural and semantic relevance among scientific literature, limiting their ability to discern the relationships between pieces of scientific knowledge, and suffer from various types of hallucinations. These methods also focus narrowly on individual downstream tasks, limiting their applicability across use cases. We propose LitFM, the first literature foundation model designed for a wide variety of practical downstream tasks on domain-specific literature, with a focus on citation information. At its core, LitFM contains a novel graph retriever that can provide accurate and diverse recommendations for LLM to integrate graph structure information and relevant literature. LitFM also leverages a knowledge-infused LLM, fine-tuned through a well-developed instruction paradigm. It enables LitFM to extract domain-specific knowledge from literature and reason relationships among them. By integrating citation graphs during both training and inference, LitFM can generalize to unseen papers and accurately assess their relevance within existing literature. Additionally, we introduce new large-scale literature citation benchmark datasets on three academic fields, featuring sentence-level citation information and local context. Extensive experiments validate the superiority of LitFM, achieving 28.1% improvement on retrieval task in precision, and an average improvement of 7.52% over state-of-the-art across six downstream literature-related tasks.

With the explosive growth of online service platforms, an increasing number of people and enterprises are undertaking personal and professional tasks online. In real applications such as trip planning and online marketing, planning sessions for a sequence of activities or services will enable social users to receive the optimal services, improving their experience and reducing the cost of their activities. These online platforms are heterogeneous, including different types of services with different attributes. However, the problem of session planning over heterogeneous platforms has not been studied so far. In this paper, we propose a Motivation-Aware Session Planning (MASP) framework for session planning over heterogeneous social platforms. Specifically, we first propose a novel HeterBERT model to handle the heterogeneity of items at both type and attribute levels. Then, we propose to predict user preference using the motivations behind user activities. Finally, we propose an algorithm together with its optimisations for efficient session generation. The extensive tests prove the high effectiveness and efficiency of MASP.

Abstract Session-based recommendation (SBR) focuses on predicting the next potential item for anonymous users based on short-click sessions. However, these interaction sessions often contain noise items, which arise from misclicks or shifts in user interests. Existing denoising methods typically presume a strong exclusionary relationship between noise items and the recommendation target, assuming that reducing noise can enhance recommendation accuracy. In contrast, our observations reveal a nuanced phenomenon: as the length of the interaction session shortens, the effect of noise removal on recommendation performance gradually transitions from positive to negative. This finding suggests that in short sessions with insufficient contextual information, relying solely on the exclusion of noise items within the session may fail to improve and could even hinder-the recommendation performance. Such complexities have been largely overlooked in prior research. To bridge this gap, we propose two solutions: (i) expanding the view of denoising from a single session to multiple sessions (i.e., from local to global), and (ii) introducing relevant contextual information into each session by employing enhancement strategies. Therefore, we design the Hybrid Prototype-based In-and-Out Flow Network (HyPro), which employs both denoising and enhancing processes for each session based on our proposed hybrid prototypes. Specifically, for each item, HyPro first learns the hybrid prototype by aggregating information from the item’s semantic and topological neighbors across all sessions. Then, based on the hybrid prototypes, HyPro employs an in-and-out flow network comprising two components: (i) the out-flow channel, which targets the removal of irrelevant information at both the data and feature levels, and (ii) the in-flow channel, which integrates global information for each session at the item and session levels. Extensive experiments conducted on three real-world datasets demonstrate that HyPro outperforms the state-of-the-art baselines. The implementation code is available at: https://github.com/jarviswww/Code4HyPro .

Predicting missing facts for temporal knowledge graphs (TKGs) is a fundamental task, called temporal knowledge graph completion (TKGC). One key challenge in this task is the imbalance in data distribution, where facts are unevenly spread across entities and timestamps. This imbalance can lead to poor completion performance for long-tail entities and timestamps, and unstable training due to the introduction of false negative samples. Unfortunately, few previous studies have investigated how to mitigate these effects. Moreover, for the first time, we found that existing methods suffer from model preferences, revealing that entities with specific properties (e.g., recently active) are favored by different models. Such preferences will lead to error accumulation and further exacerbate the effects of imbalanced data distribution. To alleviate the impacts of imbalanced data and model preferences, we introduce Booster , the first data augmentation strategy for TKGs. The unique requirements here lie in generating new samples that fit the complex semantic and temporal patterns within TKGs, and identifying hard-learning samples specific to models. Therefore, we propose a hierarchical scoring algorithm based on triadic closures within TKGs. By incorporating both global semantic patterns and local time-aware structures, the algorithm enables pattern-aware validation for new samples. Meanwhile, we propose a two-stage training approach to identify samples that deviate from the model's preferred patterns. With a frequency-based filtering strategy, this approach also helps to avoid the misleading of false negatives. Experiments justify that Booster can seamlessly adapt to existing TKGC models and achieve on average 4.5% performance improvement.

Privacy concerns in recommender systems are potentially addressed due to constitutional and commercial requirements. Centralized recommendation models are susceptible to poisoning attacks, which threaten their integrity. In this context, federated learning has emerged as an optimal solution to privacy concerns. However, recent investigations proved that Federated Recommender Systems (FedRS) are also vulnerable to model poisoning attacks. Existing attack possibilities highlighted in academic literature require a large fraction of Byzantine clients to effectively influence the training process, which is unrealistic for practical systems with millions of users. Additionally, most attack models neglected the role of the defense mechanism running at the aggregation server. To this end, we propose a novel undetectable hidden attack strategy (HidAttack) for FedRS, aiming to raise the exposure ratio of targeted items with minimum Byzantine clients. To achieve this goal, we construct a cluster of baseline attacks, on top of which a bandit model is designed that intelligently infers effective poisoned gradients. It ensures a diverse pattern of poisoned gradients and therefore, Byzantine clients cannot be distinguished from benign clients by the defense mechanism. Extensive experiments demonstrate that: 1) our attack model significantly increases the target item's exposure rate covertly without compromising the recommendation accuracy and 2) the current defenses are insufficient, emphasizing the need for better security improvements against our model poisoning attack to FedRS.

Social video anomaly is an observation in video streams that does not conform to a common pattern of dataset's behaviour. Social video anomaly detection plays a critical role in applications from e-commerce to e-Iearning. Traditionally, anomaly detection techniques are applied to find anomalies in video broadcasting. However, they neglect the live social video streams which contain interactive talk, speech, or lecture with audience. In this paper, we propose a generic framework for effectively online detecting Anomalies Over social Video LI ve Streaming (AOVLIS). Specifically, we propose a novel deep neural network model called Coupling Long Short-Term Memory (CLSTM) that adaptively captures the history behaviours of the presenters and audience, and their mutual interactions to predict their behaviour at next time point over streams. Then we well integrate the CLSTM with a decoder layer, and propose a new reconstruction error-based scoring function REI A to calculate the anomaly score of each video segment for anomaly detection. After that, we propose a novel model update scheme that incrementally maintains CLSTM and decoder. Moreover, we design a novel upper bound and ADaptive Optimisation Strategy (ADOS) for improving the efficiency of our solution. Extensive experiments are conducted to prove the superiority of AOVLIS.

Partial index tracking (PIT) is a popular passive investment strategy aiming at replicating the performance of a market index (e.g., S&P 500). Existing PIT methods typically treat it as a regression problem and divide it into two tasks: (i) asset selection (determining which assets to choose from the index constituents) and (ii) asset allocation (deciding how to allocate capital among the selected assets). However, these methods either optimize these two tasks jointly, which has been proven to be NP-hard and inefficient when tracking large-scale constituent indices (e.g., Russell 2000), or attempt an independent optimization, lacking a connection to ensure collaborative optimization. In this paper, we present a hierarchical model for partial index tracking (HIT), which formulates PIT as a hierarchical Markov decision process (MDP) and is optimized via hierarchical reinforcement learning (HRL). HIT consists of (1) a high-level policy learns to select assets from constituents to handle task (i) and (2) a low-level policy learns to allocate capital weights among the selected assets to handle task (ii). We further propose a novel cost-sensitive reward function that serves as a connection to collaboratively optimize the two policies, aiming to replicate the index closely while considering transaction cost. Compared with existing jointly optimized approaches, our model simplifies the problem by learning separate policies for the two tasks, and the reward function serves as a connection to ensure collaborative optimization between them, avoiding challenges faced by joint optimization methods in existing literature. Remarkable performance across 6 benchmarks, ranging from small to large-scale constituents demonstrate the superiority of HIT. Moreover, the experiments conducted on a real-world market dataset spanning over 10 years show its effectiveness and practicality.

Deep reinforcement learning (RL) has emerged as a promising approach for portfolio management due to its ability to make sequential decisions. However, applying RL techniques to this domain is still challenging due to the non-stationary nature of financial markets. Existing RL-based solutions fail to consider the intrinsic causes behind this non-stationary, which primarily stem from the involvement of diverse traders with distinct investment horizons and their varied investment strategies. In this paper, we tackle the non-stationary problem by examining its intrinsic causes and propose cross-insight trader, a novel two-step RL-based approach that integrates multiple trading policies with different investment horizons to adapt to the changing market conditions. In the first step, we learn multiple horizon-specific policies by providing each policy with tailored information specific to its investment horizon. This allows each policy to recognize dynamic patterns within its respective horizon and make insightful pre-decisions. In the second step, we learn a cross-insight policy to make the final trade decision by considering the investment pre-decisions made by multiple horizon-specific policies in the first step. To enable effective learning of two types of policies, our approach employs a centralized critic to evaluate the actions performed by both horizon-specific and cross-insight policies. By incorporating multiple insights from different investment horizons into the decision-making process, our approach enhances its adaptability to changing market conditions. Experimental results conducted on three stock markets demonstrate the superiority of our framework.

Group recommendation over social media streams has attracted attention due to its wide applications such as e-commerce, entertainment and online news broadcasting. However, existing stream group recommendation techniques ignore the influence of user groups, which are not effective for item propagation over social networks. To address this problem, we propose a framework for Influence-aware Group Recommendation (IGR) over high-speed social streams. Specifically, we first propose a novel GroupGCN model to capture the dynamics of user attributes and interactions which maps groups and items to their embeddings. A Temporal GroupGCN-RNN-Autoencoder (TGGCN-RA) model is designed to extend GroupGCN for sequence-based tasks, enabling the prediction of group interests over time. Then, we adopt an Independent Cascade (IC) model to predict the influence propagation of social items over user groups. Extensive experiments prove the high effectiveness and efficiency of IGR.

Temporal knowledge graphs (TKGs) are valuable resources for capturing evolving relationships among entities, yet they are often plagued by noise, necessitating robust anomaly detection mechanisms. Existing dynamic graph anomaly detection approaches struggle to capture the rich semantics introduced by node and edge categories within TKGs, while TKG embedding methods lack interpretability, undermining the credibility of anomaly detection. Moreover, these methods falter in adapting to pattern changes and semantic drifts resulting from knowledge updates. To tackle these challenges, we introduce AnoT, an efficient TKG summarization method tailored for interpretable online anomaly detection in TKGs. AnoT begins by summarizing a TKG into a novel rule graph, enabling flexible inference of complex patterns in TKGs. When new knowledge emerges, AnoT maps it onto a node in the rule graph and traverses the rule graph recursively to derive the anomaly score of the knowledge. The traversal yields reachable nodes that furnish interpretable evidence for the validity or the anomalous of the new knowledge. Overall, AnoT embodies a detector-updater-monitor architecture, encompassing a detector for offline TKG summarization and online scoring, an updater for real-time rule graph updates based on emerging knowledge, and a monitor for estimating the approximation error of the rule graph. Experimental results on four real-world datasets demonstrate that AnoT surpasses existing methods significantly in terms of accuracy and interoperability. All of the raw datasets and the implementation of AnoT are provided in https://github.com/zjs123/ANoT.

Privacy and trust are highly demanding in practical recommendation engines. Although Federated Learning (FL) has significantly addressed privacy concerns, commercial operators are still worried about several technical challenges while bringing FL into production. In addition, classical FL has several intrinsic operational limitations such as single-point failure, data and model tampering, and heterogenic clients participating in the FL process. To address these challenges in practical recommenders, we propose a responsible recommendation generation framework based on blockchain-empowered asynchronous FL that can be adopted for any model-based recommender system. In standard FL settings, we build an additional aggregation layer in which multiple trusted nodes guided by a mediator component perform gradient aggregation to achieve an optimal model locally in a parallel fashion. The mediator partitions users into K clusters, and each cluster is represented by a cluster head. Once a cluster gets semi-global convergence, the cluster head transmits model gradients to the FL server for global aggregation. In addition the trusted cluster heads are responsible to submit the converged semi-global model to a blockchain to ensure tamper resilience. In our settings, an additional mediator component works like an independent observer that monitors the performance of each cluster head, updates a reward score, and records it into a digital ledger. Finally, evaluation results on three diversified benchmarks illustrate that the recommendation performance on selected measures is considerably comparable with the standard and federated version of a well-known neural collaborative filtering recommender.

Sequential recommendations have received great attention in recent years due to their wide application in e-commerce, trip planning, and online education. Contexts reveal the intention of a user in a transaction such as consuming or purchasing an item, and highly affect her behaviour. However, existing techniques cannot fully capture the context-sensitive and dynamic sequential user behaviours for effective and efficient sequential recommendation. To address this limitation, we propose a novel Dynamic Bi-layer Graph learning (DBIG)-based framework for sequential recommendation, which learns the sequence dynamics of user behaviours and contexts. Specifically, we first propose a context graph to identify the relevant contexts to a target user. Then, we propose a transaction context graph that predicts the probability of a transaction happening under certain contexts. In addition, we apply BERT (Bidirectional Encoder Representations from Transformers) to our graph embedding for understanding the user sequential behaviours in bidirectional ways and generating recommendations. Finally, we propose a cluster-structure-based algorithm, which incrementally maintains the DBIG model and BERT over dynamic transaction updates. We evaluate our DBIG-based framework by conducting extensive experiments over five public datasets. The quantitative results demonstrate the superiority of our approach in terms of effectiveness and efficiency.

Stock prediction is an attractive topic in fintech. However, traditional solutions for stock prediction have two drawbacks: (1) Some focus on the temporal patterns of stocks and model each stock as an independent individual but neglect their relations. Some models consider the relations among stocks, but work in a two-step format (i.e., capturing the temporal patterns first and then considering the relation dependency), which makes them complex and inefficient; (2) They model the stock prediction as a regression (predicting stock price) or classification task (predicting stock trend), which cannot optimize the target of investment, i.e., selecting the best stocks from the exchange market with the highest expected revenue in the future. To fully utilize the relations among stocks and achieve the highest revenue, a relation-temporal graph convolutional network (RT-GCN) is proposed. We first model the relations among stocks and their daily features into a relation-temporal graph. Then, we apply RT-GCN and three relation-aware strategies to realize the relation-temporal feature extraction for each stock. Finally, the features are fed for score calculation in a learning-to-rank way, and the stock with the highest score represents the highest investment revenue in the future. Extensive experiments demonstrate the effectiveness and efficiency of our method.

The accurate detection of road closures is of great value for real-time updating of digital maps. The existing methods mainly follow the paradigm of detecting the drastic changes in traffic statistical values (e.g., traffic flow), but they may lead to misidentifying since 1) drastic changes of traffic statistical values are hard to be observed in low-heat roads where the passing vehicles are sparse; 2) statistical values are sensitive to noise (e.g., traffic flow for tiny roads and tunnels is prone to miscounting); and 3) statistical values are naturally delayed, and misidentifying may occur when they have not yet shown significant changes. Surprisingly, since GPS-based trajectories can also exhibit significant abnormal patterns for road closures and have the superiority in fine granularity and timeliness, they can naturally tackle the above challenges. In this paper, we present a novel road closure detection framework based on mining the semantics of trajectories, called T-Closure. We first construct a heterogeneous graph based on the trajectory and the planned route to extract the spatial-topological property of each trajectory, where a node-level auxiliary task is proposed to guide the learning of feature encoders. A multi-view heterogeneous graph neural network (MVH-GNN) with a graph-level auxiliary task is then introduced to capture the semantics of trajectories, where intra-category relevance and inter-category interaction are both considered. Finally, a sequence-level auxiliary task refines the ability of LSTM in modeling the semantic relevance among trajectories while enhancing the robustness of our framework. Experiments on four real-world road closure datasets demonstrate the superiority of T-Closure. Online performance shows that T-Closure can detect 7000+ closure events monthly, with a delay within 1.5 hours.

Recently, super-resolution (SR) performance has been improved by the stereo images since the beneficial information could be provided by another view. Transformer has shown significant performance gains for computer vision tasks, while it needs huge computing resources and training time. To alleviate this problem, we introduce an efficient Transformer feature extraction block, which can efficiently capture long-range pixel interactions with lower resource consumption. There are many kinds of cross-view interaction modules for stereo image SR, and they all have limitations of SR performance in their own models. To address the aforementioned challenge, we first propose the strong-weak cross-view interaction mechanism, which consists of strong cross-view interaction module and weak cross-view interaction module. Benefiting from the proposed mechanism, the SR performance can be improved significantly with a negligible increment of computing cost. We integrate the efficient Transformer feature extraction module and the strong-weak cross-view interaction mechanism into a unified framework named strong-weak cross-view interaction network (SWCVIN), and extensive experiments on three benchmark datasets show the proposed model achieves state-of-the-art results.

Learning representations for temporal knowledge graphs (TKGs) is a fundamental task. Most existing methods regard TKG as a sequence of static snapshots and recurrently learn representations by retracing the previous snapshots. However, new knowledge can be continuously accrued to TKGs as streams. These methods either cannot handle new entities or fail to update representations in real time, making them unfeasible to adapt to the streaming scenarios. In this paper, we propose a lightweight framework called StreamE towards the efficient generation of TKG representations in streaming scenarios. To reduce the parameter size, entity representations in StreamE are decoupled from the model training to serve as the memory module to store the historical information of entities. To achieve efficient update and generation, the process of generating representations is decoupled as two functions in StreamE. An update function is learned to incrementally update entity representations based on the newly-arrived knowledge and a read function is learned to predict the future semantics of entity representations. The update function avoids the recurrent modeling paradigm and thus gains high efficiency while the read function considers multiple semantic change properties. We further propose a joint training strategy with two temporal regularizations to effectively optimize the framework. Experimental results show that StreamE can achieve better performance than baseline methods with 100x faster in inference, 25x faster in training, and only 1/5 parameter size, which demonstrates its superiority. Code is available at https://github.com/zjs123/StreamE.

Knowledge graph (KG) enhanced recommendation, which aims to solve the cold start and explainability in recommender systems, has attracted considerable research interest recently. Existing recommender systems usually focus on implicit feedback such as purchase history without negative feedback. Most of them apply the negative sampling strategy to deal with the implicit feedback data, which may ignore the latent positive user-item interaction. Some other works adopt the non-sampling strategy that treats all non-observed interactions as negative samples and assigns a weight for each negative sample to represent the probability that this sample is a positive sample. However, they use a simple and intuitive weight assignment strategy and cannot catch the latent relationship from all interaction data. To address these problems, we consider graph structure information of both user-item interaction and knowledge graph, and propose a Graph-based Non-Sampling strategy to achieve efficient performance in Knowledge graph enhanced Recommendation (GNSKR). GNSKR utilizes node centrality to significantly improve recommendation performance with low computation cost. Meanwhile, we combine knowledge graph embedding and recommendation task with a local aggregation block, which efficiently catches the high-order connection information in KG enhanced recommendation. Experiments on three datasets show that GNSKR embraces the state-of-the-art with competitive efficiency.

Recently, many knowledge graph embedding models for knowledge graph completion have been proposed, ranging from the initial translation-based models such as TransE to recent convolutional neural network (CNN) models such as ConvE. However, these models only focus on semantic information of knowledge graph and neglect the natural graph structure information. Although graph convolutional network (GCN)-based models for knowledge graph embedding have been introduced to address this issue, they still suffer from fact incompleteness, resulting in the unconnectedness of knowledge graph. To solve this problem, we propose a novel model called deep relational graph infomax (DRGI) with mutual information (MI) maximization which takes the benefit of complete structure information and semantic information together. Specifically, the proposed DRGI consists of two encoders which are two identical adaptive relational graph attention networks (ARGATs), corresponding to catching semantic information and complete structure information respectively. Our method establishes new state-of-the-art on the standard datasets for knowledge graph completion. In addition, by exploring the complete structure information, DRGI embraces the merits of faster convergence speed over existing methods and better predictive performance for entities with small indegree.

Cold-start has become critical for recommendations, especially for sparse user-item interactions. Recent approaches based on meta-learning succeed in alleviating the issue, owing to the fact that these methods have strong generalization, so they can fast adapt to new tasks under cold-start settings. However, these meta-learning-based recommendation models learned with single and spase ratings are easily falling into the meta-overfitting, since the one and only rating$r_{ui}$to a specific item$i$cannot reflect a user's diverse interests under various circumstances(e.g., time, mood, age, etc), i.e. if$r_{ui}$equals to 1 in the historical dataset, but$r_{ui}$could be 0 in some circumstance. In meta-learning, tasks with these single ratings are called Non-Mutually-Exclusive(Non-ME) tasks, and tasks with diverse ratings are called Mutually-Exclusive(ME) tasks. Fortunately, a meta-augmentation technique is proposed to relief the meta-overfitting for meta-learning methods by transferring Non-ME tasks into ME tasks by adding noises to labels without changing inputs. Motivated by the meta-augmentation method, in this paper, we propose a cross-domain meta-augmentation technique for content-aware recommendation systems (MetaCAR) to construct ME tasks in the recommendation scenario. Our proposed method consists of two stages: meta-augmentation and meta-learning. In the meta-augmentation stage, we first conduct domain adaptation by a dual conditional variational autoencoder (CVAE) with a multi-view information bottleneck constraint, and then apply the learned CVAE to generate ratings for users in the target domain. In the meta-learning stage, we introduce both the true and generated ratings to construct ME tasks that enables the meta-learning recommendations to avoid meta-overfitting. Experiments evaluated in real-world datasets show the significant superiority of MetaCAR for coping with the cold-start user issue over competing baselines including cross-domain, content-aware, and meta-learning-based recommendations.

Long sequence time-series forecasting has become a central problem in multivariate time-series analysis due to its difficulty of consistently maintaining low prediction errors. Recent research has concentrated on developing large deep learning frameworks such as Informer and SCINet with remarkable results. However, these complex approaches were not benchmarked with simpler statistical methods and hence this part of the puzzle is missing for multivariate long sequence time-series forecasting (MLSTF). We investigate two simple statistical methods for MLSTF and provide analysis to indicate that linear regression owns a lower upper bound of error than deep learning methods and SNaive can act as an effective nonparametric method with unpredictable trends. Evaluations across six real-world datasets demonstrate that linear regression and SNaive are able to achieve state-of-the-art performance for MLSTF.

Recently, many knowledge graph embedding models for knowledge graph completion have been proposed, ranging from the initial translation-based models such as TransE to recent convolutional neural network (CNN) models such as ConvE. However, these models only focus on semantic information of knowledge graph and neglect the natural graph structure information. Although graph convolutional network (GCN)-based models for knowledge graph embedding have been introduced to address this issue, they still suffer from fact incompleteness, resulting in the unconnectedness of knowledge graph. To solve this problem, we propose a novel model called deep relational graph infomax (DRGI) with mutual information (MI) maximization which takes the benefit of complete structure information and semantic information together. Specifically, the proposed DRGI consists of two encoders which are two identical adaptive relational graph attention networks (ARGATs), corresponding to catching semantic information and complete structure information respectively. Our method establishes new state-of-the-art on the standard datasets for knowledge graph completion.

Cold-start issues have been more and more challenging for providing accurate recommendations with the fast increase of users and items. Most existing approaches attempt to solve the intractable problems via content-aware recommendations based on auxiliary information and/or cross-domain recommendations with transfer learning. Their performances are often constrained by the extremely sparse user-item interactions, unavailable side information, or very limited domain-shared users. Recently, meta-learners with meta-augmentation by adding noises to labels have been proven to be effective to avoid overfitting and shown good performance on new tasks. Motivated by the idea of meta-augmentation, in this paper, by treating a user's preference over items as a task, we propose a so-called Diverse Preference Augmentation framework with multiple source domains based on meta-learning (referred to as MetaDPA) to i) generate diverse ratings in a new domain of interest (known as target domain) to handle overfitting on the case of sparse interactions, and to ii) learn a preference model in the target domain via a meta-learning scheme to alleviate cold-start issues. Specifically, we first conduct multi-source domain adaptation by dual conditional variational autoencoders and impose a Multi-domain InfoMax (MDI) constraint on the latent representations to learn domain-shared and domain-specific preference properties. To avoid overfitting, we add a Mutually-Exclusive (ME) constraint on the output of decoders to generate diverse ratings given content data. Finally, these generated diverse ratings and the original ratings are introduced into the meta-training procedure to learn a preference meta-learner, which produces good generalization ability on cold-start recommendation tasks. Experiments on real-world datasets show our proposed MetaDPA clearly outperforms the current state-of-the-art baselines.

Nowadays, going out and participating in group activities is an indispensable part of human life, and group recommendation systems are needed to provide suggestions. In practice, group recommendation faces serious sparsity issues due to the lack of group-item interaction data, and the key challenge is to aggregate group member preference for group decision making. Conventional group recommendations applied a predefined strategy to aggregate the preferences of group members, which cannot model the group decision making process and do not address the data sparsity problem well. In this paper, we introduce knowledge graph into group recommendation as side information, and propose a novel end-to-end method named knowledge graph-based attentive group recommendation (KGAG) to solve the data sparsity and preference aggregation problems. Specifically, a graph convolution network (GCN) is employed to capture abundant structure information of items and users in knowledge graph to overcome the sparsity problem. Besides, to learn knowledge-aware group representation for inferring the group decision better, we capture the user-item connectivity and user-user connectivity in knowledge graph, and then adopt attention mechanism to learn the influence of each member according to user-user interaction in group and the candidate item, for member preference aggregation. Additionally, the attention mechanism can provide interpretability to group recommendation. Moreover, we extend the margin loss to our KGAG which forces the prediction score of positive item to be a distance larger than that of negative item. Experimental results show the superiority of the proposed KGAG and verify the efficacy of each component of KGAG.

In our daily life, we are exposed to a large number of video files. Compared with video containing only mono audio, video with stereo can provide us with better audio-visual experience. However, a large number of ordinary users do not have professional equipment to record videos with high-quality stereo. In order to make it more convenient for users to obtain videos with stereo, we propose an effective method to convert mono audio in the video into stereo. One of the keys to this task is how to effectively inject visual information extracted from video frames into the audio signal. We design a novel multi-attention fusion network (MAFNet) based on the self-attention mechanism to extract the spatial features related to the sound source in the video frames and fuse them into audio features well. Furthermore, in order to obtain stereo with higher quality, we design an additional iterative structure which can refine and optimize the generated stereo sound by several iterations. Our proposed approach is validated on two challenging video datasets (FAIR-Play and YT-MUSIC), and achieves new state-of-the-art performance.

The field of broadcast sports video analysis requires attention from the research community. Identifying the semantic actions within a broadcast sports video aids better video analysis and highlight generation. One of the key challenges posed to sports video analysis is the availability of relevant datasets. In this paper, we introduce a new dataset SP-2 related to broadcast sports video (available at https://github.com/abdkhanstd/Sports2). SP-2 is a large dataset with several annotations such as sports category (class), playfield scenario, and game action. Along with the introduction of this dataset, we focus on accurately classifying the broadcast sports video category and propose a simple yet elegant method for the classification of broadcast sports video. Broadcast sports video classification plays an important role in sports video analysis as different sports games follow a different set of rules and situations. Our method exploits and explores the true potential of capsule network with dynamic routing, which was introduced recently. First, we extract features using a residual convolutional neural network and build temporal feature sequences. Further, a cascaded capsule network is trained using the extracted feature sequence. Residual inception cascaded capsule network (RICAPS) significantly improves the performance of broadcast sports video classification as deeper features are captured by the cascaded capsule network. We conduct extensive experiments on SP-2 dataset and compare the results with previously proposed methods, and the results show that RICAPS outperforms the previously proposed methods.

Text-based person search aims at retrieving the most relevant pedestrian images from database in response to a query in form of natural language description. Existing algorithms mainly focus on embedding textual and visual features into a common semantic space so that the similarity score of features from different modalities can be computed directly. Softmax loss is widely adopted to classify textual and visual features into a correct category in the joint embedding space. However, softmax loss can only help classify features but not increase the intra-class compactness and inter-class discrepancy. To this end, we propose multiplicative angular margin (MAM) loss to learn angularly discriminative features for each identity. The multiplicative angular margin loss penalizes the angle between feature vector and its corresponding classifier vector to learn more discriminative feature. Moreover, to focus more on informative image-text pair, we propose pairwise similarity weighting (PSW) loss to assign higher weight to informative pairs. Extensive experimental evaluations have been conducted on the CUHK-PEDES dataset over our proposed losses. The results show the superiority of our proposed method. Code is available at https://github.com/pengzhanguestc/MAM_loss.

Entity resolution identifies all records in a database that refer to the same entity. The mainstream solutions rely on supervised learning or crowd assistance, both requiring labor overhead for data annotation. To avoid human intervention, we propose an unsupervised graph-theoretic fusion framework with two components, namely ITER and CliqueRank. Specifically, ITER constructs a weighted bipartite graph between terms and record-record pairs and iteratively propagates the node salience until convergence. Subsequently, CliqueRank constructs a record graph to estimate the likelihood of two records resident in the same clique. The derived likelihood from CliqueRank is fed back to ITER to rectify the edge weight until a joint optimum can be reached. Experimental evaluation was conducted among 14 competitors and results show that without any labeled data or crowd assistance, our unsupervised framework is comparable or even superior to state-of-the-art methods among three benchmark datasets.

Instance search aims at retrieving images containing a particular query instance. Recently, image features derived from pre-trained convolutional neural networks (CNNs) have been shown to provide promising performance for image retrieval. However, the robustness of these features is still limited by hard positives and hard negatives. To address this issue, this work focuses on reconstructing a new representation based on conventional CNN features to capture the intrinsic image manifold in the original feature space. After the feature reconstruction, the Euclidean distance can be applied in the new space to measure the pairwise distance among feature points. The proposed method is highly efficient, which benefits from the linear search complexity and a further optimization for speedup. Experiments demonstrate that our method achieves promising efficiency with highly competitive accuracy. This work succeeds in capturing implicit embedding information in images as well as reducing the computational complexity significantly.

Big data analytics is gaining massive momentum in the last few years. Applying machine learning models to big data has become an implicit requirement or an expectation for most analysis tasks, especially on high-stakes applications. Typical applications include sentiment analysis against reviews for analyzing on-line products, image classification in food logging applications for monitoring user's daily intake, and stock movement prediction. Extending traditional database systems to support the above analysis is intriguing but challenging. First, it is almost impossible to implement all machine learning models in the database engines. Second, expert knowledge is required to optimize the training and inference procedures in terms of efficiency and effectiveness, which imposes heavy burden on the system users. In this paper, we develop and present a system, called Rafiki, to provide the training and inference service of machine learning models. Rafiki provides distributed hyper-parameter tuning for the training service, and online ensemble modeling for the inference service which trades off between latency and accuracy. Experimental results confirm the efficiency, effectiveness, scalability and usability of Rafiki.

Recently, a new type of video understanding task called Movie-Fill-in-the-Blank (MovieFIB) has attracted many research attentions. Given a pair of movie clip and description with one blank word as input, MovieFIB aims to automatically predict the blank word. Because of the advantage in processing sequence data, Long-Short Term Memory (LSTM) has been used as a key component in existing MovieFIB methods to generate representations of videos and descriptions. However, most of these methods fail to emphasize the salient parts of videos. To address this problem, in this paper we propose to use a novel LSTM network called LSTM with Linguistic gate (LSTMwL), which exploits adaptive temporal attention for MovieFIB. Specifically, we first use LSTM to produce video features, which are then used to update the text representation. Finally, we put the updated text into two opposite directional LSTMwL layers to infer the blank word. Experimental results demonstrate that our approach outperforms state-of-the-art models for MovieFIB.

With the advance of various location-acquisition technologies, a myriad of GPS trajectories can be collected every day. However, the raw coordinate data captured by sensors often cannot reflect real positions due to many physical constraints and some rules of law. How to accurately match GPS trajectories to roads on a digital map is an important issue. Many existing methods still cannot meet stringent performance requirements, especially for low/unstable sampling rate and noisy/lost data. As in practice, some other measurements such as speed and moving direction are collected together with the spatial locations acquired, we can make use of not only location coordinates but all data collected. In this paper, we propose a novel model using the related meta-information to describe a moving object, and present an algorithm called IF-Matching for map-matching. It can handle many ambiguous cases which cannot be correctly matched by existing methods. We run our algorithm with taxi trajectory data on a city-wide road network. Compared with two state-of-the-art algorithms of ST-Matching and the winner of GIS Cup 2012, our approach achieves more accurate results.

Locality sensitive hashing (LSH) and its variants are widely used for approximate kNN (k nearest neighbor) search in high-dimensional space. The success of these techniques largely depends on the ability of preserving kNN information. Unfortunately, LSH only provides a high probability that nearby points in the original space are projected into nearby region in a new space. This potentially makes many false positives and false negatives resulting from unrelated points. Many extensions of LSH aim to alleviate the above issue by improving the distance preserving ability. In this paper, we abound improving LSH function but propose a novel idea to enhance the performance by transforming the original data to a new space before applying LSH. A preserving-ignoring transformation (PIT) function satisfying some rigorous conditions can be used to convert original points to an interim space with strict distance preserving-ignoring capacity. Based on this property, a linear order is utilized to build an efficient index structure in the interim space. Finally, LSH can be applied to candidate set searched by our index structure for final results. Experiments are conducted and the proposed approach performs better than state-of-the-art methods SK-LSH, DSH and NSH in terms of both accuracy and efficiency.

The prevalence of GPS devices has facilitated collection of large-scale trajectories. Fresh positions of moving objects can be sampled periodically and sent to servers for data analytics and query processing. Online trajectory simplification is a compression task usually conducted at the sensor side and serves as a key component to reduce network communication overhead. In this paper, we study a new trajectory simplification problem which is direction-preserving and works in an online fashion. An efficient simplification algorithm is proposed, which is guaranteed to be error-bounded and achieves O(n) time and O(1) space complexity. In an extensive experimental evaluation with two real datasets, our approach exhibits superior performance on both running time and compression rate.

Zero-shot learning (ZSL) aims to bridge the knowledge transfer via available semantic representations (e.g., attributes) between labeled source instances of seen classes and unlabelled target instances of unseen classes. Most existing ZSL approaches achieve this by learning a projection from the visual feature space to the semantic representation space based on the source instances, and directly applying it to the target instances. However, the intrinsic manifold structures residing in both semantic representations and visual features are not effectively incorporated into the learned projection function. Moreover, these methods may suffer from the inherent projection shift problem, due to the disjointness between seen and unseen classes. To overcome these drawbacks, we propose a novel framework termed transductive visual-semantic embedding (TVSE) for ZSL. In specific, TVSE first learns a latent embedding space to incorporate the manifold structures in both labeled source instances and unlabeled target instances under the transductive setting. In the learned space, each instance is viewed as a mixture of seen class scores. TVSE then effectively constructs the relational mapping between seen and unseen classes using the available semantic representations, and applies it to map the seen class scores of the target instances to their predictions of unseen classes. Extensive experiments on four benchmark datasets demonstrate that the proposed TVSE achieves competitive performance compared with the state-of-the-arts for zero-shot recognition and retrieval tasks.

Travel route planning aims to mine user's attributes and recommend personalized routes. How to build interest model for users and understand their real intention brings great challenges. This paper presents an approach which mines the user interest model by multi-source social media (e.g., travelogues and check-in records), and understands the user's real intention by active behavior such as point of interest (POI) inputs. In order to unify heterogeneous data from different sources, a topical package is built as the measurement space. Based on the topical package, user topical package is modeled to find user interest and route topical package is constructed to describe the attributes of each route. User's active behavior can also be considered during route planning, where top ranked routes are finally recommended. The proposed multi-source topical package (MSTP) approach is evaluated on a real dataset and compared with two state-of-the-art methods. The result shows that MSTP performs better for providing personalized travel routes.

With the advance of various location-acquisition technologies, a myriad of GPS trajectories can be collected every day. However, the raw coordinate data captured by sensors often cannot reflect real positions due to many physical constraints and some rules of law. How to accurately match GPS trajectories to roads on a digital map is an important issue. The problem of map-matching is fundamental for many applications. Unfortunately, many existing methods still cannot meet stringent performance requirements in engineering. In particular, low/unstable sampling rate and noisy/lost data are usually big challenges. Information fusion of different data sources is becoming increasingly promising nowadays. As in practice, some other measurements such as speed and moving direction are collected together with the spatial locations acquired, we can make use of not only location coordinates but all data collected. In this paper, we propose a novel model using the related meta-information to describe a moving object, and present an algorithm called IF-Matching for map-matching. It can handle many ambiguous cases which cannot be correctly matched by existing methods. We run our algorithm with taxi trajectory data on a city-wide road network. Compared with two state-of-the-art algorithms of ST-Matching and the winner of GIS Cup 2012, our approach achieves more accurate results.

In spite of vast business potential, targeted advertising in public transportation systems is a grossly unexplored research area. For instance, SBS Transit in Singapore can reach 1 billion passengers per year but the annual advertising revenue contributes less than $35 million. To bridge the gap, we propose a probabilistic data model that captures the motion patterns and user interests so as to quantitatively evaluate the impact of an advertisement among the passengers. In particular, we leverage hundreds of millions of bus/train boarding transaction records to quantitatively estimate the probability as well as the extent of a user being influenced by an ad. Based on the influence model, we study a top-kretrieval problem for bus/train ad recommendation, which acts as a primitive operator to support various advanced applications. We solve the retrieval problem efficiently to support real-time decision making. In the experimental study, we use the dataset from SBS Transit as a case study to verify the effectiveness and efficiency of our proposed methodologies.

Research on cognitive science indicates that humans often use different criteria for route selection. An alternative type of spatial proximity search on road networks recently has been proposed to find the easiest-to-reach neighboring object with the smallest navigation complexity. This paper presents an evaluation to compare the effectiveness of easiest-to-reach neighbor query against a classic nearest neighbor query in a real-world setting. Our user study demonstrates usability of the new spatial query type and suggests people may not always care about travel distance most. To provide flexibility to accommodate different requirements, we also show how to achieve tradeoff between navigation complexity and travel distance for advanced navigational assistance.

With the rapid development of GPS-enabled mobile devices, huge amounts of user-contributed data with location information can be collected from the Internet. With this kind of data, one promising application is travel recommendation, which has attracted a considerable number of researches recently. However, most of the previous studies only focus on one aspect of the relations among users and locations or make a coarse linear combination of the relations. Moreover, all the existing work on travel recommendation do not consider recommendation to groups, which is an important characteristic of travelers' behavior. In this paper, we present a personalized travel recommendation system named Where to Go. The novelty of the system is a 3R model which can unify user-location relation, user-user relation and location-location relation into a single framework and perform random walk with restart to analyze the model. We further extend our approach to provide recommendations for groups. To the best of our knowledge, this is the first work to use random walk with restart for group recommendation. We conduct a comprehensive performance evaluation using a real dataset collected from Flickr, which is one of the most popular online photo-sharing sites. Experimental results show that our approach provides significantly superior recommendation quality compared to other state-of-the-art travel recommendation approaches for both individuals and groups.

Image tagging, also known as image annotation and image conception detection, has been extensively studied in the literature. However, most existing approaches can hardly achieve satisfactory performance owing to the deficiency and unreliability of the manually-labeled training data. In this paper, we propose a new image tagging scheme, termed social assisted media tagging (SAMT), which leverages the abundant user-generated images and the associated tags as the "social assistance" to learn the classifiers. We focus on addressing the following major challenges: (a) the noisy tags associated to the web images; and (b) the desirable robustness of the tagging model. We present a joint image tagging framework which simultaneously refines the erroneous tags of the web images as well as learns the reliable image classifiers. In particular, we devise a novel tag refinement module for identifying and eliminating the noisy tags by substantially exploring and preserving the low-rank nature of the tag matrix and the structured sparse property of the tag errors. We develop a robust image tagging module based on the l2,p-norm for learning the reliable image classifiers. The correlation of the two modules is well explored within the joint framework to reinforce each other. Extensive experiments on two real-world social image databases illustrate the superiority of the proposed approach as compared to the existing methods.

We study the problem of finding the shortest route between two locations that includes a stopover of a given type. An example scenario of this problem is given as follows: “On the way to Bob's place, Alice searches for a nearby take-away Italian restaurant to buy a pizza.” Assuming that Alice is interested in minimizing the total trip distance, this scenario can be modeled as a query where the current Alice's location (start) and Bob's place (destination) function as query points. Based on these two query points, we find the minimum detour object (MDO), i.e., a stopover that minimizes the sum of the distances: 1) from the start to the stopover, and 2) from the stopover to the destination. In a realistic location-based application environment, a user can be indecisive about committing to a particular detour option. The user may wish to browse multiple (k) MDOs before making a decision. Furthermore, when a user moves, the k{\rm MDO} results at one location may become obsolete. We propose a method for continuous detour query (CDQ) processing based on incremental construction of a shortest path tree. We conducted experimental studies to compare the performance of our proposed method against two methods derived from existing k-nearest neighbor querying techniques using real road-network data sets. Experimental results show that our proposed method significantly outperforms the two competitive techniques.

Studies in cognitive science have shown that people have differ-ent optimization goals in mind for route selection: beyond shortest travel distance (or time), criteria such as smallest number of turns or straightest path are often considered. A common query that a traveller in a foreign city may ask is “where is a facility of type X”. When multiple facilities of the same type are available in the nearby area, usually not the nearest neighbor but the one which is easiest to find is preferred for giving instructions by locals, especially in an unfamiliar and complex urban environment. This paper studies a novel type of neighboring object selection problem, taking cog-nitive complexity of navigation into account. The main difficulty arises from incorporating spatial chunking and landmark informa-tion into neighbor comparisons. We propose an algorithm based on network expansion, which uses incremental processing of graph transformation that models instruction complexity. Our approach can efficiently find the easiest-to-reach neighbor with the guaran-teed smallest navigation cost. Through experimental evaluation on real road networks, the performance of the proposed algorithm is demonstrated under various settings. Our comparison results reveal that on average the travel distance of the easiest-to-reach neighbor is only 19.3 % longer than that of the nearest neighbor, whereas the navigation cost can achieve a 64.8 % reduction.

Video search has become a compelling research topic in recent years, due to the proliferation of online video uploading/sharing sites and the exponential explosion of video data. In this demonstration, we showcase a Web-based integrated platform which performs online detection of near-duplicate occurrences over continuous video streams, as well as retrieval of near-duplicate clips from segmented video collections. In particular, our method to detect relevant subsequences in a streaming video is characterized by a novel one-dimensional distance trajectory capturing the changes of consecutive frames. Such a trajectory is further represented by a sequence of compact signatures. An effective similarity measure is devised to compare the trajectory with multiple query videos. This system shows a number of new features compared with our previous prototype.

With the growing demand for visual information of rich content, effective and efficient manipulations of large video databases are increasingly desired. Many investigations have been made on content-based video retrieval. However, despite the importance, video subsequence identification, which is to find the similar content to a short query clip from a long video sequence, has not been well addressed. This paper presents a graph transformation and matching approach to this problem, with extension to identify the occurrence of potentially different ordering or length due to content editing. With a novel batch query algorithm to retrieve similar frames, the mapping relationship between the query and database video is first represented by a bipartite graph. The densely matched parts along the long sequence are then extracted, followed by a filter-and-refine search strategy to prune some irrelevant subsequences. During the filtering stage, maximum size matching is deployed for each subgraph constructed by the query and candidate subsequence to obtain a smaller set of candidates. During the refinement stage, sub-maximum similarity matching is devised to identify the subsequence with the highest aggregate score from all candidates, according to a robust video similarity model that incorporates visual content, temporal order, and frame alignment information. The performance studies conducted on a long video recording of 50 hours validate that our approach is promising in terms of both search accuracy and speed.

Recently, video clips have become very popular online. The massive influx of video clips has created an urgent need for video search engines to facilitate retrieving relevant clips. Different from traditional long videos, a video clip is a short video often expressing a moment of significance. Due to the high complexity of video data, efficient video clip search from large databases turns out to be very challenging. We propose a novel video clip representation model called the Bounded Coordinate System (BCS), which is the first single representative capturing the dominating content and content—changing trends of a video clip. It summarizes a video clip by a coordinate system, where each of its coordinate axes is identified by principal component analysis (PCA) and bounded by the range of data projections along the axis. The similarity measure of BCS considers the operations of translation, rotation, and scaling for coordinate system matching. Particularly, rotation and scaling reflect the difference of content tendencies. Compared with the quadratic time complexity of existing methods, the time complexity of measuring BCS similarity is linear. The compact video representation together with its linear similarity measure makes real-time search from video clip collections feasible. To further improve the retrieval efficiency for large video databases, a two-dimensional transformation method called Bidistance Transformation (BDT) is introduced to utilize a pair of optimal reference points with respect to bidirectional axes in BCS. Our extensive performance study on a large database of more than 30,000 video clips demonstrates that BCS achieves very high search accuracy according to human judgment. This indicates that content tendencies are important in determining the meanings of video clips and confirms that BCS can capture the inherent moment of video clip to some extent that better resembles human perception. In addition, BDT outperforms existing indexing methods greatly. Integration of the BCS model and BDT indexing can achieve real-time search from large video clip databases.

Searching relevant visual information based on content features in large databases is an interesting and changeling topic that has drawn lots of attention from both the research community and industry. This paper gives an overview of our investigations on effective and efficient video similarity search. We briefly introduce some novel techniques developed for two specific tasks studied in this PhD project: video retrieval in a large collection of segmented video clips, and video subsequence identification from a long unsegmented stream. The proposed methods for processing these two types of similarity queries have shown encouraging performance and are being incorporated into our prototype system of video search named UQLIPS, which has demonstrated some marketing potentials for commercialisation.

To retrieve similar database videos to a query clip, each video is typically represented by a sequence of high-dimensional feature vectors. Given a query video containing m feature vectors, an independent nearest neighbor (NN) search for each feature vector is often first performed. Completing all the NN searches, an overall similarity is then computed, i.e., a single video retrieval usually involves the searches for m times. Since normally nearby feature vectors in a video are similar, a large number of expensive random disk accesses are expected to repeatedly occur, which crucially affects the overall query performance. Batch nearest neighbor (BNN) search is stated as a single operation that performs a batch of individual NN searches. This paper presents a novel approach to efficient high-dimensional BNN search called dynamic query ordering (DQO) for advanced optimizations in both I/O and CPU cost. Observing the overlapped candidates (or search space) of a pervious query may help to further reduce the candidate sets of succeeding queries, DQO aims to progressively find a query order such that the common candidates among queries are fully utilized to maximally reduce the total number of candidates. Modelling the candidate set relationship by a candidate overlapping graph (COG), DQO iteratively selects the next query to be executed based on its estimated pruning power to the rest of queries with the dynamically updated COG. The extensive experiments show its significance.