In recent years, the success of large language models (LLMs) has driven the exploration of scaling laws in recommender systems. However, models that demonstrate scaling laws are actually challenging to deploy in industrial settings for modeling long sequences of user behaviors, due to the high computational complexity of the standard self-attention mechanism. Despite various sparse self-attention mechanisms proposed in other fields, they are not fully suited for recommendation scenarios. This is because user behaviors exhibit personalization and temporal characteristics: different users have distinct behavior patterns, and these patterns change over time, with data from these users differing significantly from data in other fields in terms of distribution. To address these challenges, we propose SparseCTR, an efficient and effective model specifically designed for long-term behaviors of users. To be precise, we first segment behavior sequences into chunks in a personalized manner to avoid separating continuous behaviors and enable parallel processing of sequences. Based on these chunks, we propose a three-branch sparse self-attention mechanism to jointly identify users' global interests, interest transitions, and short-term interests. Furthermore, we design a composite relative temporal encoding via learnable, head-specific bias coefficients, better capturing sequential and periodic relationships among user behaviors. Extensive experimental results show that SparseCTR not only improves efficiency but also outperforms state-of-the-art methods. More importantly, it exhibits an obvious scaling law phenomenon, maintaining performance improvements across three orders of magnitude in FLOPs. In online A/B testing, SparseCTR increased CTR by 1.72% and CPM by 1.41%. Our source code is available at https://github.com/laiweijiang/SparseCTR.

Click-Through Rate (CTR) models play a vital role in improving user experience and boosting business revenue in many online personalized services.However, current CTR models generally encounter bottlenecks in performance improvement.Inspired by the scaling law phenomenon of Large Language Models (LLMs), we propose a new paradigm for improving CTR predictions: first, constructing a CTR model with accuracy scalable to the model grade and data size, and then distilling the knowledge implied in this model into its lightweight model that can serve online users.To put it into practice, we construct a CTR model named SUAN (Stacked Unified Attention Network).In SUAN, we propose the unified attention block (UAB) as a behavior sequence encoder.A single UAB unifies the modeling of the sequential and non-sequential features and also measures the importance of each user behavior feature from multiple perspectives.Stacked UABs elevate the configuration to a high grade, paving the way for performance improvement.In order to benefit from the high performance of the high-grade SUAN and avoid the disadvantage of its long inference time, we modify the SUAN with sparse self-attention and parallel inference strategies to form LightSUAN, and then adopt online distillation to train the low-grade LightSUAN, taking a high-grade SUAN as a teacher.The distilled LightSUAN has superior performance but the same inference time as the LightSUAN, making it well-suited for online deployment.Experimental results show that SUAN performs exceptionally well and holds the scaling laws spanning three orders * Corresponding author.

CTR (Click-Through Rate) prediction, crucial for recommender systems and online advertising, etc., has been confirmed to benefit from modeling long-term user behaviors. Nonetheless, the vast number of behaviors and complexity of noise interference pose challenges to prediction efficiency and effectiveness. Recent solutions have evolved from single-stage models to two-stage models. However, current two-stage models often filter out significant information, resulting in an inability to capture diverse user interests and build the complete latent space of user interests. Inspired by multi-interest and generative modeling, we propose DiffuMIN (Diffusion-driven Multi-Interest Network) to model long-term user behaviors and thoroughly explore the user interest space. Specifically, we propose a target-oriented multi-interest extraction method that begins by orthogonally decomposing the target to obtain interest channels. This is followed by modeling the relationships between interest channels and user behaviors to disentangle and extract multiple user interests. We then adopt a diffusion module guided by contextual interests and interest channels, which anchor users' personalized and target-oriented interest types, enabling the generation of augmented interests that align with the latent spaces of user interests, thereby further exploring restricted interest space. Finally, we leverage contrastive learning to ensure that the generated augmented interests align with users' genuine preferences. Extensive offline experiments are conducted on two public datasets and one industrial dataset, yielding results that demonstrate the superiority of DiffuMIN. Moreover, DiffuMIN increased CTR by 1.52% and CPM by 1.10% in online A/B testing. Our source code is available at https://github.com/laiweijiang/DiffuMIN.

User interests can be viewed over different time scales, mainly including stable long-term preferences and changing short-term intentions, and their combination facilitates the comprehensive sequential recommendation. However, existing work that focuses on different time scales of user modeling has ignored the negative effects of different time-scale noise, which hinders capturing actual user interests and cannot be resolved by conventional sequential denoising methods. In this paper, we propose a Long- and Short-term Interest Denoising Network (LSIDN), which employs different encoders and tailored denoising strategies to extract long- and short-term interests, respectively, achieving both comprehensive and robust user modeling. Specifically, we employ a session-level interest extraction and evolution strategy to avoid introducing inter-session behavioral noise into long-term interest modeling; we also adopt contrastive learning equipped with a homogeneous exchanging augmentation to alleviate the impact of unintentional behavioral noise on short-term interest modeling. Results of experiments on two public datasets show that LSIDN consistently outperforms state-of-the-art models and achieves significant robustness.

User behavior sequence modeling plays a significant role in Click-Through Rate (CTR) prediction on e-commerce platforms. Except for the interacted items, user behaviors contain rich interaction information, such as the behavior type, time, location, etc. However, so far, the information related to user behaviors has not yet been fully exploited. In the paper, we propose the concept of a situation and situational features for distinguishing interaction behaviors and then design a CTR model named Deep Situation-Aware Interaction Network (DSAIN). DSAIN first adopts the reparameterization trick to reduce noise in the original user behavior sequences. Then it learns the embeddings of situational features by feature embedding parameterization and tri-directional correlation fusion. Finally, it obtains the embedding of behavior sequence via heterogeneous situation aggregation. We conduct extensive offline experiments on three real-world datasets. Experimental results demonstrate the superiority of the proposed DSAIN model. More importantly, DSAIN has increased the CTR by 2.70%, the CPM by 2.62%, and the GMV by 2.16% in the online A/B test. Now, DSAIN has been deployed on the Meituan food delivery platform and serves the main traffic of the Meituan takeout app. Our source code is available at https://github.com/W-void/DSAIN.

With the rapid increase of micro-video creators and viewers, how to make personalized recommendations from a large number of candidates to viewers begins to attract more and more attention. However, existing micro-video recommendation models rely on expensive multi-modal information and learn an overall interest embedding that cannot reflect the user's multiple interests in micro-videos. Recently, contrastive learning provides a new opportunity for refining the existing recommendation techniques. Therefore, in this paper, we propose to extract contrastive multi-interests and devise a micro-video recommendation model CMI. Specifically, CMI learns multiple interest embeddings for each user from his/her historical interaction sequence, in which the implicit orthogonal micro-video categories are used to decouple multiple user interests. Moreover, it establishes the contrastive multi-interest loss to improve the robustness of interest embeddings and the performance of recommendations. The results of experiments on two micro-video datasets demonstrate that CMI achieves state-of-the-art performance over existing baselines.

Document retrieval (DR) is a crucial task in NLP. Recently, the pre-trained BERT-like language models have achieved remarkable success, obtaining a state-of-the-art result in DR. In this paper, we come up with a new BERT-based ranking model for DR task, named TABLE. In the pre-training stage of TABLE, we present a domain-adaptive strategy. More essentially, in the fine-tuning stage, we develop a two-phase task-adaptive process, i.e., type-adaptive pointwise fine-tuning and listwise fine-tuning. In the type-adaptive pointwise fine-tuning phase, the model can learn different matching patterns regarding different query types. In the listwise fine-tuning phase, the model matches documents with regard to a given query in a listwise fashion. This task-adaptive process makes the model more robust. In addition, a simple but effective exact matching feature is introduced in fine-tuning, which can effectively compute matching of out-of-vocabulary (OOV) words between a query and a document. As far as we know, we are the first who propose a listwise ranking model with BERT. This work can explore rich matching features between queries and documents. Therefore it substantially improves model performance in DR. Notably, our TABLE model shows excellent performance on the MS MARCO leaderboard.

On-demand cinemas are a new type of offline entertainment venues which have shown the rapid expansion in the recent years. Recommending movies of interest to the potential audiences in on-demand cinemas is keen but challenging because the recommendation scenario is totally different from all the existing recommendation applications including online video recommendation, offline item recommendation and group recommendation. In this paper, we propose a novel spatio-temporal approach called Pegasus. Because of the specific characteristics of on-demand cinema recommendation, Pegasus exploits the POI (Point of Interest) information around cinemas and the content descriptions of movies, apart from the historical movie consumption records of cinemas. Pegasus explores the temporal dynamics and spatial influences rooted in audience behaviors, and captures the similarities between cinemas, the changes of audience crowds, time-varying features and regional disparities of movie popularity. It offers an effective and explainable way to recommend movies to on-demand cinemas. The corresponding Pegasus system has been deployed in some pilot on-demand cinemas. Based on the real-world data from on-demand cinemas, extensive experiments as well as pilot tests are conducted. Both experimental results and post-deployment feedback show that Pegasus is effective.

In large and medium-sized cities, detecting unusual changes of crowds of people on the streets is needed for public security, transportation management, emergency control, and terrorism prevention. As public transportation has the capability to bring a large number of people to an area in a short amount of time, real-time discovery of anomalies in passenger numbers is an effective way to detect crowd anomalies. In this paper, we devise an approach called Kochab. Kochab adopts a generative model and combines the prior knowledge about passenger flows. Hence, it can detect anomalies in the numbers of incoming and outgoing passengers within a certain time and spatial area, including anomalous events along with their durations and severities. Through well-designed inference algorithms, Kochab requires only a moderate amount of historical data to be sample data. As such, Kochab shows good performance in real time and makes prompt responses to user' s interactive analysis requests. In particular, based on the recognized anomalous events, we capture event patterns which give us hints to link to activities or status in cities. In addition, for the convenience of method evaluation and comparison, we create an open Stream Anomaly Benchmark on the basis of large-scale real-world data. This benchmark will prove useful for other researchers too. Using this benchmark, we compare Kochab with four other methods. The experimental results show that Kochab is sensitive to population flow anomalies and has superior accuracy in detecting anomalies in terms of precision, recall and the F1 score.

Vehicles on the roads have high heterogeneity in vehicle types. Real-time and full-coverage vehicle classification has always been a challenge. Existing intrusive and non-intrusive methods cannot meet the requirements with satisfaction. Considering that signaling data from mobile operators have the advantages such as the wide coverage and the low cost, a new approach named Lepus, which analyzes the signaling stream to achieve the real-time multi-class classification of vehicles on highways, is proposed. Following the Lepus, the historical GPS trajectories with vehicle types and the signaling trajectories occurring at the same time and space are first examined to establish the relation among signaling trajectories, vehicles and vehicle types and then identify signaling-recognizable vehicles. Further, the driving characteristics of these labeled signaling-recognizable vehicles are analyzed so as to determine vehicle classification features. Finally, the vehicle classification model is established and used to analyze the incoming signaling stream and classify the vehicles in real time. Extensive experiments are conducted on real data and the results show that the Lepus approach is effective in real time vehicle classification.

Analyzing and mining trajectories of moving objects (such as persons or vehicles) in the cities bring a promising way to discover the potential knowledge and therefore can foster diversified applications, including personalized travel services, intelligent transportation systems (ITSs), and etc. For increasing the intelligence of current public transit systems, the paper proposes to discover and utilize the patterns in passenger trajectory streams to optimize bus scheduling. More specifically, the paper first analyzes the real world data from bus smart cards so as to fully understand the nature of passenger trajectories and bus operations. Based on it, the paper defines a new trip pattern, i.e., the frequent bus passenger trip pattern for bus scheduling (the FBPT4BS pattern in short). Then, the paper proposes an approach. The approach gives the procedure of discovering FBPT4BS patterns from passenger trajectory streams and finds the bus lines whose capacities are not enough to satisfy the passengers' travel demands. Further, the approach gives the suggestion on the corresponding scheduling adjustment strategy for bus lines. Experiments are conducted on the data from the Beijing Public Transport Group. The experimental results show that the proposed approach can efficiently decrease the travel times of passengers.

The newly emerging location-based social networks (LBSN) such as Tinder and Momo extends social interaction from friends to strangers, providing novel experiences of making new friends. Familiar strangers refer to the strangers who meet frequently in daily life and may share common interests; thus they may be good candidates for friend recommendation. In this paper, we study the problem of discovering familiar strangers, specifically, public transportation trip companions, and their common interests. We collect 5.7 million transaction records of smart cards from about 3.02 million people in the city of Beijing, China. We first analyze this dataset and reveal the temporal and spatial characteristics of passenger encounter behaviors. Then we propose a stability metric to measure hidden friend relations. This metric facilitates us to employ community detection techniques to capture the communities of trip companions. Further, we infer common interests of each community using a topic model, i.e., LDA4HFC (Latent Dirichlet Allocation for Hidden Friend Communities) model. Such topics for communities help to understand how hidden friend clusters are formed. We evaluate our method using large-scale and real-world datasets, consisting of two-week smart card records and 901,855 Point of Interests (POIs) in Beijing. The results show that our method outperforms three baseline methods with higher recommendation accuracy. Moreover, our case study demonstrates that the discovered topics interpret the communities very well.

Ontology reuse offers great benefits by measuring and comparing ontologies. However, the state of art approaches for measuring ontologies neglects the problems of both the polymorphism of ontology representation and the addition of implicit semantic knowledge. One way to tackle these problems is to devise a mechanism for ontology measurement that is stable, the basic criteria for automatic measurement. In this paper, we present a graph derivation representation based approach (GDR) for stable semantic measurement, which captures structural semantics of ontologies and addresses those problems that cause unstable measurement of ontologies. This paper makes three original contributions. First, we introduce and define the concept of semantic measurement and the concept of stable measurement. We present the GDR based approach, a three-phase process to transform an ontology to its GDR. Second, we formally analyze important properties of GDRs based on which stable semantic measurement and comparison can be achieved successfully. Third but not the least, we compare our GDR based approach with existing graph based methods using a dozen real world exemplar ontologies. Our experimental comparison is conducted based on nine ontology measurement entities and distance metric, which stably compares the similarity of two ontologies in terms of their GDRs.