In this paper, we propose a new ranking model DR-BERT, which improves the Document Retrieval (DR) task by a task-adaptive training process and a Segmented Token Recovery Mechanism (STRM).In the task-adaptive training, we first pre-train DR-BERT to be domain-adaptive and then make the two-phase fine-tuning.In the first-phase fine-tuning, the model learns query-document matching patterns regarding different query types in a pointwise way.Next, in the second-phase finetuning, the model learns document-level ranking features and ranks documents with regard to a given query in a listwise manner.Such pointwise plus listwise fine-tuning enables the model to minimize errors in the document ranking by incorporating ranking-specific supervisions.Meanwhile, the model derived from pointwise fine-tuning is also used to reduce noise in the training data of the listwise fine-tuning.On the other hand, we present STRM which can compute OOV word representation and contextualization more precisely in BERT-based models.As an effective strategy in DR-BERT, STRM improves the matching perfromance of OOV words between a query and a document.Notably, our DR-BERT model keeps in the top three on the MS MARCO leaderboard since May 20, 2020.

Xingwu Sun, Yanling Cui, Hongyin Tang, Qiuyu Zhu, Fuzheng Zhang, Beihong Jin. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

Sponsored search ads appear next to search results when people look for products and services on search engines. In recent years, they have become one of the most lucrative channels for marketing. As the fundamental basis of search ads, relevance modeling has attracted increasing attention due to the significant research challenges and tremendous practical value. Most existing approaches solely rely on the semantic information in the input query-ad pair, while the pure semantic information in the short ads data is not sufficient to fully identify user's search intents. Our motivation lies in incorporating the tremendous amount of unsupervised user behavior data from the historical search logs as the complementary graph to facilitate relevance modeling. In this paper, we extensively investigate how to naturally fuse the semantic textual information with the user behavior graph, and further propose three novel AdsGNN models to aggregate topological neighborhood from the perspectives of nodes, edges and tokens. Furthermore, two critical but rarely investigated problems, domain-specific pre-training and long-tail ads matching, are studied thoroughly. Empirically, we evaluate the AdsGNN models over the large industry dataset, and the experimental results of online/offline tests consistently demonstrate the superiority of our proposal.

Text encoders based on C-DSSM or transformers have demonstrated strong performance in many Natural Language Processing (NLP) tasks. Low latency variants of these models have also been developed in recent years in order to apply them in the field of sponsored search which has strict computational constraints. However these models are not the panacea to solve all the Natural Language Understanding (NLU) challenges as the pure semantic information in the data is not sufficient to fully identify the user intents. We propose the TextGNN model that naturally extends the strong twin tower structured encoders with the complementary graph information from user historical behaviors, which serves as a natural guide to help us better understand the intents and hence generate better language representations. The model inherits all the benefits of twin tower models such as C-DSSM and TwinBERT so that it can still be used in the low latency environment while achieving a significant performance gain than the strong encoder-only counterpart baseline models in both offline evaluations and online production system. In offline experiments, the model achieves a 0.14% overall increase in ROC-AUC with a 1% increased accuracy for long-tail low-frequency Ads, and in the online A/B testing, the model shows a 2.03% increase in Revenue Per Mille with a 2.32% decrease in Ad defect rate.

Document retrieval (DR) is a crucial task in NLP. Recently, the pre-trained BERT-like language models have achieved remarkable success, obtaining a state-of-the-art result in DR. In this paper, we come up with a new BERT-based ranking model for DR task, named TABLE. In the pre-training stage of TABLE, we present a domain-adaptive strategy. More essentially, in the fine-tuning stage, we develop a two-phase task-adaptive process, i.e., type-adaptive pointwise fine-tuning and listwise fine-tuning. In the type-adaptive pointwise fine-tuning phase, the model can learn different matching patterns regarding different query types. In the listwise fine-tuning phase, the model matches documents with regard to a given query in a listwise fashion. This task-adaptive process makes the model more robust. In addition, a simple but effective exact matching feature is introduced in fine-tuning, which can effectively compute matching of out-of-vocabulary (OOV) words between a query and a document. As far as we know, we are the first who propose a listwise ranking model with BERT. This work can explore rich matching features between queries and documents. Therefore it substantially improves model performance in DR. Notably, our TABLE model shows excellent performance on the MS MARCO leaderboard.

Predicting the passenger flows in a city, especially in a metropolis, can guide traffic dispersion, and help assessing the risks of public safety and improving urban planning. However, it is challenging as passenger flows in a road network may vary with time and space, affected by weather conditions, urban activities, etc. In the paper, we propose a passenger flow prediction approach named Yildun, which constructs an encoder-decoder neural network and captures the spatial and temporal correlations inherent in passenger flows. More specifically, to predict the passenger flows at each and every station, a spatial attention mechanism is presented to adaptively extract inter-station correlations of flows by referring to the previous hidden state of the encoder at each time step. Meanwhile, a temporal attention mechanism is employed to capture time-dependent connections of flows by selecting relevant hidden states of the encoder across all time steps. Further, extra factors, such as POI (Point of Interest) data and day of the week, are fused in the decoder. With this spatio-temporal attention scheme, Yildun not only can make predictions effectively, but also is easily explainable. Extensive experiments are conducted on large-scale real-world data. The experimental results show that Yildun can predict passenger flows with small prediction errors and outperforms five baselines significantly.

Vehicles on the roads have high heterogeneity in vehicle types. Real-time and full-coverage vehicle classification has always been a challenge. Existing intrusive and non-intrusive methods cannot meet the requirements with satisfaction. Considering that signaling data from mobile operators have the advantages such as the wide coverage and the low cost, a new approach named Lepus, which analyzes the signaling stream to achieve the real-time multi-class classification of vehicles on highways, is proposed. Following the Lepus, the historical GPS trajectories with vehicle types and the signaling trajectories occurring at the same time and space are first examined to establish the relation among signaling trajectories, vehicles and vehicle types and then identify signaling-recognizable vehicles. Further, the driving characteristics of these labeled signaling-recognizable vehicles are analyzed so as to determine vehicle classification features. Finally, the vehicle classification model is established and used to analyze the incoming signaling stream and classify the vehicles in real time. Extensive experiments are conducted on real data and the results show that the Lepus approach is effective in real time vehicle classification.

Sensory data are often of low quality, for example, data are incomplete, ambiguous, or indirect, which has become the bottleneck of many data-driven applications. Two kinds of data which are handled in the paper for estimating traffic volumes on highways are no exception. In particular, the traffic volume data obtained from the loop detectors are accurate but sparse, and the mobile signaling data for estimating relative traffic volumes are wide in coverage and low in cost, but they are indirect and inaccurate. Keeping the characteristics of data in mind, the paper proposes a data fusion approach named Polaris which extends compressive sensing to estimate traffic volumes on highways. The Polaris analyzes the sparsity of the traffic volumes reported by detectors, mines the spatial-temporal correlations between the two kinds of data, and then gives the computational steps in the light of compressive sensing. Experiments are conducted on the large-scale real signaling data and the loop detector data. The experimental results show that the Polaris has the lowest estimation errors in comparison with several other methods. The corresponding Polaris system has been built and deployed in Fujian Province, China. It can obtain real-time traffic volumes on the highways with full coverage at a very low cost.1

The newly emerging location-based social networks (LBSN) such as Tinder and Momo extends social interaction from friends to strangers, providing novel experiences of making new friends. Familiar strangers refer to the strangers who meet frequently in daily life and may share common interests; thus they may be good candidates for friend recommendation. In this paper, we study the problem of discovering familiar strangers, specifically, public transportation trip companions, and their common interests. We collect 5.7 million transaction records of smart cards from about 3.02 million people in the city of Beijing, China. We first analyze this dataset and reveal the temporal and spatial characteristics of passenger encounter behaviors. Then we propose a stability metric to measure hidden friend relations. This metric facilitates us to employ community detection techniques to capture the communities of trip companions. Further, we infer common interests of each community using a topic model, i.e., LDA4HFC (Latent Dirichlet Allocation for Hidden Friend Communities) model. Such topics for communities help to understand how hidden friend clusters are formed. We evaluate our method using large-scale and real-world datasets, consisting of two-week smart card records and 901,855 Point of Interests (POIs) in Beijing. The results show that our method outperforms three baseline methods with higher recommendation accuracy. Moreover, our case study demonstrates that the discovered topics interpret the communities very well.

With the aid of ubiquitous sensors/devices and pervasive networks, various types of multi-source data in the form of texts, videos, pictures, audios, etc. can be collected. They can be applied to detecting the traffic conditions. For highways, it has inevitable inherent defects in detecting traffic conditions by analyzing data from a single source or user participation. Therefore, the paper proposes a data fusion approach named Phecda. Phecda combines the signaling data of mobile phones with the data from static loop detectors. Phecda works in an unobtrusive way, it not only incorporates with the characteristics of traffic flows, but also includes the strategies of setting and optimizing parameters by learning historical data. Experiments are conducted with the large-scale real-world data as input. The experimental results show that the Phecda approach has high precisions and recalls in vehicle speed estimation. The corresponding Phecda system has been built and deployed in Fujian Province, China. It achieves the highway traffic monitoring with full road segment coverage at a very low cost.

In the infotainment applications over Vehicular Ad Hoc Networks(VANETs), the Roadside Units (RSUs) often play the roles of data switching centers. However, the RSUs tend to be the bottlenecks in the data dissemination because of the limited channel bandwidth and heavy traffic loads. In order to improve the throughput of the RSUs, we devise a new MAC protocol Mizar from a cross-layer design perspective. Mizar concentrates on improving the spatio-temporal efficiency in disseminating data from an RSU to the moving vehicles. Leveraging the space diversity of wireless signals, Mizar increases the channel utilization by the cooperative transmission. Moreover, motivated by the location distribution characteristics of the vehicles, Mizar seeks to catch the concurrent transmission opportunities generated in the process of the cooperative communication. It is shown from the experimental results that Mizar can increase the throughput and decrease the transmission delay significantly in comparison with the fixed data rate scheme of IEEE 802.11p, the variable data rate scheme and the cooperative transmission scheme in Mizar.

VANETs inherit a conservative channel access control mechanism from IEEE 802.11 standard to mitigate the hidden terminal problem on the service channels, i.e., once a service channel is reserved by a node, all its neighbors in 2-hop range are prohibited from initiating the transmission simultaneously. To improve the channel utilization and increase the network capacity, we propose an optimistic distributed power control scheme, which relaxes the limitations on the 2-hop neighbor's data transmission and creates opportunities to start concurrent transmissions. The experimental results indicate that our scheme shows significant advantages over the fixed power scheme of IEEE 802.11p and Density-based Power Control (DBPC) [1] in terms of the average delay and throughput.