extended-abstract Share on International Workshop on Deep Learning Practice for High-Dimensional Sparse Data with RecSys 2023 Authors: Ruiming Tang Huawei Noah's Ark Lab, China Huawei Noah's Ark Lab, China 0000-0002-9224-2431View Profile , Xiaoqiang Zhu Mobvista Group, China Mobvista Group, China 0000-0001-7486-0853View Profile , Junfeng Ge Alibaba Group, China Alibaba Group, China 0000-0001-8435-0443View Profile , Kuang-chih Lee Alibaba Group, USA Alibaba Group, USA 0009-0007-5198-9866View Profile , Biye Jiang Alibaba Group, China Alibaba Group, China 0009-0001-5814-1581View Profile , Xingxing Wang Meituan, China Meituan, China 0000-0002-2655-3928View Profile , Han Zhu Alibaba Group, China Alibaba Group, China 0000-0002-9522-5637View Profile , Tao Zhuang Alibaba Group, China Alibaba Group, China 0000-0002-7408-8514View Profile , Weiwen Liu Huawei Noah's Ark Lab, China Huawei Noah's Ark Lab, China 0000-0002-9148-3997View Profile , Kan Ren Microsoft Research, China Microsoft Research, China 0000-0002-4032-9615View Profile , Weinan Zhang Shanghai Jiao Tong University, China Shanghai Jiao Tong University, China 0000-0002-0127-2425View Profile , Xiangyu Zhao City University of Hong Kong, China City University of Hong Kong, China 0000-0003-2926-4416View Profile Authors Info & Claims RecSys '23: Proceedings of the 17th ACM Conference on Recommender SystemsSeptember 2023Pages 1276–1280https://doi.org/10.1145/3604915.3608765Published:14 September 2023Publication History 0citation67DownloadsMetricsTotal Citations0Total Downloads67Last 12 Months67Last 6 weeks67 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Deep learning techniques have been applied widely in industrial recommendation systems. However, far less attention has been paid on the overfitting problem of models in recommendation systems, which, on the contrary, is recognized as a critical issue for deep neural networks. In the context of Click-Through Rate (CTR) prediction, we observe an interesting one-epoch overfitting problem: the model performance exhibits a dramatic degradation at the beginning of the second epoch. Such a phenomenon has been witnessed widely in real-world applications of CTR models. Thereby, the best performance is usually achieved by training with only one epoch. To understand the underlying factors behind the one-epoch phenomenon, we conduct extensive experiments on the production data set collected from the display advertising system of Alibaba. The results show that the model structure, the optimization algorithm with a fast convergence rate, and the feature sparsity are closely related to the one-epoch phenomenon. We also provide a likely hypothesis for explaining such a phenomenon and conduct a set of proof-of-concept experiments. We hope this work can shed light on the future research on training more epochs for better performance.

Recently, we have witnessed that deep learning-based approaches have been widely applied. Particularly, some applications involve data that are high dimensional, sparse or imbalanced, which are different from those applications with dense data processing, such as image classification and speech recognition, where deep learning-based approaches have been extensively studied. One of the main applications is the user-centric platform that consists of great deal of users, items and user generated tabular data which are quite high-dimensional. The characteristics of such data pose unique challenges to the adoption of deep learning in these applications, including modeling, training, and online serving, etc. More and more communities from both academia and industry have initiated the endeavors to solve these challenges. This workshop will provide a venue for both the research and engineering communities to discuss and formulate the challenges, utilize opportunities, and propose new ideas in the practice and theory of deep learning on high-dimensional, sparse and imbalanced data.

Recently, we have witnessed that deep learning-based approaches has been widely applied to empower many internet-scale applications. However, the data in these internet-scale applications are high dimensional and extremely sparse, which makes it different from those applications with dense data processing, such as image classification and speech recognition, where deep learning-based approaches have been extensively studied. One of the main applications is the user-centric platform that consists of great deal of users, items and user generated tabular data which are quite high-dimensional. The characteristics of such data pose unique challenges to the adoption of deep learning in these applications, including modeling, training, and online serving, etc. More and more communities from both academia and industry have initiated the endeavors to solve these challenges. This workshop will provide a venue for both the research and engineering communities to discuss and formulate the challenges, utilize opportunities, and propose new ideas in the practice of deep learning on high-dimensional sparse data.

Machine learning is growing in importance in industry, sciences, and many other fields. In many and perhaps most of these applications, users need to trade off competing goals. Machine learning, however, has evolved around the optimization of a single, usually narrowly-defined criterion. In most cases, an expert makes (or should be making) trade-offs between these criteria which requires high-level (human) intelligence. With interactive customization and optimization the expert can incorporate secondary criteria into the model-generation process in an interactive way. In this paper we develop the techniques to perform customized and interactive model optimization, and demonstrate the approach on several examples. The keys to our approach are (i) a machine learning architecture which is modular and supports primary and secondary loss functions, while users can directly manipulate its parameters during training (ii) high-performance training so that non-trivial models can be trained in real-time (using roofline design and GPU hardware), and (iii) highly-interactive visualization tools that support dynamic creation of visualizations and controls to match various optimization criteria.

Gibbs sampling is a workhorse for Bayesian inference but has several limitations when used for parameter estimation, and is often much slower than non-sampling inference methods. SAME (State Augmentation for Marginal Estimation) [15, 8] is an approach to MAP parameter estimation which gives improved parameter estimates over direct Gibbs sampling. SAME can be viewed as cooling the posterior parameter distribution and allows annealed search for the MAP parameters, often yielding very high quality estimates. But it does so at the expense of additional samples per iteration and generally slower performance. On the other hand, SAME dramatically increases the parallelism in the sampling schedule, and is an excellent match for modern (SIMD) hardware. In this paper we explore the application of SAME to graphical model inference on modern hardware. We show that combining SAME with factored sample representation (or approximation) gives throughput competitive with the fastest symbolic methods, but with potentially better quality. We describe experiments on Latent Dirichlet Allocation, achieving speeds similar to the fastest reported methods (online Variational Bayes) and lower cross-validated loss than other LDA implementations. The method is simple to implement and should be applicable to many other models.

Abstract Data analysts must make sense of increasingly large data sets, sometimes with billions or more records. We present methods for interactive visualization of big data, following the principle that perceptual and interactive scalability should be limited by the chosen resolution of the visualized data, not the number of records. We first describe a design space of scalable visual summaries that use data reduction methods (such as binned aggregation or sampling) to visualize a variety of data types. We then contribute methods for interactive querying (e.g., brushing & linking) among binned plots through a combination of multivariate data tiles and parallel query processing. We implement our techniques in imMens, a browser‐based visual analysis system that uses WebGL for data processing and rendering on the GPU. In benchmarks imMens sustains 50 frames‐per‐second brushing & linking among dozens of visualizations, with invariant performance on data sizes ranging from thousands to billions of records.