Large Language Models (LLMs) have demonstrated efficacy in various domains, but deploying these models is economically challenging due to extensive parameter counts. Numerous efforts have been dedicated to reducing the parameter count of these models without compromising performance, employing a technique known as model pruning. Conventional pruning methods assess the significance of weights within individual layers and typically apply uniform sparsity levels across all layers, potentially neglecting the varying significance of each layer. To address this oversight, we first propose a dual-assessment driven pruning strategy that employs both intra-layer metric and global performance metric to comprehensively evaluate the impact of pruning. Then our method leverages an iterative optimization algorithm to find the optimal layer-wise sparsity distribution, thereby minimally impacting model performance. Extensive benchmark evaluations on state-of-the-art LLM architectures such as LLaMAv2 and OPT across a variety of NLP tasks demonstrate the effectiveness of our approach. When applied to the LLaMaV2-7B model with an overall pruning sparsity of 80%, our method achieves a 50% reduction in perplexity compared to the benchmark. The results indicate that our method significantly outperforms existing state-of-the-art methods in preserving performance after pruning.

Sequential recommendation has now been more widely studied, characterized by its well-consistency with real-world recommendation situations. Most existing works model user preference as the transition pattern from the previous item to the next, ignoring the time interval between these two items. However, we find that the time intervals in different sequences may vary significantly and thus result in the ineffectiveness of user modeling due to the issue ofpreference drift. Thus we propose an assumption that a sequence with uniformly distributed time intervals (denoted as uniform sequence) is more beneficial for preference learning than that with greatly varying time intervals. We then conduct an empirical study on four real datasets and the results support this assumption. Therefore, we advocate to augment sequence data from the perspective of time intervals, which is not studied in the literature. Specifically, we design five operators (Ti-Crop, Ti-CateReorder, Ti-Mask, Ti-Substitute, Ti-Insert) to transform the original non-uniform sequence to uniform sequence with the consideration of time intervals. Then, we devise a control strategy to execute data augmentation on item sequences in different lengths and a looseness range to ensure the generalization (or diversity) of generated data. Finally, we implement these improvements on a state-of-the-art model CoSeRec and proposeTimeInterval AwareCoSeRec(TiCoSeRec). Experimental results on four datasets demonstrate that TiCoSeRec achieves significantly better performance than other 11 counterparts recommendation techniques.

Sequential recommendation is an important task to predict the next-item to access based on a sequence of interacted items. Most existing works learn user preference as the transition pattern from the previous item to the next one, ignoring the time interval between these two items. However, we observe that the time interval in a sequence may vary significantly different, and thus result in the ineffectiveness of user modeling due to the issue of preference drift. In fact, we conducted an empirical study to validate this observation, and found that a sequence with uniformly distributed time interval (denoted as uniform sequence) is more beneficial for performance improvement than that with greatly varying time interval. Therefore, we propose to augment sequence data from the perspective of time interval, which is not studied in the literature. Specifically, we design five operators (Ti-Crop, Ti-Reorder, Ti-Mask, Ti-Substitute, Ti-Insert) to transform the original non-uniform sequence to uniform sequence with the consideration of variance of time intervals. Then, we devise a control strategy to execute data augmentation on item sequences in different lengths. Finally, we implement these improvements on a state-of-the-art model CoSeRec and validate our approach on four real datasets. The experimental results show that our approach reaches significantly better performance than the other 9 competing methods. Our implementation is available: https://github.com/KingGugu/TiCoSeRec.

When doing private domain marketing with cloud services, the merchants usually have to purchase different machine learning models for the multiple marketing purposes, leading to a very high cost. We present a unified user-item matching framework to simultaneously conduct item recommendation and user targeting with just one model. We empirically demonstrate that the above concurrent modeling is viable via modeling the user-item interaction matrix with the multinomial distribution, and propose a bidirectional bias-corrected NCE loss for the implementation. The proposed loss function guides the model to learn the user-item joint probability p(u,i) instead of the conditional probability p(i|u) or p(u|i) through correcting both the users and items’ biases caused by the in-batch negative sampling. In addition, our framework is model-agnostic enabling a flexible adaptation of different model architectures. Extensive experiments demonstrate that our framework results in significant performance gains in comparison with the state-of-the-art methods, with greatly reduced cost on computing resources and daily maintenance.

User representation is essential for providing high-quality commercial services in industry. In our business scenarios, we face the challenge of learning universal (general-purpose) user representation. The universal representation is expected to be informative, and can handle various types of real-world applications without fine-tuning (e.g., applicable for both user profiling and the recall process in advertising). It shows great advantages compared to the solution of training a specific model for each downstream application. Specifically, we attempt to improve universal user representation from two points of views. First, a contrastive self-supervised learning paradigm is presented to guide the representation model training. It provides a unified framework that allows for long-term or short-term interest representation learning in a data-driven manner. Moreover, a novel multi-interest extraction module is presented. The module introduces an interest dictionary to capture principal interests of the given user, and then generate his/her interest-oriented representations via behavior aggregation. Experimental results demonstrate the effectiveness and applicability of the learned user representations. Such an industrial solution has now been deployed in various real-world tasks.

User modeling is critical for developing personalized services in industry. A common way for user modeling is to learn user representations that can be distinguished by their interests or preferences. In this work, we focus on developing universal user representation model. The obtained universal representations are expected to contain rich information, and be applicable to various downstream applications without further modifications (e.g., user preference prediction and user profiling). Accordingly, we can be free from the heavy work of training task-specific models for every downstream task as in previous works. In specific, we propose Self-supervised User Modeling Network (SUMN) to encode behavior data into the universal representation. It includes two key components. The first one is a new learning objective, which guides the model to fully identify and preserve valuable user information under a self-supervised learning framework. The other one is a multi-hop aggregation layer, which benefits the model capacity in aggregating diverse behaviors. Extensive experiments on benchmark datasets show that our approach can outperform state-of-the-art unsupervised representation methods, and even compete with supervised ones.

Precise destination prediction from partial trajectories have a huge potential impact on intelligent location-based approaches. Traditional prediction approaches, which treat trajectories as one-dimensional sequences and process them in a single scale, fail to capture diverse and rich two-dimensional patterns of trajectories in different spatial scales. Meanwhile, most models treat each portion of a trajectory equally in terms of contributing to final destination prediction. This is in conflict with our observation that there exists some albeit small local areas playing much more important roles for destination prediction than the others. To address these problems, we propose a novel prediction algorithmT-CONV, which models trajectories as two-dimensional images, and then feed them into a convolutional neural network (CNN) architecture to extract multi-scale patterns for precise destination prediction. Furthermore, we propose a method to extract regions with different relevance for final output ofT-CONV, and further explore the local patterns of important regions by integrating multi-scope local-enhancement areas based on attention mechanism. The comprehensive experiments based on two large-scale real taxi trajectory datasets show thatT-CONVcan achieve higher accuracy than the state-of-the-art methods, demonstrating the strength of the multi-scale and multi-scope feature extraction mechanisms in trajectory mining.

Data distribution of GIS has been realized through WebGIS. With the increasing need of data sharing and GIS, WebGIS can no longer meet this need. The applications of grid computing and middleware technology in GIS lay a foundation for the development of middleware grid GIS. Grid GIS not only makes spatial data distribution, also makes the functional services of GIS distribution. This paper focuses on the functional architecture of grid GIS, and brings forward a functional model of grid GIS. GML and middleware are also important factors in the development of grid GIS, GML is the message carrier in the grid of Web.

Remote sensing images have many advantages, such as containing abundant information and acquiring quickly and timely. In order to service for economy and society using remote sensing images, with which we integrate publishing. A minute description is given of making remote sensing images atlas of Jiangyin city, Jiangsu province of east China. This paper then analyses anticipations of economic and society benefit of the atlas. We can draw a conclusion that potential huge market exists in applying remote sensing images to publishing.