Learning high-quality sentence embeddings from Natural Language Inference (NLI) data is often challenged by a critical signal conflict between discrete labels and the continuous spectrum of semantic similarity, as well as information loss from discarded neutral sentence pairs during training.To address this, we introduce Rank-Awareness and Angular Optimization Embeddings (RAOE), a framework that leverages the full NLI dataset (Entailment, Neutral, Contradiction) augmented with precomputed continuous similarity scores (S).RAOE employs a novel composite objective which features: (1) a Rank Margin objective that enforces rank consistency against S using an explicit margin, and (2) a Gated Angular objective that conditionally refines embedding geometry based on NLI label (L) and S score agreement.Extensive evaluations on STS tasks and the MTEB benchmark demonstrate RAOE's effectiveness.Our generalpurpose RAOE-S1 model (BERT-base) significantly outperforms strong baselines, achieving an average Spearman's correlation of 85.11 (vs.SimCSE's 81.57and AnglE's 82.43), and shows consistent improvements on MTEB.Further STS-specialized fine-tuning (RAOE-S2) establishes new state-of-the-art performance on STS (88.17 with BERT-base).These results confirm RAOE's ability to efficiently learn robust and nuanced sentence representations through the synergy of rankawareness and conditional angular constraints.

Fine-tuning is a widely used technique for leveraging pre-trained language models (PLMs) in downstream tasks, but it can be computationally expensive and storage-intensive. To address this challenge, researchers have developed parameter-efficient methods that balance performance and resource cost. However, these methods often come with trade-offs like increased inference latency, token length usage, or limited adaptability for multitasking scenarios. This paper introduces a novel parameter-efficient method called DimA(Dimensionality Augmentation), which enhances the Transformer architecture by increasing the dimensionality. DimA achieves state-of-the-art results in GLUE and XSUM tasks while utilizing less than 1% of the original model’s parameters. Moreover, DimA introduces a novel approach to knowledge transfer that enables the simultaneous utilization of knowledge learned from multiple tasks to handle new tasks. This method significantly enhances the performance of the model on new tasks. Its versatility in model structure also enables its application to various Transformer-based models.

Named Entity Recognition (NER) models perform well on standard benchmarks, but they often lack robustness when dealing with out-of-domain data. Recent studies have highlighted their limitations in genuine sentence comprehension, often due to their reliance on memorized entities or context, leading to spurious correlations. In this paper, we present Structural Causal Model-based techniques for detecting and addressing two types of spurious correlations in NER models: contextual and entity spurious correlations. Additionally, we introduce four data augmentation methods to improve training data and enable NER models to learn contextual and entity information effectively. Experimental results reveal that several NER models suffer from spurious correlations, and our approach mitigates them effectively. Furthermore, in comparison to the baseline methods, our approach demonstrates competitive performance on the in-domain dataset while achieving state-of-the-art results on the out-of-domain dataset.

Pretrained language models (PLMs) have made significant progress on various NLP tasks recently. However, PLMs encounter challenges when it comes to domain-specific tasks such as legal AI. These tasks often involve intricate expertise, expensive data annotation, and limited training data availability. To tackle this problem, we propose a human-oriented artificial–natural parallel system for organized intelligence (HANOI)-Legal based on the parallel learning (PL) framework. First, by regarding the description in PL as the pretraining process based on a large-scale corpus, we setup an artificial system based on a PLM. Second, to adapt the PLM to legal tasks with limited resources, we propose UniPrompt as a prescription. UniPrompt serves as a unified prompt-based training framework, enabling the utilization of diverse open datasets for these tasks. Third, we labeled a few task-specific legal data through distributed autonomous operations (DAO-II) for further fine-tuning. By combining a scalable unified-task-format reformulation and a unified-prompt-based training pipeline, HANOI-Legal leverages PLMs’ linguistic capabilities acquired from a variety of open datasets to generate task-specific models. Our experiments in two legal domain tasks show that HANOI-Legal achieved an excellent performance in low-resource scenarios compared to the state-of-the-art prompt-based approach.

Transformer with the self-attention mechanism, which allows fully-connected contextual encoding over input tokens, has achieved outstanding performances in various NLP tasks, but it suffers from quadratic complexity with the input sequence length. Long-range contexts are often tackled by Transformer in chunks using a sliding window to avoid GPU memory overflow. However, how to achieve considerable performance on downstream tasks under the premise of modeling sequences as long as possible with limited GPU resources is still a problem to be explored. To address this issue, we propose a new framework using hybrid attention-based Transformer to capture long-range contextual features. More specifically, we make a combination of three types of attention, i.e. sliding window local attention, clustering-based long-range attention and specific global attention. Experiments comparing the performance of our model with mainstream efficient improved Transformers are conducted on document classification task in public datasets IMDB and CAIL-Long. Experimental results demonstrate that the proposed approach can outperform the state-of-the-art models on the adopted datasets, which verifies the effectiveness of our model on long-range document classification task.

With the widespread application of mobile phones, it has become possible to study human mobility and travel behaviors based on cellular network data. Contrary to call detail records, the data is triggered by mobile cellular signaling and can provide fine-grained information about users’ daily routines. However, it does not explicitly provide semantic details about traveling traces, e.g., trip purposes. In this paper, we propose a methodological framework to handle large-scale cellular network data and discover the underlying trip purposes in an unsupervised way. We first devise heuristic rules to identify home/work purposes. Then, a flexible latent Dirichlet allocation (LDA) model is presented to discover the activities for remaining trips, in which each trip is depicted by four attributes, i.e. arrival time, age group, stay duration, and the point of interest tag for the destination. Experimental results show that the proposed method can identify diverse trip purposes by explaining their structures over trip attributes and outperform baselines in terms of log-likelihood and perplexity. We also analyze the difference between the automatically discovered trip purposes and those estimated from household census, and the analyzed results demonstrate the feasibility of our proposed method.

Face recognition methods have achieved high accuracy when faces are captured in frontal pose and constrained scenes. However, severe drop in accuracy is observed when large pose variations exist. The main reason is that the large yaw angle leads to ID information loss. In this paper, we intend to solve the large pose variations in a generation manner. Specifically, we propose a Pose-Weighted Generative Adversarial Network (PW-GAN) for photorealistic frontal view synthesis. We find frontalizing the faces in large poses (yaw angle larger than 60°) is so difficult that the results are not photorealistic and the ID information is lost. To simplify the problem, we first frontalize the face image through 3D face model, which is then used to guide the network predicting. Second, we refine the pose code in the loss function to make the network pay more attention to large poses. Quantitative and qualitative experimental results on the Multi-PIE and LFW demonstrate our method achieves state of the art.

Estimating age through a single facial image is a classic and challenging topic in computer vision. Since facial images of the same age vary considerably, while those from different ages may look very similar. To address these problems, we propose an end-to-end deep embedding neural network for robust age estimation. Specifically, we jointly use classification loss and triplet-based ranking loss to train a deep embedding network, which maps the input facial images into an embedding metric space where features of the same age are compact and those from different ages are pushed away. Thus the deep embedding network can learn more discriminative features and improves the performance for age estimation. Additionally, to accelerate the convergence of the network, we adopt an online hard negative mining strategy during the triplet loss computation. Experimental results on public datasets MORPH II and FG-NET show the superiority of our approach compared to the state-of-the-art.