Transaction fraud detection (TFD) remains a substantial challenge in the digital economy. Feature extraction is crucial in TFD, as relying solely on raw transactional features fails to capture the complex and dynamic nature of fraudulent behavior. Existing methods primarily rely on manually aggregated features, failing to account for the evolving nature of fraud patterns. These limitations undermine the accuracy and adaptability of TFD systems. In this work, we propose a behavioral drift-aware feature extraction (BDFE) method designed to capture evolving fraud patterns by modeling behavioral drift and extracting new feature representations for TFD. BDFE consists of four synergistic modules. The first module is a feature extractor that encodes user transactional behaviors into new representations. The second and third modules are a historical behavior-aware classifier and a current behavior-aware classifier, which jointly guide the extractor to learn more discriminative features by capturing patterns from historical and current transactions, respectively. The fourth module is a behavior discriminator that distinguishes between historical and current behaviors. Through adversarial training, the discriminator encourages the feature extractor to extract unified representations that are invariant to behavioral shifts. This design enables BDFE to learn drift-resilient features, thus enhancing the detection of evolving fraud patterns. Extensive experiments are conducted on real-world transaction datasets and public ones. The results show that BDFE achieves improvements in average precision from 2.32% to 12.17%, in recall from 3.65% to 14.57%, in F1-score from 1.80% to 11.79%, and in G-mean from 2.05% to 8.17% over its peers. These gains demonstrate its superior feature extraction capability and its effectiveness in enhancing TFD performance.

In the digital era, effective Transaction Fraud Detection (TFD) is essential to ensuring financial security. The considerable class imbalance, with legitimate transactions vastly outnumbering fraudulent ones, presents a significant challenge for TFD models to accurately identify fraudulent patterns. While existing sample-balancing strategies address class imbalance effectively in many contexts, they often fall short in TFD due to fraudsters’ sophisticated concealment tactics, which lead to pronounced behavioral overlap between fraudulent and legitimate transactions. In this paper, we introduce a novel Generative Adversarial Network-based Hybrid Sampling method (GANHS) to effectively address the class imbalance issue. GANHS employs a dual-discriminator generative adversarial network to generate synthetic samples that accurately reflect the characteristics of fraudulent activity, while an adaptive neighborhood-based undersampling technique refines these samples to minimize overlap with legitimate ones. This hybrid approach not only enhances the model’s ability to learn fraud patterns by generating high-quality samples but also improves its resilience against highly concealed fraudulent activities. Experiments on real-world and public datasets demonstrate that GANHS outperforms its competitive peers, with gains of 0.5%–8.7% in average$F_{1}$-Score and 1.0%–7.0% in G-mean, highlighting its strong potential for improving the reliability and effectiveness of TFD systems in complex, high-risk financial scenarios.