This work explores efficient ultra-long context modeling.We posit that an effective solution requires three fundamental properties: sparsity, random-access flexibility, and length generalization.To achieve this, we leverage Hierarchical Sparse Attention (HSA), a novel attention mechanism that satisfies all three properties.We integrate HSA into the Transformer architecture to develop HSA-UltraLong, an 8B-parameter Mixture-of-Experts (MoE) model trained on over 8 trillion tokens.We rigorously evaluate the model across tasks with both in-domain and out-of-domain context lengths to validate its capabilities.Our model demonstrates comparable performance to full-attention baselines on in-domain sequence lengths.Crucially, it achieves over 90% accuracy on most in-context retrieval tasks with contexts up to 512 times the pre-training context length.This work reports our findings and remaining issues throughout the experiments, offering insights for future research in ultralong context modeling.

In recent years, online social networks (OSN) have rapidly gained popularity worldwide, becoming important platforms for information dissemination. Cyber manipulators use OSN bots to disseminate harmful information and manipulate public opinion, which can engage in cyber violence and conduct financial crimes. Therefore, it is crucial to propose an effective detection solution for OSN bots as a matter of urgency. Different OSN bots exhibit distinct behavioral patterns compared to regular users due to varying behavioral preferences. Analyzing network behavior patterns can reveal the fundamental rules and anomalies of OSN bots, providing support for effective detection in order to gather evidence of any illegal activities. Traditional social bot detection methods based on user profiles or social relationships pose risks of infringing on user privacy. Therefore, we propose a new detection framework for OSN bots——OBTT model, which demonstrates significant advantages in identifying bot traffic to OSN and discovering behavior patterns of different types of bots. OBTT adopts a multimodal approach, integrating graph embeddings from raw traffic with sequential features, while incorporating temporal information to explore the regularities in bot action sequences. Using large-scale unlabeled data, we pretrain a Masked Autoencoder (MAE) and fine-tune it with a small amount of labeled data to enhance the model capacity to detect various bot behavior patterns. Experiments conducted on our OSNBotTraffic5 dataset show that OBTT achieved an accuracy of 0.95, demonstrating excellent performance. Notably, this is the first time that different OSN bot behavior patterns have been identified in quasi-real time from the perspective of network traffic.