This work explores efficient ultra-long context modeling.We posit that an effective solution requires three fundamental properties: sparsity, random-access flexibility, and length generalization.To achieve this, we leverage Hierarchical Sparse Attention (HSA), a novel attention mechanism that satisfies all three properties.We integrate HSA into the Transformer architecture to develop HSA-UltraLong, an 8B-parameter Mixture-of-Experts (MoE) model trained on over 8 trillion tokens.We rigorously evaluate the model across tasks with both in-domain and out-of-domain context lengths to validate its capabilities.Our model demonstrates comparable performance to full-attention baselines on in-domain sequence lengths.Crucially, it achieves over 90% accuracy on most in-context retrieval tasks with contexts up to 512 times the pre-training context length.This work reports our findings and remaining issues throughout the experiments, offering insights for future research in ultralong context modeling.

The unique characteristics of the Internet of Things (IoT) make IoT protocols more vulnerable to security risks compared to traditional security protocols. Formal methods are widely used to analyze and verify the security properties of IoT protocols, but existing methods often fail to address the special security mechanisms required in the complex IoT network environment. A novel symbolic model based on event logic is proposed to implement the comprehensive analysis of secure IoT protocols. The model enhances the description of message flow and match, while also extending the inference capabilities of equivalence and injectivity. Aiming at the pre-shared key, cookie, raw public key, and certificate authority in the IoT protocols, a proof system of the model is proposed to prove the security properties of these mechanisms. To improve efficiency, the match buffer reduction method is also presented to eliminate the redundant proof steps. Formal analysis and verification of protocols, including DTLS, SRTP, CoAP, and PUF, uncover several attacks that violate security properties. The results reveal that the proposed symbolic model is an effective method for the formal analysis and proof of IoT protocols.