This paper addresses the important and challenging task of large-scale unsupervised semantic segmentation (LUSS). We present the first attempt to unleash the power of foundation models (FMs) for the challenging, dense prediction task LUSS, and our main objective is to present simple, effective yet efficient solutions for LUSS, namely Prompting foundation models for LUSS (PLUSS). Firstly, we proposed a cascade framework PLUSS$_\alpha$α by effectively marrying CLIPS, Grounding DINO, and SAM in a zero-shot manner. This cascade architecture automatically generates semantic and spatial prompts for SAM, establishing a strong baseline that significantly outperforms previous state-of-the-art methods. Building upon this foundation, we propose PLUSS$_\beta$β, which addresses the critical bottleneck of prompt quality through two novel tuner modules: a semantic tuner that enhances fine-grained category discrimination via visual prompt tuning, and a box tuner that improves object localization through cross-modal feature fusion. Both tuners are optimized by capitalizing on the knowledge already present within the foundation models themselves, deriving self-supervised signals from internal model consistency. This approach requires no external supervision or updates to the foundation models' parameters. Extensive experiments on ImageNet-S benchmarks demonstrate that PLUSS$_\beta$β achieves remarkable performance improvements, surpassing the previous best method by 39.6%, 27.3%, and 22.6% in mIoU for 50, 300, and 919 categories respectively. Our approach exhibits robust category-shape representation across varying object sizes and dataset scales, while maintaining strong generalization capabilities for open-vocabulary tasks. The proposed framework provides a solid baseline for adapting foundation models to downstream vision tasks.

Existing cross-domain few-shot learning (CDFSL) methods, which develop training strategies in the source domain to enhance model transferability, face challenges when applied to large-scale pre-trained models (LMs), as their source domains and training strategies are not accessible. Besides, fine-tuning LMs specifically for CDFSL requires substantial computational resources, which limits their practicality. Therefore, this paper investigates the source-free CDFSL (SF-CDFSL) problem to solve the few-shot learning (FSL) task in target domain using only a pre-trained model and a few target samples, without requiring source data or training strategies. However, the inaccessibility of source data prevents explicitly reducing the domain gaps between the source and target. To tackle this challenge, this paper proposes a novel approach, Step-wise Distribution-aligned Style Prompt Tuning (StepSPT), to implicitly narrow the domain gaps from the perspective of prediction distribution optimization. StepSPT initially proposes a style prompt that adjusts the target samples to mirror the expected distribution. Furthermore, StepSPT tunes the style prompt and classifier by exploring a dual-phase optimization process (external and internal processes). In the external process, a step-wise distribution alignment strategy is introduced to tune the proposed style prompt by factorizing the prediction distribution optimization problem into the multi-step distribution alignment problem. In the internal process, the classifier is updated via standard cross-entropy loss. Evaluation on 5 datasets illustrates the superiority of StepSPT over existing prompt tuning-based methods and state-of-the-art methods (SOTAs). Furthermore, ablation studies and performance analyzes highlight the efficacy of StepSPT.

Few-shot semantic segmentation (FSS) aims to segment objects of novel categories in the query images given only a few annotated support samples. Existing methods primarily build the image-level correlation between the support target object and the entire query image. However, this correlation contains the hard pixel noise, \textit{i.e.}, irrelevant background objects, that is intractable to trace and suppress, leading to the overfitting of the background. To address the limitation of this correlation, we imitate the biological vision process to identify novel objects in the object-level information. Target identification in the general objects is more valid than in the entire image, especially in the low-data regime. Inspired by this, we design an Object-level Correlation Network (OCNet) by establishing the object-level correlation between the support target object and query general objects, which is mainly composed of the General Object Mining Module (GOMM) and Correlation Construction Module (CCM). Specifically, GOMM constructs the query general object feature by learning saliency and high-level similarity cues, where the general objects include the irrelevant background objects and the target foreground object. Then, CCM establishes the object-level correlation by allocating the target prototypes to match the general object feature. The generated object-level correlation can mine the query target feature and suppress the hard pixel noise for the final prediction. Extensive experiments on PASCAL-${5}^{i}$ and COCO-${20}^{i}$ show that our model achieves the state-of-the-art performance.

While recent debiasing methods for Scene Graph Generation (SGG) have shown impressive performance, these efforts often attribute model bias solely to the long-tail distribution of relationships, overlooking the more profound causes stemming from skewed object and object pair distributions. In this paper, we employ causal inference techniques to model the causality among these observed skewed distributions. Our insight lies in the ability of causal inference to capture the unobservable causal effects between complex distributions, which is crucial for tracing the roots of model bias. Specifically, we introduce the Mediator-based Causal Chain Model (MCCM), which, in addition to modeling causality among objects, object pairs, and relationships, incorporates mediator variables, i.e., cooccurrence distribution, for complementing the causality. Following this, we propose the Causal Adjustment Module (CAModule) to estimate the modeled causal structure, using variables from MCCM as inputs to produce a set of adjustment factors aimed at correcting biased model predictions. Moreover, our method enables the composition of zero-shot relationships, thereby enhancing the model's ability to recognize such relationships. Experiments conducted across various SGG backbones and popular benchmarks demonstrate that CAModule achieves state-of-the-art mean recall rates, with significant improvements also observed on the challenging zero-shot recall rate metric.

Existing two-stage Scene Graph Generation (SGG) frameworks typically incorporate a detector to extract relationship features and a classifier to categorize these relationships; therefore, the training paradigm follows a causal chain structure, where the detector's inputs determine the classifier's inputs, which in turn influence the final predictions. However, such a causal chain structure can yield spurious correlations between the detector's inputs and the final predictions, i.e., the prediction of a certain relationship may be influenced by other relationships. This influence can induce at least two observable biases: tail relationships are predicted as head ones, and foreground relationships are predicted as background ones; notably, the latter bias is seldom discussed in the literature. To address this issue, we propose reconstructing the causal chain structure into a reverse causal structure, wherein the classifier's inputs are treated as the confounder, and both the detector's inputs and the final predictions are viewed as causal variables. Specifically, we term the reconstructed causal paradigm as the Reverse causal Framework for SGG (RcSGG). RcSGG initially employs the proposed Active Reverse Estimation (ARE) to intervene on the confounder to estimate the reverse causality, i.e., the causality from final predictions to the classifier's inputs. Then, the Maximum Information Sampling (MIS) is suggested to enhance the reverse causality estimation further by considering the relationship information. Theoretically, RcSGG can mitigate the spurious correlations inherent in the SGG framework, subsequently eliminating the induced biases. Comprehensive experiments on popular benchmarks and diverse SGG frameworks show the state-of-the-art mean recall rate.

State-of-the-art Active Learning (AL) methods often encounter challenges associated with a hysteretic learning process and an expensive data sampling mechanism. The former implies that data selection in the ($i+1$)-th round is solely based on the learned model’s results in the$i$-th round. The latter involves using model inference to calculate data value (e.g., uncertainty estimation based on model inference), which can be cumbersome, particularly when working with large datasets or Deep Neural Networks (DNNs). To address these challenges, we propose FastAL, an efficient and dynamic deep AL framework. Our approach includes an efficient method for calculating data value from the frequency domain perspective, generating multiple candidates. Then, we introduce the Fast Evaluation Module, which directly calculates each candidate’s contribution to future model training and selects the best options. In addition, current AL methods, particularly those based on uncertainty, are susceptible to data bias, which implies that selected data may not represent the original unlabeled data adequately. To alleviate this issue, we propose the De-similar Module, which removes partially similar data. The above three modules are model-agnostic and thus can be seamlessly integrated into any Active Learning framework. We conducted rigorous experiments on various benchmark datasets to validate our approach’s effectiveness. Our results demonstrate that FastAL outperforms other state-of-the-art methods by a significant margin, including those based on uncertainty, diversity, and expected model change.

Active learning (AL) aims to sample the most valuable data for model improvement from the unlabeled pool. Traditional works, especially uncertainty-based methods, are prone to suffer from a data bias issue, which means that selected data cannot cover the entire unlabeled pool well. Although there have been lots of literature works focusing on this issue recently, they mainly benefit from the huge additional training costs and the artificially designed complex loss. The latter causes these methods to be redesigned when facing new models or tasks, which is very time-consuming and laborious. This article proposes a feature-matching-based uncertainty that resamples selected uncertainty data by feature matching, thus removing similar data to alleviate the data bias issue. To ensure that our proposed method does not introduce a lot of additional costs, we specially design a unsupervised fusion feature matching (UFFM), which does not require any training in our novel AL framework. Besides, we also redesign several classic uncertainty methods to be applied to more complex visual tasks. We conduct rigorous experiments on lots of standard benchmark datasets to validate our work. The experimental results show that our UFFM is better than the similar unsupervised feature matching technologies, and our proposed uncertainty calculation method outperforms random sampling, classic uncertainty approaches, and recent state-of-the-art (SOTA) uncertainty approaches.

Personally identifiable information (PII) refers to any information that links to an individual. Sharing PII is extremely useful in public affairs yet hard to implement due to the worries about privacy violations. Building a PII retrieval service over multi-cloud, which is a modern strategy to make services stable where multiple servers are deployed, seems to be a promising solution. However, three major technical challenges remain to be solved. The first is the privacy and access control of PII. In fact, each entry in PII can be shared to different users with different access rights. Hence, flexible and fine-grained access control is needed. Second, a reliable user revocation mechanism is required to ensure that users can be revoked efficiently, even if few cloud servers are compromised or collapse, to avoid data leakage. Third, verifying the correctness of received PII and locating a misbehaved server when wrong data are returned is crucial to guarantee user's privacy, but challenging to realize. In this paper, we propose Rainbow, a secure and practical PII retrieval scheme to solve the above issues. In particular, we design an important cryptographic tool, called Reliable Outsourced Attribute Based Encryption (ROABE) which provides data privacy, flexible and fine-grained access control, reliable immediate user revocation and verification for multiple servers simultaneously, to support Rainbow. Moreover, we present how to build Rainbow with ROABE and several necessary cloud techniques in real world. To evaluate the performance, we deploy Rainbow on multiple mainstream clouds, namely, AWS, GCP and Microsoft Azure, and experiment in browsers on mobile phones and computers. Both theoretical analysis and experimental results indicate that Rainbow is secure and practical.

Despite the impressive performance of recent unbiased Scene Graph Generation (SGG) methods, the current debiasing literature mainly focuses on the long-tailed distribution problem, whereas it overlooks another source of bias, i.e., semantic confusion, which makes the SGG model prone to yield false predictions for similar relationships. In this paper, we explore a debiasing procedure for the SGG task leveraging causal inference. Our central insight is that the Sparse Mechanism Shift (SMS) in causality allows independent intervention on multiple biases, thereby potentially preserving head category performance while pursuing the prediction of high-informative tail relationships. However, the noisy datasets lead to unobserved confounders for the SGG task, and thus the constructed causal models are always causal-insufficient to benefit from SMS. To remedy this, we propose Two-stage Causal Modeling (TsCM) for the SGG task, which takes the long-tailed distribution and semantic confusion as confounders to the Structural Causal Model (SCM) and then decouples the causal intervention into two stages. The first stage is causal representation learning, where we use a novel Population Loss (P-Loss) to intervene in the semantic confusion confounder. The second stage introduces the Adaptive Logit Adjustment (AL-Adjustment) to eliminate the long-tailed distribution confounder to complete causal calibration learning. These two stages are model agnostic and thus can be used in any SGG model that seeks unbiased predictions. Comprehensive experiments conducted on the popular SGG backbones and benchmarks show that our TsCM can achieve state-of-the-art performance in terms of mean recall rate. Furthermore, TsCM can maintain a higher recall rate than other debiasing methods, which indicates that our method can achieve a better tradeoff between head and tail relationships.

Recent transformer-based models, especially patch-based methods, have shown huge potentiality in vision tasks. However, the split fixed-size patches divide the input features into the same size patches, which ignores the fact that vision elements are often various and thus may destroy the semantic information. Also, the vanilla patch-based transformer cannot guarantee the information communication between patches, which will prevent the extraction of attention information with a global view. To circumvent those problems, we propose an Efficient Attention Pyramid Transformer (EAPT). Specifically, we first propose the Deformable Attention, which learns an offset for each position in patches. Thus, even with split fixed-size patches, our method can still obtain non-fixed attention information that can cover various vision elements. Then, we design the Encode-Decode Communication module (En-DeC module), which can obtain communication information among all patches to get more complete global attention information. Finally, we propose a position encoding specifically for vision transformers, which can be used for patches of any dimension and any length. Extensive experiments on the vision tasks of image classification, object detection, and semantic segmentation demonstrate the effectiveness of our proposed model. Furthermore, we also conduct rigorous ablation studies to evaluate the key components of the proposed structure.

With more and more data moving to the cloud, privacy of user data have raised great concerns. Client-side encryption/decryption seems to be an attractive solution to protect data security, however, the existing solutions encountered three major challenges: low security due to encryption with low-entropy PIN, inconvenient data sharing with traditional encryption algorithms, and poor usability with dedicated software/plugins that require certain types of terminals. This work designs and implements WebCloud, a practical browser-side encryption solution, leveraging modern Web technologies. It solves all the above three problems while achieves several additional remarkable features: robust and immediate user revocation, fast data processing with offline encryption and outsourced decryption. Notably, our solution works on any device equipped with a Web user agent, including Web browsers, mobile and PC applications. We implement WebCloud based on ownCloud for basic file management utility, and utilize WebAssembly and Web Cryptography API for complex cryptographic operations integration. Finally, comprehensive experiments are conducted with many well-known browsers, Android and PC applications, which indicates that WebCloud is cross-platform and efficient. As an interesting by-product, the design of WebCloud naturally embodies a dedicated and practical ciphertext-policy attribute-based key encapsulation mechanism (CP-AB-KEM) scheme, which can be useful in other applications.

As an innovative technique for cloud storage services, attribute based encryption (ABE) enables fine-grained access control over encrypted data in many cloud computing applications. However, there exist two main drawbacks that restrict the development of ABE. One is that most of the existing user revocation mechanisms cannot achieve high efficiency, immediacy and robustness simultaneously. Another is that the decryption of ABE contains expensive pairing operations which often grow with the complexity of access policy. In this work, we propose a practical server-aided revocable fine-grained access control mechanism with the help of cloud’s storage, computing and management capabilities, which not only achieves efficient fine-grained attribute based access control, but also actualizes immediate and robust user revocation. Moreover, most of the complicated operations in decryption are outsourced to the public cloud server, leaving one exponentiation for the users. At last, we implement our proposed mechanism with$\mathsf{Charm}$Charmframework. The benchmark results demonstrate the high efficiency and practicality of our proposed mechanism.

Abstract As the only approved Identity-Based Encryption scheme in China that is also standardized by ISO, SM9-IBE has been widely adopted in many real-world applications. However, similar to other IBE standard algorithms, SM9-IBE currently lacks revocation mechanism, which is vital for a real system. Worse still, we find that existing revocable techniques may not be suitable and efficient when applying to SM9-IBE. Given the widespread use of SM9-IBE, an efficient and robust user revocation mechanism becomes an urgent issue.In this work, we propose a dedicated server-aided revocation mechanism, which for the first time achieves the secure, immediate and robust user revocation for SM9-IBE. Provided with a compact system model, the proposed method leverages an existing server to perform all heavy workloads during user revocation, thus leaving no communication and computation costs for the key generation center and users. Moreover, the mechanism supports key-exposure resistance, meaning the user revocation mechanism is robust even if the revocation key leaks. We then formally define and prove the security. At last, we present theoretical comparisons and an implementation in terms of computational latency and throughput. The results indicate the efficiency and practicability of the proposed mechanism.

SPHINCS was recently proposed as a stateless, quantum-resilient hash-based signature scheme. However, one possible limitation of SPHINCS is its signing speed, namely, the best known implementation merely produces a few hundred of signatures per second, which is not good enough, e.g., for a social website with a huge amount of users. Aiming at improving the singing throughput, we present highly parallel and optimized implementations of SPHINCS, which can be deployed on various multi-core platforms. As a first step, we give an elementary implementation on ×86/64 processors, which proves the effectiveness and correctness of our implementations. To obtain a significantly higherthroughput, we implement SPHINCS on Graphics Processing Units (GPUs). Furthermore, we develop a few general and hardware-specific techniques to take full advantage of the computing power of targeted platforms. We instantiate the underlying hash functions with three primitives. Our comprehensive benchmark shows that our work outperforms all the state-of-the-art implementations of SPHINCS regarding throughput with reasonable latency, and has scalability on multiple cores and multiple GPU cards. For instance, forthe key generation algorithm instantiated with ChaCha running on a GeForce GTX 1080, we obtain 5152 signatures per second which is 7.88x speedup fasterthan a recent FPGA implementation. When upgrade to TITAN Xp, 6,651 signatures are generated in one second. With four TITAN Xp GPUs, the obtained throughput satisfies vast majority scenarios.

Authenticated key exchange (AKE) is one of the most important applications in applied cryptography, where a user interacts with a server to set up a session key where pre-registered information (aka. authentication factor), such as a password or biometrics, of the user is stored. While single-factor AKE is widely used in practice, higher security concerns call for multi-factor AKE (MFAKE) schemes, e.g., combining both passwords and biometrics simultaneously. However, in some casually designed schemes, security is even weakened in the sense that leakage of one authentication factor will defeat the whole MFAKE protocol. Furthermore, an inevitable by-product arise that the usability of the protocol often drop greatly. To summarize, the existing multi-factor protocols did not provide enough security and efficiency simultaneously. In this paper, we make one step ahead by proposing a very efficient MFAKE protocol. We define the security model and give the according security analysis. We also implement our protocol on a smartphone and a cloud server. The theoretic comparisons and the experimental results show that our scheme achieves both security and usability.